finish the questions under the document Assessment Instrument Overview Activity(4) (1)
xiabuantjngo1
measurement_reliability.pptxMeasurement Reliability
Unit 6
Assessment Instruments Measurement Reliability
Assessment instruments help us identify concerns or problems
when conducting a needs assessment. They give us a snapshot of
where the company stands relative to the concept being assessed.
So, for example, we could use an assessment instrument to
determine how loyal employees feel. Or we could use an
assessment to see how effective managers are in communicating
with staff.
Regardless of the specifics, the process works the same.
Assessment Instruments Measurement Reliability
When we use assessment instruments we are collecting data.
When the data are quantitative (i.e., numerical) we are able to
check on how well our instrument has performed.
The idea of determining how well an instrument has performed is
known as measurement reliability.
Reliability Coefficient Measurement Reliability
It is important to understand and realize that assessment
instruments are forms of measurement and that as such their
performance may vary. That is, they could work well in some
settings and with some groups and not so well in others.
Thus, it is helpful to determine how well they have performed by
checking on the reliability of the assessment instrument. This can
be done by computing what is known as a reliability coefficient.
The reliability coefficient is a statistic that we depend on to
determine how reliably our measure, or in this case assessment
instrument, has performed.
Reliability Coefficient Measurement Reliability
The reliability statistic can be called by any of the following names:
the reliability coefficient
the coefficient alpha
or Cronbach’s alpha
At times it is simply represented by the Greek symbol for alpha, which is α.
We may see it reported in any of the following ways:
The Organizational Dissent Scale had a reliability coefficient of .86.
Cronbach’s alpha for the Leader Accountability Scale was .96.
Workplace diversity was measured with the Workplace Diversity Scale (α =.65).
Reliability Coefficient Measurement Reliability
Regardless of what it is called or how it appears in a given
sentence, the reliability coefficient is always a numerical value
that ranges from .00 to 1.00.
Whereby higher scores indicate that the measure has performed
well and lower scores indicate that it has performed poorly.
As a rule, values above .70 are considered reliable, whereas
those below — particularly those far below .70 — are considered
unreliable. Thus, a reliability coefficient of .85 is considered
reliable whereas a reliability coefficient of .57 is considered
unreliable.
Reliability Coefficient Measurement Reliability
But what does it mean when we say an instrument is reliable?
With regard to T&D, primarily two things:
That an assessment instrument has little measurement error.
That all the items (remember that most instruments are made up of multiple items) in the instrument are working well together.
Measurement Error Measurement Reliability
Measurement error comes from two places: random error and
measurement error.
Random error is error that is due to uncontrollable aspects of the measurement administration such as fatigue in respondents, lack of attentiveness, etc.
This is not something the person collecting the data has much control over. If respondents are tired, antsy, irritated by something that just happened in the office, etc. that could affect how they respond to an assessment instrument. Unfortunately, there is little we can do about it.
Measurement Error Measurement Reliability
In contrast, people conducting assessments do have control over
the types of assessment instruments they use and how well
crafted they are. The old adage garbage in-garbage out is relevant
here. Better designed instruments have less error and perform
well compared to those that are poorly designed.
Measurement error, then, is the amount of error in the measurement instrument.
The trainer conducting the needs assessment has control over this
type of error. Careful design of measurement instruments and
techniques serves to reduce or minimize measurement error.
Measurement Error Measurement Reliability
So, the reliability coefficient tells us how much error there is in
an assessment instrument.
Now, again looking at the examples from previously, we can
determine that certain measures had more error than others.
The Organizational Dissent Scale (α =.86)
Leader Accountability Scale (α =.96)
Workplace Diversity Scale (α =.65).
In this case the Workplace Diversity Scale had more error
than either of the other two instruments. In contrast, it appears
that the Leader Accountability Scale had very little error.
Consistency of Items Measurement Reliability
To measure reliably all of the items included in an assessment instrument need
to be working well together. Thus, the numerical value of a reliability
coefficient also can serve as an indicator of how closely items are working
together.
Higher reliability values suggest that items are working well together, that they
are in fact all getting at or measuring something similar. A reliability
coefficient, for example of .93 for a 10-item instrument indicates that all 10
items are getting at or measuring the same concepts/idea.
Contrastingly, lower reliability values (e.g., .48) suggest that some of the items
are probably measuring a different concept and therefore are working against
the others in the set of items comprising the instrument.
Consistency of Items Measurement Reliability
When we have a low reliability it is permissible to run a reliability analysis. This is a
statistical procedure whereby we can ask the reliability statistic to be produced for the
entire scale but also for the scale minus each of the individual items.
So, for example, say a 5-item instrument produced a coefficient alpha of .56. The
reliability analysis would show us…
Overall Coefficient Reliability .56
Alpha if Item 1 Deleted .57
Alpha if Item 2 Deleted .58
Alpha if Item 3 Deleted .55
Alpha if Item 4 Deleted .82
Alpha if Item 5 Deleted .54
We can see by examining this table that the coefficient alpha would change little if we
dropped some of the items, moving only a fraction up or down. But it would jump
substantially if we removed Item 4 — increasing to .82 and going from something
unacceptable to something quite strong.
Dimensionality Measurement Reliability
A final consideration when thinking about measurement reliability is the
dimensionality of the assessment instrument. That is, is the instrument
unidimensional (only measuring one concept) or multidimensional
(measuring several concepts).
If it is multidimensional we have to be careful to treat the dimensions separate,
as different measures even though they are part of the same assessment
instrument. So, for example the Organizational Dissent Scale is
multidimensional. Overall it measures organizational dissent, but does so
along three dimensions (articulated, latent, and displaced dissent).
Thus, when we have a multidimensional instrument we must compute and
check the reliability for each dimension separately.
