ADD5106- Week 2 Discussion 2: Reliability and Validity

CurrentStandardsforReliabilityChapter5.pdf

Home >Human Resource Management homework help >ADD5106- Week 2 Discussion 2: Reliability and Validity

Objectives 1. Identify different types of reliability. 2. Define key terms related to reliability. 3. Understand the role of reliability in determining the usefulness of assessment

results. 4. Interpret reliability estimates and apply them to case conceptualization.

Current Standards for Reliability

Defining Reliability Inherent in the practice of assessment is the repeated administration of an instrument, either to a single individual across time or across multiple individuals. An example of the former is a counselor who may wish to use the Beck Depression Inventory- II (BDI- II) to evaluate whether clients have experienced a reduction in depressive symptoms after a period of time. As an example of assessing multiple individuals, the expectation is that individuals with similar characteristics, such as adults diagnosed with depression and par- ticipating in group counseling, will demonstrate similar scoring patterns. Thus, reliability is the consistency of scores on a measure (American Educational Research Association [AERA], American Psychological Association, & National Council of Measurement in Education, 2014; Cohen et al., 2013). In these examples, consistency refers to stability over time and across populations.

When dealing with objective measures, such as an individual’s weight, an indi- vidual can step on a scale, read the weight, step back on, and more than likely read the same weight. There is very little variance on the weight. Individuals of similar build will likely have similar weights, so the scale is a reliable measure of weight. However, the

Balkin, Richard S., and Gerald A. Juhnke. Assessment in Counseling : Practice and Applications, Oxford University Press, Incorporated, 2018. ProQuest Ebook Central, http://ebookcentral.proquest.com/lib/capella/detail.action?docID=5179769. Created from capella on 2023-10-17 21:18:57.

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

current standards for rel iab il ity | 89

same cannot be said for measuring constructs. Not every phenomenon of interest can be directly observed. Constructs are theoretically guided phenomena that cannot be directly observed or measured. Intelligence and various mood states (e.g., depression, happiness, stress, antisocial personality) serve as constructs in the counseling profession. For example, an athlete may be labeled as brilliant for an ability to think quickly and react in creative ways, but such behavior may not translate into high scores on an intelligence test. Yet, such behavior is an act of creativity that could be recognized as a measure of intelligence. Therefore, constructs are limited by an operational definition, a method of explaining and limiting how a construct will be measured. Referring to the previous example of using the BDI- II, clients completing the instruments are evaluated for depression partially based on how each symptom is defined on the BDI- II. Other instruments, such as the Hamilton Rating Scale for Depression or the Suicide Probability Scale, also evaluate depression, but they do so differently. Although correlations may exist between or among the instruments, each instrument employs a separate operational definition based on having different items evaluating depression.

Reliability is a term extended from classical test theory (CTT), derived from Spearman (1904), as well as other more contemporary contributors (e.g., Guttman, Likert, Lord, Thurstone). The premise of CTT was based on two postulates:

1. the measurement of attributes of an individual that contribute to a consistent response set.

2. the measurement of attributes unrelated to the construct being measured but that affect the test scores (Gregory, 2014).

So although the first postulate relates to attributes of the construct, the second postulate is a reflection of extraneous factors that contribute to measurement error.

Although CTT is the more popular theory related to assessment and measurement, other theories are present, particularly item response theory (IRT), also known as latent- trait theory. IRT addresses the extent to which an item measures a particular trait (Cohen et al., 2013).

One important aspect of reliability is that reliability is a function of scores, not the scale. In other words, an instrument is never reliable. Rather, scores on the measure are accurate and consistent; the scores are the indication of reliability. In reference to the previ- ous chapter on validity, reliability and validity complement each other, but they are sepa- rate concepts. Reliability and validity are essential to responsible test use. However, sole evaluation of reliability would be a mistake. An instrument may be reliable without being valid. For example, Lawson (2007) asserted that counselor wellness affects the quality of services clients receive. Although items that measure wellness may be consistent and accu- rate, would the presence of such items on a licensure exam compromise the validity of the licensure exam? So would it be appropriate for evaluating counselor competence if a licen- sure exam included an item such as, “How many times per week do you exercise for 30 min- utes or more?” In this case, the validity of the licensure exam may be compromised even

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

90 | assessment in counsel ing

if the scores on the item were consistent across numerous test- takers. Hence, while scores on the item may be reliable, inclusion of such an item for determining whether someone is licensed as a counselor compromises the validity of the exam.

True Score Note that the postulates of CTT indicate that an individual’s responses are measures of an attribute but that unanticipated events or factors also contribute to the measurement. Therefore, measurement is fraught with error. Any score obtained from the measurement of a construct includes three elements: (a) the observed score, (b) the true score, and (c) error. If O represents an observed score, T represents a true score, and E represents error, then the following equation expresses the relationship of the observed score to the true score and error:

O T E= +

The equation is theoretical in nature. The true score is never actually known (Gregory, 2014). For example, say an aspiring college student takes the SAT and scores 510 on the quantitative section. The student decides to retake the SAT in an effort to get a higher score and obtains a 530. Which score is the true measure of the aspiring student’s aptitude? According to CTT, the true score is within a range of scores in which 510 and 530 are included.

With respect to the previous equation, error can be positive or negative. Assume that the true score in the aforementioned example is 520 (again, we will never know the true score in actuality). In one administration, the error term is positive, denoting the aspiring student’s aptitude to be higher than what was initially measured; whereas the error term is negative in the other administration, indicating an overestimation of aptitude.

Error To understand the relationship of an observed score to a true score, it may be helpful to rearrange the equation:

T O E= −

Notice that in this equation, the smaller the error term, the more accurate the observed score is to the true score. If it were possible for no error to be present in a measure (i.e., E = 0), then the true score and the observed score would be equal. Measurement error is the difference between the observed score and the true score (AERA et al., 2014).

E O T= −

Measurement error occurs because of random error, chance, unplanned phenomena, or events that affect the measure of a construct. Stanley (1971) provided a comprehensive

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

current standards for rel iab il ity | 91

overview of measurement error, but the following issues were regarded as the most perti- nent and likely in assessment: (a) construction of the instrument, (b) administration of the instrument, (c) scoring of the instrument, and (d) interpretation of the instrument (Cohen et al., 2013; Gregory, 2014).

Construction of the Instrument The construct being measured by an instrument is based on a finite number of items con- sistent with the operational definition of the construct. With any construct of interest, there are an infinite number of possibilities for items, and the construct will ultimately be defined by a limited number of items chosen by the author(s) of the measure. For example, the BDI- II uses items that reflect one of two aspects of depression: affective symptoms and somatic symptoms. Eight items were chosen to reflect the affect (or mood) of the client and 13 items were selected to reflect somatic issues (or physical complaints). Could more of these items been created? Naturally, there are certainly many possibilities in adding to or revising the questions asked. Generally, authors of instruments seek to identify items that will provide information to measuring the construct based on the operational def- inition. Items that tend to elicit the same information as another item or fail to provide new information about the construct may be eliminated. Certainly, the failure to eliminate such items using statistical procedures and expert review will lead to measurement error.

Because the operational definition of a construct may lend subjectivity to the measure of the construct, instruments that measure the same construct may produce varying results. For example, an adolescent who is administered the Reynolds Adolescent Depression Scale (RADS) may score differently than if the BDI- II had been administered. That each instru- ment has a different item pool for measuring the same construct contributes to different results for the same construct being measured. Furthermore, each instrument may employ variations in the operational definition of depression.

When items are developed, a respondent answers each item from a subjective inter- pretation of the item. Any ambiguity in the interpretation or system of scoring contributes to random error. For example, the RADS uses a response pattern of four choices across 30 items to assess depressive symptoms using the following response format: 1 = almost never, 2 = hardly ever, 3 = sometimes, 4 = most of the time. Notice the subjective nature of the response format. The decision, for instance, to choose a 3 = sometimes versus a 4 = most of the time is not universal. What one person views as a 3 for a specific symptom may be viewed as a 2 or a 4 by another person. Although reliability coefficients attempt to estimate the consistency of the responses, random error affects the accuracy of these estimates.

Scaling items are similar to multiple- choice items, except that these types of items are usually used to discern a degree to which a behavior, thought, or action exists. In this case, there is no correct response but rather a response that may describe how a client thinks, feels, or behaves. One of the more common types of scales used in assess- ment is a Likert- type scale. In a Likert- type scale, items range from 1 to 5, in which lower scores indicate disagreement or negativity toward a construct and higher scores

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

92 | assessment in counsel ing

are indicative of agreement or affirmation toward a construct. A classic example is items in which a respondent indicates 1 = strongly disagree, 2 = disagree, 3 = neither agree nor disagree, 4 = agree, and 5 = strongly agree. Likert- type items may be scored reliably, but differences between the negative components (strongly disagree and dis- agree) or positive components (strongly disagree and disagree) may be difficult to dif- ferentiate. In addition, justifying that the measures are truly interval, that the degree of difference between two scores is universal for each respondent, is a limitation. For example, Rye et al. (2001) developed the Forgiveness Scale as part of a study involv- ing college women who had previously been wronged in a relationship. A sample item included “I spend time thinking about ways to get back at the person who wronged me” in which the respondent indicated 1 for strongly disagree to 5 for strongly agree. Can researchers assume that respondents who indicate 4, agree, are the same? The extent to which an individual indicates agree may not be universal. A respondent who answers agree may experience the same degree of a construct as another respondent who answers strongly agree. Think of it this way— imagine going to a comedy club and the comedian tells a joke. Some in the audience laugh; others do not. Yet, each audi- ence member was subjected to the same event. Some members may say the joke was funny; others may say the joke was very funny. The extent to which one finds a joke funny or very funny may not be the same across each individual. This is a limitation with Likert- type items, because such items are treated as interval data but are really more ordinal in nature.

Other types of common scales include Guttman scaling and Thurstone scaling (Trochim, 2000). Guttman scaling includes developing dichotomous items (yes or no responses) that build on one another to develop a cumulative measure. For example, a researcher could develop a series of items to measure religious tolerance that would be placed in a logical order:

1. I appreciate perspective from individuals of different faiths. 2. I would have no problem if my son/ daughter dated an individual from a different faith. 3. I would have no problem if my son/ daughter married an individual from a different faith.

Notice that each item can be answered with yes or no. In addition, an argument could be made that the items increase in intensity. The number of items in which the respondent answers yes could be the scale score. Usually a scale will consist of a larger series of items than the three used in this example.

The Thurstone scale is similar to the Guttman scale in that the items are dichotomous (yes or no responses; agree or disagree responses), but the development of the scale is much more complex and involves evaluating a large series of items and weighting the items to draw conclusions from respondents. Unlike the Guttman scale, the items are not necessar- ily presented in an ascending fashion and the scoring of the items is more involved.

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

current standards for rel iab il ity | 93

Administration of the Instrument Variability in the administration of instruments is common and can have a haphazard effect on the results. Even though instrument developers may pay careful attention to stan- dardizing the instrument through the gathering and analyzing of scores across a chosen population, random errors related to the testing environment, the individual(s) complet- ing the instrument, or the individual(s) administering the instrument may occur. Although items on an instrument can be evaluated for consistency, evaluation of the scores based on other factors simply does not occur on a consistent basis and is rarely considered when an instrument is scored and interpreted.

Errors within the testing environment include the variability within each testing environment. Areas may be spacious or cramped; temperatures may fluctuate; participants could be exposed to uncomfortably high or low temperatures. Rooms may be noisy or there could be disruptive activities outside of the testing area, such as construction work. Characteristics of the room in which the instrument is administered may have qualities that distract examinees, such as posters or carvings on walls or desks. Lighting in the room may be poor. Some desks may have poor writing surfaces, or perhaps issues exist with tech- nology when computer- based instruments are administered. Older students in particular may be less adept at computer- based administrations. These attributes, as well as others, could have an effect on the participants’ abilities to concentrate and be comfortable in the testing environment.

Counselors are responsible for identifying nonstandard environmental conditions or favorable testing conditions when interpreting assessment results (American Counseling Association [ACA], 2014). When nonstandard testing conditions occur, such conditions should be disclosed and reflected in the interpretation of the scores. Indeed, counselors should be cautious in the interpretation and subsequent use of test scores obtained from nonstandard testing conditions, as such conditions compromise reliability of the scores.

Individuals responsible for administering group- based assessments, such as school counselors who coordinate testing for schools, may encourage students to get adequate rest and food before the examination. However, the fact that each individual has unique char- acteristics related to the administration of the instrument cannot be overlooked. Students may come to the examination with varying levels of motivation, energy, and health. Some students may be overly tired or hungry; others may be ill. Issues related to test anxiety might play a role in test performance. Inadequate attention to testing protocol, such as poor time management on timed exams or mistakes made in the reading or interpretation of the items, may result in random error. Participants may mistakenly blacken the wrong oval corresponding to the item number (e.g., mistakenly mark the oval for Item 6 when answering Item 5) and thereby make the following items incongruent with the intended responses. Items may be mistakenly skipped or omitted. Such random errors can have dis- astrous consequences for scoring, interpreting, and even placing participants in appropri- ate programs or services.

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

94 | assessment in counsel ing

Individuals administering instruments also contribute to random errors. Any acci- dental departure from the standardized procedures may contribute to random error. Verbal and nonverbal communications may have an impact on how participants respond to a protocol. For example, an abrupt verbal response or nod could indicate to a participant that an answer is incorrect (Gregory, 2014). In particular, any type of exam that has an oral administration, such as various intelligence tests, is highly susceptible to random errors on the part of the administrator. Cadence, rhythm, and accent in the examiner’s speech may affect performance on an instrument and lend to variability from other examinees that have a different administrator for an instrument.

High- stakes testing has had a serious impact on some individuals who adminis- ter assessment instruments. Hacker (2007) documented evidence of cheating from 700 schools in Texas on the Texas Assessment of Knowledge and Skills, an achievement test used to comply with No Child Left Behind policies. In addition to teachers leaking ques- tions out to other teachers in order to facilitate test preparation for the students, Hacker (2007) stated the following:

In most cases, the cheating involved individual pairs or small groups of stu- dents . . . [b] ut in a few cases . . . an overwhelming number of students’ answers were incredibly similar. So aside from the statistical equivalent of lightning striking the same place 10 times, those students were either all copying one source, or an adult was doctoring answer sheets. (p. 20)

The ramifications of such errors are serious. Consumers of research use test scores to make policy decisions in education. Parents use test scores to make decisions on where to send their children to school. Test scores are often used as evidence for appropriate student placement in programs for various schools, such as gifted and talented programs. Often counselors play a pivotal role in the procedures of administering various assessments, and training for administering instruments in a standardized format cannot be overlooked.

Scoring of the Instrument Computer- based scoring may reduce random errors by increasing consistency to the scor- ing process. However, counselors typically use many instruments that are hand- scored or scored by scantrons. Problems may persist, especially when score sheets contain erasure marks or lightly blackened answers that may be misread by scantrons. Although instru- ments that rely on open- ended items, such as intelligence tests, often include substantial training to standardize scoring methods, subjectivity in scoring items may still occur, thereby compromising instrument reliability.

However, another type of scoring issue that is often overlooked when evaluating the psychometric properties of the instrument is the response format on an instrument. Forced- choice items, such as using a Guttman scale, may have less subjectivity in terms of participants understanding the item and choosing a response, yet the limited responses

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

current standards for rel iab il ity | 95

available may not accurately reflect the construct of interest for the participant. In contrast, Likert scale items (i.e., 5- point scales ranging from strongly disagree to strongly agree), as mentioned earlier, are often used as interval scale items, which can be summed and incor- porated into mean scores. However, the response format is quite subjective, as some par- ticipants may choose agree while others choose strongly agree with no verifiable measure of whether the intensity of the construct is truly different from participant to participant. Another example would be scales that measure chronic pain. Some individuals may have a higher pain tolerance and provide lower scores yet still be in as much or more pain as someone who is endorsing higher levels of pain. Technically these items could be consid- ered ordinal, but researchers who use these instruments treat such items as interval scales to facilitate the use of parametric statistics, many of which are included in the test manuals to support reliability of the scores and validity of the instrument. So, although the practice of quasi- interval scales is common among instrument developers, the potential for random error and subsequent effect on reliability is evident.

Interpretation of the Instrument Many instruments provide the opportunity for counselors to use computer- generated reports once scores are tabulated. When pregenerated reports are used to communicate test results to the client, the onus of responsibility for the accuracy of the report lies with the counselor who assumes ultimate responsibility for the communication of results (ACA, 2014). In the case of using assessments for diagnostic purposes, counselors admin- istering and creating reports are accountable for the accuracy and errors of the assessment report. However, in the case of participants contracting with testing companies to provide an assessment, such as an aptitude test, the test company assumes responsibility for the accuracy of the results.

Another source of measurement error is systematic error, when the instrument mea- sures something other than the construct. In an attempt to measure commitment to safety for adolescents admitted to a crisis unit, Balkin (2004) created items that also loaded on an individual’s ability to process coping skills. The end result was the reliability for scores on the scale designed to measure commitment to safety was compromised. The development of scales that consistently assess one and only one construct is improbable and compounded by the fact that systematic errors may go unnoticed. Cronin and Goodman (2008) docu- mented the legislative approval of using the SAT to predict first- year success in college as the exit exam for high school students. In other words, an instrument designed to assess aptitude was implemented to assess academic achievement. Despite efforts from Maine counselors, the ACA, and the Association of Assessment in Counseling and Education, a major systematic error was placed into educational policy. The good intentions of legisla- tors to boost college admissions has resulted in students being evaluated on material that may not be covered in a high school academic curriculum and therefore serves as an invalid indicator of academic achievement.

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

96 | assessment in counsel ing

Estimating Reliability Because of the presence of error, estimating the reliability of scores on an instrument (i.e., the consistency in which a construct is measured) can be problematic. Moreover, different methods are used, and more than one reliability estimate may be reported. Three terms are common in estimating reliability: (a) reliability coefficient, (b) standard error of measure- ment (SEM), and (c) reliability index.

Reliability Coefficient To estimate reliability, a reliability coefficient is computed to quantify the relation- ship of observed scores on an instrument: rxx is the correlation between two observed scores. Reliability coefficients typically range from zero to 1. A perfect correlation between observed scores is 1.0, meaning that a set of examinees will obtain the same score each time the test is administered. Consequently, a reliability coefficient of 1.0 is not likely to occur for instruments commonly used to measure psychological con- structs. Scores may be similar for individuals who retake an instrument, but slight variations are expected. Ultimately, perfect reliability is difficult to obtain, even when more objective measures are used. For example, measures of blood pressure, resting heart rate, or even weight rarely show the exact values when done repeatedly in a given time frame. The reliability coefficient is the most commonly reported estimate of reli- ability. There are different methods of reporting the reliability coefficient, which are discussed later in the chapter.

Standard Error of Measurement Recall that the standard deviation typically refers to the average amount of error from the mean for a given sample or population. A mean, therefore, represents the average score for a particular group and the standard deviation indicates how much each individual will differ from the group mean, on average. When administering an instrument, the standard deviation provides an indication of how a particular individual’s score is similar or different from a given group, but the standard deviation is not an indication of the instrument being a consistent measure of a construct for the individual.

For example, the BDI- II for adults in outpatient settings has a mean of 22.45, a stand- ard deviation of 12.75, and a reliability coefficient of .92 (Beck, Steer, & Brown, 1996). Beck et al. suggested the guidelines of total scores listed in Table 5.1 for diagnosing major depression.

A client who scores a 20 on the BDI- II would be classified in the moderate range for major depression according to Beck et al.’s (1996) guidelines. But how likely would the cli- ent be to get the same score on a second administration of the BDI- II if no other interven- tion or change in life circumstances has taken place? To answer this question, counselors use the standard error of measurement (SEM), which indicates the average amount of error for an individual if the instrument were to be administered repeatedly. So, although the

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

current standards for rel iab il ity | 97

standard deviation indicates variability within a group, the SEM indicates variability of a score for an individual.

The SEM can be computed with the following formula:

σ σe xxr= −1

where σe is the standard error of measurement, σ is the population standard deviation, and rxx is the reliability coefficient. So, although an instrument has error, participants may not obtain the same score when administered an instrument repeatedly under similar condi- tions, and although the true score of an individual is never really known, the SEM can be computed to indicate the range in which the true score lies.

Referring back to the previous example of the BDI- II, if the test has a standard devia- tion of 12.75 for individuals receiving counseling services in an outpatient setting and a reliability index of .92, then the SEM for the BDI- II can be computed as follows:

σe = − =12 75 1 92 3 61. . .

Notice the attributes of the SEM. If the standard deviation remains constant and the reliability coefficient increases (moves closer to 1, demonstrating higher consistency), the SEM becomes smaller; likewise, scores on instruments that are less reliable have more error indicated by a larger SEM. For example, the reliability of the BDI- II for a college sample was .93:

σe = − =12 75 1 93 3 37. . .

When the reliability coefficient remains constant and the standard deviation decreases, the SEM once again becomes smaller and would increase if the standard deviation were to increase. When the BDI- II was administered to participants who had a previous diagnosis of a mood disorder, the mean score was 26.57 and the standard deviation was 12.15.

σe = − =12 15 1 92 3 44. . .

TABLE 5.1 Cut Scores

on the Beck Depression

Inventory- II

Total Scores Range

0– 13 Minimal

14– 19 Mild

20– 28 Moderate

29– 63 Severe

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

98 | assessment in counsel ing

On any given assessment period the SEM provides information about the true score within a specific range of confidence, known as a confidence interval. Now, we can identify the range in which the true score lies using the information we learned in Chapter 3. Within 1 standard deviation, we can be 68% confident that the true score will lie ±1σe of the observed score; within 2 standard deviations, we can be 95% confident that the true score will lie ±2σe of the observed score; within 3 standard deviations, we can be 99% confident that the true score will lie ±3σe of the observed score. The range for the true score can be expressed in the following way:

For 68% confidence, T = O ±1σe

For 95% confidence, T = O ±2σe

For 99% confidence, T = O ±3σe

In our example for the client who scored 20 on the BDI- II:

We can be 68% confident that the client’s true score is between 20 ± 3.61 or between 16.39 and 23.61, inclusive.

We can be 95% confident that the client’s true score is between 20 ± (2)3.61 or between 12.78 and 27.22, inclusive.

We can be 99% confident that the client’s true score is between 20 ± (3)3.61 or between 9.17 and 30.83, inclusive.

Referring back to the suggested interpretative guidelines of the BDI- II, the SEM has some implications for the client, as the client may fall in between the mild to moderate range of depression, again reinforcing the importance of caution in the interpretation of assessment results.

Indeed, SEM can play a pivotal role when assessments are used. All too often educa- tional settings overrely on scores to address placement and service issues. A student who tests in the range of borderline intellectual functioning may be refused services on the basis of the score when in fact the SEM indicates that the student may have tested in the extremely low range of intellectual functioning.

Reliability Index The reliability index is the relationship between the true score and the observed score and is a less common term than the reliability coefficient to provide estimates of reliability. The reliability index, rTX, quantifies correlation of the true score (T) to the observed score (O), as opposed to the reliability coefficient, which identifies the relationship between two forms or administrations of an instrument. The reliability index is related directly to the reliability coefficient and is computed as follows:

r rTX xx=

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

current standards for rel iab il ity | 99

As the reliability coefficient increases or decreases, the reliability index increases or decreases, respectively. A perfect reliability coefficient, 1.0, would indicate no error in consistency between administrations; therefore, a perfect reliability index, 1.0, would also be present as there would be no error in predicting the true score from the observed score.

Types of Reliability Measurement Several methods are used to calculate reliability coefficients (rxx). As rxx approaches 1.0, scores on the instrument are deemed more consistent. This section addresses four differ- ent forms of computing reliability coefficients: (a) test– retest, (b) parallel/ alternate forms, (c) internal consistency, and (d) interscorer reliability. When reliability estimates are reported for scores on an instrument, common practice includes the use of more than one method to demonstrate reliability.

Consistency Over Time: Test– Retest Reliability Test– retest reliability refers to the correlation of two administrations of the same instru- ment. Often referred to as stability over time, reliability is evaluated by examining the relationship of the same instrument measuring the same construct at two different time periods. A Pearson product– moment correlation coefficient can be computed between the two scores to determine the relationship. This is an appropriate measure to use when the construct being measured remains stable (i.e., does not change) over time.

Psychosocial constructs, however, may change over time— even if no intervention has occurred. Balkin, Tietjen- Smith, Caldwell, and Shen (2007) studied the effect of exer- cise on depression for young adult women and noted a nonsignificant decrease in scores on the BDI- II for the control group (nonexercise group) when the BDI- II was adminis- tered for baseline and then six weeks later. Although depression may decrease over time when there is no intervention, that no statistically significant decrease was evident may be linked to the high test– retest reliability of the BDI- II. Beck et al. (1996) reported a test– retest reliability for the BDI- II at r = .93 when the BDI- II was administered twice at an interval of one week apart. Test– retest reliability is related to the amount of time between the two administrations (Trochim, 2000). Shorter time periods between administrations may yield higher reliability coefficients. A limitation in the test– retest methodology is the assumption that no meaningful changes have occurred that would alter the measurement of the construct being investigated. In addition, the presence of a testing effect, previous exposure to the instrument by the examinee, may alter the manner in which the exam- inee responds. For example, say after an initial administration of the BDI- II an examinee decides to look up symptoms of depression. In an effort to appear less depressed, the exam- inee could answer the items differently, because previous exposure to the items from the first administration took place.

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

100 | assessment in counsel ing

Estimates of Equivalency: Parallel or Alternate Forms When more than one form of the same instrument exists, the equivalency of the forms can be assessed by correlating scores on the two forms of the instrument. Although the terms parallel or alternate are used interchangeably, there is a technical difference. Parallel forms maintain the same means and variances across the various forms; alternate forms are constructed with the intention of being parallel but may not have the same descriptive information (Cohen et al., 2013).

A Pearson product– moment correlation coefficient can be computed between the two scores on each of the forms to determine the relationship. In this case, highly consist- ent forms will have different items that cover the same content. Instruments that meas- ure a specific knowledge base or aptitude (e.g., SAT, ACT, Graduate Record Examination [GRE], National Counselor Examination [NCE]) should not differ across item difficulty, the percentage of participants answering an item correctly, or item discrimination, the extent to which an item distinguishes those who vary on a given construct.

The challenge in using parallel forms to evaluate the reliability of the scores on an instrument is the development of items that measure the same aspects of a construct for different forms of the instrument. A much larger item pool is necessary to develop equiva- lent forms. For example, a counselor who is required to take the NCE for a second time would likely encounter items that are different from the first administration but cover the same content areas. Developing items that fit the criteria for an alternate form would be much easier than developing items for a parallel form, as providing evidence for similar item difficulty and discrimination would be necessary.

Internal Consistency Both test– retest and equivalent forms can be time- consuming methods to estimate reli- ability of scores, as either the instrument must be administered twice or another form of the instrument must be created. However, another way to estimate reliability may be to examine the internal consistency of the scores on the instrument— analyzing the relation- ships of the scores for each of the items on the instrument. There are several methods to evaluating the internal consistency of an instrument: (a) split- half, (b) coefficient alpha, and (c) Kuder- Richardson (K- R) formulas.

Split- Half Assuming that all of the items measure the same construct, the instrument can be split into equivalent halves. A Pearson product– moment correlation coefficient can be computed between the two halves on each of the forms to determine the relationship or reliability coefficient. Splitting the instrument into two equivalent halves can be complicated. For example, the BDI- II may not be a good instrument with which to use this method of reli- ability estimation. The BDI- II has 21 items, and each item measures a distinct characteristic

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

current standards for rel iab il ity | 101

of depression. So identifying two equivalent halves of the instrument may not be possible. However, to use this method on the NCE may be easier, as there could be several items that measure knowledge in ethics, the helping relationship, group theory, and so forth, and these items could be equally divided between two halves of the exam.

An additional problem occurs in reliability estimation when the split- half method is used. Reliability estimates fluctuate depending on the length of the exam. An increase in items leads to an increase in reliability estimates (Gage & Damrin, 1950), and scores on shorter tests are less reliable. When an instrument is split in half, the reliability coefficient will be underestimated. The Spearman- Brown formula adjusts for the underestimation of the split- half method and can be computed as follows:

r r rsb hh

= + 2

where rsb is the reliability coefficient using the Spearman- Brown formula and rhh is the reliability coefficient using the split- half method. Recall that the split- half method will compare two half- tests, while the Spearman- Brown adjusts for this error by providing a reliability estimate for the whole test. Using the previous formula, the adjustments found in Table 5.2 can be noted.

Because of the underestimation of the split- half method, Cohen et al. (2013) rec- ommended that the Spearman- Brown formula always be used when estimating reliability using the split- half method.

Coefficient Alpha As mentioned previously, test developers may have difficulty justifying how an instru- ment can be divided into two equivalent halves. However, there can also be many ways to divide an instrument in half. Cronbach (1951) devised a mathematical formula, coef- ficient alpha or Cronbach’s alpha, to take into account all possible split- half methods to evaluate the internal consistency of an instrument. The formula for coefficient alpha is as follows:

r n

n i

α σ

σ =

−  

  −

 

 1

1 2

TABLE 5.2 Reliability Estimates Using

Split- Half and Spearman- Brown Formulas

Split- Half Reliability

Coefficient

Spearman- Brown

Reliability Coefficient

0.70 0.80 0.90

0.82 0.89 0.95

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

102 | assessment in counsel ing

where rα is coefficient alpha (this term is often referred to as α when discussing reliability estimates in published research), n is the number of items on the instrument, Σσi

2 is the sum of the variance for each item, and σ2 is the total variance of the instrument. Although this statistic is quite labor intensive when computed by hand, the use of computer pro- grams has made coefficient alpha the most widely reported and preferred method for esti- mating reliability.

Kuder- Richardson The coefficient alpha formula is actually an extension of an earlier formula developed to evaluate internal consistency for dichotomous items. Whereas coefficient alpha can be used to estimate reliability for items that have a range of responses (i.e., Likert scale items— strongly agree to strongly disagree), the KR- 20 is used to evaluate internal consis- tency when items can be scored a 1 or zero (e.g., right or wrong; relapse or no relapse). The KR- 20 formula is as follows:

r n

n pq

KR20 21 1=

−  

  −

  

Σ σ

where rKR20 is the Kuder- Richardson reliability coefficient, n is the number of items, p is the proportion of participants who answer the item correctly or positively, q is the proportion of participants who answer the item incorrectly or negatively, and σ2 is the total variance of the instrument. Although the KR- 20 can only be used for dichotomous items, coefficient alpha will produce the same results as KR- 20 for dichotomous items and can be extended to nondichotomous items as well.

Interscorer Reliability Some measures are dependent on scoring from standardized procedures. Measures of intelligence, performance, or other subjective indicators may vary as a result of the scorers, as opposed to actual variance in the construct. For example, each year at the Olympics, a controversy ensues over scores by various judges. Sports such as figure skat- ing, gymnastics, and boxing often experience questionable scoring procedures. These types of issues may also exist in many types of tests in which the presence or absence or pass or failure of an attribute is dependent upon a scorer’s perspective. Interscorer reliability, often referred to as interrater reliability or interrater agreement, refers to the relationship between or among scores issued by raters. Consistency among scorers is dependent on the use and training of objective criteria to rate a construct. When low correlations exist among raters, some type of training is needed to prompt the raters to use similar criteria.

The Pearson product– moment correlation coefficient may be computed to assess the consistency between two judges. When more than two judges are being evaluated, a more sophisticated statistic, called the intraclass correlation coefficient, can be computed using

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

current standards for rel iab il ity | 103

computer programs. The intraclass correlation coefficient provides the average rating for a single judge. To account for more than one judge in the average, adjustments can be made using a Spearman- Brown correction:

j icc j icc ( )

( )1 1+ −

where j is the number of judges and icc is the intraclass correlation coefficient. An approxi- mation of the icc may be determined from the average of the Pearson product– moment correlation coefficients from all raters.

Despite the attempt to make scores or determinations from experts accurate and consistent, there are some endeavors in counseling that simply lack evidence of accuracy and consistency, and one of those is diagnosis. According to a review by Vanheule et al. (2014) that covered selected clinical field trials in 1974, 1992, 1995, and 2013, the accu- racy and consistency to diagnose has not changed or improved, even though the meta- morphosis of the diagnostic system from second to the fifth edition of the Diagnostic and Statistical Manual of Mental Disorders (DSM). In other words, although our understand- ing of diagnosis has changed, the ability for clinicians throughout the helping professions to accurately and consistently diagnose has not improved (Aboraya, 2007). But that has not stopped professionals from suggesting that accuracy and consistency in diagnosis has improved. Rather, what we see is that any improvement in diagnosis is due to the change of the standard and what constitutes excellence in accuracy and consistency in diagnosis. Let’s explain this a little more.

Vanheule et al. (2014, p. 314) used a kappa coefficient (k), which measures the agreement of two raters between zero (no agreement) and 1 (perfect agreement) and noted the following evolution of k as it pertains to the interpretation of reliability estimates in diagnosis:

• 1974: k ≥ .90, excellent; k = .70 – .90, good; k ≤ .70, unacceptable; • 1977: k ≥ .75, excellent; k = .40 – .70, fair to good; k ≤ .40, poor; • 2010: k ≥ .70, excellent; k = .60 – .70, good; k = .41 - 49, questionable; k ≤ .40, poor; • 2013: k ≥ .80, excellent; k = .60 – .79, very good; k = .40 - 59, good; k = .20 – 39, ques-

tionable; k ≤ .20, unacceptable.

So how does the interpretation of the k translate to the agreement between diagnosti- cians in clinical trials? According to Vanheule et al. (2014), a clinical field trial by Williams et al. (1992) would have used the third edition of the DSM. Based on the 1974 interpre- tation of the k, 72% (n = 13) of the interrater agreements were in unacceptable range, and 28% (n = 5) were in the good range. However, using the 2013 interpretation of the k of the interrater agreements, 50% were in the good range, 33% in the very good range, and 17% in the excellent range. In comparison, the more recent 2013 clinical field trial for the fifth

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

104 | assessment in counsel ing

edition of the DSM indicated that 93% (14 out of 15) of the interrater agreements would have been categorized as questionable or unacceptable in 1974, but by using the 2013 stan- dards, only 33% (n = 5) are identified as questionable or unacceptable and the remaining 67% (n = 10) are identified as good or very good.

So what does this all mean? Essentially, the ability for experts to agree on diagnosis is a challenge that has been problematic since psychiatry attempted to standardize this process. The actual statistics of agreement on diagnoses have not changed substantially. Rather, the standards by which to judge agreement have been altered in order demonstrate improvements that do not really exist.

Interpretation of Reliability When deciding whether or not to use a particular psychosocial instrument, the interpreta- tion of reliability data is pertinent. Counselors should be aware of reliability estimation methods, the conditions in which reliability estimates were derived, and the description of the participants from whom the data were collected. “General statements to the effect that a test is ‘reliable’ or that it is ‘sufficiently reliable to permit interpretations of individual scores’ are rarely, if ever, acceptable” (AERA et al., 2014, p. 41).

Each method for determining reliability contains sources of error related to time, content, scoring error, and sampling variance. Test– retest is limited by time, as longer peri- ods of time between administrations may decrease reliability estimates. Reliability esti- mates related to using equivalent forms may be limited by content, as alternate content may be inadvertently used because of the need to generate larger item pools. Although coefficient alpha appears to address limitation in test content that occurs with split- half methods, items that are less likely to measure the homogeneous nature of a construct will lower reliability estimates. Identifying items that measure more heterogeneous attributes of a construct may need to be eliminated. Reliability estimates, with respect to interscorer reliability, are affected by inherent biases, as well as subjective scoring procedures. Objective criteria and training may increase reliability estimates but not eliminate the error variance.

An important consideration outside of computing reliability estimates is the nature of the group in which the reliability estimates are obtained. As a rule of thumb, a het- erogeneous group will provide higher reliability estimates, regardless of the method used. Imagine if a group being administered the BDI- II all scored in the severe range. There would be no way to correlate this group’s depression with other characteristics, because there was no difference evident from the scores in depression— everyone scored similarly. Having a diverse sample provides evidence that attributes can be consistently measured, as they vary from person to person.

In another example, consider the construct of introversion– extroversion. There would be little relationship to any other construct, such as propensity for substance abuse or self- esteem, if each participant scored high on extroversion. Correlations would be low

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

current standards for rel iab il ity | 105

because of the lack of variability in the sample. With respect to interscorer reliability, if everyone obtained the highest score possible on a construct such as creativity, there would be no way to rank the participants with respect to creativity. The issue of variance in the sample underlies the importance of counselors being familiar with whom the instrument was normed.

The nature of the instrument is another important consideration. Many instruments, especially instruments geared toward measuring ability (e.g., aptitude, achievement, intel- ligence), rely on speed and/ or power measures. Speed measures contain simple items that the examinee will likely answer correctly but the time limit is restricted, preventing the examinee from completing all of the items. Power measures provide adequate time to com- plete the instrument but include items of difficulty that may prevent one from obtain- ing a perfect score. Many instruments (ACT, SAT, GRE, Wechsler Intelligence Scale for Children– Fifth Edition) employ a combination of these measures. Because of the nature of speeded tests, traditional split- half methods, such as comparing odd and even items, may produce very high reliability estimates. A better method would be to use a test– retest method on two separately timed tests or correlate to half- tests with a Spearman- Brown formula (Gregory, 2014).

Given the limitations of reliability estimates, what constitutes adequate reliability? Hopkins (1998) indicated that standardized tests, such as those used for placement and college admission, should have reliability coefficients of .90 or higher. Yet many psycho- social instruments are used with reliability estimates near .70. Certain constructs, such as psychosis, have been difficult to measure, and reliability estimates may be lower. In the case of interrater reliability, some constructs simply lack expert agreement, as in the case of diagnosis.

Reliability estimates should not be used alone to assess the consistency of the scores. Standard error of measurement should also be considered. Although reliability estimates account for consistency of the instrument, standard error of measurement provides an indication of accuracy. Recall that the standard error of measurement incorporates two terms: a reliability coefficient and measurement error. Therefore, scores on an instrument may be consistent but could also be inaccurate.

What Are the Implications for Reliability? Because of the many different aspects of reliability, determining whether an instrument is reliable is not a simple matter and requires a multifaceted approach. Instruments may have strong reliability evidence in one area yet be lacking in another area. Such an issue is appar- ent in the Child Behavior Checklist (CBCL). Although the CBCL is a popular instru- ment, the reliability of the scores on the instrument may be questioned in some regards. Test– retest after one week and internal consistency scores for composite scales average .80; but internal consistency scores for the subscales may be as low as .50. The attributes of

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

106 | assessment in counsel ing

the raters may also be a factor with interrater reliability averaging .66 on the parent forms (Doll, 2004). Similar to the CBCL, the Minnesota Multiphasic Personality Inventory- 2 maintains strong test– retest reliability, averaging .83, but internal consistency reliability for the clinical subscales has a broader range, .34 to .87 with a median range of .63 (Matz, Altepeter, & Perlman, 1992).

Educational and cognitive tests may have higher reliability estimates because of the nature of measuring academic performance or intelligence, as opposed to psycho- pathology, which may be considered more diverse and complex in nature to measure. Reliability estimates for the Wechsler Adult Intelligence Scale– IV are quite adequate to strong (e.g., internal consistency estimates were .71– .96 for scores on the subtests and .97– .98 for the full scale IQ scores; Canivez, 2010). The Wechsler Intelligence Scales have a standard error of measurement of ±5 points, so this needs to be consid- ered when applying labels and determining services for individuals. For example, if a school district employs a cut- score of 70 IQ to provide services and a student scores 72, the SEM indicates that the student could fall in the range of borderline to extremely low functioning.

Again, the popularity of an instrument is not a guarantee of reliable assessment results. Each of the assessments used in this sample profile are more widely known and can easily be referenced in the Mental Measurement Yearbook (Lincoln, NE: Buros Institute of Mental Measurements) or peer- reviewed literature. Counselors should be aware of the error related to measurement and assessment. Standardized assessments are only a tool and should never stand alone in determining treatment or diagnosis.

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .