Creating a Code Book and inputting Data Entry

60

A. RELIABILITY AND VALIDITY IN QUANTITATIVE RESEARCH

Reliability

What is reliability? If you think about how we use the word reliable in everyday language you

might get a hint about its meaning. For instance, we often speak about a machine as reliable, as in, “I

have a reliable car.” Or, news people talk about a “reliable source”. In both cases, the word reliable

usually means “dependable” or “trustworthy”. In research, the term reliable also means dependable in

a general sense, but there’s more to the definition. In research, the term reliability means repeatability

or consistency. A measure is considered reliable if it would give you the same result over and over

again (assuming that what you are measuring isn’t expected to change). You cannot use a

measurement in your research without first showing that it is reliable. There are four ways of

assessing reliability. Each is used in a different circumstance but they are all ways of estimating the

reliability of the measures you are using in your research study.

Test-Retest Reliability

You assess test-retest reliability when you administer the same test to the same sample on two

different occasions. You would use this procedure for examining reliability if you were measuring

something that you were not expecting to change between the two testing occasions. The amount of

time allowed between the measurements is critical. If you measure the same thing twice, the

consistency between the two measurements will depend in part on how much time has elapsed

between the two measurement occasions. The shorter the time gap, the more the scores will be the

same; the longer the time gap, the more the scores might differ because there might be changes to the

individual during the time interval.

Parallel Forms (Alternate Forms) Reliability

In parallel forms reliability, you first have to create alternate forms of the same measure. One way

to accomplish this is to start with a large set of questions that address the same thing and then

randomly divide the questions into two sets. You then administer both instruments to the same sample

of people; the correspondence between the two parallel forms is the estimate of reliability. One major

problem with this approach is that you have to be able to generate lots of items that reflect what you

want to measure, which is no easy feat. Further, this approach makes the assumption that the

randomly divided halves are equivalent, or parallel. Even by chance, this sometimes will not be the

case.

Internal Consistency Reliability

When you assess reliability using the internal consistency method, you use your single

measurement instrument and administer it to a group of people on one occasion. What you are doing

by this method of estimating reliability is assessing how well the items on the instrument reflect the

same thing. You are in essence looking at how consistent the results are for different items for

measuring the same thing within the measurement instrument.

Inter-observer (Inter-rater) Reliability

Whenever you use humans to conduct observations of others as part of your measurement

procedure, you have to worry about whether the results you get are reliable or consistent. People are

notorious for their inconsistency. We are easily distractible. We get tired of doing repetitive tasks. We

61

daydream. We misinterpret. So how do you determine whether two observers are being consistent in

their observations? You should first establish inter-observer (inter-rater) reliability before you start

your actual study in a pilot study. That way, if you find your observers don’t agree with each other

(which means the reliability will be low) you can conduct additional training with them rather than

having to discard all of the data that’s been collected by them.

There are two ways to go about estimating inter-observer reliability. If the measurement they are

taking consists of categories – the raters are checking off which categories each observation falls in –

you can calculate the percent of agreement between the raters. For example, suppose you had 100

observations that were being assessed by two raters. For each observation, the rater could check one

of three categories. Imagine that on 86 of the observations the raters checked the same category. In

this case, the percent of agreement would be 86%. This might seem like a crude measure, but it does

give an idea of how much agreement exists, and it works no matter how many categories are used for

each observation.

The other major way to calculate inter-observer reliability is appropriate when the measurement is

not categories, but rather is a numerically continuous score. If this is the case, all you would do is

calculate a statistic (called a correlation) which would tell you the exact correspondence between the

two observers, and thus provides a precise assessment of inter-observer reliability.

Validity

When examining a measurement instrument, the first question you must ask yourself is if the

scores it produces are stable and consistently reliable. If your answer is “yes”, the second question is

then whether you can draw meaningful and useful inferences from the measuring instrument – you are

asking about its validity. Seen in this way, reliability is an antecedent to validity: A measure cannot be

valid unless it is first reliable. Reliability is a necessary but not sufficient condition for validity. This

perspective also characterizes validity as the larger, more encompassing issue when you make a

decision about the choice of an existing instrument, or design your own.

The clearest way to define validity is to state that you are asking the question “Is this instrument

measuring what it is supposed to be measuring?” You might have an instrument that repeatedly yields

the same scores on the same sample members (that is, it’s reliable), but those scores aren’t a measure

of what you think they are (that is, the instrument is NOT valid). Validity inquires into the meaning of

the instrument – whether it’s actually measuring what it was designed to measure. There are several

different types of validity, but for our purposes we will address only the most frequently found ones.

Types of Validity

Criterion Validity

An existing measure that we accept as the best indicator of the target concept or behavior we are

trying to study is called a criterion. Criterion validity involves assessing the correspondence

(correlation) between this criterion measure and the new measure that we are trying to devise. If the

correspondence between the two is very high, we can conclude that our new measurement instrument

is high in criterion validity.

Criterion validity has a number of variants, one of which is of particular interest. Suppose we were

trying to design a new measurement instrument that could predict future behavior, such as the tests

that high school juniors take to determine how well they’ll perform in college, or those tests taken by

those aspiring to medical or law school a year or more in advance. We would need to know that the

scores on our test do a good job of predicting future behavior – that they are high in predictive

62

validity. To assess this, we would give a large sample of high school students our test and use their

scores to predict how well they are likely to do in their choice of schools in several years. If there is a

strong correspondence between the criterion of later success and the earlier test scores, we can

conclude that our measurement was high in predictive validity.

Content Validity

There are some concepts that we choose to study that have many component parts to them. For

example, job performance for the Director of a park and recreation agency consists of many domains.

If we were to develop an assessment and only included budgetary skills, for example, we would be

missing the majority of what that job entailed. That performance assessment instrument would not be

content valid because it did not cover all of the areas of the job. If we are trying to design a

measurement instrument for a concept that is comprised of multiple components, we must assess its

content validity – that is, we must be certain that the instrument has covered the entire domain of

content and asked questions about all of the components that make up that concept. Content validity is

assessed by inviting a team of experts in the subject matter of the concept we are researching and

asking if we have included all parts of it in the measuring instrument. A unanimous opinion from the

experts in the affirmative would allow us to conclude that the measurement is high in content validity.

Construct Validity

Construct validation asks how well the test reflects the concept (or more abstract “construct”) we

are trying to measure. Because of its abstract nature, this is a difficult type of validity to assess. At

best we can gather evidence that would strengthen or weaken our confidence in the construct validity

of the measure, but often times it is difficult to find a perfect measure to which we can compare the

one we are designing. To assess construct validity we need to find some objective valid measure of

what we are trying to measure, and administer both measures (the known valid one and ours which is

under development) to members of a large sample. If the correspondence between the measures is

very high, then we conclude that our new measure is high in construct validity. It is frequently

difficult to find another measurement instrument that is high in validity to which we can use for

comparison, so other types of criterion measures can be used if they are valid, such as certain

behaviors. For example, if we return to our desire to design a measure of job performance for the

position of Director of a parks and recreation agency, we might use measures such as increases in

revenue or participation from previous years, evaluations from department heads or input from the

public, etc. The important and challenging issue is to be able to find that objective valid criterion

measure against which to compare our developing measure in an effort to assess its construct validity.