Week 6

Reliability and Validity

In: Reliability and Validity in Qualitative Research

By: Jerome Kirk & Marc L. Miller

Pub. Date: 2011

Access Date: July 15, 2019

Publishing Company: SAGE Publications, Inc.

City: Newbury Park

Print ISBN: 9780803924703

Online ISBN: 9781412985659

DOI: https://dx.doi.org/10.4135/9781412985659

Print pages: 14-21

© 1986 SAGE Publications, Inc. All Rights Reserved.

This PDF has been generated from SAGE Research Methods. Please note that the pagination of the

online version will vary from the pagination of the print book.

Reliability and Validity

Despite the prestige and success of natural science in recent years, application of science as a model for

social “science” is not inevitable. Many have argued that social science has an intrinsically different set of

goals that call for an altogether separate collection of methods. Others (nonscientists, it should be noted)

contend that recent developments in the natural sciences entirely discredit the fundamental notions (such as

objectivity) of an earlier and outdated science.

Yet, whatever their detailed goals, the natural and social sciences share an aspiration to cumulative collective

knowledge that is of interest on its own merits to those other than the friends and admirers of its creators.

This goal is exactly objectivity. In the natural sciences, objectivity is obtained in two ways. First, experience is

reported in such a way that it is accessible to others, for example, when reporting an experiment every effort

is made to describe the way the experiment was carried out, just in case somebody else would like to try the

same thing. Second, the results of the experiment are reported in terms of theoretically meaningful variables,

measured in ways that are themselves justifiable in terms of the relevant theories.

Since Wilhelm Dilthey and George Herbert Mead, the vast majority of social scientists have agreed that

objectivity, in this sense, is an admirable goal. Yet, the description of reliability and validity ordinarily provided

by nonqualitative social scientists rarely seems appropriate or relevant to the way in which qualitative

researchers conduct their work.

It is the purpose of this book to reconcile the means-ends discrepancy. The remainder of this chapter will

pursue the argument that, subject to clearly specifiable differences in goals and practice, social science is in

every sense of the word fully as “scientific” as physics, and has fully as much need for reliability and validity

as any other science.

The “Positivist” View

In recent decades, the social science literature has incorporated a great deal of discussion of an epistemology

called “positivism.” (The term is generally employed by those advocating some alternative view of knowledge,

and often amounts to a straw man.) In its strongest form, positivism denies objectivity as defined here by

assuming not only that there is an external world, but that the external world itself determines absolutely the

one and only correct view that can be taken of it, independent of the process or circumstances of viewing. No

one seriously defends such an ontology, but scholars attentive to the social and cultural construction of social

things (including social science) point out that much research (particularly nonqualitative research) makes

sense only in terms of a set of unexamined positivist assumptions.

Most often, these assumptions pertain to the “naturalness” of the measurement procedure employed. Thus

a survey researcher may interview a large number of people about their political attitudes, and conclude that

SAGE

1986 SAGE Publications, Ltd. All Rights Reserved.

SAGE Research Methods

Page 2 of 7 Reliability and Validity in Qualitative Research

“public opinion” says something.

Such an assertion obviously concerns the investigator's theoretical view of the world as much as it does

the psychic organization of the interviewees. The investigator's theory contains categories not imposed by

the structure of empirical reality. Elements such as “attitudes” and “public opinion” serve rather to organize

understanding of the world. Certainly, political and psychological theories that do not use these constructs (or

even deny their meaningfulness) are possible, and treating analytic devices as though they are facts is the

well-known fallacy of reification.

In response to the propensity of so many nonqualitative research traditions to use such hidden positivist

assumptions, some social scientists have tended to overreact by stressing the possibility of alternative

interpretations of everything to the exclusion of any effort to choose among them. This extreme relativism

ignores the other side of objectivity—that there is an external world at all. It ignores the important distinction

between knowledge and opinion, and results in everyone having a separate insight that cannot be reconciled

with anyone else's.

Metaphysical polemics, often directed against caricatures of the opposing views, largely miss the point. As is

shown in the next chapter, the problem is not so much one of metaphysics as it is a pragmatic question of

the validity of measurements. The survey researcher who discusses attitudes is not wrong to do so. Rather,

the researcher is wrong if he or she fails to acknowledge the theoretical basis on which it is meaningful to

make measurements of such entities and to do so with survey questions addressed to a probability sample of

voters.

For any observation (or measurement) to yield discovery, it must generate data that is (a) not already

known and (b) identifiable as “new” by the theory already in place.2 Most of the technology of “confirmatory”

nonqualitative research in both the social and natural sciences is aimed at preventing discovery. When

confirmatory research goes smoothly, everything comes out precisely as expected. Received theory is

supported by one more example of its usefulness, and requires no change. As in everyday social life,

confirmation is exactly the absence of insight.

In science, as in life, dramatic new discoveries must almost by definition be accidental (“serendipitous”).

Indeed, they occur only in consequence of some kind of mistake.

The Discovery of the New

Henri Becquerel was studying the phenomenon of phosphorescence by exposing metal salts first to the

sun and then to photographic plates. When the sky clouded over for an extended period, he tossed the

uranium salts into a drawer with his photographic materials and knocked off work for a while (Badash, 1965).

“Merde! Je me suis plantér!” he must have muttered when he discovered that the film was ruined, but he was

sufficiently prepared and alert to realize that he had discovered radioactivity.

SAGE

1986 SAGE Publications, Ltd. All Rights Reserved.

SAGE Research Methods

Page 3 of 7 Reliability and Validity in Qualitative Research

More recently, the first men to hear the echo of the origin of the universe thought they were listening to guano.

In 1964, Arno Penzias and Robert Wilson set out to measure the radio waves emitted from different latitudes

in our disk-shaped galaxy. First, they had to identify what portions of the signal they received originated in

the instrument itself. When they received a strong signal in microwave frequencies (where galaxies emit

virtually no radiation), their first move was to devote considerable time and expense to cleaning out the “white

dielectric material” deposited by pigeons in the antenna throat. This produced only a negligible decrease in

signal strength (Penzias and Wilson, 1965). While Penzias and Wilson were in the antenna throat, Dicke et

al. (1965) were proposing the hypothesis that traces of the high temperatures that occurred shortly after the

(or a) “big bang” should still be observable, and predicting that they should sound very much like the signal

heard by Penzias and Wilson. This “cosmic microwave radiation” is now considered the basic evidence for

the truth of the “standard model” of the universe.

The history of the biomedical sciences, too, is full of examples of this particular kind of serendipity. Fleming

(1946) discusses the irritation he felt when some kind of mold got into his staphylococcus culture and ruined

the bacteria. He named the mold penicillin. Miller et al. (1955) inadvertently used a four-year-old bottle of

DNA and discovered the hormonal element that provokes cell division in plants. Paul Ehrlich discovered the

acid-fast method of staining tubercle bacilli only because he accidentally lit the stove on which his culture

was resting; somewhat later, Hans Christian Joachim Gram accidentally grabbed the bottle of Lugol's iodine

instead of the gentian violet, and only some of the bacteria (the “Gram-negative” ones) yielded up their purple

color when he washed them off (Beveridge, 1950). And so on.

These historical examples illustrate how one feature of the hypothetico-deductive model of scientific progress

is misleading: Hypothesis testing is not the only research activity in any scientific discipline. Indeed, the most

dramatic discoveries necessarily come about some other way, because in order to test a hypothesis, the

investigator must already know what it is he or she is going to discover.

The majority of nonqualitative methods in the social sciences are designed primarily for the logical testing of

hypotheses.3 Testing hypotheses is a useful, often essential element of research. It is also a useful model for

the training of researchers, for it accustoms the novice to subject his or her predictions to the risk of empirical

refutation.

As social scientists have come to recognize in recent decades, however, hypothesis testing is appropriate to

only a small proportion of the questions they ask. Qualitative research has always retained the proper ideals

of hypothesis-testing research—sound reasoning and the empirical risking of theory. But, in being intrinsically

exploratory, it explicitly departs from certain strictures of the hypothetico-deductive model.

Formal logic, for instance, is not the only kind of sound reasoning. In fact, formal logic possesses certain

flaws, such as its perverse insistence on the analytic “truth” of such statements as “everybody over twelve

feet tall is named Fred,” and “if Durkheim lives, then he is a rock star.” (Formal logic is merely an arbitrary

set of conventions. One of these conventions is that any false statement implies every other statement.)

The prior explicit statement of hypotheses and null hypotheses is not the only way to subject predictions

SAGE

1986 SAGE Publications, Ltd. All Rights Reserved.

SAGE Research Methods

Page 4 of 7 Reliability and Validity in Qualitative Research

to empirical test. Each time Chauncey greets his old friend Ricky, he does expose himself to the unlikely

possibility that he has mistaken a perfect stranger for Ricky. Much social research deliberately seeks out such

“embarrassing” interaction; Agar (1982) has applied the hermeneutic term “breakdown” to these informative

gaffes. The general commitment of qualitative researchers to interacting with their objects of study on the

latter's home ground strongly encourages the discovery that what the researcher takes for granted at his

home does not apply in the new situation. The anthropologist who returns alive from some exotic place must

know something nontrivial about it.

Relaxing certain of the narrow definitions of the hypothetico-deductive model, then, facilitates discovery of the

new and unexpected. It would be an error, however, to drop the scientific concern for objectivity. The scientific

credo is one good way to permit the resolution of a conflict of opinion. It is not the only way; the scholastic

solution, still prevalent in many disciplines called “humanities,” relies on argument and rhetoric rather than

on argument and demonstration. Another alternative is argumentum ad imperium—“might makes right.” One

attractive feature of the scientific solution is that it is an extension of the ordinary processes of inference that

people use in everyday life (Piaget, 1954). As Wilhelm Dilthey pointed out, it is impossible to account for the

observed reality of human interaction without acknowledging that human beings have an innate capacity to

understand one another. Thus striving forever-greater objectivity is as much a part of people's everyday social

inference as it is of their everyday physical inference.

Components of Objectivity

The analogy between qualitative research and other scientific methods and traditions has its limitations. Yet

the ability of practitioners of certain kinds of scientific endeavor to talk about what it is they do is much

more advanced than that of qualitative researchers (Van Maanen, 1979). Indeed, a primary purpose of this

monograph is to remedy that situation. It is often useful to examine methodological formulations from other

traditions to assess their adaptability to qualitative research.

One appropriate and useful device first used in psychometrics (the field of tests and measurements) is the

partitioning of objectivity into two components: reliability and validity. Loosely speaking, “reliability” is the

extent to which a measurement procedure yields the same answer however and whenever it is carried out;

“validity” is the extent to which it gives the correct answer.4 These concepts apply equally well to qualitative

observations.5

A standard physical example of reliability and validity involves the use of thermometers to measure

temperature. A thermometer that shows the same reading of 82 degrees each time it is plunged into

boiling water gives a reliable measurement. A second thermometer might give readings over a series

of measurements that vary from around 100 degrees. The second thermometer would be unreliable but

relatively valid, whereas the first would be invalid but perfectly reliable.

The standard example of the thermometer is neither very qualitative nor very familiar to social scientists. A

SAGE

1986 SAGE Publications, Ltd. All Rights Reserved.

SAGE Research Methods

Page 5 of 7 Reliability and Validity in Qualitative Research

rather homier (if artificial) example occurs when Chauncey sees a blond man across the room at a large

cocktail party, and has the uncertain feeling that he knows him from somewhere. He looks again, sees the

same thing, and continues to have the feelings of uncertainty. Chauncey has perfectly reliable data, and it is

of no use. Is his feeling valid? (As in ordinary language, the technical use of the term “valid” is as a properly

hedged weak synonym for “true.”)

Chauncey might ask himself whether it seems he would know a person who looks like that, moves like that,

dresses like that, and so on. Is the blond, in other words, apparently the sort of person Chauncey would

know? Or he might ask himself subtler questions, such as whether people who look like that frequent the

places he does. At a cocktail party, such a search for validity will probably fail because the guest list is

deliberately socially homogeneous, and any two members are likely to have been in the same other places.

So Chauncey must resort to empirical research if he is to discover whether his feeling is useful.

Perhaps Chauncey's least costly pilot project would be to ask the host what the blond man's name is, or

whether in fact the host has relevant information (e.g., that the blond has just arrived in the country for the

first time from a place Chauncey has never been, or that the three of them had a conversation last week).

Another strategy would be to make ambiguous eye contact with the blond, in such a way as to assign to the

other responsibility for acknowledging the acquaintance. Ultimately, it may prove necessary to confront him

and ask, “Don't I know you?”

If Chauncey devotes as much time to worrying about his problem as it requires to read about it, we would

conclude that he is socially inept, or at least painfully shy. This is one of the problems of methodological

discussion: detailing the inferential steps in getting the job done looks picky and absurd. If we suppose this

computation passes very quickly through Chauncey's mind as he gives the blond a second glance, we might

better empathize with him. When discussing the validity checks of social research, it is useful to remember

that a careful description of what is done generally tends to suggest an obsessive preoccupation with detail on

the part of the researcher. This is an artifact of the fact of description, not a recommendation for compulsive

behavior.

Objectivity, though the term has been taken by some to suggest a naive and inhumane version of vulgar

positivism, is the essential basis of all good research. Without it, the only reason the reader of the research

might have for accepting the conclusions of the investigator would be an authoritarian respect for the person

of the author. Objectivity is the simultaneous realization of as much reliability and validity as possible.

Reliability is the degree to which the finding is independent of accidental circumstances of the research, and

validity is the degree to which the finding is interpreted in a correct way.

Reliability and validity are by no means symmetrical. It is easy to obtain perfect reliability with no validity at all

(if, say, the thermometer is broken, or it is plunged into the wrong flask). Perfect validity, on the other hand,

would assure perfect reliability, for every observation would yield the complete and exact truth.

As a means to the truth, social science has relied almost entirely on techniques for assuring reliability, in part

SAGE

1986 SAGE Publications, Ltd. All Rights Reserved.

SAGE Research Methods

Page 6 of 7 Reliability and Validity in Qualitative Research

because “perfect validity” is not even theoretically attainable. Most nonqualitative research methodologies

come complete with a variety of checks on reliability, and none on validity.

http://dx.doi.org/10.4135/9781412985659.n2

SAGE

1986 SAGE Publications, Ltd. All Rights Reserved.

SAGE Research Methods

Page 7 of 7 Reliability and Validity in Qualitative Research