QBRDM #6

Reliability and Validity In: Reliability and Validity in Qualitative Research

By: Jerome Kirk & Marc L. Miller Pub. Date: 2011 Access Date: November 5, 2019 Publishing Company: SAGE Publications, Inc. City: Newbury Park Print ISBN: 9780803924703 Online ISBN: 9781412985659 DOI: https://dx.doi.org/10.4135/9781412985659 Print pages: 14-21

© 1986 SAGE Publications, Inc. All Rights Reserved. This PDF has been generated from SAGE Research Methods. Please note that the pagination of the online version will vary from the pagination of the print book.

Reliability and Validity

Despite the prestige and success of natural science in recent years, application of science as a model for social “science” is not inevitable. Many have argued that social science has an intrinsically different set of goals that call for an altogether separate collection of methods. Others (nonscientists, it should be noted) contend that recent developments in the natural sciences entirely discredit the fundamental notions (such as objectivity) of an earlier and outdated science.

Yet, whatever their detailed goals, the natural and social sciences share an aspiration to cumulative collective knowledge that is of interest on its own merits to those other than the friends and admirers of its creators. This goal is exactly objectivity. In the natural sciences, objectivity is obtained in two ways. First, experience is reported in such a way that it is accessible to others, for example, when reporting an experiment every effort is made to describe the way the experiment was carried out, just in case somebody else would like to try the same thing. Second, the results of the experiment are reported in terms of theoretically meaningful variables, measured in ways that are themselves justifiable in terms of the relevant theories.

Since Wilhelm Dilthey and George Herbert Mead, the vast majority of social scientists have agreed that objectivity, in this sense, is an admirable goal. Yet, the description of reliability and validity ordinarily provided by nonqualitative social scientists rarely seems appropriate or relevant to the way in which qualitative researchers conduct their work.

It is the purpose of this book to reconcile the means-ends discrepancy. The remainder of this chapter will pursue the argument that, subject to clearly specifiable differences in goals and practice, social science is in every sense of the word fully as “scientific” as physics, and has fully as much need for reliability and validity as any other science.

The “Positivist” View

In recent decades, the social science literature has incorporated a great deal of discussion of an epistemology called “positivism.” (The term is generally employed by those advocating some alternative view of knowledge, and often amounts to a straw man.) In its strongest form, positivism denies objectivity as defined here by assuming not only that there is an external world, but that the external world itself determines absolutely the one and only correct view that can be taken of it, independent of the process or circumstances of viewing. No one seriously defends such an ontology, but scholars attentive to the social and cultural construction of social things (including social science) point out that much research (particularly nonqualitative research) makes sense only in terms of a set of unexamined positivist assumptions.

Most often, these assumptions pertain to the “naturalness” of the measurement procedure employed. Thus a survey researcher may interview a large number of people about their political attitudes, and conclude that

SAGE 1986 SAGE Publications, Ltd. All Rights Reserved.

SAGE Research Methods

Page 2 of 7 Reliability and Validity

“public opinion” says something.

Such an assertion obviously concerns the investigator's theoretical view of the world as much as it does the psychic organization of the interviewees. The investigator's theory contains categories not imposed by the structure of empirical reality. Elements such as “attitudes” and “public opinion” serve rather to organize understanding of the world. Certainly, political and psychological theories that do not use these constructs (or even deny their meaningfulness) are possible, and treating analytic devices as though they are facts is the well-known fallacy of reification.

In response to the propensity of so many nonqualitative research traditions to use such hidden positivist assumptions, some social scientists have tended to overreact by stressing the possibility of alternative interpretations of everything to the exclusion of any effort to choose among them. This extreme relativism ignores the other side of objectivity—that there is an external world at all. It ignores the important distinction between knowledge and opinion, and results in everyone having a separate insight that cannot be reconciled with anyone else's.

Metaphysical polemics, often directed against caricatures of the opposing views, largely miss the point. As is shown in the next chapter, the problem is not so much one of metaphysics as it is a pragmatic question of the validity of measurements. The survey researcher who discusses attitudes is not wrong to do so. Rather, the researcher is wrong if he or she fails to acknowledge the theoretical basis on which it is meaningful to make measurements of such entities and to do so with survey questions addressed to a probability sample of voters.

For any observation (or measurement) to yield discovery, it must generate data that is (a) not already

known and (b) identifiable as “new” by the theory already in place.2 Most of the technology of “confirmatory” nonqualitative research in both the social and natural sciences is aimed at preventing discovery. When confirmatory research goes smoothly, everything comes out precisely as expected. Received theory is supported by one more example of its usefulness, and requires no change. As in everyday social life, confirmation is exactly the absence of insight.

In science, as in life, dramatic new discoveries must almost by definition be accidental (“serendipitous”). Indeed, they occur only in consequence of some kind of mistake.

The Discovery of the New

Henri Becquerel was studying the phenomenon of phosphorescence by exposing metal salts first to the sun and then to photographic plates. When the sky clouded over for an extended period, he tossed the uranium salts into a drawer with his photographic materials and knocked off work for a while (Badash, 1965). “Merde! Je me suis plantér!” he must have muttered when he discovered that the film was ruined, but he was sufficiently prepared and alert to realize that he had discovered radioactivity.

SAGE 1986 SAGE Publications, Ltd. All Rights Reserved.

SAGE Research Methods

Page 3 of 7 Reliability and Validity

More recently, the first men to hear the echo of the origin of the universe thought they were listening to guano. In 1964, Arno Penzias and Robert Wilson set out to measure the radio waves emitted from different latitudes in our disk-shaped galaxy. First, they had to identify what portions of the signal they received originated in the instrument itself. When they received a strong signal in microwave frequencies (where galaxies emit virtually no radiation), their first move was to devote considerable time and expense to cleaning out the “white dielectric material” deposited by pigeons in the antenna throat. This produced only a negligible decrease in signal strength (Penzias and Wilson, 1965). While Penzias and Wilson were in the antenna throat, Dicke et al. (1965) were proposing the hypothesis that traces of the high temperatures that occurred shortly after the (or a) “big bang” should still be observable, and predicting that they should sound very much like the signal heard by Penzias and Wilson. This “cosmic microwave radiation” is now considered the basic evidence for the truth of the “standard model” of the universe.

The history of the biomedical sciences, too, is full of examples of this particular kind of serendipity. Fleming (1946) discusses the irritation he felt when some kind of mold got into his staphylococcus culture and ruined the bacteria. He named the mold penicillin. Miller et al. (1955) inadvertently used a four-year-old bottle of DNA and discovered the hormonal element that provokes cell division in plants. Paul Ehrlich discovered the acid-fast method of staining tubercle bacilli only because he accidentally lit the stove on which his culture was resting; somewhat later, Hans Christian Joachim Gram accidentally grabbed the bottle of Lugol's iodine instead of the gentian violet, and only some of the bacteria (the “Gram-negative” ones) yielded up their purple color when he washed them off (Beveridge, 1950). And so on.

These historical examples illustrate how one feature of the hypothetico-deductive model of scientific progress is misleading: Hypothesis testing is not the only research activity in any scientific discipline. Indeed, the most dramatic discoveries necessarily come about some other way, because in order to test a hypothesis, the investigator must already know what it is he or she is going to discover.

The majority of nonqualitative methods in the social sciences are designed primarily for the logical testing of

hypotheses.3 Testing hypotheses is a useful, often essential element of research. It is also a useful model for the training of researchers, for it accustoms the novice to subject his or her predictions to the risk of empirical refutation.

As social scientists have come to recognize in recent decades, however, hypothesis testing is appropriate to only a small proportion of the questions they ask. Qualitative research has always retained the proper ideals of hypothesis-testing research—sound reasoning and the empirical risking of theory. But, in being intrinsically exploratory, it explicitly departs from certain strictures of the hypothetico-deductive model.

Formal logic, for instance, is not the only kind of sound reasoning. In fact, formal logic possesses certain flaws, such as its perverse insistence on the analytic “truth” of such statements as “everybody over twelve feet tall is named Fred,” and “if Durkheim lives, then he is a rock star.” (Formal logic is merely an arbitrary set of conventions. One of these conventions is that any false statement implies every other statement.) The prior explicit statement of hypotheses and null hypotheses is not the only way to subject predictions

SAGE 1986 SAGE Publications, Ltd. All Rights Reserved.

SAGE Research Methods

Page 4 of 7 Reliability and Validity

to empirical test. Each time Chauncey greets his old friend Ricky, he does expose himself to the unlikely possibility that he has mistaken a perfect stranger for Ricky. Much social research deliberately seeks out such “embarrassing” interaction; Agar (1982) has applied the hermeneutic term “breakdown” to these informative gaffes. The general commitment of qualitative researchers to interacting with their objects of study on the latter's home ground strongly encourages the discovery that what the researcher takes for granted at his home does not apply in the new situation. The anthropologist who returns alive from some exotic place must know something nontrivial about it.

Relaxing certain of the narrow definitions of the hypothetico-deductive model, then, facilitates discovery of the new and unexpected. It would be an error, however, to drop the scientific concern for objectivity. The scientific credo is one good way to permit the resolution of a conflict of opinion. It is not the only way; the scholastic solution, still prevalent in many disciplines called “humanities,” relies on argument and rhetoric rather than on argument and demonstration. Another alternative is argumentum ad imperium—“might makes right.” One attractive feature of the scientific solution is that it is an extension of the ordinary processes of inference that people use in everyday life (Piaget, 1954). As Wilhelm Dilthey pointed out, it is impossible to account for the observed reality of human interaction without acknowledging that human beings have an innate capacity to understand one another. Thus striving forever-greater objectivity is as much a part of people's everyday social inference as it is of their everyday physical inference.

Components of Objectivity

The analogy between qualitative research and other scientific methods and traditions has its limitations. Yet the ability of practitioners of certain kinds of scientific endeavor to talk about what it is they do is much more advanced than that of qualitative researchers (Van Maanen, 1979). Indeed, a primary purpose of this monograph is to remedy that situation. It is often useful to examine methodological formulations from other traditions to assess their adaptability to qualitative research.

One appropriate and useful device first used in psychometrics (the field of tests and measurements) is the partitioning of objectivity into two components: reliability and validity. Loosely speaking, “reliability” is the extent to which a measurement procedure yields the same answer however and whenever it is carried out;

“validity” is the extent to which it gives the correct answer.4 These concepts apply equally well to qualitative

observations.5

A standard physical example of reliability and validity involves the use of thermometers to measure temperature. A thermometer that shows the same reading of 82 degrees each time it is plunged into boiling water gives a reliable measurement. A second thermometer might give readings over a series of measurements that vary from around 100 degrees. The second thermometer would be unreliable but relatively valid, whereas the first would be invalid but perfectly reliable.

The standard example of the thermometer is neither very qualitative nor very familiar to social scientists. A

SAGE 1986 SAGE Publications, Ltd. All Rights Reserved.

SAGE Research Methods

Page 5 of 7 Reliability and Validity

rather homier (if artificial) example occurs when Chauncey sees a blond man across the room at a large cocktail party, and has the uncertain feeling that he knows him from somewhere. He looks again, sees the same thing, and continues to have the feelings of uncertainty. Chauncey has perfectly reliable data, and it is of no use. Is his feeling valid? (As in ordinary language, the technical use of the term “valid” is as a properly hedged weak synonym for “true.”)

Chauncey might ask himself whether it seems he would know a person who looks like that, moves like that, dresses like that, and so on. Is the blond, in other words, apparently the sort of person Chauncey would know? Or he might ask himself subtler questions, such as whether people who look like that frequent the places he does. At a cocktail party, such a search for validity will probably fail because the guest list is deliberately socially homogeneous, and any two members are likely to have been in the same other places. So Chauncey must resort to empirical research if he is to discover whether his feeling is useful.

Perhaps Chauncey's least costly pilot project would be to ask the host what the blond man's name is, or whether in fact the host has relevant information (e.g., that the blond has just arrived in the country for the first time from a place Chauncey has never been, or that the three of them had a conversation last week). Another strategy would be to make ambiguous eye contact with the blond, in such a way as to assign to the other responsibility for acknowledging the acquaintance. Ultimately, it may prove necessary to confront him and ask, “Don't I know you?”

If Chauncey devotes as much time to worrying about his problem as it requires to read about it, we would conclude that he is socially inept, or at least painfully shy. This is one of the problems of methodological discussion: detailing the inferential steps in getting the job done looks picky and absurd. If we suppose this computation passes very quickly through Chauncey's mind as he gives the blond a second glance, we might better empathize with him. When discussing the validity checks of social research, it is useful to remember that a careful description of what is done generally tends to suggest an obsessive preoccupation with detail on the part of the researcher. This is an artifact of the fact of description, not a recommendation for compulsive behavior.

Objectivity, though the term has been taken by some to suggest a naive and inhumane version of vulgar positivism, is the essential basis of all good research. Without it, the only reason the reader of the research might have for accepting the conclusions of the investigator would be an authoritarian respect for the person of the author. Objectivity is the simultaneous realization of as much reliability and validity as possible. Reliability is the degree to which the finding is independent of accidental circumstances of the research, and validity is the degree to which the finding is interpreted in a correct way.

Reliability and validity are by no means symmetrical. It is easy to obtain perfect reliability with no validity at all (if, say, the thermometer is broken, or it is plunged into the wrong flask). Perfect validity, on the other hand, would assure perfect reliability, for every observation would yield the complete and exact truth.

As a means to the truth, social science has relied almost entirely on techniques for assuring reliability, in part

SAGE 1986 SAGE Publications, Ltd. All Rights Reserved.

SAGE Research Methods

Page 6 of 7 Reliability and Validity

because “perfect validity” is not even theoretically attainable. Most nonqualitative research methodologies come complete with a variety of checks on reliability, and none on validity.

http://dx.doi.org/10.4135/9781412985659.n2

SAGE 1986 SAGE Publications, Ltd. All Rights Reserved.

SAGE Research Methods

Page 7 of 7 Reliability and Validity