8010week 8
Educational Evaluation and Policy Analysis Winter, Vol. 6, No. 4, 1984, pp. 379-391
The Problem of Criteria for Judging Interpretive Inquiry
John K. Smith University of Northern Iowa
Over the years various interpretive (or qualitative, naturalistic, etc.) investiga- tors have been concerned with develop- ing criteria for judging, and defining pro- cedures for doing, this type of investiga- tion. Apparently the aim is to establish some set of foundational criteria to sepa- rate the good interpretive study from the not so good. These criteria and procedures would help advocates of this approach refute charges of subjectivity and allow them to lay claim to that certitude that is available, at least as it is generally pre- sumed, to their quantitatively oriented counterparts. Given the assumptions con- ventionally made to distinguish interpre- tive from quantitative inquiry, this paper takes the position that the quest for foun- dations may be a misguided one. If the assumptions and their implications are taken seriously, arguments that attempt to establish such nonarbitrary criteria and/or procedures will, perhaps inevita- bly, be marked by confusion and incon- sistency. To defend this contention, this paper will begin with a brief historical overview of the issue. This discussion will then be followed by a specific example of the problem and an examination of why inconsistency mars our attempts to de- velop criteria given certain assumptions.
T h e a u t h o r wishes to t h a n k Lous Heshusius, Jack Yates, Joe Blase, and three a n o n y m o u s reviewers for their insightful c o m m e n t s of earlier versions of this paper.
Historical Context In that this issue of assumptions and
criteria (and thus the problem of certi- tude) is not a new one for this area of interpretive inquiry, a brief historical note will help to clarify the nature of the problem. One might easily argue that the tenor of our present discussions was es- tablished around the turn of this century by, among others, Dilthey and Weber. Dil- they deserves mention in that he pre- sented one of the first serious and sus- tained challenges to positivism and, in doing so, elaborated a different approach to social inquiry based on assumptions quite different from those of positivism (Hodges, 1944, 1952; Smith, 1983). Weber merits attention in this instance not for his numerous important achievements, but for the legacy left by his substantial, but nonetheless unsuccessful, attempt to develop a social science that combined a focus on subjective meaning with the de- sire for universally valid judgments (Aron, 1967, pp. 185-216).
Dilthey argued that social inquiry should not be conducted with the meth- ods of the physical sciences because there was a fundamental difference in subject matter. Whereas the physical sciences dealt with a series of inanimate objects that could be seen as existing independ- ently of us (a world of independent but objectively knowable facts), the cultural sciences focused on the products of hu- man minds. Those products were inevit-
379
380 John K. Smith
ably and intimately connected to human minds—including those of the investiga- tors—with all of their emotions, values, subjectivity, and so on. Accordingly, since society was the result of conscious human intention, the interrelationships of what was investigated and the investigator were impossible to separate. For Dilthey, there simply was no objective social real- ity as such, that is, divorced from the people who participate with others in con- structing or interpreting reality. Bergner (1981), in his discussion of Dilthey, has put the situation as follows: "We cannot meaningfully stand outside of the events and occurrences of life and say with cer- tainty what they signify. What we can know, however, is that historical and so- cial events do signify things to those who enact them and to those who endure them" (p. 64).
Dilthey clearly believed that positivism erred by making the discovery of social laws, analogous to physical laws, the prin- ciple goal of social inquiry. He said that, if nothing else, the complexity of the so- cial makes it impossible to find such laws. More important, he also argued that to find laws should not even be the point of the cultural studies. The primary concern in this case was understanding (verste- hen). This point was the basis for his dis- cussion of the ideas of lived experience and that understanding required a her- meneutical perspective. In his discussion of the latter idea he noted that, at least at one level, to understand the experiences of another required a constant movement or interplay between the individual expression and the context (hermeneuti- cal circle)—a process with no absolute beginning or absolute ending points.
It is here, however, that Dilthey faced a problem that he was unable to resolve. If meaning must be taken in context and understanding must be hermeneutical, clearly the interpretation of expressions could vary. Given this position, could there be any such thing as a correct inter- pretation? Dilthey could find no founda- tional standards to apply for sorting out conflicting possibilities and, at the same time, he rejected an appeal to transcen- dental values or to a metaphysical solu- tion. Whereas positivism had developed standards against which to judge true
from false or correct from incorrect, Dil- they was caught between assumptions that are epistemologically antifounda- tional and a desire for criteria that are foundational. In other words, his idealist tendencies led to a relativism which he found unacceptable, but from which he could find no escape (Hughes, 1958, p. 199).
Even though Weber obviously disa- greed with Dilthey in various ways (e.g., Weber felt the two sciences differed not because of an inherent subject matter dif- ference, but rather because of a different interest taken in the subject matter), he was influenced by Dilthey's ideas of un- derstanding and interpretation (Roche, 1973; Wrong, 1970). However, Weber re- alized that this idealist-oriented position, in contrast to a realist-oriented positivist position, brought with it a problem of certitude. He therefore attempted to find a middle ground, in the sense of a synthe- sis, between the two approaches. In his conception the problem was that a Dil- thean approach paid too little attention to existent social reality, whereas the other perspective devoted too little attention to the ideas of subjective meaning and social reality as an interpreted reality. Aron (1967) has concisely summarized Weber's project in this area: "How can there be an objective science—one not distorted by our value judgments—of the value- charged productions of man? This is the central question Weber asked himself and to which he tried to provide an answer" (p. 193).
The answer Weber provided cannot be discussed in all of its detail and richness. Suffice it to say that one of the central tenets of his argument (along with the concept of "ideal type") was the separa- tion of the idea of value relevance from that of value judgment. Weber accepted that social reality is meaningful only to the extent that it is based on values. More- over, he said that "without the investiga- tor's evaluative ideas, there would be no principle of selection and no meaningful knowledge of the concrete reality" (1949, p. 82). Yet, at the same time he argued that universally valid knowledge is pos- sible because "all research in the cultural sciences . . . once it is oriented to a given subject matter . . . and has established its
Judging Interpretive Inquiry 381
methodological principles, will consider t h e analysis of data as a n end in i t s e l f (1949, p. 112). In other words, Weber held that even though values are t h e basis for m e a n i n g a n d are crucial to t h e content of inquiry, this does not m e a n that t h e in- quirer must evaluate or m a k e value judg- m e n t s about w h a t is investigated.
That Weber's arguments for a distinc- tion b e t w e e n value relevance and value judgment, and m o r e generally for t h e sep- aration of facts and values, cannot be sus- tained (and even ends in contradiction), is not unrecognized. Various scholars, ar- guing from a r a t h e r wide variety of differ- ent positions, h a v e m a d e substantial cri- tiques of Weber's distinctions and thereby, at least indirectly, of his attempt to synthesize the two perspectives (see, e.g., Habermas, 1968; Hindess, 1977; Lu- kacs, 1971; Rorty, 1982; Strauss, 1953). Simey (1969), however, has put the situ- ation as clearly as anyone: "Weber, t h e r e - fore, was attempting to find a via media between, on the one hand, realism (in its positivistic form), and, on the other, phil- osophical idealism. He accepted both al- ternatives, in t h e sense that h e sought to establish a synthesis b e t w e e n t h e m . But t h e synthesis in fact eluded h i m " (p. 83). T h e question of importance for our pur- poses is, w i t h w h a t consequences for the development of social inquiry?
Two things, very m u c h related, follow from Weber's "failure of synthesis." First, his inability to find a via media m a y well indicate that such a position cannot be found. Certainly since his time social in- quiry has b e e n characterized far less by synthesis and far more by "an e n d e m i c conflict b e t w e e n competing frameworks" (Outhwaite, 1983, p. 2). T h e two ap- proaches have moved along separate lines, differing not only in t e c h n i q u e , b u t also in terms of their philosophical tem- p e r a m e n t s or "logics of justification." What w e refer to as t h e quantitative ap proach is based on an external perspec- tive—facts separate from values, t r u t h as a m a t t e r of correspondence to an inde- p e n d e n t l y existing reality, and t h e idea that w e may adopt a "God's eye" point of view. T h e qualitative perspective is based on an internalist position—truth as so- cially and historically conditioned agree- ment, social reality as m i n d constructed,
and the idea that t h e only point of view w e can take is that of various people based on their various interests, purposes, and so on (Putnam, 1981, pp. 49-74).
Second, in one way Weber's failure is an expression, albeit in different form, of the Dilthean problem of combining as- sumptions that are epistemologically an- tifoundational (meaning as subjective, etc.) with the attempt to find epistemolog- ically foundational criteria (certitude). Weber's via media was really a matter of bringing together a frame of reference based on subjective choice and proce- dures that would yield universally valid judgments. In other words, how can one maintain or have the kind of certitude claimed by and generally granted to the realist perspective in the face of a concern over meaning and intention? Dilthey's problem became, even if in a different form, Weber's problem and, more impor- tant for our purposes, it is the problem that continues to be at the center of any discussion of interpretive approaches to social inquiry.
Finally, one additional c o m m e n t is needed about this historical context. Over the years n u m e r o u s schools of thought h a v e b e e n put forth as t h e basis for t h e interpretive forms of inquiry. T h e n a m e s of these schools are quite well-known (phenomenological, ethnomethodologi- cal, h e r m e n e u t i c a l , etc.). Clearly it would be a mistake not to recognize that t h e r e are differences among these schools. However, it would also be a mistake, pos- sibly even a more serious one, not to rec- ognize their similarities. A good case can be m a d e that the " i n s i g h t s . . . from [these] schools of t h o u g h t . . . stand close to phil- osophical idealism" (Giddens, 1976, p. 155). They all, in at least some important aspects such as the idea that meaning must be grasped hermeneutically, are part of a broad m o v e m e n t away from empiri- cism along the lines of an idealist orien- tation. As Benton (1977) s u m m a r i z e s the situation, "The philosophical sources of h u m a n i s t approaches in the social sci- ences at first glance appear to be very heterogeneous—phenomenological, eth- nomethodological, existentialist. . . But it t u r n s out that most of these philosophical traditions are related, through either his- torical origin, or conceptual affinities or
382 John K. Smith
both, to the work of. . . Kant" (p. 101; see also, Bernstein, 1976; Bleicher, 1982; Hughes, 1980; Levy, 1981; O u t h w a i t e , 1975, 1983; Psathas, 1973; Roche, 1973).
A Contemporary Example
To clarify, in more tangible terms t h a n have b e e n employed u p to this point, the n a t u r e of this problem of criteria, an ex- ample of some length and detail is neces- sary. While m a n y people have discussed, in different forms and with different de- grees of aggressiveness, the topic of crite- ria a n d / o r procedures for interpretive in- quiry (e.g., Bogdan & Biklen, 1982; Bruyn, 1966; Glaser & Strauss, 1967; LeCompte & Goetz, 1982; Johnson, 1975; Wolcott, 1975), an article by Guba (1981) merits special attention (see also, Guba, 1978; Guba & Lincoln, 1981, 1982a, 1982b, 1982c). Unlike some authors, Guba dis- cusses some of the specific assumptions underlying a naturalistic approach and t h e n elaborates the criteria to be used and the procedures to be employed for this mode of inquiry. His article is one of the more precise and concise t r e a t m e n t s of this topic to be found, especially in the educational research literature. Hence, Guba's work will serve as an excellent vehicle with w h i c h to demonstrate the problems that can occur w h e n one ac- cepts certain assumptions and t h e n at- tempts to elaborate certain criteria for judging naturalistic research.
Guba (1981) begins by noting that the rationalist and naturalist paradigms are based on different assumptions (pp. 7 6 - 79). Rationalism assumes that a single reality exists, w h e r e a s naturalism as- sumes there are multiple realities. T h e relationship of the investigator to w h a t is investigated is a n o t h e r area of contrast. In rationalism the object of study is seen as existing i n d e p e n d e n t l y of the investiga- tor. Reality is assumed to be self-subsis- tent, a separate entity w h e t h e r or not it is an object of interest or study. In natural- ism, the investigator and w h a t is investi- gated are seen as i n t e r d e p e n d e n t . Accord- ing to this view, w h a t we treat as real is in large m e a s u r e m i n d - d e p e n d e n t as op- posed to m i n d - i n d e p e n d e n t . As Guba puts it, "The rationalistic paradigm rests on t h e assumption that the inquirer can main- tain a discrete (and discreet) distance from
the objects of t h e inquiry . . . the relation- ship . . . is essentially one of independ- ence" (p. 77), w h e r e a s for naturalism, "the inquirer a n d t h e r e s p o n d e n t . . . are inter- related, with each influencing the other" (p. 77).
Another important difference b e t w e e n the two paradigms concerns t h e n a t u r e of "truth statements." In this case the ration- alist paradigm assumes that generaliza- tions—"enduring t r u t h statements that are context free" (p. 77)—are possible. Naturalism, however, holds that such generalizations are not possible and "that at best w h a t one can hope for are 'working hypotheses' that relate to a particular con- text" (p. 77). T h e implication of this dif- ference in position is that while a n e u t r a l (in the sense of nonevaluative) posture is possible from a rationalist perspective, es- pecially in regard to physical science, from a naturalist perspective neutrality is "impossible w h e n the objects of investi- gation are people" (p. 78). T h e reasons for this impossibility are that h u m a n behav- ior is never context free and that both respondent and investigator exert a recip- rocal influence on one another.
Guba has found the two paradigms dif- fer significantly in terms of their basic assumptions. He s u m m a r i z e s the under- lying assumptions of the naturalistic ap- proach quite well w h e n he says, "Social/ behavioral p h e n o m e n a exist chiefly in the minds of people, and t h e r e are as m a n y realities as persons" (p. 77). Social reality is m i n d - d e p e n d e n t in the sense that people (including investigators) shape or construct reality and people (in- cluding investigators) may construct their realities in their own different ways at different times and places.
Given such a world of multiply con- structed realities, how is it possible to find some versions of reality "trustworthy" and others not? Although he does not specifically define the term trustworthy, Guba does discuss four major aspects of it (pp. 79-82). First, the findings of a trust- worthy study exhibit "isomorphism to the respondents' perceptions" (p. 80). In other words, the investigator's statements ac- curately reflect the respondents' percep- tions. Second, the findings are neutral in that they are a "function solely of subjects (respondents) and conditions of t h e in-
Judging Interpretive Inquiry 383
quiry and not of the biases, motivations, interests, perceptions, and so on of the inquirer" (p. 80). Third, if the inquiry is repeated with the same or similar sub- jects, the findings should be consistent with those of the first inquiry. Fourth, the results must be transferable to other sim- ilar situations. Taken together these points seem to represent an attempt to separate inquiry that tells us how people really perceive their own situations from inquiry that distorts "reality" by the in- volvement of the investigator's percep- tions, biases, and interests. Using this interpretation of trustworthiness, Guba then describes a series of procedures, such as member checks, structural corrobora- tion, and dependability audits, which are used in the naturalist mode to deal with this problem.
Given his assumptions, what Guba is attempting to do by specifying criteria is not possible. The assumptions of multiple realities and reality as mind-involved se- riously undermine the notion of applying foundational criteria to distinguish trust- worthy results from untrustworthy ones. These assumptions and foundational cri- teria are, in a word, incompatible. To ac- cept that social reality is mind-con- structed and that there are multiple real- ities is to deny that there are any "givens" upon which to found knowledge. If one accepts these assumptions, different claims about reality result not from incor- rect procedures but may simply be a case of one investigator's interpretation of real- ity versus another's. In a world of multi- ply constructed, mind-dependent reali- ties, there may be no "court of last resort" to appeal to to sort out trustworthy inter- pretations from untrustworthy ones.
Some examples of the inconsistencies that result from accepting antifounda- tional assumptions while attempting to pose foundational criteria can now be pre- sented. Guba's problems with consistency are clearly evident in his varied use and definition of the term criterion. By formal definition criterion can mean slightly dif- ferent things. Among other possibilities, the term may refer to a "characterizing trait." In this usage it has, at best, mild implications as a prescription for inquirer behavior and does not necessarily refer to something that is held to be foundational.
Criterion may also, however, mean "a standard against which to make a judg- ment." Here the term brings with it a very strong implication of something founda- tional—it is that touchstone that can be employed to sort out the good from the bad, the correct from the incorrect.
In some instances Guba seems to be employing the former definition and seems aware of the antifoundational im- plications of his assumptions. Toward the end of the article he says his criteria are not prescriptions and that he does not want them to become an orthodoxy (p. 90): "But—most emphatically—the [cri- teria] are not prescriptions of how inquiry must be done" (p. 90). If this is the case, then he is discussing simply the charac- terizing traits of a particular approach to inquiry and saying that these criteria are merely the way inquirers seem to do it. The elements he discusses cannot there- fore be thought of in such terms as mini- mums, absolutes, and essentials.
Throughout the paper, however, there is considerable evidence that Guba de- sires criteria to carry more weight than as simply characterizing traits. His first sen- tence demonstrates that he is concerned with "criteria for judging the trustworthi- ness of inquiries conducted with the na- turalistic inquiry paradigm" (p. 75). In fact, the entire paper is directed at dem- onstrating that naturalistic researchers also have standards. For example, Guba poses his criteria as equivalent to or par- allel to those of a rationalist paradigm. The tone is one of "we are equally well protected from the problems of subjectiv- ity." More specifically, he uses terms such as "newer rules" (p. 90), implies that his efforts are an attempt to "codify the safe- guards" (p. 76), and says that he has elab- orated criteria that are to be generally applied by editors, referees, and research- ers to sort out the trustworthy from the untrustworthy: "When a naturalistic study is to be judged, it is these criteria that ought to be brought to bear" (p. 88). This is all rather strongly put and defi- nitely goes beyond describing simply how inquirers proceed through this type of study. Even though at times his use of language makes it difficult to discern his exact meaning or intention, it is clear that he desires standards that are neutral and
384 John K. Smith
nonarbitrary. In the end, his antifounda- tionalist assumptions push his use of cri- teria in one direction, while his desire for "something to hang on to" pushes it in the other direction.
A second major area of inconsistency involves the status of t h e procedures Guba lists. At one point he seems to rec- ognize the epistemological consequences of his assumptions, but at other points he does not. He contends that procedures such as triangulation, peer debriefing, and prolonged engagement will not guarantee that a naturalistic study will be trustwor- thy. These procedures do not constitute, as Guba feels is the case for their ration- alist counterparts, a "theoretically com- plete w h o l e " and will therefore not in- sure, but only increase, the probability of a study's trustworthiness (p. 88). How- ever, shortly after discussing these pro- cedures in terms of "ambiguity" and "in- d e t e r m i n a n c y " (p. 88), he gives some of the m e m b e r s on the list the status of sine qua nons (p. 88). As he notes, "It is incon- ceivable that one would be persuaded of t h e trustworthiness of a study that in- volved no triangulation and no m e m b e r checks" (p. 88). This is quite clearly an attempt to pose an absolute (a necessary if not sufficient condition) on w h i c h to found naturalistic inquiry. T h e establish- m e n t of absolute conditions for trustwor- thiness negates the fact that only indeter- minate procedures can follow from the relativism of multiply constructed reali- ties. T h e assumptions on w h i c h natural- istic research is based are violated by pos- iting procedures that are indispensable. T h e r e is a great deal of difference b e t w e e n describing w h a t one might do and man- dating w h a t one must do.
Inherent in Guba's assumptions and specifically stated by him (p. 78), is the significant idea that the investigator can- not be neutral. This idea immediately pro- vokes problems since earlier h e argues for neutrality as one of the key elements for trustworthiness. This point aside, this is- sue of neutrality appears w h e n Guba at- tempts to preserve the idea of inquirer "distance" from the subject matter and even claims there is such a thing as "op- timal distance" (p. 77). Once it is granted that reality is constructed by people (among w h o m are investigators), this
prospect of maintaining distance is very difficult to defend. To do so, the object of study must exist i n d e p e n d e n t l y of the in- quirer, and the inquirer must have inde- p e n d e n t access to that object. Once the investigator is seen as mind-involved with the construction of reality that he or she investigates, there is no way for the m i n d to stand apart (that God's eye view) and measure distance. The mind, after all, cannot be distanced from itself. Once one grants that reality is m i n d - d e p e n d e n t and that there are as m a n y realities as t h e r e are persons, it follows that it may be im- possible to sort out w h o has maintained proper distance and w h o has not.
Guba's work therefore provides an ex- cellent example in tangible terms of w h a t happens w h e n antifoundational assump- tions are combined with the attempt to pose foundational criteria. T h e inevitable result is inconsistency of argument. It is now time to discuss w h y this result is inevitable.
The Issue of Assumptions and Criteria
Given the assumptions of naturalistic or interpretive inquiry, w h y is it impos- sible to develop and apply foundational criteria to this type of investigation? To answer this question, it is necessary to review briefly how the dominant mode of inquiry—quantitative—has dealt with this issue of assumptions and criteria. One reason for this digression is that interpre- tive inquiry, at least since the time of the challenge initiated by Dilthey and others to positivism, has very often been para- sitic (i.e., taken its strength from and de- veloped through critique of the quantita- tive approach). As Rorty (1979) notes, "Hermeneutics is always parasitic upon the possibility of epistemology" (p. 365). Therefore, to u n d e r s t a n d the claim that there can be no criteria to reconcile dis- course (to sort out the trustworthy from the u n t r u s t w o r t h y results) for interpre- tive inquiry, we must first e x a m i n e the opposing claim that t h e r e are indeed foundations on w h i c h to base epistemo- logical claims.
A n u m b e r of assumptions, some of w h i c h are often implicit, are crucial to the assertion that t h e r e are nonarbitrary criteria for judging and nonarbitrary pro- cedures for doing quantitative inquiry.
Judging Interpretive Inquiry 385
Even though other terms, such as exter- nalism, objectivism, and so on, could be used here (Putnam, 1981, pp. 49-50; Ru- binstein, 1981, pp. 24-25), for our pur- poses the assumptions of this approach can be located within the general philo- sophical perspective or temperament known as realism. A most critical feature of a realist perspective is the idea that "what exists does so whether men con- ceive of it or not" (Trigg, 1980, p. vii). In other words, reality is self-subsistent; it exists "out there" independently of us and of our interest in it. Following the lead of Durkheim (1938), social and educational facts are considered external to us, exist- ing prior to an investigator's study of them, awaiting discovery. While this po- sition grants that minds are necessary to conceptualize reality (by definition only minds can do so), it does not accept the idea that reality is mind-dependent in the sense of being created or constructed. As Trigg notes, "The realist must hold that the mind and the world are separate items" (p. 93).
Given this separation of the knower from what is or can be known, the basic epistemological thrust of realism is that to know reality is to be able to describe or reflect it accurately. "According to most realists . . . a judgment is true when it corresponds to an independent reality and false when it does not" (Ewing, 1974, p. 195). Not surprisingly, this position is la- beled as the correspondence theory of truth. True statements are judged to have accurately reflected the qualities and characteristics of what is out there, whereas false ones are those judged to have in some way distorted the nature of that independently existing reality. For example, if one says the cat is on the mat and the cat is observed to be there, we accept that one has produced a true state- ment. Similarly, if one says that school performance and self-concept are posi- tively related in first graders and on ob- servation this turns out to be the case, one has made a statement we judge to be true. The significant point is that truth has its source in an independently existing real- ity—a reality that can be known for what it really is.
As these examples demonstrate, the process of observation, or empirical veri-
fication, is essential to our ability to judge whether or not statements are true. How- ever, there are many ways of observing the world, and some are considered less subject to error and distortion than others. From the quantitative perspective, one of the greatest dangers to sound empirical verification is subjectivity. Paraphrasing from Nagel's (1981) discussion of the topic, to be subjective is to see the world from one's own particular place within it (pp. 196-213). To approach the world in this way is to let one's personal values and dispositions distort or bias an accurate description of what really exists. The re- sult of such subjectivity, if allowed to dominate our views of the world, would allow that everyone could create his or her own little private reality. To avoid this danger, the demand of quantitative inquiry is for objectivity—that we take a God's eye point of view (Putnam, 1981, p. 49)—or see the world from no particular place within it. How this approach is able to insure objectivity requires that inves- tigators adhere to method, in particular the scientific method.
First and foremost, the methods we con- ventionally think of as scientific serve to constrain the investigator and hence pro- tect him or her from the perils of distort- ing the qualities and characteristics of the object of study. At the level of the practice of inquiry, adherence to these procedures will insure that the study has been prop- erly conducted. And if this is the case, we must accept the results, regardless of how we might feel about them. In other words, these are results to which all "rational" people must accede, and disagreements can be reconciled by a properly done ap- peal to an independent reality. These pro- cedures are neutral and nonarbitrary in that they stand over and beyond any one individual's interests, dispositions, or place in the world. The assumptions of this approach allow for a foundational epistemology, one that holds we can know reality for what it really is and that we have criteria for judging, and procedures to employ as we go about discovering, that reality.
A brief restatement of these issues in slightly different terms will be helpful. The ideal of verification and of a founda- tional epistemology is dependent on the
386 John K. Smith
possibility of building knowledge free from a never-ending circle of interpreta- tion (Taylor, 1971). The nonarbitrary ver- ification of propositions requires that there be such a thing as "brute data" or "givens." These data, and the inferences based on them, stand on their own in the sense that they are "data whose validity cannot be questioned by offering another interpretation or reading, data whose va- lidity cannot be founded on or under- mined by further reasoning" (Taylor, 1971, p. 8). Therefore, to have founda- tional criteria requires one to assume that there is an independent reality out there that can be read for what it is, and, if this is the case, then the claim of certitude may follow.
The assumptions of naturalistic in- quiry—that reality is mind-dependent and that there are multiple realities—are incompatible with the idea of an inde- pendently existing reality that can be known through a neutral set of proce- dures. While various labels, such as inter- nalism, subjectivism, and so on, can be employed here, this approach to inquiry can be discussed within the context of an idealist philosophical temperament. Even though there are several forms of ideal- ism, for our purposes suffice it to say that they all share one point in common: "To the idealist it seems inconceivable that anyone should talk of knowledge in any intelligible sense if the supposed object of knowledge is wholly independent of the knower—in which there is not in some way and in some degree mutual involve- ment of mind and object" (Urban, 1949, p. 16). There is no independent reality or data free from interpretation on which to found our knowledge.
The most striking epistemological con- sequence of this perspective is that cor- respondence or accuracy of reflection is not an acceptable way to define truth. The idea of a mind-dependent reality is in- compatible with the correspondence the- ory. For interpretive inquiry, the basis of truth or trustworthiness is social agree- ment; what is judged true or trustworthy is what we can agree, conditioned by time and place, is true or trustworthy.1 This
1 A coherence theory of t r u t h is often associated with an idealist perspective. Simply put, for this
appeal to common understanding as op- posed to correspondence results in defi- nitions of such basic terms as objective and subjective that are quite different from those used in the quantitative ap- proach to inquiry. To be objective is not a matter of representing things as they really are. As Rorty (1979) notes, "The application of such honorifics as 'objec- tive' and 'cognitive' is never anything more than an expression of the presence of, or the hope for, agreement among in- quirers" (p. 335). Similarly, to be subjec- tive does not mean representing things influenced by the emotions, interests, and so on; it is to "bring in considerations which others think beside the point" (Rorty, 1979, p. 339). What we agree with and on is objective; what we find unfa- miliar we label subjective.
The implications of this idea of agree- ment can be seen if we imagine a situation in which competing claims have been made. From a realist/quantitative per- spective this conflict can be resolved by referring to the extralinguistic facts—that external, independently existing reality. From an idealist/interpretive perspective, however, reference to the facts will not provide a solution, since disagreement is not founded on "error of reflection or cor- respondence" but is simply the result of interpreting things in a different way. While the different interpretations may be subject to questions of internal consis- tency (coherence), there is no nonarbi- trary set of criteria that can be employed to sort out the difference and no brute data to appeal to to reconcile the issue. In a world where "social/behavioral phe-
theory a proposition is judged true if it coheres (is connected and consistent) with other propositions in a scheme or network. Coherence is t h u s a matter of internal relations. One question arises, however, that leads back to the idea of agreement as the basis for truth. That is, how can one choose from among various coherent schemes? As Grayling (1982) puts the problem, "It is logically possible for there to be any n u m b e r of internally coherent systems of be- liefs, and since there is no external criteria for choos- ing among them, it cannot be k n o w n which is the 'right' one" (p. 137). Since idealism holds that there is no independent reality to w h i c h w e have inde- p e n d e n t access (no facts of an extralinguistic nature), a coherence approach must ultimately respond to the idea of common understanding. A scheme or network is held to be true only to the extent we have come to agree, conditioned by time and place, that it is true.
Judging Interpretive Inquiry 387
nomena exist chiefly in the minds of peo- ple, and there are as many realities as persons" (Guba, 1981, p. 77), we are faced with the prospect of one interpretation of reality coexisting with another interpre- tation of reality. No safeguards, rules, or absolute minimums are available to work out our differences. Clearly this position is epistemologically antifoundational.
What does all of this mean for the prac- tice of naturalistic inquiry? According to Taylor (1971), the sciences of man are hermeneutical. A hermeneutical science is directed at interpreting the intentional, meaningful expressions of people and pro- vides interpretations of interpretations people have already given to their own situations. Furthermore, a defining char- acteristic of a hermeneutical science is that it is, in principle, impossible to de- velop a precise and nonarbitrary way to verify results. All that an investigator can do is offer an interpretation (of interpre- tations) and hope others will "catch on" or accept those readings. If they do not, nothing outside of the hope for common understanding can be used to compel agreement. As Taylor puts it, "We can only convince an interlocuter if at some point he shares our understanding of the language concerned. If he does not, there is no further step to take in rational ar- gument; we can try to awaken these in- tuitions in him, or we can simply give up; argument will advance us no further" (p. 6).
Hermeneutics and the idea of the her- meneutical circle have been subject to various interpretations over the years.2
Generally speaking, these terms mean that any reading by the inquirer (or lay- person for that matter) is the product of an interpretive movement back and forth
2 As used here, h e r m e n e u t i c s is not a successor to method as m e t h o d is defined for the quantitative mode of inquiry. H e r m e n e u t i c s is not an alternative, but equally foundational way to obtain the truth. Even though there have been attempts (from Dilthey to Betti) to m a k e h e r m e n e u t i c s the method of the cultural sciences, I think these efforts have not been successful. Thus, the position h e r e is that of Gada- mer (1975): "Hermeneutics . . . is not . . . a m e t h o d - ology of the h u m a n sciences, but an attempt to u n d e r s t a n d w h a t the h u m a n sciences truly are, be- yond their methodological self-consciousness" (p. xiii). See Palmer (1969) for an excellent discussion of these issues.
between parts and whole. In the interpre- tation of a text, for example, individual sentences can only be understood in re- lation to the whole text, and the whole text can be understood only in relation to the individual expressions. To give a read- ing to an expression requires the appli- cation of mental categories to provoke an understanding of the expression and of the context of the expression. As Lessnoff (1974) notes in a discussion based on the work of Melden (1961), "In order to see an event as a human action, it is necessary to interpret its empirically observable fea- tures in terms of mental categories, to assume the applicability of these cate- gories to what is observed" (p. 37). In interpretive understanding, a constant movement back and forth between parts and whole is necessary. However, this process responds to no formal rules of procedure and has no absolute beginning point and no absolute ending point. There is no time at which one can say with assurance, or in a foundationalist manner, "It is over and I have got it right."
The implications of this hermeneutical- interpretive approach to inquiry can be seen in a situation in which one person does not accept the readings of another. Will it not be easier to convince the other person if the inquirer has used some pro- cedures as opposed to others in achieving the result? Not necessarily. If the results are based on particular procedures, such as Guba's member checks and so on, these will be convincing only if these proce- dures make sense to the reader. The prob- lem is that what makes sense depends on how one reads the situation—not only the general situation of interpretive inquiry, but the specific situation of any particular inquiry; not only in terms of the actual interpretation, but also in terms of how the interpretation was obtained. As Tay- lor (1971) notes in a slightly different con- text, "Ultimately, a good explanation is one which makes sense of the behavior; but then to appreciate a good explanation, one has to agree on what makes good sense; what makes good sense is a func- tion of one's readings; and these in turn are based on the kind of sense one under- stands" (p. 14).
What all of this means is that we have no way, in a foundational sense, to know a good interpretation from a bad interpre-
388 John K. Smith
tation. One can try to convince another of the appropriateness of a particular read- ing, but this can only be done through an appeal to common understanding. If such an understanding does not exist, there is no "higher authority" to which one can appeal to resolve the issue. Naturalistic inquiry then, given these idealist assump- tions, begins to look very relativistic. And, once accepted, the idea of relativism can- not be suspended in the quest for foun- dational criteria by which to judge the results of this form of inquiry.
Member Checks as Absolute Minimum A more specific discussion of one pro-
cedure advocated by Guba should clarify some of the preceding points. Guba (1981) says that member checks are an absolute minimum for naturalistic inquiry (p. 88) and that they are the most important thing a naturalistic inquirer can do (p. 85). This is the same as saying the procedure is foundational in that it is a necessary, if not sufficient, condition for trustworthi- ness. If an inquirer has confirmed his or her interpretations with the subjects (and produced a record of how this process has influenced the results), the study has a chance of being judged trustworthy. If not, it is unlikely the results will be granted such status.
Is this claim defensible? If we accept the assumptions of interpretive inquiry, then we must accept that the need for member checks will depend on the per- ceptions of the people involved. If a reader feels member checks are essential, he or she will not accept a study in which this procedure has not been done. Someone else, however, may believe either in gen- eral or for a specific study that they make no difference at all. For example, it is very possible for a person to argue that the most important and original insights come during the initial contacts with subjects. To bring one's interpretations back to them will only serve to attenuate these insights. The point is, since we are not dealing with brute data there is no un- ambiguous way to know what makes sense in this regard.
The question of member checks must also be seen in regard to what we might call a form of reflexivity. Since it can be inferred that the purpose of member
checks, to those who advocate them, is to verify the extent to which the inquirer has accurately described or reflected what the subjects did or said, this proce- dure looks very much at home with the correspondence theory. In other words, implicit in the procedure is the assump- tion of an independent, external reality (the subjects' expressions) that can be known for what it really is. However, when subjects are asked about their re- sponses, it is always possible that this is new information which will lead them to change their self-definition and how they interpret their own past and present sit- uations. This may in turn lead the in- quirer to reinterpret his or her findings in what can be a never-ending process. Sup- pose, for example, that the subject tells the inquirer that the latter's interpreta- tion of what was said is not correct and what was really meant was . . . . The prob- lem is that there is no definitive way to decide between (a) the inquirer did not originally reflect the subject's true mean- ing but now is getting closer, and (b) the inquirer's comments provoked a change in the member's definitions of the situa- tion and hence a change in his or her interpretations. Further questioning will only push the question back, not answer it.
A simple, but illustrative, hypothetical example will be helpful.3 Suppose that a researcher has been a nonparticipant ob- server of a number of "staffings" in which school personnel discuss with parents the special education placement of their chil- dren. The inquirer notes that after about 30 minutes the principal will often say to the parents, "I do not mean to rush you, but there are only two spots left in the resource room and you must make a de- cision soon." The inquirer interprets this to mean, "We have spent a half hour on this, we are busy, and we must get this meeting over." When the inquirer returns for a member check, he or she says, "You said the former but could have really meant the latter." The principal insists he meant the former and that the latter never entered his mind. In the name of iso-
3 T h e idea for, but not the reality of, this example is based on a study by Law (1981).
Judging Interpretive Inquiry 389
morphism or an accurate reflection of reality, must the inquirer accept what the principal said or can he or she continue to offer the alternative interpretation? There is simply no clear-cut way to decide the issue. And if we have no foundational basis from which to judge how to locate the information resulting from a member check, it seems unlikely that this proce- dure, or any other by extension, can be raised to the level of absolute minimum.
One objection to this example requires comment.4 One can argue that an inquirer would ask the questions both directly and indirectly and would ask related ques- tions so that a pattern of responses, inter- preted within the context that occasioned them, could be developed. This process would then yield results that have truth value or are scientifically relevant. Two things can be said in response to this sit- uation. First, that it is a good description of actual fieldwork practice almost goes without saying. Second, that it will pro- vide for results that have truth value or are scientifically relevant, if these terms are taken in a foundationalist sense, is debatable. This approach to the interpre- tation of responses is in essence an expres- sion of the idealist injunction that mean- ing must be grasped hermeneutically. And, if this is the case, than the Dilthean problem returns: Is there any point at which interpretation ends and one can say, "I have got it right?" or, more gener- ally, can there be any such thing as a "correct" interpretation? One might say that the process ends when one has a coherent interpretation that makes sense of the situation. However, given the idealist assumption that there is no inde- pendent reality to which we have inde- pendent access, there can be numerous, yet different, coherent interpretations, and what makes sense depends on the kind of sense one understands. In other words, the antifoundationalist nature of interpretive inquiry means that no epis- temological privilege can be attached to any procedure for doing or criterion for judging this approach to inquiry.
4 This objection was raised by an anonymous re- viewer of a previous version of this paper.
Summary and Implications
This paper has attempted to describe the inconsistencies that occur and to dem- onstrate why they must occur, when the assumptions of naturalistic or interpretive inquiry meet with a desire to establish neutral, nonarbitrary criteria for separat- ing trustworthy from untrustworthy re- sults. This problem of consistency arises from the incompatibility of assumptions that are epistemologically antifounda- tional and the desire for criteria (defined as standards) that are epistemologically foundational. Given the idea that social reality is mind-dependent, that facts can- not be separated from values, that the only point of view we have is that of various people based on various interests and purposes, and so on, it is impossible to be foundational or to "ground" our find- ings in the sense of "getting it right." In this situation what is trustworthy or true is nothing more than what we can agree, at any given time and place, is trustwor- thy or true.
The question now is, What does this mean for the nature of social inquiry? Presuming that consistency is important, we are left with a choice between two options: either give up the assumptions or give up the desire for "something to hang on to." In the first instance, if the claim to certitude is compelling, then we must re- consider, or simply not heed, the assump- tions of this perspective. To some extent there has been movement, even though not necessarily intended, in this latter direction. Guba was clearly drawn by this possibility with his emphasis on iso- morphism, neutrality, accurate represen- tation of respondents' perceptions, and so on. All of these points are more at home as terms of correspondence in that they are sustained by the prospect that an in- vestigator can stand apart from or inde- pendent of social reality and discover and describe that reality as it really is. Le- Compte and Goetz (1982), in another ex- ample, take Guba one step further. For them the difference between the two ap- proaches is one of technique or investi- gator strategy with both based on the same general "logic of justification." They define validity in the same way for both perspectives (as accuracy of representa-
390 John K. Smith
tion) and hold that the difference is sim- ply one of procedure to achieve such accuracy. Thus, one way out of the con- sistency problem is to ease aside the interpretive assumptions and, given the emphasis on "grounding" inquiry, indi- rectly accept those of a realist orientation.
However, if the interpretive assump- tions are more satisfying, then we must put aside the desire to be foundational. And if this is the case, then we will be compelled to rethink the nature of social inquiry. What this rethinking will lead to is impossible to specify. At this stage only a most general sort of speculation is pos- sible. Taking the lead from Rorty (1979, 1982), maybe the situation of assumptions and criteria indicates that we should dis- pense with the traditional ideas of objec- tivity and truth and realize that we are "beyond method." There are no proce- dures or criteria exclusive to or particu- larly appropriate for social inquiry and, accordingly, one cannot simply tell an- other it must be done in this way or that way because to do so will insure objectiv- ity and lead to truth (in any sense other than as a description for a sentence with which one agrees). From this antifounda- tionalist perspective social inquiry might be best seen as "continuous with litera- ture—as interpreting other people to us, and thus enlarging and deepening our sense of community" (Rorty, 1982, p. 203). Sociologists interpret the interpretations of the poor and other marginal peoples, educational psychologists interpret the interpretations of children, and educa- tional evaluators do the same for those affected by particular educational policies and practices. Yet, these interpretations are not about certitude or the discovery of how things really are—they are an at- tempt to enlarge the conversation and to keep it going.
All of this, of course, is very speculative. Not only the solutions, but even the pre- cise nature of the problems, associated with such an antifoundationalist position are extremely unclear. What is clear, however, is that the assumptions of in- terpretive inquiry are incompatible with the desire for foundational criteria. How we are to work out this problem, one way or another, would seem to merit serious attention.
References
A R O N , R. (1967). Main currents in sociological thought 2 (R. Howard & H. Weaver, Trans.). New York: Penguin.
BENTON, T. (1977). Philosophical foundations of the three sociologies. London: Routledge and Kegan Paul.
BERGNER, J. (1981). The origins of formalism in social science. Chicago: University of Chicago Press.
BERNSTEIN, R. (1976). The restructuring of social and political theory. New York: Harcourt, Brace, Jov- anovich.
BLEICHER, J. (1982). The hermeneutic imagination. London: Routledge and Kegan Paul.
BOGDAN, R., & BIKLEN, S. (1982). Qualitative research for education: An introduction to theory and practice. Boston: Allyn and Bacon.
BRUYN, S. (1966). H u m a n perspective in sociology. Englewood Cliffs, NJ: Prentice-Hall.
DURKHEIM, E. (1938). Rules of the sociological method (S. Solovay & J. Mueller, Trans., & G. E. G. Catlin, Ed.). Chicago: University of Chicago Press.
EWING, A. C. (1974). Idealism: A critical survey. Lon- don: M e t h u e n .
GADAMER, H-G. (1975). Truth and method (G. Borden & J. Cumming, Trans.). New York: Seabury Press.
GIDDENS, A. (1976). New rules of sociological method. New York: Basic Books.
GLASER, B., & STRAUSS, A. (1967). Discovery of grounded theory. Chicago: Aldine.
GRAYLING, A. C. (1982). An introduction to philosoph- ical logic. Totowa, NJ: Barnes and Noble.
GUBA, E. (1978). Toward a methodology of naturalis- tic inquiry in educational evaluation (Mono- graph Series No. 8). Los Angeles: Center for the Study of Evaluation, University of California, Los Angeles.
GUBA, E. (1981). Criteria for assessing the trust- worthiness of naturalistic inquiry. Educational Communication and Technology Journal, 29, 7 5 - 9 1 .
GUBA, E., & LINCOLN, Y. (1981). Effective evaluation. San Francisco: Jossey-Bass.
GUBA, E., & LINCOLN, Y. (1982a, March). Causality versus plausibility: Alternative stances for in- quiry into h u m a n behavior. Paper presented at the a n n u a l meeting of the American Educa- tional Research Association, New York.
GUBA, E., & LINCOLN, Y. (1982b, March). Establishing dependability and confirmability in naturalistic inquiry through an audit. Paper presented at the a n n u a l meeting of the American Educational Research Association, New York.
GUBA, E., & LINCOLN, Y. ( 1 9 8 2 C , March). The nature
of naturalistic inquiry. Paper presented at the a n n u a l meeting of the American Educational Research Association, New York.
HABERMAS, J. (1968). Knowledge and h u m a n interests (J. Shapiro, Trans.). Boston: Beacon Press.
HINDESS, B. (1977). Philosophy and methodology in the social sciences. Atlantic Highlands, NJ: Hu- manities Press.
HODGES, H. (1944). Wilhelm Dilthey: An introduction. London: Routledge and Kegan Paul.
HODGES, H. (1952). The philosophy of Wilhelm Dil- they. London: Routledge and Kegan Paul.
Judging Interpretive Inquiry 391
HUGHES, H. (1958). Consciousness and society. New York: Knopf.
HUGHES, J. (1980). The philosophy of social research. London: Longman.
JOHNSON, J. (1975). Doing field research. New York: Free Press.
LAW, S. (1981). The staffing: A naturalistic study of the educational placement process. Unpublished Ed.S. thesis, University of N o r t h e r n Iowa, Cedar Falls.
LECOMPTE, M., & GOETZ, J. (1982). Problems of relia- bility and validity in ethnographic research. Re- view of Educational Research, 52, 31-60.
LESSNOFF, M. (1974). The structure of social science. London: George Allen and Unwin.
LEVY, D. (1981). Realism: An essay in interpretation and social reality. Atlantic Highlands, NJ: Hu- manities Press.
LUKACS, G. (1971). History and class consciousness (R. Livingstone, Trans.). London: Merlin Press.
MELDEN, A. (1961). Free action. London: Rout- ledge and Kegan Paul.
NAGEL, T. (1981). Mortal questions. New York: Cam- bridge University Press.
OUTHWAITE, W. (1975). Understanding social life: The method called verstehen. London: Allen and Un- win.
OUTHWAITE, W. (1983). Concept formation in social science. London: Routledge and Kegan Paul.
PALMER, R. (1969). Hermeneutics. Evanston, IL: Northwestern University Press.
PSATHAS, G. (Ed.). (1973). Phenomenological sociol- ogy. New York: Wiley.
PUTNAM, H. (1981). Reason, truth, and history. Cam- bridge: Cambridge University Press.
ROCHE, M. (1973). Phenomenology, language, a n d the social sciences. London: Routledge and Kegan Paul.
RORTY, R. (1979). Philosophy and the mirror of na- ture. Princeton, NJ: Princeton University Press.
RORTY, R. (1982). Consequences of pragmatism. Min- neapolis: University of Minnesota Press.
RUBINSTEIN, D. (1981). Marx a n d Wittgenstein. Lon- don: Routledge and Kegan Paul.
SIMEY, T. (1969). Social science and social purpose. New York: Schocken.
SMITH, J. (1983). Quantitative versus qualitative re- search: An attempt to clarify the issue. Educa- tional Researcher, 12, 6-13.
STRAUSS, L. (1953). Natural right and history. Chi- cago: University of Chicago Press.
TAYLOR, C. (1971). Interpretation and the sciences of man. Review of Metaphysics, 25, 3 - 5 1 .
TRIGG, R. (1980). Reality at risk. Totowa, NJ: Barnes and Noble.
URBAN, W. (1949). Beyond realism a n d idealism. Lon- don: George Allen and Unwin.
WEBER, M. (1949). The methodology of the social sciences (E. Shils & H. Finch, trans, and eds.). Glencoe, IL: Free Press.
WOLCOTT, H. (1975). Criteria for an ethnographic approach to research in schools. H u m a n Orga- nization, 34, 111-127.
WRONG, D. (Ed.). (1970). Max Weber. Englewood Cliffs, NJ: Prentice-Hall.
Author JOHN K. SMITH, Associate Professor of
Education, Department of Educa- tional Psychology and Foundations, University of Northern Iowa, Cedar Falls, IA 50614. Specializations: Edu- cational research, educational soci- ology.