U10A1-68 - Qualitative Research Plan - ***TUTOR FOLLOW ALL INSTRUCTIONS AS OUTLINED TO COMPLETE THIS WORK. READ ATTACHMENTS.

6/2/2018 Bookshelf Online: Qualitative Inquiry and Research Design: Choosing Among Five Approaches

https://online.vitalsource.com/#/books/9781506330228/cfi/6/44!/4/2/4/2@0:0 1/22

10 Standards of Validation and Evaluation

Qualitative researchers strive for “understanding,” that deep structure of knowledge that comes from visiting personally with participants, spending extensive time in the field, and probing to obtain detailed meanings. During or after a study, qualitative researchers ask, “Did we get it right?” (Stake, 1995, p. 107) or “Did we publish a ‘wrong’ or inaccurate account?” (Thomas, 1993, p. 39). Is it possible to even have a right answer? To answer these questions, researchers need to look to themselves, to the participants, and to the readers. There are multi- or polyvocal discourses at work here that provide insight into the validation and evaluation of a qualitative narrative.

In this chapter, we address two interrelated questions: Is the account valid, and by whose standards? How do we evaluate the quality of qualitative research? Answers to these questions will take us into the many perspectives on validation to emerge within the qualitative community and the multiple standards for evaluation discussed by authors with procedural, interpretive, and postmodern perspectives.

Questions for Discussion What are some perspectives on validation within the qualitative community? What are some alternative procedures useful in establishing validation in qualitative research? How is reliability realized in qualitative research? What are some alternative stances on evaluating the quality of qualitative research? How do these stances differ by types of approaches to qualitative inquiry?

Validation and Reliability in Qualitative Research

Perspectives on Validation Many perspectives exist regarding the importance of validation in qualitative research, the definition of it, terms to describe it, and procedures for establishing it. Our view of validation as an evolving construct means that a broad understanding of both traditional and contemporary perspectives is essential for informing the work of qualitative researchers and readers of qualitative research. In Table 10.1, we illustrate several of the perspectives, arranged chronologically, available on validation in the qualitative literature. These perspectives position validation in qualitative research in terms of quantitative equivalents, postmodern and interpretive lenses, and importance of the construct. Most use qualitative terms to describe validation that are distinct from quantitative terms; some combine or synthesize many perspectives or use a metaphor for visualizing it. Evidence of evolving thinking about validation in qualitative research is apparent within as well as across authors and we conclude this section by forwarding our own stance.

Writers have searched for and found qualitative equivalents that parallel traditional quantitative approaches to validation. LeCompte and Goetz (1982) took this approach when they compared the issues of validation and reliability to their counterparts in experimental design and survey research. They contend that qualitative research has garnered much criticism in the scientific ranks for its failure to “adhere to canons of reliability and validation” (LeCompte & Goetz, 1982, p. 31) in the traditional sense. They apply threats to internal validation in experimental research to ethnographic research (e.g., history and maturation, observer effects, selection and regression, mortality, spurious conclusions). They further identify threats to external validation as “effects that obstruct or reduce a study’s comparability or translatability” (LeCompte & Goetz, 1982, p. 51).

6/2/2018 Bookshelf Online: Qualitative Inquiry and Research Design: Choosing Among Five Approaches

https://online.vitalsource.com/#/books/9781506330228/cfi/6/44!/4/2/4/2@0:0 2/22

Some writers argue that authors who use positivist terminology facilitate the acceptance of qualitative research in a traditionally focused quantitative world. Ely and colleagues (Ely, Anzul, Friedman, Garner, & Steinmetz, 1991) assert that using quantitative terms tend to be a defensive measure that muddies the waters and that “the language of positivistic research is not congruent with or adequate to qualitative work” (p. 95). Lincoln and Guba (1985) offer alternative terms that, they contend, adhere more to naturalistic research. To establish the “trustworthiness” of a study, Lincoln and Guba (1985) use unique terms, such as credibility, authenticity, transferability, dependability, and confirmability as “the naturalist’s equivalents” for internal validation, external validation, reliability, and objectivity (p. 300). To operationalize these new terms, they propose techniques such as prolonged engagement in the field and the triangulation of data sources, methods, and investigators to establish credibility. To make sure that the findings are transferable between the researcher and those being studied, thick description is necessary. Rather than reliability, one seeks dependability that the results will be subject to change and instability. The naturalistic researcher looks for confirmability rather than objectivity in establishing the value of the data. Both dependability and confirmability are established through an auditing of the research process. We find the Lincoln and Guba criteria still popular today in qualitative reports.

Rather than using the term validation, Eisner (1991) constructed standards such as structural corroboration, consensual validation, and referential adequacy as evidence for asserting the credibility of qualitative research. In structural corroboration, the researcher uses multiple types of data to support or contradict the interpretation. As Eisner (1991) states, “We seek a confluence of evidence that breeds credibility, that allows us to feel confident about our observations, interpretations, and conclusions” (p. 110). He further illustrates this point with

6/2/2018 Bookshelf Online: Qualitative Inquiry and Research Design: Choosing Among Five Approaches

https://online.vitalsource.com/#/books/9781506330228/cfi/6/44!/4/2/4/2@0:0 3/22

an analogy drawn from detective work: The researcher compiles bits and pieces of evidence to formulate a “compelling whole.” At this stage, the researcher looks for recurring behaviors or actions and considers disconfirming evidence and contrary interpretations. Moreover, Eisner recommends that to demonstrate credibility, the weight of evidence should become persuasive. Consensual validation seeks the opinion of others, and Eisner (1991) refers to “an agreement among competent others that the description, interpretation, evaluation, and thematics of an educational situation are right” (p. 112). Referential adequacy suggests the importance of criticism, and Eisner describes the goal of criticism as illuminating the subject matter and bringing about more complex and sensitive human perception and understanding.

Qualitative researchers also have reconceptualized validation with a postmodern sensibility. Lather (1991) initially identified four types of validation (i.e., triangulation, construct validation, face validation, and catalytic validation) as a “reconceptualization of validation.” Lather comments that current “paradigmatic uncertainty in the human sciences is leading to the re-conceptualizing of validation” and calls for “new techniques and concepts for obtaining and defining trustworthy data which avoids the pitfalls of orthodox notions of validation” (p. 66). For Lather, the character of a social science report changes from a closed narrative with a tight argument structure to a more open narrative with holes and questions and an admission of situatedness and partiality. In Getting Smart, Lather (1991) describes triangulation as drawing upon multiple data sources, methods, and theoretical schemes, construct validation as recognizing the constructs that exist rather than imposing theories or constructs on informants or the context, face validation citing Kidder (1982) as “a ‘click of recognition’ and a ‘yes, of course,’ instead of ‘yes, but’ experience” (p. 56) and catalytic validation as energizing participants toward knowing reality to transform it.

In a later article, Lather’s (1993) terms became more unique and closely relate to feminist research in “four frames of validation.” The first, ironic validation, is where the researcher presents truth as a problem. The second, paralogic validation, is concerned with undecidables, limits, paradoxes, and complexities, a movement away from theorizing things and toward providing direct exposure to other voices in an almost unmediated way. The third, rhizomatic validation, pertains to questioning proliferations, crossings, and overlaps without underlying structures or deeply rooted connections. The researcher also questions taxonomies, constructs, and interconnected networks whereby the reader jumps from one assemblage to another and consequently moves from judgment to understanding. The fourth type is situated, embodied, or voluptuous validation, which means that the researcher sets out to understand more than one can know and to write toward what one does not understand.

Other writers, such as Wolcott (1990), have little use for validation, suggesting that “validation neither guides nor informs” his work (p. 136). He does not dismiss validation but rather places it in a broader perspective. Wolcott’s (1990) goal is to identify “critical elements” and write “plausible interpretations from them” (p. 146). He ultimately tries to understand rather than convince, and he voices the view that validation distracts from his work of understanding what is really going on. Wolcott (1990, 1994) claims that the term validation does not capture the essence of what he seeks, adding that perhaps someone would coin a term appropriate for the naturalistic paradigm. But for now, he says, the term understanding seems to encapsulate the idea as well as any other (Wolcott, 1994).

Validation has also been cast within an interpretive approach to qualitative research marked by a focus on the importance of the researcher; a lack of truth in validation; a form of validation based on negotiation and dialogue with participants; and interpretations that are temporal, located, and always open to reinterpretation (Angen, 2000). Angen (2000) suggests that within interpretative research, validation is “a judgment of the trustworthiness or goodness of a piece of research” (p. 387). She espouses an ongoing, open dialogue on the topic of what makes interpretive research worthy of our trust. Considerations of validation are not definitive as the final word on the topic, nor should every study be required to address them. Further, she advances two types of validation: ethical validation and substantive validation. Ethical validation means that all research agendas must question their underlying moral assumptions, their political and ethical implications, and the equitable treatment of diverse voices. It also requires research to provide some practical answers to questions. Angen (2000) also proposes that our interpretive qualitative research should have a “generative promise” (p. 389) and raise new possibilities, open up new questions, stimulate new dialogue, and provide non-dogmatic answers to

6/2/2018 Bookshelf Online: Qualitative Inquiry and Research Design: Choosing Among Five Approaches

https://online.vitalsource.com/#/books/9781506330228/cfi/6/44!/4/2/4/2@0:0 4/22

the questions we pose. In so doing, interpretive qualitative research must have transformative value leading to action and change. Substantive validation means understanding one’s own topic, understandings derived from other sources, and the documentation of this process in the written study. Self-reflection contributes to the validation of the work. The interpretive qualitative researcher, as a sociohistorical interpreter, interacts with the subject matter to co-create the interpretations derived. Understandings derived from previous research give substance to the inquiry. Interpretive research also is a chain of interpretations that must be documented for others to judge the trustworthiness of the meanings arrived at in the end. Written accounts must resonate with their intended audiences, and must be compelling, powerful, and convincing.

A synthesis of validation perspectives comes from Whittemore, Chase, and Mandle (2001), who analyze 13 writings about validation and extract key validation criteria from these studies. They organize these criteria into primary and secondary criteria. They find four primary criteria: credibility (Are the results an accurate interpretation of the participants’ meaning?); authenticity (Are different voices heard?); criticality (Is there a critical appraisal of all aspects of the research?); and integrity (Are the investigators self-critical?). Secondary criteria relate to explicitness, vividness, creativity, thoroughness, congruence, and sensitivity. In summary, with these criteria, it seems like the validation standard has moved toward the interpretive lens of qualitative research, with an emphasis on researcher reflexivity and on researcher challenges that include raising questions about the ideas developed during a research study.

A postmodern perspective draws on the metaphorical image of a crystal. Richardson (in Richardson & St. Pierre, 2005) describes this image:

I propose that the central imaginary for “validation” for postmodern texts is not the triangle—a rigid, fixed, two-dimensional object. Rather the central imaginary is the crystal, which combines symmetry and substance with an infinite variety of shapes, substances, transmutations, multidimensionalities, and angles of approach. Crystals grow, change, and are altered, but they are not amorphous. Crystals are prisms that reflect externalities and refract within themselves, creating different colors, patterns, and arrays casting off in different directions. What we see depends on our angle of response—not triangulation but rather crystallization. (p. 963)

A final perspective is that drawn from Lincoln, Lynham, and Guba (2011). They capture the many perspectives developed through the years. They suggest that the question of validity criteria is not whether we should have such criteria or whose criteria the scientific community might adopt but rather how criteria need to be developed within the projected transformations being suggested by social scientists. To this end, they review their focus on establishing authenticity but frame it within the perspectives of a balance of views, raising the level of awareness among participants and other stakeholders, and advancing the ability of inquiry to lead to action on the part of research participants and training those participants to take action. Lincoln and colleagues (2011) also see a role for validity in understanding hidden assumptions through the image (also shared by Richardson and St. Pierre, 2005) of a crystal that reflects and refracts the processes of research, such as discovery, seeing, telling, storying, and representation. Finally, for these authors validity is an ethical relationship with research participants through such standards as positioning themselves, having discourses, encouraging voices, and being self-reflective.

Given these many perspectives, we summarize our own stance. We consider “validation” in qualitative research to be an attempt to assess the “accuracy” of the findings, as best described by the researcher, the participants, and the readers (or reviewers). This view also suggests that any report of research is a representation by the author. We also view validation as a distinct strength of qualitative research in that the account made through extensive time spent in the field, the detailed thick description, and the closeness of the researcher to participants in the study all add to the value or accuracy of a study. We use the term validation to emphasize a process (see Angen, 2000), rather than verification (which has quantitative overtones) or historical words such as trustworthiness and authenticity (recognizing that many qualitative writers do return to these words, suggesting the “staying power” of Lincoln and Guba’s, 1985, standards; see Whittemore et al., 2001).

6/2/2018 Bookshelf Online: Qualitative Inquiry and Research Design: Choosing Among Five Approaches

https://online.vitalsource.com/#/books/9781506330228/cfi/6/44!/4/2/4/2@0:0 5/22

We acknowledge that there are many types of validation in qualitative research and that authors need to choose the types and terms with which they are comfortable. We recommend that writers reference their validation terms and strategies. The subject of validation arises in several of the approaches to qualitative research (e.g., Corbin & Strauss, 2015; Riessman, 2008; Stake, 1995), but we do not think that distinct validation approaches exist for the five approaches to qualitative research. At best, there might be less emphasis on validation in narrative research and more emphasis on it in grounded theory, case study, and ethnography, especially when the authors of these approaches want to employ systematic procedures. We recommend using multiple validation strategies regardless of type of qualitative approach.

Our framework for thinking about validation in qualitative research is to suggest that researchers employ accepted strategies to document the accuracy of their studies. These we call validation strategies.

Validation Strategies It is not enough to gain perspectives and terms; ultimately, these ideas are translated into practice as strategies or techniques. Whittemore and colleagues (2001) organize the techniques into 29 forms that apply to design consideration, data generating, analysis, and presentation. We describe nine strategies frequently used by qualitative researchers during the process of validation adapted from the work of Creswell and Miller (2000) and provide some general guidance about how we go about implementing these strategies (see Figure 10.1).

The strategies are not presented in any specific order of importance but are organized in three groups by the lens the strategy represents: researcher’s lens, participant’s lens, and reader’s or reviewer’s lens (Creswell, 2016). Our recommendation that qualitative researchers engage in at least two of the validation strategies in any given study is further discussed next.

Researcher’s Lens

Among the many roles a researcher undertakes is to check the accuracy of a qualitative account and any of the following validation strategies can assist the researcher in this role:

Corroborating evidence through triangulation of multiple data sources. The researcher makes use of multiple and different sources, methods, investigators, and theories to provide corroborating evidence (Bazeley, 2013; Ely et al., 1991; Erlandson, Harris, Skipper, & Allen, 1993; Glesne, 2016; Lincoln & Guba, 1985; Miles & Huberman, 1994; Patton, 1980, 1990, 2015; Yin, 2014). Typically, this process involves corroborating evidence from different sources to shed light on a theme or perspective. When qualitative researchers locate evidence to document a code or theme in different sources of data, they are triangulating information and providing validity to their findings. For this validation strategy, we begin considering how various data sources can be used in tandem when planning the study. Then, as data is collected, we further explore evidence of corroboration and use these insights in our interpretation and writing.

Figure 10.1 Strategies for Validation in Qualitative Research

6/2/2018 Bookshelf Online: Qualitative Inquiry and Research Design: Choosing Among Five Approaches

https://online.vitalsource.com/#/books/9781506330228/cfi/6/44!/4/2/4/2@0:0 6/22

Discovering negative case analysis or disconfirming evidence. The researcher refines working hypotheses as the inquiry advances in light of negative or rival evidence (Ely et al., 1991; Lincoln & Guba, 1985; Miles & Huberman, 1994; Patton, 1980, 1990, 2015; Yin, 2014). Not all evidence will fit the pattern of a code or a theme. It is necessary then to report this negative analysis, and in doing so, the researcher provides a realistic assessment of the phenomenon under study. In real life, not all evidence is either positive or negative; it is some of both. For this validation strategy, we both admit that we tend to be attentive to such evidence and make a point of following what we call “points of intrigue” throughout the study. We find these points are often our key points of discussion in our writing. Clarifying researcher bias or engaging in reflexivity. The researcher discloses their understandings about the biases, values, and experiences that he or she brings to a qualitative research study from the outset of the study so that the reader understands the position from which researcher undertakes the inquiry (Hammersley & Atkinson, 1995; Merriam & Tisdell, 2015). In this clarification, according to Weiner- Levey and Popper-Giveon (2013), the researcher illuminates what they call the “dark matter” that is often omitted in qualitative research by commenting on past experiences, biases, prejudices, and orientations that have likely shaped the interpretation and approach to the study. For this validation strategy, we embed opportunities throughout a study for writing and discussing connections that emerge with our past experiences and perspectives.

Participant’s Lens

Participants can play an important role in the following validation strategies:

Member checking or seeking participant feedback. The researcher solicits participants’ views of the credibility of the findings and interpretations (Bazeley, 2013; Ely et al., 1991; Erlandson et al., 1993; Glesne, 2016; Lincoln & Guba, 1985; Merriam & Tisdell, 2015; Miles & Huberman, 1994). This technique is considered by Lincoln and Guba (1985) to be “the most critical technique for establishing credibility” (p. 314). This approach, writ large in most qualitative studies, involves taking data, analyses, interpretations, and conclusions back to the participants so that they can judge the accuracy and credibility of the account. According to Stake (1995), participants should “play a major role directing as well as

6/2/2018 Bookshelf Online: Qualitative Inquiry and Research Design: Choosing Among Five Approaches

https://online.vitalsource.com/#/books/9781506330228/cfi/6/44!/4/2/4/2@0:0 7/22

acting in case study” research (p. 115). They should be asked to examine rough drafts of the researcher’s work and to provide alternative language, “critical observations or interpretations” (Stake, 1995, p. 115). In so doing, participants play a critical role because they are asked “how well the ongoing data analysis represents their experience” (Hays & Singh, 2012, p. 206). For this validation strategy, we convene a focus group made up of participants in the study and ask them to reflect on the accuracy of the account. We do not take back to participants the transcripts or the raw data, but the preliminary analyses consisting of description or themes. We are interested in their views of these written analyses as well as what was missing (see also Richards, 2015, for practical guidance about how to interpret participant feedback). Prolonged engagement and persistent observation in the field. The researcher makes field-based decisions about what is salient to study, relevant to the purpose of the study, and of interest for focus. Researchers build rapport with participants and gatekeepers, learn the culture and context, and check for misinformation that stems from distortions introduced by themselves or informants (Ely et al., 1991; Erlandson et al., 1993; Glesne, 2016; Lincoln & Guba, 1985; Merriam & Tisdell, 2015). Fetterman (2010) contends that “participant observation requires close, long-term contact with the people under study” (p. 39). For this validation strategy, we spend as much time in the field as is feasible during the study and prior to beginning data collection we familiarize ourselves with the site and participants. Collaborating with participants. The researcher embeds opportunities for participants to be involved throughout the research process in varying ways and degrees. Among the various ways is involvement in key research decisions such as developing data collection protocols and contributing to data analysis and interpretation. The degree to which participants are involved can vary along a continuum from minimal to extensive. Participant involvement is based on the idea (and ever-growing body of research) that the study is more likely to be supported and findings used when participants are involved (Patton, 2011, 2015). For this validation strategy, we are often guided by community-based participatory research practices that involve participants as co-researchers in the study (see, e.g., Hacker, 2013, for further discussions).

Reader’s or Reviewer’s Lens

Including others beyond the researcher and those involved in the research contribute in the following validation strategies:

Enabling external audits. The researcher facilitates auditing by an external consultant, the auditor, to examine both the process and the product of the account to assess their accuracy (Erlandson et al., 1993; Lincoln & Guba, 1985; Merriam & Tisdell, 2015; Miles & Huberman, 1994). This auditor should have no connection to the study. In assessing the product, the auditor examines whether or not the findings, interpretations, and conclusions are supported by the data. Lincoln and Guba (1985) compare this, metaphorically, with a fiscal audit, and the procedure provides a sense of interrater reliability to a study. This process can be assisted by the creation of documentation, sometimes referred to as an audit trail and described by Silver and Lewins (2014) as “comprising a log of all the processes followed, describing the small analytic leaps contributing to the analysis as a whole” (p. 140). For this validation strategy, we engage in two processes; first, we create a tracking document at the beginning of a study on which we detail our key decisions including rationale and potential consequence. Second, when resources permit, we use an auditor to review our process and findings. Generating a rich, thick description. The researcher allows readers to make decisions regarding transferability because the writer describes in detail the participants or setting under study (Erlandson et al., 1993; Lincoln & Guba, 1985; Merriam & Tisdell, 2015). With such a detailed description, the researcher enables readers to transfer information to other settings and to determine whether the findings can be transferred “because of shared characteristics” (Erlandson et al., 1993, p. 32). Thick description means that the researcher provides details when describing a case or when writing about a theme. According to Stake (2010), “a description is rich if it provides abundant, interconnected details . . .” (p. 49). Detail can emerge through physical description, movement description, and activity description. It can also involve describing from the general ideas to the narrow, interconnecting the details, using strong action verbs, and quotes. For this validation strategy, we devote time to revisiting our raw data soon after its collection to add further descriptions that might be helpful during the analysis—for example, contextual descriptions like atmosphere.

6/2/2018 Bookshelf Online: Qualitative Inquiry and Research Design: Choosing Among Five Approaches

https://online.vitalsource.com/#/books/9781506330228/cfi/6/44!/4/2/4/2@0:0 8/22

Having a peer review or debriefing of the data and research process. The researcher seeks an external check by “someone who is familiar with the research or the phenomenon explored” (Creswell & Miller, 2000, p. 129), in much in the same spirit as interrater reliability in quantitative research (Ely et al., 1991; Erlandson et al., 1993; Glesne, 2016; Lincoln & Guba, 1985; Merriam & Tisdell, 2015). Lincoln and Guba (1985) define the role of the peer debriefer as a “devil’s advocate,” an individual who keeps the researcher honest; asks hard questions about methods, meanings, and interpretations; and provides the researcher with the opportunity for catharsis by sympathetically listening to the researcher’s feelings. For this validation strategy, we involve colleagues and students as reviewers (and for our students we play this role) in what Lincoln and Guba (1985) called peer debriefing sessions during which both the reviewers and the researcher keep written accounts of the sessions.

Examining these nine procedures as a whole (just discussed and outlined in Figure 10.1), we advise that researchers engage in at least two of them in any given qualitative study. Unquestionably, procedures such as triangulating among different data sources (assuming that the investigator collects more than one), writing with detailed and thick description, and taking the entire written narrative back to participants in member checking all are reasonably easy procedures to conduct. They also are the most popular and cost-effective procedures. Other procedures, such as peer audits and external audits, are more time consuming in their application and may also involve substantial costs to the researcher. We also point out that differences among the validity lenses (i.e., researchers, participants, readers, and reviewers) can be attributed to the philosophical orientation of the researcher and may therefore impact his or her use of specific validation strategies (see Creswell, 2016, for further discussion).

Reliability Perspectives and Procedures Reliability can be addressed in qualitative research in several ways (Silverman, 2013). Reliability can be enhanced if the researcher obtains detailed field notes by employing good-quality recording devices and by transcribing the digital files. Also, the recording needs to be transcribed to indicate the trivial, but often crucial, pauses and overlaps. Further coding can be done “blind” with the coding staff and the analysts conducting their research without knowledge of the expectations and questions of the project directors. Silverman also supports the use of computer programs to assist in recording and analyzing the data.

Our focus on reliability here will be on intercoder agreement based on the use of multiple coders to analyze transcript data. In qualitative research, reliability often refers to the stability of responses to multiple coders of data sets. It is important to develop codes and assess the reliability among coders as part of the analysis process (Kuckartz, 2014; Richards & Morse, 2012). We find this practice especially used in qualitative health science research and within the form of qualitative research in which inquirers want an external check on the highly interpretive coding process. What seems to be largely missing in the literature (with the exception of Armstrong, Gosling, Weinman, & Marteau, 1997; Campbell, Quincy, Osserman, & Pederson, 2013; Miles & Huberman, 1994; and Miles, Huberman, & Saldaña, 2014) is a discussion about the procedures of actually conducting intercoder agreement checks. One of the key issues is determining what exactly the coders agree on, whether they seek agreement on code names, the coded passages, or the same passages coded the same way. We also need to decide on whether researchers seek agreement based on codes, themes, or both codes and themes (see Armstrong et al., 1997). Finally, we need to carefully interpret our findings—as Richards (2015) wisely advises that we cannot expect to find complete consistency in coding over time or between coders. Undoubtedly, there is flexibility in the process, and researchers need to fashion an approach consistent with the resources and time to engage in coding.

Drawing upon our experiences working with multiple coders, we propose the following procedures for assessing intercoder agreement in coding of qualitative research (see Figure 10.2):

Establish a common platform for coding, and develop a preliminary code list. The researchers decide which software program or paper-based methods they will use; a common platform is essential to be able to easily share the results of their initial read and preliminary coding efforts. In our recent work, we have often used computer-assisted qualitative data analysis software packages (MAXQDA, ATLAS.ti, or

6/2/2018 Bookshelf Online: Qualitative Inquiry and Research Design: Choosing Among Five Approaches

https://online.vitalsource.com/#/books/9781506330228/cfi/6/44!/4/2/4/2@0:0 9/22

NVivo, depending on familiarity across researchers) and have implemented training sessions at the beginning of the study. Each researcher then reads several transcripts independently and develops a list of preliminary codes. As discussed in Chapter 8, computer software programs have features that facilitate the creation of these lists. Develop and share the initial codebook among coders. The researchers develop a shared understanding of codes to create a codebook that is stable and represents the coding analysis of four independent coders. To do this, after coding, say, three transcripts (i.e., A, B, and C), we then meet and examine the codes, their names, and the text segments that each researcher codes. We begin developing an initial codebook of the major codes. This codebook contains a definition of each code and the text segments that we assigned to each code. In this initial codebook, we had main codes and subcodes. In this initial codebook, we focus on the main codes we were finding in the database than in an exhaustive list. Additional codes are added as the analyses proceeds. Apply the codebook to additional transcripts, and compare coding across multiple researchers. The researchers independently apply the shared codebook to additional transcripts and then compare their coding to assess consistency. To do this, the researchers had to agree upon a unit of text to compare (e.g., phrase, sentence, and paragraph), and then each researcher independently codes three additional transcripts (i.e., D, E, and F). With no clear guidance in the literature as to what the appropriate unit should be for coding, consensus can be difficult to reach (Hruschka et al., 2004). To streamline the process in their exploratory study, Campbell and colleagues (2013) suggest having the lead researcher identify the segments of data into coding units. Often in our work we feel it is more important to have agreement on the text segments we were assigning to codes than to have the same, exact passages coded. The decision about the unit of text is crucial because it becomes the basis on which intercoder agreement is defined, and we are ready to actually compare our coding. Thus, often in our work, intercoder agreement means that we agree that when we assign a code word to a passage, we all assign this same code word to the passage. It does not mean that we all code the same passages—an ideal that we believe would be hard to achieve because some people code short passages and others longer passages. Nor does it mean that we all bracket the exact same lines for our code word, another ideal difficult to achieve. Assess and report the intercoder agreement among researchers. The researchers define individual instances of intercoder agreement before assessing overall intercoder agreement. In our work, we tend to take a realistic stance, look at the passages that are coded across researchers, and ask ourselves whether we all assign the same code word to the passage based on our tentative definitions in the codebook. The decision would be either a yes or a no, and we could calculate the percentage of agreement among all researchers on this passage that we all code. We seek to establish an 80% agreement of coding on these passages informed by Miles and Huberman’s (1994) recommendation of an 80% agreement. More recently, Miles and colleagues (2014) suggest achieving 85% to 90% agreement depending on the “size and range of the coding scheme” (p. 85). Many computer-assisted qualitative data analysis software packages include features to facilitate the calculation among multiple coders. Other researchers might actually calculate a kappa reliability statistic on the agreement, but we feel that a percentage is sufficient to report in our published studies (see also Creswell, 2016, for practical guidance about three different approaches to intercoder agreement). Revise and finalize the codebook to inform further coding. The researchers review and refine the codebook to further differentiate code definitions. In our work, after the process continues through several more transcripts, we then revise the codebook and conduct anew an assessment of passages that all researchers code and determine if we apply the same or different codes. With each phase in the intercoder agreement process, we hope to achieve a higher percentage of agreed upon codes and themes for text segments. Then we can collapse codes into broader themes and can conduct the same process with themes, to see if the passages all coded as themes by the researchers were consistent in the use of the same theme.

Evaluation Criteria

Qualitative Perspectives

6/2/2018 Bookshelf Online: Qualitative Inquiry and Research Design: Choosing Among Five Approaches

https://online.vitalsource.com/#/books/9781506330228/cfi/6/44!/4/2/4/2@0:0 10/22

In reviewing validation in the qualitative research literature, we are struck by how validation is sometimes used in discussing the quality of a study (e.g., Angen, 2000). Although validation is certainly an aspect of evaluating the quality of a study, other criteria are useful as well. In reviewing the criteria, we find that here, too, the standards vary within the qualitative community (see also Creswell, 2012, for contrast of three approaches to qualitative evaluation). We will first review three general standards and then turn to specific criteria within each of our five approaches to qualitative research.

A methodological perspective comes from Howe and Eisenhardt (1990), who suggest that only broad, abstract standards are possible for qualitative (and quantitative) research. Moreover, to determine, for example, whether a study is a good ethnography cannot be answered apart from whether the study contributes to our understanding of important questions. Silverman (2013) advances four criteria that good research must satisfy. The following five standards are adapted from Howe and Eisenhardt (1990) in question form for researchers to ask:

Does the research questions drive the data collection and analysis? (That is, rather than the reverse?) Silverman (2013) states similar criteria about good research using “methods which are demonstrably appropriate to the research problem” (p. 322). To what extent are the data collection and analysis techniques competently applied? (That is, in a technical sense?) Silverman (2013) states similar criteria about good research developing “empirically sound, reliable, and valid findings” (p. 322). Are the researcher’s assumptions made explicit? (That is, such as the researcher’s own subjectivity?) Does the study have overall warrant? (That is, is it robust, does it use respected theoretical explanations, and does it discuss disconfirmed theoretical explanations?) Silverman (2013) states similar criteria about good research thinking “theoretically through and with data” (p. 322). Does the study have value both in informing and improving practice and in protecting the confidentiality, privacy, and truth telling of participants conducting in an ethical manner? (That is, does it contribute to the “so what?” question and is it conducted ethically?) Silverman (2013) states similar criteria about good research contributing “where possible, to practice and policy” (p. 322).

Figure 10.2 Procedures for Reliability of Intercoder Agreement in Qualitative Research

A postmodern, interpretive framework forms a second perspective, from Lincoln (1995), who thinks about the quality issue in terms of emerging criteria. She tracks her own thinking (and that of her late colleague, Guba) from early approaches of developing parallel methodological criteria (Lincoln & Guba, 1985) to establishing the criteria of “fairness” (a balance of stakeholder views), sharing knowledge, and fostering social action (Guba & Lincoln, 1989) to her current stance. The new emerging approach to quality is based on three new commitments: to emergent relations with respondents, to a set of stances, and to a vision of research that enables and promotes justice. Based on these commitments, Lincoln (1995) then proceeds to identify several standards:

6/2/2018 Bookshelf Online: Qualitative Inquiry and Research Design: Choosing Among Five Approaches

https://online.vitalsource.com/#/books/9781506330228/cfi/6/44!/4/2/4/2@0:0 11/22

The standard is set in the inquiry community, such as by guidelines for publication. These guidelines admit that within diverse approaches to research, inquiry communities have developed their own traditions of rigor, communication, and ways of working toward consensus. These guidelines, she also maintains, serve to exclude legitimate research knowledge and social science researchers. The standard of positionality guides interpretive or qualitative research. Drawing on those concerned about standpoint epistemology, this means that the text should display honesty or authenticity about its own stance and about the position of the author. Another standard is under the rubric of community. This standard acknowledges that all research takes place in, is addressed to, and serves the purposes of the community in which it is carried out. Such communities might be feminist thought, Black scholarship, Native American studies, or ecological studies. Interpretive or qualitative research must give voice to participants so that their voice is not silenced, disengaged, or marginalized. Moreover, this standard requires that alternative or multiple voices be heard in a text. Critical subjectivity as a standard means that the researcher needs to have heightened self-awareness in the research process and create personal and social transformation. This “high-quality awareness” enables the researcher to understand his or her psychological and emotional states before, during, and after the research experience. High-quality interpretive or qualitative research involves reciprocity between the researcher and those being researched. This standard requires that intense sharing, trust, and mutuality exist. The researcher should respect the sacredness of relationships in the research-to-action continuum. This standard means that the researcher respects the collaborative and egalitarian aspects of research and “make[s] spaces for the lifeways of others” (Lincoln, 1995, p. 284). Sharing of the privileges acknowledges that in good qualitative research, researchers share their rewards with persons whose lives they portray. This sharing may be in the form of royalties from books or the sharing of rights to publication.

A final perspective utilizes interpretive standards of conducting qualitative research. Richardson (in Richardson & St. Pierre, 2005) identifies multiple criteria she uses when she reviews papers or monographs submitted for social science publication:

Substantive contribution. Does this piece contribute to our understanding of social life? Demonstrate a deeply grounded social scientific perspective? Seem “true”? Aesthetic merit. Does this piece succeed aesthetically? Does the use of creative analytical practices open up the text and invite interpretive responses? Is the text artistically shaped, satisfying, complex, and not boring? Reflexivity. How has the author’s subjectivity been both a producer and a product of this text? Is there self-awareness and self-exposure? Does the author hold himself or herself accountable to the standards of knowing and telling of the people he or she has studied? Impact. Does this piece affect me emotionally or intellectually? Generate new questions or move me to write? Try new research practices or move me to action? (p. 964)

As applied research methodologists, we prefer the methodological standards of evaluation, but we can also support the postmodern and interpretive perspectives. We also agree with Flick (2014), who states, “the problem of how to assess qualitative research has not yet been solved” (p. 480). What seems to be missing in all of the approaches discussed thus far is their connection to the five approaches of qualitative inquiry. What standards of evaluation, beyond those already mentioned, would signal a high-quality narrative study, a phenomenology, a grounded theory study, an ethnography, and a case study?

Narrative Research In discussing what makes for a good narrative study, both Riessman (2008) and Clandinin (2013) point to seeking coherence of participants’ narratives yet acknowledge that it may not always be possible. To that end, Riessman (2008) proposes questions for assessing coherence (p. 189):

6/2/2018 Bookshelf Online: Qualitative Inquiry and Research Design: Choosing Among Five Approaches

https://online.vitalsource.com/#/books/9781506330228/cfi/6/44!/4/2/4/2@0:0 12/22

Do episodes of a life story hang together? Are sections of a theoretical argument linked and consistent? Are there major gaps and inconsistencies? Is the interpreter’s analytic account persuasive?

Clandinin (2013) proposes a different way for “judging and responding to narrative inquiries” (p. 211) by describing what she and Vera Caine define as touchstones:

While one meaning directs our attention to a touchstone as a quality of example that is used to test the excellence or genuineness of others, we were also drawn to a touchstone as a hard black stone, such as jasper or basalt, that was used to test the quality of gold or silver by comparing the streak left on the stone by one of these metals with that of a standard alloy. We wondered, if we metaphorically touched or scratched a narrative inquiry, what kinds of streaks or marks would be left. (Clandinin & Caine, 2013, p. 191)

Figure 10.3 Guiding Aspects for a “Good” Narrative Study

Clandinin (2013) lists twelve touchstones and considers them as evolving (see also Clandinin & Caine, 2013, for in-depth touchstone descriptions). In Figure 10.3, we advance five aspects of what we would look for in a “good” narrative study.

When writing an interpretive biography, Denzin (1989) is primarily interested in the problem of “how to locate and interpret the subject in biographical materials” (p. 26). He advances several guidelines for writing:

The lived experiences of interacting individuals are the proper subject matter of sociology. The meanings of these experiences are best given by the persons who experience them; thus, a preoccupation with method, validation, reliability, generalizability, and theoretical relevance of the biographical method must be set aside in favor of a concern for meaning and interpretation.

6/2/2018 Bookshelf Online: Qualitative Inquiry and Research Design: Choosing Among Five Approaches

https://online.vitalsource.com/#/books/9781506330228/cfi/6/44!/4/2/4/2@0:0 13/22

Students of the biographical method must learn how to use the strategies and techniques of literary interpretation and criticism (i.e., bring their method in line with the concern about reading and writing of social texts, where texts are seen as “narrative fictions.” (Denzin, 1989, p. 26)

When an individual writes a biography, he or she writes himself or herself into the life of the subject about whom the individual is writing; likewise, the reader reads through her or his perspective.

Thus, within a humanistic, interpretive stance, Denzin (2001) identifies “criteria of interpretation” as a standard for judging the quality of a biography. These criteria are based on respecting the researcher’s perspective as well as on thick description. Denzin (2001) advocates for the ability of the researcher to illuminate the phenomenon in a thickly contextualized manner (i.e., thick description of developed context) so as to reveal the historical, processual, and interactional features of the experience. Also, the researcher’s interpretation must engulf what is learned about the phenomenon and incorporate prior understandings while always remaining incomplete and unfinished.

This focus on interpretation and thick description is in contrast to criteria established within the more traditional approach to biographical writing. For example, Plummer (1983) asserts that three sets of questions related to sampling, the sources, and the validation of the account should guide a researcher to a good life history study:

Is the individual representative? Edel (1984) asks a similar question: How has the biographer distinguished between reliable and unreliable witnesses? What are the sources of bias (about the participant, the researcher, and the participant–researcher interaction)? Or, as Edel (1984) questions, how has the researcher avoided making himself or herself simply the voice of the subject? Is the account valid when subjects are asked to read it, when it is compared to official records, and when it is compared to accounts from other participant?

Phenomenological Research What criteria should be used to judge the quality of a phenomenological study? From the many readings about phenomenology, one can infer criteria from the discussions about steps (Giorgi, 1985) or the “core facets” of transcendental phenomenology (Moustakas, 1994, p. 58). We have found direct discussions of the criteria to be missing, but perhaps Polkinghorne (1989) and van Manen (2014) come the closest in our readings when Polkinghorne discusses whether the findings are “valid” (p. 57) and when van Manen outlines validation and evaluative criteria.

For Polkinghorne, validation refers to the notion that an idea is well grounded and well supported. He asks, “Does the general structural description provide an accurate portrait of the common features and structural connections that are manifest in the examples collected?” (Polkinghorne, 1989, p. 57). He then proceeds to identify five questions that researchers might ask themselves:

Did the interviewer influence the contents of the participants’ descriptions in such a way that the descriptions do not truly reflect the participants’ actual experience? Is the transcription accurate, and does it convey the meaning of the oral presentation in the interview? In the analysis of the transcriptions, were there conclusions other than those offered by the researcher that could have been derived? Has the researcher identified these alternatives? Is it possible to go from the general structural description to the transcriptions and to account for the specific contents and connections in the original examples of the experience? Is the structural description situation specific, or does it hold in general for the experience in other situations? (Polkinghorne, 1989)

van Manen (2014) points to questions as a way to “test its [a phenomenology’s] level of validity” (p. 350).

6/2/2018 Bookshelf Online: Qualitative Inquiry and Research Design: Choosing Among Five Approaches

https://online.vitalsource.com/#/books/9781506330228/cfi/6/44!/4/2/4/2@0:0 14/22

Is the study based on a valid phenomenological question? In other words, does the study ask, “What is this human experience like?” “How is this or that phenomenon or event experienced?” A phenomenological question should not be confused with empirical studies of a particular population, person(s), or group of people at a particular time and location. Also, phenomenology cannot deal with causal questions or theoretical explanations. However, a particular individual or group may be studied for the understanding of a phenomenological theme—such as gender phenomenon, a sociopolitical event, or the experience of a human disaster. Is the analysis performed on experientially descriptive accounts, transcripts? (Does the analysis avoid empirical material that mostly consists of perceptions, opinions, beliefs, views, and so on?) Is the study properly rooted in primary and scholarly phenomenological literature—rather than mostly relying on questionable secondary and tertiary sources? Does the study avoid trying to legitimate itself with validation criteria derived from sources that are concerned with other (non-phenomenological) methodologies? (van Manen, 2014, pp. 350–351)

van Manen (2014) also provides criteria for evaluative appraisal of phenomenological studies.

Heuristic questioning: Does the text induce a sense of contemplative wonder and questioning attentiveness —ti estin (the wonder what this is) and hoti estin (the wonder that something exists at all)? Descriptive richness: Does the text contain rich and recognizable experiential material? Interpretive depth: Does the text offer reflective insights that go beyond the taken-for-granted understandings of everyday life? Distinctive rigor: Does the text remain constantly guided by a self-critical question of distinct meaning of the phenomenon or event? Strong and addressive meaning: Does the text “speak” to and address our sense of embodied meaning? Experiential awakening: Does the text awaken prereflective or primal experience through vocative and presentative language? Inceptual epiphany: Does the study offer us the possibility of deeper and original insight, and perhaps, an intuitive or inspirited grasp of the ethics and ethos of life commitments and practices? (pp. 355–356)

In Figure 10.4, we advance our five standards for assessing the quality of a phenomenology.

Grounded Theory Research Strauss and Corbin (1990) identified criteria by which one judges the quality of a grounded theory study. They describe seven criteria related to the general research process and six criteria related to the empirical grounding of a study. Corbin and Strauss (2015) advance the term checkpoint in place of criteria saying “dislike using the word criteria because that makes the evaluative process seem so dogmatic, an “all nothing” approach to evaluation” (p. 350, emphasis in original).

Figure 10.4 Standards for Assessing the Quality of a Phenomenology

6/2/2018 Bookshelf Online: Qualitative Inquiry and Research Design: Choosing Among Five Approaches

https://online.vitalsource.com/#/books/9781506330228/cfi/6/44!/4/2/4/2@0:0 15/22

Corbin and Strauss (2015) describe 16 checkpoints for guiding researchers and reviewers in evaluating the methodological consistency of a grounded theory study:

1. What was the target sample population? How was the original sample selected? 2. How did sampling proceed? What kinds of data were collected? Were there multiple sources of data and

multiple comparative groups? 3. Did data collection alternate with analysis? 4. Were ethical considerations taken into account in both data collection and analysis? 5. Were the concepts driving the data collection arrived at through analysis (based on theoretical sampling),

or were concepts derived from the literature and established before the data were collected not true theoretical sampling)?

6. Was theoretical sampling used, and was there a description of how it proceeded? 7. Did the research demonstrate sensitivity to the participants and to the data? 8. Is there evidence or examples of memos? 9. At what point did data collection end or a discussion of saturation end?

10. Is there a description of how coding proceeded along with examples of theoretical sampling, concepts, categories, and statements of relations? What were some of the events, incidents, or actions (indicators) that pointed to some of these major categories?

11. Is there a core category, and is there a description of how that core category was arrived at? 12. Were there changes in design as the research went along based on findings? 13. Did the researcher(s) encounter any problems while doing the research? Is there any mention of a negative

case, and how was that data handled? 14. Are methodological decisions made clear so that readers can judge their appropriateness for gathering data

(theoretical sampling) and doing analysis? 15. Was there feedback on the findings from other professionals and from participants? And were changes

made in the theory based on this feedback? 16. Did the researcher keep a research journal or notebook? (Corbin & Strauss, 2015, pp. 350–351)

They also advance 17 checkpoints for researchers and reviewers to evaluate the quality and applicability of a grounded theory study:

1. What is the core category, and how do the major categories relate to it? Is there a diagram depicting these relationship?

6/2/2018 Bookshelf Online: Qualitative Inquiry and Research Design: Choosing Among Five Approaches

https://online.vitalsource.com/#/books/9781506330228/cfi/6/44!/4/2/4/2@0:0 16/22

2. Is the core category sufficiently broad so that it can be used to study other populations and similar situations beyond this setting?

3. Are each of the categories developed in terms of their properties and dimensions so that they show depth, breadth, and variation?

4. Is there descriptive data given under each category that brings that theory to life so that it provides understanding and can be used in a variety of situations?

5. Has context been identified and integrated into the theory? Conditions and consequences should be listed merely as background information in a separate section but woven into the actual analysis with explanations of how they impact and flow from action–interaction in the data.

6. Has process been incorporated into the theory in the form of changes in action-interaction in relationship to changes in conditions? Is action–interaction matched to different situations, demonstrating how the theory might vary under different conditions and therefore be applied to different situations?

7. How is saturation explained, and when and how was it determined that categories were saturated? 8. Do the findings resonate or fit with the experience of both the professionals for whom the research ended

and the participants who took part in the study? Can participants see themselves in the story even if not every detail applies to them?

9. Are there gaps, or missing links, in the theory, leaving the reader confused and with a sense that something is missing?

10. Is there an account of extremes or negative cases? 11. Is variation built into the theory? 12. Are the findings presented in a creative and innovative manner? Does the research say something new or

put old ideas together in new ways? 13. Do findings give insight into situation and provide knowledge that can be applied to develop policy,

change practice, and add to the knowledge base of a profession? 14. Do the theoretical findings seem significant, and to what extent? It is entirely possible to complete a

theory-generating study, or any research investigation, yet not produce findings that are significant. 15. Do the findings have the potential to become part of the discussion and ideas exchanged among relevant

social and professional groups? 16. Are the limitations of the study clearly spelled out? 17. Are there suggestions for practice, policy, teaching, and application of the research? (Corbin & Strauss,

2015, pp. 351–352)

Charmaz (2014) reflects on the quality of the theory developed in a grounded theory study. She suggests that grounded theorists look at their theory and ask themselves the following evaluative questions:

Are the definitions of major categories complete? Have I raised major categories to concepts in my theory? How have I increased the scope and depth of the analysis in this draft? Have I established strong theoretical links between categories and between categories and their properties, in addition to the data? How have I increased understanding of the studied phenomenon? How does my grounded theory study make a fresh contribution? With which theoretical, substantive, or practical problems is this analysis most closely aligned? Which audiences might be most interested in it? Where shall I go with it? What implications does this analysis hold for theoretical each, depth, and breadth? For methods? For substantive knowledge? For actions or interventions? (Charmaz, 2014, pp. 337–338)

See also Charmaz (2014, pp. 337–338) for guiding questions for assessing criteria for grounded theory studies organized by four categories: credibility, originality, resonance, and usefulness. In Figure 10.5, we describe features of the general process and a relationship among the concepts we look for when evaluating a grounded theory study.

Ethnographic Research

6/2/2018 Bookshelf Online: Qualitative Inquiry and Research Design: Choosing Among Five Approaches

https://online.vitalsource.com/#/books/9781506330228/cfi/6/44!/4/2/4/2@0:0 17/22

Few ethnographic resources identify criteria for quality ethnographies. Instead, the preference, it seems, for ethnographers is to describe the “basics” of ethnographical studies as prolonged fieldwork that generate thick, contextual descriptions reflective of the triangulation of multiple data sources (Fetterman, 2010; Wolcott, 2008a, 2010). Two exceptions are Richardson (2000) and Spindler and Spindler (1987).

Richardson (2000) describes her criteria for evaluating ethnographies:

Substantive contribution: Does this piece contribute to our understanding of social life? Does the writer demonstrate a deeply grounded (if embedded) human world understanding and perspective? How has this perspective informed the construction of the text?

Figure 10.5 Features for Evaluating a Grounded Theory Study

Aesthetic merit: Does this piece succeed aesthetically? Does the use of creative analytical practices open up the text and invite interpretive responses? Is the text artistically shaped, satisfying, complex, and not boring? Reflexivity: How did the author come to write this text? How was the information gathered? Ethical issues? How has the author’s subjectivity been both a producer and a product of this text? Is there adequate self-awareness and self-exposure for the reader to make judgments about the point of view? Do authors hold themselves accountable to the standards of knowing and telling of the people they have studied? Impact: Does this affect me? Emotionally? Intellectually? Generate new questions? Move me to write? Move me to try new research practices? Move me to action.

6/2/2018 Bookshelf Online: Qualitative Inquiry and Research Design: Choosing Among Five Approaches

https://online.vitalsource.com/#/books/9781506330228/cfi/6/44!/4/2/4/2@0:0 18/22

Expresses a reality: Does this text embody a fleshed out, embodied sense of lived experience? Does it seem “true”—a credible account of a cultural, social, individual, or communal sense of the “real”? (p. 254)

The ethnographers Spindler and Spindler (1987) emphasize that the most important requirement for an ethnographic approach is to explain behavior from the “native’s point of view” (p. 20) and to be systematic in recording this information using note taking, tape recorders, and cameras. This requires that the ethnographer be present in the situation and engage in constant interaction between observation and interviews. These points are reinforced in Spindler and Spindler’s (1987) nine criteria for a “good ethnography”:

Criterion I. Observations are contextualized. Criterion II. Hypotheses emerge in situ as the study goes on. Criterion III. Observation is prolonged and repetitive. Criterion IV. Through interviews, observations, and other eliciting procedures, the native view of reality is obtained. Criterion V. Ethnographers elicit knowledge from informant-participants in a systematic fashion. Criterion VI. Instruments, codes, schedules, questionnaires, agenda for interviews, and so forth are generated in situ as a result of inquiry. Criterion VII. A transcultural, comparative perspective is frequently an unstated assumption. Criterion VIII. The ethnographer makes explicit what is implicit and tacit to informants. Criterion IX. The ethnographic interviewer must not predetermine responses by the kinds of questions asked. (p. 18)

This list, grounded in fieldwork, leads to a strong ethnography. Moreover, as Lofland (1974) contends, the study is located in wide conceptual frameworks; presents the novel but not necessarily new; provides evidence for the framework(s); is endowed with concrete, eventful interactional events, incidents, occurrences, episodes, anecdotes, scenes, and happenings without being “hyper-eventful”; and shows an interplay between the concrete and analytical and the empirical and theoretical. In Figure 10.6, we advance seven criteria we would look for in a “good” ethnography.

Figure 10.6 Criteria for a “Good” Ethnography

6/2/2018 Bookshelf Online: Qualitative Inquiry and Research Design: Choosing Among Five Approaches

https://online.vitalsource.com/#/books/9781506330228/cfi/6/44!/4/2/4/2@0:0 19/22

Case Study Research Evaluative criteria for a case study are provided by Stake’s (1995) rather extensive “critique checklist.” In so doing, Stake (1995) shares criteria for assessing a good case study report:

Is the report easy to read? Does it fit together, each sentence contributing to the whole? Does the report have a conceptual structure (i.e., themes or issues)? Are its issues developed in a serious and scholarly way? Is the case adequately defined? Is there a sense of story to the presentation? Is the reader provided some vicarious experience? Have quotations been used effectively? Are headings, figures, artifacts, appendixes, and indexes used effectively? Was it edited well—and then again with a last-minute polish? Has the writer made sound assertions, neither over- nor misinterpreting? Has adequate attention been paid to various contexts? Were sufficient raw data presented?

6/2/2018 Bookshelf Online: Qualitative Inquiry and Research Design: Choosing Among Five Approaches

https://online.vitalsource.com/#/books/9781506330228/cfi/6/44!/4/2/4/2@0:0 20/22

Were data sources well chosen and in sufficient number? Do observations and interpretations appear to have been triangulated? Is the role and point of view of the researcher nicely apparent? Is the nature of the intended audience apparent? Is empathy shown for all sides? Are personal intentions examined? Does it appear that individuals were put at risk? (p. 131)

Yin (2014) reflects on the quality of the description presented in a case study. He describes several characteristics for an exemplary case study.

Significant. Has the researcher focused the case(s) “unusual and of general public interest” or underlying issues “nationally important—either in theoretical terms or in policy or practical terms” (p. 201)? Complete. Has the researcher clearly defined the case(s) boundaries, collected extensive evidence, and conducted the study absent “of certain artefactual conditions” (p. 203)—for example, if a time or resource constraint unwittingly ended the study? Consider alternative perspectives. Has the researcher considered rival propositions and sought to collect evidence from differing perspectives in the case(s)? Display sufficient evidence. Has the researcher reported the case(s) in such a way that a reader can “reach an independent judgment regarding the merits” (p. 205)? Composed in an engaging manner. Has the researcher presented the case(s) in a way that communicates “the results widely”—either in writing or performance (p. 206)?

In Figure 10.7, we describe our six criteria for evaluating a “good” case study.

Comparing the Evaluation Standards of the Five Approaches The standards discussed for each approach differ slightly depending on the procedures of the approaches. Certainly less is mentioned about narrative research and its standards of quality, and more is available about the other approaches. From within the major books used for each approach, we have attempted to extract the evaluation standards recommended for their approach to research. To these, we have added our own standards that we use in our qualitative classes when we evaluate a project or study presented within each of the five approaches. We can compare these standards across the five approaches relating five dimensions summarized in Table 10.2. At the most fundamental level, the five differ in the focus of the study. A couple of potential similarities should be noted. Phenomenology, grounded theory, and ethnography typically focus the study on a singular phenomenon, process (or action or interaction), and culture-sharing group respectively. True, one may also focus narrative research or a case study on a single individual or case but it also possible to focus on two or three individuals or multiple cases. The study procedures offer distinguishing features for each approach whereby some research features the collection of stories (i.e., narrative), specification of cultural themes (i.e., ethnography), and selection of cases (i.e., case study)—others such as phenomenology are distinguishable by initial conveyance of an understanding of the philosophical tenets or integration of analysis into data collection activities and use of memoing (i.e., grounded theory). When examining a study, the approach can be identifiable by its presentation and outcomes of a story chronology for narrative research, theoretical diagramming for grounded theory research, explanation of how a culture-sharing group works for ethnography, and the assertions for a case study. Common across all the qualitative approaches is the use of reflexive and self-disclosing practices for embedding what the researcher brings to the study.

Figure 10.7 Evaluative Criteria for a Case Study

6/2/2018 Bookshelf Online: Qualitative Inquiry and Research Design: Choosing Among Five Approaches

https://online.vitalsource.com/#/books/9781506330228/cfi/6/44!/4/2/4/2@0:0 21/22

6/2/2018 Bookshelf Online: Qualitative Inquiry and Research Design: Choosing Among Five Approaches

https://online.vitalsource.com/#/books/9781506330228/cfi/6/44!/4/2/4/2@0:0 22/22