A7 - Field Notes Interviews

Creswell, J. W., & Poth, C. N. (2025). Qualita ve inquiry & research design: Choosing among five approaches (5th ed.). Sage Publica ons.

10 STANDARDS OF VALIDATION AND EVALUATION

Qualita ve researchers strive for “understanding,” that deep structure of knowledge that comes from visi ng personally with par cipants, spending extensive me in the field, and probing to obtain detailed meanings. During or a er a study, qualita ve researchers ask, “Did we get it right?” (Stake, 1995, p. 107) or “Did we publish a ‘wrong’ or inaccurate account?” (J. Thomas, 1993, p. 39). Is it possible to even have a right answer? To answer these ques ons, researchers need to look to themselves, to the par cipants, and to the readers. There are mul - or polyvocal discourses at work here that provide insight into the valida on and evalua on of a qualita ve research account. In this chapter, we address two interrelated ques ons: Is the account valid, and by whose standards? and How do we evaluate the quality of qualita ve research? Answers to these ques ons take us into the many perspec ves on valida on to emerge within the qualita ve community and the mul ple standards for evalua on discussed by authors with procedural, interpre ve, and postmodern perspec ves. Then, we examine the quality criteria for each of the five approaches to inquiry and compare standards of evalua on across the approaches. To represent the diversity of qualita ve perspec ves, we weave our own stances in with those of others from the literature. VALIDATION AND RELIABILITY IN QUALITATIVE RESEARCH Valida on of qualita ve research occurs throughout the research process, yet it is o en only in the wri en study descrip on that a reader becomes aware of the procedures researchers take to ensure the accuracy of their findings and generate evidence of credibility. It is important that we define our qualita ve approach to validity and reliability. We define qualita ve validity as the researcher checking for the accuracy of the findings by employing certain procedures (Creswell & Creswell, 2023). Qualita ve reliability indicates that the researcher’s approach is consistent across different researchers and among different projects (Gibbs, 2018). In the sec ons to follow, we introduce some of the diverse perspec ves that exist regarding the importance of valida on in qualita ve research and the terms and procedures used by

researchers to describe and establish a valid account. We then discuss reliability related to the procedures qualita ve researchers use to promote and assess coding consistency across mul ple researchers and data sets. Perspec ves on Valida on Within Qualita ve Research We begin by recognizing the diverse perspec ves within the qualita ve community for descrip on of and procedures for establishing the accuracy of a research account. Our view of valida on as an evolving construct means that a broad understanding of both tradi onal and contemporary perspec ves is essen al for informing the work and reading of qualita ve research. In Table 10.1, we present a sample of the perspec ves, arranged chronologically, available on valida on in the qualita ve literature. These perspec ves represent posi oning valida on in qualita ve research in a range of ways including (but not limited to) terms of quan ta ve equivalents, postmodern and interpre ve lenses, metaphors used for visualiza on, and present validity use. Evidence of evolving thinking about valida on in qualita ve research is apparent within and across authors, and we conclude this sec on by advancing a ra onale for our own stance.

Writers have searched for and found qualita ve equivalents that parallel tradi onal quan ta ve approaches to valida on. LeCompte and Goetz (1982) took this approach when they compared the issues of valida on and reliability to their counterparts in experimental design and survey research. They contended that qualita ve research had garnered much cri cism in the scien fic ranks for its failure to “adhere to canons of reliability and valida on” (LeCompte & Goetz, 1982, p. 31) in the tradi onal sense. They applied threats to internal valida on in experimental research to ethnographic research (e.g., history and matura on, observer effects, selec on and regression, mortality, spurious conclusions). They further iden fied threats to external valida on as “effects that obstruct or reduce a study’s comparability or translatability” (LeCompte & Goetz, 1982, p. 51). Some writers argue that authors who use posi vist terminology facilitate the acceptance of qualita ve research in a tradi onally focused quan ta ve world. Ely and colleagues asserted that using quan ta ve terms tended to be a defensive measure that muddied the waters and that “the language of posi vis c research is not congruent with or adequate to qualita ve work” (1991, p. 95). Lincoln and Guba (1985) offered alterna ve terms that, they contended, adhere more to naturalis c research. To establish the “trustworthiness” of a qualita ve study, Lincoln and Guba (1985) advanced the use of unique terms, such as credibility, authen city, transferability, dependability, and confirmability as “the naturalist’s equivalents” for the qualita ve equivalent of quan ta ve internal valida on, external valida on, reliability, and objec vity (p. 300). To opera onalize these new terms, they proposed techniques such as prolonged engagement in the field and the triangula on of data sources, methods, and inves gators to establish credibility. They described “thick descrip on” as necessary to ensure the transferability of findings between the researcher and those being studied. Rather than a focus on the reliability of an approach, researchers sought dependability that the results would be subject to change and instability. The naturalis c researcher looked for confirmability rather than objec vity in establishing the value of the data. Both dependability and confirmability were established through an audi ng of the research process. We find the Lincoln and Guba criteria s ll popular today in qualita ve reports.

Rather than using the term valida on, Eisner (1991) constructed standards such as structural corrobora on, consensual valida on, and referen al adequacy as evidence for asser ng the credibility of qualita ve research. In structural corrobora on, the researcher used mul ple types of data to support or contradict the interpreta on. Eisner described his purpose to “seek a confluence of evidence that breeds credibility, that allows us to feel confident about our observa ons, interpreta ons, and conclusions” (1991, p. 110). He further illustrated this purpose with an analogy drawn from detec ve work: The researcher compiles bits and pieces of evidence to formulate a “compelling whole.” At this stage, the researcher looked for recurring behaviors or ac ons and considered disconfirming evidence and contrary interpreta ons. Moreover, Eisner recommended that to demonstrate credibility, the weight of evidence should become persuasive. In consensual valida on, the researcher sought the opinion of others, which Eisner described as “an agreement among competent others that the descrip on, interpreta on, evalua on, and thema cs of an educa onal situa on are right” (1991, p. 112). Referen al adequacy suggested the importance of cri cism, and Eisner described the goal of cri cism as illumina ng the subject ma er and bringing about more complex and sensi ve human percep on and understanding. Qualita ve researchers also have reconceptualized valida on with a postmodern perspec ve. Lather (1991) ini ally iden fied four types of valida on (i.e., triangula on, construct valida on, face valida on, and cataly c valida on) as a “reconceptualiza on of valida on.” Lather commented that “paradigma c uncertainty in the human sciences is leading to the re- conceptualizing of valida on” and called for “new techniques and concepts for obtaining and defining trustworthy data which avoids the pi alls of orthodox no ons of valida on” (1991, p. 66). For Lather, the character of a social science report changed from a closed narra ve with a

ght argument structure to a more open narra ve with holes and ques ons and an admission of situatedness and par ality. In Ge ng Smart, Lather (1991) described triangula on as drawing upon mul ple data sources, methods, and theore cal schemes. Construct valida on recognized the constructs that existed rather than imposing theories or constructs on informants or the context. Face valida on was best explained by Kidder (1982) as “a ‘click of recogni on’ and a ‘yes, of course,’ instead of ‘yes, but’ experience” (p. 56). Cataly c valida on energized par cipants to consider knowing reality to transform it. In a later ar cle, Lather’s (1993) terms became more unique and closely related to feminist research in “four frames of valida on.” The first, ironic valida on, was where the researcher presented truth as a problem. The second, paralogic valida on, was concerned with undecidables, limits, paradoxes, and complexi es. This valida on moved away from theorizing things and toward providing direct exposure to other voices in an almost unmediated way. The third, rhizoma c valida on, pertained to ques oning prolifera ons, crossings, and overlaps without underlying structures or deeply rooted connec ons. The researcher should also ques on taxonomies, constructs, and interconnected networks in which the reader jumped from one assemblage to another and consequently moved from judgment to understanding. The fourth type was situated, embodied, or voluptuous valida on, which meant that the researcher sets out to understand more than one can know and to write toward what one does not understand.

Other writers, such as Wolco (1990a), had li le use for valida on, sugges ng that “valida on neither guides nor informs” his work (p. 136). He did not dismiss valida on but rather placed it in a broader perspec ve. Wolco ’s (1990a) goal was to iden fy “cri cal elements” and write “plausible interpreta ons from them” (p. 146). He ul mately tried to understand rather than convince, and he voiced the view that valida on distracted from his work of understanding what was really going on. Wolco (1990a, 1994) claimed that the term valida on did not capture the essence of what he sought, adding that perhaps someone would coin a term more appropriate for the naturalis c paradigm. But for now, he said, the term understanding seemed to encapsulate the idea as well as any other (Wolco , 1994). Valida on has also been cast within an interpre ve approach to qualita ve research marked by a focus on the importance of the researcher; a lack of truth in valida on; a form of valida on based on nego a on and dialogue with par cipants; and interpreta ons that are temporal, located, and always open to reinterpreta on (Angen, 2000). Angen suggested that within interpreta ve research, valida on was “a judgment of the trustworthiness or goodness of a piece of research” (2000, p. 387). She espoused an ongoing, open dialogue on the topic of what made interpre ve research worthy of our trust. Considera ons of valida on were not defini ve as the final word on the topic, nor should every study be required to address them. Further, she advanced two types of valida on: ethical valida on and substan ve valida on. Ethical valida on meant that all research agendas must ques on their underlying moral assump ons, their poli cal and ethical implica ons, and the equitable treatment of diverse voices. It also required research to provide some prac cal answers to ques ons. Angen (2000) also proposed that our interpre ve qualita ve research should have a “genera ve promise” (p. 389) and raise new possibili es, open up new ques ons, s mulate new dialogue, and provide nondogma c answers to the ques ons posed. In so doing, interpre ve qualita ve research must have transforma ve value leading to ac on and change. Substan ve valida on meant understanding one’s own topic, understandings derived from other sources, and the documenta on of this process in the wri en study. Self-reflec on contributed to the valida on of the work. The interpre ve qualita ve researcher, as a sociohistorical interpreter, interacted with the subject ma er to co-create the interpreta ons derived. Understandings derived from previous research gave substance to the inquiry. Interpre ve research also was a chain of interpreta ons that must be documented for others to judge the trustworthiness of the meanings arrived at in the end. Wri en accounts must resonate with their intended audiences, and must be compelling, powerful, and convincing. A synthesis of valida on perspec ves comes from Whi emore et al. (2001), who analyzed 13 wri ngs about valida on and extracted key valida on criteria from these studies. They organized these criteria into primary and secondary criteria. They found four primary criteria: credibility (Are the results an accurate interpreta on of the par cipants’ meaning?); authen city (Are different voices heard?); cri cality (Is there a cri cal appraisal of all aspects of the research?); and integrity (Are the inves gators self-cri cal?). Secondary criteria related to explicitness, vividness, crea vity, thoroughness, congruence, and sensi vity. In summary, with these criteria, it seemed like the valida on standard had moved toward the interpre ve lens of

qualita ve research, with an emphasis on researcher reflexivity and on researcher challenges that included raising ques ons about the ideas developed during a research study. A postmodern perspec ve drew on the metaphorical image of a crystal. Richardson (in Richardson & St. Pierre, 2005) described this image:

A final perspec ve was drawn from Lincoln et al. (2011). They captured the many perspec ves developed through the years. They suggested that the ques on of validity criteria is not whether we should have such criteria or whose criteria the scien fic community might adopt, but rather how criteria needed to be developed within the projected transforma ons being suggested by social scien sts. To this end, they reviewed establishing authen city but framed it within the perspec ves of a balance of views, raising the level of awareness among par cipants and other stakeholders, and advancing ac on on the part of research par cipants and their training. Lincoln et al. (2011) also saw a role for validity in understanding hidden assump ons and ethical rela onships with research par cipants through such standards as posi oning themselves, having discourses, encouraging voices, and being self-reflec ve. Given the many perspec ves of valida on within the qualita ve research community, we acknowledge the ongoing debate regarding the many types of and terms for valida on in qualita ve research (e.g., Beck, 2020). We suggest the need for authors to choose the types and terms with which they are comfortable and reference the terms and strategies writers use. To summarize our own stance, we consider “valida on” in qualita ve research to be an a empt to assess the “accuracy” of the findings, as best described by the researcher, the par cipants, and the readers (or reviewers). This view also suggests that any report of research is a representa on by the author. We also view valida on as a dis nct strength of qualita ve research by the researcher spending extensive me in the field se ng, detailing thick descrip on in their work, and remaining close to par cipants in the study. We use the term valida on to emphasize a process (see Angen, 2000; Hayashi et al., 2019), rather than verifica on (which has quan ta ve overtones) or historical words such as trustworthiness and authen city (recognizing that many qualita ve writers do return to these words, sugges ng the “staying power” of Lincoln and Guba’s, 1985, standards; see Whi emore et al., 2001). Our adop on of a “process view” to valida on is based on the longstanding premise that the ques on of whether the account is valid cannot and should not be based on a single strategy. Instead, valida on strategies genera ng evidence should be embedded throughout the qualita ve research process and be sufficiently flexible to the mul ple contexts in which the

qualita ve research takes place. As an illustra on, we suggest Hayashi et al.’s (2019) reflec ons highligh ng contextual influences on valida on strategies throughout their research process. The subject of valida on arises in several of the approaches to qualita ve research (e.g., Corbin & Strauss, 2015; Riessman, 2008; Stake, 1995), but we do not think that dis nct valida on approaches exist for the five approaches to qualita ve research. At best, there might be less emphasis on valida on in narra ve research and more emphasis on it in grounded theory, case study, and ethnography, especially when the authors of these approaches want to employ systema c procedures. We recommend using mul ple valida on strategies regardless of the type of qualita ve approach. Our framework for thinking about valida on in qualita ve research is to suggest that researchers employ accepted strategies to document the accuracy of their studies. These we call valida on strategies.

Valida on Strategies It is not enough for researchers to be able to define and describe the essen al role of valida on in qualita ve research. They need to be able to translate their understandings of valida on into prac ce as strategies. We describe nine strategies frequently used by qualita ve researchers during the process of valida on adapted from the work of Creswell and Miller (2000) and provide some general guidance about how we go about implemen ng these strategies (see Figure 10.1).

Figure 10.1 Strategies for Valida on in Qualita ve Research The strategies are not presented in any specific order of importance but are organized in three groups by the lens the strategy represents: researcher’s lens, par cipant’s lens, and reader’s or reviewer’s lens (Creswell & Báez, 2021). Researcher’s Lens Among the many roles a researcher undertakes is checking the accuracy of a qualita ve account and any of the following valida on strategies can assist the researcher in this effort:

Corrobora ng evidence through triangula on of mul ple data sources. The researcher makes use of mul ple and different sources, methods, inves gators, and theories to provide corrobora ng evidence (Bazeley, 2021; Ely et al., 1991; Erlandson et al., 1993; Gibbs, 2018; Glesne, 2016; Lincoln & Guba, 1985; Miles & Huberman, 1994; Pa on, 1980, 1990, 2015; Yin, 2017). Typically, this process involves corrobora ng evidence from different sources or researchers to shed light on a theme or perspec ve. When qualita ve researchers locate evidence to document a code or theme in different sources of data, they are triangula ng informa on and providing validity to their findings. For this valida on strategy, we begin considering how various data sources can be used in tandem when planning the study. Then, as data is collected, we further explore evidence of corrobora on and use these insights in our interpreta on and wri ng. See Goodrum et al. (2022; Appendix E) for a descrip on of their data triangula on efforts involving mul ple components and official documents. Discovering a nega ve case analysis or disconfirming evidence. The researcher refines working hypotheses as the inquiry advances in light of nega ve or rival evidence (Ely et al., 1991; Lincoln & Guba, 1985; Miles & Huberman, 1994; Pa on, 1980, 1990, 2015; Yin, 2017). Not all evidence will fit the pa ern of a code or a theme. It is necessary then to report this nega ve evidence, and in doing so, the researcher provides a realis c assessment of the phenomenon under study. In real life, not all evidence is either posi ve or nega ve; it is some of both. For this valida on strategy, we both admit that we tend to be a en ve to such evidence and make a point of following what we call “points of intrigue” throughout the study. We find these points are o en our key points of discussion in our wri ng. Clarifying researcher bias or engaging in reflexivity. The researcher discloses their understandings about the biases, values, and experiences that they bring to a qualita ve research study from the outset of the study so that the reader understands the posi on from which the researcher undertakes the inquiry (Hammersley & Atkinson, 1995, 2019; Merriam & Tisdell, 2015). In this clarifica on, according to Weiner-Levey and Popper- Giveon (2013), the researcher illuminates what they call the “dark ma er” that is o en omi ed in qualita ve research by commen ng on past experiences, biases, prejudices, and orienta ons that have likely shaped the interpreta on and approach to the study. For this valida on strategy, we embed opportuni es throughout a study for wri ng and

discussing connec ons that emerge with our past experiences and perspec ves. See Trip et al. (2019; Appendix C) for their descrip on of how memoing helped them engage in reflexivity.

Par cipant’s Lens Par cipants can play an important role in the following valida on strategies:

Member checking or seeking par cipant feedback. The researcher solicits par cipants’ views of the credibility of the findings and interpreta ons (Bazeley, 2021; Ely et al., 1991; Erlandson et al., 1993; Glesne, 2016; Lincoln & Guba, 1985; Merriam & Tisdell, 2015; Miles & Huberman, 1994). This technique is considered by Lincoln and Guba (1985) to be “the most cri cal technique for establishing credibility” (p. 314). This approach, writ large in most qualita ve studies, involves taking data, analyses, interpreta ons, and conclusions back to the par cipants so that they can judge the accuracy and credibility of the account. According to Stake (1995), par cipants should “play a major role direc ng as well as ac ng in case study” research (p. 115). They should be asked to examine rough dra s of the researcher’s work and to provide alterna ve language, “cri cal observa ons or interpreta ons” (Stake, 1995, p. 115). In so doing, par cipants play a cri cal role because they are asked “how well the ongoing data analysis represents their experience” (Hays & Singh, 2012, p. 206). For this valida on strategy, we o en convene a focus group made up of par cipants in the study either in person or virtually and ask them to reflect on the accuracy of the account. We do not take back to par cipants the transcripts or the raw data, but the preliminary analyses consis ng of descrip on or themes. We are interested in their views of these wri en analyses as well as what was missing. Another op on is to use e-mail to share a summary of the preliminary findings with focus group par cipants and seek wri en feedback. See Richards (2021) for prac cal guidance about how to interpret par cipant feedback. Also see Chance (2022; Appendix B) for her descrip on of member checking with par cipants. Having a prolonged engagement and persistent observa on in the field. The researcher makes field-based decisions about what is salient to study, relevant to the purpose of the study, and of interest for focus. Researchers build rapport with par cipants and gatekeepers, learn the culture and context, and check for misinforma on that stems from distor ons introduced by themselves or informants (Ely et al., 1991; Erlandson et al., 1993; Glesne, 2016; Lincoln & Guba, 1985; Merriam & Tisdell, 2015). Fe erman (2019) contended that “par cipant observa on requires close, long-term contact with the people under study” (p. 50). For this valida on strategy, we spend as much me in the field as feasible during the study. We begin data collec on by familiarizing ourselves with the site and par cipants. Throughout the study we also reflect upon our observa ons and documents as our understandings evolve and emerge. See Mac an Ghaill and Haywood (2015; Appendix D) for their descrip on of their 3-year ethnographic work and how access to the community was “greatly enabled by our being

known for our social commitment to the local area, working with families in the local community” (p. 111) Collabora ng with par cipants. The researcher embeds opportuni es for par cipants to be involved throughout the research process in varying ways and degrees. Among the various ways is involvement in key research decisions such as developing data collec on protocols and contribu ng to data analysis and interpreta on. The degree to which par cipants are involved can vary along a con nuum from minimal to extensive. Par cipant involvement is based on the idea (and ever-growing body of research) that the study is more likely to be supported and findings used when par cipants are involved (Pa on, 2011, 2015). For this valida on strategy, we are o en guided by community-based par cipatory research prac ces that involve par cipants as co- researchers in the study (see, e.g., Hacker, 2013, Wallerstein et al., 2018, for further discussions).

Reader’s or Reviewer’s Lens Including others beyond the researcher and those involved in the research contribute in the following valida on strategies:

Enabling external audits. The researcher facilitates audi ng by securing the services of an external consultant, the auditor, to examine both the process and the product of the account to assess their accuracy (Erlandson et al., 1993; Lincoln & Guba, 1985; Merriam & Tisdell, 2015; Miles & Huberman, 1994). This auditor should have no connec on to the study. In assessing the study, the auditor examines whether or not the findings, interpreta ons, and conclusions are supported by the data. Lincoln and Guba (1985) compared this, metaphorically, with a fiscal audit, and the procedure provided a sense of interrater reliability to a study. This process can be assisted by the crea on of documenta on, some mes referred to as an audit trail and described by Silver and Lewins (2014) as “comprising a log of all the processes followed, describing the small analy c leaps contribu ng to the analysis as a whole” (p. 140). For this valida on strategy, we engage in two processes: First, we create a tracking document at the beginning of a study on which we detail our key decisions including ra onale and poten al consequences. Second, when resources permit, we use an auditor to review our process and findings. Genera ng a rich, thick descrip on. The researcher allows readers to make decisions regarding transferability because the writer describes in detail the par cipants or se ng under study (Erlandson et al., 1993; Lincoln & Guba, 1985; Merriam & Tisdell, 2015). With such a detailed descrip on, the researcher enables readers to transfer informa on to other se ngs and to determine whether the findings can be transferred “because of shared characteris cs” (Erlandson et al., 1993, p. 32). Thick descrip on means that the researcher provides details when describing a case or when wri ng about a theme. According to Stake (2010), “a descrip on is rich if it provided abundant, interconnected details” (p. 49). Detail can emerge through physical descrip on, movement descrip on,

and ac vity descrip on. It can also involve describing from the general ideas to the narrow, interconnec ng the details, and using strong ac on verbs and quotes. For this valida on strategy, we devote me to revisi ng our raw data soon a er its collec on to add further descrip on that might be helpful during the analysis—for example, contextual descrip ons like atmosphere. See Chan (2010; Appendix A) for her descrip on of how she was able to generate such a detailed narra ve story about an individual’s lived experiences from her varied interac ons over 2 years that took place at home, at school, with those around her, and the use of extensive field notes. Having a peer review or debriefing of the data and research process. The researcher seeks an external check by “someone who is familiar with the research or the phenomenon explored” (Creswell & Miller, 2000, p. 129), in much the same spirit as interrater reliability in quan ta ve research (Ely et al., 1991; Erlandson et al., 1993; Glesne, 2016; Lincoln & Guba, 1985; Merriam & Tisdell, 2015). Lincoln and Guba (1985) defined the role of the peer debriefer as a “devil’s advocate,” an individual who kept the researcher honest; asked hard ques ons about methods, meanings, and interpreta ons; and provided the researcher with the opportunity for catharsis by sympathe cally listening to the researcher’s feelings. For this valida on strategy, we involve colleagues and students as reviewers (and for our students we play this role) in what Lincoln and Guba (1985) called peer debriefing sessions during which both the reviewers and the researcher keep wri en accounts of the sessions.

Examining these nine validity strategies as a whole (just discussed and outlined in Figure 10.1), we advise that researchers engage in at least two of them in any given qualita ve study. Unques onably, procedures such as triangula ng among different data sources (assuming that the inves gator collects more than one), wri ng with detailed and thick descrip on, and taking the en re wri en narra ve back to par cipants in member checking all are reasonably easy procedures to conduct. They also are the most popular and cost-effec ve procedures. Other procedures, such as peer audits and external audits, are more me-consuming in their applica on and may also involve substan al costs to the researcher. We also point out that differences among the validity lenses (i.e., researchers, par cipants, readers, and reviewers) can be a ributed to the philosophical orienta on of the researcher and may therefore impact their use of specific valida on strategies (see Creswell & Báez, 2021, for further discussion).

Reliability Perspec ves and Procedures

Reliability can be addressed in qualita ve research in several ways (Bazeley, 2021). Richards (2021) described the need for conveying to readers transparent details about the data-handling processes. This informa on assured the reliable examina on of data records and consistent applica on of codes and categories. Reliability can be enhanced if the researcher kept detailed field notes and checked the accuracy and completeness of transcrip ons of recordings. This completeness included men oning the trivial, but o en crucial, pauses and overlaps. Further coding checks can be done by researchers using a codebook (possibly held by individuals not previously involved in the study). Bazeley (2021) emphasized the need for training a research team and documen ng coding and the reliability of coding through computer QDAS programs. Our focus on reliability here will be on intercoder agreement based on the use of mul ple coders to analyze transcript data. In qualita ve research, reliability o en refers to the stability of responses by mul ple coders of data sets. It is important to develop codes and assess the reliability among coders as part of the analysis process (Kuckartz, & Rädiker, 2023; Richards & Morse, 2013; Saldaña, 2021). Saldaña (2021) described intercoder agreement as providing a “crowd-sourcing reality check” for each other in the data analysis process (p. 53). We find this prac ce especially used in qualita ve health science research and within the form of qualita ve research in which inquirers want an external check on the highly interpre ve coding process. What seems to be largely missing in the literature (with the excep on of Armstrong et al., 1997; Bernard, 2017; Campbell et al., 2013; Miles & Huberman, 1994; and Miles et al., 2014, 2019) is a discussion about the procedures of actually conduc ng intercoder agreement checks. One of the key issues is determining what exactly the coders agree on, whether they seek agreement on code names, the coded passages, or the same passages coded the same way. We also need to decide on whether researchers seek agreement based on codes, themes, or both codes and themes (see Armstrong et al., 1997). Finally, we need to carefully interpret our findings—as Richards (2021) wisely advised, we cannot expect to find complete consistency in coding over me or between coders. Undoubtedly, there is flexibility in the process, and researchers need to fashion an approach consistent with the resources and me to engage in coding. Drawing upon our experiences working with mul ple coders, we propose the following procedures for assessing intercoder agreement in coding of qualita ve research (see Figure 10.2):

Establish a common pla orm for coding and develop a preliminary code list. The researchers decide which so ware program or paper-based methods they will use; a common pla orm is essen al to be able to easily share the results of their ini al read and preliminary coding efforts. In our recent work, we have o en used computer- assisted qualita ve data analysis so ware packages (MAXQDA, ATLAS. , or NVivo, depending on familiarity across researchers) and have implemented training sessions at the beginning of the study. Each researcher then reads several transcripts independently and develops a list of preliminary codes. As discussed in Chapter 8, computer so ware programs have features that facilitate the crea on of these lists.

Develop and share the ini al codebook among coders. The researchers develop a shared understanding of codes to create a codebook that is stable and represents the coding analysis of four independent coders. To do this, a er coding, say, three transcripts (i.e., A, B, and C), we then meet and examine the codes, their names, and the text segments that each researcher codes. We begin developing an ini al codebook of the major codes. This codebook contains a defini on of each code and the text segments that we assigned to each code. In this ini al codebook, we had main codes and subcodes. In this ini al codebook, we focus on the main codes we were finding in the data rather than forming an exhaus ve list. Addi onal codes are added as the analyses proceed. Apply the codebook to addi onal transcripts and compare coding across mul ple researchers. The researchers independently apply the shared codebook to addi onal transcripts and then compare their coding to assess consistency. To do this, the researchers had to agree upon a type of data segment to compare, whether it be a phrase, sentence, or paragraph, and then each researcher independently codes three addi onal transcripts (i.e., D, E, and F). With no clear guidance in the literature as to what the appropriate data segment should be for coding, consensus can be difficult to reach (Hruschka et al., 2004). To streamline the process in their exploratory study, Campbell and colleagues (2013) suggested having the lead researcher determine the type of data segments to be coded. We feel it is more important to have agreement on the type of data segments we were assigning to codes than to have the same, exact passages coded. The decision about the type of data segment is crucial because it becomes the basis on which intercoder agreement is defined, when we are ready to actually compare our coding. Thus, intercoder agreement means that we agree that when we assign a code word to a passage, we all assign this same code word to the passage. It does not mean that we all code the same passages—an ideal that we believe would be hard to achieve because some people code short passages and others longer passages. Nor does it mean that we all bracket the exact same lines for our code word, another ideal difficult to achieve. Assess and report intercoder agreement among researchers. The researchers define individual instances of intercoder agreement before assessing overall intercoder agreement. In our work, we tend to take a realis c stance, look at the passages that are coded across researchers, and ask ourselves whether we all assign the same code word to the passage based on our tenta ve defini ons in the codebook. The decision would be either a yes or a no, and we could calculate the percentage of agreement among all researchers on this passage that we all code. We seek to establish a high level of agreement on coding of these passages informed by an 80%-to-90% range that seems a minimal benchmark by many (Miles & Huberman, 1994; Saldaña, 2021). This range is similar to the recommenda on by Miles et al. (2014, 2019). We suggest keeping in mind that there is no agreed-upon standard to indicate a high level of intercoder agreement and that the size and range of the coding scheme means it can be difficult (or even impossible) for researchers to reach the 80%-to-90% range of agreement. Many QDAS packages include features to facilitate the calcula on among mul ple coders. Other

researchers might actually calculate a kappa reliability sta s c on the agreement, but we feel that a percentage is sufficient to report in our published studies (see also Creswell & Báez, 2021, for prac cal guidance about three different approaches to intercoder agreement). Revise and finalize the codebook to inform further coding. The researchers review and refine the codebook to further differen ate code defini ons. In our work, a er the process con nues through several more transcripts, we then revise the codebook and conduct anew an assessment of passages that all researchers code to determine if we apply the same or different codes. With each phase in the intercoder agreement process, we hope to achieve a higher percentage of agreed-upon codes and themes for data segments. Then we can collapse codes into broader themes and can conduct the same process with themes, to see if the passages all coded as themes by the researchers were consistent in the use of the same theme.

Figure 10.2 Procedures for Reliability of Intercoder Agreement in Qualita ve Research

EVALUATION CRITERIA FOR QUALITATIVE RESEARCH In reviewing valida on in the qualita ve research literature, we are struck by how valida on is some mes used in discussing the quality of a study (e.g., Angen, 2000). Although valida on is certainly an aspect of evalua ng the quality of a study, other criteria are also useful. In reviewing evalua on criteria, we find that here, too, the standards vary within the qualita ve community (see Creswell & Gue erman, 2019, who contrast three approaches to qualita ve evalua on). To discuss evalua on, we reprise the features of a “good” qualita ve study introduced in Chapter 3, and review three standards (i.e., general standards, interpre ve and postmodern standards, and publica on standards), and then turn to specific criteria within each of our five approaches to qualita ve research. In Figure 10.3, we present the criteria of a “good” qualita ve study presented earlier in our Chapter 3 discussion about the characteris cs of qualita ve research. Also in Figure 10.3, we reference key content covered in different chapters of this book for each of the “good” features. In this way, readers can review detailed content for each feature.

Figure 10.3 Reprising the Features of a “Good” Qualita ve Study

Other writers provide addi onal thoughts about helpful evalua on criteria. A methodological perspec ve comes from Howe and Eisenhardt (1990), who suggested that only broad, abstract standards are possible for qualita ve (and quan ta ve) research. Moreover, to determine, for example, whether a study is a good ethnography cannot be answered apart from whether the study contributes to our understanding of important ques ons. Silverman (2022) advanced four criteria that good research must sa sfy. The following five standards are adapted from Howe and Eisenhardt (1990) and are put in ques on form for researchers to apply to their work:

Does the research ques on drive the data collec on and analysis? (That is, rather than the reverse?) Silverman (2022) states similar criteria about good research using “methods which are demonstrably appropriate to the research problem” (p. 425). To what extent are the data collec on and analysis techniques competently applied? (That is, in a technical sense?) Silverman (2022) states similar criteria about good research developing “empirically sound, reliable, and valid findings” (p. 425). Are the researcher’s assump ons made explicit? (That is, such as the researcher’s own subjec vity?) Does the study have overall warrant? (That is, is it robust, does it use respected theore cal explana ons, and does it discuss disconfirmed theore cal explana ons?) Silverman (2022) states similar criteria about good research thinking “theore cally through and with data” (p. 425). Does the study have value both in informing and improving prac ce in an ethical manner? (That is, does it contribute to the “so what?” ques on and is the study conducted ethically? Are par cipants protected from harm by taking all measures to protect the confiden ality, privacy, and truth telling of par cipants?) Silverman (2022) states similar criteria about good research contribu ng “where possible, to prac ce and policy” (p. 425).

A postmodern, interpre ve framework forms a second perspec ve. Lincoln (1995) thought about the quality issue in terms of emerging criteria. She tracked her own thinking (and that of her late colleague, Guba). Her thinking extended from early approaches of developing parallel methodological criteria (Lincoln & Guba, 1985) to establishing the criteria of “fairness” (a balance of stakeholder views), to sharing knowledge and fostering social ac on (Guba & Lincoln, 1989), and on to her current stance. Her new emerging approach to quality was based on three new commitments: emergent rela ons with respondents, a set of stances, and a vision of research that enables and promotes jus ce. Based on these commitments, Lincoln (1995) then proceeded to iden fy several standards of quality:

The standard is set in the inquiry community, such as by guidelines for publica on. These guidelines admit that within diverse approaches to research, inquiry communi es have

developed their own tradi ons of rigor, communica on, and ways of working toward consensus. These guidelines, she also maintains, serve to exclude legi mate research knowledge and social science researchers. The standard of posi onality guides interpre ve or qualita ve research. Drawing on those concerned about standpoint epistemology, this means that the text should display honesty or authen city about its own stance and about the posi on of the author. Another standard is under the rubric of community. This standard acknowledges that all research takes place in, is addressed to, and serves the purposes of the community in which it is carried out. Such communi es might be feminist thought, Black scholarship, Na ve American studies, or ecological studies. Interpre ve or qualita ve research must give voice to par cipants so that their voice is not silenced, disengaged, or marginalized. Moreover, this standard requires that alterna ve or mul ple voices be heard in a text. Cri cal subjec vity as a standard means that the researcher needs to have heightened self-awareness in the research process and create personal and social transforma on. This “high-quality awareness” enables the researcher to understand his or her psychological and emo onal states before, during, and a er the research experience. High-quality interpre ve or qualita ve research involves reciprocity between the researcher and those being researched. This standard requires that intense sharing, trust, and mutuality exist. The researcher should respect the sacredness of rela onships in the research-to-ac on con nuum. This standard means that the researcher respects the collabora ve and egalitarian aspects of research and “make[s] spaces for the lifeways of others” (Lincoln, 1995, p. 284). Sharing of the privileges acknowledges that in good qualita ve research, researchers share their rewards with persons whose lives they portray. This sharing may be in the form of royal es from books or the sharing of rights to publica on.

A final perspec ve uses interpre ve standards for conduc ng qualita ve research. Drawing from their own experience reviewing papers or monographs submi ed for social science publica on, Richardson (in Richardson & St. Pierre, 2018) suggests the following criteria:

Substan ve contribu on. Does this piece contribute to our understanding of social life? Does the writer demonstrate a deeply grounded (if embedded) social scien fic perspec ve? Does this piece seem “true”—a credible account of a cultural, social, individual, or communal sense of the “real”?

Aesthe c merit. Does this piece succeed aesthe cally? Does the use of crea ve analy cal prac ces open up the text and invite interpre ve responses? Is the text ar s cally shaped, sa sfying, complex, and not boring? Reflexivity. How has the author’s subjec vity been both a producer and a product of this text? Is there adequate self-awareness and self-exposure for the reader to make judgments about the point of view? Does the author hold himself or herself accountable to the standards of knowing and telling of the people he or she has studied? Impact. Does this piece affect me emo onally or intellectually? Does it generate new ques ons or move me to write? Does it move me to try new research prac ces or move me to ac on? (p. 823)

As applied research methodologists, we prefer the methodological standards of evalua on, but we can also support the postmodern and interpre ve perspec ves. We also agree with Flick (2014), who stated, “the problem of how to assess qualita ve research has not yet been solved” (p. 480). S ll, we return to our criteria for a “good” qualita ve study as men oned in Figure 10.3. Our features highlight the choice of a methodological approach (i.e., one of the five approaches men oned in this book) and strong methods for data collec on and analysis in a qualita ve study. Next we discuss specific evalua ve criteria for each of the five approaches. EVALUATION CRITERIA SPECIFIC TO EACH OF THE FIVE APPROACHES In the sec on that follows, we provide prac cal guidance of evalua on criteria specific to each of the five approaches of qualita ve inquiry. What standards of evalua on, beyond those already men oned, would signal a high-quality narra ve study, a phenomenology, a grounded theory study, an ethnography, and a case study? Narra ve Research In discussing what makes for a good narra ve study, both Riessman (2008) and Clandinin (2013, 2023) pointed to seeking coherence of par cipants’ narra ves yet acknowledged that it may not always be possible. To that end, Riessman (2008) proposed ques ons for assessing coherence (p. 189):

Do episodes of a life story hang together? Are sec ons of a theore cal argument linked and consistent? Are there major gaps and inconsistencies? Is the interpreter’s analy c account persuasive?

In 2013, Clandinin proposed a way for “judging and responding to narra ve inquiries” (p. 211) by describing what she and Vera Caine defined as touchstones:

Clandinin (2023, p. 146) listed the following 12 touchstones and considered them as evolving (see also Clandinin & Caine, 2013, for in-depth touchstone descrip ons):

1. Rela onal responsibili es 2. In the midst 3. Nego a on of rela onships 4. Narra ve beginnings 5. Nego a ng entry to the field 6. Moving from field to field texts 7. Moving from field texts to interim and final research texts 8. Represen ng narra ves of experience in ways that show temporality, sociality, and place 9. Rela onal response communi es 10. Jus fica ons—personal, prac cal, social 11. A en ve to mul ple audiences 12. Commitment to understanding lives in mo on

When wri ng an interpre ve biography, Denzin (1989) was primarily interested in the problem of “how to locate and interpret the subject in biographical materials” (p. 26). Denzin (1989) advanced several guidelines for wri ng:

Thus, within a humanis c, interpre ve stance, Denzin (2001) iden fied “criteria of interpreta on” as a standard for judging the quality of a biography. These criteria were based on respec ng the researcher’s perspec ve as well as thick descrip on. Denzin (2001) advocated for the ability of the researcher to illuminate the phenomenon in a thickly contextualized manner (i.e., thick descrip on of developed context) so as to reveal the historical, processual, and interac onal features of the experience. Also, the researcher’s interpreta on engulfed what was learned about the phenomenon and incorporated prior understandings while always remaining incomplete and unfinished.

This focus on interpreta on and thick descrip on was in contrast to criteria established within the more tradi onal approach to biographical wri ng. For example, Plummer (1983) asserted three sets of ques ons for a good life history study:

Is the individual representa ve? Edel (1984) asked a similar ques on: How has the biographer dis nguished between reliable and unreliable witnesses? What are the sources of bias (about the par cipant, the researcher, and the par cipant– researcher interac on)? Or, as Edel (1984) ques ons, how has the researcher avoided making himself or herself simply the voice of the subject? Is the account valid when subjects are asked to read it, when it is compared to official records, and when it is compared to accounts from other par cipants?

In Figure 10.4, we advance five criteria for guiding what we would look for in a narra ve study.

Figure 10.4 Guiding Evalua ve Criteria for a Narra ve Study Phenomenological Research What criteria should be used to judge the quality of a phenomenological study? From the many readings about phenomenology, one can infer criteria from the discussions about steps (Giorgi, 1985) or the “core facets” of transcendental phenomenology (Moustakas, 1994, p. 58). We have found direct discussions of the criteria to be missing, but perhaps Polkinghorne’s (1989) discussion of whether the findings are “valid” (p. 57) and van Manen’s (2014) outline of valida on and evalua ve criteria come the closest in our readings.

For Polkinghorne, valida on referred to the no on that an idea was well grounded and well supported. He asked, “Does the general structural descrip on provide an accurate portrait of the common features and structural connec ons that are manifest in the examples collected?” (Polkinghorne, 1989, p. 57). He then proceeded to iden fy five ques ons that researchers might ask themselves:

Did the interviewer influence the contents of the par cipants’ descrip ons in such a way that the descrip ons do not truly reflect the par cipants’ actual experience? Is the transcrip on accurate, and does it convey the meaning of the oral presenta on in the interview? In the analysis of the transcrip ons, were there conclusions other than those offered by the researcher that could have been derived? Has the researcher iden fied these alterna ves? Is it possible to go from the general structural descrip on to the transcrip ons and to account for the specific contents and connec ons in the original examples of the experience? Is the structural descrip on situa on specific, or does it hold in general for the experience in other situa ons? (Polkinghorne, 1989)

Van Manen (2014) pointed to ques ons as a way to “test [a phenomenology’s] level of validity” (p. 350).

Is the study based on a valid phenomenological ques on? In other words, does the study ask, “What is this human experience like?” “How is this or that phenomenon or event experienced?” A phenomenological ques on should not be confused with empirical studies of a par cular popula on, person(s), or group of people at a par cular

me and loca on. Also, phenomenology cannot deal with causal ques ons or theore cal explana ons. However, a par cular individual or group may be studied for the understanding of a phenomenological theme—such as a gender phenomenon, a sociopoli cal event, or the experience of a human disaster. Is the analysis performed on experien ally descrip ve accounts, transcripts? (Does the analysis avoid empirical material that mostly consists of percep ons, opinions, beliefs, views, and so on?) Is the study properly rooted in primary and scholarly phenomenological literature— rather than mostly relying on ques onable secondary and ter ary sources? Does the study avoid trying to legi mate itself with valida on criteria derived from sources that are concerned with other (non-phenomenological) methodologies (van Manen, 2014, pp. 350–351)?

Van Manen (2014) also provided criteria for evalua ve appraisal of phenomenological studies.

Heuris c ques oning: Does the text induce a sense of contempla ve wonder and ques oning a en veness— es n (the wonder [of] what this is) and ho es n (the wonder that something exists at all)? Descrip ve richness: Does the text contain rich and recognizable experien al material? Interpre ve depth: Does the text offer reflec ve insights that go beyond the taken-for- granted understandings of everyday life? Dis nc ve rigor: Does the text remain constantly guided by a self-cri cal ques on of dis nct meaning of the phenomenon or event? Strong and addressive meaning: Does the text “speak” to and address our sense of embodied meaning? Experien al awakening: Does the text awaken prereflec ve or primal experience through voca ve and presenta ve language? Inceptual epiphany: Does the study offer us the possibility of deeper and original insight, and perhaps, an intui ve or inspirited grasp of the ethics and ethos of life commitments and prac ces? (pp. 355–356)

In Figure 10.5, we advance our five standards for assessing the quality of a phenomenology.

Figure 10.5 Standards for Assessing the Quality of a Phenomenology

Grounded Theory Research Strauss and Corbin (1990) iden fied criteria by which one judges the quality of a grounded theory study. They described seven criteria related to the general research process and six criteria related to the empirical grounding of a study. Corbin and Strauss (2015) advanced the term checkpoint in place of criteria, saying they “dislike using the word criteria because that makes the evalua ve process seem so dogma c, an ‘all [or] nothing’ approach to evalua on” (p. 350, emphasis in original). Corbin and Strauss (2015) described 16 checkpoints for guiding researchers and reviewers in evalua ng the “methodoloical consistency” of a grounded theory study:

1. What was the target sample popula on? How was the original sample selected? 2. How did sampling proceed? What kinds of data were collected? Were there mul ple

sources of data and mul ple compara ve groups? 3. Did data collec on alternate with analysis? 4. Were ethical considera ons taken into account in both data collec on and analysis? 5. Were the concepts driving the data collec on arrived at through analysis (based on

theore cal sampling), or were concepts derived from the literature and established before the data were collected not true theore cal sampling)?

6. Was theore cal sampling used, and was there a descrip on of how it proceeded? 7. Did the research demonstrate sensi vity to the par cipants and to the data? 8. Is there evidence or examples of memos? 9. At what point did data collec on end or a discussion of satura on end? 10. Is there a descrip on of how coding proceeded along with examples of theore cal

sampling, concepts, categories, and statements of rela ons? What were some of the events, incidents, or ac ons (indicators) that pointed to some of these major categories?

11. Is there a core category, and is there a descrip on of how that core category was arrived at?

12. Were there changes in design as the research went along based on findings? 13. Did the researcher(s) encounter any problems while doing the research? Is there any

men on of a nega ve case, and how was that data handled? 14. Are methodological decisions made clear so that readers can judge their

appropriateness for gathering data (theore cal sampling) and doing analysis? 15. Was there feedback on the findings from other professionals and from par cipants? And

were changes made in the theory based on this feedback? 16. Did the researcher keep a research journal or notebook? (Corbin & Strauss, 2015, pp.

350–351) Corbin and Strauss (2015) also advanced 17 checkpoints for researchers and reviewers to evaluate the “quality and applicability” of a grounded theory study:

1. What is the core category, and how do the major categories relate to it? Is there a diagram depic ng these rela onship?

2. Is the core category sufficiently broad so that it can be used to study other popula ons and similar situa ons beyond this se ng?

3. Are each of the categories developed in terms of their proper es and dimensions so that they show depth, breadth, and varia on?

4. Is there descrip ve data given under each category that brings that theory to life so that it provides understanding and can be used in a variety of situa ons?

5. Has context been iden fied and integrated into the theory? Condi ons and consequences should be listed merely as background informa on in a separate sec on but woven into the actual analysis with explana ons of how they impact and flow from ac on–interac on in the data.

6. Has process been incorporated into the theory in the form of changes in ac on– interac on in rela onship to changes in condi ons? Is ac on–interac on matched to different situa ons, demonstra ng how the theory might vary under different condi ons and therefore be applied to different situa ons?

7. How is satura on explained, and when and how was it determined that categories were saturated?

8. Do the findings resonate or fit with the experience of both the professionals for whom the research ended and the par cipants who took part in the study? Can par cipants see themselves in the story even if not every detail applies to them?

9. Are there gaps, or missing links, in the theory, leaving the reader confused and with a sense that something is missing?

10. Is there an account of extremes or nega ve cases? 11. Is varia on built into the theory? 12. Are the findings presented in a crea ve and innova ve manner? Does the research say

something new or put old ideas together in new ways? 13. Do findings give insight into situa on and provide knowledge that can be applied to

develop policy, change prac ce, and add to the knowledge base of a profession? 14. Do the theore cal findings seem significant, and to what extent? It is en rely possible to

complete a theory-genera ng study, or any research inves ga on, yet not produce findings that are significant.

15. Do the findings have the poten al to become part of the discussion and ideas exchanged among relevant social and professional groups?

16. Are the limita ons of the study clearly spelled out? 17. Are there sugges ons for prac ce, policy, teaching, and applica on of the research?

(Corbin & Strauss, 2015, pp. 351–352) Charmaz (2014) reflected on the quality of the theory developed in a grounded theory study. She suggested that grounded theorists look at their theory and ask themselves the following evalua ve ques ons: Charmaz (2014) reflected on the quality of the theory developed in a grounded theory study. She suggested that grounded theorists look at their theory and ask themselves the following evalua ve ques ons:

Are the defini ons of major categories complete?

Have I raised major categories to concepts in my theory? How have I increased the scope and depth of the analysis in this dra ? Have I established strong theore cal links between categories and between categories and their proper es, in addi on to the data? How have I increased understanding of the studied phenomenon? How does my grounded theory study make a fresh contribu on? With which theore cal, substan ve, or prac cal problems is this analysis most closely aligned? Which audiences might be most interested in it? Where shall I go with it? What implica ons does this analysis hold for theore cal reach, depth, and breadth? For methods? For substan ve knowledge? For ac ons or interven ons? (Charmaz, 2014, pp. 337–338)

See also Charmaz (2014, pp. 337–338) for guiding ques ons for assessing criteria for grounded theory studies organized by four categories: credibility, originality, resonance, and usefulness. In Figure 10.6, we describe features of the general process and a rela onship among the concepts we look for when evalua ng a grounded theory study.

Figure 10.6 Features for Evalua ng a Grounded Theory Study Ethnographic Research Few ethnographic writers iden fy criteria for quality ethnographies. Instead, the preference, it seems, for ethnographers is to describe the “basics” of ethnographical studies as prolonged fieldwork that generate thick and contextual descrip ons reflec ve of triangula ng mul ple data sources (Fe erman, 2010, 2019; Wolco , 2008a, 2010). An excep on is Spindler and Spindler (1987). The ethnographers Spindler and Spindler (1987) emphasized that the most important requirement for an ethnographic approach was to explain behavior from the “na ve’s point of view” (p. 20) and to be systema c in recording this informa on using note taking, tape recorders, and cameras. This required that the ethnographer be present in the situa on and engage in constant interac on between observa on and interviews. These points were reinforced in Spindler and Spindler’s (1987) nine criteria for a “good ethnography”:

Criterion I. Observa ons are contextualized. Criterion II. Hypotheses emerge in situ as the study goes on. Criterion III. Observa on is prolonged and repe ve. Criterion IV. Through interviews, observa ons, and other elici ng procedures, the na ve view of reality is obtained. Criterion V. Ethnographers elicit knowledge from informant-par cipants in a systema c fashion. Criterion VI. Instruments, codes, schedules, ques onnaires, agenda for interviews, and so forth are generated in situ as a result of inquiry. Criterion VII. A transcultural, compara ve perspec ve is frequently an unstated assump on. Criterion VIII. The ethnographer makes explicit what is implicit and tacit to informants. Criterion IX. The ethnographic interviewer must not predetermine responses by the kinds of ques ons asked. (p. 18)

This list, grounded in fieldwork, leads to a strong ethnography. Moreover, as Lofland (1974) contended, the study was located in wide conceptual frameworks; presented the novel but not necessarily new; provided evidence for the framework(s); was endowed with concrete, even ul interac onal events, incidents, occurrences, episodes, anecdotes, scenes, and happenings without being “hyper-even ul”; and showed an interplay between the concrete and analy cal and the empirical and theore cal. In Figure 10.7, we advance seven evalua ve criteria for an ethnography.

Figure 10.7 Criteria for Evalua ng an Ethnography Case Study Research Yin (2017) reflected on the quality of the descrip on presented in a case study. He described several characteris cs for an exemplary case study.

Significant: Has the researcher focused the case(s) as “unusual and of general public interest” or underlying issues as “na onally important—either in theore cal terms or in policy or prac cal terms” (p. 242)? Complete: Has the researcher clearly defined the case(s) boundaries, collected extensive evidence, and conducted the study absent “of certain artefactual condi ons” (p. 244)— for example, if a me or resource constraint unwi ngly ended the study? Consider alterna ve perspec ves: Has the researcher considered rival proposi ons and sought to collect evidence from differing perspec ves in the case(s)?

Display sufficient evidence: Has the researcher reported the case(s) in such a way that a reader can “reach an independent judgment regarding the merits” (p. 246)? Composed in an engaging manner: Has the researcher presented the case(s) in a way that communicates “the results widely”—either in wri ng or performance (p. 247)?

G. Thomas (2021) asked case study researchers to consider the following three ques ons:

1. How well has the case been chosen? Thomas’s discussions of quality are aptly focused on appropriate sampling and procedural decisions.

2. How well has the context for the study been explained and jus fied? Thomas pointed to the need for detailed contextual descrip ons to be able to assess decisions jus fying how analysis has been undertaken.

3. How well have arguments been made? Thomas explained the role of rival explana ons to provide “space to think about alterna ve storylines for the same plot” (p. 81).

Stake (1995) described evalua ve criteria for a case study using a rather extensive “cri que checklist.” In so doing, Stake (1995) shared criteria for assessing a good case study report:

Is the report easy to read? Does it fit together, each sentence contribu ng to the whole? Does the report have a conceptual structure (i.e., themes or issues)? Are its issues developed in a serious and scholarly way? Is the case adequately defined? Is there a sense of story to the presenta on? Is the reader provided some vicarious experience? Have quota ons been used effec vely? Are headings, figures, ar facts, appendixes, and indexes used effec vely? Was it edited well—and then again with a last-minute polish? Has the writer made sound asser ons, neither over- nor misinterpre ng? Has adequate a en on been paid to various contexts? Were sufficient raw data presented? Were data sources well-chosen and in sufficient number? Do observa ons and interpreta ons appear to have been triangulated? Is the role and point of view of the researcher nicely apparent? Is the nature of the intended audience apparent? Is empathy shown for all sides? Are personal inten ons examined? Does it appear that individuals were put at risk? (p. 131)

In Figure 10.8, we describe our criteria for evalua ng a case study.

Figure 10.8 Evalua ve Criteria for a Case Study COMPARING EVALUATION STANDARDS ACROSS THE FIVE APPROACHES The standards discussed for each approach differ slightly depending on the procedures of the approaches. Certainly, less is men oned about narra ve research and its standards of quality, and more is available about the other approaches. From within the major books used for each approach, we have a empted to extract the evalua on standards recommended for their approach to research. To these, we have added our own standards that we use in our qualita ve classes when we evaluate a project or study presented within each of the five approaches. We can compare these standards across the five approaches rela ng five dimensions summarized in Table 10.2. At the most fundamental level, the five differ in the focus of the study. A couple of poten al similari es should be noted. Phenomenology, grounded theory, and ethnography typically focus the study on a singular phenomenon, process (or ac on or interac on), and culture-sharing group, respec vely. One may also focus narra ve research or a case study on a single individual or case but it also possible to focus on two or three individuals or mul ple cases. The study procedures offer dis nguishing features for each approach in which some research features the collec on of stories (i.e., narra ve), specifica on of cultural themes (i.e., ethnography), and selec on of cases (i.e., case study). Others such as phenomenology are dis nguishable by ini al conveyance of an understanding of the philosophical tenets or integra on of analysis into data collec on ac vi es and use of memoing (i.e., grounded theory). When examining a study, the approach can be iden fiable by its presenta on and outcomes of

a story chronology for narra ve research, theore cal diagramming for grounded theory research, explana on of how a culture-sharing group works for ethnography, and the asser ons for a case study. Common across all the qualita ve approaches is the use of reflexive and self- disclosing prac ces for embedding what the researcher brings to the study. Table 10.2 Comparing the Evalua on Standards Across the Five Qualita ve Approaches

SUMMARY In this chapter, we discussed valida on, reliability, and standards of evalua on in qualita ve research. Valida on approaches vary considerably, such as strategies that emphasize using qualita ve terms comparable to quan ta ve terms, the use of dis nct terms, perspec ves from postmodern and interpre ve lenses, syntheses of different perspec ves, descrip ons based on metaphorical images, or some combina on of these perspec ves on validity. Reliability is used in qualita ve research in several ways, one of the most popular being the use of intercoder agreements when mul ple coders analyze and then compare their code segments to establish the reliability of the data analysis process. A detailed procedure for establishing intercoder agreement is described in this chapter. Also, diverse standards exist for evalua ng the quality of qualita ve research, and these criteria are based on methods perspec ves, general research perspec ves, postmodern perspec ves, and interpre ve perspec ves. Within each of the five approaches to inquiry, specific standards also exist; these were reviewed in this chapter. Finally, we advanced standards that we use to assess the quality of studies presented in each approach and we compare the evalua ng standards across the five approaches. Descrip ons of Images and Figures In the center lies the “Strategies for Valida on” which is surrounded by other methods in a circular sequence such as “Corrobora ng evidence through triangula on, Discovering a nega ve case analysis or disconfirming evidence, and Clarifying researcher bias or engaging in reflexivity” comes under “Researcher’s Lens.” The “Member checking or seeking par cipant feedback, Having a prolonged engagement and persistent observa on in the field and Collabora ng with par cipants” comes under “Par cipant’s Lens.” The “Enabling external audits, Genera ng a rich, thick descrip on, and Having a peer review or debriefing of the data and research process” comes under “Reader’s or Reviewer’s Lens.” The flowchart includes from top to bo om:

Establish a common pla orm for coding and develop a preliminary code list. Define and share the ini al codebook among coders. Apply the codebook to addi onal transcripts and compare coding across mul ple researchers. Assess and report intercoder agreement among researchers. Revise and finalize the codebook to inform further coding. Does the qualita ve study do the following? Meet the assump ons and dis nguishing features of the qualita ve approach to research? • The author provides evidence of the defining characteris cs described in Table 1.1. Provide evidence of an ethical study? • The author provides evidence of having an cipated and addressed the ethical issues described in Tables 3.1, 7.2, 8.3, and 9.1. Use procedures of a recognizable approach to inquiry? • The author may refer to the procedures described in Figures 4.3 (narra ve), 4.5 (phenomenology), 4.7 (grounded theory), 4.9 (ethnography), and 4.11 (case study). Explore a single focus or concept? • The author describes the study focus using research purpose statements such as those in Examples 6.1–6.5. Employ rigorous data collec on procedures? • The author provides evidence of data collec on ac vi es described in Figure 7.1 and Table 7.1. Detail rigorous data analysis methods? • The author provides evidence of data spiral ac vi es described in Figure 8.1 and Table 8.5. Present evidence of mul ple levels of abstrac on? • The author provides evidence of the valida on strategies in Figure 10.1 and Table 10.2. Convey the reader experience of “being there” persuasively? • The author provides evidence of wri ng structures described in Table 9.2. Present the researcher’s situated role? • The author provides evidence of their guiding philosophical assump ons and interpre ve frameworks described in Tables 2.1, 2.2, and 2.3. Back to Figure Does the narra ve study do the following? Focus on an individual? • The author may focus on one or more individuals.

Collect stories about a significant issue? • The author may focus on the stories told by the individual or individuals. Develop a chronology? • The author may use a chronology to connect different phases or aspects of a story. Tell a story? • The author may, through the story, report what was said (themes), how it was said (unfolding story), or how speakers interact or perform the narra ve. Embed reflexivity? • The author may use reflexive thinking and wri ng to bring himself or herself into the study. Back to Figure Does the phenomenology do the following? Ar culate a clear “phenomenon” to study in a concise way? • The author may use a phenomenological ques on to guide the study. Convey an understanding of the philosophical tenets of phenomenology? • The author may ground the study in primary and scholarly phenomenological literature. Use procedures of data analysis in phenomenology? • The author may refer to procedures recommended by Moustakas (1994) or van Manen (1990, 2023). Communicate the overall essence of the experience of the par cipants? • The author may describe the context in which the experience occurred. Embed reflexivity throughout the study? • The author may explain the process and outcomes of reflexive thinking. Back to Figure Does the grounded theory study do the following? Focus on the study of a process, an ac on, or an interac on as the key element in the theory? • The author may focus on the steps that unfold when studying a central phenomenon as a process, ac on, or interac on among individuals. Integrate a coding process that works from the data to a larger theore cal model? • The author may describe the data collec on as alterna ng with data analysis to build a theore cal model.

Present the theore cal model in a figure or diagram? • The author may use innova ve means for presen ng the theory in a crea ve and innova ve manner. Advance a story line or proposi on connected with the categories in the theore cal model that presents further ques ons to be answered? • The author may refer to the overall picture emerging in the current study as a springboard for future direc ons of research. Use memoing through the process of research? • The author may describe the different types of memos or ways of recording emerging ideas throughout the process of conduc ng the study. Embed evidence of reflexivity for self-disclosure by the researcher about their study “posi oning”? • The author may describe how a research journal documented their reflexive thinking during the study. Does the ethnography do the following? Convey evidence of clear iden fica on of a culture-sharing group? • The author may describe the group in some detail—how the group was selected, how access to the group was facilitated including any involvement of gatekeepers, how the group interacts, how the group communicates, and so forth. Specify a cultural theme that will be examined in light of this culture-sharing group? • The author may iden fy a cultural theme and the ra onale for choosing it. Describe the cultural group in detail? • The author may use crea ve analy cal prac ces to convey these descrip ons. Communicate themes derived from an understanding of the cultural group? • The author may organize a thema c narra ve or tale. Iden fy issues that arose during fieldwork that reflect on the rela onship between the researcher and the par cipants, the interpre ve nature of repor ng, and sensi vity and reciprocity in the co-crea ng of the account? • The author may describe a working set of rules or generaliza ons as to how the culture sharing group func ons. Explain how the culture-sharing group works overall? • The author may describe a working set of rules or generaliza ons as to how the culture sharing group func ons. Integrate self-disclosure and reflexivity by the researcher about their posi on in the research?

• The author may describe their background experiences with the group and describe their reflec ons about interac ons with the group. Does the case study do the following? Iden fy the case studied? • The author may iden fy the boundaries and me parameters of a single case or mul ple cases. Present a ra onale for the case(s) selec on? • The author may iden fy ra onales for selec ng the case, such as the intent to understand a research issue or describe intrinsic merit. Describe the case(s) in detail? • The author may begin with a rich descrip on of the case(s) and its se ng or context(s). Ar culate the themes iden fied for the case(s)? • The author may focus on a few key thema c issues for an individual case(s) or across cases. Report asser ons or generaliza ons from the case analysis? • The author may interpret how the case(s) provides insight into the issue, or the findings can be generalized to other cases. Some mes this may take the form of a summary statement or a vigne e. Embed researcher self-disclosure about their posi on in the study? • The author may use reflexive prac ces and wri ng throughout the study