U10A1-68 - Qualitative Research Plan - ***TUTOR FOLLOW ALL INSTRUCTIONS AS OUTLINED TO COMPLETE THIS WORK. READ ATTACHMENTS.

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/44!/4/2/4/2@0:0 1/39

CHAPTER

9 Enhancing the Quality and Credibility ofQualitative Studies

The Medieval alchemical symbol for fire was a single triangle, also the modern symbol for triangulation in geometry, trigonometry, and surveying: the process of locating an unknown point by measuring angles to it from known points. Triangulation in qualitative inquiry involves gathering and analyzing multiple perspectives, using diverse sources of data, and during analysis, using alternative frameworks.

The double-triangle symbol, shown here, represented strong fire in alchemy. Strong fire was needed to ensure that the transformative process would work. Building and sustaining a strong fire required quality materials, good ventilation, and ongoing monitoring. Using a strong fire required skill, experience, and rigorous implementation of the transformative process to achieve the desired effects. Strong fire produces both intense heat and bright illumination. Alchemists who could properly build, sustain, and appropriately use strong fire were held in high esteem, had great credibility, and produced much-valued products.

Interpreting Truth A young man traveling through a new country heard that a great Mulla, a Sufi guru with unequaled insight into the mysteries of the world, was also traveling in that region. The young man was determined to become his disciple. He found his way to the wise man and said, “I wish to place my education in your hands that I might learn to interpret what I see as I travel through the world.”

After six months of traveling from village to village with the great teacher, the young man was confused and disheartened. He decided to reveal his frustration to the Mulla.

“For six months I have observed the services you provide to the people along our route. In one village you tell the hungry that they must work harder in their fields. In another village you tell the hungry to give up their preoccupation with food. In yet another village you tell the people to pray for a richer harvest. In each village the problem is the same, but always your message is different. I can find no pattern of Truth in your teachings.”

The Mulla looked piercingly at the young man.

“Truth? When you came here you did not tell me you wanted to learn Truth. Truth is like the Buddha. When met on the road it should be killed. If there were only one Truth to be applied to all villages, there would be no need of Mullahs to travel from village to village.”

“When you first came to me you said you wanted to ‘learn how to interpret’ what you see as you travel through the world. Your confusion is simple. To interpret and to state Truths are two quite different things.”

Having finished his story Halcolm smiled at the attentive youths. “Go, my children. Seek what you will, do what you must.”

—From Halcolm’s Evaluation Parables

Chapter Preview

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/44!/4/2/4/2@0:0 2/39

This chapter concludes the book by addressing ways to enhance the quality and credibility of qualitative analysis. Module 76 discusses and demonstrates analytical processes for enhancing credibility by systematically engaging and questioning the data. Module 77 presents four triangulation processes for enhancing credibility. Modules 78 and 79 present alternative and competing criteria for judging the quality of qualitative studies. Module 80 discusses how and why the credibility of the inquirer is critical to the overall credibility of qualitative findings. Module 81 examines core issues of generalizability, extrapolations, transferability, generating principles, and harvesting lessons. Module 82 concludes the chapter and the book by addressing philosophy of science issues related to the credibility and utility of qualitative inquiry.

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/44!/4/2/4/2@0:0 3/39

MODULE

76 Analytical Processes for Enhancing Credibility: Systematically Engaging and Questioning the Data

The credibility of qualitative inquiry depends on four distinct but related inquiry elements:

1. Systematic, in-depth fieldwork that yields high-quality data 2. Systematic and conscientious analysis of data with attention to issues of credibility 3. Credibility of the inquirer, which depends on training, experience, track record, status, and presentation of

self 4. Readers’ and users’ philosophical belief in the value of qualitative inquiry—that is, a fundamental

appreciation of naturalistic inquiry, qualitative methods, inductive analysis, purposeful sampling, and holistic thinking (indeed, all 12 core qualitative strategies presented in Exhibit 2.1, pp. 46–47)

The first of the elements that determine credibility, systematic, in-depth fieldwork that yields high-quality data, was covered in Chapter 5 (purposeful qualitative designs), Chapter 6 (in-depth fieldwork and rich observational data), and Chapter 7 (high-quality, skillful interviewing).

This module and the next focus on the remaining three elements of quality: systematic and conscientious analysis of data. Module 80 discusses credibility of the inquirer, and Module 82 examines readers’ and users’ philosophical belief in the value of qualitative inquiry.

Strategies for Enhancing the Credibility of Analysis

Chance favors the prepared mind. —Louis Pasteur (1822–1895)

French microbiologist (known as the “father of microbiology”) who discovered the process for pasteurizing milk, named after him

Chapter 8 presented analytical strategies for coding qualitative data, identifying patterns and themes, creating typologies, determining substantive significance, and reporting findings. However, at the heart of much controversy about qualitative findings are doubts about the nature of qualitative analysis because it is so judgment dependent. Statistical analysis follows formulas and rules, while, at the core, qualitative analysis depends on the insights, conceptual capabilities, and integrity of the analyst. Qualitative analysis is driven by the capacity for astute pattern recognition from beginning to end. Staying open to the data, for example, involves aggregating and integrating the data around a particular expected pattern while also watching for unexpected patterns. This process is epitomized in health research by the scientist working on one problem who suddenly notices a pattern related to a quite different problem—and thus discovers Viagra; as Pasteur explained when he was asked how he happened to discover how to stop bacterial contamination of milk, “Chance favors the prepared mind.” Here, then, are some techniques that prepare the mind for insight while also enhancing the credibility of the resulting analysis.

Integrity in Analysis: Generating and Assessing Alternative Conclusions and Rival Explanations

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/44!/4/2/4/2@0:0 4/39

One barrier to credible qualitative findings stems from the suspicion that the analyst has shaped findings according to his or her predispositions and biases. Being able to report that you engaged in a systematic and conscientious search for alternative themes, divergent patterns, and rival explanations enhances credibility, not to mention that it is simply good analytical practice and the very essence of being rigorous in analysis. This can be done both inductively and logically. Inductively, it involves looking for other ways of organizing the data that might lead to different findings. Logically, it means thinking about other logical possibilities and then seeing if those possibilities can be supported by the data. When considering rival organizing schemes and competing explanations, your mind-set should not be one of attempting to disprove the alternatives; rather, you look for data that support alternative explanations.

In evaluation of a training program for chronically unemployed men of color, we conducted case studies of a group of successes. The program model was based on training in both hard skills (e.g., machine tooling, keyboarding, welding, and accounting) and soft skills (showing up to work on time, dressing appropriately, and respecting supervisors and coworkers). The cases studied validated the importance of both kinds of skills, but an additional explanation emerged in later cases, namely, that the program experience and peer support led to an identity shift: Successful trainees began to think of themselves as capable of holding a job. They were used to being labeled as “losers.” The opportunity to think of themselves as “winners” involved more than acquiring “soft skills.” It involved a shift in identity. We went back to earlier cases to find out if that phenomenon was evident there as well. It was, as was evidence for how that shift in identity occurred. Might this change be simply a function of participants being older by the time they entered this particular program (a maturation effect)? No, the change was evident in younger participants as well as older ones. We continued in this fashion, looking for alternative explanations and checking them out against the case data.

Failure to find strong supporting evidence for alternative ways of presenting data or contrary explanations helps increase confidence in the initial, principal explanation you generated. Comparing alternative patterns will not typically lead to clear-cut “yes there is support” versus “no there is no support” kinds of conclusions. You’re searching for the best fit, the preponderance of evidence. This requires assessing the weight of evidence and looking for those patterns and conclusions that fit the preponderance of data. Keep track of and report alternative classification systems, themes, and explanations that you considered and “tested” during data analysis. This demonstrates intellectual integrity and lends considerable credibility to the final set of findings and explanations offered. Analysis of rival explanations in case studies is analogous to counterfactual analysis in experimental designs.

Searching for and Analyzing Negative or Disconfirming Evidence and Cases Closely related to testing alternative constructs is the search for and analysis of negative cases. Where patterns and trends have been identified, our understanding of those patterns and trends is increased by considering the instances and cases that do not fit within the pattern. These may be exceptions that illuminate the boundaries of the pattern. They may also broaden understanding of the pattern, change the conceptualization of the pattern, or cast doubt on the pattern altogether.

In qualitative analysis you need to keep analyzing the data to check any explanations and generalizations that you wish to make, to ensure that you have not missed anything that might lead you to question their applicability. Essentially this means looking for negative or deviant cases—situations and examples that just do not fit the general points you are trying to make. However, the discovery of negative cases or counter-evidence to a hunch in qualitative analysis does not mean its immediate rejection. You should investigate the negative cases and try to understand why they occurred and what circumstances produced them. As a result, you might extend the idea behind the code to include the circumstances of the negative case and thus extend the richness of your coding. (Gibbs, 2007, p. 96)

In the Southwest Field Training Project involving wilderness education, virtually all participants reported significant “personal growth” as a result of their participation in the wilderness experiences; however, the two people who reported “no change” provided particularly useful insights into how the program operated and

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/44!/4/2/4/2@0:0 5/39

affected participants. These two had crises going on back home that limited their capacity to “get into” the wilderness experiences. The project staff treated the wilderness experiences as fairly self-contained, closed- system experiences. The two negative cases opened up thinking about “baggage carried in from the outside world,” “learning-oriented mind-sets,” and a “readiness” factor that subsequently affected participant selection and preparation.

Negative cases also provide instructive opportunities for new learning in formative evaluations. For example, in a health education program for teenage mothers where the large majority of participants complete the program and show knowledge gains, an important component of the analysis should include examination of reactions from dropouts, even if the sample is small for the dropout group. While the small proportion of dropouts may not be large enough to make a difference in a statistical analysis, qualitatively the dropout feedback may provide critical information about a niche group or a specific subculture, and/or clues to program improvement.

No specific guidelines can tell you how and how long to search for negative cases or how to find alternative constructs and hypotheses in qualitative data. Your obligation is to make an “assiduous search . . . until no further negative cases are found” (Lincoln & Guba, 1986, p. 77). You then report the basis for the conclusions you reach about the significance of the negative or deviant cases.

SIDEBAR

In 1587, the Roman Catholic Church created advocacy–adversary roles to test the validity of evidence in support of the canonization process for elevating someone to sainthood. The Devil’s Advocate (Latin: advocatus diaboli) in this process (officially designated the Promoter of the Faith) was a canon lawyer whose job was to argue against the canonization by presenting doubts about or holes in the evidence, for example, to argue that any miracles attributed to the candidate were unsubstantiated or even fraudulent. The Devil’s Advocate opposed God’s Advocate, whose job was to present evidence supporting and make the argument in favor of canonization. This advocacy–adversary process endured until 1983, when it was abolished by Pope John Paul II as overly adversarial and contentious.

ADVOCACY–ADVERSARY ANALYSIS

Advocacy–Adversary Analysis in Evaluation

A formal and forced approach to engaging rival conclusions draws on the legal system’s reliance on opposing perspectives battling it out in the courtroom. The advocacy‑adversary model suggested by Wolf (1975) developed in response to concerns that evaluators could be biased in their conclusions. Also called the Judicial Model of Evaluation (Datta, 2005), to balance possible evaluator biases, two teams engage in debate. The advocacy team gathers and presents information that supports the proposition that the program is effective; the adversary team gathers information that supports the conclusion that the program ought to be changed or terminated.

Some years ago, I served as the judge for what would constitute admissible evidence in an advocacy– adversary evaluation of an innovative education program in Hawaii. The task of the advocacy team was to gather and present data supporting the proposition that the program was effective and ought to be continued. The adversaries were charged with marshalling all possible evidence demonstrating that the program ought to be terminated. When I arrived on the scene, I immediately felt the exhilaration of the competition. I wrote in my journal,

No longer staid academic scholars, these are athletes in a contest that will reveal who is best; these are lawyers prepared to use whatever means necessary to win their case. The teams have become openly secretive about their respective strategies. These are experienced evaluators engaged in a battle not only of data but also of wits.

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/44!/4/2/4/2@0:0 6/39

As the two teams prepared their final reports, a concern emerged among some about the narrow focus of the evaluation. The summative question concerned whether the program should be continued or terminated. Education officials were asking how to improve the program without terminating it. Was it possible that a great amount of time, effort, and money was directed at answering the wrong question? Was it appropriate to force the data into a simple save-it-or-scrap-it choice? In fact, middle-ground positions were more sensible. But the advocacy–adversary analytical process design obliged opposing teams to do battle on the unembellished question of whether to maintain or terminate a program. A systematic assessment of strengths and weaknesses, with ideas for improvement, gave way to an all-good, all-bad framing, and that’s how the results were presented (Patton, 2008, pp. 142–143).

The weakness of the advocacy–adversary approach is that it emphasizes contrasts and opposite conclusions, to the detriment of appreciating and communicating nuances in the data and accepting and acknowledging genuine and meaningful ambiguities. Advocacy–adversary analysis forces data sets into combat with each other. Such oversimplification of complex and multifaceted findings is a primary reason why advocacy–adversary evaluation is rarely used (in addition to being expensive and time-consuming). Still, it highlights the importance of engaging in some systematic analysis of alternative and rival conclusions, and as one approach (but not the only one) to testing conclusions, it can be useful and revealing.

Practical Analytical Variations on a Theme

1. A variation of the overall advocacy–adversary approach would be to arbitrarily create advocacy and adversary teams only during the analysis stage so that both teams work with the same set of data but each team organizes and interprets those data to support different and opposite conclusions, including identifying ambiguous findings.

2. Another variation would be for a lone analyst to organize data systematically into pro and con sets of evidence to see what each yielded.

Readers of a qualitative study will make their own decisions about the plausibility of alternate explanations and the reasons why deviant cases do not fit within dominant patterns. But I would note that the section of the report that involves exploration of alternative explanations and consideration of why certain cases do not fall into the main pattern can be among the most interesting sections of a report to read. When well written, this section of a report reads something like a detective study in which the analyst (detective) looks for clues that lead in different directions and tries to sort out which direction makes the most sense given the clues (data) that are available. Such writing adds credibility by showing the analyst’s authentic search for what makes most sense rather than marshalling all the data toward a single conclusion. Indeed, the whole tone of a report feels different when the qualitative analyst is willing to openly consider other possibilities than those finally settled on as most reasonable in accordance with the preponderance of evidence. Compare the approach of weighing alternatives with the report where all the data lead in a single-minded fashion, in a rising crescendo, toward an overwhelming presentation of a single point of view. Perfect patterns and omniscient explanations are likely to be greeted skeptically—and for good reason: The human world is not perfectly ordered, and human researchers are not omniscient. Humility can do more than certainty to enhance credibility. Dealing openly with the complexities and dilemmas posed by negative cases is both intellectually honest and politically strategic.

SIDEBAR

ANALYTIC INDUCTION: HYPOTHESIS TESTING WITH NEGATIVE CASES

Analytic induction emphasizes giving special attention to negative or deviant cases for testing propositions that should, based on the theory being examined, apply to all cases that have been sampled in the design

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/44!/4/2/4/2@0:0 7/39

to manifest the phenomenon of interest. Analytic induction works through one case at a time. If the case data fit the hypothesis, the inductive analyst takes up the next case. If a case isn’t consistent with the hypothesis—that is, it is a negative or deviant case—then the hypothesis is revised or the case is rejected as not actually relevant to the phenomenon being studied. The analytical focus is examining the extent to which every case confirms the hypothesis and to either refine the hypothesis or the statement of the problem to account for all cases. No cases can be ignored. All must be accounted for and used in the analysis.

Here’s an example of testing a hypothesis about the effect of mother–daughter relationships on anorexia. The proposition being tested was “If mother was critical of daughter’s body image and mother–daughter relationship was strained and daughter experiences weight loss, then count that as an example of mother’ s negative influence on daughter’s self-image.” Once particular interviews were identified as containing the codes identified in the hypothesis, the qualitative data from interviews and cases could be examined to determine whether support for this causal interpretation could be justified for each case (Hesse-Biber & Dupuis, cited in Silverman & Marvasti, 2008, p. 252). The rigor of this approach is that finding even a single disconfirming case disconfirms the hypothesis requiring either refinement or reformulation, for the goal is to identify and confirm a generalizable, universal, causal explanation for the phenomenon of interest (Flick, 2007a, p. 30; Schwandt, 2007, p. 6).

Avoid the Numbers Game Philosopher of science Thomas H. Kuhn (1970), having studied extensively the value systems of scientists, observed that “the most deeply held values concern predictions” and “quantitative predictions are preferable to qualitative ones” (pp. 184–185). The methodological status hierarchy in science ranks “hard data” above “soft data,” where “hardness” refers to the precision of statistics. Qualitative data can carry the stigma of “being soft.” This carries over into the public arena, especially in the media and among policymakers, creating what has been called the tyranny of numbers (Eberstadt, 1995).

How can one deal with a lingering bias against qualitative methods? A starting point is helping people understand that qualitative methods are not weaker or softer than quantitative approaches. Qualitative methods are different. Making the case for the value of qualitative inquiries involves being able to communicate the particular strengths of qualitative methods (Chapters 1 and 2) and the kinds of evaluation and other applications for which qualitative data are especially appropriate (Chapter 4). But those understandings can only open the door to dialogue. The fact is that numbers have a special allure in modern society. Statistics are seductive—so precise, so clear. Numbers convey that sense of precision and accuracy, even if the measurements that yielded the numbers are relatively unreliable, invalid, and meaningless (e.g., see Hausman, 2000; Silver, 2012).

Quantitizing

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/44!/4/2/4/2@0:0 8/39

©2002 Michael Quinn Patton and Michael Cochran

Quantitizing, commonly understood to refer to the numerical translation, transformation, or conversion of qualitative data, has become a staple of mixed-methods research (Sandelowski, Voils, & Knafl, 2009, p. 208). Quantitized qualitative data are analyzed statistically, including using statistical significance tests (Collingridge, 2013).

There are different techniques by which quantitization may be achieved. Two common strategies are (1) dichotomizing and (2) counting. Dichotomizing refers to assigning a binary value (e.g., 0 and 1) to variables with two mutually exclusive and exhaustive categories, such as assigning “0” to participants who did not express a particular theme and “1” to participants who did express the theme. In contrast, counting involves calculating the number of themes expressed by each participant, as in the case of determining that a participant expressed two out of four themes in a study. Counting also includes calculating the number of qualitative codes assigned to specific themes, as in the case of determining that a participant expressed 10 qualitative codes associated with a theme (Collingridge, 2013, p. 82).

In Chapter 8, I devoted my MQP Rumination to why I consider this kind of quantizing to be generally a bad idea and advocated keeping qualitative analysis qualitative (see pp. 557–559). I won’t repeat that argument here. Still, it strikes me as a worrisome trend. What’s driving it? Partly, it’s simply the cultural and political allure of numbers. But there’s more.

Pragmatic and ecumenical impulses, and the advent of computerized software programs to manage both qualitative and quantitative data, have served to promote a largely technical view of quantitizing. Moreover, the rhetorical appeal of numbers—their cultural association with scientific precision and rigor—has served to reinforce the necessity of converting qualitative into quantitative data.

A systematic literature review of quantitizing studies—that is, studies featuring quantitative analysis of qualitative interviews—shows the widespread nature of the phenomenon and some of the problems that arise, especially applying statistics to small sample sizes. Quantitative analyses of qualitative data are done to disaggregate results by background characteristics of participants (cross-tabs and correlations), to statistically test hypotheses, and to determine the prevalence of themes. But the overall problem is precisely what one would expect: “The conversion of the qualitative information to frequency counts has reduced the rich interpretation of people’s experience that was expressed through their interviews” (Fakis, Hilliam, Stoneley, & Townend, 2014, p. 156). That is the crux of the issue, as is replacing a determination of substantive significance with the safe fallback position of replying on statistical significance. Moreover, those engaged in quantiziting seem oblivious to the issues involved.

Typically glossed, however, are the foundational assumptions, judgments, and compromises involved in converting qualitative into quantitative data and whether such conversions advance inquiry. . . . Such conversions “are by no means transparent, uncontentious, or apolitical” (Love, Pritchard, Maguire, McCarthy, & Paddock, 2005, p. 287; Sandelowski, Voils, & Knafl, 2009, p. 28).

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/44!/4/2/4/2@0:0 9/39

Substantive Significance Trumps Statistical Significance The point, however, is not to be anti-numbers. The point is to be pro-meaningfulness.

I’m not numbers phobic. I have used numbers regularly in titling exhibits throughout this book: Exhibit 8.1 Twelve Tips for Ensuring a Strong Foundation for Qualitative Analysis (pp. 522–523). Exhibit 8.10 Ten Types of Qualitative Analysis (see pp. 551–552). Exhibit 9.1 Ten Systematic Analysis Strategies to Enhance Credibility and Utility (pp. 659–660). Module 77 presents four triangulation processes for enhancing credibility.

When there is something meaningful to be counted, then count. As sample sizes increase, especially in mixed-methods studies, quantizing is likely to become even more pervasive. One study in the systematic review of quantizing articles had a sample size of 400 (Fakis et al., 2014, p. 146). Such studies will quantitize and do so appropriately. Weaver-Hightower (2014) studied political influence by reviewing public policy documents; from 1,459 transcript pages, he coded 2,294 unique arguments and relied heavily on quantitative analysis. That’s understandable and appropriate, though reporting the results to two decimal places, “the average agreement score was 5.22%” (p. 125), illustrates the allure of pretentious precision. Or maybe just habit.

So while I advocate keeping qualitative analysis qualitative and focusing on substantive significance when interpreting findings, this is no hard-and-fast rule (my Chapter 8 MQP Rumination notwithstanding). Do what is appropriate. It doesn’t make sense to report percentages in a sample of 10 interviewees; it does make sense with a sample of 400. By knowing the strengths and weaknesses of both quantitative and qualitative data, you can help those with whom you dialogue focus on really important questions rather than, as sometimes happens, focusing primarily on how to generate numbers. The really important questions are about what the findings mean. A single illuminative case or interview may be more substantively meaningful and insightful than 20 routine cases. That 5% level of insight is not a reason to pay more attention to the 95% degree of mediocrity just because there’s more of it. Information-rich cases stand out not because there are lots of them but precisely because they are so rare—and rich with revelation (the very definition of being information- rich). Rare, precious gems are valued over widely available (and less expensive), semiprecious stones for the same reason. Qualitative analysis must include the analytical insight to distinguish signal from noise and valuable insights from commonplace ones.

SIDEBAR

CONSTANT COMPARISON

A lot of qualitative analysis involves comparisons: comparing cases, comparing quotations, comparing observations, and comparing findings in others studies with your own findings.

The point about these comparisons is that they are constant; they continue throughout the period of analysis and are used not just to develop theory and explanations but also to increase the richness of description in your analysis and thus ensure that it closely captures what people have told you and what happened.

There are two aspects to this constant process:

1. Use the comparisons to check the consistency and accuracy of application of your codes, especially as you first develop them. Try to ensure that the passages coded the same way are actually similar. But at the same time, keep your eyes open for ways in which they are different. Filling out the detail of what is coded in this way may lead you to further codes and to ideas about what is associated with any variation. This can be seen as a circular or iterative process. Thus, develop your code, check for other

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/44!/4/2/4/2@0:0 10/39

occurrences in your data, compare these with the original, and then revise your coding (and associated memos) if necessary.

2. Look explicitly for differences and variations in the activities, experiences, actions and so on that have been coded. In particular, look for variation across cases, settings and events (Gibbs, 2007, p. 96).

Constant comparison is an ongoing analysis of similarities and differences: What things go together in the data? What things are different? What explains these similarities and differences? What are the implications for your overall inquiry purpose and conclusions?

Design Checks: Keeping Methods and Data in Context

One issue that can arise during analysis is concerns about how design decisions affect results. For example, purposeful sampling strategies provide a limited number of cases for examination. When interpreting findings, it becomes important to reconsider how design constraints may have affected the data available for analysis. This means considering the rival methodological hypothesis that the findings are due to methodological idiosyncrasies.

By their nature, qualitative findings are highly context and case dependent. Three kinds of sampling limitations typically arise in qualitative research designs:

1. There are limitations in the situations (critical events or cases) that are sampled for observation (because it is rarely possible to observe all situations even within a single setting).

2. There are limitations from the time periods during which observations took place—that is, constraints of temporal sampling.

3. The findings will be limited based on selectivity in the people who were sampled for observations or interviews, or selectivity in document sampling.

In reporting how purposeful sampling decisions affect findings, the analyst returns to the reasons for having made the initial design decisions. Purposeful sampling involves studying information-rich cases in depth and detail to understand and illuminate important cases rather than generalizing from a sample to a population (see Chapter 5). For instance, sampling and studying highly successful and unsuccessful cases in an intervention yields quite different results from studying a “typical” case or a mix of cases. People unfamiliar with purposeful samples may think of small, purposeful samples as “biased,” a perception that undermines credibility in their minds. In communicating findings, then, it becomes important to emphasize that the issue is not one of dealing with a distorted or biased sample but rather one of clearly delineating the purpose, strengths, and limitations of the sample studied—and therefore being careful about not inappropriately extrapolating the findings to other situations, other time periods, and other people—a caution we’ll return to later in this chapter. Reporting both methods and results in their proper contexts will avoid many controversies that result from yielding to the temptation to overgeneralize from purposeful samples. Keeping findings in context is a cardinal principle of qualitative analysis. Design decisions are context for analysis.

The wise fool in Sufi tales, Mulla Nasrudin, was once called on to make this point to his monarch. Although he was supposed to be a wise man, Nasrudin was accused of being illiterate. Nagged to action by skeptics, the monarch decided to test him.

“Write something for me, Nasrudin,” said the king.

“I would willingly do so, but I have taken an oath never to write so much as a single letter again,” replied Nasrudin.

“Well, write something in the way in which you used to write before you decided not to write, so that I can see what it was like.”

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/44!/4/2/4/2@0:0 11/39

“I cannot do that, because every time you write something, your writing changes slightly through practice. If I wrote now, it would be something written for now.”

“Then,” addressing the crowd, the king commanded: “Bring me an example of Nasrudin’s writing, anyone who has something he’s written.”

Someone brought a terrible scrawl that Nasrudin had once written to him.

“Is this your writing?” asked the monarch.

“No,” said Nasrudin. “Not only does writing change with time, but reasons for writing change. You are now showing a piece of writing done by me to demonstrate to someone how he should not write.” (Shah, 1973, p. 92)

Summary of Strategies for Systematically Analyzing Qualitative Data to Enhance Credibility

Qualitative analysis aims to make sense of qualitative data: detecting patterns, identifying themes, answering the primary questions framing the study, and presenting substantively significant findings. In this chapter, we’ve been looking at ways of enhancing the credibility of findings by deepening the analysis, reexamining initial findings, and continuously working back and forth between the findings and the data to validate findings against data. Exhibit 9.1 summarizes the analytical techniques we’ve just covered and looks ahead to the four kinds of triangulation I’ll present and discuss in the next module (Items 7–10 in Exhibit 9.1).

EXHIBIT 9.1 Ten Systematic Analysis Strategies to Enhance Credibility and Utility

1. Generate and assess alternative conclusions and rival explanations. Don’t settle quickly on initial conclusions. Go back to the data. What are other ways of explaining what you’ve found? Look for the explanation that best fits the preponderance of evidence.

2. Advocacy–adversary analysis uses a debate format for testing the viability of conclusions. What are the evidence and arguments that support your conclusions? What are the contrary evidence and counterarguments? Get another analyst to play the “Devil’s Advocate” role, or switch back and forth in advocacy and adversary roles yourself. The aim is to surface doubts and weaknesses as well as build on strengths and confirm solid conclusions.

3. Search for and analyze negative or disconfirming evidence and cases. There are “exceptions that prove the rule” and exceptions that question the rule. In either case, look for and learn from exceptions to the patterns you’ve identified.

4. Make constant comparison your constant companion. All analysis is ultimately comparative. You compare the data that fit into a category, pattern, or theme with the data that don’t fit. You compare alternative explanations, conclusions, and chains of evidence. Compare and contrast. Then compare and contrast some more.

5. Keep analysis connected to purpose and design. When deeply enmeshed in cataloguing, classifying, and comparing the trees in your qualitative data—that is, the depth and details of rich, thick qualitative data—change perspectives now and again to see the forest—that is, reconnect with the big picture. Purpose drives design. Purpose and design drive data collection. Purpose, design, and the data collected, in combination, drive analysis. Make sure that your analysis is serving the purpose of the inquiry. A well-chosen, thoughtful design will have anticipated how analysis would unfold. Keep those linkages in mind so that analysis doesn’t become isolated from the inquiry’s overall purpose and context.

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/44!/4/2/4/2@0:0 12/39

Keeping findings in context is a cardinal principle of qualitative analysis.

6. Keep qualitative analysis qualitative. Paraphrasing poet Dylan Thomas, do not go gently into that numerical night. Quantitize thoughtfully, carefully, and even reluctantly. Do so when it’s appropriate and enhances understanding, all the while aware of the allure of numbers and the danger of losing the richness of qualitative data in the parsimony of numerical reduction.

7. Integrate and triangulate diverse sources of qualitative data: interviews, observations, document analysis. Any single source of data has strengths and weaknesses. Consistency of findings across types of data increases confidence in the confirmed patterns and themes. Inconsistency across types of data invites questions and reflection about why certain methods produced certain findings.

8. Integrate and triangulate quantitative and qualitative data in mixed-methods studies. The logic of triangulation (see Item 7) applies in mixed-methods designs when the strengths and weaknesses of qualitative and quantitative data are used together to illuminate the inquiry.

9. Triangulate analysts. Having more than one pair of eyes look at and think about the data, identify patterns and themes, and test conclusions and explanations reduces concerns about the potential biases and selective perception of a single analyst.

10. Undertake theory triangulation. Look at the findings and conclusions through the lens of alternative theoretical frameworks. How would a symbolic interactionist interpret the data compared with a phenomenologist or realist? How would a behavioral psychologist interpret the findings compared with a humanistic psychologist? What does a mechanistic display reveal compared with a systems graphic? The point is not to conduct an endless set of such theoretical comparisons but to select only those theoretical frameworks most germane to your inquiry to see what the alternative perspectives yield by way of insight and explanation.

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/44!/4/2/4/2@0:0 13/39

MODULE

77 Four Triangulation Processes for Enhancing Credibility

By combining multiple observers, theories, methods and data sources, [researchers] can hope to overcome the intrinsic bias that comes from single-methods, single-observer, and single-theory studies.

—Norman K. Denzin (1989c, p. 307)

Chapter 5 on design discussed the benefits of using multiple data-collection techniques, a form of triangulation, to study the same setting, issue, or program. You may recall from that discussion that the term triangulation is taken from land surveying. Knowing a single landmark only locates you somewhere along a line in a direction from the landmark, whereas with two landmarks you can take bearings in two directions and locate yourself at their intersection. The notion of triangulating also works metaphorically to call to mind the world’s strongest geometric shape—the triangle, which in its double alchemical form serves as the symbol for this chapter. The logic of triangulation is based on the premise that no single method ever adequately solves the problem of rival explanations. Because each method reveals different aspects of empirical reality and social perception, multiple methods of data collection and analysis provide more grist for the analytical mill. Combinations of interviewing, observation, and document analysis are expected in most fieldwork. Mixed qualitative–quantitative studies are increasingly valued as more credible than single-method studies. Studies that use only one method are more vulnerable to errors linked to that particular method (e.g., loaded interview questions, biased or untrue responses) than studies that use multiple methods, in which different types of data provide cross-data consistency checks.

Four Kinds of Analytical Triangulation It is in data analysis that the strategy of triangulation really pays off, not only in providing diverse ways of looking at the same phenomenon, but in adding to credibility by strengthening confidence in whatever conclusions are drawn. Four kinds of triangulation can contribute to the verification and validation of qualitative analysis:

1. Triangulation of qualitative sources: Checking out the consistency of different data sources within the same method (consistency across interviewees)

2. Mixed qualitative–quantitative methods triangulation: Checking out the consistency of findings generated by different data collection methods

3. Analyst triangulation: Using multiple analysts to review findings 4. Theory/perspective triangulation: Using multiple perspectives or theories to interpret data

By triangulating with multiple data sources, methods analysts, and/or theories, qualitative analysts can make substantial strides in overcoming the skepticism that greets singular methods, lone analysts, and single- perspective interpretations.

Interpreting Triangulation Results: Making Sense of Conflicting and Inconsistent Patterns

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/44!/4/2/4/2@0:0 14/39

A common misconception about triangulation involves thinking that the purpose is to demonstrate that different data sources or inquiry approaches yield essentially the same result. The point is to test for such consistency. Different kinds of data may yield somewhat different results because different types of inquiry are sensitive to different real-world nuances. Thus, understanding inconsistencies in findings across different kinds of data can be illuminative and important. Finding such inconsistencies ought not to be viewed as weakening the credibility of results but, rather, as offering opportunities for deeper insight into the relationship between inquiry approach and the phenomenon under study. I’ll comment briefly on each of the four types of triangulation.

1. Triangulation of Qualitative Data Sources

Four kinds of persons: zeal without knowledge; knowledge without zeal; neither knowledge nor zeal; both zeal and knowledge.

—Pascal, Pensées

Four kinds of qualitative triangulation: interviews with observations; interviews with documents; observations with documents; and interviews from multiple sources with observations of diverse events and documents of many kinds.

—Halcolm, Qualitative Pensées

Triangulation of data sources within and across different qualitative methods means comparing and cross- checking the consistency of information derived at different times and by different means from interviews, observations, and documents. It can include

• comparing observations with interviews; • comparing what people say in public with what they say in private; • checking for the consistency of what people say about the same thing over time; • comparing the perspectives of people from different points of view—for example, in an evaluation,

triangulating staff views, participants’ views, funder views, and views expressed by people outside the program; and

• checking interviews against program documents and other written evidence that can corroborate what interview respondents report.

Quite different kinds of data can be brought together in a case study to illuminate various aspects of a phenomenon. In a classic evaluation of an innovative educational project, historical program documents, in- depth interviews, and ethnographic participant observations were triangulated to illuminate the roles of powerful actors in supporting adoption of the innovation (Smith & Kleine, 1986). The evaluation of the Paris Declaration on development aid triangulated interviews with a variety of key informants, government reports, donor agency reports, and observations of donor–recipient decision-making meetings (Wood et al., 2011).

Maxwell (2012) is especially insightful about the interrelationship of interview and observation data in qualitative inquiry and analysis.

One belief that inhibits triangulation is the widespread (though often implicit) assumption that observation is mainly useful for describing behavior and events, while interviewing is mainly useful for obtaining the perspectives of actors. It is true that the immediate result of observation is description, but this is equally true of interviewing: The latter gives you a description of what the informant said, not a direct understanding of their perspective. Generating an interpretation of someone’s perspective is inherently a matter of inference from descriptions of their behavior

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/44!/4/2/4/2@0:0 15/39

(including verbal behavior), whether the data are derived from observations, interviews, or some other source such as written documents.

While interviewing is often an efficient and valid way of understanding someone’s perspective, observation can enable you to draw inferences about this perspective that you couldn’t obtain by relying exclusively on interview data. . . . For example, watching how a teacher responds to boys’ and girls’ questions in a science class may provide a much better understanding of the teacher’s actual views about gender and science than what the teacher says in an interview.

Conversely, although observation often provides a direct and powerful way of learning about people’s behavior and the context in which this occurs, interviewing can also be a valuable way of gaining a description of actions and events—often the only way, for events that took place in the past or to which you can’t gain observational access. Interviews can provide additional information that was missed in observation, and can be used to check the accuracy of the observations. However, in order for interviews to be useful for this purpose, you need to ask about specific events and actions rather than posing questions that elicit only generalizations or abstract opinions. . . . In both of these situations, triangulation of observations and interviews can provide a more complete and accurate account than either could alone. (pp. 106–107)

Triangulation of data sources within qualitative methods may not lead to a single, totally consistent picture. The point is to study and understand when and why differences appear. The fact that observational data produce different results from interview data does not mean that either or both kinds of data are “invalid,” although that may be the case. More likely, it means that different kinds of data have captured different things and so the analyst attempts to understand the reasons for the differences. Either consistency in overall patterns of data from different sources or reasonable explanations for differences in data from divergent sources can contribute significantly to the overall credibility of findings.

SIDEBAR

ETHNOGRAPHIC TRIANGULATION

In ethnographic research practice, triangulation of data sorts and methods and of theoretical perspectives leads to extended knowledge potentials, which are fed by the convergences, and even more by the divergences, they produce.

As in other areas of qualitative research, triangulation in ethnography is a way of promoting quality of research. . . . Good ethnographies are characterized by flexible and hybrid use of different ways of collecting data and by a prolonged engagement in the field. As in other areas of qualitative research, triangulation can help reveal different perspectives on one issue in research such as knowledge about and practices with a specific issue. Thus, triangulation is again a way to promote quality of qualitative research in ethnography also and more generally a productive approach to managing quality in qualitative research. (Flick, 2007b, p. 89)

2. Mixed-Methods Triangulation: Integrating Qualitative and Quantitative Data

Tis not the many oaths that makes the truth,

But the plain single vow that is vow’d true. —William Shakespeare (written 1604–1605)

Diana in All’s Wells That Ends Well

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/44!/4/2/4/2@0:0 16/39

Mixed-methods triangulation often involves comparing and integrating data collected through some kind of qualitative methods with data collected through some kind of quantitative method. Such efforts flow from a pragmatic approach to mixed-methods analysis that assumes potential compatibility and seeks to discover the degree and nature of such compatibility (Tashakkori & Teddlie, 1998; Teddlie & Tashakkori, 2003, 2011). This is seldom straightforward because certain kinds of questions lend themselves to qualitative methods (e.g., developing hypotheses or theory in the early stages of an inquiry, understanding particular cases in depth and detail, getting at meanings in context, and capturing changes in a dynamic environment), while other kinds of analyses lend themselves to quantitative approaches (e.g., generalizing from a sample to a population, testing hypotheses for statistical significance, and making systematic comparisons on standardized criteria). Thus, it is common that quantitative methods and qualitative methods are used in a complementary fashion to answer different questions that do not easily come together to provide a single, well-integrated picture of the situation.

Given the varying strengths and weaknesses of qualitative versus quantitative approaches, the researcher using different methods to investigate the same phenomenon should not expect that the findings generated by those different methods will automatically come together to produce some nicely integrated whole. Indeed, the evidence is that one ought to expect initial conflicts in findings from qualitative and quantitative data and expect those findings to be received with varying degrees of credibility. It is important, then, to consider carefully what each kind of analysis yields and thereby giving different interpretations the chance to arise, with each considered on its merits, before favoring one result over the other based on methodological biases.

Critical Multiplism as an Analytical Strategy

Critical multiplism is a research strategy that advocates designing packages of imperfect methods and theories in a manner that minimizes the respective and inevitable biases of each. Multiplism, applied to analysis, acknowledges that any analysis can usually be conducted in any one of several ways, but in many cases, no single way is known to be uniformly the best. Under such circumstances, a multiplist advocates making heterogeneous those aspects of analysis about which uncertainty exists, so that the task is conducted in several different ways, each of which is subject to different biases.

Critical refers to rational, empirical, and social efforts to identify the assumptions and biases present in the options chosen. Putting the two concepts together, we can say that the central tenet of critical multiplism is this: When it is not clear which of several defensible options for a scientific task is least biased, we should select more than one, so that our options reflect different biases, avoid constant biases, and leave no plausible bias overlooked. (Shadish, 1993, p. 18)

When multiple analytical approaches yield similar results across different analytical biases, confidence in the resulting findings is increased. If different results occur when the analysis is done in different ways, then we have to try to explain the differences.

Different Findings From Different Methods

In a classic article, Shapiro (1973) described in detail her struggle to resolve basic differences between qualitative data and quantitative data in her study of Follow Through Classrooms; she eventually concluded that some of the conflicts between the two kinds of data were the result of measuring different things, although the ways in which different things were measured were not immediately apparent until she worked to sort out the conflicting findings. She began with greater trust in the data derived from quantitative methods and ended by believing that the most useful information came from the qualitative data.

Another pioneering article, by M. G. Trend (1978) of ABT Associates, has become required reading for anyone becoming involved in a team project that will involve collecting and analyzing both qualitative and quantitative data, where different members of the team have responsibilities for different kinds of data. The Trend study involved an analysis of three social experiments designed to test the concept of using direct-cash housing allowance payments to help low-income families obtain decent housing on the open market. The analysis of qualitative data from a participant observation study produced results that were at variance with

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/44!/4/2/4/2@0:0 17/39

those generated by analysis of quantitative data. The credibility of the qualitative data became a central issue in the analysis.

The difficulty lay in conflicting explanations or accounts, each based largely upon a different kind of data. The problems we faced involved not only the nature of observational versus statistical inferences, but two sets of preferences and biases within the entire research team. . . .

Though qualitative/quantitative tension is not the only problem which may arise in research, I suggest that it is a likely one. Few researchers are equally comfortable with both types of data, and the procedures for using the two together are not well developed. The tendency is to relegate one type of analysis or the other to a secondary role, according to the nature of the research and the predilections of the investigators. . . . Commonly, however, observational data are used for “generating hypotheses,” or “describing process.” Quantitative data are used to “analyze outcomes,” or “verify hypotheses.” I feel that this division of labor is rigid and limiting. (Trend, 1978, p. 352)

Early Efforts at Quantitative–Qualitative Triangulation

SIDEBAR

STRATEGY FOR ACHIEVING QUALITY IN MIXED-METHODS STUDIES

The quantitative researchers work side by side every step of the way as full members of the case study team, bringing the analytic rigor of their quantitative frameworks to bear on case study and observation design, data collection, analysis, integration with other methods, and reporting. The qualitative researchers, in turn, are full members of the quantitative team (analysis of administrative data, survey research, and time series assessments), bringing their own rigor to survey designs, data reduction decisions, and interpretations. As a result, assumptions are more rigorously examined, methodological lacunae more clearly (and early) identified, and the team leaders become sufficiently methodologically multilingual so that they can discuss both qualitatively and quantitatively based findings with equal confidence (Datta, 2006, p. 427).

©2002 Michael Quinn Patton and Michael Cochran

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/44!/4/2/4/2@0:0 18/39

Anthropologists participating in teams in which both quantitative and qualitative data were being collected applied their inquiry skills to examine the nature of the experience in the 1970s. The problems they have shared were stark evidence that qualitative methods at that time were typically perceived as exploratory and secondary when used in conjunction with quantitative/experimental approaches. When qualitative data supported quantitative findings, that was the icing on the cake. When qualitative data conflicted with quantitative data, the qualitative data have often been dismissed or ignored (Society of Applied Anthropology, 1980).

A strategy of methods triangulation, then, doesn’t magically put everyone on the same page. While valuing and endorsing triangulation, Trend (1978) suggested that

we give different viewpoints the chance to arise, and postpone the immediate rejection of information or hypotheses that seem out of joint with the majority viewpoint. Observationally derived explanations are particularly vulnerable to dismissal without a fair trial. (pp. 352–353)

From Separation to Integration

Qualitative and quantitative data can be fruitfully combined to elucidate complementary aspects of the same phenomenon. For example, a community health indicator (e.g., teenage pregnancy rate) can provide a general and generalizable picture of an issue, while case studies of a few pregnant teenagers can put faces on the numbers and illuminate the stories behind the quantitative data; this becomes even more powerful when the indicator is broken into categories (e.g., those under the age of 15, those 16 and above), with case studies illustrating the implications of and rationale for such categorization.

In essence, triangulation of qualitative and quantitative data constitutes a form of comparative analysis. The question is “What does each analysis contribute to our understanding?” Areas of convergence increase confidence in findings. Areas of divergence open windows to better understanding of the multifaceted, complex nature of a phenomenon. Deciding whether results have converged remains a delicate exercise subject to both disciplined and creative interpretation. Focusing on the degree of convergence rather than forcing a dichotomous choice—that the different kinds of data do or do not converge—yields a more balanced overall result.

Mixed-Methods Analysis and Triangulation in the Twenty-First Century While difficulties still arise in triangulating and integrating qualitative and quantitative data, advances in mixed methods have propelled integrated analyses into the spotlight, especially in applied and interdisciplinary areas like policy analysis, program evaluation, environmental studies, international development, and global health. Where disciplinary barriers have yielded to genuine interdisciplinary engagement, traditional methodological divisions have yielded to collaboration and integration. Exhibit 8.27 (pp. 618–619) presented mixed-methods challenges and solutions. Exhibit 9.2 presents 10 developments that are making mixed-methods triangulation both valued and, increasingly, expected in applied social science.

3. Triangulation With Multiple Analysts A third kind of triangulation is investigator or analyst triangulation—that is, using multiple as opposed to singular observers or analysts. This is the core of qualitative team research (Guest & MacQueen, 2008). Triangulating observers or using several interviewers helps reduce the potential bias that comes from a single person doing all the data collection and provides means of more directly assessing the consistency of the data obtained. Triangulating observers provides a check on potential bias in data collection.

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/44!/4/2/4/2@0:0 19/39

A related strategy is triangulating analysts—that is, having two or more persons independently analyze the same qualitative data and compare their findings. In the traditional social science approach to qualitative inquiry, engaging multiple analysts and computing the interrater reliability among these different analysts is valued, even expected, as a means of establishing credibility of findings (Silverman & Marvasti, 2008, pp. 238–239).

SIDEBAR

A STORY OF MIXED-METHODS TRIANGULATION: TESTING CONCLUSIONS WITH MORE FIELDWORK

Economists Lawrence Katz and Jeffrey Liebman of Harvard, and Jeffrey R. Kling of Princeton, were trying to interpret data from a federal housing experiment that involved randomly assigning people to a program that would help them get out of the slums. The evaluation focused on the usual outcomes of improved school and job performance. However, to get beyond the purely statistical data, they decided to conduct interviews with residents in an inner-city poverty community.

Professor Lieberman commented to a New York Times reporter,

I thought they were going to say they wanted access to better jobs and schools, and what we came to understand was their consuming fear of random crime; the need the mothers felt to spend every minute of their day making sure their children were safe. (Uchitelle, 2001, p. 4)

By adding qualitative, field-based interview data to their study, Kling, Liebman, and Katz (2001) came to a new and different understanding of the program’s impacts and participants’ motivations based on interviewing the people directly affected, listening to their perspectives, and including those perspectives in their analysis.

EXHIBIT 9.2 Ten Developments Enhancing Mixed-Methods Triangulation

1. Designs that are truly mixed-methods inquiries are demonstrating the value of systematic, planned triangulation. Increased understanding of the strengths and weaknesses of qualitative and quantitative data has led to both the commitment and capacity to build on the strengths of each at the design stage.

2. Asking integrating questions of the data supports triangulation. Triangulation is most powerful when mixed-methods studies are designed for integration, which begins by asking the same questions of both methods and gathering both qualitative and quantitative data on those questions. That is happening at a level unprecedented in applied social science research and evaluation.

3. Mixed-methods sampling strategies anticipate and facilitate triangulation. Sampling with triangulation in mind is a collaborative strategy that anticipates and lays the foundation for mixed- methods analysis.

4. Specific methods are incorporating mixed data intentionally to support triangulation. Surveys ask both closed- and open-ended questions. Case studies collect both quantitative and qualitative data. Strong experimental designs gather both standardized intervention and quantitative effects data plus qualitative process data.

5. Mixed methods are proving especially appropriate for studying complex issues. Mixed-methods researchers are extending our understandings of how to understand complex social phenomena as well as how to use research to develop effective interventions to address complex social problems (Mertens, 2013; Patton, 2011).

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/44!/4/2/4/2@0:0 20/39

6. Team approaches are being created and implemented with mixed-methods skills and capabilities in mind. High-quality mixed-methods designs often require teams because individuals lack the full skill set needed. Knowing how to form and manage such teams has advanced significantly as experience has accumulated about what to do—and what not to do (Guest & MacQueen, 2008; Morgan, 2014).

7. Software supports mixed-methods data analysis and triangulation. As data analysis software has become more sophisticated, flexible, and responsive to analysts’ needs, techniques and processes for triangulation are becoming more common and easier to use.

8. Resources available for mixed-methods designs and analysis have burgeoned. The Journal of Mixed Methods began publishing in 2007, with an opening editorial by Abbas Taskhakkori and John Creswell proclaiming, “The New Era of Mixed Methods.” This means that there are more outlets for publishing mixed-methods studies. The Handbook of Mixed Methods was published in 2003 (Tashakkori & Teddlie). Excellent mixed-methods texts provide guidance on the full process from designing mixed-methods studies to analyzing and triangulating mixed data (Bamberger, 2013; Bergman, 2008; Greene, 2007; Mertens, 1998; Mertens & Hesse-Biber, 2013; Morgan, 2014).

9. Researchers are developing mixed skills, capabilities, and capacities—and being recognized and valued for their mixed-methods expertise. In 2014, the International Association of Mixed Methods Research was launched and hailed as “a momentous development in mixed-methods research” (Mertens, 2014).

10. Mixed-methods exemplars show what is possible. Early experiences with qualitative–quantitative triangulation were mixed at best—and many were quite negative, as indicated in the cautionary tales reported preceding the exhibit. When I was doing earlier editions of this book, there were more bad examples and negative experiences than good and positive exemplars. That balance has shifted for all the reasons listed here. The momentum is building as funders of research and evaluation are coming to demand mixed-methods studies.

Here, however, is a perfect example of how different criteria for judging quality lead to different practices. In a lead editorial for the journal Qualitative Health Research, Janet Morse (1997) took on “the myth of inter- rater reliability” from a social constructionist perspective. She begins by distinguishing standardized interview formats from more flexible and open interview guide approaches.She acknowledges that interrater reliability may be acceptable when everyone is asked the same question in the same way (the preferred interviewing approach to meet traditional social science concerns about validity and reliability), but in the more adaptive, personalized, individualized, and flexible approach of interview guides and conversational interviewing, what constitutes coherent passages for coding is more problematic and depends on the analyst’s interpretive framework. Multiple analysts might still discuss what they see in the data, share insights, and consider what emerges from their different perspectives, but that’s quite different from computing a statistical interrater reliability coefficient. (See the sidebar on “the myth of interrater reliability” for her full argument.)

SIDEBAR

PERFECTLY HEALTHY BUT DEAD: THE MYTH OF INTERRATER RELIABILITY

—Janet M. Morse (1997)

Qualitative researchers seem to have inherited a host of habits from quantitative researchers and have adopted them into the qualitative paradigm without considering the appropriateness of their purpose, rationale, or underlying assumptions. On the surface, these practices seem right, so they are unquestioningly maintained. One of these adopted habits is the practice of obtaining interrater reliability of coding decisions used in qualitative research when coding unstructured, interactive interviews.

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/44!/4/2/4/2@0:0 21/39

The argument goes something like this: To be reliable, coding should be replicable. Replication is checked by duplication; if coding decisions are explicit and communicated to another researcher, that researcher should be able to make the same coding decisions as the first researcher. The result is reliable research. Right?

Wrong. Interrater reliability is appropriate with semistructured interviews, wherein all participants are asked the same questions, in the same order, and data are coded all at once at the end of the data collection period. But this does not hold for unstructured interactive interviews. Recall that unstructured, interactive interviews are used in research because the researcher does not know enough about the topic or its parameters to construct interview questions. With unstructured, interactive interviews, the researcher first assumes a listening stance and learns about the topic as she or he goes along. Thus, once the researcher has learned something about the phenomenon from the first few participants, the substance of the interview then changes and becomes targeted on another aspect of the phenomenon. Importantly, unlike semistructured interviews, all participants are not asked the same questions. Participants are used to verify the information learned in the first interviews and are encouraged both to speak from their own experience and to speak for others. Each interview may overlap with the others but may also have a slightly different focus and different content.

This notion, learning from participants as the study progresses, is crucial to the understanding of the fluid nature of coding unstructured interviews. Initially, coding decisions may be quite superficial—by topic, for instance—but later coding decisions are made with the knowledge of, and in consideration of, information gained from all the previously analyzed interviews. Such coding schemes are not superficial, and in light of all the knowledge gained, small pieces of data may have monumental significance. The process is not necessarily superficially objective: It is conducted in light of comprehensive understanding of the significance of each piece of text. The coding process is highly interpretative.

This comprehensive understanding of data bits cannot be acquired in a few objective definitions of each category. Moreover, it cannot be conveyed quickly and in a few definitions to a new member of the research team who has been elected for the purpose of determining a percentage agreement score. This new coder does not have the same knowledge base as the researcher, has not read all the interviews, and therefore does not have the same potential for insight or depth of knowledge required to code meaningfully. Maintaining a simplified coding schedule for the purposes of defining categories for an interrater reliability check will maintain the coding scheme at a superficial level. It will simplify the research to such an extent that all of the richness attained from insight will be lost. Ironically, it forcibly removes each piece of data from the context in which each coding decision should be made. The study will become respectably reliable with an interrater reliability score, but this will be achieved at the cost of losing all the richness and creativity inherent in analysis, ultimately producing a superficial product.

The cost of such an endeavor is equivalent to Mrs. Frisby, who, when the farmer commented that the poisoned rat looked perfectly healthy, said sadly, “Perfectly healthy, but dead!” Your research will be perfectly reliable, but trivial.

There is often a shocked silence when I discuss this with students. But then I ask two questions: “How many of you have written a literature review lately?” Almost every hand is raised. I then ask, “How many of you took a second person to the library with you to make sure you interpreted each article in a manner that was replicable?” Not a single hand remains raised. “Aren’t you concerned?” I ask, “How do you know that your analysis, your interpretation of those articles, was reliable?”

The analysis of unstructured, interactive interviews is exactly the same case. Researchers must learn to trust themselves and their judgments and be prepared to defend their interpretations and analyses. But it is death to one’s study to simplify one’s insights, coding, and analyses so that another person may place the same piece of datum in the same category.

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/44!/4/2/4/2@0:0 22/39

Triangulation Through Distinct Evaluation Teams: The Goal-Free Approach

In program evaluation, an interesting form of team triangulation has been used. Michael Scriven (1972b) has advocated and used two separate teams, one that conducts a traditional goals-based evaluation (assessing the stated outcomes of the program) and a second that undertakes a “goal-free evaluation” in which the evaluators assess clients’ needs and program outcomes without focusing on stated goals (see Chapter 4, p. 206). Comparing the results of the goals-based team with those of the goal-free team provides a form of analytical triangulation for determining program effectiveness (Youker & Ingraham, 2014).

Review by Inquiry Participants Having those who were studied review the findings offers another approach to analytical triangulation. Researchers and evaluators can learn a great deal about the accuracy, completeness, fairness, and perceived validity of their data analysis by having the people described in that analysis react to what is described and concluded. To the extent that participants in the study are unable to relate to and confirm the description and analysis in a qualitative report, questions are raised about the credibility of the findings. In what became a classic study of how evaluations were used, key informants in each case study were asked for both verbal and written reactions to the accuracy and comprehensiveness of the cases. The evaluation report then included those written reactions (Alkin et al., 1979). In her study of homeless youth, Murphy (2014) met with each of the 14 youth to go over the details of the case study she created from their transcribed interviews to affirm accuracy, add additional details and reflections if they so desired, and choose a pseudonym that they wanted to be called in the study, if they had not already done so. (See Thmaris’s case study example, pp. 511–516.)

Obtaining the reactions of respondents to your working drafts is time-consuming, but respondents may (1) verify that you have reflected their perspectives; (2) inform you of sections that, if published, could be problematic for either personal or political reasons; and (3) help you to develop new ideas and interpretations. (Glesne, 1999, p. 152)

Different Purposes Drive Different Review Procedures

Different kinds of studies have different participant review processes, some none at all. Collaborative and participatory inquiry builds in participants’ review of cases, quotations, and findings as a matter of course; that’s part of what collaboration and participation mean. However, investigative inquiries (Douglas, 1976) aimed at exposing what goes on beyond the public eye are often antagonistic to those in power, so their responses would not typically be used to revise conclusions but might be used to at least offer them an opportunity to provide context and an alternative interpretation. Some traditional social science researchers and evaluators worry that sharing findings with participants for their reactions will undermine the independence of their analysis. Others view it as an important form of triangulation. In an Internet listserv discussion of this issue, one researcher reported this experience:

I gave both transcripts and a late draft of findings to participants in my study. I wondered what they would object to. I had not promised to alter my conclusions based on their feedback, but I had assured them that my aim was to be sure not to do them harm. My findings included some significant criticisms of their efforts that I feared/expected they might object to. Instead, their review brought forth some new information about initiatives that had not previously been mentioned. And their primary objection was to my not giving the credit for their successes to a wider group in the community. What I learned was not to make assumptions about participants’ thinking.

Exhibit 9.3 summarizes three contrasting views of involving those studied in reviewing findings and conclusions.

Critical Friend Review

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/44!/4/2/4/2@0:0 23/39

A critical friend can be defined as a trusted person who asks provocative questions, provides data to be examined through another lens, and offers critiques of a person’s work as a friend. A critical friend takes the time to fully understand the context of the work presented and the outcomes that the person or group is working toward. The friend is an advocate for the success of that work. (Costa & Kallick, 1993, p. 49)

EXHIBIT 9.3 Different Perspectives on Triangulation by Those Who Were Studied

Tessie Tzavaras Catsambasis is president of EnCompass LLC, an international evaluation research company. She is active in evaluation capacity building around the world, including leadership service with the International Organization for Cooperation in Evaluation. She also plays the role of critical friend with colleagues’ projects and within her own organization. Here’s an example she shared with me (and kindly gave permission to include here) that nicely illustrates the critical friend role as a form of analyst triangulation.

My team and I conducted an evaluation of a UN organization’s Internet-based system that countries could download to track their own HIV/AIDS activities in any sector and area, nationally down to district level. A previous organizational review of the UN organization recommended discontinuing this program based on resource constraints and rumors about problems, but without looking at it closely. The department supporting this program decided to evaluate it first, because they had invested in it significantly and wanted to make a final decision based on evidence. The evaluation we conducted (country visits, focus groups, interviews, survey, benchmarking) revealed many, many problems. But interestingly, some 20 countries were using it (the tracking system). My colleagues who did the data collection were ready to push the button to kill it, citing all the problems we had found. I got involved at the last stage of the data analysis process.

I grilled my colleagues, asking them to justify every conclusion. Their perspective was clear: “This program has so many operational obstacles in the field, no Internet, low capacity, we should recommend discontinuing it.” Then, I asked what turned out to be the “turning point” questions: “If this program has so many problems, why are 20 countries choosing to use it?” and “How are those countries addressing the problems you have documented?”

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/44!/4/2/4/2@0:0 24/39

This kind of question (at its best, how is it working?) is an appreciative analytical question asked as a critical friend from a systems dynamics perspective (change the shoes you are wearing, and from the perspective of a country, what do you see?). In response, my colleagues listed many innovations that countries were undertaking to make this tracking program work, and then they concluded, “It is the only option out there that they can control fully, and it is cheap.” So “country controlled” was also important, and so was “low cost.” Then, I asked them, “Imagine you hold the button to kill the program, do you push it?” They each said, “No, but we would . . . ” and proceeded to give me three fabulous recommendations. Their responses enabled us to present to the client the findings, and engage the client in grappling with a tough decision.

Essentially, we said, “This program is filling a demand, and 20 counties are using it in spite of significant operational problems. We know you have resource constraints, and this system requires more technical assistance, but if you decide to stop supporting it, consider transferring the system’s administration to another funding agency, and also consider certifying independent consultants as technical assistance providers, so countries can contract with them directly for help on the system. And, if you cannot even do that, think about how you will transition in a way that will not hurt countries.”

From an evaluation point of view, two things are important: (1) if it were not for these two questions in the analysis, the team would have concluded something very different from the same data, and (2) asking these questions enabled us to facilitate the client to face these challenging findings, have an internal debate about what to do, and own the final decision.

Audience Review as Credibility Triangulation Reflexive triangulation (Exhibit 2.5, p. 72) includes the audience’s reactions to the triangulation mix: (1) the inquirer’s reflexive perspective, (2) the perspectives of those studied, and (3) the perspectives of those who received the findings. The opening module of this chapter emphasized that different readers of qualitative reports will apply different criteria to judge quality and credibility. Audience reactions constitute additional data. Whenever possible, I prefer to present draft findings to multiple audiences to learn how they react, what they focus on, what is clear and unclear, and what questions are inadequately answered. In a sense, this is equivalent to theater or movie previews when producers and directors get to gauge audience reaction to a performance or film before it is released. Time and procedures for audience previews and reactions have to be planned in advance, but whenever I’ve done them, I’ve been glad I did.

In a study of a community development effort in an inner-city, low-income neighborhood, focus groups were done with diverse groups: African Americans, Native Americans, Hispanics, Hmong residents, and low- income whites. Age-based focus groups were also done: youth under the ages of 25, 25- to 55-year-olds, and those over 55 years. A community advisory group reviewed the study design and voiced no objections to focus groups done homogeneously by either ethnicity or age. In fact, they thought such focus groups were a good idea. But when the draft results were reported in a public meeting that included community people and public officials, the focus group results made it appear that there were great divisions and differences of perspectives among neighborhood ethnic and age-groups. Audience members outside the community were especially focused in on conflicts and differences reported in the findings. Similarities and important areas of agreement got lost amid the reports’ overemphasis on differences. Moreover, perspectives within ethnic group and age-group appeared much more monolithic and homogeneous than, in fact, they were. As a result of this feedback, we went back to the field and added heterogeneous focus groups to the data and then drafted a more balanced report. This was in no way undermining inquirer independence. It was making sure we got it right.

Evaluation Audiences and Intended Users

Program evaluation constitutes a particular challenge in establishing credibility because the ultimate test of the credibility of an evaluation report is the response of primary intended users and readers of that report. Their reactions often revolve around face validity. On the face of it, is the report believable? Are the data reasonable? Do the results connect to how people understand the world? In seriously soliciting intended users’ reactions, the evaluator’s perspective is joined to the perspective of the people who must use the findings.

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/44!/4/2/4/2@0:0 25/39

Evaluation theorist Ernie House (1977) has suggested that the more “naturalistic” (qualitative) the evaluation, the more it relies on its audiences to reach their own conclusions, draw their own generalizations, and make their own interpretations:

Unless an evaluation provides an explanation for a particular audience, and enhances the understanding of that audience by the content and form of the argument it presents, it is not an adequate evaluation for that audience, even though the facts on which it is based are verifiable by other procedures. One indicator of the explanatory power of evaluation data is the degree to which the audience is persuaded. Hence, an evaluation may be “true” in the conventional sense but not persuasive to a particular audience for whom it does not serve as an explanation. In the fullest sense, then, an evaluation is dependent both on the person who makes the evaluative statement and on the person who receives it [italics added]. (p. 42)

Understanding the interaction and mutuality between the evaluator and the people who use the evaluation, as well as relationships with participants in the program, is critical to understanding the human side of evaluation. This is part of what gives evaluation—and the evaluator—situational and interpersonal “authenticity” (Lincoln & Guba, 1986). Exhibit 9.16, at the end of this chapter (pp. 736–741), provides an experiential account from an evaluator dealing with issues of credibility while building relationships with program participants and evaluation users; her reflections provide a personal, in-depth description of what authenticity is like from the perspective of one participant-observer.

Expert Audit Review

A final review alternative involves using experts to assess the quality of analysis or, where the stakes for external credibility are especially high, performing a meta-evaluation or process audit. An external audit by a disinterested expert can render judgment about the quality of data collection and analysis. “That part of the audit that examines the process results in a dependability judgment [italics added], while that part concerned with the product (data and reconstructions) results in a confirmability judgment [italics added]” (Lincoln & Guba, 1986, p. 77). Such an audit would need to be conducted according to appropriate criteria. For example, it would not be fair to audit an aesthetic and evocative qualitative presentation by traditional social science standards or vice versa. But within a particular framework, expert reviews can increase credibility for those who are unsure how to distinguish high-quality work. That, of course, is the role of the doctoral committee for graduate students and peer reviewers for scholarly journals. Problems arise when peer reviewers apply traditional scientific criteria to constructivist studies, and vice versa. In such cases, the review or audit itself lacks credibility. Exhibit 9.4 on the next page presents an example of an expert meta-evaluation (evaluation of the evaluation) to independently judge the quality and establish credibility for a high-stakes international mixed-methods evaluation.

The challenge of getting the right expert, one who can apply an appropriately critical eye, is wittily illustrated by a story about the great French artist Pablo Picasso. Marketing of fakes of his paintings plagued Picasso. His friends became involved in helping check out the authenticity of supposed genuine originals. One friend in particular became obsessed with tracking down frauds and brought several paintings to Picasso, all of which the master identified as fake. A poor artist who had hoped to profit from having obtained a Picasso before the great artist’s works had become so valuable sent his painting for inspection via the friend. Again Picasso pronounced it a forgery.

“But I saw you paint this one with my very own eyes,” protested the friend.

“I can paint false Picassos as well as anyone,” retorted Picasso.

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/44!/4/2/4/2@0:0 26/39

©2002 Michael Quinn Patton and Michael Cochran

4. Theory Triangulation Greek legend tells of the fearsome hotelier Procrustes who would adjust his guests to match the length of his bed, stretching the short and trimming off the legs of the tall. Guides to program theory that are too prescriptive risk creating such a Procrustean bed. When the same approach to program theory is used for all types of interventions and all types of purposes, the risk is that the interventions will be distorted to fit into a preconceived format. Important aspects may be chopped off and ignored, and other aspects may be stretched to fit into preconceived boxes of a factory model, with inputs, processes, outcomes, and impacts.

Purposeful program theory requires thoughtful assessment of circumstances, asking in particular, “Who is going to use the program theory and for what purposes?” and “What is the nature of the intervention and the situation in which it is implemented?” It requires a wide repertoire, not a one-size-fits-all approach to program theory.

Purposeful program theory also requires attention to the limitations of any one program theory, which must necessarily be a simplification of reality and a willingness to revise it as needed to address emerging issues.

—Funnell and Rogers (2011, p. xxi) Purposeful Program Theory

Having discussed triangulation of qualitative data sources, mixed-methods triangulation, and multiple analyst triangulation, we turn now to the fourth and final kind of triangulation: using different theoretical perspectives to look at the same data. Chapter 3 presented a number of general theoretical frameworks derived from diverse intellectual and disciplinary traditions. More concretely, multiple theoretical perspectives can be brought to bear on specialized substantive issues. For example, one might examine interviews with therapy clients from different psychological perspectives: psychotherapy, Gestalt, Adlerian, and behavioral psychology. Observations of a group, community, or organization can be examined from a Marxian or Weberian perspective, a conflict or functionalist point of view. The point of theory triangulation is to understand how differing assumptions and premises affect findings and interpretations.

EXHIBIT 9.4 Metaevaluation: Evaluating the Evaluation of the Paris Declaration on Development Aid

It has become a standard in major high-stakes evaluations to commission an independent review to determine whether the evaluation meets generally accepted standards of quality and, in so doing, to identify strengths, weaknesses, and lessons (Stufflebeam & Shrinkfield, 2007, p. 649). The major addition

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/44!/4/2/4/2@0:0 27/39

to the Joint Committee Standards for Evaluation, when revised in 2010, was that of “Evaluation Accountability Standards” focused on meta-evaluation.

Evaluation Accountability Standards

E1 Evaluation documentation: Evaluations should fully document their negotiated purposes and implemented designs, procedures, data, and outcomes. E2 Internal meta-evaluation: Evaluators should use these and other applicable standards to examine the accountability of the evaluation design, procedures employed, information collected, and outcomes. E3 External meta-evaluation: Program evaluation sponsors, clients, evaluators, and other stakeholders should encourage the conduct of external meta-evaluations using these and other applicable standards (Joint Committee on Standards, 2010; Yarbrough, Shulha, Hopson, & Caruthers, 2010).

Evaluating the Evaluation of the Paris Declaration

Given the historic importance of the Evaluation of the Paris Declaration on Development Aid (Dabelstein & Patton, 2013b), the Management Group overseeing the evaluation commissioned an independent assessment of the evaluation. Prior to undertaking this review, we had no prior relationship with any members of the Management Group or the Core Evaluation Team. We had complete and unfettered access to any and all evaluation documents and data, and to all members of the International Reference Group, the Management group, the Secretariat, and the Core Evaluation Team. Our evaluation of the evaluation included reviewing data collection instruments, templates, and processes; reviewing the partner country and donor evaluation reports on which the synthesis of findings was based; directly observing two meetings of the International Reference Group where the evidence was examined and the conclusions refined and sharpened accordingly; engaging International Reference Group participants in a reflective practice, lessons-learned session; surveying participants about the evaluation process and partner country evaluations; and interviewing key people involved in and knowledgeable about how the evaluation was conducted. The evaluation of the evaluation included assessing both the evaluation report’s findings and the technical appendix that details how the findings were generated. The Development Assistance Committee (DAC) of the Organization for Economic Co-operation and Development (OECD) established international standards for evaluation in 2010, and those were the standards used for the meta-evaluation (OECD-DAC, 2010). A meta-evaluation audit statement confirming the quality, credibility, and usability of the evaluation was included as a preface to the full evaluation reports. The meta-evaluation report (Patton & Gornick, 2011a) was published and made available online two weeks after the Final Evaluation report was published. This timing was possible because the meta-evaluation began halfway through the Paris Declaration Evaluation and the meta-evaluation team had access to draft versions of the final report at each stage of the report’s development. The process for conducting the meta-evaluation and its uses are discussed in detail in Patton (2013).

The Paris Declaration Evaluation received the 2012 American Evaluation Association (AEA) Outstanding Evaluation Award. At the award ceremony, the chair of the AEA Awards Committee, Frances Lawrenz (2013), summarized the merits of the evaluation that led to the award selection and recognition:

The success of the Paris Declaration Phase 2 Evaluation required an unusually skilled, knowledgeable and committed evaluation team; a visionary, well-organized, and well-connected Secretariat to manage the logistics, international stakeholder meetings, and financial accounts; and a highly competent and respected Management Group to provide oversight and ensure the Evaluation’s independence and integrity. This was an extraordinary partnership where all involved understood their roles, carried out their responsibilities fully and effectively, and respected the contributions of other members of the collaboration.

Examples of Theory Triangulation

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/44!/4/2/4/2@0:0 28/39

Let’s suppose we are studying famine in a drought-afflicted region of an African country. We have quantitative data on food production (sorghum and millet), nutrition data from household surveys, health data from clinics, rainfall data over many years, interviews with villagers (males and females), key informant interviews (e.g., government officials, agricultural experts, aid agency staff members, and village leaders), and case studies of purposefully sampled villages telling the story of their agricultural and nutritional situations and experiences before and during the famine. Put all of these data together and we have an in-depth description of the extent and nature of the famine, its effects on subsistence agriculture families, food and agricultural assistance provided, and the interventions of government and international agencies. We have (a) mixed-methods triangulation and (b) multiple sources of qualitative data (interviews, observations, case studies, documents), and (c) our team members have analyzed the patterns independently to confirm the findings as well as had the findings externally reviewed by experts. Thus, we can make a credible case for the nature, extent, and impacts of the famine. What does theory triangulation add?

When we move from description to interpretation, we need a framework to make sense of and explain the patterns in the data. Why is the region experiencing famine? Why aren’t interventions more effective? Different theoretical frameworks emphasize different explanatory variables.

• Climate change theory would emphasize long-term weather and climate trends. • Malthusian theory would emphasize overpopulation.

“I envy your confidence. Even after decades of evaluations, these metaevalutions still make me feel naked.”

• Marxian theory would emphasize power dynamics (Who controls the means of production? How do the powerful benefit from famine?).

• Weberian theory would emphasize organizational competence and incompetence (How does the functioning and activities of government and international agencies exacerbate or alleviate famine?).

• Ecological systems theory would call for examining the interactions between the ecosystem, farming practices, soil and water conditions, and markets.

• Cultural systems theory would emphasize the way in which cultural beliefs and norms affect the experience of and responses to famine by the people affected.

• Feminist theory would point to the role of women in the system as a factor in how the famine affects families and their responses to the crisis (Podems, 2014b).

• Cognitive theory would focus on how people make decisions in the face of changing conditions.

When designing the famine study, these various theoretical perspectives would inform the kinds of questions to be asked and data to be collected. When analyzing the findings and explaining results, these

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/44!/4/2/4/2@0:0 29/39

diverse theoretical perspectives provide competing interpretations for explaining the patterns and observed impacts. Theory triangulation involves examining the data through different theoretical lenses to see what theoretical framework (or combination) aligns most convincingly with the data (best fit).

Theory triangulation for evaluation can involve examining the data from the perspectives of various stakeholder positions. It is common for diverse stakeholders to disagree about program purposes, goals, and means of attaining goals. These differences represent different “theories of action” (Patton, 2012a) that can cast the same findings in different perspective-based lights. When we were seeking explanations to explain dropout rates for adult literacy programs in Minnesota, the predominant staff theory was that low-income people led chaotic lives and couldn’t manage regular attendance and follow-through in a program. Political explanations included laziness, effects of multigenerational poverty, lack of good jobs to motivate participants to complete programs, and cultural deprivation theories. But the explanation that best fit the data (interviews with dropouts) was that the adult literacy programs were lousy learning experiences: large class sizes; disinterested and disrespectful teachers, poorly paid and exhausted from having already taught all day in their regular jobs; uninteresting and outdated curriculum materials; and an all-around depressing environment. Most traditional explanations blamed the participants or the larger societal problems that affected the participants, but the actual data pointed to ineffective programs, something that was actionable. Changes were made, and dropout rates went down significantly.

SIDEBAR

THEORY INTEGRATION MEETS THEORY TRIANGULATION

Different criteria for evaluating the quality of qualitative remain fluid as qualitative inquirers move back and forth among genres, ignoring the boundaries, much as birds ignore human fences—except to use them occasionally as convenient places to rest. Consider the reflections on working across and integrating multiple genres and theoretical orientations of self-described “critical educators” Patricia Burdell and Beth Blue Swadener (1999). They combine autobiographical narratives with a variety of theoretical perspectives, including critical, dialogic, phenomenological, feminist, and semiotic perspectives. They speculate that “it is perhaps both the intent and effect of many of these texts to broaden the ‘acceptable’ or give voice to the intellectual contradictions and tensions in everyday lives of scholar-teachers and researchers” (p. 23).

Our research has used narrative inquiry, collaborative ethnography, and applied semiotics. Between us, we share an identity and scholarship in critical and feminist curriculum theory. We are frequent border-crossers. We seek texts that allow us to enter the world of others in ways that have us more present in their experience, while better understanding our own. (p. 23).

They call this border-crossing genre “critical personal narrative and autoethnography.” The real world in which inquiry occurs is not a very neat and orderly place. Nor is it likely to become so. Theoretical and methodological border crossers are natural and determined triangulators.

Thoughtful, Systematic Triangulation All four of these different types of triangulation—(1) mixed-methods triangulation, (2) triangulation of qualitative data sources, (3) analyst triangulation, and (4) theory or perspective triangulation—offer strategies for reducing systematic bias and distortion during data analysis, and thereby increasing credibility. In each case, the strategy involves checking findings against other sources and perspectives. Triangulation, in whatever form, increases credibility and quality by countering the concern (or accusation) that a study’s findings are simply an artifact of a single method, a single source, or a single investigator’s blinders. Exhibit 9.1 (p. 660) reviews and summarizes the four types of triangulation (items 7–10 in Exhibit 9.1).

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/44!/4/2/4/2@0:0 30/39

Exhibit 9.5 presents a model for rigorous analysis that broadens and deepens triangulation processes in high-stakes, high-visibility situations. Eight attributes of a rigorous analysis process were identified by studying experienced intelligence analysts from multiple U.S. federal investigative agencies. The researchers used a cognitive systems approach in which professional intelligence analysts were engaged in going beyond assessment of the quality of an analysis based on product quality to examine the analytic processes necessary to generate a high-quality, credible, and useful product. The understanding of rigor that emerged was that it is not about following a standardized, highly prescribed analytical process (a formula or recipe) but, rather, “assessing the contextual sufficiency of many different aspects of the analytic process” (Zelik et al., 2007). The researchers posited that these dimensions could be relevant to any process where analysts must make sense of complex data, and the rigor and resulting credibility of their analytical process will affect the utility of the findings for decision making. Examine Exhibit 9.5 carefully and thoughtfully. There’s a lot there pulled together in a comprehensive, coherent, and integrated triangulation model: The Rigor Attribute Model. What comes across most powerfully from the work that generated the model is that a product (report, findings, or presentation of results) cannot be assessed for quality and credibility without knowing the nature and rigor of the analytical process that generated the findings. That insight is consistent with the focus of my MQP Rumination, avoiding research rigor mortis, in this chapter (see pp. 701–703).

EXHIBIT 9.5 Dimensions of Rigorous Analysis and Critical Thinking

The Rigor Attribute Model

Eight attributes of a rigorous analysis process were identified by studying experienced intelligence analysts from multiple U.S. federal investigative agencies. The researchers used a cognitive systems approach in which professional intelligence analysts were engaged in going beyond assessment of the quality of an analysis based on product quality to examine the analytic process that generated the product. The understanding of rigor that emerged was that it is not about following a standardized process but, rather, “assessing the contextual sufficiency of many different aspects of the analytic process” (Zelik et al., 2007). The researchers posited that these dimensions could be relevant to any process where analysts must make sense of complex data, and the rigor and resulting credibility of their analytical process will affect the utility of the findings for decision making.

Overview of the Eight Dimensions of Rigorous Analysis

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/44!/4/2/4/2@0:0 31/39

SOURCE: Adapted and revised from Zelik, Patterson, and Woods (2007).

SIDEBAR

INTERPRETING TRIANGULATION RESULTS: MAKING SENSE OF CONFLICTING AND INCONSISTENT CONCLUSIONS

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/44!/4/2/4/2@0:0 32/39

A common misconception about triangulation involves thinking that the purpose is to demonstrate that different data sources or inquiry approaches yield essentially the same result. The point is to test for such consistency. Different kinds of data may yield somewhat different results because different types of inquiry are sensitive to different real-world nuances. Different theoretical frameworks will likely foster different interpretations of the same findings. Different analysts may well interpret the same patterns in different ways. Thus, understanding inconsistencies in findings across different kinds of triangulation can be illuminative and important. Finding such inconsistencies ought not to be viewed as weakening the credibility of results but, rather, as offering opportunities for deeper insight into the relationship between inquiry approach and the phenomenon under study.

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/44!/4/2/4/2@0:0 33/39

MODULE

78 Alternative and Competing Criteria for Judging the Quality of Qualitative Inquiries, Part 1

Universal Criteria, and Traditional Scientific Research Versus Constructivist Criteria

Every way of seeing is also a way of not seeing. —David Silverman (2000, p. 825)

Judging Quality: The Necessity of Determining Criteria

It all depends on criteria. Judging quality requires criteria. Credibility flows from those judgments. Quality and credibility are connected in that judgments of quality constitute the foundation for perceptions of credibility.

Diverse approaches to qualitative inquiry—phenomenology, ethnomethodology, ethnography, hermeneutics, symbolic interaction, heuristics, critical theory, realism, grounded theory, and feminist inquiry, to name but a few—remind us that issues of quality and credibility intersect with audience and intended inquiry purposes. Research directed to an audience of independent feminist scholars, for example, may be judged by somewhat different criteria from research addressed to an audience of government economic policymakers. Formative research or action inquiry for program improvement involves different purposes and therefore different criteria of quality compared with summative evaluation aimed at making fundamental continuation decisions about a program or policy. Thus, it is important to acknowledge at the outset that particular philosophical underpinnings or theoretical orientations and special purposes for qualitative inquiry will generate different criteria for judging quality and credibility.

Despite this, efforts to generate universal criteria and checklists for quality abound. The results are as follows: multiple possibilities, no consensus, and ongoing debate.

A Review of Quality Assurance Recommendations for Qualitative Research

An interdisciplinary team of health researchers engaged in worldwide malaria prevention and treatment set out to identify quality criteria for qualitative research and evaluation (Reynolds et al., 2011). They found 93 papers published between 1994 and 2010 that offered and discussed quality criteria, 37 of which were sufficiently detailed to merit further analysis. (The 56 papers that were rejected focused only on review criteria for publication or guidance on a specific qualitative method or single stage of the research process, such as data analysis.) They found no consensus about how to ensure the quality of qualitative research. However, they were able to categorize approaches into two “narratives” about quality: (1) an output-oriented approach versus (2) a process-oriented approach:

1. The most dominant narrative detected was that of an output-oriented approach. Within this narrative, quality is conceptualized in relation to theoretical constructs such as validity or rigor, derived from the positivist paradigm, and is demonstrated by the inclusion of certain recommended methodological techniques: the use of triangulation, member (or participant) validation of findings, peer review of findings, deviant or negative case analysis, and multiple coders of data.

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/44!/4/2/4/2@0:0 34/39

Strengths of the output-oriented approach for assuring quality of qualitative studies include the acceptability and credibility of this approach within the dominant positivist environment where decision making is based on “objective” criteria of quality. Checklists equip those unfamiliar with qualitative research with the means to assess its quality.

The weakness of this approach is that “following of check-lists does not equate with understanding of and commitment to the theoretical underpinnings of qualitative paradigms or what constitutes quality within the approach. The privileging of guidelines as a mechanism to demonstrate quality can mislead inexperienced qualitative researchers as to what constitutes good qualitative research. This runs the risk of reducing qualitative research to a limited set of methods, requiring little theoretical expertise and diverting attention away from the analytic content of research unique to the qualitative approach. Ultimately, one can argue that a solely output- oriented approach risks the values of qualitative research becoming skewed towards the demands of the positivist paradigm without retaining quality in the substance of the research process.”

2. By contrast, the second, process-oriented narrative, presented conceptualizations of quality that were linked to principles or values considered inherent to the qualitative approach, to be understood and enacted throughout the research process. Six common principles were identified across the narrative: (1) reflexivity of the researcher’s position, assumptions, and practice; (2) transparency of decisions made and assumptions held; (3) comprehensiveness of approach to the research question; (4) responsibility toward decision making acknowledged by the researcher; (5) upholding good ethical practice throughout the research; and (6) a systematic approach to designing, conducting, and analyzing a study.

Strengths of the process-oriented approach include the ability of the researcher to address the quality of their research in relation to the core principles or values of qualitative research. The core principles identified in this narrative also represent continuous, researcher-led activities rather than externally determined indicators such as validity, or end-points. Reflexivity, for example, is an active, iterative process—an attitude of attending systematically to the context of knowledge construction . . . at every step of the research process. As such, this approach emphasises the need to consider quality throughout the whole course of research, and locates the responsibility for enacting good qualitative research practice firmly in the lap of the researcher(s).

Need for a Flexible Quality Framework

The review team (Reynolds et al., 2011) found that “there is an increasing demand for the qualitative research field to move forward in developing and establishing coherent mechanisms for quality assurance of qualitative research.” They concluded with a recommendation for “the development of a flexible framework to help qualitative researchers to define, apply and demonstrate principles of quality in their research.” They further recommended that “the strengths of both the output-oriented and process-oriented narratives be brought together to create guidance that reflects core principles of qualitative research but also responds to expectations of the global health field for explicitly assured quality in research.”

We recommend the development of a framework that helps researchers identify their core principles, appropriate for their epistemological and methodological approach, and ways to demonstrate that these have been upheld throughout the research process. . . . We propose that this framework be flexible enough to accommodate different qualitative methodologies without dictating essential activities for promoting quality. (Reynolds et al., 2011)

This chapter addresses this recommendation, offering both a generic quality framework as well as specialized quality criteria for specific types of qualitative inquiry.

SIDEBAR

THE PURPOSE OF AND DEBATE ABOUT CRITERIA

Criteria are standards, benchmarks, norms, and, in some cases, regulative ideals that guide judgments about the goodness, quality, validity, truthfulness, and so forth of competing claims (or methodologies, theories,

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/44!/4/2/4/2@0:0 35/39

interpretations, etc.). . . .

Criteria that have been proposed for judging the processes and products of social inquiry include truth, relevance, validity, credibility, plausibility, generalizability, social action, and social transformation, among others. Some of these criteria are epistemic (i.e., concerned with justifying knowledge claims as true, accurate, correct), others are political (i.e., concerned with warranting the power, use, and effects of knowledge claims or the inquiry process more generally); still others are moral or ethical standards (i.e., concerned with the right conduct of the inquirer and the inquiry process in general). . . .

Poststructuralist and postmodernist approaches to qualitative inquiry are also shaping the way we conceive of criteria. Given the growing influence of narrative approaches and experimental texts in qualitative inquiry, it is becoming more common to find discussions of rhetorical and aesthetic criteria replacing discussions of epistemic criteria. Other scholars argue that epistemological criteria cannot be neatly decoupled from political and critical agendas and ethical concerns. Some scholars in qualitative inquiry have little patience for discussing criteria within different epistemological frameworks and theoretical perspectives and prefer to focus on the craft of using various methodological procedures for producing “quality” work.

—Schwandt (2007, pp. 49–50) The Sage Dictionary of Qualitative Inquiry

Judging the Quality of Alternative Approaches to Qualitative Inquiry There can be no universal, generic, standardized, and all-encompassing criteria for judging the quality of qualitative studies because qualitative inquiry is not monolithic, uniform, or standardized. It’s as if someone set out to create a universal checklist for beauty that ignored culture, human variability, variety, and differences in taste, socialization, and values (oh yes, the “Miss Universe” and “Miss World” contests notwithstanding). The common core elements across all kinds of qualitative inquiry are attention to language, words, narrative, description, stories, cases, worldviews, and how people make sense of their worlds. Tracy (2010), for example, identified “eight ‘big-tent’ criteria for excellent qualitative research”: (1) worthy topic, (2) rich rigor, (3) sincerity, (4) credibility, (5) resonance, (6) significant contribution, (7) ethics, and (8) meaningful coherence. But approaches to inquiring into and judging attainment of these general criteria are diverse and multifaceted, serve competing purposes, and are, ultimately, a matter of debate (Gordon & Patterson, 2013).

It is possible to specify quality criteria for research generally. These are not unique to qualitative inquiry but apply to scientific inquiries of all kinds. Exhibit 9.6 presents these general, science-based quality criteria.

EXHIBIT 9.6 General Scientific Research Quality Criteria

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/44!/4/2/4/2@0:0 36/39

EXHIBIT 9.7 Alternative Sets of Criteria for Judging the Quality and Credibility of Qualitative Inquiry

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/44!/4/2/4/2@0:0 37/39

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/44!/4/2/4/2@0:0 38/39

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/44!/4/2/4/2@0:0 39/39