U8D1-68 - Data Analysis Strategies Reviewed by Peers - Please follow the instructions as outlined below. See attachments for readings.

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 1/37

Summary: Interpretation

This module has focused on interpreting findings, determining substantive significance, elucidating phenomenological essence, understanding hermeneutic interpretation, and distinguishing how different theory-based inquiries yield certain kinds of findings. The core theme has been interpretation. Interpretation is how we make sense of data. Interpretation is both an inevitability and a necessity. But as philosopher Friedrich Nietzsche added, “Necessity is not an established fact, but an interpretation.” The same can be said of causal inference, our next subject.

Permission is included in documents already submitted for Claudius Ceccon (Brazil).

SOURCE: Brazilian cartoonist Claudius Ceccon. Used with permission.

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 2/37

MODULE

71 Causal Explanation Through Qualitative Analysis

A scientist puts a frog on a table and yells, “Jump!” The frog jumps. He then surgically removes a leg from the frog, puts the frog back on the table, and yells, “Jump!” The frog jumps again. The scientist surgically removes another limb and repeats the experiment. The frog jumps when commanded, just like before. He does it a third time. “Jump!” he exclaims after removing another limb, and the frog jumps. He removes the last limb on the frog and tries again. “Jump!” he yells, but the frog remains still. “Jump!” he repeats, but no response.

The scientist writes his findings in his notebook: “Upon removing all four limbs, the frog becomes deaf.”

This rather grotesque frog story is meant to facilitate the transition from Module 70, which focused on interpreting what qualitative data mean, to this module, which focuses on causal explanation. In essence, we are moving from interpreting findings to explaining them. Causal inference is a particularly treacherous and important form of qualitative analysis and interpretation.

Causality as Explanation

We construct reality, not exclusively but importantly, in terms of cause and effect. We must—else our individual worlds would be chaotic jumbles of actions unrelated to consequences.

—Northcutt and McCoy (2004, p. 169) Interactive Qualitative Analysis

Description tells what happened and how the story unfolded. Interpretation elucidates what the description means and judges what makes it significant. Then comes the “why?” question. Why did things unfold as they did? The answer: Because.

“Causation is intimately related to explanation; asking for explanation of an event is often to ask why it happened” (Schwandt, 2001, p. 23).

• The homeless youth moved into housing and enrolled in school or got a job because of what they experienced, how they were treated, and what they learned in the program serving homeless youth.

• The crime rate in the community is low because people look out for and support each other. • Climate change is occurring because of human activity. • The gap between rich and poor is increasing because of public policies favoring wealth accumulation by

those already rich.

To use the verb “because” is to posit an explanation and assert a causal connection. We make such assertions all the time in casual conversation. “You caused me to be late for my meeting because you didn’t wake me.” “You caused the car accident because you were texting on your phone and not paying attention to the road.” When researchers and evaluators make causal claims, however, evidence is required to support the assertion. Yet what counts as credible evidence is a matter of vociferous debate (Donaldson, Christie, & Mark, 2008). “Exactly how to define a causal relationship is one of the most difficult topics in epistemology and the philosophy of science. There is little agreement on how to establish causation” (Schwandt, 2001, p. 23).

Stake (1995) has emphasized that

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 3/37

explanations are intended to promote understanding and understanding is sometimes expressed in terms of explanation—but the two aims are epistemologically quite different . . . , a difference important to us, the difference between case studies seeking to identify cause and effect relationships and those seeking understanding of human experience. (p. 38)

Appreciating and respecting this distinction, once case studies have been written and descriptive typologies have been developed and supported, the tasks of organization and description are largely complete. Whether to take the next step in analysis and posit causality depends on the purpose of the inquiry and whether it has been designed to answer causal questions. As emphasized at the beginning of this chapter, analysis is driven by purpose. Purpose determines design. Design frames and focuses analysis.

In program evaluation, for example, explanations about which things appear to lead to other things, which aspects of a program produce certain effects, and how processes lead to outcomes are natural areas for analysis. When careful study of the data gives rise to ideas about causal linkages, there is no reason to deny those interested in the study’s results the benefit of those insights, with presentation of the supporting evidence from interviews, case studies, and field observations.

SIDEBAR

WHY WE’RE SO OBSESSED WITH CAUSALITY

We are pattern seekers, believers in a coherent world, in which regularities appear not by accident but as a result of mechanical causality or of someone’s intention.

—Daniel Kahneman (cognitive scientist) (Quoted in Pomeroy, 2013, p. 1)

Humans are creatures of causality. We like effects to have causes, and we detest incoherent randomness. Why else would the quintessential question of existence give rise to so many sleepless nights, endear billions to religion, or single-handedly fuel philosophy?

This predisposition for causation seems to be innate. In the 1940s, psychologist Albert Michotte theorized that “we see causality, just as directly as we see color,” as if it is omnipresent. To make his case, he devised presentations in which paper shapes moved around and came into contact with each other. When subjects— who could only see the shapes moving against a solid-colored background—were asked to describe what they saw, they concocted quite imaginative causal stories. . . .

Humanity’s need for concrete causation likely stems from our unceasing desire to maintain some iota of control over our lives. That we are simply victims of luck and randomness may be exhilarating to a madcap few, but it is altogether discomforting to most. By seeking straightforward explanations at every turn, we preserve the notion that we can always affect our condition in some meaningful way. Unfortunately, that idea is a facade. Some things don’t have clear answers. Some things are just random. Some things simply can’t be controlled. (Pomeroy, 2013, p. 1)

Reporting the Causal Assertions of Others Versus Inferring Causality Yourself

To the extent that you describe and report the causal linkages suggested by and believed in by those you’ve interviewed, you haven’t crossed the line from description into causal interpretation. You are simply reporting their beliefs and assertions.

Much qualitative inquiry stops at reporting the explanations of the people studied. This can take the form of presenting indigenous explanations through quotations, presenting case data, and/or identifying cross-case patterns and themes that describe how people explain key phenomena of interest, but without the analyst

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 4/37

adding any additional explanation of “why” the indigenous causal assertions take the form they do. People and groups construct causal explanations, for example, the gods are angry when it thunders. How people explain things can suffice. Indeed, the study of the attributions people make (how they explain things) offers a vast panorama for qualitative inquiry.

Attributions refer to people’s understandings of themselves and their environment. . . . Someone who has just passed a driving test at the first attempt may attribute his or her success to good fortune; another might attribute it to a happy choice of driving school; a third might attribute it to their own natural driving talent; and so on. Attributions such as these are often made about matters of moment in our lives. . . .

Interest in attribution stems from its crucial role as a mediator of perceptions, emotions, motivations and behaviours. Indeed, how people see cause and effect has implications for or may influence their interpersonal relations, their psychopathology, their response to psychotherapy, their decision making, and their adjustment to illness. . . . Attributions may serve many other functions, including enhancement of one’s perception of control, preservation of self-esteem, presentation of a particular picture of the self, and emotional release. (Harvey, Turnquist, & Agostinell, 1998, pp. 32–33)

Causal Explanation Grounded in Fieldwork

A researcher who has lived in a community for an extensive period of time will likely have insights into why things happen as they do there. A qualitative analyst who has spent hours interviewing people will likely come away from the analysis with possible explanations for how the phenomenon of interest takes the forms and has the effects it does. An evaluator who has studied a program, lived with the data from the field, and reflected at length about the patterns and themes that run through the data is in as good a position as anyone else at that point to interpret meanings, make judgments about significance, and offer explanations about relationships. Moreover, if decision makers and evaluation users have asked for such information—and in my experience they virtually always welcome causal analyses—there is no reason not to share insights with them to help them think about their own causal presuppositions and hypotheses, and to explore what the data do and do not support in the way of interconnections and potential causal relationships. But doing so remains controversial among those who lack training in qualitative causal analysis. I take the position in this module that qualitative causal analysis has now advanced in rigor and credibility to a point where it should be valued on its merits—and this module will demonstrate those merits. In this module, we’ll examine different perspectives on qualitative causal explanations and ways of increasing the credibility and utility of such analyses.

Historical Context: Approaching Causation Cautiously

The qualitative/quantitative debate of the past century positioned qualitative methods as primarily descriptive and quantitative methods as explanatory. Internal validity in experimental designs concerns the extent to which causal claims are warranted and can be substantiated. Randomized controlled trials (RCTs) involve manipulating a variable (the independent variable) to measure what effect it has on a second variable of interest (the dependent variable). This quantitative/experimental method, first used in agricultural and pharmaceutical studies and then applied in psychology and social science, became advocated as the “gold standard” for establishing causality. My MQP Rumination opposing this gold standard designation is in Chapter 3 (pp. 93–95). For our purpose here, the historically important point is that the experimental method became synonymous with causal research. RCTs had high internal validity. Case studies had low to no internal validity, meaning that they could describe but not explain (Campbell & Stanley, 1963; Shadish, Cook, & Campbell, 2001).

This remains the dominant perspective to this day, but developments in qualitative methods and analysis over the past two decades have demonstrated that qualitative analysis can yield causal explanations rigorously and credibly. This assertion is controversial and remains controversial largely because the advocates of experimental designs have been so successful in positioning experiments as the gold standard and the only way to establish causality. This module will present and explain how qualitative analysis can be

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 5/37

used to generate causal explanations. Exhibit 8.19, in Module 72 (pp. 600–601), summarizes approaches to causal explanation in qualitative analysis.

SIDEBAR

THE CAUSAL WARS ARE RAGING

The causal wars are still raging, and the amount of collateral damage is increasing.

The causal wars are about what is to count as scientifically impeccable evidence of a causal connection, usually in the context of the evaluation of interventions into human affairs.

The collateral damage comes from the policy that the RCT camp has been supporting with considerable success, here referred to as “the exclusionary policy,” which recommends that no (or almost no) programs be funded whose claims of good effects cannot be supported by randomized controlled trials (RCT)-based evidence. This means terminating many demonstrably excellent programs currently saving huge numbers of life-years.

After reviewing the causal wars and alternatives for making causal inferences, Scriven (2008) concludes, “In sum, there is absolutely nothing imperative, and nothing in general superior, about . . . RCT designs” (p. 23).

SOURCE: From the introduction to “A Summative Evaluation of RCT Methodology and an Alternative Approach to Causal Research,” by philosopher of science and evaluation research pioneer Michael Scriven (2008, p. 11).

Qualitative Causal Analysis as Speculative: Classic Advice to Proceed With Caution

Qualitative researchers . . . have generally denied that they were seeking causal explanations, arguing that their goal was the interpretive understanding of meanings rather than the identification of causes.

—Maxwell and Mittapalli (2008, pp. 322–323)

Lofland’s (1971) advice in his classic and influential book Analyzing Social Settings offers a cautionary perspective on the role of causal “speculation” in qualitative analysis. He argued that the strong suit of the qualitative researcher is the ability “to provide an orderly description of rich, descriptive detail” (p. 59); the consideration of causes and consequences using qualitative data should be a “tentative, qualified, and subsidiary task” (p. 62).

It is perfectly appropriate that one be curious about causes, so long as one recognizes that whatever account or explanations he develops is conjecture. In more legitimacy-conferring terms, such conjectures are called hypotheses or theories. It is proper to devote a portion of one’s report to conjectured causes of variations so long as one clearly labels his conjectures, hypotheses or theories as being that. (p. 62)

Neuendorf (2002) follows in this cautious tradition by setting such a high standard for determining causality, especially from content analysis of qualitative data, that it “is generally impossible to fully achieve . . . ; true causality is essentially an unattainable goal” (pp. 47–48).

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 6/37

In contrast to this cautious approach to drawing causal inferences from qualitative data, other qualitative analytical frameworks focus on causal explanation as a primary purpose, an attainable outcome, and even the strength of case studies.

Rigorous Qualitative Causal Analysis

The Case for Valuing Direct Observation

The most straightforward form of causal attribution involves direct observation in a short time frame where you can link the cause and effect in a straightforward manner. For example, you go out to a restaurant with friends, and all eat raw oysters. After the meal, all of you experience stomach sickness. It’s not a wild speculation to conclude that the oysters caused the sickness.

During a site visit to an employment training program, I witnessed a staff member yelling at a participant. The participant immediately left the building. I followed him and asked if I could talk with him for a moment. He said, “No, I’m finished with that program. Done. I won’t stand to be yelled at like that. I’m so out of there.” He turned and went his way. I checked subsequent attendance. He didn’t return. It seems reasonable to conclude that being yelled at was at least a contributing cause of his dropping out.

The idea of establishing causality has taken on such heavy meaning philosophically, methodologically, and epistemologically that the very idea can be daunting and intimidating (Maxwell, 2004).

Thus, qualitative analysts have often been advised to eschew using causal language and avoid making assertions. Reclaiming causal analysis as a reasonable and valid form of qualitative analysis begins with a recognition that direct, critical observation yields causal understandings.

The real “gold standard” for causal claims is the same ultimate standard as for all scientific claims; it is critical observation. Causation can be directly observed, in [a] lab or home or field, usually as one of many contextually embedded observations, such as lead being melted by heating a crucible, eggs being fried in a pan, or a hawk taking a pigeon. And causation can also be inferred from non-causal direct observations with no experimentation, as by the forensic pathologist performing an autopsy to determine the cause of death. (Scriven, 2008, p. 18)

One end of the observational continuum is seeing an action and a reaction that allows a direct, immediate conclusion that the action caused the reaction. At the other end of the continuum is long-term observation, including participant observation. Such in-depth qualitative studies yield detailed, comprehensive data about what happened in the observed settings, both how and why what happened happened. Spending a long time in a field setting engaging in ongoing observations, interviewing people, and tracking the details of what occurred makes it possible to connect the dots to explain what has occurred. When I read a classic high- quality, in-depth, comprehensive, field-based case study (e.g., Becker, Geer, Hughes, & Strauss, 1961; Goffman, 1961), the causal explanations offered strike me as valid, warranted, well supported, and consistent with the great volume of evidence presented.

What Constitutes Credible Evidence?

The fact that I find in-depth case studies persuasive does not mean that others do. Valuing different kinds of evidence is what the causal attribution war is about. Consider the challenge of eradicating intestinal worms in children, a widespread problem in many developing countries. Suppose we want to evaluate an intervention in which school-age children with diarrhea are given deworming medicine to increase their school attendance and performance. To attribute the intervention to the desired outcome, advocates of RCTs would insist on an evaluation design in which students suffering from diarrhea are randomly divided into a treatment group (those who receive worm medicine) and a control group (those who do not receive the medicine). The school attendance and test performance of the two groups would then be compared. If, after a month on the medicine, those receiving the intervention show higher attendance and school performance at a statistically significant

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 7/37

level compared with the control group (the counterfactual), then the increased outcomes can be attributed to the intervention (the worm medicine).

Advocates of qualitative inquiry would question the value of the control group in this case. Suppose that students, parents, teachers, and local health professionals are interviewed about the reasons why students miss school and perform poorly on tests. Independently, each of these groups assert that diarrhea is a major cause of the poor school attendance and performance. Gathering data separately from different informant groups (students, parents, teachers, and health professionals) is a form of triangulation, a way of checking the consistency of findings from different data sources. Following the baseline interviews, students are given a regimen of worm medicine. Those taking the medicine show increased school attendance and performance, and in follow-up interviews, the students, parents, teachers, and health professionals independently affirm their belief that the changes can be attributed to taking the worm medicine and being relieved of the symptoms of diarrhea. Is this credible, convincing evidence?

Those who find such a design sufficient would argue that the results are both reasonable and empirical and that the high cost of adding a control group is not needed to establish causality. Nor, they would assert, is it ethical to withhold medicine from students with diarrhea when relieving their symptoms has merit in and of itself. The advocates of RCTs would respond that without the control group, other unknown factors may have intervened to affect the outcomes and that only the existence of a counterfactual (control group) will establish with certainty the impact of the intervention.

As this example illustrates, those evaluators and methodologists on opposite sides of this debate have different worldviews about what constitutes sufficient evidence for attribution and action in the real world. This is not simply an academic debate. At stake are millions of dollars of evaluation funds and the credibility of different kinds of and approaches to evaluations around the world.

Thus far, I’ve been presenting the case for using high-quality, detailed, context-specific qualitative data to make causal inferences. When causal findings from fieldwork are explained through relevant theory, the explanations move to a higher level of generalizability. We turn now to theory-based approaches to causal inference.

Theory-Based Causal Analysis

Realist Analysis to Identify Causal Mechanisms

Qualitative inquiry grounded in realist philosophy and methods makes causal analysis the central focus. The overarching question for realist inquiry is “What are the causal mechanisms that explain how and why reality unfolds as it does in a particular context?” (See Chapter 3, pp. 111–114.) Causal explanation comes from carefully documenting and analyzing the actual mechanisms and processes that are involved in particular events and situations.

These mechanisms and processes can include mental phenomena as well as physical phenomena and can be identified in unique events as well as through regularities. This position’s emphasis on understanding processes, rather than on simply showing an association between variables, provides an alternative approach to causal explanation that is particularly suited to qualitative research. It incorporates qualitative researchers’ emphasis on meaning for actors and on unique contextual circumstances, and by treating causal processes as real events, it implies that these may be observed directly rather than only inferred. Thus, it removes the restriction that causal inference requires the comparison of situations in which the presumed cause is present or absent. (Maxwell & Mittapalli, 2008, p. 323)

Emmel (2013) does not hesitate to make drawing causal inferences a priority purpose of realist inquiries because a “scientific realist sampling strategy” (p. 95) is always based in theory.

Explanation and interpretation in a realist sampling strategy tests and refines theory. Sampling choices seek out examples of mechanisms in action, or inaction towards being able to say something explanatory about their causal

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 8/37

powers. Sampling . . . is both pre-specified and emergent, it is driven forward through an engagement with what is already known about that which is being investigated and ideas catalyzed through engagement with empirical accounts. (p. 85)

This illustrates how purpose and design drive analysis. Cases sampled are chosen purposefully because they will illuminate causal mechanisms. The inquiry involves detailed description and analysis of the real connection between events in such a way that they can be validly explained as “causally connected” (Emmel, 2013, p. 99). Emmel (2013) uses the image of splitting open a chicken (“spatchcocking”) to study its inner parts as analogous to realist qualitative analysis:

Splitting a chicken down its breastbone and opening it up to reveal the details of its thoracic and abdominal cavities. In a similar way, in research we will split these things, these variables . . . open and lay bare their anatomy for scrutiny and explanation through theorisation and empirical investigation. In the process of which we will be able to better describe, interpret, and, ultimately, explain the sample. (p. 100)

SIDEBAR

WHY THE GERMAN BOMBING OF LONDON IN WORLD WAR II DID NOT DEMORALIZE THE BRITISH

Both British and German leaders expected the intense aerial bombing of London in World War II to create panic. London’s residents were expected to flee to the countryside, leaving the city abandoned. Over an eight-month period of incessant bombing in 1940 and 1941, more than 40,000 people were killed, 46,000 injured, and a million buildings damaged or destroyed (Gladwell, 2013, p. 129). But the panic never came. Why?

Canadian psychiatrist J. T. MacCurdy (1943) offered an explanation in his book The Structure of Morale. He categorized three groups of people affected by the bombing: (1) those killed, (2) near misses, and (3) remote misses. He interviewed and conducted case studies of near misses and remote misses. His qualitative comparative analysis led to a causal explanation.

So why were Londoners so unfazed by the Blitz? Because forty thousand deaths and forty-six thousand injuries —spread across a metropolitan area of more than eight million people—means that there were many more remote misses who were emboldened by the experience of being bombed than there were near misses who were traumatized by it. (Gladwell, 2013, p. 133)

“We are all of us not merely liable to fear,” MacCurdy went on,

we are also prone to be afraid of being afraid, and the conquering of fear produces exhilaration. . . . When we have been afraid that we may panic in an air-raid, and, when it has happened, we have exhibited to others nothing but a calm exterior and we are now safe, the contrast between the previous apprehension and the present relief and feeling of security promotes a self-confidence that is the very father and mother of courage. (MacCurdy, quoted by Gladwell, 2013, p. 133)

In the midst of the Blitz, a middle-aged laborer in a button-factory was asked if he wanted to be evacuated to the countryside. He had been bombed out of his house twice. But each time he and his wife had been fine. He refused. “What, and miss all this?” he exclaimed. “Not for all the gold in China! There’s never been nothing like it! Ever! And never will be again.” (Gladwell, 2013, pp. 132–133)

Grounded Theory and Causal Analysis

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 9/37

Theory denotes a set of well-developed categories (e.g., themes, concepts) that are systematically interrelated through statements of relationship to form a theoretical framework that explains some relevant social, psychological, educational, nursing, or other phenomenon. The statements of relationship explain who, what, when, where, why, how, and with what consequences an event occurs. Once concepts are related through statements of relationship into an explanatory theoretical framework, the research findings move beyond conceptual ordering to theory. . . . A theory usually is more than a set of findings; it offers an explanation about phenomena. (Strauss & Corbin, 1998, p. 22)

Chapter 3 provided an overview of grounded theory in the context of other theoretical perspectives like ethnography, constructivism, phenomenology, and hermeneutics. As I noted in Chapter 3, grounded theory has opened the door to qualitative inquiry in many traditional academic social science and education departments, especially as a basis for doctoral dissertations. I believe this is, in part, because of its overt emphasis on the importance of and specific procedures for generating explanatory theory. Being systematic gets particular emphasis.

By systematic, I still mean systematic every step of the way; every stage done systematically so the reader knows exactly the process by which the published theory was generated. The bounty of adhering to the whole grounded theory method from data collection through the stages to writing, using the constant comparative method, show how well grounded theory fits, works, and is relevant. Grounded theory produces a core category and continually resolves a main concern, and through sorting the core category organizes the integration of the theory. . . . Grounded theory is a package, a lock-step method that starts the researcher from a “know nothing” to later become a theorist with a publication and with a theory that accounts for most of the action in a substantive area. The researcher becomes an expert in the Substantive area. . . . And if an incident comes his way that is new he can humbly through constant comparisons modify his theory to integrate a new property of a category.

Grounded theory methodology leaves nothing to chance by giving you rules for every stage on what to do and what to do next. If the reader skips any of these steps and rules, the theory will not be as worthy as it could be. The typical falling out of the package is to yield to the thrill of developing a few new, capturing categories and then yielding to use them in unending conceptual description and incident tripping rather that analysis by constant comparisons. (Glaser, 2001, pp. 1–2)

SIDEBAR

NO PRECONCEPTIONS: THE GROUNDED THEORY DICTUM

Preconceived questions, problems, and codes all block emergent coding, thereby undermining the foundation of Grounded Theory. Entering the field without preconceptions is the fundamental grounded theory dictum. The grounded theory researcher begins an inquiry without knowing the participant’s issues, worldview, basic concepts, or sense-making framework. These emerge through the course of the inquiry.

—Barney G. Glaser (2014b) No Preconceptions: The Grounded Theory Dictum

Other grounded theory resources are as follows:

STOP, WRITE!: Writing Grounded Theory (Glaser, 2012)

Memoing: A Vital Grounded Theory Procedure (Glaser, 2014a)

In their book on techniques and procedures for developing grounded theory, Strauss and Corbin (1998) emphasize that analysis is the interplay between the researcher and data, so what grounded theory offers as a framework is a set of “coding procedures” to “help provide some standardization and rigor” to the analytical process. Grounded theory is meant to “build theory rather than test theory.” It strives to “provide researchers

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 10/37

with analytical tools for handling masses of raw data.” It seeks to help qualitative analysts “consider alternative meanings of phenomenon.” It emphasizes being “systematic and creative simultaneously.” Finally, it elucidates “the concepts that are the building blocks of theory.” Grounded theory operates from a correspondence perspective in that it aims to generate explanatory propositions that correspond to real-world phenomena. The characteristics of a grounded theorist, they posit, are these:

1. The ability to step back and critically analyze situations 2. The ability to recognize the tendency toward bias 3. The ability to think abstractly 4. The ability to be flexible and open to helpful criticism 5. Sensitivity to the words and actions of respondents 6. A sense of absorption and devotion to the work process (p. 7)

According to Strauss and Corbin (1998), grounded theory begins with basic description, then moves to conceptual ordering (organizing data into discrete categories “according to their properties and dimensions and then using description to elucidate those categories” [p. 19]) and then theorizing: “conceiving or intuiting ideas—concepts—then also formulating them into a logical, systematic, and explanatory scheme” (p. 21).

In doing our analyses, we conceptualize and classify events, acts, and outcomes. The categories that emerge, along with their relationships, are the foundations for our developing theory. This abstracting, reducing, and relating is what makes the difference between theoretical and descriptive coding (or theory building and doing description). Doing line-by-line coding through which categories, their properties, and relationships emerge automatically takes us beyond description and puts us into a conceptual mode of analysis. (p. 66)

Strauss and Corbin (1998) have defined terms and processes in ways that are quite specific to grounded theory. It is informative to compare the language of grounded theory with the language of phenomenological analysis presented in the previous module. Here’s a sampling of important terminology.

Microanalysis: “The detailed line-by-line analysis necessary at the beginning of a study to generate initial categories (with their properties and dimensions) and to suggest relationships among categories; a combination of open and axial coding” (p. 57). Theoretical sampling: “Sampling on the basis of the emerging concepts, with the aim being to explore the dimensional range or varied conditions along which the properties of concepts vary” (p. 73). Theoretical saturation: “The point in category development at which no new properties, dimensions, or relationships emerge during analysis” (p. 143). Range of variability: “The degree to which a concept varies dimensionally along its properties, with variation being built into the theory by sampling for diversity and range of properties” (p. 143). Open coding: “The analytic process through which concepts are identified and their properties and dimensions are discovered in data” (p. 101). Axial coding: “The process of relating categories to their subcategories, termed ‘axial’ because coding occurs around the axis of the category, linking categories of the level of properties and dimensions” (p. 123). Relational statements: “We call these initial hunches about how concepts relate ‘hypotheses’ because they make two or more concepts, explaining the what, why, where, and how of phenomenon” (p. 135).

Comparative Analysis

According to Strauss and Corbin (1998), comparative analysis is a core technique of grounded theory development. Making theoretical comparisons—systematically and creatively—engages the analyst in

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 11/37

“raising questions and discovering properties and dimensions that might be in the data by increasing researcher sensitivity” (p. 67). Theoretical comparisons are one of the techniques used when doing microscopic analysis. Such comparisons enable

identification of variations in the patterns to be found in the data. It is not just one form of a category or pattern in which we are interested but also how that pattern varies dimensionally, which is discerned through a comparison of properties and dimensions under different conditions. (p. 67)

Strauss and Corbin (1998) offer specific techniques to increase the systematic and rigorous processes of comparison, for example, “the flip-flop technique”:

This indicates that a concept is turned “inside out” or “upside down” to obtain a different perspective on the event, object, or actions/interaction. In other words, we look at opposites or extremes to bring out significant properties (p. 94).

In the course of conducting a grounded theory analysis, one moves from lower-level concepts to higher- level theorizing:

Data go to concepts, and concepts get transcended to a core variable, which is the underlying pattern. Formal theory is on the fourth level, but the theory can be boundless as the research keeps comparing and trying to figure out what is going on and what the latent patterns are. (Glaser, 2000, p. 4)

Glaser (2000) worries that the popularity of grounded theory has led to a preponderance of lower-level theorizing without completing the job. Too many qualitative analysts, he warns, are satisfied to stop when they’ve merely generated “theory bits.”

Theory bits are a bit of theory from a substantive theory that a person will use briefly in a sentence or so. . . .

Theory bits come from two sources. First, they come from generating one concept in a study and conjecturing without generating the rest of the theory. With the juicy concept, the conjecture sounds grounded, but it is not; it is only experiential. Second, theory bits come from a generated substantive theory. A theory bit emerges in normal talk when it is impossible to relate the whole theory. So, a bit with grab is related to the listener. The listener can then be referred to an article or a report that describes the whole theory. . . .

Grounded theory is rich in imageric concepts that are easy to apply “on the fly.” They are applied intuitively, with no data, with a feeling of “knowing” as a quick analysis of a substantive incident or area. They ring true with great credibility. They empower conceptually and perceptually. They feel theoretically complete (“Yes, that accounts for it.”). They are exciting handles of explanation. They can run way ahead of the structural constraints of research. They are simple one or two variable applications, as opposed to being multivariate and complex. . . . They are quick and easy. They invade social and professional conversations as colleagues use them to sound knowledgeable. . . . The danger, of course, is that they might be just plain wrong or irrelevant unless based in a grounded theory. Hopefully, they get corrected as more data come out. The grounded theorist should try to fit, correct, and modify them even as they pass his or her lips.

Unfortunately, theory bits have the ability to stunt further analysis because they can sound so correct. . . . Multivariate thinking stops in favor of a juicy single variable, a quick and sensible explanation. . . . Multivariate thinking can continue these bits to fuller explanations. This is the great benefit of trusting a theory that fits, works, and is relevant as it is continually modified. . . . But a responsible grounded theorist always should finish his or her bit with a statement to the effect that “Of course, these situations are very complex or multivariate, and without more data, I cannot tell what is really going on.” (pp. 7–8)

As noted throughout this chapter in commenting on how to learn qualitative analysis, it is crucial to study examples. Bunch (2001) has published a grounded theory study about people living with HIV/AIDS. Glaser (1993) and Strauss and Corbin (1997) have collected together in edited volumes a range of grounded theory exemplars that include several studies of health (life after heart attacks, emphysema, chronic renal failure, chronically ill men, tuberculosis, and Alzheimer’s disease), organizational headhunting, abusive relationships,

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 12/37

women alone in public places, selfhood in women, prison time, and the characteristics of contemporary Japanese society. The journal Grounded Theory Review began publication in 2000.

The theory generated can take the form of a model and be told as a story based on the coding categories used to organize the data. So, for example, causal conditions will have been identified that produced the phenomenon that is being studied. Strategies show how these causal conditions operated in particular contexts. These strategies are mediated by intervening conditions and produce action and interactions that result in consequences. The model, which articulates a theory, also tells a causal story.

Narrative Analysis and Case Studies: Telling a Coherent Causal Story Narrative analysis (see Chapter 3, pp. 128–131) interprets stories, life history narratives, historical memoirs, and creative nonfiction to reveal cultural and social patterns through the lens of individual experiences. Chapter 2 featured an interpretation of the significance of the story of Henrietta Lacks, whose cells were taken without her knowledge and used for medical research (pp. 78–80). Her story takes much of its meaning from what it reveals about the African American community at that time, the nature and norms of medical research, the policies of a major university, and the larger political, social, and economic context within which her story unfolded. The story explains why her cells were taken for medical research without her consent. The case study traces the consequences that resulted—consequences for the researchers, her family, the university, and people suffering from a great variety of diseases whose treatment came from what was learned from her cells. Case studies can be written in a variety of ways, one of which is “causal narratives” specifically constructed to “elucidate the processes at work in one case, or a small number of cases, using in-depth intensive analysis and a narrative presentation of the argument” (Maxwell & Mittapalli, 2008, p. 324).

Yin (2012) distinguishes descriptive case studies from explanatory case studies. Descriptive case studies generate “rich and revealing insights into the social world of a particular case” (p. 49). An explanatory case study, in contrast, “seeks to explain how and why a series of events occurred. In real-world settings, the explanations can be quite complex and can cover an extended period of time. Such conditions create the need for using the case study method rather than conducting, say, an experiment or a survey” (p. 89). Explanatory case studies use causal reasoning to create a coherent, data-based explanation of how one thing led to another.

The internal validity of an explanatory case study depends on the richness of the explanation, the detailed depiction of processes and actions that lead to observed outcomes and consequences, triangulation of data sources, and exploration of rival explanations to arrive at the one that best fits the data. Causal reasoning is the basis for organizing and making sense of a case study or narrative through systematic process tracing (see Exhibit 8.18).

Qualitative Comparative Analysis (QCA) Qualitative comparative analysis, developed and championed by political sociologist Charles Ragin (1987, 2000), focuses on systematically making comparisons to generate explanations. He has developed a systematic approach for making comparisons of large case units, like nation-states and historical periods, or macrosocial phenomena, like social movements. He constructs and codes each case as a combination of causal and outcome conditions. These combinations can be compared with each other and then logically simplified through a bottom-up process of paired comparison. He aims to draw on the strength of holistic analysis manifest in context-rich individual cases while making possible systematic cross-case comparisons of relatively large numbers of cases, for example, 15 to 25 or more. Ragin (2000, 2008) draws on fuzzy set theory and calls the result “diversity-oriented research” because he systematically codes and takes into account case variations and uniquenesses as well as commonalities, thereby elucidating both similarities and differences. The comparative analysis involves constructing a “truth table” in which the analyst codes each case for the presence or absence of each attribute of interest (Fielding & Lee, 1998, pp. 158–159). The

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 13/37

information in the truth table displays the different combinations of conditions that produce a specific outcome. To deal with the large number of comparisons needed, QCA is done using a customized software program.

Analysts conducting diversity-oriented research are admonished to assume maximum causal complexity by considering the possibility that no single causal condition may be either necessary or sufficient to explain the outcome of interest. Different combinations of causal conditions might produce the observed result, though singular causes can also be considered, examined, and tested. Despite reducing large amounts of data to broad patterns represented in matrices or some other form of shorthand, Ragin (1987) stresses repeatedly that these representations must ultimately be evaluated by the extent to which they enhance understanding of specific cases. A cause–consequence comparative matrix, then, can be thought of as a map providing guidance through the terrain of multiple cases and causes.

EXHIBIT 8.18 Process Tracing for Causal Analysis

Process tracing identifies, tests, elucidates, and validates causal mechanisms by describing the “causal chain between an independent variable (or variables) and the outcome of the dependent variable” (George & Bennett, 2005, p. 206–207).

Beach and Pedersen (2013) assert that process-tracing methods are arguably the only methods that allow us to study causal mechanisms:

Studying causal mechanisms with process-tracing methods enables the researcher to make strong within-case inferences about the causal process whereby outcomes are produced, enabling us to update the degree of confidence we hold in the validity of a theorized causal mechanism . . . [and] enabling us to open up the black box of causality using in-depth case study methods to make strong within-case inferences about causal mechanisms based on in-depth single-case studies that are arguably not possible with other social science methods. (pp. 1–2)

Beach and Pedersen (2013) differentiate three approaches to process tracing: (1) theory testing, (2) theory building, and (3) explaining outcomes:

Theory-testing process tracing deduces a theory from the existing literature and then tests whether evidence shows that each part of a hypothesized causal mechanism is present in a given case, enabling within-case inferences about whether the mechanism functioned as expected in the case and whether the mechanism as a whole was present. . . .

Theory-building process-tracing seeks to build a generalizable theoretical explanation from empirical evidence, inferring that a more general causal mechanism exists from the facts of a particular case. . . .

Finally, explaining-outcome process-tracing attempts to craft a minimally sufficient explanation of a puzzling outcome in a specific historical case. Here the aim is not to build or test more general theories but to craft a (minimally) sufficient explanation of the outcome of the case where the ambitions are more case-centric than theory-oriented. This distinction reflects the case-centric ambitions of many qualitative scholars. (p. 3)

QCA [qualitative comparative analysis] seeks to recover the complexity of particular situations by recognizing the conjunctural and context-specific character of causation. Unlike much qualitative analysis, the method forces researchers to select cases and variables in a systematic manner. This reduces the likelihood that “inconvenient” cases will be dropped from the analysis or data forced into inappropriate theoretical moulds. . . .

QCA clearly has the potential to be used beyond the historical and cross-national contexts originally envisioned by Ragin. (Fielding & Lee, 1998, pp. 160, 161–162)

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 14/37

SIDEBAR

CROSS-CULTURAL CASE ANALYSIS COMPARABILITY

In cross-cultural research, the challenge of determining comparable units of analysis has created controversy. For example, when definitions of “family” vary dramatically, can one really do systematic comparisons? Are extended families in nonliterate societies and nuclear families in modern societies such different entities that, beyond the obvious surface differences, they cease to be comparable units for generating theory? “The main problem for ethnologists has been to define and develop adequate and equivalent cultural units for cross-cultural comparison” (De Munck, 2000, p. 279).

©2002 Michael Quinn Patton and Michael Cochran

Analytic Induction The analysis of set relations is critically important to social research. . . . Qualitative analysis is fundamentally about set relations. Consider this simple example: if all (or almost all) of the anorectic teenage girls I interview have highly critical mothers (that is, the anorectic girls constitute a consistent subset of the girls with highly critical mothers), then I will no doubt consider this connection when it comes to explaining the causes and contexts of anorexia. This attention to consistent connections (e.g., causally relevant commonalities that are more or less uniformly present in a given set of cases) is characteristic of qualitative inquiry. It is the cornerstone of the technique commonly known as analytic induction. (Ragin, 2008, p. 2)

Analytic induction also involves cross-case analysis in an effort to seek explanations. Ragin’s qualitative comparative analysis formalized and moderated the logic of analytic induction (Ryan & Bernard, 2000, p. 787), but it was first articulated as a method of “exhaustive examination of cases in order to prove universal, causal generalizations” (Peter Manning, quoted by Vidich & Lyman, 2000, p. 57). In Norman Denzin’s sociological methods classic The Research Act (1978b), he identified analytic induction based on comparisons of carefully done case studies as one of the three primary strategies available for dealing with and sorting out rival explanations in generating theory; the other two are experiment-based inferences and multivariate analysis. Analytic induction as a comparative case method

was to be the critical foundation of a revitalized qualitative sociology. The claim to universality of the causal generalizations is . . . derived from the examination of a single case studied in light of a preformulated hypothesis

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 15/37

that might be reformulated if the hypothesis does not fit the facts. . . . Discovery of a single negative case is held to disprove the hypothesis and to require its reformulation. (Vidich & Lyman, 2000, p. 57)

Over time, those using analytic induction have eliminated the emphasis on discovering universal causal generalizations and have, instead, emphasized it as a strategy for engaging in qualitative inquiry and comparative case analysis that includes examining preconceived hypotheses—that is, without the pretense of the mental blank slate advocated in purer forms of phenomenological inquiry and grounded theory.

In analytic induction, researchers develop hypotheses, sometimes rough and general approximations, prior to entry into the field or, in cases where data already are collected, prior to data analysis. These hypotheses can be based on hunches, assumptions, careful examination of research and theory, or combinations. Hypotheses are revised to fit emerging interpretations of the data over the course of data collection and analysis. Researchers actively seek to disconfirm emerging hypotheses through negative case analysis, that is, analysis of cases that hold promise for disconfirming emerging hypotheses and that add variability to the sample. In this way, the originators of the method sought to examine enough cases to assure the development of universal hypotheses.

Originally developed to produce universal and causal hypotheses, contemporary researchers have de-emphasized universality and causality and have emphasized instead the development of descriptive hypotheses that identify patterns of behaviors, interactions and perceptions. . . . Bogdan & Biklen (1992) have called this approach modified analytic induction. (Gilgun, 1995, pp. 268–269)

Jane Gilgun (1995) used modified analytic induction in a study of incest perpetrators to test hypotheses derived from the literature on care and justice and to modify them to fit an in-depth subjective account of incest perpetrators. She used the literature-derived concepts to sensitize herself throughout the research while remaining open to discovering concepts and hypotheses not accounted for in the original formulations. And she did gain new insights:

Most striking about the perpetrators’ accounts was that almost all of them defined incest as love and care. The types of love they expressed ranged from sexual and romantic to care and concern for the welfare of the children. These were unanticipated findings. I did not hypothesize that perpetrators would view incest as caring and as romantic love. Rather, I had assumed that incest represented lack of care and, implicitly, an inability to love [literature- derived hypotheses]. It did not occur to me that perpetrators would equate incest and romance, or even incest and feelings of sexualized caring. From previous research, I did assume that incest perpetrators would experience profound sexual gratification through incest. Ironically, their professed love of whatever type was contradicted by many other aspects of their accounts, such as continuing the incest when children wanted to stop, withholding permission to do ordinary things until the children submitted sexually, and letting others think the children were lying when the incest was disclosed. These perpetrators, therefore, did not view incest as harmful to victims, did not reflect on how they used their power and authority to coerce children to cooperate, and even interpreted their behavior in many cases as forms of care and romantic love. (p. 270)

Analytic induction reminds us that qualitative inquiry can do more than discover emergent concepts and generate new theory. A mainstay of science has always been examining and reexamining and reexamining yet again those propositions that have become the dominant belief or explanatory paradigm within a discipline or group of practitioners. Modified analytic induction provides a name and guidance for undertaking such qualitative inquiry and analysis.

Forensic Causal Analysis On August 1, 2007, the major six-lane I-35 interstate highway bridge connecting Minneapolis and Saint Paul collapsed during rush hour. Thirteen people died, and another 145 were injured. The National Transportation Safety Board investigators studied evidence from the accident, including a security video camera that recorded the collapse. The investigators determined that the bridge’s steel gusset plates were undersized and inadequate to support the load of the bridge, a load that had had increased substantially over time since the bridge’s construction. During the recovery of the wreckage, investigators discovered gusset plates at eight different joint locations that were fractured. The investigation turned up photos from a June 2003 inspection

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 16/37

of the bridge that showed gusset plate bowing. The investigation concluded that the primary cause of the collapse was the undersized gusset plates. Contributing to the collapse was the fact that 2 inches of concrete was added to the road surface over the years, increasing the dead load by 20%. Also contributing was the weight of construction equipment and material resting on the bridge just above its weakest point at the time of the collapse. That load was estimated at 578,000 pounds (262,000 kilograms), consisting of sand, water, and vehicles.

This kind of in-depth case analysis is mandatory for accidents of all kinds: vehicular accidents, fires, building collapses, and so on. Crimes are investigated. Deaths in hospitals are investigated. What all such investigations share are retrospective case study methods that consider possible causes and, based on the preponderance of evidence, the most probable cause. Exhibit 5.12 (p. 296) describes the rigorous forensic case analyses of railroad switching operations fatalities. As noted in the exhibit, 55 railroad employees died in switching yard accidents from 2005 to 2010. To analyze the causes of these deaths, investigators from the railroad industry, labor unions, locomotive engineers, and federal regulators worked together to look for patterns of causes of the accidents. They carefully reviewed every case, coding a variety of variables related to conditions, contributing factors, and kinds of operations involved. They then looked for patterns in the qualitative case data and correlations in the quantitative cross-case data. They spent three to four hours coding each case and hours analyzing patterns across cases.

They found that fatalities happen for a reason. Accidents are not random occurrences, unfortunate events, or just plain bad luck. The risks to employees engaged in switching operations are real, ever present, and preventable. The data showed patterns in why switching fatalities occur. Findings about the causes of railroad accidents accumulated through this rigorous analysis over time and across cases identified five common behaviors and 10 common hazards that contribute to fatal accidents.

General Elimination Method and Modus Operandi Analysis

The General Elimination Methodology (GEM) approach for case analysis involves identifying rival alternative causal explanations, comparing each with the evidence, and eliminating those that don’t conform to the evidence, until that causal explanation remains that best fits the preponderance of the evidence (Scriven, 2008).

To take an example from work in which I have been involved, when looking at the effect of aid given by Heifer or Gates to extremely poor farmers in East Africa, after determining that a substantial improvement in welfare has followed the arrival of aid, and has been sustained for a few years, we check for the presence of more than a dozen other possible causes of this observed subsequent increase in welfare, including: efforts by the country’s government that have actually trickled down to the village level, analogous efforts by other philanthropies, self-help gains resulting from inspired leadership in the local communities, increased income from family members traveling to well-paid job openings elsewhere and remitting money back home, increased prices for milk or calves in the local markets, the beneficial results of a few years of good weather or of improved water supply, or of technology-driven improvements in the quality of available commercial feed, veterinary meds or services, or grass seed for improving pastures. This requires considerable systematic effort, but no sophisticated experimental design, no sophisticated statistics or risk analysis. (Scriven, 2008, p. 22)

Modus operandi (or method of operating, MO) analysis was also conceptualized by evaluation theorist Michael Scriven (1976) as a way of inferring causality when experimental designs are impractical or inappropriate. The MO approach, drawing from forensic science, makes the inquirer a detective. Detectives match clues discovered at a crime scene with known patterns of possible suspects. Those suspects whose MO does not fit the crime scene pattern are eliminated from further investigation. Translated to research and evaluation, the inquirer/detective observes some pattern and makes a list of possible causes. Evidence from the inquiry is matched with the list of suspects (possible causes). Those possible causes that do not fit the pattern of evidence can be eliminated from further consideration. Following the autopsy-like logic of Occam’s razor, as each possible cause is matched with the evidence, that cause supported by the preponderance of evidence and offering the simplest causal inference among competing possibilities is chosen as most likely.

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 17/37

This is also known as inference to the best explanation: If, based on the facts of the case and cumulative evidence, including interpretation, knowledge, and explanatory theory brought to the analysis, it is possible to identify one explanation as better than the others, then inferences based on that explanation will be warranted as the best explanation.

Forensic case analysis findings are accepted as evidence in courts of law for both criminal and civil proceedings. Clearly, such analyses and their use throughout the world demonstrate that thorough, systematic, and independent case analysis can identify causes at a reasonable level of confidence (preponderance of evidence) for individual cases. Cross-case analyses are used to generate regulations, policies, and operational procedures in fields as varied as transportation, hospitals, utilities, and construction, to name but a few examples.

Permission is included in documents already submitted for Claudius Ceccon (Brazil).

SOURCE: Brazilian cartoonist Claudius Ceccon. Used with permission.

SIDEBAR

GENERAL ELIMINATION CASE STUDY METHOD EXAMPLE: EVALUATING AN ADVOCACY CAMPAIGN AIMED AT INFLUENCING A SUPREME COURT DECISION

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 18/37

Several foundations funded a campaign aimed at influencing a Supreme Court decision. The collaboration of foundations committed more than $2 million to a focused advocacy effort within a window of nine months to potentially influence the Court. The evaluation case study examined the following question: To what extent, if at all, did the final-push campaign influence the Supreme Court’s decision?

The method we used in evaluating the Supreme Court advocacy campaign is what Scriven (2008) has called General Elimination Methodology, or GEM. It is a kind of “inverse epidemiological method.” Epidemiology begins with an effect and searches for its cause. In this application of GEM, we had both an effect (the Supreme Court decision in favor of the campaign position) and an intervention (the advocacy campaign), and we were searching for connections between the two. In doing so, we conducted a retrospective case study. Using evidence gathered through fieldwork—interviews, document analysis, detailed review of the Court arguments and decision, news analysis, and the documentation of the campaign itself—we aimed to eliminate alternative or rival explanations until the most compelling explanation, supported by the evidence, remained. This is also called the forensic method or MO (modus operandi, or method of operating) approach. Scriven brought the concept into evaluation from detective work, in which a criminal’s MO is established as a “signature trace” that connects the same criminal to different crimes (Davidson, 2005, p. 75). The modus operandi method works well in tracing the effects of interventions that have highly distinctive patterns of effects.

The evidence brought to bear in the evaluation of the judicial advocacy campaign was organized and presented as an in-depth case study of the campaign in four sections: (1) the litigation work; (2) the coordinated, targeted state organizing campaigns; (3) the communications and public education strategies; and (4) the overall coalition coordination. The case study involved detailed examination of campaign documents and interviews with 45 people directly involved in and knowledgeable about the campaign and/or the case, including the attorneys who argued both sides of the case before the Supreme Court. Several key people were interviewed more than once. The case also involved examining and analyzing hundreds of documents, including legal briefs, the Court’s opinions, more than 30 other court documents, more than 20 scholarly publications and books about the Supreme Court, media reports on the case, and confidential campaign files and documents, including three binders of media clips from campaign files. The case also drew on reports and documents describing related cases, legislative activity, and policy issues. Group discussions with key campaign strategists and advocates were especially helpful in clarifying important issues in the case.

Given the multifaceted and omnibus nature of the total campaign, a particular value of constructing this kind of in-depth case study is that none of the informants completely knew the full story. And, of course, different informants about the same events and processes had varying perspectives about what occurred and what it meant. A case study, then, involves ongoing comparative analysis—the sorting out, comparing, and reporting of different perspectives.

The full case did not emerge all at once. Indeed, it took time, including follow-up interviews, rereading documents, and continuous fact checking, for the full story to emerge. In a retrospective case study of this kind, we are often talking to people about events that they have “moved beyond” in their busy lives. Documentation is useful in returning to the past, but the critical judgments and perceptions stored in the memories of key players often take time and care to reignite. Developing relationships with key players was critical to this process.

Evaluation Conclusion

Based on a thorough review of the campaign’s activities, interviews with key informants and key knowledgeables, and careful analysis of the Supreme Court decision, we conclude that the coordinated final-push campaign contributed significantly to the Court’s decision.

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 19/37

See Exhibit 8.24 (p. 614) in Module 73 for a graphic depiction of the finding about the factors that contributed to the campaign’s success.

SOURCES: Patton (2008) and Patton and Sherwood (2007).

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 20/37

MODULE

72 New Analysis Directions: Contribution Analysis, Participatory Analysis, and Qualitative Counterfactuals

Look carefully for words and phrases that indicate attributes and various kinds of causal or conditional relations . . . .

Causal relations: “because” and its variants, ’cause, ’cuz, as a result, since, and the like. For example, “Y’know, we always take [highway] 197 there ’cuz it avoids all that traffic at the mall.” But notice the use of the word “since” in the following: “Since he got married, it’s like he forgot his friends.” Text analysis that involves the search for linguistic connectors like these requires very strong skills in the language of the text because you have to be able to pick out very subtle differences in usage.

—Bernard and Ryan (2010, p. 60) Analyzing Qualitative Data

Contribution Analysis One qualitative critique of traditional causal attribution approaches is that the language and concepts are overly deterministic. The word cause connotes singular, direct, and linear actions leading to clear, precise, and verifiable results: X caused Y. But such a direct, singular, linear causation is rare in the complex and dynamic interactions of human beings. More often, there are multiple causal influences and multiple outcomes.

Reframing Explanation From Cause and Effect to Influences and Interactions

Contribution analysis (Mayne, 2007, 2011, 2012; Patton, 2012b) was developed as an approach in program evaluation to examine a causal hypothesis (theory of change) against logic and evidence to examine what factors could explain the findings. Attribution questions are different from contribution questions, as follows.

Traditional evaluation causality questions (attribution)

• Has the program caused the outcome? • To what extent has the program caused the outcome? • How much of the outcome is caused by the program?

Contribution questions

• Has the program made a difference? That is, has the program made an important contribution to the observed result? Has the program influenced the observed result?

• How much of a difference has the program made? How much of a contribution?

The result of a contribution analysis is not definitive proof that the intervention or program has made an important contribution but rather evidence and argumentation from which it is reasonable to draw conclusions about the degree and importance of the contribution, within some level of confidence. The aim is to get

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 21/37

plausible association based on a preponderance of evidence, as in the judicial and forensic traditions. The question is whether a reasonable person would agree from the evidence and argument that the program has made an important contribution to the observed result. Contribution analysis can be used in impact evaluations for interventions in complex development situations with multiple actors (Stern et al., 2012).

A contribution analysis produces a contribution story that presents the evidence and other influences on program outcomes. A major part of that story may tell about behavioral changes that intended beneficiaries have made as a result of the intervention.

Attributes of a Credible Contribution Story

A credible statement of contribution would entail the following:

• A well-articulated context of the program, discussing other influencing factors • A plausible theory of change (no obvious flaws) that is not disproven • A description of implemented activities and resulting outputs of the program • A description of the observed results • The results of contribution analysis

The evidence in support of the assumptions behind the key links in the theory of change Discussion of the roles of the other influencing factors

• A discussion of the quality of the evidence provided, noting weaknesses

Contribution analysis focuses on identifying likely influences. Such causes, which on their own are neither necessary nor sufficient, represent the kind of contribution role that many interventions play. Contribution analysis, like detective work, requires connecting the dots between what was done and what resulted, examining a multitude of interacting variables and factors, and considering alternative explanations and hypotheses, so that in the end, we can reach an independent, reasonable, and evidence-based judgment based on the cumulative evidence. That is what we did in evaluating the judicial advocacy campaign featured earlier in a sidebar (see p. 595). From a contribution perspective, the question became how much influence the campaign appeared to have had rather than whether the campaign directly and singularly produced the observed results.

Outcome mapping (IDRC, 2010) and outcome harvesting (Wilson-Grau & Britt, 2012) are well-developed frameworks that use contribution analysis for evaluating outcomes in complex dynamic systems characterized by multiple influences, multiple outcomes, and multiple interrelationships.

Collaborative and Participatory Causal Analyses Collaborative and participatory approaches to qualitative inquiry include working with nonresearchers and nonevaluators not only in collecting data but also in analyzing data. When dealing with making judgments about the extent to which the preponderance of evidence supports certain causal conclusions, or what contributory factors explain the results, the people in the setting studied can serve as the equivalent of an inquiry jury, rendering their own interpretations and judgments about causality. In a major study of “the difficult methodological and theoretical challenges faced by those who wish to evaluate the impacts of international development policies,” aimed at “broadening the range of designs and methods for impact evaluations,” participatory approaches were highlighted as an important design and analysis option.

Impact Evaluation (IE) aims to demonstrate that development programmes lead to development results, that the intervention as cause has an effect. . . . On the basis of literature and practice, a basic classification of potential

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 22/37

designs is outlined [with] . . . five design approaches identified—Experimental, Statistical, Theory-based, Case- based and Participatory. (Stern et al., 2012, pp. i–ii)

Participatory approaches to causal inference do not see recipients of aid as passive recipients but rather as active “agents.” Within this understanding, beneficiaries have “agency” and can help “cause” successful outcomes by their own actions and decisions. (Stern et al., 2012, p. 29)

Chapter 4 discussed collaborative and participatory approaches at some length, including Exhibit 4.13: Principles of Fully Participatory and Genuinely Collaborative Inquiry (page 222). Participatory approaches require special facilitation skills to help those involved adopt analytical thinking. Some of the challenges include the following:

• Deciding how much involvement nonresearchers will have, for example, whether they will simply react and respond to the researcher’s analysis or whether they will be involved in the generative phase of analysis (Determining this can be a shared decision. “In participatory research, participants make decisions rather than function as passive subjects” [Reinharz, 1992, p. 185].)

• Creating an environment in which those collaborating feel that their perspective is genuinely valued and respected

• Demystifying research • Combining training in how to do analysis with the actual work of analysis • Managing the difficult mechanics of the process, especially where several people are involved • Developing processes for dealing with conflicts in interpretations (e.g., agreeing to report multiple

interpretations) • Determining how to maintain confidentiality with multiple analysts

A good example of these challenges concerns how to help lay analysts deal with counterintuitive findings and counterfactuals—that is, data that don’t fit primary patterns, negative cases, and data that oppose primary preconceptions or predilections. Morris (2000) found that shared learning, especially the capacity to deal with counterfactuals, was reduced when participants feared judgment by others, especially those in positions of authority.

In analyzing hundreds of open-ended interviews with parents who had participated in early-childhood parent education programs throughout the state of Minnesota, I facilitated a process of analysis that involved some 40 program staff. The staff worked in groups of two and three, each analyzing 10 pre and post paired interviews at a time. No staff analyzed interviews with parents from their own programs. The analysis included coding interviews with a framework developed at the beginning of the study as well as inductive, generative coding in which the staff could create their own categories. Following the coding, new and larger groups engaged in interpreting the results and extracting central conclusions. Everyone worked together in a large center for three days. I moved among the groups, helping resolve problems. Not only did we get the data coded, but also the process, as is intended in collaborative and participatory research processes, proved to be an enormously stimulating and provocative learning experience for the staff participants. The process forced them to engage deeply with parents’ perceptions and feedback, as well as to engage with each other’s reactions, biases, and interpretations. In that regard, the process also facilitated communication among diverse staff members from across the state, another intended outcome of the collaborative analysis process. Finally, the process saved thousands of dollars in research and evaluation costs, while making a staff and program development contribution. The results were intended primarily for internal program improvement use. As would be expected in such a nonresearcher analysis process, external stakeholders placed less value on the results than did those who participated in the process (Mueller, 1996; Mueller & Fitzpatrick, 1998; Program Evaluation Division, 2001). However, as participatory processes have become better facilitated and understood, external stakeholders are coming to appreciate and value them more.

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 23/37

Qualitative Counterfactuals A central issue in establishing causality is addressing the counterfactual: What would have happened without the cause or intervention? A counterfactual analysis looks at the case that is counter to the facts actually observed. In experimental designs, the control group constitutes the counterfactual, providing evidence of what would happen without the treatment received by the treatment group. Because qualitative designs do not assign people to treatment and control groups, critics of qualitative inquiry assert that causal claims cannot be made. Yet in many cases, it is neither feasible nor ethical to conduct experiments. One qualitative solution is to construct a hypothetical counterfactual case.

SIDEBAR

FRAMEWORK FOR CAUSAL ANALYSIS IN EVALUATION USING PROGRAM THEORY

A systematic approach to causal analysis for program theory evaluation consists of three components: (1) congruence, (2) comparisons, and (3) critical review.

1. Congruence with program theory. Do the observed and documented results match the program theory? 2. Counterfactual comparisons. What would have happened without the intervention? (See discussion of

qualitative and scenario-based counterfactuals, p. 599.) 3. Critical review. Are there plausible explanations for the results?

SOURCE: Funnell and Rogers (2011, pp. 473–499).

Assessing the plausible outcome of a combination of conditions that does not exist and instead must be imagined may seem esoteric. However, this analytic strategy has a long and distinguished tradition in the history of social science. A causal combination that lacks empirical instances and therefore must be imagined is a counterfactual case; evaluating its plausible outcome is counterfactual analysis.

To some, counterfactual analysis is central to case-oriented inquiry because such research typically embraces only a handful of empirical cases. If only a few instances exist (e.g., of social revolution), then researchers must compare empirical cases to hypothetical cases. The affinity between counterfactual analysis and case-oriented research, however, derives not simply from its focus on small Ns, but from its configurational nature. Case-oriented explanations of outcomes are often combinatorial in nature, stressing specific configurations of causal conditions. Counterfactual cases thus often differ from empirical cases by a single causal condition, thus creating a decisive, though partially imaginary, comparison. (Ragin, 2008, p. 150)

An example of a historical counterfactual case is Moore’s (1966) creation of an alternative history of the United States in which the South, rather than the North, won the U.S. Civil War. He used this counterfactual creation to support his theory that a “revolutionary break with the past” is an essential part of becoming a modern democracy.

Such counterfactuals constitute thought experiments. German social science pioneer Max Weber is commonly credited with being the first to advocate the use of thought experiments in social research to gain insight into causal relationships. Ragin (2008) offers an extensive discussion of counterfactuals in his configurational framework of qualitative comparative analysis, discussed earlier. In qualitative comparative analysis, counterfactual cases are created and used as substitutes for matched empirical cases when the real world offers no matching case. The hypothetical matched cases are identified by their configurations of causal conditions to illuminate the comparative analysis and causal patterns. This moves the qualitative analyst from

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 24/37

writing fieldwork-based empirical case studies to creating relevant comparison cases through thought experiments and alternative case study scenarios.

Scenario-Based Counterfactuals

One innovative and intriguing approach to constructing counterfactual comparisons is a collaborative and participatory process being applied in program evaluation. Scenario-based counterfactuals are negotiated alternative scenarios developed jointly with decision makers and stakeholders as part of a collaborative analysis process. The negotiated and constructed scenario-based counterfactual specifies reasonable projected alternative processes and any differences that would have occurred under the negotiated alternative scenario, for example, in timing or scale. This amounts to creating a plausible alternative hypothesis and examining the degree of plausibility.

• A community adopts an energy conservation plan built around a collaboration of churches, businesses, schools, nonprofits, and government units. How important was the collaboration to the adoption and implementation of the plan? Knowledgeable key informants construct the alternative scenario as a fabricated case study—no collaboration, business as usual—as a counterfactual.

• Exhibit 7.20 (pp. 511–516) is a case study of Thmaris, a homeless youth who received services and describes the difference it had made to him. In the case, he asserts that he would have landed in jail given the path he was on before he received services. A full case study alternative scenario could be constructed to fill in the details of a possible counterfactual. The credibility and validity of the counterfactual scenario are negotiated in a participatory process with key stakeholders.

The negotiated scenario-based counterfactual is based on a constructed alternative to what was implemented. The negotiated alternative scenario is a plausible (to decision makers and stakeholders) alternative; it is feasible in the sense that there are no budgetary, timing, or technical reasons why it could not have occurred, and it is legal in the sense that the alternative represents one of the options available within current law or plausible changes to relevant law (Rowe, Colby, Hall, & Niemeyer, in press).

Overview and Summary: Causal Explanation Thorough Qualitative Analysis This module has examined a variety of ways of approaching casual explanation in qualitative analysis. Exhibit 8.19 summarizes these different approaches.

I opened this module on causal analysis with observations from two eminent philosophers of science. First, Tom Schwandt (2001), University of Illinois, warned that “exactly how to define a causal relationship is one of the most difficult topics in epistemology and the philosophy of science. There is little agreement on how to establish causation” (p. 23). Then, Michael Scriven (2008), Claremont Graduate School, observed that “the causal wars are still raging, and the amount of collateral damage is increasing” (p. 11); he was referring to the gold standard debate that pits RCTs against all the alternatives. He concluded, as I do, that “there is absolutely nothing imperative, and nothing in general superior, about . . . RCT designs” (p. 23).

The overall conclusion I reach is that developments in qualitative methods and analysis over the past two decades have demonstrated that qualitative analysis can yield causal explanations rigorously and credibly.

That said, I close this module with a cautionary tale and a reminder that some question the whole notion of simple linear causality, doubting both its accuracy and its utility.

EXHIBIT 8.19 Twelve Approaches to Qualitative Causal Analysis

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 25/37

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 26/37

From Linear Causality to Complex Interrelationships

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 27/37

The law of causality, I believe, like much that passes muster among philosophers, is a relic of a bygone age, surviving, like the monarchy, only because it is erroneously supposed to do no harm.

—Philosopher Bertrand Russell (1872–1970) Selected Papers

Simple causal explanations are alluring, even seductive. We seem unable to escape simple linear modeling. We fall back on the linear assumptions of much of quantitative analysis and specify isolated independent and dependent variables that are mechanically linked together out of context. In contrast, the challenge of qualitative inquiry involves portraying a holistic picture of what the phenomenon, setting, or program is like and struggling to understand the fundamental nature of a particular set of activities and people in relationship in a specific context. “Particularization is an important aim, coming to know the particularity of the case,” as qualitative case study expert Bob Stake (1995) admonishes us to remember (p. 39). Simple statements of linear relationships may be more distorting than illuminating. The ongoing challenge of qualitative analysis is moving between the phenomenon of interest constantly moving between the phenomenon of interest and our abstractions of that phenomenon, between the descriptions of what has occurred and our interpretations of those descriptions, between the complexity of reality and our simplifications of those complexities, between the circularities and interdependencies of human activity and our need for linear, ordered statements of cause and effect.

Distinguished social scientist Gregory Bateson traced at least part of the source of our struggle to the ways in which we have been taught to think about things. We are told that a “noun” is the “name of a person, place, or thing.” We are told that a “verb” is an “action word.” These kinds of definitions, Bateson (1977) argues, were the beginning of teaching us that “the way to define something is by what it supposedly is in itself—not by its relations to other things.”

Today all that should be changed. Children could be told a noun is a word having a certain relationship to a predicate. A verb has a certain relationship to a noun, its subject, and so on. Relationship could now be used as a basis for definition, and any child could then see that there is something wrong with the sentence “‘Go’ is a verb.” . . . We could have been told something about the pattern which connects: that all communication necessitates context, and that without context there is no meaning. (p. 13)

Without belaboring this point about the difference between linear causal analysis (x causes y) and a holistic perspective that describes the interdependence and interrelatedness of complex phenomena, I would simply offer the reader a Sufi story. I suggest trying to analyze the data represented by the story in two ways. First, try to isolate specific variables that are important in the story, deciding which are the independent variables and which the dependent variable, and then write a statement of the form “These things caused this thing.” Then, read the story again. For the second analysis, try to distinguish among and label the different meanings of the situation expressed by the characters. Then, write a statement of the form “These things and these things came together to create ______.” Don’t try to decide that one approach is right and the other is wrong; simply try to experience and understand the two approaches.

Walking one evening along a deserted road, Mulla Nasrudin saw a troop of horsemen coming towards him. His imagination started to work; he imagined himself captured and sold as a slave, robbed by the oncoming horsemen, or conscripted into the army. Fearing for his safety, Nasrudin bolted, climbed a wall into a graveyard, and lay down in an open tomb.

Puzzled at this strange behavior, the men—honest travelers—pursued Nasrudin to see if they could help him. They found him stretched out in the grave, tense and quivering.

“What are you doing in that grave? We saw you run away and see that you are in a state of great anxiety and fear. Can we help you?”

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 28/37

Seeing the men up close, Nasrudin realized that they were honest travelers who were genuinely interested in his welfare. He didn’t want to offend them or embarrass himself by telling them how he had misperceived them, so Nasrudin simply sat up in the grave and said, “You ask what I’m doing in this grave. If you must know, I can tell you only this: I am here because of you, and you are here because of me.” (Adapted from Shah, 1972, p. 16)

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 29/37

MODULE

73 Writing Up and Reporting Findings, Including Using Visuals

At one time, one blade of grass is as effective as a sixteen-foot golden statue of Buddha. At another time, a sixteen-foot golden statute of Buddha is as effective as a blade of grass.

—Zen master Wumen Huikai (1183–1260)

Some reports are thin as a blade of grass; others feel 16 feet thick. Size, of course, is not the issue. Quality is. But given the volume of data involved in qualitative inquiry and the challenges of data reduction already discussed, reporting qualitative findings is the final step in data reduction, and size is a real constraint, especially when writing in forms other than research monographs and book-length studies, like journal articles and newsletter summaries. Each step in completing a qualitative project presents quality challenges, but the final step is completing a report so that others can know what you’ve learned and how you learned it. This means finding and writing the story that has emerged from your analysis. It also means dealing with what Lofland (1971) called the “the agony of omitting”—deciding what material to leave out of the story.

It can happen that an overall structure that organizes a great deal of material happens also to leave out some of one’s most favorite material and small pieces of analysis. . . . Unless one decides to write a relatively disconnected report, he must face the hard truth that no overall analytic structure is likely to encompass every small piece of analysis and all the empirical material that one has on hand. . . .

The underlying philosophical point, perhaps, is that everything is related to everything else in a flowing, even organic fashion, making coherence and organization a difficult and problematic human task. But in order to have any kind of understanding, we humans require that some sort of order be imposed upon that flux. No order fits perfectly. All order is provisional and partial. Nonetheless, understanding requires order, provisional and partial as it may be. It is with that philosophical view that one can hopefully bring himself to accept the fact that he cannot write about everything that he has seen (or analyzed) and still write something with overall coherence or overall structure. (p. 123)

Purpose Guides Writing and Reporting This chapter opened with the reminder that purpose guides analysis. Purpose also guides report writing and dissemination of findings. The key to all writing starts with (a) knowing your audience and (b) knowing what you want to say to them—a form of strategic communication (Weiss, 2001).

Dissertations have their own formats and requirements (Biklen & Casella, 2007; Bloomberg & Volpe, 2012; Heer & Anderson, 2005; Piantanida & Garman, 2009). Scholarly journals in various disciplines and applied research fields have their own standards and norms for what they publish. The best way to learn them is to read and study them, and study specialized qualitative methods journals like Qualitative Inquiry, Qualitative Research, Field Methods, Symbolic Interaction, Journal of Contemporary Ethnography, Grounded Theory Review, Qualitative Health Research, and American Journal of Evaluation. The format for evaluation reports is a matter of negotiation and is usually specified in the contract that commissions the evaluation. In all of these cases, the guiding principles remain (a) know your audience and (b) know what you want to say to them. (For in-depth guidance on writing the results of qualitative analysis, see Goodall, 2008, Writing Qualitative Inquiry; Holliday, 2007, Doing and Writing Qualitative Research; Wolcott, 2009, Writing Up Qualitative Research.)

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 30/37

Reflexivity and Voice

I do not put that note of spontaneity that my critics like into anything but the fifth draft. —Economist John Kenneth Galbraith (1986) (In the interview “The Art of Good Writing”)

In Chapter 2, when presenting the major strategic themes of qualitative inquiry, I included as one of the 12 primary themes that of “Voice, Perspective, and Reflexivity.”

The qualitative analyst owns and is reflective about her or his own voice and perspective; a credible voice conveys authenticity and trustworthiness; the inquirer’s focus becomes balance—understanding and depicting the world authentically in all its complexity while being self-analytical, politically aware, and reflexive in consciousness. This reiterates that the qualitative inquirer is the instrument of inquiry. (See Exhibit 2.1, pp. 46–47.)

SIDEBAR

PRESENTATIONS AND FEEDBACK BEFORE A FINAL REPORT OR PUBLICATION

You may be called on to present your findings to colleagues or at seminars or conferences before you’ve completed a final report or formally written up your results for publication. Such occasions can be helpful in testing out your conclusions and getting feedback about how they are received. Indeed, it is wise to seek such opportunities to help you get outside your own perspective and find out what questions your analysis raises in the minds of others. You can use what you learn to focus and fine-tune your report.

Program Evaluation Feedback

A different but related challenge arises in evaluation when, as is typical, intended users (especially program staff and administrators) want preliminary feedback while fieldwork is still under way or as soon as data collection is over. Providing preliminary feedback provides an opportunity to reaffirm with intended users the final focus of the analysis and nurture their interest in findings. Academic social scientists have a tendency to want to withhold their findings until they have polished their presentation. Use of evaluation findings, however, does not necessarily center on the final report, which should be viewed as one element in a total utilization process, sometimes a minor element, especially in formative and developmental evaluation (Patton, 2012a; Torres et al., 1996).

Evaluators who prefer to work diligently in the solitude of their offices until they can spring a final report on a waiting world may find that the world has passed them by. Feedback can inform ongoing thinking about a program instead of serving only as a one-shot information input for a single decision point. However, sessions devoted to reestablishing the focus of the evaluation analysis and providing initial feedback need to be handled with care. The evaluator will need to explain that analysis of qualitative data involves a painstaking process requiring long hours of careful work, going over notes, organizing data, looking for patterns, checking emergent patterns against the data, cross-validating data sources and findings, and making linkages among the various parts of the data and the emergent dimensions of the analysis. Thus, any early discussion of findings can only be preliminary, directed at the most general issues and the most striking, obvious results. If, in the course of conducting the more detailed and complete analysis of the data, the evaluator finds that statements made or feedback given during a preliminary session were inaccurate, evaluation users should be informed about the discrepancy at once.

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 31/37

Analysis and reporting are where reflexivity comes to the fore. As discussed in Chapter 2, the term reflexivity has entered the qualitative lexicon as a way of emphasizing the importance of deep introspection, political consciousness, cultural awareness, and ownership of one’s perspective. Reflexivity calls on us to think about how we think and inquire into our thinking patterns even as we apply thinking to making sense of the patterns we observe around us. Being reflexive involves self-questioning, self-understanding, and

interpretation of interpretation and the launching of a critical self-exploration of one’s own interpretations . . . , a consideration of the perceptual, cognitive, theoretical, linguistic, (inter)textual, political and cultural circumstances that form the backdrop to—as well as impregnate–the interpretations. (Alvesson & Sköldberg, 2009, p. 9)

Reflexivity reminds the qualitative inquirer to be attentive to and conscious of the cultural, political, social, linguistic, and economic origins of one’s own perspective and voice as well as the perspective and voices of those one interviews and those to whom one reports. To be reflexive, then, is to undertake an ongoing examination of what I know and how I know it.

So I repeat—analysis and reporting are where reflexivity comes to the fore. Throughout analysis and reporting, as indeed throughout all of qualitative inquiry, questions of reflexivity and voice must be asked as part of the process of engaging the data and extracting findings. Triangulated reflexive inquiry involves three sets of questions (see Exhibit 2.1 in Chapter 2, pp. 46–47.):

1. The Self-Reflexivity question What do I know? How do I know what I know? What shapes and has shaped my perspective? How have my perceptions and my background affected the data I have collected and my analysis of those data? How do I perceive those I have studied? With what voice do I share my perspective? What do I do with what I found? These questions challenge the researcher to also be a learner, to reflect on our “personal epistemologies”—the ways we understand knowledge and the construction of knowledge (Rossman & Rallis, 1998, p. 25).

2. Reflexive questions about those studied How do those studied know what they know? What shapes and has shaped their worldview? How do they perceive me, the inquirer? Why? How do I know?

3. Reflexivity about the audience How do those who receive my findings make sense of what I give them? What perspectives do they bring to the findings I offer? How to they perceive me? How do I perceive them? How do these perceptions affect what I report and how I report it?

SIDEBAR

REFLEXIVITY WHEN RESEARCHING TRAUMA

Connolly and Reilly (2007) examined the impact of conducting narrative research focused on trauma and healing. They recounted what they found through three voices: (1) the study participants, who experienced the trauma; (2) the researchers, who shared their personal experiences of conducting the inquiry; (3) and “an academic colleague who acted as a reflective echo making sense of and normalizing the researcher’s experience” (p. 522). Issues that emerged included

harmonic resonance between the story of the participant and the life experiences of the researcher; emotional reflexivity; complex researcher roles and identities; acts of reciprocity that redress the balance of power in the research relationship; the need for compassion for the participants; and self-care for the researcher when researching trauma. (p. 522)

Based on their reflexive process, the researchers concluded that when researching trauma, the researcher is a member of both a scholarly community and a human community and that maintaining the stance of a member of the human community is an essential element of conducting trauma research.

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 32/37

Reflexivity in the Methods Section of a Report

Because the qualitative inquirer is the instrument of qualitative inquiry and analysis, especially analysis, the methods section of a qualitative report should include some degree of reflexive discussion to acknowledge the perspective, skills, and experiences the inquirer has brought to the work. Such reflexivity is rare to nonexistent in quantitative reports, but qualitative reporting brings a different voice and perspective to the work. The methods section should reflect that difference. That’s also why qualitative reports more often use the first- person voice. (For a discussion on using the first-person, active voice in reporting versus the third-person, passive voice, see the section on reflexivity meets voice [pp. 70–74 in Chapter 2].)

Self-awareness, even a certain degree of self-analysis, has become a requirement of qualitative inquiry. As the reflexive questions above suggest, attention to voice applies not only to intentionality about the voice of the analyst but also to intentionality and consciousness about whose voices and what messages are represented in the stories and interviews we report. Qualitative data “can be used to relay dominant voices or can be appropriated to ‘give voice’ to otherwise silenced groups and individuals” (Coffey & Atkinson, 1996, p. 78). Eminent qualitative sociologist Howard Becker (1967) posed this classically as the question “Whose side are we on?” Societies, cultures, organizations, programs, and families are stratified. Power, resources, and status are distributed differentially. How we sample in the field, and then sample again during analysis in deciding who and what to quote, involves decisions about whose voices will be heard.

Finally, as we report findings, we need to anticipate how what we report will be heard and understood. We need strategies for thinking about the nature of the reporter–audience interaction, for example, understanding how “six basic tendencies of human behavior come into play in generating a positive response: reciprocation, consistency, social validation, liking, authority, and scarcity” (Cialdini, 2001, p. 76). Some writers eschew this responsibility, claiming that they write only for themselves. But researchers and evaluators have larger social responsibilities to present their findings for peer review and, in the cases of applied research, evaluation, and action research, to present their findings in ways that are understandable and useful.

Triangulated reflexive inquiry provides a framework for sorting through these issues during analysis and report writing—and then including in the methods section of your report how these reflections informed your findings. (For examples of qualitative writings centered on illuminating issues of reflexivity and voice, see Hertz, 1997.) We turn now to the content of writing and reporting.

Balancing Description and Interpretation One of the major decisions that has to be made in reporting is how much description to include. Rich, detailed description and direct quotations constitute the foundation of qualitative inquiry. Sufficient description and direct quotations should be included to allow the reader to enter into the situation observed and the thoughts of the people represented in the report. Description should stop short, however, of becoming trivial and mundane. The reader does not have to know everything that was done or said. Focus comes from having determined what’s substantively significant (see p. 572) and providing enough detail and evidence to illuminate and make that substantive case.

Yet the description must not be so “thin” as to remove context or meaning. Qualitative analysis, remember, is grounded in “thick description”:

A thick description does more than record what a person is doing. It goes beyond mere fact and surface appearances. It presents detail, context, emotion, and the webs of social relationships that join persons to one another. Thick description evokes emotionality and self-feelings. It inserts history into experience. It establishes the significance of an experience, or the sequence of events, for the person or persons in question. In thick description, the voices, feelings, actions, and meanings of interacting individuals are heard. (Denzin, 1989c, p. 83)

From Thick Description to Thick Interpretation

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 33/37

Thick description sets up and makes possible thick interpretation. By “thick interpretation,” Denzin (1989c) means, in part, connecting individual cases to larger public issues and to the programs that serve as the linkage between individual troubles and public concerns: “The perspectives and experiences of those persons who are served by applied programs must be grasped, interpreted, and understood if solid, effective, applied programs are to be put into place” (p. 105).

Description precedes and is then balanced by analysis and interpretation. Endless description becomes its own muddle. The purpose of analysis is to organize the description so that it is manageable. Description provides the skeletal frame for analysis that leads to interpretation. An interesting and readable report provides sufficient description to allow the reader to understand the basis for an interpretation and sufficient interpretation to allow the reader to appreciate the description.

Details of verification and validation processes (topics of the next chapter) are typically placed in a separate methods section of a report, but parenthetical remarks throughout the text about findings that have been validated can help readers value what they are reading. For example, if I describe some program process and then speculate on the relationship between that process and client outcomes, I may mention that (a) staff and clients agreed with this analysis when they read it, (b) I experienced this linkage personally as a participant- observer in the program, or (c) this connection was independently arrived at by two analysts looking at the data separately.

The report should help readers understand the different degrees of significance of various findings, if these exist. Since qualitative analysis lacks the parsimonious statistical significance tests of statistics, the qualitative analyst must make judgments that provide clues for the reader as to the writer’s belief about variations in the credibility and importance of different findings: When are patterns “clear”? When are they “strongly supported by the data”? When are the patterns “merely suggestive”? Readers will ultimately make their own decisions and judgments about these matters based on the evidence you’ve provided, but your analysis-based opinions and speculations deserve to be reported and are usually of interest to readers given that you’ve struggled with the data and know them better than anyone else.

Exhibit 8.35, at the end of this chapter (pp. 643–649), presents portions of a report describing the effects on participants of their experiences in the wilderness education program. The data come from in-depth, open- ended interviews. This excerpt illustrates the centrality of quotations in supporting and explaining thematic findings.

Communicating With Metaphors and Analogies

All perception of truth is the detection of an analogy. —Henry David Thoreau (1817–1862)

The museum study reported earlier in the discussion of analyst-generated typologies differentiated different kinds of visitors by using metaphors: the “commuter,” the “nomad,” the “cafeteria type,” and the “VIP.” In the dropout study, we relied on metaphors to depict the different roles we observed teachers playing in interacting with truants: the “cop,” the “old-fashioned schoolmaster,” and “the ostrich.” Language not only supports communication but also serves as a form of representation, shaping how we perceive the world (Chatterjee, 2001; Patton, 2000).

Metaphors and analogies can be powerful ways of connecting with readers of qualitative studies. But some analogies offend certain audiences, so they must be selected with some sensitivity to how those being described would feel and how intended audiences will respond. At a meeting of the Midwest Sociological Society, distinguished sociologist Morris Janowitz was asked to participate in a panel on the question “What is the cutting edge of sociology?” Janowitz (1979), having written extensively on the sociology of the military, took offense at the “cutting edge” metaphor. He explained,

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 34/37

Paul Fussell, the humanist, has prepared a powerful and brilliant sociological study of the literary works of the great wars of the 20th century which he entitled The Great War and Modern Memory. It is a work which all sociologists should read. His conclusion is that World War I and World War II, Korea and Vietnam have militarized our language. I agree and therefore do not like the question “Where is the cutting edge of sociology?” “Cutting Edge ”is a military term. I am put off by the very term cutting edge. Cutting edge, like the parallel term breakthrough, are slogans which intellectuals have inherited from the managers of violence. Even if they apply to the physical sciences, I do not believe that they apply to the social sciences, especially sociology, which grows by gradual accretion. (p. 591)

Of particular importance, in this regard, is avoiding metaphors with possible racist and sexist connotations, for instance, “It’s black and white.” One external reviewer of this book felt that this point deserves special emphasis, so I yield the floor, so to speak:

A lack of respect for people is conveyed in insensitive metaphors. So let’s avoid metaphors that are insensitive to regional differences, health and mental health differences, sexual orientation, and so on and so forth. No need to list them all (who could?), but at least make the comment about being respectful of people rather than seeming to be specific to just two types of insensitivity [racist and sexist insensitivities]. It is just too easy to inadvertently fall into habits of speech that can be hurtful.

At an Educational Evaluation and Public Policy Conference sponsored by the Far West Laboratory for Educational Research and Development, the women’s caucus expressed concern about the analogies used in evaluation and went on to suggest some alternatives:

To deal with diversity is to look for new metaphors. We need no new weapons of assessment—the violence has already been done! How about brooms to sweep away the attic-y cobwebs of our male/female stereotypes? The tests and assessment techniques we frequently use are full of them. How about knives, forks, and spoons to sample the feast of human diversity in all its richness and color. Where are the techniques that assess the delicious-ness of response variety, independence of thought, originality, uniqueness? (And lest you think those are female metaphors, let me do away with that myth—at our house everybody sweeps and everybody eats!) Our workgroup talked about another metaphor—the cafeteria line versus the smorgasbord banquet of styles of teaching/learning/assessing. Many new metaphors are needed as we seek clarity in our search for better ways of evaluating. To deal with diversity is to look for new metaphors. (Hurty, 1976, p. 1)

When employing a metaphor, it is important to make sure that it serves the data and not vice versa. Don’t manipulate the data to fit the metaphor. Moreover, because metaphors carry implicit connotations, it is important to make sure that the data fit the most prominent of those connotations so that what is communicated is what the analyst wants to communicate. Finally, one must avoid reifying metaphors and acting as if the world were really the way the metaphor suggests it is.

The metaphor is chiefly a tool for revealing special properties of an object or event. Frequently, theorists forget this and make their metaphors a real entity in the empirical world. It is legitimate, for example, to say that a social system is like an organism, but this does not mean that a social system is an organism. When metaphors, or concepts, are reified, they lose their explanatory value and become tautologies. A careful line must be followed in the use of metaphors, so that they remain a powerful means of illumination. (Denzin, 1978b, p. 46)

How “real” metaphors are may turn out to be a specific manifestation of Thomas’s theorem: What is perceived as real is real in its consequences. Brain scans are revealing that when we read a detailed description, an evocative metaphor, or an emotional story, the brain is stimulated. A team of brain researchers from Emory University found that

when subjects in their laboratory read a metaphor involving texture, the sensory cortex, responsible for perceiving texture through touch, became active. Metaphors like “The singer had a velvet voice” and “He had leathery hands” roused the sensory cortex, while phrases matched for meaning, like “The singer had a pleasing voice” and “He had strong hands,” did not. (Paul, 2012, p. SR6)

This invites research into the larger question of what happens to the brain when one is analyzing qualitative data or reading a full qualitative case study. Stay tuned.

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 35/37

Creating and Incorporating Visuals The raw data of qualitative inquiry take the form of words, narratives, recorded observations, documents, and stories. This chapter has discussed how these data are analyzed and interpreted through content, pattern, and theme analysis; constructing cases studies and cross-case analyses; creating typologies; and depicting causal connections and interrelationships. At the center of all these analytical approaches have been words. Now, we turn to those things that have legendarily and metaphorically been worth a thousand words: pictures, visuals, and graphics.

No trend is more pronounced in the past decade than the visualization of data and findings. The capability to create meaningful and powerful visuals is a skill that is likely to become increasingly important in our short-attention-span world. Visualization rules. But before we exalt its place in contemporary analysis and reporting, let’s pause and give a nod to the visualization pioneers who couldn’t just go on the Internet and download some impressive photos and graphics. The legendary nurse Florence Nightingale (1820–1910) gathered data over a period of a year showing deaths in the Crimean War (1855–1856) due to battle wounds compared with deaths due to infections and disease (a much larger number). She converted the data into a color-coded visual display that proved hugely influential in changing sanitation practices and paved the way for attention to preventing infections, which saved hundreds of thousands of lives (Magnello, 2012). In 1894, George Waring Jr. was appointed Street Commissioner of the City of New York and began a systematic sanitation program that cleared the streets of New York of shin-deep garbage and animal and human waste. He took and had published in the newspapers before and after photographs showing the visual difference his reforms brought (Waring, 1897; Wells, 2012). In 1896, New York City honored him with a parade of appreciation for his contributions to the city’s quality of life, which included draining the wetlands of Manhattan Island to create Central Park. That is the vision of visualization we build on.

The best way to present qualitative data visualization is not to talk about such visuals but to provide actual illustrations. Exhibit 3.13 (pp. 142–143) in Chapter 3 presented a theory-of-change baseline systems graphic depicting the situation at the beginning of a light rail construction project; the key system actors, structures, and processes; and the lack of integration among these subsystem elements. The purpose of that visual graphic was to provide an example of how qualitative inquiry can be used to depict a system and support the conceptualization and evaluation of system change. Take another look at that graphic from the perspective of visually depicting findings, in that case the results of focus groups with key knowledgeables.

In this section, I’ll present examples of visual displays of qualitative findings with a minimum of accompanying verbiage, inviting you instead to engage with each type of illustration as a way of stimulating your thinking about making visualizations a part of your qualitative reporting (see Exhibits 8.20–8.27).

SIDEBAR

PICTURING DISABILITIES

Qualitative sociologist Robert Bogdan (2012) has assembled, analyzed, and published more than 200 historical photographs of people with disabilities. Beginning in the 1860s, when photography was emerging as a commercial enterprise, up to the 1970s, when the disability rights movement forced change, he shows how people with disabilities were portrayed. In one photo, a young woman with no arms wears a sequined tutu and smiles for the camera as she holds a teacup with her toes. In another, a man holds up two prosthetic legs while his own legs are bared to the knees to show his missing feet. Such photos were used as promotional material for circus sideshows and charity drives and hung in art galleries. They were found on “begging cards” and in family albums.

Bogdan’s (2012) analysis includes an inquiry into the perspective, role, and values of the photographers who took these photos and the contexts within which such photographs were created and people with disabilities were exploited. He examines a wide range of purposes and uses of disability photographs,

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 36/37

from sideshow souvenirs to clinical photographs. The photographs are both data and visualization of findings.

EXHIBIT 8.20 Photos Before and After to Illustrate Change

Vietnam Helmet Law

• Road traffic injuries have long been a leading cause of death and disability in Vietnam. • 60% of fatalities occur in motorcycle riders and passengers. • Vietnam has had a partial motorcycle helmet legislation since 1995, However implementation and

enforcement had been limited. • On 15 December 2007, Vietnam first comprehensive mandatory helmet law came into effect, covering

all riders and passengers on all roads nationwide. Penalties increased ten-fold and cohorts of police were mobilized for enforcement.

Photos on street corners the day before the law took effect

Photos on the same street corners the day after the law took effect

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 37/37

Results

The Asia Injury Prevention Foundation reported: “Nearly 100% of Vietnam’s motorbike users left home wearing a helmet. It was an unbelievable sight with a near instantaneous effect. Major hospitals report the number of patients admitted for traumatic brain injuries in the two days after the law’s enactment was much lower than on previous weekends. In Ho Chi Minh City alone, serious traffic accident injuries fell by almost 50 percent compared with pre-helmet weekends.”

SOURCE: McDonnell, Tran, and McCoy (2010).