U8A1-68 - TUTOR PLEASE FOLLOW THE INSTRUCTIONS AS OUTLINED TO COMPLETE THIS HOMEWORK....Data Analysis Strategies...SEE ATTACHED FILES

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 1/39

These descriptions are progressive in that each new category identifies a person more serious about the exhibit hall.

• The commuter: A person who merely uses the hall as a vehicle to get from the entry point to the exit point. . . . • The nomad: A casual visitor, a person who is wandering through the hall, apparently open to become interested

in something. The Nomad is not really sure why he or she is in the hall and not really sure that s/he is going to find anything interesting in this particular exhibit hall. Occasionally the Nomad stops, but it does not appear that the nomadic visitor finds any one thing in the hall more interesting than any other thing.

• The cafeteria type: This is the interested visitor who wants to get interested in something, and so the entire museum and the hall itself are treated as a cafeteria. Thus, the person walks along, hoping to find something of interest, hoping to “put something on his or her tray” and stopping from time to time in the hall. While it appears that there is something in the hall that spontaneously sparks the person’s interest, we perceive this visitor has a predilection to becoming interested, and the exhibit provides the many things from which to choose.

• The V.I.P.—very interested person: This visitor comes into the hall with some prior interest in the content area. This person may not have come specifically to the hall, but once there, the hall serves to remind the V.I.P.’s that they were, in fact, interested in something in that hall beforehand. The V.I.P. goes through the hall much more carefully, much slower, much more critically—that is, they move from point to point, they stop, they examine aspects of the hall with a greater degree of scrutiny and care. (pp. 10–11)

This typology of types of visitors became important in the full evaluation because it permitted analysis of different kinds of museum experiences. Moreover, the evaluators recommended that when conducting interviews to get museum visitors’ reactions to exhibits, the interview results should be differentially valued depending on the type of person being interviewed—commuter, nomad, cafeteria type, or VIP.

A different typology was developed to distinguish how visitors learn in a museum, “Museum Encounters of the First, Second, and Third Kind,” a takeoff on the popular science fiction movie Close Encounters of the Third Kind, which referred to direct human contact with visitors from outer space.

• Museum encounters of the first kind: This encounter occurs in halls that use display cases as the primary approach to specimen presentation. Essentially, the visitor is a passive observer to the “objects of interest.” Interaction is visual and may occur only at the awareness level. The visitor is probably not provoked to think or consider ideas beyond the visual display.

• Museum encounters of the second kind: This encounter occurs in halls that employ a variety of approaches to engage the visitor’s attention and/or learning. The visitor has several choices to become active in his/her participation. . . . The visitor is likely to perceive, question, compare, hypothesize, etc.

• Museum encounters of the third kind: This encounter occurs in halls that invite high levels of visitor participation. Such an encounter invites the visitor to observe phenomena in process, to create, to question the experts, to contribute, etc. Interaction is personalized and within the control of the visitor. (Wolf & Tymitz, 1978, p. 39)

Here’s a sample of a quite different classification scheme, this one developed from fieldwork by sociologist Rob Rosenthal (1994) as “a map of the terrain” of the homeless.

• Skidders: Most often women, typically in their 30s, who grew up middle or upper class but “skidded” into homelessness as divorced or separated parents

• Street people: Mostly men, often veterans, rarely married; highly visible and know how to use the resources of the street

• Wingnuts: People with severe mental problems, occasionally due to long-term alcoholism, a visible subgroup (Note to readers: Including this example and the label “wingnuts” is not an endorsement of its insensitivity. The label is offensive. Labeling is treacherous and will appear especially inappropriate when removed from the context in which it was generated; in this case, the term was sometimes used among homeless people themselves.)

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 2/39

• Transitory workers: People with job skills and a history of full-time work who travel from town to town, staying months or years in a place and then heading off to greener pastures

EXHIBIT 8.10 Ten Types of Qualitative Analysis

These varying types of qualitative analysis are distinct, not mutually exclusive. An analysis can include, and typically does include, several approaches. It is worth distinguishing them because they involve different ways of approaching the challenge of making sense of qualitative data.

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 3/39

SIDEBAR

COMPLEMENTARY PAIRS AS CONCEPTUALLY SENSITIZING CONTINUA

Nobel laureate Niels Bohr’s maxim is as follows:

Contraria sunt complementa (“Contraries are complementary”)

Contraries are not contradictory: . . . We replace all related but slightly different terms like contraries, polar opposites, duals, opposing tensions, binary oppositions, dichotomies, and the like with the all-encompassing term “complementary pairs.” (Kelso & Engstrom, 2006, p. 7)

Sampling of Complementary Pairs From Various Field of Endeavor

Anatomy: organ/organism; form/function

Art: foreground/background; original/reproduction

Culture: permissible/taboo; public/private

Economics: boom/bust; equilibrium/disequilibrium

Education: knowledge/ignorance; student/teacher

Entertainment: amateur/professional; comedy/tragedy

Mathematics: problem/solution; finite/infinite

Medicine: curative/palliative; invasive/noninvasive; prevention/cure

Military: all/enemy; defensive/offensive; peace/war

Mythology: hero/villain; beauty/ugliness

Philosophy: faith/reason; physical/spiritual; truth/falsehood

Politics: conservative/liberal; rights/responsibilities

Psychology: abnormal/normal; extraversion/introversion

Sociology: folk/urban; general/particular; task oriented/process oriented; social/antisocial

SOURCE: Kelso and Engstrom (2006, pp. 257–262).

Categories of How Homeless People Spend Their Time:

• Hanging out • Getting by • Getting ahead

As these examples illustrate, the first purpose of typologies is to distinguish aspects of an observed pattern or phenomenon descriptively. Once identified and distinguished, these types can later be used to make interpretations, and they can be related to other observations to draw conclusions, but the first purpose is

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 4/39

description based on an inductive analysis of patterns that appear in the data. Kenneth Bailey’s (1994) classic on typologies and taxonomies in qualitative analysis remains an excellent resource.

Summary of Pattern, Theme, and Content Analysis Purpose drives analysis. Design frames analysis. Purposeful sampling strategies determine the unit of analysis. Different analytical approaches will yield different kinds of findings based on distinct analysis procedures and priorities. There is no single right way to engage in qualitative analysis. Distinguishing signal from noise (detecting patterns and identifying themes) results from immersion in the data, systematic engagement with what the data reveal, and judgment about what is meaningful and useful. The next module gets into some of the nitty-gritty operational processes involved in analysis. Exhibit 8.10 presents the 10 analytical approaches reviewed in this module.

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 5/39

MODULE

68 The Intellectual and Operational Work of Analysis

Classification is Ariadne’s clue through the labyrinth of nature. —George Sand (1869)

Nouvelles Lettres d’un Voyageur

Coding Data, Finding Patterns, Labeling Themes, and Developing Category Systems Thus far, I’ve provided lots of examples of the fruit of qualitative inquiry: patterns, themes, categories, and typologies. Let’s back up now to consider how you recognize patterns in qualitative data and turn those patterns into meaningful categories and themes. This chapter could have started with this section, but I think it’s helpful to understand what kinds of findings can be generated from qualitative analysis before delving very deeply into the mechanics and operational processes, especially because the mechanics vary greatly and are undertaken differently by analysts in different disciplines working from divergent frameworks. That said, some guidance can be offered.

Raw field notes and verbatim transcripts constitute the undigested complexity of reality. Simplifying and making sense out of that complexity constitutes the challenge of content analysis. Developing some manageable classification or coding scheme is the first step of analysis. Without classification, there is chaos and confusion. Content analysis, then, involves identifying, coding, categorizing, classifying, and labeling the primary patterns in the data. This essentially means analyzing the core content of interviews and observations to determine what’s significant. In explaining the process, I’ll describe it as done traditionally, which is without software, to highlight the thinking involved. Software programs provide different tools and formats for coding, but the principles of the analytical process are the same whether doing it manually or with the assistance of a computer program.

I begin by reading through all of my field notes or interviews and making comments in the margins or even attaching post-it notes that contain my notions about what I can do with the different parts of the data. This constitutes the first cut at organizing the data into topics and files. Coming up with topics is like constructing an index for a book or labels for a file system: You look at what is there and give it a name, a label. The copy on which these topics and labels are written becomes the indexed copy of the field notes or interviews. Exhibit 8.11 shows examples of codes from the field note margins of the evaluation of a wilderness education program. You create your own codes, or in a team situation, you create codes together.

EXHIBIT 8.11 First-Cut Coding Examples

P is for participants, S is for staff.

Sample codes from the field note margins of the evaluation of a wilderness education program

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 6/39

The shorthand codes (abbreviations) are written in the margins directly on the relevant data passages or quotations. The full labels in the second column of the above table are the designations for separate files that contain all similarly coded passages.

The shorthand codes (abbreviations) are written directly on the relevant data passages, either in the margins or with an attached tab on the relevant page. Many passages will illustrate more than one theme or pattern. The first reading through the data is aimed at developing the coding categories or classification system. Then a new reading is done to actually start the formal coding in a systematic way. Several readings of the data may be necessary before field notes or interviews can be completely indexed and coded. Some people find it helpful to use colored highlighting pens—color coding different idea or concepts. Using self-adhesive colored dots or post-it notes is another option.

If sensing a pattern or “occurrence” can be called seeing, then the encoding of it can be called seeing as. That is, you first make the observation that something important or notable is occurring, and then you classify or describe it. . . . The seeing as provides us with a link between a new or emergent pattern and any and all patterns that we have observed and considered previously. It also provides a link to any and all patterns that others have observed and considered previously through reading. (Boyatzis, 1998, p. 4)

Where more than one person is working on the analysis, it is helpful to have each person (or small teams for large projects) develop the coding scheme independently, then compare and discuss similarities and differences. Important insights can emerge from the different ways in which two people look at the same set of data—a form of analytical triangulation.

Often an elaborate classification system emerges during coding, particularly in large projects where a formal scheme must be developed that can be used by several trained coders. In our study of evaluation use, which is the basis for Utilization-Focused Evaluation (Patton, 2008), graduate students in the evaluation program at the University of Minnesota conducted lengthy interviews with 60 project officers, evaluators, and federal decision makers. We developed a comprehensive classification system that would provide easy access to the data by any of the student or faculty researchers. Had only one investigator been intending to use the

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 7/39

data, such an elaborate classification scheme would not have been necessary. However, to provide access to several students for different purposes, every paragraph in every interview was coded using a systematic and comprehensive coding scheme made up of 15 general categories with subcategories. Portions of the codebook used to code the utilization of evaluation data appear as Exhibit 8.34 at the end of this chapter (pp. 642–643), as an example of one kind of qualitative analysis codebook. This codebook was developed from four sources: (1) the standardized open-ended questions used in interviewing; (2) review of the utilization literature for ideas to be examined and hypotheses to be reviewed; (3) our initial inventory review of the interviews, in which two of us read all the data and added categories for coding; and (4) a few additional categories added during coding when passages didn’t fit well into the available categories.

Every interview was coded twice by two independent coders. Each individual code, including redundancies, was entered into our qualitative analysis database so that we could retrieve all passages (data) on any subject included in the classification scheme, with brief descriptions of the content of those passages. The analyst could then go directly to the full passages and complete interviews from which the passages were extracted to keep quotations in context. In addition, the computer analysis permitted easy cross-classification and cross-comparison of passages for more complex analyses across interviews.

Some such elaborate coding system is routine for very rigorous analysis of a large amount of data. Complex coding systems with multiple coders categorizing every paragraph in every interview constitutes a labor- intensive form of coding, one that would not be used for most small-scale formative evaluation or action research projects. However, where data are going to be used by several people or where data are going to be used over a long period of time, including additions to the data set over time, such a comprehensive and computerized system can be well worth the time and effort required. This is the case, for example, where an action research project involves a number of people working together in an organizational or community context where the stakes are high.

Classifying and coding qualitative data produces a framework for organizing and describing what has been collected during fieldwork. (For published examples of coding schemes, see Bernard, 2000, pp. 447–450; Bernard & Ryan, 2010, pp. 325–328, 387–389, 491–492, 624; Boyatzis, 1998; Miles, Huberman, & Saldaña, 2014; Strauss & Corbin, 1998.) This descriptive phase of analysis builds a foundation for the interpretative phase, when meanings are extracted from the data, comparisons are made, creative frameworks for interpretation are constructed, conclusions are drawn, significance is determined, and, in some cases, theory is generated.

Convergence and Divergence in Coding and Classifying In developing codes and categories, a qualitative analyst must first deal with the challenge of “convergence” (Guba, 1978)—figuring out what things fit together. Begin by looking for recurring regularities in the data. These regularities reveal patterns that can be sorted into categories. Categories should then be judged by two criteria: (1) internal homogeneity and (2) external heterogeneity. The first criterion concerns the extent to which the data that belong in a certain category hold together or “dovetail” in a meaningful way. The second criterion concerns the extent to which differences among categories are bold and clear. “The existence of a large number of unassignable or overlapping data items is good evidence of some basic fault in the category system” (Guba, 1978, p. 53). The analyst then works back and forth between the data and the classification system to verify the meaningfulness and accuracy of the categories and the placement of data in categories. If several different possible classification systems emerge or are developed, some priorities must be established to determine which are more important and illuminative. Prioritizing is done according to the utility, salience, credibility, uniqueness, heuristic value, and feasibility of the classification schemes. Finally, the category system or set of categories is tested for completeness.

SIDEBAR

ERRORS TO AVOID OR FIX WHEN FOUND

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 8/39

In a clever graphic comic titled The Good, the Bad, and the Data: Shane the Lone Ethnographer’s Basic Guide to Qualitative Data Analysis, Sally Galman (2013) identifies an alphabet soup of qualitative errors to be avoided or fixed. Here’s a sample:

A is for Anemia. As you code, you find that your data are terribly thin. Go back to the field to beef up your data.

J is for Jaws (the shark). Something is lurking under the surface—you are in denial of the disconfirming evidence, so you avoid it.

O is for “Oh no! I didn’t OBSERVE.” All you have are notes filled with interpretations rather than actual observations.

P is for Procrastination. Don’t let too much time pass before you analyze.

X is for Extra Stuff. You’ve completed your analysis, and you have all this extra data that do not seem to fit. Maybe it’s time to revisit your questions and design.

Z is for Zealotry. Did you make room for discovery or did you only confirm your own ideas? (pp. 82–85)

©2002 Michael Quinn Patton and Michael Cochran

1. The set should have internal and external plausibility, a property that might be termed “integratability.” Viewed internally, the individual categories should appear to be consistent; viewed externally, the set of categories should seem to comprise a whole picture. . . .

2. The set should be reasonably inclusive of the data and information that do exist. This feature is partly tested by the absence of unassignable cases, but can be further tested by reference to the problem that the inquirer is investigating or by the mandate given the evaluator by his client/sponsor. If the set of categories did not appear to be sufficient, on logical grounds, to cover the facets of the problem or mandate, the set is probably incomplete.

3. The set should be reproducible by another competent judge. . . . The second observer ought to be able to verify that a) the categories make sense in view of the data which are available, and b) the data have been appropriately arranged in the category system. . . . The category system auditor may be called upon to attest that the category system “fits” the data and that the data have been properly “fitted into” it.

4. The set should be credible to the persons who provided the information which the set is presumed to assimilate. . . . Who is in a better position to judge whether the categories appropriately reflect their issues and concerns than the people themselves? (Guba, 1978, pp. 56–57)

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 9/39

After analyzing for convergence, the mirror analytical strategy involves examining divergence. By this, Guba means that the analyst must “flesh out” the patterns or categories. This is done by the processes of extension (building on and going deeper into the patterns and themes already identified), bridging (making connections among different patterns and themes), and surfacing (proposing new categories that ought to fit and then verifying their existence in the data). The analyst brings closure to the process when sources of information have been exhausted, when sets of categories have been saturated so that new sources lead to redundancy, when clear regularities have emerged that feel integrated, and when the analysis begins to “overextend” beyond the boundaries of the issues and concerns guiding the it. Divergence also includes careful and thoughtful examination of data that do not seem to fit, including deviant cases that don’t fit the dominant identified patterns.

This sequence, convergence then divergence, should not be followed mechanically, linearly, or rigidly. The processes of qualitative analysis involve both technical and creative dimensions. As noted early in this chapter, no abstract processes of analysis, no matter how eloquently named and finely described, can substitute for the skill, knowledge, experience, creativity, diligence, and work of the qualitative analyst. “The task of converting field notes and observations about issues and concerns into systematic categories is a difficult one. No infallible procedure exists for performing it” (Guba, 1978, p. 53).

(For in-depth guidance on qualitative coding and analysis see Bernard & Ryan, 2010, Analyzing Qualitative Data: Systematic Approaches; Boeije, 2010, Analysis in Qualitative Research; Guest, MacQueen, & Namey, 2012, Applied Thematic Analysis; Northcutt & McCoy, 2004, Interactive Qualitative Analysis: A Systems Method for Qualitative Research; Saldaña, 2009, The Coding Manual for Qualitative Researchers.)

SIDEBAR

FEAR OF FINDING NOTHING

Students beginning dissertations often ask me, their anxiety palpable and understandable, “What if I don’t find out anything?” Bob Stake, of “responsive evaluation” and “case study” fame, said at his retirement, “Paraphrasing Milton: They also serve who leave the null hypothesis tenable. . . . It is a sophisticated researcher who beams with pride having, with thoroughness and diligence, found nothing there” (Stake, 1998, p. 364, with a nod to Michael Scriven for inspiration).

True enough. But in another sense, it’s not possible to find nothing there, at least not in qualitative inquiry. The case study is there. It may not have led to new insights or confirmed one’s predilections, but the description of that case at that time and in that place is there. That is much more than nothing. The interview responses and observations are there. They, too, may not have led to headline-grabbing insights or confirmed someone’s eminent theory, but the thoughts and reflections from those people at that time and in that place are there, recorded and reported. That is much more than “nothing.”

Halcolm will tell you this:

You can only find nothing if you stare at a vacuum. You can only find nothing if you immerse yourself in nothing. You can only find nothing if you go nowhere. Go to real places. Talk to real people. Observe real things. You will find something. Indeed, you will find much, for much is there. You will find the world.

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 10/39

MQP Rumination # 8

Make Qualitative Analysis First and Foremost Qualitative

I am offering one personal rumination per chapter. These are issues that have persistently engaged, sometimes annoyed, occasionally haunted, and often amused me over more than 40 years of research and evaluation practice. Here’s where I state my case on the issue and make my peace.

Here’s the scenario. I’ve conducted 15 key informant interviews with executive directors of nonprofit agencies that receive funds from the same philanthropic foundation. I’m presenting the results to the foundation’s senior staff and trustees. I report as follows:

Most of those I interviewed report being quite frustrated with your evaluation reporting requirements. They don’t think you’re asking the most important questions and they are dubious that anyone here is reading or using their reports. Most said that they get no feedback after submitting the required reports.

I then share three examples of direct quotes supporting this overall conclusion:

• “I do the reports because we’re required to, and we take them seriously and answer seriously. But there are important things we’d like to report and think they’d like to know that aren’t asked, and there’s no space for. That feels like a lost opportunity.”

• “Look, I’ve been at this for years. It’s very frustrating. We know it’s just a compliance thing. No one reads our reports. We do them because they’re required. That’s it. End of story.”

• “Truth be told, it’s a waste of time, a frustrating waste of time.”

I then invite questions, comments, and reactions.

The board chair asks, “How many said it was a waste of time?”

I take a deep breath, and bite my tongue (metaphorically) to stop myself from saying, “You have a problem here. Does it really matter whether it’s 7 people or 9 or 12? You have a problem! It’s not about the number. It’s about the substance. YOU HAVE A PROBLEM!”

The Allure of Precision

This scenario occurs over and over again. It’s the knee-jerk response to the ambiguities of qualitative findings: “Many said,” “some said,” “ a few said,” and so on. When presenting findings at a major international evaluation that involved 20 key informant interviews, the response from the conference chair was to dismiss the report as “evaluation by adjective.” He wanted to know how many said what? “What are the percentages?” he demanded.

I refused. I invite you to refuse. Here are 5 reasons why. (Count them. There are exactly 5 reasons. Now I could have generated 10 reasons or just offered my top 3. But I decided on 5. Elsewhere, I’ve offered lists of 10, 12, or 3, but 5 struck me as about right for a rumination. So that’s what you get: 5.)

1. Open-ended interviews generate diverse responses. That’s the purpose of an open-ended question, to find out what’s salient in the interviewees’ own words. We then group together those responses that manifest a common theme. The three quotes above all fall into a category of Feeling Frustrated. Only one person used the phrase “waste of time.” Another said, “I put it off as long as I can and do it

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 11/39

just in time to meet the deadline for submission, because I have a lot of more important things to do and it’s not a great use of my time. But I do it.” Not quite “waste of time,” but pretty close. What responses go together is a matter of interpretation and judgment. Coding, categorizing, and theme analysis are not precise. The result is qualitative. Stay qualitative.

2. The adjectives “most,” “many,” “some,” or “a few” are actually more accurate than a precise number. It’s common to have a couple of responses that could be included in the category or left out, thus changing the number. I don’t want to add a quote to a category just to increase the number. I want to add it because, in my judgment, it fits. So when I code 12 of 20 saying some version of “feeling frustrated,” I’m confident in reporting that “many” felt frustrated. It could have been 10, or it could have been 14, depending on the coding. But it definitely was many.

3. Percentages may be misleading. With a key informant sample of 20, each response is 5%. Thus, going from 12 of 20 to 14 of 20 is a jump from 60% to 70%. In a survey of 300 respondents, a 10% difference is significant. In a small purposeful sample, it’s not. Going from 12 to 14 is still “many.”

4. The “how many” question can distract from dealing with substantive significance. I regularly conduct workshops with 20 to 40 participants. The workshop sponsors usually have some standardized evaluation form that solicits ratings and then invites an overall open-ended response. Over the years, a single, particularly insightful and specific response has proved more valuable to me than a large number of general comments (e.g., “I learned a lot”). The point of qualitative analysis is not to determine how many said something. The point is to generate substantive insight into the phenomenon. One or two very insightful and substantive responses can easily trump 15 general responses. Here’s an example. I interviewed 15 participants in an employment training program. Two female participants said they were on the verge of dropping out because of sexual harassment by a staff member. That’s “only” 13%. That’s just 2 of 15. But any sexual harassment is unacceptable. The program has a problem, a potentially quite serious problem.

5. Small purposeful samples pose confidentiality challenges. When I’m reporting qualitative findings, I say in the methods section that I will not report that “all” or “no one” responded in a certain way because that would potentially break the confidentiality pledge. In the example that opened this rumination, all the 15 agency directors I interviewed complained about the foundation’s evaluation reporting process, especially the lack of feedback. But I reported that “many” complained, and refused attempts to get me to provide a number (which would have been 20 of 20), so as not to put any of the directors at risk.

Reasons Galore

So there you have 5 reasons to keep qualitative analysis qualitative. But maybe that doesn’t seem like enough. Maybe you’d be more persuaded and feel more confident if I gave you 10 reasons. No sooner asked, than done. Here are 5 more rumination-inspired reasons to keep qualitative analysis first and foremost qualitative:

6. Doing so demonstrates integrity. 7. It reinforces the message that the inquiry is qualitative. 8. It requires people to think about meanings. 9. Numbers are easily manipulated and analysis is corrupted under pressure to increase the number.

(Hmmm, is that 2 reasons or just 1?) 10. Meaning is essentially qualitative and about qualities.

And a bonus item: Generating numbers is not the purpose of qualitative inquiry. If someone wants precise numbers, tell them to do a survey and ask closed questions and count the responses. That’s what quantitative methods are for!

Pragmatism

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 12/39

Readers of this book know by now that I’m fundamentally a pragmatist. Thus, the prior points notwithstanding, sometimes numbers are appropriate, sometimes they are illuminative, and sometimes they are simply demanded by those who commission evaluations. My point is not to be rigid but to place the burden of proof on justifying quantitizing. Do not go gently down that primrose path. Use numbers when appropriate, and then in moderation.

Here’s an example where numbers are appropriate. Psychologist Marvin Eisenstadt studied the link between career achievement and loss of a parent in childhood by identifying famous people from ancient Greece through to modern times whose lives merited significant entries in encyclopedias. He generated a list of 573 eminent people and did extensive research on their childhoods, an inquiry that took 10 years. “A quarter had lost at least one parent before the age of ten. By age 15, 34.5 percent had at least one parent die, and by the age of twenty, 45 percent” (Gladwell, 2013, p. 141). This conversion of qualitative codes to quantitative distributions is appropriate because the sample size is large, the numbers are accurate, and the focus of the inquiry is on a single variable. When there is something meaningful to be counted, then count. As sample sizes increase, especially in mixed-methods studies, quantizing is likely to become even more pervasive. Now let me offer an example where quantitizing strikes me as considerably less appropriate and meaningful.

Feeding the Quantitative Beast

The opening scenario in this rumination involved a board chair reacting to my qualitative presentation by asking how many said what. But those involved in qualitative studies exacerbate the problem by turning their reports into numbers even before being asked to do so. As I was completing this chapter, I received an analysis from a graduate student who had taken interviews I had given and counted how many times I used various words, a form of so-called content analysis that actually diminishes the meaning of both “content” and “analysis.” Having counted my use of various words, he then correlated them. He was seeking my interpretation of a couple of statistical correlations that he couldn’t explain. My response was that the entire analytical approach struck me as meaningless since I adapt my language in an interview to context, audience, and whatever I’m working on at the time. To lose the contextual meaning of words by counting them as isolated data points strikes me as highly problematic—and certainly not qualitative meaning making.

I receive a substantial number of qualitative evaluations to review each year. The most common pattern I see, and criticize in my review, is a qualitative study filled with numbers. Here’s an example that just came to me the very week I was writing this rumination. I’m afraid my response was rather intemperate.

Qualitative Report Excerpts

• Of the 20 students interviewed, 14 mentioned gaining leadership skills; 8 of 21 staff said leadership skills were important; 7 out of 13 field personnel said this, as did 4 out of 6 community leaders.

• Eighteen of the 20 students said they were more committed to scholarly publication; 2 said they didn’t want to be university scholars.

• Two out of the seven program directors at different universities felt that the purpose of the professional development program was mainly to train advanced students how to write for academic publication; the other five emphasized writing for policymakers.

• Out of the 14 university researchers interviewed, 7 had no opinion about students becoming better teachers because they were not sure what the program was doing to train students as teachers. Four other interviewees claimed that combining teaching skills with research skills caused confusion. Three said combining the two made sense and was valuable.

The 20-page report was filled with this kind of quantitative gibberish—I’m sorry, analytical reporting. Qualitative software easily generates such numbers, so that may feed this trend and give it the appearance of being appropriate and expected. It is not appropriate and should not be expected. Indeed,

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 13/39

I urge those involved in qualitative evaluations to make it clear at the outset to those who will be receiving the findings that numbers will generally not be reported. The focus will be on substantive significance. The point is not to be anti-numbers. The point is to be pro-meaningfulness.

Keep qualitative analysis first and foremost qualitative.

SOURCE: © Chris Lysy—freshspectrum.com

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 14/39

MODULE

69 Logical and Matrix Analyses, and Synthesizing QualitativeStudies

Logic: The art of thinking and reasoning in strict accordance with the limitations and incapacities of the human misunderstanding.

—Satirist Ambrose Bierce (1842–1914)

Contrariwise, if it was so, it might be; and if it were so, it would be; but as it isn’t, it ain’t. That’s logic.

—Author Lewis Carroll (1832–1898)

Logical Analysis While working inductively, the analyst is looking for emergent patterns in the data. These patterns, as noted in the preceding sections, can be represented as dimensions, categories, classification schemes, and themes. Once some dimensions have been constructed, using either participant-generated constructions or analyst- generated constructions, it is sometimes useful to cross-classify different dimensions to generate new insights about how the data can be organized and to look for patterns that may not have been immediately obvious in the initial, inductive analysis. Creating cross-classification matrices is an exercise in logic.

The logical process involves creating potential categories by crossing one dimension or typology with another and then working back and forth between the data and one’s logical constructions, filling in the resulting matrix. This logical system will create a new typology all parts of which may or may not actually be represented in the data. Thus, the analyst moves back and forth between the logical construction and the actual data in search of meaningful patterns.

In the high school dropout program described earlier, the focus of the program was reducing absenteeism, skipping classes, and tardiness. An external team of change agents worked with teachers in the school to help them develop approaches to the dropout problem. Observations of the program and interviews with the teachers gave rise to two dimensions. The first dimension distinguished teachers’ beliefs about what kind of programmatic intervention was effective with dropouts—that is, whether they primarily favored maintenance (i.e., caretaking or warehousing of kids to just keep the schools running), rehabilitation efforts (helping kids with their problems), or punishment (no longer letting them get away with the infractions they had been committing in the past). Teachers’ behaviors toward dropouts could be conceptualized along a continuum from taking direct responsibility for doing something about the problem at one end to shifting responsibility to others at the opposite end. Exhibit 8.12 shows what happens when these two dimensions are crossed. Six cells are created, each of which represents a different kind of teacher role in response to the program.

The qualitative analyst working with these data had been struggling in the inductive analysis to find the patterns that would express the different kinds of teacher roles manifested in the program. He had tried several constructions, but none of them quite seemed to work. The labels he came up with were not true to the data. When he described to me the other dimensions he had generated, I suggested that he cross them, as shown in Exhibit 8.12. When he did, he said that “the whole thing immediately fell into place.” Working back and forth between the matrix and the data, he generated a full descriptive analysis of diverse and conflicting teacher roles.

The description of teacher roles served several purposes. First, it gave teachers a mirror image of their own behaviors and attitudes. It could thus be used to help teachers make more explicit their own understanding of

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 15/39

roles. Second, it could be used by the external team of consultants to more carefully gear their programmatic efforts toward different kinds of teachers who were acting out the different roles. The matrix makes it clear that an omnibus strategy for helping teachers establish a program that would reduce dropouts would not work in this school; teachers manifesting different roles would need to be approached and worked with in different ways. Third, the description of teacher roles provided insights into the nature of the dropout problem. Having identified the various roles, the evaluator–analyst had a responsibility to report on the distribution of roles in this school and the observed consequences of that distribution.

Abductive Analysis One must be careful about purely logical analysis. It is tempting for an analyst using a logical matrix to force data into the categories created by the cross-classification to fill out the matrix and make it work. Logical analysis to generate new sensitizing concepts must be tested out and confirmed by the actual data. Such logically derived sensitizing concepts provide conceptual possibilities to test. Levin-Rozalis (2000), following American philosopher Charles Sanders Pierce of the pragmatic school of thought, suggests labeling the logical generation and discovery of hypotheses and findings abduction to distinguish such logical analysis from data- based inductive analysis and theory-derived deductive analysis.

EXHIBIT 8.12 An Empirical Typology of Teacher Roles in Dealing With High School Dropouts

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 16/39

Denzin (1978b) has explained abduction in qualitative analysis as a combination of inductive and deductive thinking with logical underpinnings:

Naturalists inspect and organize behavior specimens in ways which they hope will permit them to progressively reveal and better understand the underlying problematic features of the social world under study. They seek to ask the question or set of questions which will make that world or social organization understandable. They do not approach that world with a rigid set of preconceived hypotheses. They are initially directed toward an interest in the routine and taken-for-granted features of that world. They ask how it is that the persons in question know about producing orderly patterns of interaction and meaning. . . . They do not use a full-fledged deductive hypothetical scheme in thinking and developing propositions. Nor are they fully inductive, letting the so-called facts speak for themselves. Facts do not speak for themselves. They must be interpreted. Previously developed deductive models seldom conform with empirical data that are gathered. The method of abduction combines the deductive and inductive models of proposition development and theory construction. It can be defined as working from consequence back to cause or antecedent. The observer records the occurrence of a particular event, and then works back in time in an effort to reconstruct the events (causes) that produced the event (consequence) in question. (pp. 109–110)

The famous fictional detective Sherlock Holmes relied on abduction more than deduction or induction, at least according to William Sanders’s (1976) review of Holmes’s analytical thinking in The Sociologist as Detective. We’ve already suggested that the qualitative analyst is part scientist and part artist. Why not add the qualitative analyst as detective? Here’s an example.

An Example of Abductive Qualitative Analysis

In the evaluation of the rural community leadership program, we did follow-up interviews with participants to find out how they were using their training when they returned to their home communities. We found ourselves with a case not unlike the “Silver Blaze” story, in which Sherlock Holmes made much of the fact that the dog at the scene of the crime had not barked during the night while the crime was being committed. He inferred that the criminal was someone known to the dog. In our case, we discovered that graduates of the leadership program were not leading. In fact, they weren’t doing much of anything. Were their skills inadequate after only 1 week of intensive training? Did they lack confidence? Were they discouraged? Disinterested? Intimidated? Incompetent? Unmotivated?

So we had a finding. We had an outcome—or more precisely, the lack of an outcome. We worked backward from the experience of the training, examined what had happened during the training right up to the final session, and tried to connect the dots between what had happened and this unexpected result. We also returned to the participants for further reflections that might explain this general lack of follow-up action.

The participants expressed great interest in and commitment to exercising community leadership and engaging in community development. They expressed confidence in their abilities and felt they were competent to use the skills they had learned. But at perhaps the most teachable moment of all, in the final session of training, as the participants enthusiastically prepared to return to their communities and begin to use their learnings, the director of the program had offered a closing word of caution:

Take your time when you return. Don’t go back like a cadre of activists invading your community. You’ve had an intense experience together. Let things settle. It can be pretty overwhelming to the people back home when they get a sense that you’ve been through what for many of you has been a transformative experience. So go easy. Take your time. Resettle.

And so they did—more than he imagined. What he had neglected was any guidance about how to know when it was time to begin engaging after the reentry period of resettling. So they waited. And waited. And waited. Not wanting to get it wrong.

Of all the explanations we considered, that one fit the evidence the best. Its accuracy was further borne out when the director changed his parting advice at the end of subsequent programs and we found a different

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 17/39

result in the communities. A qualitative inquirer is a detective, using both data and reasoning to solve the mystery of how a story has unfolded and why it has unfolded in the way documented.

Abductive Analysis Caution

Abduction is not widely known, understood, or appreciated. I think it provides a distinct and important alternative to deductive and inductive reasoning. But one external reviewer of this book worried that including abduction might confuse novice researchers and weaken the emphasis on induction as the core of qualitative reasoning. I disagree but think the caution is worth acknowledging, so here it is:

Although interesting, the discussion of abductive analysis will likely only confuse the novice researcher. Perhaps this section should start with a stronger disclaimer that advises novice qualitative researchers to give their undivided attention to the skills needed for inductive inquiry and clearly identify potential logical positivist pitfalls.

Abductive Matrix Analysis

The empty cell of a logically derived matrix (the cell created by crossing two dimensions for which no name or label immediately occurs) creates an intersection of a possible consequence and antecedent that begs for abductive exploration and explanation. Each such intersection of consequence and antecedent sensitizes the analyst to the possibility of a category of activity or behavior that has either been overlooked in the data or that is logically a possibility in the setting but has not yet been documented. The latter cases are important to note for their importance derives from the fact that they did not occur. The next section will look in detail at a process–outcomes matrix ripe with abductive possibilities. First, Exhibit 8.13 shows a matrix for mapping stakeholders’ stakes in a program or policy. This matrix can be used to guide data collection as well as analysis.

A Process–Outcomes Matrix The linkage between processes and outcomes constitutes such a fundamental issue in many program evaluations that it provides a particularly good focus for illustrating qualitative matrix analysis. As discussed in Chapter 4, qualitative methods can be particularly appropriate for evaluation where program processes, impacts, or both are largely unspecified or difficult to measure. This can be the case because the outcomes are meant to be individualized; sometimes one is simply uncertain as to what a program’s outcomes will be; and in many programs, neither processes nor impacts have been carefully articulated. Under such conditions, one purpose of the evaluation may be to illuminate program processes, program impacts, and the linkages between the two. This task can be facilitated by constructing a process–outcomes matrix to organize the data.

EXHIBIT 8.13 Power Versus Interest Grid for Analyzing Diverse Stakeholders’ Engagement With a Program, a Policy, or an Evaluation

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 18/39

Players. High interest, high power: They make things happen.

Context setters. High power, low interest: They watch what unfolds and can become players if they get interested.

Subjects. High interest, low power: They do not act unless organized and empowered.

Crowd. Low interest, low power: They are disengaged until they become aware that they have a stake in what is unfolding.

This matrix can be used to map the stakeholder environment for any initiative by gathering data about the perspectives, interests, and nature of power of people in diverse relationships to the initiative (Bryson & Patton, 2010, p. 42; Patton, 2008, p. 80).

Exhibit 8.14 shows how such a matrix can be constructed. Major program processes or identified implementation components are listed along the left side. Types or levels of outcomes are listed across the top. The category systems for program processes and outcomes are developed from the data in the same way that other typologies are constructed (see previous sections). The cross-classification of any process with any outcome produces a cell in the matrix—for example, the first cell in Exhibit 8.14 is created by the intersection of Process 1 with Outcome a. The information that goes in Cell 1-a (or any other cell in the matrix) describes linkages, patterns, themes, experiences, content, or actual activities that help us understand the relationships between processes and outcomes. Such relationships may have been identified by participants themselves during interviews or discovered by the evaluator in analyzing the data. In either case, the process–outcomes matrix becomes a way of organizing, thinking about, and presenting the qualitative connections between program implementation dimensions and program results.

EXHIBIT 8.14 Process–Impact Matrix

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 19/39

Application of the Process–Impact Matrix: An Example

An example will help make the notion of the process–outcomes matrix (Exhibit 8.14) more concrete and, hopefully, useful. Suppose we have been evaluating a juvenile justice program that places delinquent youth in foster homes. We have visited several foster homes, observed what the home environments are like, and interviewed the juveniles, the foster home parents, and the probation officers. A regularly recurring process theme concerns the importance of “letting kids learn to make their own decisions.” A regularly recurring

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 20/39

outcomes theme involves “keeping the kids straight” (reduced recidivism). Crossing the program process (“kids making their own decisions”) with the program outcome (“keeping kids straight”) creates a data analysis question: What actual decisions do juveniles make that are supposed to lead to reduced recidivism? We then carefully review our field notes and interview quotations, looking for data that help us understand how people in the program have answered this question based on their actual behaviors and practices. When we describe what decisions juveniles actually make in the program, the decision makers to whom our findings are reported can make their own judgments about the strength or weakness of the linkage between this program process and the desired outcome. Moreover, once the process–outcomes descriptive analysis of linkages has been completed, the evaluator is at liberty to offer interpretations and judgments about the nature and quality of this process–outcome connection.

An In-Depth Analysis Example: Recognizing Processes, Outcomes, and Linkages in Qualitative Data

Because of the centrality of the sensitizing concepts “program process” and “program outcome” in evaluation research, it may be helpful to provide a more detailed description of how these concepts can be used in qualitative analysis. How does one recognize a program process? Learning to identify and label program processes is a critical evaluation skill. This sensitizing notion of “process” is a way of talking about the common action that cuts across program activities, observed interactions, and program content. The example I shall use involves data from the wilderness education program I evaluated and discussed throughout the observations chapter (Chapter 6). That program, titled the Southwest Field Training Project, used the wilderness as a training arena for professional educators in the philosophy and methods of experiential education by engaging those educators in their own experiential learning process. Participants went from their normal urban environments into the wilderness for 10 days at a time, spending at least 1 day and night completely alone in some wilderness spot “on solo.” At times, while backpacking, the group was asked to walk silently so as not to be distracted from the wilderness sounds and images by conversation. In group discussions, participants were asked to talk about what they had observed about the wilderness and how they felt about being in the wilderness. Participants were also asked to write about the wilderness environment in journals. What do these different activities have in common, and how can that commonality be expressed?

We begin with several different ways of abstracting and labeling the underlying process:

• Experiencing the wilderness • Learning about the wilderness • Appreciating the wilderness • Immersion in the environment • Developing awareness of the environment • Becoming conscious of the wilderness • Developing sensitivity to the environment

Any of these phrases, each of which consists of some verb form (experiencing, learning, developing, etc.) and some noun form (wilderness, environment, etc.), captures some nuance of the process. The qualitative analyst works back and forth between the data (field notes and interviews) and his or her conception of what it is that needs to be expressed to find the most fitting language to describe the process. What language do people in the program use to describe what those activities and experiences have in common? What language comes closest to capturing the essence of this particular process? What level of generality or specificity will be most useful in separating out this particular set of things from other things? How do program participants and staff react to the different terms that could be used to describe the process?

It’s not unusual during analysis to go through several different phrases before finally settling on the exact language that will go into a final report. In the Southwest Field Training Project, we began with the concept label “Experiencing the Wilderness.” However, after several revisions, we finally described the process as

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 21/39

“Developing Sensitivity to the Environment” because this broader label permitted us to include discussions and activities that were aimed at helping participants understand how they were affected by and acted in their normal institutional environments. “Experiencing the wilderness” became a specific subprocess that was part of the more global process of “developing sensitivity to the environment.” Program participants and staff played a major role in determining the final phrasing and description of this process.

Other processes identified as important in the implementation of the program were as follows:

• Encountering and managing stress • Sharing in group settings • Examining professional activities, needs, and commitments • Assuming responsibility for articulating personal needs • Exchanging professional ideas and resources • Formally monitoring experiences, processes, changes, and impacts

As you struggle with finding the right language to communicate themes, patterns, and processes, keep in mind that there is no absolutely “right” way of stating what emerges from the analysis. There are only more and less useful ways of expressing what the data reveal.

Identifying and conceptualizing program outcomes and impacts can involve induction, deduction, abduction, and/or logical analysis. Inductively, the qualitative analyst looks for changes in participants, expressions of change, program ideology about outcomes and impacts, and ways that people in the program make distinctions between “those who are getting it” and “those who aren’t getting it” (where it is the desired outcome). In highly individualized programs, the statements about change that emerge from program participants and staff may be global. Outcomes such as “personal growth,” increased “awareness,” and “insight into self” are difficult to operationalize and standardize. That is precisely the reason why qualitative methods are particularly appropriate for capturing and evaluating such outcomes. The task for the qualitative analyst, then, is to describe what actually happens to people in the program and what they say about what happens to them.

Logically (or abductively), constructing a process–outcomes matrix can suggest additional possibilities. That is, where data on both program processes and participant outcomes have been sorted, analysis can be deepened by organizing the data through a logical scheme that links program processes to participant outcomes. Such a logically derived scheme was used to organize the data in the Southwest Field Training Project. First, a classification scheme that described different types of outcomes was conceptualized:

a. Changes in knowledge b. Changes in attitudes c. Changes in feelings d. Changes in behaviors e. Changes in skills

These general themes provided the reader of the report with examples of and insights into the kinds of changes that were occurring and how those changes were perceived by participants to be related to specific program processes. I emphasize that the process–outcomes matrix is merely an organizing tool; the data from participants themselves and from field observations provide the actual linkages between processes and outcomes.

What was the relationship between the program process of “developing sensitivity to the environment” and these individual-level outcomes? Space permits only a few examples from the data.

Skills: “Are you kidding? I learned how to survive without the comforts of civilization. I learned how to read the terrain ahead and pace myself. I learned how to carry a heavy load. I learned how to stay dry when

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 22/39

it’s raining. I learned how to tie a knot so that it doesn’t come apart when pressure is applied. You think those are metaphors for skills I need in my work? You’re damn right they are.” Attitudes: “I think it’s important to pay attention to the space you’re in. I don’t want to just keep going through my life oblivious to what’s around me and how it affects me and how I affect it.” Feelings: “Being out here, especially on solo, has given me confidence. I know I can handle a lot of things I didn’t think I could handle.” Behaviors: “I use my senses in a different way out here. In the city you get so you don’t pay much attention to the noise and the sounds. But listening out here, I’ve also begun to listen more back there. I touch more things too, just to experience the different textures.” Knowledge: “I know about how this place was formed, its history, the rock formations, the effects of the fires on the vegetation, where the river comes from, and where it goes.”

A different way of thinking about organizing data around outcomes was to think of the different levels of impact: (a) effects at the individual level, (b) effects on the group, and (c) effects on the institutions from which participants came into the program. The staff hoped to have impacts at all of these levels. Thus, it also was possible to organize the data by looking at what themes emerged when program processes were crossed with levels of impact. How did “developing sensitivity to the environment” affect individuals? How did the process of “developing sensitivity to the environment” affect the group? What was the effect of “developing sensitivity to the environment” on the institutions to which participants returned after their wilderness experiences? The process–outcomes matrix thus becomes a way of asking questions of the data, an additional source of focus in looking for themes and patterns in the hundreds of pages of field notes and interview transcriptions. Exhibit 8.35, at the end of this chapter (pp. 643–649), presents an extended excerpt from the qualitative evaluation report.

A Three-Dimensional Qualitative Analysis Matrix To study how schools used planning and evaluation processes, Campbell (1983) developed a 500-cell matrix (Exhibit 8.15) that begins (but just begins) to reach the outer limits of what one can do in a three-dimensional space. Campbell used this matrix to guide data collection and analysis in studying how the mandated, statewide educational planning, evaluation, and reporting system in Minnesota was being used. She examined five levels of use (high school, . . . , community, district), 10 components of the statewide project (planning, goal setting, . . . , student involvement), and 10 factors affecting utilization (personal factor, political factors, . . . ). Exhibit 8.15 again illustrates matrix thinking for both data organization and analytical/conceptual purposes.

Miles et al. (2014) have provided a rich source of ideas and illustrations on how to use matrices in qualitative analysis. Their Sourcebook provides a variety of ideas for analytical approaches to qualitative data, including a variety of mapping and visual display techniques.

Synthesizing Qualitative Studies We turn now to analysis across a different unit of analysis: synthesizing patterns, themes, and findings across qualitative studies where a completed study is the unit of analysis. In Chapter 5, Exhibit 5.8 (pp. 266–272), I introduced two purposeful sampling strategies that involve sampling completed studies as the unit of analysis: (1) qualitative research synthesis and (2) systematic qualitative evaluation reviews. One is research focused, the other evaluation focused. I distinguished these as different purposeful sampling strategies because they serve different purposes and select different kinds of qualitative studies for synthesis. (1) A qualitative research synthesis selects qualitative studies to analyze for cross-cutting research findings and contributions to theory. For example, there have been a substantial number of separate and independent ethnographic studies of coming-of-age and initiation ceremonies across cultures. A synthesis involves analyzing and interpreting

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 23/39

findings across those myriad studies. (2) In contrast, a systematic qualitative evaluation review seeks patterns across diverse qualitative evaluations to reach conclusions about what is effective.

EXHIBIT 8.15 Conceptual Guide for Data Collection and Analysis: Utilization of Planning, Evaluation, and Reporting

Qualitative Research Synthesis

A qualitative research synthesis involves seeking patterns across and integrating different qualitative studies (Finlayson & Dixon, 2008; Hannes & Lockwood, 2012; Saini & Shlonsky, 2012). Which studies are included in the final synthesis will depend on what emerges as relevant and meaningful during the synthesis. “Assessing quality is also about examining how study findings fit (or do not fit) with the findings of other studies. How study findings fit with the findings of other studies cannot be assessed until the synthesis is completed” (Harden & Gough, 2012, p. 160). In essence, quality criteria in a qualitative synthesis can be emergent.

Hannes and Lockwood (2012) have examined diverse approaches and frameworks for synthesizing qualitative research. The diversity of synthesis approaches reflects the diversity of qualitative methods and the variety of theoretical perspectives that inform qualitative inquiries (see Chapter 3). What all syntheses share in

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 24/39

common is having to address at a minimum three separate processes: (1) identifying and aggregating studies to include; (2) analyzing patterns, themes, and findings across the studies; and (3) interpreting the results.

In one sense, each qualitative study is a case. Synthesis of different qualitative studies on the same subject is a form of cross-case analysis. Such a synthesis is much more than a literature review. Noblit and Hare (1988) describe synthesizing qualitative studies as “meta-ethnography,” in which the challenge is to “retain the uniqueness and holism of accounts even as we synthesize them in the translations” (p. 7).

Systematic Qualitative Evaluation Reviews

Systematic qualitative evaluation serves a meta-evaluation function and is aimed at increasing confidence in actions to be taken to solve particular problems by synthesizing patterns of effectiveness across separate and independent evaluation studies. Such a systematic review seeks to identify, appraise, select, and synthesize all high-quality evaluation research evidence relevant to a particular arena of knowledge that is the basis for interventions (Funnell & Rogers, 2011, pp. 508–514).

Systematic reviews of randomized controlled trial (RCT) studies have been the basis for evidence-based medicine. The cases being synthesized are completed, usually published, studies. The process involves identifying all high-quality, peer-reviewed studies on a problem and synthesizing findings across those separate and diverse studies to reach conclusions about what is effective in dealing with the problem of concern, for example, female hormone replacement therapy or effective treatments for prostate cancer. Informing and setting guidelines for treatment is the instrumental use. The important new direction in systematic reviews is including qualitative evaluation studies (Gough et al., 2012; Wright, 2013). For example, the internationally prestigious Cochrane Collaboration Qualitative & Implementation Methods Group supports the synthesis of qualitative evidence and the integration of qualitative evidence with other evidence (mixed methods) in Cochrane intervention reviews on the effectiveness of health interventions. In effect, systematic reviews serve a meta-evaluation function.

Systematic Evaluation Reviews of Lessons Learned

Evaluators can synthesize lessons from a number of case studies to generate generic factors that contribute to program effectiveness—as, for example, Lisbeth Schorr (1988) did for poverty programs in her review and synthesis Within Our Reach: Breaking the Cycle of Disadvantage. Three decades ago, the U.S. Agency for International Development began commissioning lessons-learned synthesis studies on subjects such as irrigation (Steinberg, 1983), rural electrification (Wasserman & Davenport, 1983), food for peace (Rogers & Wallerstein, 1985), education development efforts (Warren, 1984), contraceptive social marketing (Binnendijk, 1986), agricultural policy analysis and planning (Tilney & Riordan, 1988), and agroforestry (Chew, 1989). In synthesizing separate evaluations to identify lessons learned, evaluators build a store of knowledge for future program development, more effective program implementation, and enlightened policy making.

SIDEBAR

QUALITATIVE RESEARCH SYNTHESES VERSUS SYSTEMATIC QUALITATIVE EVALUATION REVIEWS

I distinguish qualitative research syntheses from systematic qualitative evaluation reviews because they involve different purposeful sampling strategies that serve different purposes. You won’t find this distinction elsewhere. I make the distinction, and believe it is worth making, because this book addresses both research and evaluation methods. Qualitative research synthesis serves research purposes—selecting qualitative studies to analyze for cross-cutting findings and contributions to theory. Systematic qualitative evaluation reviews serve evaluation purposes—analyzing diverse qualitative evaluations to reach conclusions about patterns of effectiveness. The criteria for what studies to include in each case will be

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 25/39

different, as will the criteria for judging the final result: contribution to theory (qualitative research syntheses) versus contributions to practice (systematic evaluation reviews).

For scholarly inquiry, the qualitative research synthesis is a way to build theory through induction, deduction, interpretation, and integration. For evaluators, a qualitative systematic review can identify and extrapolate lessons learned to inform future program designs, identify effective intervention approaches, and build program theory.

The sample for synthesis studies usually consists of case studies with a common focus, for example, elementary education, health care for the elderly, and so on. However, one can also learn lessons about effective human intervention processes more generically by synthesizing case studies on quite different subjects. I synthesized three quite different qualitative evaluations conducted for The McKnight Foundation: (1) a major family housing effort, (2) a downtown development endeavor, and (3) a graduate fellowship program for minorities. Before undertaking the synthesis, I knew nothing about these programs, nor did I approach them with any particular preconceptions. I was not looking for any specific similarities, and none were suggested to me by either McKnight or program staff. The results were intended to provide insights into The McKnight Foundation’s operating philosophy and strategies as exemplified in practice by real operating programs. Independent evaluations of each program had already been conducted and presented to The McKnight Foundation, showing that these programs had successfully attained and exceeded the intended outcomes. But why were they successful? That was the intriguing and complex question on which the synthesis study focused.

The synthesis design included fieldwork (interviews with key players and site visits to each project) as well as an extensive review of their independent evaluations. I identified common success factors that were manifest in all three projects. Those were illuminating but not surprising. The real contribution of the synthesis was in how the success factors fit together, an unanticipated pattern that deepened the implications for understanding effective philanthropy.

The 10 success factors common to all three programs were as follows:

1. Strong leadership, developed, engaged, and supported throughout the initiative 2. A sizeable amount of money ($15 million), able to attract attention and generate support 3. Effective use of leverage at every level of program operation (McKnight insisted on sizable matching

funds and use of local in-kind resources from participating universities.) 4. A long-term perspective on and commitment to a sustainable program with cumulative impact over

time—in perpetuity (Support for the programs was converted to an endowment.) 5. A carefully melded public–private partnership 6. A program based on a vision made real through a carefully designed model that was true to the vision 7. Taking the time and effort to carefully plan in a process that generated broad-based community and

political support throughout the state 8. The careful structuring of local board control so that responsibility and ownership resided among key

influentials 9. Taking advantage of the right timing and climate for this kind of program 10. Clear accountability and evaluation so that problems could be corrected and accomplishments could

be recognized

While each of these factors provided insight into an important element of effective philanthropic programming, the unanticipated pattern was how these factors fit together to form a constellation of excellence. I found that I couldn’t prioritize these factors because they worked together in such a way that no

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 26/39

one factor was primary or sufficient; rather, each made a critical contribution to an integrated, effectively functioning whole. The lesson that emerged for effective philanthropy was not a series of steps to follow but rather a mosaic to create; that is, effective philanthropy appears to be a process of matching and integrating elements so that the pieces fit together in a meaningful and comprehensive way as a solution to complex problems. This means matching people with resources; bringing vision and values to bear on problems; and nurturing partnerships through leverage, careful planning, community involvement, and shared commitments —and doing all these things in mutually reinforcing ways. The challenge for effective philanthropy, then, is putting all the pieces and factors together to support integrated, holistic, and high-impact efforts and results— and to do so creatively (Storm & Vitt, 2000, pp. 115–116).

Another example of a major synthesis focused on evaluation of the 2005 Paris Declaration on Aid Effectiveness, endorsed by more than 150 countries and international organizations. An independent evaluation examined what difference, if any, the Paris Declaration made to development processes and results (Wood et al., 2011). The final report was a synthesis of case studies done in 22 developing countries and 18 donor agencies. The synthesis identified the factors that contributed to international aid reform and barriers to more effective aid. (For the lessons learned, see Dabelstein & Patton, 2013a.)

Qualitative synthesis has become a major and important approach to making sense of multiple and diverse qualitative studies. It is possible only because there are now many qualitative studies in diverse fields of interest available for synthesis. Synthesis findings are elevating the contribution of qualitative results to both research and evaluation.

SIDEBAR

REALIST SYNTHESIS

“Realist synthesis might best be thought of as a way of assembling rocks, or nonstandardized pieces of knowledge” (Funnell & Rogers, 2011, p. 514). Developed by British sociologist and evaluator Ray Pawson (2013), realist synthesis selects and integrates any quality evidence on a topic, including experiments, quasi-experimental studies, and case studies. “Quality is not assessed with reference to a hierarchy of research designs for the entire study but by assessing whether threats to validity have been adequately addressed in terms of the specific piece of evidence being used” (p. 515). Realist synthesis uses purposeful sampling of diverse kinds of evidence available, both quantitative and qualitative, to develop, refine, and test theories about how, for whom, and in what contexts policies will be effective . . . [and to] identify causal mechanisms that operate only in particular contexts. Realist synthesis is inherently a process of building, testing, and refining program theory. (p. 515)

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 27/39

MODULE

70 Interpreting Findings, Determining Substantive Significance, Elucidating Phenomenological Essence, and Hermeneutic Interpretation

Simply observing and interviewing do not ensure that the research is qualitative; the qualitative researcher must also interpret the beliefs and behaviors of participants.

—Valerie J. Janesick (2000, p. 387)

Interpreting for Meaning Qualitative interpretation begins with elucidating meanings. The analyst examines a story, a case study, a set of interviews, or a collection of field notes and asks, “What does this mean? What does this tell me about the nature of the phenomenon of interest?” In asking these questions, the analyst works back and forth between the data or story (the evidence) and his or her own perspective and understandings to make sense of the evidence. Both the evidence and the perspective brought to bear on the evidence need to be elucidated in this choreography in the search for meaning. Alternative interpretations are tried and tested against the data.

Interpretation, by definition, involves going beyond the descriptive data. Interpretation means attaching significance to what was found, making sense of findings, offering explanations, drawing conclusions, extrapolating lessons, making inferences, considering meanings, and otherwise imposing order on an unruly but surely patterned world. The rigors of interpretation and bringing data to bear on explanations include dealing with rival explanations, accounting for disconfirming cases, and accounting for data irregularities as part of testing the viability of an interpretation. All of this is expected—and appropriate—as long as the researcher owns the interpretation and makes clear the difference between description and interpretation. A good example is Reid Zimmerman’s (2014) description and interpretation of the seven deadly sayings of a nonprofit leader.

Schlechty and Noblit (1982) concluded that an interpretation may take one of three forms:

1. Making the obvious obvious 2. Making the obvious dubious 3. Making the hidden obvious

This captures rather succinctly what research colleagues, policymakers, and evaluation stakeholders expect: (1) confirm what we know that is supported by data, (2) disabuse us of misconceptions, and (3) illuminate important things that we didn’t know but should know. Accomplish these three things, and those interested in the findings can take it from there.

Explaining findings is an interpretive process. For example, when we analyzed follow-up interviews with participants who had gone through intensive community leadership training, we found a variety of expressions of uncertainty about what they should do with their training. In the final day of a six-day retreat, after learning how to assess community needs, work with diverse groups, communicate clearly, empower people to action, and plan for change, they were cautioned to go easy in transitioning back to their communities and to take their time in building community connections before taking action. What program staff meant as a last-day warning about not returning to the community as a bull in a china shop and charging ahead destructively had, in fact, paralyzed the participants and made them afraid to take any action at all. The program, which intended

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 28/39

to position participants for action, had inadvertently left graduates in “action paralysis” for fear of making mistakes. The meaning-laden phrase “action paralysis” emerged from the data analysis through interpretation. No one used that specific phase. Rather, we interpreted action paralysis as the essence of what the interviewees were reporting through a haze of uncertainties, ambiguities, worried musings, and wait-and-see- before-acting reflections.

Interpreting Findings at Different Levels of Analysis In the early 1990s, The McKnight Foundation in Minnesota invested $13 million in an innovative initiative titled The Aid to Families in Poverty Program. The initiative funded 34 programs using a variety of strategies. Our evaluation team conducted evaluations at the program level, the overall initiative level (synthesis of findings across all 34 programs), and the level of the broader policy environment that set the context for antipoverty interventions in Minnesota and the nation. The qualitative synthesis team interviewed program staff, conducted site visits, reviewed individual program evaluation reports, met with foundation program officers, and reviewed the national literature about poverty issues and programs. Here are examples of major findings and how we interpreted them. I invite you to pay special attention to how the findings and interpretations change at different levels of analysis for different units of analysis.

SIDEBAR

GENERAL THEORY OF INTERPRETATION

Interpretation is dependent on value, and judgments in all domains requiring interpretation are necessarily value judgments of some kind or other—though they start out from the descriptive facts.

Whenever we are faced with a normative system like law, whose point is to govern conduct, we cannot understand it simply as a pattern of behavior. We have to see it as an internalized set of standards and principles that the participants take to justify their behavior. But we need not share that point of view in order to understand it, even though we must rely on our own capacity for value judgments when we interpret how others see things as right that we believe to be wrong, and vice versa. We will not understand a bad system unless we see how its participants see it as good.

—Dworkin’s General Theory of Interpretation From Ronald Dworkin: The Moral Quest (Nagel, 2013, p. 56)

1. Outcomes for Families in Poverty

Finding: Program participants and staff emphasized the importance of helping families move out of crisis. Taking first steps and arresting decline were consistently reported as important outcomes. A common early outcome was increased intentionality—helping families in poverty come up with a plan, a sense of direction, and a commitment to making progress. Interpretation: Funders, policymakers, and evaluators place heavy emphasis on achieving long-term outcomes—getting families out of poverty. In doing so, they undervalue the huge amount of work it takes to establish trust with families in need, help stabilize families, and begin the process toward long-term outcomes. Early indicators of progress foreshadow longer-term outcomes and should be reported and valued.

2. Characteristics of Effective Program Staff

Finding: Effective staff approach program participants on a case-by-case basis. Their approach is respectful and individualized. They recognize that progress occurs in varying ways and at different rates,

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 29/39

and what may be limited progress for one family may represent enormous strides for another. Moreover, effective staff are highly responsive to individual participants’ situations, needs, capabilities, interests, and family context. In being responsive and respectful, they work to raise hopes and empower participants by helping them make concrete, intentional changes. Interpretation: Effective staff are the key to effective programs and achieving desired program outcomes for families in poverty. It wasn’t the program model or approach that made the difference. More important than the conceptual model or theory of change being implemented was how staff interacted with program participants. This has implications for staff recruitment, training, support, and performance evaluation.

3. Characteristics of Effective Programs

Finding: Effective programs support staff responsiveness by being flexible and giving staff discretion to take whatever actions assist participants to climb out of poverty. Flexible, responsive programs affect the larger systems of which they are a part by pushing against boundaries, arrangements, rules, procedures, and attitudes that hinder their capability to work flexibly and responsively—and therefore effectively— with participants. Interpretation: Patterns of effectiveness cut across different levels of operation and impact and showed up in what we would call the program culture. How people are treated affects how they treat others. How staff are treated affects how they treat program participants. Responsiveness reinforces responsiveness, flexibility supports individualization, and empowerment breeds empowerment. Program directors, professional staff, and organizational administrators will often need training and technical assistance in setting up and working with flexible, responsive approaches.

4. Philanthropic Foundation Lessons

Finding: The foundation chose the initiative’s name without consultation and review in the community. Our interviews found that many program staff and participants reacted negatively to the initiative’s name: Aid to Families in Poverty Program. Interpretation: Language matters to people. What an initiative is called sends messages. A name for the initiative that conveyed hope and strength rather than deficiency would have been received more positively. Failure to consult with people outside the foundation about the name increased the risk that the initiative’s title would inadvertently carry negative connotations (Patton, 1993).

Substantive Significance In lieu of statistical significance, qualitative findings are judged by their substantive significance. The analyst makes an argument for substantive significance in presenting findings and conclusions, but readers and users of the analysis will make their own value judgments about significance. In determining substantive significance, the analyst addresses these kinds of questions:

• To what extent and in what ways do the findings increase and deepen understanding of the phenomenon studied (verstehen)?

• To what extent are the findings useful for their intended purpose, for example, contributing to theory, informing policy, improving a program, informing decision making about some action, or problem solving in action research?

• To what extent are the findings consistent with other knowledge? A finding supported by and supportive of other work has confirmatory significance. A finding that breaks new ground has discovery or innovative significance.

• How solid, coherent, and consistent is the evidence in support of the findings? Triangulation, for example, can be used in determining the strength of evidence in support of a finding.

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 30/39

Examples of Substantive Significance

Here are three examples of findings that, when interpreted, can be judged as substantively significant:

1. Case studies of 30 postdoctoral fellowship recipients found only 2 whose lives had been disrupted during the fellowship. Those 2 permanently lost the fellowship funds and were deemed failures by the fellowship’s sponsor and the fellows’ institutions. The case studies showed that the failures were due to the inflexibility of the funders in having no process for allowing a temporary sabbatical from the fellowship under conditions of sudden hardship. The consequences for the fellows were dramatic and long term, while a solution was readily at hand that could alleviate such dire consequences, which were likely to occur again in the future.

2. During a polio immunization campaign in India, a community of Muslim mothers heard a rumor that the immunization was a Hindu plot to sterilize Muslim children. So they hid their children from the vaccinators. A year later, those children were the source of an outbreak of polio. The numbers who resisted were small, but the consequences were great. The implication was that immunization campaigns need to be inquiring into community perceptions in real time so as to intervene and correct misperceptions in real time.

3. An international funder spent a large sum to build typhoon shelters in Bangladesh in low-lying areas near the ocean, where thousands of poor people were especially vulnerable to storms. When a typhoon hit, the shelters went largely unused because animals were prohibited and the resources of these poor people were their animals, which they would not abandon. The funding agency had been told of this potential problem when it was identified from a few key informant interviews, but the agency dismissed the findings because of the small sample size.

Determining substantive significance requires critical thinking about the broader consequences of findings. Exhibit 8.16 provides another example of substantive significance.

Interpretation Requires Both Critical Thinking and Creativity

Identifying patterns, themes, and categories involves using both creative and critical faculties in making carefully considered judgments about what is meaningful and substantively significant in the data. Since as a qualitative analyst you do not have a statistical test to help tell you when an observation or pattern is significant, you must rely first on your own sense making, understandings, intelligence, experience, and judgment; second, you should take seriously the responses of those who were studied or who participated in the inquiry about what they have reported to you as meaningful and significant; and third, you should consider the responses and reactions of those who read and review the results. Where all three—the qualitative analyst, those studied, and reviewers—agree, one has consensual validation of the substantive significance of the findings. Where disagreements emerge, which is more usual, you get a more interesting life and the joys of debate.

EXHIBIT 8.16 Substantive Significance Example: Minimally Disruptive Medicine

Chronic disease requires ongoing, lifetime management. This means that a patient must find a way to fit medical and lab appointments, exercise, medications, and dietary changes into a life already busy with family and work. Victor Montori, a professor of medicine at Mayo Clinic, Carl May, a professor of medical sociology at Newcastle University, and Francis Mair, a professor of primary care research at University of Glasgow, decided to study the burden of treatment by interviewing patients with multiple chronic comorbidities or cognitive impairment—two groups that are often excluded from studies examining compliance.

Case Study Examples

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 31/39

• A man who in the previous two years had visited specialist clinics for appointments, tests, and treatment 54 times (the equivalent of a full day every two weeks)

• A woman whose doctors had prescribed medications to be taken at 11 separate times during the day and was having trouble managing this (Montori, 2014)

Though initially from a small purposeful sample, the findings were sufficiently substantive to inspire conceptualization of a new category of health intervention: Minimally Disruptive Medicine. Minimally disruptive medicine is minimally disruptive not because it is minimal but because it is designed to fit naturally into a patient’s life and be manageable on an ongoing basis.

Determining substantive significance involves distinguishing signal from noise, which involves risking two kinds of errors. First, the analyst may decide that something is not a signal—that is, is not significant—when in fact it is, or second, and conversely, the analyst may attribute significance to something that is meaningless (just noise). The Halcolm story presented as a graphic comic at the end of Chapter 6 (pp. 418–419)is worth repeating here to illustrate this challenge of making judgments about what is really significant.

Halcolm was approached by a woman who handed him something. Without hesitation, Halcolm returned the object to the woman. The many young disciples who followed Halcolm to learn his wisdom began arguing among themselves about the special meaning of this interchange. A variety of interpretations were offered.

When Halcolm heard of the argument among his young followers, he called them together and asked each one to report on the significance of what they had observed. They offered a variety of interpretations. When they had finished, he said, “The real purpose of the exchange was to enable me to show you that you are not yet sufficiently masters of observation to know when you have witnessed a meaningless interaction.”

SIDEBAR

INTEROCULAR SIGNIFICANCE

If we are interested in real significance, we ignore little differences . . . . We ignore them because, although they are very likely real, they are very unlikely to hold up in replications. Fred Mosteller, the great applied statistician, was fond of saying that he did not care much for statistically significant differences, he was more interested in interocular differences, the differences that hit us between the eyes. (Scriven, 1993, p. 71)

Phenomenology as an Interpretative Framework: Elucidating Essence

Phenomenology asks for the very nature of a phenomenon, for that which makes a some-“thing” what it is—and without which it could not be what it is.

—Van Manen (1990, p. 10)

Phenomenology as a qualitative theoretical framework was discussed at length in Chapter 3 (pp. 115–118). In this module, we’re going to focus on phenomenological analysis as an interpretative framework. Phenomenological analysis seeks to grasp and elucidate the meaning, structure, and essence of the lived experience of a phenomenon for a person or group of people. Before I present the steps of one particular approach to phenomenological analysis, it is important to note that phenomenology has taken on a number of meanings, has a number of forms, and encompasses varying traditions, including transcendental

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 32/39

phenomenology, existential phenomenology, and hermeneutic phenomenology (Schwandt, 2007). Moustakas (1994) further distinguishes empirical phenomenology from transcendental phenomenology. Gubrium and Holstein (2000) add the label social phenomenology. Van Manen (1990) prefers “hermeneutical phenomenological reflection.” Sonnemann (1954) introduced the term phenomenography to label phenomenological investigation aimed at “a descriptive recording of immediate subjective experience as reported” (p. 344). Harper (2000) talks of looking at images through “the phenomenological mode”—that is, from the perspective of the self: “From the phenomenological perspective, photographs express the artistic, emotional, or experiential intent of the photographer” (p. 727). To this confusion of terminology is added the difficulty of distinguishing phenomenological philosophy from phenomenological methods and phenomenological analysis, all of which increases the tensions and contradictions in qualitative inquiry (Gergen & Gergen, 2000).

The use of the term phenomenology in contemporary versions of qualitative inquiry in North America tends to reflect a subjectivist, existentialist, and non-critical emphasis not present in the Continental tradition represented in the work of Husserl and Heidegger. The latter viewed the phenomenological project, so to speak, as an effort to get beneath or behind subjective experience to reveal the genuine, objective nature of things, and as a critique of both taken-for-granted meanings and subjectivism. Phenomenology, as it is commonly discussed in accounts of qualitative research, emphasizes just the opposite: It aims to identify and describe the subjective experiences of respondents. It is a matter of studying everyday experience from the point of view of the subject, and it shuns critical evaluation of forms of social life. (Schwandt, 2001, p. 192)

Phenomenological analysis involves and emphasizes different elements depending on which type of phenomenology you are using as a framework. I have chosen to focus on the phenomenological approach to analysis taken by Clark Moustakas, founder of The Center for Humanistic Studies (Detroit, Michigan). More than most, he has focused on the analytical process itself (Douglass & Moustakas, 1985; Moustakas, 1961, 1988, 1990, 1994, 1995). As we go deeper into the perspective and language of phenomenological analysis, let me warn you that the terminology and distinctions can be hard to grasp at first. But don’t skip over them lightly. These distinctions constitute windows into the world of phenomenological analysis. They matter. See if you can figure out why they matter. If you can, you will have grasped phenomenological interpretation.

Consciousness, Intentionality, Nomea, and Noesis Husserl’s transcendental phenomenology is intimately bound up in the concept of intentionality. In Aristotelian philosophy the term intention indicates the orientation of the mind to its object; the object exists in the mind in an intentional way. . . .

Intentionality refers to consciousness, to the internal experience of being conscious of something; thus the act of consciousness and the object of consciousness are intentionally related. Included in understanding of consciousness are important background factors such as stirrings of pleasure, shapings of judgment, or incipient wishes. Knowledge of intentionality requires that we be present to ourselves and things in the world, that we recognize that self and world are inseparable components of meaning.

Consider the experience of joy on witnessing a beautiful landscape. The landscape is the matter. The landscape is also the object of the intentional act, for example, its perception in consciousness. The matter enables the landscape to become manifest as an object rather than merely exist in consciousness.

The interpretive form is the perception that enables the landscape to appear; thus the landscape is self-given; my perception creates it and enables it to exist in my consciousness. The objectifying quality is the actuality of the landscape’s existence, as such, while the non-objectifying quality is a joyful feeling evoked in me by the landscape.

Every intentionality is composed of a nomea and noesis. The nomea is not the real object but the phenomenon, not the tree but the appearance of the tree. The object that appears in perception varies in terms of when it is perceived, from what angle, with what background of experience, with what orientation of wishing, willing, or judging, always from the vantage point of the perceiving individual. . . . The tree is out there present in time and space while the perception of the tree is in consciousness. . . .

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 33/39

Every intentional experience is also noetic. . . .

In considering the nomea-noesis correlate, . . . the “perceived as such” is the nomea; the “perfect self-evidence” is the noesis. Their relationship constitutes the intentionality of consciousness. For every nomea, there is a noesis; for every noesis, there is a nomea. On the noematic side is the uncovering and explication, the unfolding and becoming distinct, the clearing of what is actually presented in consciousness. On the noetic side is an explication of the intentional processes themselves. . . .

Summarizing the challenges of intentionality, the following processes stand out:

1. explicating the sense in which our experiences are directed;

2. discerning the features of consciousness that are essential for the individuation of objects (real or imaginary) that are before us in consciousness (Noema);

3. explicating how beliefs about such objects (real or imaginary) may be acquired, how it is that we are experiencing what we are experiencing (Noesis); and

4. integrating the noematic and noetic correlates of intentionality into meanings and essences of experience. (Moustakas, 1994, pp. 28–32)

Epoche

If those are the challenges, what are the steps for meeting them? The first step in phenomenological analysis is called epoche.

Epoche is a Greek word meaning to refrain from judgment, to abstain from or stay away from the everyday, ordinary way of perceiving things. In a natural attitude we hold knowledge judgmentally; we presuppose that what we perceive in nature is actually there and remains there as we perceive it. In contrast, Epoche requires a new way of looking at things, a way that requires that we learn to see what stands.

In the Epoche, the everyday understandings, judgments, and knowings are set aside, and the phenomena are revisited, visually, naively, in a wide-open sense, from the vantage point of a pure or transcendental ego. (Moustakas, 1994, p. 33)

In taking on the perspective of epoche, the researcher looks inside to become aware of personal bias, eliminate personal involvement with the subject material—that is, eliminate, or at least gain clarity about, preconceptions. Rigor is reinforced by a “phenomenological attitude shift” accomplished through epoche.

The researcher examines the phenomenon by attaining an attitudinal shift. This shift is known as the phenomenological attitude. This attitude consists of a different way of looking at the investigated experience. By moving beyond the natural attitude or the more prosaic way phenomena are imbued with meaning, experience gains a deeper meaning. This takes place by gaining access to the constituent elements of the phenomenon and leads to a description of the unique qualities and components that make this phenomenon what it is. In attaining this shift to the phenomenological attitude, Epoche is a primary and necessary phenomenological procedure.

Epoche is a process that the researcher engages in to remove, or at least become aware of prejudices, viewpoints or assumptions regarding the phenomenon under investigation. Epoche helps enable the researcher to investigate the phenomenon from a fresh and open view point without prejudgment or imposing meaning too soon. This suspension of judgment is critical in phenomenological investigation and requires the setting aside of the researcher’s personal viewpoint in order to see the experience for itself. (Katz, 1987, pp. 36–37)

According to Ihde (1977), “Epoche requires that looking precede judgment and that judgment of what is ‘real’ or ‘most real’ be suspended until all the evidence (or at least sufficient evidence) is in” (p. 36). As such, epoche is an ongoing analytical process rather than a single fixed event. The process of epoche epitomizes the data-based, evidential, and empirical (vs. empiricist) research orientation of phenomenology.

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 34/39

Phenomenological Reduction, Bracketing, and Theme Analysis

Following epoche, the second step is phenomenological reduction. In this analytical process, the researcher brackets out the world and presuppositions to identify the data in pure form, uncontaminated by extraneous intrusions.

Bracketing is Husserl’s (1913) term. In bracketing, the researcher holds the phenomenon up for serious inspection. It is taken out of the world where it occurs. It is taken apart and dissected. Its elements and essential structures are uncovered, defined, and analyzed. It is treated as a text or a document; that is, as an instance of the phenomenon that is being studied. It is not interpreted in terms of the standard meanings given to it by the existing literature. Those preconceptions, which were isolated in the deconstruction phase, are suspended and put aside during bracketing. In bracketing, the subject matter is confronted, as much as possible, on its own terms. Bracketing involves the following steps:

(1) Locate within the personal experience, or self-story, key phrases and statements that speak directly to the phenomenon in question.

(2) Interpret the meanings of these phrases, as an informed reader.

(3) Obtain the subject’s interpretations of these phrases, if possible.

(4) Inspect these meanings for what they reveal about the essential, recurring features of the phenomenon being studied.

(5) Offer a tentative statement, or definition, of the phenomenon in terms of the essential recurring features identified in step 4. (Denzin, 1989b, pp. 55–56)

Imaginative Variation and Textural Portrayal

Once the data are bracketed, all aspects of the data are treated with equal value—that is, the data are “horizontalized.” The data are spread out for examination, with all elements and perspectives having equal weight. The data are then organized into meaningful clusters. Then, the analyst undertakes a delimitation process whereby irrelevant, repetitive, or overlapping data are eliminated. The researcher then identifies the invariant themes within the data to perform an imaginative variation on each theme. This can be likened to moving around a statue to see it from differing views. Through imaginative variation, the researcher develops enhanced or expanded versions of the invariant themes.

Using these enhanced or expanded versions of the invariant themes, the researcher moves to the textural portrayal of each theme—a description of an experience that doesn’t contain that experience (i.e., the feelings of vulnerability expressed by rape victims). The textural portrayal is an abstraction of the experience that provides content and illustration but not yet essence.

Phenomenological analysis then involves a “structural description” that contains the “bones” of the experience for the whole group of people studied, “a way of understanding how the co-researchers as a group experience what they experience” (Moustakas, 1994, p. 142). In the structural synthesis, the phenomenologist looks beneath the affect inherent in the experience to deeper meanings for the individuals who, together, make up the group.

Synthesis and Essence

The final step requires “an integration of the composite textual and composite structural descriptions, providing a synthesis of the meanings and essences of the experience” (Moustakas, 1994, p. 144). In summary, the primary steps of the Moustakas transcendental phenomenological model are as follows: (a) Epoche, (b) phenomenological reduction, (c) imaginative variation, and (d) synthesis of texture and structure. Other detailed analytical techniques are used within each of these stages (see Moustakas, 1994, pp. 180–181).

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 35/39

Heuristic Inquiry According to Moustakas (1990), heuristic inquiry applies phenomenological analysis to one’s own experience. As such, it involves a somewhat different, highly personal analytical process. Moustakas describes five basic phases in the heuristic process of phenomenological analysis: (1) immersion, (2) incubation, (3) illumination, (4) explication, and (5) creative synthesis.

Immersion is the stage of steeping oneself in all that is—of contacting the texture, tone, mood, range, and content of the experience. This state “requires my full presence, to savor, appreciate, smell, touch, taste, feel, know without concrete goal or purpose” (Moustakas, 1988, p. 56). The researcher’s total life and being are centered on the experience. He or she becomes totally involved in the world of the experience—questioning, mediating, dialoging, daydreaming, and indwelling.

The second state, incubation, is a time of “quiet contemplation” where the researcher waits, allowing space for awareness, intuitive or tacit insights, and understanding. In the incubation stage, the researcher deliberately withdraws, permitting meaning and awareness to awaken in their own time. One “must permit the glimmerings and awakenings to form, allow the birth of understanding to take place in its own readiness and completeness” (Moustakas, 1988, p. 50). This stage leads the way toward a clear and profound awareness of the experience and its meanings.

In the phase of illumination, expanding awareness and deepening meaning bring new clarity of knowing. Critical textures and structures are revealed so that the experience is known in all of its essential parameters. The experience takes on a vividness, and understanding grows. Themes and patterns emerge, forming clusters and parallels. New life and new visions appear along with new discoveries.

In the explication phase, other dimensions of meanings are added. This phase involves a full unfolding of the experience. Through focusing, self-dialogue, and reflection, the experience is depicted and further delineated. New connections are made through further explorations into universal elements and primary themes of the experience. The heuristic analyst refines emergent patterns and discovered relationships.

It is an organization of the data for oneself, a clarification of patterns for oneself, a conceptualization of concrete subjective experience for oneself, and integration of generic meanings for oneself, and a refinement of all these results for oneself. (Craig, 1978, p. 52)

What emerges is a depiction of the experience and a portrayal of the individuals who participated in the study. The researcher is ready now to communicate findings in a creative and meaningful way. Creative synthesis is the bringing together of the pieces that have emerged into a total experience, showing patterns and relationships. This phase points the way for new perspectives and meanings, a new vision of the experience. The fundamental richness of the experience and of the experiencing participants is captured and communicated in a personal and creative way. In heuristic analysis, the insights and experiences of the analyst are primary, including drawing on “tacit” knowledge that is deeply internal (Polanyi, 1967).

These brief outlines of phenomenological and heuristic analysis can do no more than hint at the in-depth living with the data that is intended. The purpose of this kind of disciplined analysis is to elucidate the essence of the experience of a phenomenon for an individual or a group. The analytical vocabulary of phenomenological analysis is initially alien, and potentially alienating, until the researcher becomes immersed in the holistic perspective, rigorous discipline, and paradigmatic parameters of phenomenology. As much as anything, this outline reveals the difficulty of defining and sequencing the internal intellectual processes involved in qualitative analysis more generally.

Phenomenology seeks to describe, elucidate, and interpret human experience. The product is a deep understanding of the nature and essence of the phenomenon studied. We turn now to a quite different analytical priority: not just describing and interpreting but also explaining the world.

The Hermeneutic Circle and Interpretation

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 36/39

Hermes was messenger to the Greek gods. . . . Himself the god of travel, commerce, invention, eloquence, cunning, and thievery, he acquired very early in his life a reputation for being a precocious trickster. (On the day he was born he stole Apollo’s cattle, invented the lyre, and made fire.) His duties as messenger included conducting the souls of the dead to Hades, warning Aeneas to go to Italy, where he founded the Roman race, and commanding the nymph Calypso to send Odysseus away on a raft, despite her love for him. With good reason his name is celebrated in the term “hermeneutics,” which refers to the business of interpreting. . . . Since we don’t have a godly messenger available to us, we have to interpret things for ourselves. (Packer & Addison, 1989, p. 1)

©2002 Michael Quinn Patton and Michael Cochran

Heuristic Inquiry Reactivity

Hermeneutics focuses on interpreting something of interest, traditionally a text or work of art; but in the larger context of qualitative inquiry, it has also come to include interpreting interviews and observed actions. The emphasis throughout concerns the nature of interpretation, and various philosophers have approached the matter differently, some arguing that there is no method of interpretation per se because everything involves interpretation (Schwandt, 2000, 2001). For our purposes here, the hermeneutic circle, as an analytical process aimed at enhancing understanding, offers a particular emphasis in qualitative analysis, namely, relating parts to wholes and wholes to parts.

Construing the meaning of the whole meant making sense of the parts, and grasping the meaning of the parts depended on having some sense of the whole. . . . The hermeneutic circle indicates a necessary condition of interpretation, but the circularity of the process is only temporary—eventually the interpreter can come to something approximating a complete and correct understanding of the meaning of a text in which whole and parts are related in perfect harmony. Said somewhat differently, the interpreter can, in time, get outside of or escape the hermeneutic circle in discovering the “true” meaning of the text. (Schwandt, 2001, p. 112)

The method involves playing the strange and unfamiliar parts of an action, text, or utterance off against the integrity of the action, narrative, or utterance as a whole until the meaning of the strange passages and the meaning of the whole are worked out or accounted for. (Thus, for example, to understand the meaning of the first few lines of a poem, I must have a grasp of the overall meaning of the poem, and vice versa.) In this process of applying the hermeneutic method, the interpreter’s self-understanding and socio-historical location neither affects nor is affected by the effort to interpret the meaning of the text or utterance. In fact, in applying the method, the interpreter abides by a set of procedural rules that help insure that the interpreter’s historical situation does not distort the bid to uncover the actual meaning embedded in the text, act, or utterance, thereby helping to insure the objectivity of the interpretation (Schwandt, 2001, p. 114).

The circularity and universality of hermeneutics (every interpretation is layered in and dependent on other interpretations, like a series of dolls that fit one inside the other, and then another and another) pose for the qualitative analyst the problem of where to begin. How and where do you break into the hermeneutic circle of interpretation? Packer and Addison (1989), in adapting the hermeneutic circle as an inquiry approach for psychology, suggest beginning with “practical understanding”:

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 37/39

Practical understanding is not an origin for knowledge in the sense of a foundation; it is, instead, the starting place for interpretation. Interpretive inquiry begins not from an absolute origin of unquestionable data or totally consistent logic, but at a place delineated by our everyday participatory understanding of people and events. We begin there in full awareness that this understanding is corrigible, and that it is partial in the twin senses of being incomplete and perspectival. Understanding is always moving forward. Practical activity projects itself forward into the world from its starting place, and shows us the entities we are home among. This means that neither commonsense nor scientific knowledge can be traced back to an origin, a foundation. . . . (p. 23)

The circularity of understanding, then, is that we understand in terms of what we already know. But the circularity is not, Heidigger argues, a “vicious” one where we simply confirm our prejudices, it is an “essential” one without which there would be no understanding at all. And the circle is complete; there is accommodation as well as assimilation. If we are persevering and open, our attention will be drawn to the projective character of our understanding and—in the backward arc, the movements of return—we gain an increased appreciation of what the forestructure involves, and where it might best be changed. . . . (p. 34)

Hermeneutic inquiry is not oriented toward a grand design. Any final construction that would be a resting point for scientific inquiry represents an illusion that must be resisted. If all knowledge were to be at last collected in some gigantic encyclopedia this would mark not the triumph of science so much as the loss of our human ability to encounter new concerns and uncover fresh puzzles. So although hermeneutic inquiry proceeds from a starting place, a self-consciously interpretive approach to scientific investigation does not come to an end at some final resting place, but works instead to keep discussion open and alive, to keep inquiry under way. (p. 35)

At a general level and in a global way, hermeneutics reminds us of the interpretive core of qualitative inquiry, the importance of context and the dynamic whole–part interrelations of a holistic perspective. At a specific level and in a particularistic way, the hermeneutic circle offers a process for formally engaging in interpretation.

Theory-Driven Qualitative Findings This module has looked in some depth at how two theory-based inquiry perspectives, phenomenology and hermeneutics, prescribe different analytical processes and produce different kinds of findings. To further emphasize this point, Exhibit 8.17 highlights how 10 different theoretical perspectives yield different kinds of findings due to the distinct focus of inquiry embedded in each theoretical perspective. Ethnography directs the inquiry to elucidate the nature of culture, whether for tribes, organizations, or programs. Social constructionism captures mental models and worldviews. Realism documents contextually operative causal mechanisms. Theories of change differentiate patterns of change and change trajectories. Systems theory illuminates interrelationships. Complexity theory invites studies of adaptation patterns, emergence, and simple rules. Phenomenology aims to elucidate the essence of the phenomenon studied. Grounded theory inquiries yield theoretical propositions and hypotheses. Hermeneutics interprets the meanings of texts. Pragmatism enquires into how things work and how they are used. Thus, operating within a particular theoretical orientation (see Chapter 3) provides focus, inquiry processes, and analytical procedures and yields certain kinds of findings that are a matter of core interest and priority for the community of inquiry engaged in studying the world through the lenses offered by that shared theoretical perspective. Exhibit 8.17 cites research and evaluation examples illuminating the differences in types of findings among these theoretical orientations.

EXHIBIT 8.17 Findings Yielded by Various Theoretical Perspectives With Research and Evaluation Examples

Different theoretical perspectives yield different kinds of findings due to the distinct focus of inquiry embedded in each theoretical perspective. Conducting research or evaluation within a particular theoretical orientation provides focus, inquiry processes, and analytical procedures and yields certain kinds of findings that are a matter of core interest and priority for the community of inquiry engaged in

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 38/39

studying the world through the lenses offered by that shared theoretical perspective. This exhibit cites research and evaluation examples illuminating the differences in types of findings among 10 theoretical orientations.

6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice

https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 39/39