U7D1-68 - **READ AND FOLLOW ALL INSTRUCTIONS AS OUTLINED BELOW.. Research Methodology Peer Review.

CHAPTER

5 Designing Qualitative Studies

Symbol of continuous quest for knowledge NEA ONNIM NO SUA A, OHU One who does not know can learn.

—Arthur (2001) Cloth as Metaphor: (Re)reading the Adinkra Cloth Symbols of the Akan

of Ghana

A Design Is a Plan There’s a reason you’re undertaking the study you’re going to do. You have a purpose. You presumably have a question or questions you want to answer in keeping with that purpose. The design sets forth how you will fulfill your purpose and answer the questions you’ve identified. A design is a plan. As John Steinbeck has his character assert at a moment of desperation in the Great Depression novel Of Mice and Men, “A plan is a real thing.” It sets direction. It gets you moving into the field. But it is also a flexible and emergent thing. The tension between following the plan you create at the beginning and adapting it as you learn along the way means that design is not a contained, bounded, do-it-and-be-done step. Yes, you do it. Then, you implement it. But it’s not a mechanical, linear, set-in-stone plan. Rather, design is a process and a way of thinking.

You don’t just do a design. You think design. You engage design. You follow the Rule of the Loop (Schell, 2008, p. 80): The more times you test and improve your design, the better it will be. Better, but not perfect. As you think through design options and their implications, improve your design, and think through the improvements and their implications, you come to know deeply, even intimately, your design’s strengths and limitations. Designs are inevitably constrained by limited resources, time, and the complexities of the real world that do not yield easily to our design parameters. But thoughtful designs are also laden with the energy of potential. Feed on that energy. You are creating something. Something real, palpable. A plan is a real thing. A design is a real thing. Be guided by the Thomas theorem: What is perceived as real is real in its consequences. Designs have consequences. Your design will determine what you learn. Your design will accompany you as a fellow traveler throughout your inquiry. You’ll think you understand it well at the beginning, but as the design unfolds, becomes data, guides analysis, and turns into findings, you’ll find that you’ve come a long way together, and that you’ve changed, and developed and learned together. So prepare for design immersion. Think design. Engage design. Create a worthy fellow traveler to both guide and accompany you throughout your inquiry.

Chapter Preview

Module 28 takes a deep dive into design thinking. Inquiry questions derive from the purpose, and design answers questions. Module 29 examines data collection options, parameters, and decisions, which leads to Module 30 on purposeful sampling and case selection. This is a critical design discussion because what you sample is what you have something to say about at the end. Strategic and purposeful case selection is hugely important, hugely misunderstood, and therefore often hugely controversial. This module sets the stage for the most comprehensive presentation of qualitative case selection options ever assembled, a recognition of and tribute to the flowering of qualitative applications and approaches in the past decade.

Module 31 Single-Significant-Case Sampling as a Design Strategy Module 32 Comparison-Focused Sampling Options Module 33 Group Characteristics Sampling Strategies and Options Module 34 Concept and Theoretical Sampling Strategies and Options Module 35 Instrumental-Use Multiple-Case Sampling Module 36 Sequential and Emergence-Driven Sampling Strategies and Options Module 37 Analytically Focused Sampling Module 38 Mixed, Stratified, and Nested Purposeful Sampling Strategies Module 39 Information-Rich Cases

The final three modules discuss sample size for qualitative designs and mixed-methods designs, with a concluding review of methods choices and decisions.

This chapter is the pivot chapter of both the book and qualitative inquiry. Chapters 1 through 4 presented examples of qualitative studies, the 12 core strategies of qualitative inquiry, diverse theoretical traditions that inform alternative frameworks for qualitative methodology, and a panorama of practical applications. Chapters 6 thorough 9 will discuss data collection and analysis. Chapter 5 is the pivot chapter on design; it builds on the foundation of the previous chapters and anticipates the fieldwork and analysis of the remaining chapters. Designs must be theoretically and conceptually strong; methodologically feasible, rigorous, and credible; and appropriate to the inquiry question, the available resources, the context for the study, and your own interests, capacities, and capabilities. Here’s where the rubber hits the road, where it gets real, where trade- offs are negotiated, where competing ideas funnel into a design, and where the nature of the eventual results is determined.

So prepare for design immersion. Think design. Engage design. Game on.

MODULE

28 Design Thinking: Questions Derive From Purpose, Design Answers Questions

The First Evaluation, the First Design The young people gathered around Halcolm. “Tell us again, Teacher of Many Things, about the first evaluation.”

“The first evaluation,” he began, “was conducted a long, long time ago, in Ancient Babylon when Nebuchadnezzar was King. Nebuchadnezzar had just conquered Jerusalem in the third year of the reign of Jehoiakim, King of Judah. Now Nebuchadnezzar was a shrewd ruler. He decided to bring carefully selected children of Israel into the palace for special training so that they might be more easily integrated into Chaldean culture. This special program was the forerunner of the gifted- and-talented education programs that would become so popular in the twentieth century. The three- year program was royally funded with special allocations and scholarships provided by Nebuchadnezzar. The ancient text from the Great Book records that

the king spake unto Ashpenaz the master of his eunuchs that he should bring certain of the children of Israel, and of the King’s seed, and of the princes; Children in whom was no blemish, but well-favored and skillful in all wisdom, and cunning in knowledge, and understanding science, and such as had ability in them to stand in the king’s palace, and whom they might teach the learning and the tongue of the Chaldeans.

And the king appointed them a daily provision of the king’s meat, and of the wine which he drank; so nourishing them three years, that at the end thereof they might stand before the king. (Daniel 1:3–5)

“Now this program had scarcely been established when the program director, Ashpenaz, who happened also to be prince of the eunuchs, found himself faced with a student rebellion led by a radical named Daniel, who decided for religious reasons that he would not consume the king’s meat and wine. This created a serious problem for the director. If Daniel and his coconspirators did not eat their dormitory food, they might fare poorly in the program and endanger not only future program funding but also the program director’s head! The Great Book says:

But Daniel purposed in his heart that he would not defile himself with the portion of the king’s meat, nor with the wine which he drank; therefore he requested of the prince of the eunuchs that he might not defile himself.

And the prince of the eunuchs said unto Daniel, I fear my lord the king, who hath appointed your meat and your drink; for why should he see your faces worse liking than the children which are of your sort? Then shall ye make me endanger my head to the king. (Daniel 1:8, 10)

“At this point, Daniel proposed history’s first educational experiment and program evaluation. He and three friends (Hananiah, Mishael, and Azariah) asked to be placed on a strict grain legume (pulse) and water diet for ten days, while other students continued on the king’s rich diet of meat and wine. At the end of ten days the program director would inspect the treatment group for any signs of physical deterioration and judge the value of Daniel’s alternative diet plan. Daniel proposed the experiment thusly:

Prove thy servants, I beseech thee, ten days; and let them give us pulse to eat, and water to drink. Then let our countenances be looked upon before thee, and the countenance of the children that eat of the portion of the king’s meat: and as thou seest, deal with thy servants.

So he consented to them in this matter, and proved them ten days. (Daniel 1:12–14)

“During the ten days of waiting Ashpenaz had a terrible time. He couldn’t sleep, he had no appetite, and he had trouble working because he was preoccupied worrying about how the evaluation would turn out. He had a lot at stake. Besides, in those days they hadn’t quite worked out the proper division of labor so he had to play the roles of both program director and evaluator. You see. . . . ”

The young listeners interrupted Halcolm. They sensed that he was about to launch into a sermon on the origins of the division of labor when they still wanted to hear the end of the story about the origins of evaluation. “How did it turn out?” they asked. “Did Daniel end up looking better or worse from the new diet? Did Ashpenaz lose his head?”

“Patience, patience,” Halcolm pleaded. “Ashpenaz had no reason to worry. The results were quite amazing. The Great Book says that

at the end of ten days their countenances appeared fairer and fatter in flesh than all the children which did eat the portion of the king’s meat.

Thus Melzar took away the portion of their meat, and the wine that they should drink; and gave them pulse.

As for these four children, God gave them knowledge and skill in all learning and wisdom; and Daniel had understanding in all visions and dreams. Now at the end of the days that the king had said he should bring them in, then the prince of the eunuchs brought them in before Nebuchadnezzar. And in all matters of wisdom and understanding, that the king inquired of them, he found them ten times better than all the magicians and astrologers that were in all his realm. (Daniel 1:15–18, 20)

“And that, my children, is the story of the first evaluation. Those were the good old days when evaluations really got used. Made quite a difference to Ashpenaz and Daniel. Now off with you—and see if you can do as well.”

—From Halcolm’s Evaluation Histories

A Meta-evaluation A meta-evaluation is an evaluation of an evaluation. A great deal can be learned about qualitative designs by conducting a meta-evaluation of history’s first program evaluation. Let us imagine a panel of experts conducting a rigorous critique of this evaluation of Babylon’s compensatory education program for Israeli students:

1. Small sample size (n = 4) 2. Selectivity bias, because recruitment into the program was done by “creaming,” that is, only

the best prospects among the children of Israel were brought into the program 3. Sampling bias, because students were self-selected into the treatment group (diet of pulse and

water)

4. Failure to clearly specify and control the nature of the treatment, thus allowing for the possibility of treatment contamination because we don’t know what other things, apart from a change in diet, either group was involved in that might have explained the outcomes observed

©2002 Michael Quinn Patton and Michael Cochran Observing Countenance

5. Possibility of interaction effects between the diet and the students’ belief system (i.e., potential Hawthorne and halo effects)

6. Outcome criteria vague: just what is “countenance” 7. Outcome measurement poorly operationalized and nonstandardized 8. Single observer with deep personal involvement in the program, introducing the possibility of

selective perception and bias in the observations 9. Validity and reliability data not reported for the instruments used to measure the final outcome

(“He found them ten times better than all the magicians and astrologers”) 10. Possible reactive effects from the students’ knowledge that they were being evaluated

(Hawthorne and halo effects)

Despite all of these threats to internal validity, not to mention external validity, the information generated by the evaluation appears to have been used. The 10-day evaluation was used to make a major decision about the program, namely, to change the diet for Daniel and his friends. The end- of-program evaluation conducted by the king was used to judge the program a success. (Daniel was placed first in his class.) Indeed, it would be difficult to find a more exemplary model for the uses of evaluation in making educational policy decisions than this “first evaluation” conducted under the auspices of Nebuchadnezzar so many years ago. This case study is an exemplar of evaluation research having an immediate, decisive, and lasting impact on an educational program. Modern evaluation researchers, flailing away in seemingly futile efforts to affect contemporary governmental decisions, can be forgiven a certain nostalgia for the “good old days” in Babylon when evaluation really made a difference.

But should the results have been used? Given the apparent weakness of the evaluation design, was it appropriate to make a major program decision on the basis of data generated by such a seemingly weak research design?

I would argue that not only was use impressive in this case, it was also appropriate because the research design was exemplary. Yes, exemplary, because the study was set up in such a way as to provide precisely the information needed by the program director to make the decision he needed to make. Certainly, it is a poor research design to study the relationship between nutrition and educational achievement. It is even a poor design to decide if all students should be placed on a vegetarian diet. But those were not the issues. The question the director faced was whether to place four specific students on a special diet at their request. The information he needed concerned the consequences of that specific change and only that specific change. He showed no interest in generalizing the results beyond those four students, and he showed no interest in convincing others that the measures he made were valid and reliable. Only he and Daniel had to trust the measures used, and so data collection (observation of countenance) was done in such a way as to be meaningful and credible to the primary intended evaluation users, namely, Ashpenaz and Daniel. If any bias existed in his observations, given what he had at stake, the bias would have operated against a demonstration of positive outcomes rather than in favor of such outcomes.

While there are hints of whimsy in the suggestion that this first evaluation was exemplary, I do not mean to be completely facetious. I am serious in suggesting that the Babylonian example is an exemplar of utilization-focused evaluation. It contains and illustrates all the factors modern evaluation researchers have verified as critical from studies of utilization (Patton, 2008b, 2012a). The decision makers who were to use the findings generated by the evaluation were clearly identified and were deeply involved in every stage of the evaluation process. The evaluation question was carefully focused on needed information that could be used in the making of a specific decision. The evaluation methods and design were appropriately matched to the evaluation question. The results were understandable, credible, and relevant. Feedback was immediate, and utilization was decisive. Few modern evaluations can meet the high standards for evaluation set by Ashpenaz and Daniel more than 3,000 years ago.

This chapter discusses some ways in which evaluation and research designs can be appropriately matched to inquiry questions, in an attempt to emulate the exemplary match between evaluation problem and research design achieved in the Babylonian evaluation. As in the previous chapters, I shall emphasize the importance of being both strategic and practical in creating evaluation and research designs. Being strategic begins with being clear about the purpose of the intended research or evaluation.

Clarity About Purpose: A Typology Purpose is the controlling force in research. Decisions about design, measurement, analysis, and reporting all flow from purpose. Therefore, the first step in a research process is getting clear about purpose. The centrality of purpose in making methods decisions becomes evident from examining alternative purposes along a continuum from theory to action.

1. Basic research: contribute to fundamental knowledge and theory 2. Applied research: illuminate a societal concern or problem in the search for solutions 3. Summative evaluation: determine if a solution (policy or program) works 4. Formative evaluation: improve a policy or program as it is being implemented 5. Action research: understand and solve a specific problem as quickly as possible

Basic and applied researchers publish in scholarly journals, where their audience is other researchers who will judge their contributions using disciplinary standards of rigor, validity, and theoretical import. In contrast, evaluators and action researchers publish reports for specific stakeholders who will use the results to make decisions, improve programs, and solve problems.

Standards for judging quality vary among these five different types of research. Expectations and audiences are different. Reporting and dissemination approaches are different. Because of these differences, the researcher must be clear at the beginning about which purpose has priority. No single study can serve all of these different purposes and audiences equally well. With clarity about purpose and primary audience, the researcher can go on to make specific design, data-gathering, and analysis decisions to meet the priority purpose and to address the intended audience.

In the Babylonian example, the purpose was simply to find out if a vegetarian diet would negatively affect the healthy appearances (countenances) of four participants—not why their countenances appeared healthy or not (a causal question) but whether the dietary change would affect countenance (a descriptive question). The design, therefore, was appropriately simple to yield descriptive data for the purpose of making a minor program adjustment. No contribution to general knowledge. No testing or development of theory. No generalizations. No scholarly publication. No elaborate report on methods. Just find out what would happen to inform a single decision about a possible program change. The participants in the program were involved in the study; indeed, the idea of putting the diet to an empirical test originated with Daniel. In short, we have a very nice example of simple formative evaluation.

The king’s examination of the program participants at the end of three years was quite different. We might infer that the king was judging the overall value of the program. Did it accomplish his objectives? Should it be continued? Could the outcomes he observed be attributed to the program? This is the kind of study we have come to call summative evaluation—summing up judgments about a program to make a major decision about its value, whether it should be continued, and whether the demonstrated model can or should be generalized to and replicated for other participants or in other places.

Now imagine that researchers from the University of Babylon wanted to study the diet as a manifestation of culture in order to develop a theory about the role of diet in transmitting culture. Their sample, their data collection, their questions, the duration of fieldwork, and their presentation of results would all be quite different from the formative evaluation undertaken by Ashpenaz and Daniel. The university study would have taken much longer than 10 days and might have yielded empirical generalizations and contributions to theory, yet it would not have helped Ashpenaz make his simple decision. On the other hand, we might surmise that University of Babylon scholars would have scoffed at a study done in just 10 days with such problematic (from their perspective) measures. Different purposes. Different criteria for judging the research contribution. Different methods. Different audiences. Different kinds of inquiry.

These are examples of how purpose can vary. This is not the only typology of purpose distinctions, but it serves to illustrate and emphasize the critical importance of matching design to purpose and audience. Previous chapters have presented the nature and strategies of qualitative inquiry, its philosophical and theoretical foundations and practical applications. In effect, you have been presented with a large array of options, alternatives, and variations. How do you sort it all out to decide what to do in a specific study? The answer is to get clear about purpose.

Exhibit 5.1 summarizes some of the major differences among the different kinds of research and evaluation. The framework provided in Exhibit 5.1 is meant to facilitate clarity about purpose and the implications for design, quality criteria, use of findings, and publication. The framework also

illustrates how one can organize a mass of observations into a coherent typology—a major analytical tool of qualitative inquiry.

©2002 Michael Quinn Patton and Michael Cochran Are they doing research or evaluation?

The Purpose of Purpose Distinctions Different purposes typically lead to different ways of conceptualizing problems, different designs, different types of data gathering, and different ways of publicizing and disseminating findings. Researchers engaged in inquiry at various points along the continuum can have very strong opinions and feelings about researchers at other points along the continuum, sometimes generating opposing opinions and strong emotions. Basic and applied researchers, for example, would often dispute calling formative and action research by the name “research.” The standards that basic researchers apply to what they would consider “good” research exclude even some applied research because it may not manifest the conceptual clarity and theoretical rigor in real-world situations that basic researchers value. Formative and action researchers, on the other hand, may attack basic research for being esoteric, academic, and irrelevant.

EXHIBIT 5.1 A Typology of Research Purposes

EXHIBIT 5.2 Family Research Example: Research Questions Matched to Research Category

Debates about the meaningfulness, rigor, significance, and relevance of various approaches to research are regular features of university life. On the whole, within universities and among scholars, the status hierarchy in science attributes the highest status to basic research, secondary status to applied research, little status to summative evaluation research, and virtually no status to formative and action research. The status hierarchy is reversed in real-world settings, where people with problems attribute the greatest significance to action and formative research that can help them

solve their problems in a timely way and attach the least importance to basic research, which they consider remote and largely irrelevant to what they are doing on a day-to-day basis.

Examples of Types of Research Questions: A Family Research Example To further clarify these distinctions, it may be helpful to take a particular issue and look at how it would be approached for each type of research. For illustrative purposes, let’s examine the different kinds of questions that can be asked about families for different inquiry purposes. All of the inquiry questions in Exhibit 5.2 focus on families, but the purpose and focus of each type of inquiry are quite different. With clarity about purpose, it is possible to turn to formulating the overarching inquiry questions.

With the overarching inquiry questions formulated, the design can be constructed. The design specifies what data will be collected to answer the inquiry questions. This will immediately raise challenges of choosing among alternative strategies and methods, including decisions about critical trade-offs in design, our next topic.

Framing Qualitative Inquiry Questions

Judge a man by his questions rather than his answers. —Voltaire (1694–1778)

French Enlightenment philosopher

Every year, I review scores of research and evaluation proposals with a range of purposes on a wide variety of topics. A common weakness, I find, is how the inquiry question is framed. Here are the seven most common forms of feedback I find myself giving about formulating the qualitative inquiry question.

1. First the question, then the methods, then back to the question: Question formulation is not a simple sequential process. You begin with a problem statement or question, but as you move to how to answer the question, the methods deliberations are likely to reshape and sharpen the question. Robert Stake (2010) has described the iterative and evolving nature of question formulation in qualitative inquiry:

SIDEBAR

FAMILY STORIES SUPPORT RESILIENCE

Marshall Duke, a psychologist at Emory University, and Robyn Fivush, director of Emory’s Family Narratives Lab, ask school- children 20 questions about their families. They have found that kids who know the most about their families tend to be the most resilient when they face adversity. Here are some examples of the questions they ask:

Do you know where your grandparents grew up?

Do you know where your parents went to school and how they met?

Do you know the story of your birth?

Duke and Fivush have found that children who know their family history have a strong sense of their “intergenerational selves” and feel that they belong to something bigger than themselves. Family storytelling, they hypothesize, may be a key to resilience in children (Kurylo, 2013). Happy families have stories that bind them together (Feiler, 2013a, 2013b).

Better, first, to ask what you need to know; then how to go about finding it. (p. 72)

For most of us, most of the time, the research problem should have first priority—but a question cannot be conceptualized without some thought of method and place of study. One cannot think deeply about the content of research without thinking of its meanings as studied one way or another. And the reality of studying it one place rather than others quickly forms in our minds. In other words, first conceptualization of the study happens pretty much all together, the focus shifting from question to method to place and back to question, each time hopefully refining the idea. And the refining will continue well into the time you are gathering data and writing up patches for the report. (p. 74)

2. Frame a specific study’s inquiry questions within a larger inquiry context and make the linkage explicit: Chapter 3 reviewed a variety of paradigmatic, philosophical, and theoretical frameworks like phenomenology, ethnography, hermeneutics, or systems and complexity theory. These distinct frameworks serve to provide an inquiry context that should guide the formulation of specific study questions. Yet I regularly see proposals and reports that discuss both the overarching theoretical framework and a study’s specific focus but don’t connect the two. Each inquiry framework in Chapter 3 is distinguished by a core question. The question asked by a specific study working within that general framework should be derived from and informed by that overarching question. Likewise, the practical and actionable inquiry frameworks in Chapter 4 also provide overarching perspectives and inquiry traditions that can and should inform specific study questions that serve a pragmatic and actionable purpose. The degree of alignment between an overarching inquiry framework or purpose and a specific study focus constitutes a test of the relevance of the general framework to guide the specific inquiry you are proposing as well as the likelihood that your specific study will contribute to knowledge within a more general and established inquiry tradition.

3. Formulate one or more questions to guide your overall inquiry: I see lots of proposals that muse about a journey of inquiry without ever bringing the study into focus with actual questions. Here’s a recent example:

I want to study leadership by engaging with some leaders about leadership and inviting them to reflect on and share their leaderships perceptions and experiences. It’s important to capture the voices of leaders about leadership and the best way to do that is to interview them about leadership so that they can express how they view leadership and what they’ve learned.

And the overarching question is what? We can infer the question. It’s implicit. But reread the proposed inquiry. What would you say is the overarching inquiry question? I’ve used this example in qualitative methods workshops and asked participants to formulate an overall inquiry question based on the musings above. They come up with different questions. What questions you ask

matters. Questions determine the realm where you’ll be traveling on your inquiry journey—and the nature and range of the answers you’ll find. Ask carefully. Ask thoughtfully. But most of all, ask.

4. Ask open-ended questions: Qualitative inquiry begins with descriptive questions. Avoid dichotomous (yes/no) questions. This is true in qualitative interviewing (Chapter 7), and it is equally true in framing the overall inquiry. Surveys that grammatically pose yes/no questions are inherently overly simplistic and reflect narrow categorical thinking that is inconsistent with the complexity and richness of qualitative inquiry, not to mention the complex and diverse nature of the world. It’s the difference between asking,

“Do university students rely on social media to nurture their primary relationships?”

versus

“How do university students use social media with their primary relationships?”

Linger for a moment on the huge differences between these two questions. Think about what kind of data would most appropriately answer each question. Think about what the results would look like. Think about the thinking process involved in engaging these two questions.

Asking dichotomous questions invites Type III errors: getting the right answer to the wrong question. A Type I error is concluding that there is a difference between groups when there is not. A Type II error is concluding that there is no difference between groups when there is. These errors are associated with statistical analysis based on significance tests, though conceptually, these errors can occur in qualitative analysis. A Type III error, however, is more fundamental than drawing an incorrect inference. A Type III error is introduced right at the beginning of an inquiry by asking the wrong question. Asking dichotomous questions constitutes one such error in qualitative inquiry: the right answer for the wrong question. Exhibit 5.3 gives examples of open-ended questions compared with dichotomous questions.

5. Avoid a laundry list of questions: I recently reviewed a proposal that listed 30 questions. This would have been too much for a detailed interview protocol much less for an overall inquiry proposal. This researcher had fallen into an all-too-common trap of believing that asking lots and lots of questions showed an inquisitive mind. What it actually showed was a lack of overall direction. It is about such people that the observation is made that he or she “can’t see the forest for the trees.” When I encounter a proposal with a laundry list of questions, my feedback is “Less is more.” And I often share the guidance of American writer Richard Bach, which is both instructional and inspirational: “You don’t want a million answers as much as you want a few forever questions. The questions are diamonds you hold in the light.”

EXHIBIT 5.3 Asking Open-Ended Questions

Qualitative Inquiry Begins With Descriptive Questions. Avoid Dichotomous (Yes/No) Questions.

6. Distinguish questions from hypotheses: The language of hypothesis testing is quantitative language. Statistical analysis is built around hypothesis testing. Statistical significance tests reveal whether the null hypothesis is confirmed or rejected and, in either case, with what degree of confidence. There are precise rules for statistical hypothesis testing. Qualitative analysis lacks both those rules and that degree of precision. Certainly, there are qualitative analysts working comfortably within the positivist tradition who unabashedly use the quantitative language of independent and dependent variables and hypothesis testing (e.g., George & Bennett, 2005; Gerring, 2007; King, Keohane, & Verba, 1994). But my preference (and advice) is to leave the language of hypothesis testing to quantitative/experimental designs, where it has specific, agreed-on meaning. Qualitative inquiry explores questions. We do not test hypotheses in the way that phrase is normally understood, so to use the language of hypothesis testing invites both confusion and criticism. An alternative phrase preferred by some is foreshadowed problems: “a qualitative version of hypotheses . . . in which the researcher enters a setting with topics to explore” (Hays & Singh, 2012, pp. 41, 423). I think it’s simplest to articulate questions.

7. Questions can evolve: Questions are beginning points. Don’t treat them as written in cement. All aspects of qualitative inquiry can be emergent, including inquiry questions. Part of the inquiry journey can be discovering new questions. In evaluating a Minnesota program for high school dropouts, we began with the staff’s concern about how to better serve a sudden influx of recent immigrants. The program was individualized and portfolio based so that students could pursue their own interests. The purpose of the inquiry was to adapt the program to this new, diverse population. The developmental evaluation question was “What do immigrants want to learn?”

In interviews and focus groups, we found that immigrants were confused by the question. Why were they being asked what they wanted to learn? They had enrolled in the program to get a high school degree. Didn’t the American teachers know what they were supposed to learn? They came from countries where educational authorities and teachers dictated the curriculum. It was unsettling to them that they, new to the United States, were being asked what they should learn.

We asked them what questions they thought program staff and the evaluation should ask. The following questions emerged: In what ways, with what impacts, can the program support immigrants to share and build on their experiences and knowledge? How can Minnesotans learn and benefit from interactions with recent immigrants? The program staff had not imagined asking those questions, but once they emerged during the inquiry, the staff embraced them, and the evaluation shifted focus.

Inquiry as Asking and Pursuing Answers to Questions

The art and science of asking questions is the source of all knowledge. —Thomas Berger

American writer

There’s a lot of ways to take a lot of data, mangle what you’re doing with it, not ask good questions, and get yourself in trouble. . . . People blame the data, when they should be asking better questions.

—Nate Silver (2012) The Signal and the Noise: Why So Many Predictions Fail—but Some Don’t

Questions matter. The quality of questions matters. The thoughtfulness of questions matters. It’s not true that there are no dumb questions. I see them in proposals and reports all the time. Over more than 40 years of reviewing designs, I have observed a huge and varied panorama of sentences that have one thing in common: They end with a question mark. As I’ve provided feedback on revising questions, and received both appreciation and pushback, I’ve concluded that dumb questions derive from ignorance and thoughtlessness, not stupidity. After reading this section, you can no longer plead ignorance and have no excuse for thoughtlessness. So, by way of review, here’s my short, nonexhaustive list of dumb questions.

Dichotomous questions are dumb in that they frame the complexity of the world as reducible to yes/no simplicities. Laundry lists of questions are dumb in that they confuse and complicate the inquiry rather than guiding it. Treating questions as rigid and unalterable is dumb because qualitative inquiry is a journey of discovery, and that includes learning what deeper questions to ask as the inquiry unfolds. Exhibit 5.3 (p. 253) compares appropriately worded, open-ended questions with inappropriate, closed-ended questions.

Designs are built around the questions we ask. Then, understanding, insight, and knowledge emerge from inquiry into the questions we ask. That means determining what data to collect and what cases to study.

MODULE

29 Data Collection Decisions

Constructing a research design successfully means to define who or what shall be studied (and who or what shall not).

—Flick (2007a, p. 44)

Qualitative inquiry collects data from in-depth interviews, focus groups, open-ended questions on surveys, postings in social media, direct observations in the field, and analysis of documents. A mixed-methods design can add any of the broad panorama of quantitative data to be collected in conjunction with qualitative data. Subsequent chapters will examine these specific data collection approaches in depth. We’ll look at different ways of making observations and engaging in fieldwork. We’ll examine options for interviewing and the implications of different approaches. First, however, we’re going to look at the strategic design decisions that must be made about the depth of inquiry given the resource and time constraints.

Nature of Data Collection: One Point in Time Versus Longitudinal Inquiry Two fundamentally different approaches to data collection involve how much contact the inquirer will have with the people and places from which data will be collected. Exhibit 5.4 compares these approaches. The one-point-in-time approach involves one interview per person or one site visit per place. For example, it is quite common in a program evaluation to interview a sample of program participants at the end of the program to document their experiences, reactions, and outcomes. The end-of-program interview is the only point of contact with these participants.

A longitudinal study, in contrast, involves multiple points of contact over some period of time. Again, using a program evaluation example, a longitudinal study would interview participants several times: (a) at the beginning of the program to document their backgrounds and expectations; (b) once or more during the program to find out how the experience is unfolding, the reactions to the experience, and what the participant is getting from the program; (c) at the end of the program to capture perceptions and outcomes as the program experience concludes; and (d) one or more follow-up interviews to get at lasting impacts and digested perceptions.

EXHIBIT 5.4 Contrasting Designs: One-Point-in-Time Data Collection Versus Longitudinal Data Collection

SIDEBAR

THE ULTIMATE LONGITUDINAL STUDY

In 1938, Harvard University began following 268 male undergraduate students in what became the longest-running longitudinal study of human development in history. The study aimed to determine what factors contribute most strongly to “human flourishing.” The men were followed into their 90s, documenting life from college through the years until long after retirement and studying a wide range of experiences, including relationships, politics, religion, coping strategies, and health, both physical and mental.

Highlights of the Findings:

• Those who do well in old age did not necessarily prosper in midlife, and vice versa. • Recovery from an unhappy childhood is possible, but a happy childhood is a source of

strength throughout life. • Marriages bring much more contentment after age 70. • Physical aging after 80 is determined less by heredity than by habits formed prior to age 50.

—George E. Vaillant (2013) Triumphs of Experience

These two quite different inquiry strategies have major time and resource implications. Each has strengths and weaknesses. Deciding which design to implement involves trade-offs between depth and breadth.

Critical Trade-Offs in Design Purposes, strategies, data collection options, and trade-offs—these themes go together. A discussion of design strategies and trade-offs is necessitated by the fact that there are no perfect research designs. There are always trade-offs. Limited resources, limited time, and limits on the human ability to grasp the complex nature of social reality necessitate trade-offs.

The very first trade-offs come in framing the research or evaluation questions to be studied. The problem here is to determine the extent to which it is desirable to study one or a few questions in great depth or to study more questions (though not a laundry list) but in less depth—the “boundary problem” in naturalistic inquiry (Guba, 1978). Once a potential set of inquiry questions has been generated, it is necessary to begin the process of prioritizing those questions in order to decide which of them ought to be pursued. For example, for an evaluation, should all parts of the program be studied or only certain parts? Should all clients be interviewed or only some subset of clients? Should the evaluator aim to describe all program processes or only certain selected processes in depth? Should all outcomes be examined or only certain outcomes of particular interest to inform a pending decision? These are questions that are discussed and negotiated with intended users of the evaluation. In basic research, these kinds of questions are resolved by the nature of the theoretical contribution to be made. In dissertation research, the doctoral committee provides guidance on focusing. And always there are fundamental constraints of time and resources.

Converging on focused priorities typically proves more difficult than the challenge of generating potential questions at the beginning of a study or evaluation. Doctoral students can be especially adept at avoiding focus, conceiving instead to propose sweeping, comprehensive studies that make the whole world their fieldwork oyster. In evaluations, once the involved users begin to take seriously the notion that they can learn from finding out whether what they think is being accomplished by a program is what is really being accomplished, they soon generate a long list of things they’d like to find out. The evaluation facilitator’s role is to help them move from a rather extensive list of potential questions to a much shorter list of realistically possible questions and finally to a focused list of essential and necessary questions.

Review of the relevant literature can also bring focus to a study. What is already known? Unknown? What are the cutting-edge theoretical issues? Yet reviewing the literature can present a quandary in qualitative inquiry because it may bias the researcher’s thinking and reduce openness to whatever emerges in the field. Thus, sometimes a literature review may not take place until after data collection. Alternatively, the literature review may go on simultaneously with fieldwork, permitting a creative interplay among the processes of data collection, literature review, and researcher introspection (Marshall & Rossman, 2011, pp. 38–40). As with other qualitative design issues, trade-offs appear at every turn, for there are decided advantages and disadvantages to reviewing the literature before, during, or after fieldwork—or on a continual basis throughout the study.

A specific example of possible variations in focus will illustrate the kinds of trade-offs involved in designing a study. Suppose some educators are interested in studying how a school program affects the social development of school-age children. They want to know how the interactions of children with others in the school setting contribute to the development of social skills. They believe that those social skills will be different for different children, and they are not sure of the range of social interactions that may occur, so they are interested in a qualitative inquiry that will capture variations in program experience and relate those experiences to individualized outcomes. What, then, are the trade-offs in determining the final focus?

We begin with the fact that any given child has social interactions with a great many people. The first problem in focusing, then, is to determine how much of the social reality experienced by children we should attempt to study. In a narrowly focused study, we might select one particular set of interactions and limit our study to those—for example, the social interactions between teachers and children. Broadening the scope somewhat, we might decide to look at only those interactions that occur in the classroom, thereby increasing the scope of the study to include interactions not only between teacher and child but also among peers in the classroom and between any volunteers and visitors to the classroom and the children. Broadening the scope of the study still more, we might decide to look at all of the social relationships that children experience in schools; in this case, we would move beyond the classroom to look at interactions with other personnel in the school—for example, the librarian, school counselors, special-subject teachers, the custodian, and/or school administrative staff. Broadening the scope of the study still further, the educators might decide that it is important to look at the social relationships children experience at home as well as at school so as to better understand how children experience and are affected by both settings, so we would include in our design interactions with parents, siblings, and others in the home. Finally, one might look at the social relationships experienced throughout the full range of societal contacts that children have, including church, clubs, and even mass media contacts.

A case could be made for the importance and value of any of these approaches, from the narrowest focus, looking at only student–teacher interactions, to the broadest focus, looking at students’ full, complex social world. Now let’s add the real-world constraint of limited resources—say, $50,000 and three months—to conduct the study. At some level, any of these research endeavors could be undertaken for $50,000. But it becomes clear, immediately, that there are trade-offs between breadth and depth. A highly focused study of student–teacher interactions could consume our entire budget but allow us to investigate the issue in great depth. On the other hand, we might attempt to look at all social relationships that children experience, but to look at each of them in a relatively cursory way in order, perhaps, to explore which of those relationships is primary. (If school relationships have very little impact on social development in comparison with relationships outside the school, policymakers could use that information to decide whether the school program ought to be redesigned to have greater impact on social development or, alternatively, if the school should forget about trying to directly affect social development at all.) The trade-offs involved are the classic ones between breadth and depth.

Breadth Versus Depth In some ways, the differences between quantitative and qualitative methods involve trade-offs between breadth and depth. Qualitative methods permit inquiry into selected issues in great depth with careful attention to detail, context, and nuance; that data collection need not be constrained by predetermined analytical categories contributes to the potential breadth of qualitative inquiry. Quantitative instruments, on the other hand, ask standardized questions that limit responses to predetermined categories (less breadth and depth). This has the advantage of making it possible to measure the reactions of many respondents to a limited set of questions, thus facilitating comparison and statistical aggregation of the data. In contrast, qualitative methods typically produce a wealth of detailed data about a much smaller number of people and cases.

However, the breadth-versus-depth trade-off also applies within qualitative design options. Human relations specialists tell us that we can never fully understand the experience of another person. The design issue is how much time and effort we are willing to invest in trying to increase our understanding about any single person’s experiences. So, for example, we could look at a

narrow range of experiences for a larger number of people or a broader range of experiences for a smaller number of people. Take the case of interviews. Interviewing with an instrument that provides respondents with largely open-ended stimuli typically takes a great deal of time. In an education study, we developed an open-ended interview for elementary students consisting of 20 questions that included items like “What do you like most about school?” and “What don’t you like about school?” These interviews took between half an hour and two hours depending on the students’ ages and how articulate they were. It would certainly have been possible to have longer interviews. Indeed, I have conducted in-depth interviews with people that ran from 6 to 16 hours over a period of a couple of days. On the other hand, it would have been possible to ask fewer questions, make the interviews shorter, and probe in less depth.

Or consider another example with a fuller range of possibilities. It is possible to study a single individual over an extended period of time—for example, the study, in depth, of one week in the life of one child. This involves gathering detailed information about every occurrence in that child’s life and every interaction involving that child during the week of the study. With more focus, we might study several children during that week but capture fewer events. With a still more limited approach, say a daily half-hour interview, we could interview a yet larger number of children on a smaller number of issues. The extreme case would be to spend all of our resources and time asking a single question of as many children as we could interview given the time and resource constraints.

No rule of thumb exists to tell a researcher precisely how to focus a study. The extent to which a research or evaluation study is broad or narrow depends on the purpose, the resources available, the time available, and the interests of those involved. In brief, these are not choices between good and bad but choices among alternatives, all of which have merit.

EXHIBIT 5.5 Design Trade-Off Example: Depth Versus Breadth

Inquiry questions: What is the nature and variety of social interactions and relationships that secondary school students experience? What are the functions and implications of students’ various interactions and relationships?

Design Constraints: Limited Time and Resources That Necessitate Trade-Offs and Choices

Exhibit 5.5 summarizes with examples some of the primary trade-offs involved between depth and breadth. Especially important in focusing a study and navigating trade-offs is clarity about what case or cases will be studied and how they will be selected. We turn now to these critical design issues.

Case Study Designs: What Is a Case? First, the bad news: Social scientists and methodologists do not agree about what constitutes a case. Eminent case study methodologist Robert Stake (2006) states the problem succinctly and bluntly: “The terms ‘case’ and ‘study’ defy clear definition. . . . Here and there, researchers will call anything they please a case study” (p. 8).

In an intriguing and provocative book titled What Is a Case? Ragin and Becker (1992) document the variety of perspectives and definitions.

To the question “What is a case?” most social scientists would have to give multiple answers. A case may be theoretical or empirical or both; it may be a relatively bounded object or a process; and it may be generic and universal or specific in some way. Asking “What is a case?” questions many different aspects of empirical social science. (Ragin, 1992, p. 3)

Cases can be “empirical units” (individuals, families, organizations) or “theoretical constructs” (resilience, excellence, living with HIV). They can be quite specific, for example, the U.S. war in

Vietnam, or general, such as military campaigns (Engberg, 2013). Cases can be physically real (deaths of children in foster care), socially constructed (excellence, medical “mistakes”), or historical/political (case study of the Congressional Budget Office, Joyce, 2011). Ragin (1992) expands the definition of what is a case to treat any study of any kind on any topic using any methods as a case:

While it is tempting to see the case study as a type of qualitative analysis, and perhaps even to equate the two, virtually every social scientific study is a case study or can be conceived as a case study, often from a variety of viewpoints. At a minimum, every study is a case study because it is an analysis of social phenomena specific to time and place. . . . The tendency to conflate qualitative study and case study should be resisted. (pp. 2–4)

Different views on case studies are also tied to varying theoretical traditions (Chapter 3). A phenomenological case is different from an ethnographic case. A symbolic interaction case is different from a systems case. A positivist case is different from a constructivist case, which is in turn different from case study research in complexity science (Anderson, Crabtree, Steele, & McDaniel, 2005). Given the varied assumptions and approaches of these diverse theoretical orientations, “it is increasingly obvious that there are quite different understandings of case study research” (Blatter, 2008, p. 71).

Now, the good news: The variety of approaches to defining a case gives you an opportunity (and responsibility) to define what a case is within the context of your own field and focus of inquiry. To help you do so, let’s review some of the issues and options.

Case studies are often talked about as a product. The case study stands on its own as a detailed and rich story about a person, organization, event, campaign, or program—whatever the focus of study (unit of analysis). From this perspective, the prime meaning of a case study is the case, not the methods by which the case is created. “The first objective of a case study is to understand the case. . . . The prime referent in case study is the case, not the methods by which the case operates (Stake, 2006, p. 2).

With a different emphasis, Merriam (1995) focuses on the case study as a method of inquiry in which the researcher examines in depth a program, event, activity, process, or one or more individuals, using a variety of data collection procedures over a sustained period of time.

Creswell (1998) elevates the case study to a distinct “qualitative inquiry and research design tradition” that is both an object of study and a methodology in which the inquirer bounds the case by time and place.

King et al. (1994), in their highly influential book in political science Designing Social Inquiry, and George and Bennett (2005), in Case Studies and Theory Development in the Social Sciences, focus on case studies as a method for developing and testing theory.

Flyvbjerg (2011), describing himself as commonsensical, finds the dictionary definition of a case study adequate: “An intensive analysis of an individual unit (as a person or community) stressing development factors in relation to environment” (p. 301).

Despite differences in emphasis, a common thread in defining a case for study is the necessity of placing a boundary around some phenomenon of interest—and where the boundary is placed is both inevitably arbitrary and fundamentally critical because that boundary-setting process determines what the case is and therefore the focus of inquiry.

• A case study is an exploration of a “bounded system” or a case (or multiple cases) over time through detailed, in-depth data collection involving multiple sources of information rich in context. This bounded system is bounded by time and place, and it is the case being studied—a program, an event, an activity, or individuals (Creswell, 1998, p. 61).

• A “case” is a bounded entity (person, organization, behavioral condition, event, or other social phenomenon), but the boundary between the case and its contextual condition—in both spatial and temporal dimension—may be blurred (Yin, 2012, p. 6; see also Yin, 2009, 2011).

The inevitable arbitrary nature of bounding cases means that what constitutes a case study varies broadly:

When presenting their results, investigators manipulate both empirical cases and theoretical cases, and these different cases may vary by level, as when they are nested or hierarchically arrayed, and they may vary in specificity. (Ragin, 1992, p. 2)

An excellent way to experience the diversity of types of and approaches to case studies is to consult the Encyclopedia of Case Study Research (Mills, Durepos, & Wiebe, 2010) or to read Yin’s (2004) The Case Study Anthology. The 19 cases include a public health scare (the feared 1976 “swine flu” epidemic), the 1962 Cuban Missile Crisis, communities (“Middletown” and “Yankee City”), an implementation process, a Korean technological development, two organization case studies (the U.S. Department of Defense and the New York City Police Department), two educational innovations, a labor union, a school reform, a technology company, a riot, a public health intervention program (methadone), military base closures, and an urban school district. Yin characterizes these as “some of the best case studies that may ever have been done” (p. xi).

What this diversity of types of case studies means is that, as I said earlier and here reiterate:

The variety of approaches to defining a case gives you an opportunity (and responsibility) to define what a case is within the context of your own field and focus of inquiry.

To help you do this, the next section looks at cases through the concept of unit of analysis. I’ll then present an in-depth discussion of alternative strategies for selecting cases to study: purposeful sampling.

Units of Analysis A design specifies the unit or units of analysis to be studied. Decisions about what cases to study—issues of both sampling strategies and sample size—depend on prior decisions about the appropriate units of analysis to study. Exhibit 5.6 presents a range of unit of analysis options.

Often, individual people, participants in programs, or students are the unit of analysis. This means that the primary focus of data collection will be on what is happening to individuals in a setting and how individuals are affected by the setting. Individual case studies and variation across individuals would focus the analysis.

Comparing groups of people in a program or across programs involves a different unit of analysis. One may be interested in comparing demographic groups (males compared with females, whites compared with African Americans) or programmatic groups (dropouts versus people who complete the program, people who do well versus people who do poorly, people who experience group therapy versus people who experience individual therapy). One or more groups are selected as the unit of analysis when there is some important characteristic that separates people into groups and when that characteristic has important implications for the program.

A different unit of analysis involves focusing on different parts of a program. Different classrooms within a school might be studied, making the classroom the unit of analysis. Outpatient and inpatient programs in a medical facility could be studied. The intake part of a program might be studied separately from the service delivery part of a program as separate units of analysis. Entire programs can become the unit of analysis. In state and national programs where there are a number of local sites, the appropriate unit of analysis may be local projects. The analytical focus in such multisite studies is on variations among project sites more than on variations among individuals within projects.

EXHIBIT 5.6 Examples of Units of Analysis for Case Studies, Comparisons, and Response Analysis

Note: These are not mutually exclusive categories.

Different units of analysis are not mutually exclusive. However, each unit of analysis implies a different kind of data collection, a different focus for the analysis of data, and a different level at which statements about findings and conclusions would be made. Neighborhoods can be units of analysis, or communities, cities, states, cultures, and even nations in the case of international programs.

One of the strengths of qualitative analysis is looking at program units holistically. This means doing more than aggregating data from individuals to get overall program results. When a program, group, organization, or community is the unit of analysis, qualitative methods involve observations and descriptions focused directly on that unit: The program, organization, or community becomes the case study focus, not just the individual people in those settings.

Particular events, occurrences, or incidents may also be the focus of study (unit of analysis). For example, a quality assurance effort in a health or mental health program might focus only on those critical incidents in which a patient fails to receive quality treatment according to established standards of quality. A criminal justice evaluation could focus on violent events or instances in which juveniles run away from treatment. A cultural study may focus on celebrations.

Time-Bounded Units of Analysis

A time period can also be the unit of analysis, for example, studying farming practices during spring seeding or harvest practices at the end of the growing season. Studying the orientation period for new employees can reveal a great deal about organizational culture. Studying new parents during the first month after their first child is born would examine how couples adapt to a child; couples might be compared with single parents during the same one-month (postpartum) period.

SIDEBAR

INTIMATE PARTNERS AS THE UNIT OF ANALYSIS:

An HIV Example Me, my intimate partner, and HIV:

Fijian self-assessments of transmission risks.

—Hammer (2011a)

The study’s aim was to strengthen Fiji’s response to HIV and AIDS by collecting and analyzing qualitative data about Fijian perceptions of their risks of HIV transmission and of other sexually transmitted diseases. The project interviewed 74 couples, with the respondents being interviewed separately and by different researchers so as to protect their confidentiality and anonymity. They belonged to one of six “target groups”: people in sex work, gays and lesbians, Christian pastors, university students, taxicab drivers, and health care workers.

Both halves of each couple were recruited, enrolled, and interviewed separately but simultaneously using multiple research methods and instruments. Not all of the intimate partners of the Christian pastors were wives, some of the health care workers were also gays and lesbians, and not all of the people in sex work were heterosexual. All of the taxicab drivers were married males, and many of the intimate partners of the university students were themselves students.

The research team also interviewed 20 key informants, including expatriates; conducted 14 audiotaped focus groups; led 74 face-to-face interviews; collected 148 drawings by research participants that depicted “Me, My Intimate Partner, and HIV” and “How I Try to Prevent HIV

Transmission”; and collected another 222 drawings depicting “Risky Behaviour,” “Risky Person,” and “Risky Setting.”

Highlights of the Findings:

• The respondents talked easily and openly about sex. • Of those who reported being HIV positive, some had not yet disclosed their status to their

intimate partner. (One partner found out during a focus group.) • General awareness of HIV and AIDS was high. • Beliefs in the abilities of “deep seawater,” “faith healing,” “prayer,” and “Fiji medicine” to

cure if not also prevent HIV infection and/or transmission were common, and they were just about as commonly shared by health care workers as by sex workers.

• Specific knowledge of the signs, symptoms, names for, and causes of various STDs was minimal, including among health care workers.

• Rates of condom use were low, especially when omitting university students, who didn’t tend to use them with intimate partners anyway.

• Low-likelihood sources of HIV transmission (e.g., rugby matches, car crashes, blood donation, and “renegade, syringe-wielders on the dance floor”) were exaggerated.

• Only four (all females) perceived HIV or STD transmission risks to emanate from their intimate partner; only one (a female) drew herself as being in a risky situation; and not one respondent perceived HIV or STD transmission risk to emanate from self (Hammer, 2011b).

In observational studies, continuous and ongoing observation in a setting contrasts with fixed- interval sampling, in which one treats units of time (e.g., 15-minute segments) as the unit of observation. “The advantages of fixed interval sampling over continuous monitoring are that field workers experience less fatigue and can collect more information at each sampling interval than they could on a continuous observation routine” (Johnson & Sackett, 1998, p. 318). However, the decision about whether and how to sample using units of time should be based primarily on the nature of the phenomenon being observed. Sensitivity to time (sampling periods or units of time) can be especially important in evaluation because programs, organizations, and communities may function in different ways at different times during the year. Of course, in some programs, there never seems to be a good time to collect data. In doing school evaluations in the United States, I’ve been told by educators to avoid collecting data before Halloween because the school year is just getting started and the kids and teachers need time to get settled in. But the period between Halloween and Thanksgiving is really too short to do very much, and then, of course, after Thanksgiving, everybody’s getting ready for the holidays, so that’s not a typical or convenient period. It then takes students a few weeks after the new year to get their attention focused back on school, and then the winter malaise sets in and both teachers and students become deeply depressed with the endlessness of winter (at least in northern climes). Then, of course, once spring hits, attention turns to the close of school, and the kids want to be outside, so that’s not an effective time to gather data either. In African villages, I was given similar scenarios about the difficulties of data collection for every month in the annual cycle of the agricultural season. A particular period of time, then, is both an important context for a study and a sampling issue.

Cross-Case Item Analysis

Case studies make the whole case the unit of analysis, but for some inquiries, the specific interview questions are the unit of interest for data collection and analysis. In such inquiries, the analysis focuses on analyzing patterns across interview questions, focus group responses, open-ended questionnaire items, or site visit observations. For example, a program evaluation could do case studies of participants to capture their holistic experiences of a program and the resulting outcomes. In such a study, the program participants are the unit of analysis. But it is also quite common to interview participants and focus the analysis on the patterns that are found question by question: (a) patterns in how participants learned about the program, (b) patterns in what activities they found most valuable, and (c) patterns in outcomes. Responses to these specific questions become the unit of analysis.

Documents

The unit of analysis can also be documents: diary entries, letters, media items (e.g., news clippings), crime reports, medical records, school portfolios, clinical files, e-mails, blog entries, and social media postings.

Unit of Analysis Decisions

Choosing among all of the possible options for units of analysis and cases for study can become overwhelming, whether the decision is about which time periods to sample, which activities to observe, or which people to interview. The trick is to keep coming back to the criterion of usefulness. What data collected during what time period describing what activities will most likely illuminate the inquiry? What focus will be most useful given the purpose of the inquiry? There are no perfect inquiry designs, only more and less useful ones.

The key to making decisions about the appropriate unit of analysis and cases to study is in determining what you want to be able to say something about at the end of the study. Do you want to have findings about individuals, families, groups, or some other unit of analysis? For scholarly inquiries, disciplinary traditions provide guidance about relevant units of analysis. For evaluations, one has to determine what decision makers and primary intended users really need information about. Do they want findings about the different experiences of individuals in programs, or do they want to know about variations in program processes at different sites? Or both?

Clarity about the unit of analysis is needed to select a study sample. In Chapter 2, I identified purposeful sampling as one of the core distinguishing strategic themes of qualitative inquiry. The next section presents variations in, rationales for, and details of how to design a study based on a purposeful sample.

MODULE

30 Purposeful Sampling and Case Selection: Overview of Strategies and Options

Sampling Purposefully

It is necessary to locate excellent participants to obtain excellent data. —Janice Morse (2010, p. 231)

Purposeful sampling: Selecting information-rich cases to study, cases that by their nature and substance will illuminate the inquiry question being investigated.

Perhaps nothing better illustrates the difference between quantitative and qualitative methods than the different logics that undergird their sampling approaches. Qualitative inquiry typically focuses in depth on relatively small samples, even single cases (n = 1), selected for a quite specific purpose. Quantitative methods typically depend on larger samples selected randomly. Not only are the techniques for sampling different, but the very logic of each approach is unique because the purpose of each strategy is different. While qualitative methodologists prefer the term purposeful sampling, quantitative methodologists are more likely to label these strategies “nonprobability sampling,” making explicit the contrast to probability sampling (e.g., American Association for Public Opinion Research, 2013).

The logic and power of random sampling derives from statistical probability theory. A random and statistically representative sample permits confident generalization from a sample to a larger population. Random sampling also controls for selection bias. The purpose of probability-based random sampling is generalization from the sample to a population and control of selectivity errors.

What would be “bias” in statistical sampling, and therefore a weakness, becomes intended focus in qualitative sampling, and therefore a strength. The logic and power of purposeful sampling lies in selecting information-rich cases for in-depth study. Information-rich cases are those from which one can learn a great deal about issues of central importance to the purpose of the inquiry, thus the term purposeful sampling. Studying information-rich cases yields insights and in-depth understanding rather than empirical generalizations. For example, if the purpose of an evaluation is to increase the effectiveness of a program in reaching lower-socioeconomic groups, one may learn a great deal by studying in depth a small number of carefully selected poor families. Purposeful sampling focuses on selecting information-rich cases whose study will illuminate the questions under study.

Alternative Purposeful Sampling Strategies Case selection is the foundation of qualitative inquiry. What you find from your inquiry will be determined by the cases you study. The type of sample you select should follow from and support inquiry into the questions you are asking. The purpose of a purposeful sample is to focus case

selection strategically in alignment with the inquiry’s purpose, primary questions, and data being collected. Exhibit 5.7 illustrates this relationship.

The previous edition of this book presented 15 purposeful sampling strategies. This edition expands the options to 40. This reflects the emergence of more distinct and nuanced strategic options over the past decade. Because of the larger number of options, I’ve organized them into eight categories:

1. Single significant case 2. Comparison-focused sampling 3. Group characteristics sampling 4. Concept or theoretical sampling 5. Instrumental-use multiple-case sampling

6. Sequential and emergence-driven sampling strategies during fieldwork 7. Analytically focused sampling 8. Mixed, stratified, and combination sampling strategies

Exhibit 5.8 presents all 40 sampling options distributed within these eight categories. Each type of sampling is discussed following Exhibit 5.8. The importance of understanding sampling options is that they constitute design options, in essence, different ways of thinking and strategizing about what to study. In the opening of this chapter, I wrote, “Prepare for design immersion. Think design. Engage design.” Now, we focus more narrowly on one core qualitative design issue. So prepare for sampling immersion. Think of sampling as a core design issue. Engage sampling as purposeful and strategic thinking.

SIDEBAR

PURPOSEFUL VERSUS PURPOSIVE SAMPLING VERSUS NONPROBABILITY SAMPLING

Purposeful sampling: Strategically selecting information-rich cases to study, cases that by their nature and substance will illuminate the inquiry question being investigated

Purposeful sampling is also called purposive sampling. There is generally no difference in meaning. It’s a matter of which term one prefers. In the first edition of this book (Patton, 1980), I substituted “purposeful” sampling for “purposive” sampling. I did so for three reasons:

1. My evaluation and applied research work involved close collaboration with nonresearchers who told me that they found the term purposive too academic, off-putting, jargon-ish, and unclear. One said, “Who talks like that?” In contrast, they could understand “purposeful.” It meant for a purpose. No one I knew, including myself, had ever used “purposive” in common conversation. Ordinary people did know, understand, and sometimes used “purposeful.” My aim was to communicate clearly and be user-friendly, so I introduced “purposeful sampling” and ceased using “purposive.” However, readers should know that purposive remains the term of choice among academic qualitative researchers.

2. In deciding whether to abandon “purposive” for “purposeful,” I investigated the origin of the term and found that it originated as a type of statistical sampling. The 1925 meeting in Rome of the International Statistics Institute included vigorous debate on various sampling approaches. The delegates ended by adopting a formal resolution that distinguished two principal kinds of representative sampling: random and purposive. Purposive sampling involved sampling population elements such that the chosen groups should have average values approximately equal to the population averages for the characteristics already known of the population. By the 1930s, however, random sampling was ascendant, and “purposive sampling lost its appeal” (Kruska & Mosteller, 1980, p. 188). Subsequently, quota sampling emerged as

a kind of purposive sampling that attempts to mimic the population in particular respects. Like all purposive sampling, quota sampling procedures suffer from potential bias because the similarity to the population in some respects by no means implies similarity in others. (Kruska & Mosteller, 1980, p. 190)

This origin of purposive sampling can still be found in some qualitative definitions of the approach. For example, in Barbour’s (2001) checklist for improving rigor in qualitative research, purposive sampling is defined as aiming to capture the diversity within a population. Given the historical origins of purposive sampling as an attempt to get a statistically representative sample of a population in order to generalize, I thought it appropriate to abandon the term altogether. So I introduced purposeful sampling as a specifically qualitative approach to case selection. My mistake, I now realize (and confess), was in not explaining the substitution when I originally made it in 1980. I simply started using the new term, and have ever since. Now it’s your choice.

3. While qualitative methodologists use the terms purposeful or purposive sampling, quantitative methodologists are more likely to label these strategies “nonprobability sampling,” making explicit the contrast to probability sampling (e.g., American Association for Public Opinion Research, 2013). This defines qualitative sampling by what it is not (nonprobability) rather than by what it is (strategically purposeful).

EXHIBIT 5.7 Steps for Design Alignment

EXHIBIT 5.8 Purposeful Sampling Strategies

Selecting Information-Rich, Illuminative Cases for Qualitative Inquiry

(For detailed discussion and references, see the accompanying text following this exhibit.)

Discussion of the 40 purposeful sampling options in Exhibit 5.8 follows in Modules 31 through 38.

MODULE

31 Single-Significant-Case Sampling as a Design Strategy

Sample purpose: One in-depth case (n = 1) that provides rich and deep understanding of the subject and breakthrough insights, and/or has distinct, stand-out importance

What can possibly be learned from a single case? We live in a world of big data and large samples. An n of 1? Worthless. More is better. Lots more is lots better. Can less be more? Can one make a difference? It turns out that, yes, single cases can provide quite powerful breakthroughs and insights. On March 3, 2013, headlines around the world proclaimed,

Scientists Report First Cure of HIV in a Child, Say It’s a Game-Changer

The baby was the first child in the world known to have been cured since the virus touched off a global pandemic three decades earlier. Until this case, children born with HIV were considered permanently infected, their only hope being lifelong treatment with antiviral drugs to prevent HIV from becoming AIDS, destroying their immune system, and leading to death. World health statistics estimate that 330,000 children around the world get infected with HIV at or around birth every year. Part of what made this one case significant and credible was careful documentation of the circumstances of the mother and child, and their health status during pregnancy and immediately after the child’s birth, and thorough documentation of an innovative treatment approach (Knox, 2013; Pollack & McNeil, 2013).

Index Case

This is what is called an index case: the first documented case to manifest a phenomenon. In epidemiology, an index case is the first person exhibiting a condition, syndrome, or cure. An index case often becomes the classic case in the literature on the phenomenon. I discussed the story of Henrietta Lacks and the discovery and impact of her HeLa cells in Chapter 1 as an example of the power and importance of an in-depth case study. She constitutes an index case, the person whose cells could be grown and sustained alive in a laboratory culture, which enabled major medical breakthroughs on a number of frontiers (Skloot, 2010).

Examples of index cases from several fields:

• The first human landing on the moon, Apollo 11 spacecraft, July 20, 1969. • The first electronic general-purpose computer, ENIAC (Electronic Numerical Integrator and

Computer), 1949. • Sirimavo Ratwatte Dias Bandaranaike, the modern world’s first female head of government,

served as prime Minister of Ceylon and Sri Lanka three times, 1960 to 1965, 1970 to 1977, and 1994 to 2000.

• Lorenzo Odone suffered from adrenoleucodystrophy, a genetic disease that progressively destroys the brain of young boys. He was the first to be treated with oleic acid, which lowered his fatty acids more effectively than any other medical approach that had been tried. His treatment led to clinical trials and the discovery of a preventative protocol for boys genetically at risk of the life-threatening disease (BBC, 2004).

SIDEBAR

TYPHOID MARY: A RENOWNED INDEX CASE

Mary Mallon (1869–1938), an Irish immigrant cook, was the first person in the United States identified as an asymptomatic carrier of the pathogen associated with typhoid fever. As public health officials investigated contacts of people who contracted typhoid fever, she was the one common contact, though she never showed any symptoms herself. She was estimated to have infected more than 50 people, at least three of whom died; thus she was dubbed Typhoid Mary. She refused to cooperate with health authorities, withheld information or lied about her past, and made up names when she moved around trying to escape the investigators. She was quarantined twice by public health authorities and eventually spent nearly three decades in isolation. How she was able to infect others without succumbing to the illness herself is still a matter of scientific inquiry 75 years after her death (Huffington Post, 2013).

Sampling an Exemplar of a Phenomenon of Interest

But a single case doesn’t have to be the first of its kind to be significant and to merit in-depth study and analysis. Any exemplar of a phenomenon of interest can be a worthy single-case study. In his classic The Art of Case Study Research, Robert Stake (1995) emphasized the value of what he called intrinsic cases in which the case offers insights that stand alone as important. Consider the case study of Eugene Pauly, who provided breakthroughs in our understanding of habit formation and behavior change in the face of memory loss and reduced cognitive functioning. The case study extended over 15 years and documented in day-by-day detail the life of an elderly man and his family coping with and adapting to severe memory loss, including the various medical, behavioral, social, and psychological interventions attempted over the years. The Pauly case provided significant advances in understanding the nature and power of habit and “revolutionized the scientific community’s understanding of how the brain works by proving, once and for all, that it’s possible to learn and make unconscious choices without remembering anything about the lesson or decision making” (Duhigg, 2013, pp. 24–25). Breakthroughs like the Pauly case study findings show why the “science of habit formation has exploded into a major field of study” (Duhigg, 2013, p. 25).