U1D1-68 -compare and contrast the characteristics of quantitative and qualitative research, addressing the philosophical assumptions of both. Based on your review of the assigned readings, identify and describe what you believe to be meaningful about the

CHAPTER

1 The Nature, Niche, Value, and Fruit of Qualitative Inquiry

Nyansapo, “wisdom knot,” Adinkra (West Africa) symbol of wisdom, ingenuity, intelligence, and patience. This symbol conveys the idea that a wise person has the capacity to select the best means to achieve a goal. Being wise means knowing how to apply broad knowledge, learning, and experience for practical purposes (Willis, 1998).

Book Overview and Chapter Preview Part 1 of this book—this journey deep into qualitative inquiry—provides an overview of qualitative methodology in four chapters—on (1) the nature, niche, value, and fruit of qualitative inquiry; (2) strategic themes in qualitative inquiry; (3) a variety of qualitative inquiry frameworks (paradigmatic, philosophical, and theoretical orientations); and (4) practical and actionable qualitative applications. Part 2 covers qualitative designs and data collection, with chapters on (5) design options, (6) fieldwork and observation, and (7) in- depth interviewing. Part 3 completes the book with chapters on (8) qualitative analysis and (9) enhancing the quality and credibility of qualitative studies.

In this first chapter, Module 1 presents examples of how qualitative inquiry contributes to our understanding of the world. Module 2 examines what makes qualitative data qualitative. Module 3 provides an overview of the issues involved in making methods decisions. Module 4 concludes the chapter with a summary of the fruit of qualitative methods, that is, a look at what comes out of qualitative studies.

A thick tree grows from a tiny seed. A tall building arises from a mound of earth. A journey of a thousand miles starts with one step.

—Lao-tzu Philosopher and poet of ancient China

QUALITATIVE WISDOM

A Portuguese professional from Barcelona was driving in a remote area of his country when he came upon a sizable herd of sheep being driven along the country road by a shepherd. Seeing that he would be delayed until the sheep could be turned off the road, he got out of the car and struck up a conversation with the shepherd. “How many sheep do you have?” he asked. “I don’t know,” responded the young man. The professional was embarrassed for having exposed what he assumed was the young shepherd’s lack of formal schooling, and therefore his inability to count such a large number. But he was also puzzled. “How do you keep track of the flock if you don’t know how many sheep there are? How would you know if one was missing?”

The shepherd, in turn, seemed puzzled by the question. Then he explained, “I don’t need to count them. I know each one, and I know the whole flock. I would know if the flock was not whole.”

MODULE

1 How Qualitative Inquiry Contributes to Our Understanding of the World

This opening chapter will offer an overview of the nature, niche, value, and fruit of qualitative inquiry. In the spirit of the Adinkra Nyansapo, symbol of wisdom, our journey together through various purposes for and contributions of qualitative inquiry aims to enhance your capacity to select the best methods and design to achieve a particular research or evaluation purpose. This chapter will offer a sampling of findings from qualitative studies. In this regard, it will be like a wine tasting, meant to introduce possibilities and support developing a more sophisticated palate, or like appetizers, as an opening to the fuller feast yet to come in later chapters.

In this chapter, we are especially attentive to the fruit of qualitative inquiry. It is important to know what qualitative data yield, what findings look like, and how they are produced, so that you will know what you are seeking to find out and produce when you undertake your own qualitative inquiry. Let’s begin, then, with seven ways in which qualitative inquiry contributes to our understanding of the world. The first contribution is illuminating meaning.

Illuminating Meanings: From Birth to Death and In-Between What makes us different from other animals is our capacity to assign meaning to things. The essence of being human is integrating and making sense of experience (Loevinger, 1976). Language has developed, and continues to develop, as a uniquely human way to express meaning (Halliday, 1978)—and to disguise meaning. As Shakespeare observed in Measure for Measure, “It oft falls out, to have what we would have, we speak not what we mean.”

Being a person is the activity of meaning-making. —Robert Kegan (1982, p. 11)

Developmental psychologist Harvard University

Qualitative research inquires into, documents, and interprets the meaning-making process. Let me illustrate how this occurs and explain why it is so important—indeed, why it is the core of qualitative inquiry and analysis. I’ll begin with a personal example. During the writing of this book, my first grandchild was born, and this book is dedicated to her. The hospital records document her weight, height, health, and Apgar score— activity (muscle tone), pulse, grimace (reflex response), appearance, and respiration. The mother’s condition, length of labor, time of birth, and hospital stay are all documented. These are physiological and institutional metrics. When aggregated across many babies and mothers, they provide trend data about the beginning of life—birthing. But nowhere in the hospital records will you find anything about what the birth of Calla Quinn means. Her name is recorded but not why it was chosen by her parents and what it means to them. Her existence is documented but not what she means to our family, what decision-making process led up to her birth, the experience and meaning of the pregnancy, the family experience of the birth process, and the familial, social, cultural, political, and economic context that is essential to understanding what her birth means to family and friends in this time and place. A qualitative case study of Calla’s birth would capture and interpret the story and meaning of her entry into the world from the perspectives of those involved in and touched by her coming into our lives. This might, or might not, include the fact that at the moment she was born I was in the midst of conducting a webinar on qualitative evaluation and those participants from around

the world became part of the experience as I took a break to rush to the hospital and meet Calla for the first time. Several participants subsequently sent me e-mails that Calla’s birth made the webinar more meaningful for them. This example of the meaning of her birth as a potential qualitative case study was born during that webinar.

I open with this personal story for another reason. Qualitative inquiry is personal. The researcher is the instrument of inquiry. What brings you to an inquiry matters. Your background, experience, training, skills, interpersonal competence, capacity for empathy, cross-cultural sensitivity, and how you, as a person, engage in fieldwork and analysis—these things undergird the credibility of your finings. Reflection on how your data collection and interpretation are affected by who you are, what’s going on in your life, what you care about, how you view the world, and how you’ve chosen to study what interests you is a part of qualitative methodology. The obligation and commitment to acknowledge and take into account the personal and interpersonal nature of qualitative inquiry will be a recurring theme of this book. I’ve been at this for more than 40 years. My granddaughter’s birth infused my writing with new energy and urgency as I imagined addressing a new generation of qualitative researchers and evaluators.

So let us turn now to the other end of the human existential continuum. Systematically gathering data on deaths began in the Black Plague, when, from 1347 to 1351, a third or more of all Europeans died. From that time, England began tracking deaths, eventually developing the death certificate, which specifies cause of death.

Of the roughly fifty million people who will die this year, approximately half will get a death certificate. That figure includes every fatality in every developed nation on earth: man, woman, child, infant. The other half, death’s dark matter, expire in the world’s poorest places, which lack the medical and bureaucratic infrastructures for end-of-life documentation. (Schulz, 2014, p. 32)

The death certificate has become a crucial source of epidemiological data documenting trends in causes of death, which has influenced policymaking, research priorities, and allocation of public health resources. Epidemiological studies go beyond death certificates to estimate deaths caused by poverty, low levels of education, smoking, obesity, and inactivity—causes in the same range as deaths from heart attacks and cancer (Galea, Tracy, Hoggatt, DiMaggio, & Karpati, 2011). But as with birth certificates, death certificates and epidemiological studies do not capture what the death of someone means to those touched by that death. Only an in-depth case study can even begin to do that. To understand how humans face death and make sense of dying under the most extreme conditions, Viktor E. Frankl (2006), a neurologist, psychiatrist, and Holocaust survivor, studied the search for meaning in World War II concentration and death camps. The capacity to find meaning in suffering and death, he concluded, was the key to survival.

So aggregate statistics on mortality reveal causes of death but don’t tell us how people find meaning in dying and how cultures make sense of death. That kind of inquiry is the focus of the anthropology of death, a specialized area of cross-cultural and cross-institutional inquiry.

The anthropology of death takes as its task to understand the phrase: “All humans die,” yet in every culture, each dies in their own way. . . .

Death is an intensely emotional and often taboo subject, so that studying death raises special dilemmas and emotional challenges for the fieldworker. . . . [Anthropologist] Hortense Powdermaker, working in a matrilineal society in New Ireland, described her own extreme distress when she began taking field notes at her first funeral. She imagined how intrusive such an ethnographic presence would be in her own house of mourning. She, however, discovered to her great surprise that these non-literate people felt no such intrusions, but rather that her writing added prestige to the ritual. They demanded her presence at every subsequent funeral, even long after she had constructed a complete account of the funeral process.

Many ethnographers have discussed the emotional strain of participating closely in the grief of others. . . . Perhaps the most moving account is by Rosaldo who connects how his overwhelming grief at the accidental death of his wife, (also an eminent anthropologist) helped him understand more deeply [the] headhunter’s rage of the Ilongot in the Philippines. (Abramovitch, 2014, p. 1)

An exemplar of an in-depth qualitative inquiry into death is a study by Karen Martin (2007) of sudden infant death syndrome (SIDS), the leading cause of death among apparently healthy infants between the ages of one week and one year. Her case studies document the painful experience of bereaved parents, who frequently blame themselves for their baby’s death. She looks at how parents grieve, the meanings and casual explanations they attribute to a SIDS death, the effects of their grief on family relationships, and the strategies they use to cope and carry on.

The anthropology of death includes how different cultures explain, talk about, and deal with death. Americans appear to have particular difficulty dealing with death. Debate about the wisdom and costs of extending life a few months with hugely expensive medical technology, surgery, and drugs has become not only a difficult matter of medical ethics but also a volatile political issue (Brown, 2014). These are matters for qualitative inquiry.

We construct and attach meaning to births. We try to make sense of death—and culture tells us how to do so. Now let’s look at five diverse examples between birth and death of how qualitative inquiry contributes to understanding human meaning making.

• Bodily meaning making: Qualitative inquirers have studied the meanings attached to male and female bodies, disabled and injured bodies, bodies of different colors and sizes, how and why people adorn their bodies (e.g., with jewelry, tattoos, piercings, intentional scars), and how and why they mutilate them (e.g., by circumcision, female genital mutilation, cutting off limbs in war, scalping, cannibalism, and sexual abuse).

Because our bodies serve as a site of meaning making within our culture, they also serve as a site of scholarly investigation. . . .

Wanda Pillow discovered the centrality of the body in her research . . . on pregnant teenagers and their experiences. . . . The body, the changing body, the experience of the pregnant body, structural responses to girls’ changing bodies, and the perceptions of others toward girls’ pregnant bodies became central to her research. In fact, without focusing on the body, it would not have been possible to understand much of the experience of this population. Accordingly, Pillow modified her research to focus on the bodies and bodily experiences of the girls she was studying. In other words, she developed a body-centered methodology. . . . A shift to the body allowed her to ask and answer research questions that would otherwise be impossible to address. Likewise, she was able to access knowledge that would otherwise remain invisible. (Hesse-Biber & Leavy, 2006a, pp. xx–xxi)

• Evaluative meaning making: Evaluation involves making judgments about what is meaningful. One important form of evaluation is assessing students’ academic achievement. Magolda and King (2012) interviewed nearly 2,000 college students to find out how they make learning meaningful. They found that many students fail to achieve complex learning goals because they rely too heavily on others’ opinions about what to believe, who to be, and how to relate to others. In other words, peer pressure trumps individual meaning making, especially early in the college experience. Over time, successful students learn to decide for themselves what is meaningful, what Magolda and King call “self-authorship.” They conclude that understanding and assessing students’ meaning making is essential for interpreting students’ academic performance and other behaviors and should inform the design of new programs and services.

• What objects mean: Humans attach meaning to things, what anthropologists call material culture. Art, food, toys, jewelry, land, cars, perfume, clothes . . . anything can become meaningful to those people within a setting who attach value to it. The dictum that “an Englishman’s home is his castle” attaches special meaning, legal protections, and social status to one’s place of abode. National flags are symbols full of meaning. Music has meaning. The Olympic medal presentations combining flags and music evoke strong emotions. Qualitative inquiry includes studying the meaning making associated with things as diverse as Smartphones, Facebook, and hair dye (Berger, 2014).

• Meaning in meaninglessness: Social groups are typically defined by their shared meaning making. In an ironic twist, some groups find meaning around a commitment to meaninglessness. Nihilism is a philosophical assertion that life has no meaning. Nihilists find common meaningfulness in asserting meaninglessness. How and why this occurs, and its effects on those involved, is a matter well suited for

qualitative inquiry. Distinguished British philosopher and author Aldous Huxley (1894–1963) studied and reflected on the political and moral attraction of a philosophy of meaninglessness in England during the 1920s and 1930s. He interpreted it as essentially a means of liberation from conservative morality and politics, resisting being told what to believe by the powerful (religious, corporate, and government leaders). Adherents of a philosophy of meaninglessness justified their political and erotic revolt by denying that the world had any meaning at all (Huxley, 1937).

• Qualitative interpretation as meaning making: Qualitative inquiries study how people and groups construct meaning. In so doing, qualitative methodology devotes considerable attention to how qualitative analysts determine what is meaningful. Qualitative analysis involves interpreting interviews, observations, and documents—the data of qualitative inquiry—to find substantively meaningful patterns and themes. Doing so is an act of interpretation. Distinguished qualitative methodologist Robert Stake (2010) explains what this means:

Interpretation is an act of composition. The interpreter takes descriptions and makes them more complex, drawing upon a few conceptual relationships. He or she might take the term work and give it muscle, durability, remuneration, and self-respect. These can be some of the larger meanings of work. He or she might take an episode observed at the workplace and give it personality, history, tension, and implication. The best interpretations will be logical extensions of the simple description but also will include contemplative, speculative, even aesthetic extension. The reader would be deceived if allowed to think that these interpretations had been agreed upon, certified in some way. They are contributions of the researcher, written so as to make it clear they are personal interpretations. All people make interpretations. All research requires interpretations. Qualitative research relies heavily on interpretive perceptions throughout the planning, data gathering, analysis, and write-up of the study. (p. 55)

The first contribution of qualitative inquiry, then, is illuminating meanings and how humans engage in meaning making—in essence, making sense of the world. Science fiction author Piers Anthony could have been talking about the challenge of qualitative inquiry when he observed, “All things make sense; you just have to fathom how they make sense.”

Studying How Things Work Michael Scriven is a founder of the transdisciplinary profession of evaluation. Research can involve studying how anything works. Program evaluation involves studying how a program works and what results it gets to render a judgment about its effectiveness. Scriven (1998) tells about being invited to evaluate a computer- based approach used by the counseling center at the University of California at Irvine. He accepted, and then things got interesting:

I ran three of my graduate students through the program, and its disastrous failings emerged readily. From the administrator’s desk, dazzled by the computers, these failings—of content as well as of the machinery—were invisible. In any case, they refused payment in order to not have my critical report in their files. I said I would be happy not to charge them and instead use it as the theme for my next published article. So they called and said they had appointed a negotiator. I called the negotiator and asked if he was empowered to negotiate to the full amount of the contract and he said, “Absolutely.” So I said fine, that I would not charge them since they did not think it worth paying for, but I would use the example in every future speech that I made on a related topic. (p. 13)

In that story are two examples of how things work. First is a glimpse into how the counseling center’s program worked—or rather didn’t work. The second story is how negotiating settlement of the evaluation contract worked.

Students of anatomy examine how the body works. Social scientists study how human groups and institutions work. The contribution of qualitative research and evaluation to understanding how things work is highlighted by the opposite phenomenon expressed in the title of Nigerian Chinua Achebe’s (1994) classic story of the clash between Western and traditional African values during and after the colonial era: Things Fall Apart.

SOURCE: From Scriven, M. (1991). Evaluation thesaurus (4th ed., p. 1). Thousand Oaks, CA: Sage. Used by permission of cartoonist Chris Lysy.

Robert Stake, quoted earlier about the centrality of interpretation in making sense of the world, subtitled his book on qualitative research Studying How Things Work. Here’s what Stake (2010) says it means:

Understanding the social and professional worlds around us comes from paying attention to what people are doing and what they are saying. Some of what they do and say is unproductive and silly, but we need to know that, too. A lot of what people do is motivated by their love for their families and a desire to help people, and we need to know that, too. We won’t just ask them. We will look closely to see how their productivity and love are manifested. I put “Studying How Things Work” in the title . . . to help you improve your ability to examine how things are working. Most of the things I have in mind are small things—small but not simple, such as classrooms and offices and committees. But also gerundial things, nursing and mainstreaming and fund-raising, in particular situations. And some special things, such as ordering chairs for a classroom, and “labor and delivery,” and personal privacy. (p. 2)

The possibilities for studying how things work is vast. How does culture work? How do families work? Small groups? Universities? Movements? Systems? What is meant by qualitatively studying how things work is getting inside the phenomenon of interest to get detailed, descriptive data and perceptions about the variations in what goes on and the implications of those variations for the people and processes involved. A major way to do that is to capture people’s stories about how things work.

Capturing Stories to Understand People’s Perspectives and Experiences

The universe is made of stories, not atoms. —Muriel Rukeyser (1913–1980)

American poet and political activist

Stories make us human. —Jonathan Gottschall (2012)

The Storytelling Animal

If you want to know how much children can read, give them a reading test. If you want to know what reading means to them, you have to talk with them, listen to them, and hear their stories about the stories they love. Exhibit 1.1 gives examples of the kinds of questions you might ask.

These are qualitative inquiry questions aimed at getting an in-depth, individualized, and contextually sensitive understanding of reading for each child interviewed. Of course, the actual questions asked would have to be adapted to the child’s age and language skills, the school and family situation, and the purpose of the inquiry. But regardless of the precise wording and sequence of the questions, the purpose is to hear children talk about reading in their own words; find out about their reading behaviors, attitudes, and experiences; and get them to tell stories that illuminate what reading means to them. You might talk to groups of kids about reading as a basis for developing more in-depth, personalized questions for individual interviews. While doing fieldwork (actually visiting schools and classrooms), you would observe children reading and the interactions between teachers and children around reading. You would also observe what books and reading materials are there in a classroom and how they are arranged, handled, and used. In a comprehensive inquiry, you would also interview teachers and parents to get their perspective on the meaning and practice of reading, both for children and for themselves, as models their children are likely to emulate.

In analyzing your classroom observations and interviews with children, parents, and teachers, you would provide illustrative case examples of variations in reading practices and what it means to those interviewed. You would report and explain any patterns or themes that emerged in the responses to your interview questions. For example, eight-year-old boys I interviewed told me that good readers are bad at sports. They believed that a little reading was okay, but if you read too much, it interferes with your muscles getting stronger. They were careful to read just enough to do okay in school but not so much as to hurt their aspirations to become good athletes. Teachers had heard this, they told me, but didn’t take it seriously. The boys I talked with took it very seriously. The eight-year-old girls thought the boys were just dumb and had silly and stupid ideas.

EXHIBIT 1.1 Open-Ended Interview Questions About Reading

If you want to know how much children can read, give them a reading test. If you want to know what reading means to them, you have to talk with them. Here are examples of open-ended interview questions about reading:

Tell me about something you’re reading now.

What do you like to read in school? How does reading relate to other subjects in school? What do you read on your own, outside school? When do you read?

What do you like about reading? What don’t you like?

Tell me about reading in your family. What do people in your family say about reading? What do your friends say about it?

Some kids seem to be good readers, and some have trouble reading. Why is this, do you think? From what you’ve seen, what’s the difference between good readers and not so good readers?

What about you, how would you describe your level of reading? Why?

How is your reading today different from a year ago, if at all?

Why do you think so much emphasis is placed on reading in school? When you think about the future, how important do you think reading will be to what you do? Why?

Tell me one of your favorite stories. What makes it a favorite?

The results from your in-depth qualitative inquiry into reading could be used to adapt and improve approaches to reading, both in school and at home. Understanding how students view reading is critical to

dealing with the development of reading skills as well as supporting positive attitudes about reading and motivation to get better at it.

© 2002 Michael Quinn Patton and Michael Cochran

Discovery of a 4,000-Year-Old Prehistoric Cave Painting of a Wooly Mammoth Encounter: An Early Example of a Qualitative Report

Making sense of and communicating research findings involves “the art of storytelling” (Hastings & Domegan, 2014, p. 117). Capturing and understanding diverse perspectives, observing and analyzing behaviors in context, looking for patterns in what human beings do and think—and examining the implications of those patterns—these are some of the basic contributions of qualitative inquiry.

Elucidating How Systems Function and Their Consequences for People’s Lives Why do people do what they do even when it doesn’t seem to make sense to an outsider? Sometimes the answer lies within the individual enmeshed in a person’s background, personality, upbringing, worldview, and conditioned behaviors. Qualitative research often inquires into the stories of individuals to capture and understand their perspectives, as just discussed. But often the answer to why people do what they do is found not just within the individual but, rather, within the systems of which they are a part: social, family, organizational, community, religious, political, and economic systems.

Atul Gawande (2007), a Harvard Medical School surgeon, tells of visiting the Walter Reed military hospital early in the Iraq war. He participated in a session interpreting eye injury statistics. The doctors were having considerable success saving some soldiers from blindness, a positive outcome. But digging deeper, the doctors asked why so many severe eye injuries were occurring. Interviewing their patients, they learned that the young soldiers weren’t wearing their protective goggles because they were considered too ugly and uncool. They recommended that the military switch to “cooler-looking Wiley X ballistic eyewear. The soldiers wore their eye gear more consistently and the eye-injury rate dropped immediately” (p. A23). By asking these kinds

of deeper questions about what’s really going on and inquiring into assumptions about why things are happening, qualitative researchers and evaluators contribute to knowledge about what works, what doesn’t, and why.

Let’s place this example in a larger context. Those involved in solving problems repeatedly share frustrations about the stream of ever more sophisticated ambulances sent to accidents at the bottom of a cliff, when simply building a fence at the top would prevent those accidents. Why doesn’t the fence get built? A qualitative researcher would likely find a number of different perspectives and explanations and learn that building a fence involves political jurisdictions, financial priorities, environmental issues, contextual complexities, and conflicts about who is involved in such a decision and who actually decides what kind of fence is to be built by whom, as well as some interesting and important local history and stories about the cliff, the road that runs along it, and the people who use that road, maybe even stories about the fence or fences that used to be there. What might begin as a seemingly simple question about why a fence hasn’t been built becomes, with in-depth inquiry, a case study of complex system dynamics. Moreover, social, cultural, and political perspectives about how to solve problems all come into play. In reviewing numerous such stories, distinguished Australian action research scholar and practitioner Yolande Wadsworth (2010) has commented that they are reminders about our repeated tendency to go for the short-term quick fix rather than examine, come to understand, and take action to change how a system is functioning that creates the very problems being addressed. In-depth qualitative inquiry can illuminate system and systemic issues and potential solutions.

Understanding Context: How and Why It Matters When qualitative inquirers study how systems function and the consequences of system dynamics, they include attention to context. Context refers to what’s going on around the people, groups, organizations, communities, or systems of interest. If we’re studying family farming systems in northwest Minnesota or eastern Burkina Faso, we’ll need to attend to larger-context issues such as climate change, political and economic trends, and public health threats, such as the obesity epidemic in the United States or the spread of HIV/AIDS in Africa.

The theme of the 2009 annual conference of the American Evaluation Association (AEA) was “Context and Evaluation.” AEA president Debra Rog (2009) articulated the challenge of taking context seriously:

As evaluators, we recognize that context matters. . . . Context has multiple layers and is dynamic, changing over time. Increasingly, we are aware of the need to shape our methods and overall approach to the context. Each of the dimensions within the context of an evaluation influences the approaches and methods that are possible, appropriate, and likely to produce actionable evidence. In tandem, some evaluations embrace context and include it within the study, rather than simply attempting to control its effects. Attention to context helps to produce findings that are generalizable and useful to a broader set of stakeholders outside the local decision-making context.

When new cases of polio emerged in a remote area of India, epidemiologists accompanied the revaccination team and interviewed people in the area to find out how and why some children had missed being vaccinated in the recent campaign. They learned that some Muslim mothers had resisted the vaccination and had hidden their children because they’d heard rumors of a Hindu plot to sterilize their boys (Gawande, 2004). Understanding the resistance to vaccination or other health practices, and developing approaches to overcome that resistance, requires an in-depth understanding of the cultural, social, and political systems within a particular context.

Context includes attention to and understanding of important nuances of culture, politics, economy, history, geography, resources, and institutions. Qualitative inquiry makes attention to context a priority both for data collection and for reporting findings. This means documenting diversity and the contextual factors that explain particular variations even while identifying cross-cutting patterns and themes.

Bottom line: Sensitivity to context is central in qualitative inquiry and analysis. Sounds reasonable, I trust. Perhaps even matter-of-fact. But maybe a bit abstract. So let’s make it concrete. What is the context for your

reading this book? For me, or anyone, to understand how you engage with this book—indeed, how you engage with these very words as you read them—I would need to know the context within which you are reading. Are you taking a required course and are required to read this? Are you doing, or considering doing, a qualitative thesis or dissertation? Are you undertaking an evaluation project that includes qualitative methods? Are you new to qualitative inquiry, or are you an experienced researcher reading this as a refresher or to see how these methods have developed in the past few years? Do you come to this book with a strong quantitative methods background? Are you working or studying in a context that values qualitative data or one of skepticism about the value and credibility of qualitative findings? Are you just studying qualitative methods at this point for possible future use, or are you engaged in or about to engage in a qualitative study? If the latter, are you doing so alone or as part of a team? These are questions that illuminate the context within which you are reading this book. If I am to understand your reading of this book, and your reactions to it, I need to know the context within which you are reading it.

Identifying Unanticipated Consequences

That there will be unanticipated consequences is the one sure thing we can anticipate. —Halcolm

The great delusion of our times is that we can control what happens. Politicians routinely promise to bring about changes in things such as the economy, over which they have little or no control. People running programs of all kinds establish objectives and implementation strategies and then follow the admonition to plan your work and work your plan. Good advice. But things seldom work out quite as planned. I’ve seen literacy programs aimed at helping high school dropouts learn to read that made them hate reading. I’ve seen juvenile justice programs aimed at rescuing delinquents from a potential life of crime that propelled them inadvertently on a pathway to becoming hard-core criminals. I’ve seen interventions that intended to support welfare recipients become economically self-sufficient that contributed to multigenerational poverty. I’ve seen international agricultural development programs aimed at increasing food production and income from cash crops turn fragile soils into desert through the introduction of inappropriate technological and management schemes. None of these programs were run by uncaring or incompetent people. Quite the contrary. In each case, I found the program leaders and staff to be committed, motivated, hardworking, and deeply engaged in bringing about change but, in the end, ineffective. Indeed, they were not just ineffective but counterproductive, having the opposite of their intended effects.

Context Matters: The Example of the Paris Declaration on Aid Effectiveness

The 2005 Paris Declaration on Aid Effectiveness was a landmark agreement between aid donors and developing countries aimed at reforming aid processes and increasing aid results. The Paris Declaration Principles were endorsed by more than 150 countries and organizations, including the more developed aid donor countries, such as the United States, developing countries from around the world, and international development institutions, such as the World Bank, the United Nations Development Group, and the Organization for Economic Co-operation and Development. The five principles are as follows:

1. Ownership: Developing countries set their own strategies for poverty reduction, improve their institutions, and tackle corruption. 2. Alignment: Donor countries align behind these objectives and use local systems. 3. Harmonization: Donor countries coordinate, simplify procedures, and share information to avoid duplication.

4. Results: Developing countries and donors shift focus to development results, and results get measured. 5. Mutual accountability: Donors and partners are accountable for development results. (OECD, 2005)

An independent evaluation examined what difference, if any, the Paris Declaration made to development processes and results (Wood et al., 2011). The scope of the evaluation was immense: case studies in 22 developing countries and in-depth reviews in 18 donor agencies as well as studies on special themes such as health sector aid. The evaluation was conducted in two phases over four years, between 2007 and 2011. The case studies used mixed methods, including interviews with “key knowledgeables” (people in a position to know firsthand about the subject of an inquiry), reviews of documents and official reports, and surveys of people involved with aid initiatives. The evaluation received the 2012 Outstanding Evaluation Award from the American Evaluation Association.

I became involved as the evaluator of the evaluation, to independently assess the rigor of the evaluation processes and render judgment about the credibility and utility of the findings (Patton & Gornick, 2011a). I observed two international meetings of those involved from developing countries and donor agencies engaged in international development assistance. As the findings were reported and discussed, one theme dominated: Context matters. The five reform principles provided a shared global commitment and a standard inquiry framework, but Indonesia is different from Vietnam, Colombia is different from the Cook Islands, and Afghanistan is different from South Africa. Development aid from Sweden is handled differently than aid from the United States, just as Australia, Switzerland, and Japan have different priorities, processes, and relationships with recipient countries.

Thus, the evaluation of the Paris Declaration not only included 22 country case studies but also offered cross-cutting judgments, for example, that country ownership (Principle 1 above) showed significant progress over the past decade, while mutual accountability (Principle 5) has lagged. Qualitative inquiry balances the particular (specific case studies in context) with the general (findings that cut across cases and contexts).

Qualitative inquiry is especially valuable for identifying unintended consequences and side effects. If all a program evaluator looks at is whether the intended outcomes are attained, especially using standard performance indicators such as reading tests, employment statistics, and health outcome data, then other, unintended effects will be missed. To find unanticipated effects, you have to go into the field where things are happening, observe what is really going on, interview program participants about what they’re experiencing, and find out through open inquiry what is happening, both intended and unintended. Consider these examples:

• A program for chronically unemployed men wanted to improve its diagnosis of participants’ needs, so an extensive battery of assessment tests was added to the intake process. But the men coming into the program were so turned off by all the testing that the drop-out rate soared. The tests were supposed to communicate a deep concern with understanding the particular needs of each man. What the men experienced was quite different. The tests made them feel stupid, reminded them of years of failure in school, and came across as impersonal, mechanistic, and alienating.

• A parent education program for young, poor, first-time mothers aimed to increase their knowledge and skills as parents. Those outcomes were attained, and the mothers were grateful. But when asked what was the most important thing they got out of the program, they said that it wasn’t the knowledge and skills attained, as important as those were, but the new relationships with other mothers. They came into the program feeling isolated, lonely, abandoned, and afraid. They came out with friends, playmates for their children, and a network of support. Those were unintended side effects from the perspective of the program funders and staff. They were what made the program worthwhile from the perspective of the mothers.

• The economy of Kiribati, an island nation in the central Pacific, depends on coconut oil and fishing. But overfishing threatened the island’s future. The government began subsidizing the coconut oil industry to increase the islanders’ income from coconut production and reduce overfishing. The idea was that if

people spent more time growing coconuts, they would spend less time fishing. What happened? Fishing increased dramatically, and the reef fish population dropped precipitously, putting the whole ecosystem at risk. It turned out that paying people more to do coconut agriculture actually increased fishing because as people earned more money making coconut oil, they could work less to support themselves and spend more leisure time fishing. They didn’t just fish for income. They fished because they liked to fish, and so having more income from coconut production gave them more time to fish (Walsh, 2009).

• Economists and financial analysts spent decades building sophisticated statistical models to predict market behavior and manage both the domestic and the global economy, yet those models did not predict the financial crisis of 2008 that wiped out $50 trillion in global wealth and increased poverty and human suffering around the world. The sophisticated prediction models used by financial planners and government policymakers unexpectedly contributed to and deepened the crisis by promulgating a false sense of security that what actually unfolded could never happen (Brooks, 2010). In-depth interviews with the financial sector, government, and corporate leaders, who, in deep crisis mode, were trying to figure out what to do to avert a worldwide economic collapse, consistently reported a sense of disbelief and shock: “I don’t know how this happened.” “This was never supposed to happen.” “This can’t be happening.” But it did happen. And to this day, the full story of what happened and its long-term implications for the global economy are still unfolding.

• A social service program was created in Detroit to support young inner-city African Americans living with AIDS. Interviews with participants discovered that some of these destitute young people, when they heard about the services provided, had intentionally contracted HIV/AIDS to become eligible for social services. Despite knowing that it was a potentially fatal disease, they felt so hopeless that the risk felt worth taking to have someone care for them for once (Tourigny, 1998).

• The mission of the United States Forest Service highlighted prevention of forest fires. The Smokey Bear campaign was the centerpiece of a highly effective public campaign to prevent forest fires—by setting aside national forests and parks with the aim of protecting them from fire. The unanticipated consequence has been that the successful prevention of forest fires created a widespread habitat for fires that are much more destructive and devastating than the smaller, natural fires of the past. “So,” according to modern forest ecologists, “instead of a few dozen trees per acre, the Southwestern mountains of New Mexico, Arizona, Colorado and Utah are now choked with trees of all sizes, and grass and shrubs. Essentially, it’s fuel. And now fires are burning bigger and hotter. They’re not just damaging forests—they’re wiping them out. [In 2011] more than 74,000 wildfires burned over 8.7 million acres in the U.S.” (Joyce, 2012, p. 1).

The particular niche and contribution of qualitative methods in uncovering unanticipated consequences come from the openness of inquiry: asking open-ended interview questions, doing fieldwork in a way that is open to whatever turns up, studying documents to discover patterns that are hidden in the details, and observing with open eyes and an open mind. There is a lot of rhetoric in research and evaluation about the importance of looking for unintended consequences, but the rhetoric rings hollow unless the design includes sufficient time, money, and investigator skills to do fieldwork and undertake genuinely open-ended inquiry to find out what is actually happening. You can’t find this out from surveys and performance indicators because you don’t know what you don’t know, so you can’t ask questions about it or measure it. The dirty little secret in much of research and evaluation is that the designs do not give serious attention to the emergent and unexpected because those who design studies are primarily interested in testing their predetermined hypotheses and analyzing established indicators rather than openly inquiring into the complex and dynamic ways in which the real world unfolds.

The mind-set that is critical in open inquiry is to expect the unexpected, look for it, and see where it leads you. That is the nature, niche, and value of qualitative inquiry. And, as noted in the previous section, context comes into play. As the famed and infamous actress Angelina Jolie explained in summarizing her youth, “You’re young, you’re drunk, you’re in bed, you have knives; shit happens.”

Stuff happens everywhere. Qualitative inquiry documents the stuff that happens among real people in the real world in their own words, from their own perspectives, and within their own contexts; it then makes sense of the stuff that happens by finding patterns and themes among the seeming chaos and idiosyncrasies of lots of

stuff. Woody Allen, award-winning filmmaker, once witticized, “If you want to make God laugh, tell him about your plans.” Qualitative researchers and evaluators document the plans and study the consequences of attempting to carry out those plans, both intended and unintended.

Making Case Comparisons to Discover Important Patterns and Themes

The Internet is a big deal, but electricity was bigger.

Building a great company requires adherence to principles predating both. —Jim Collins (2000)

“The Timeless Physics of Great Companies”

Jim Collins identifies principles of organizational effectiveness by comparing successful and unsuccessful companies. He is one of the most influential management scholars and consultants of the twenty-first century. His books have sold more than 10 million copies worldwide.

• Built to Last: Successful Habits of Visionary Companies compares businesses that have endured over time with those that have failed (Collins & Porras, 2004).

• Good to Great: Why Some Companies Make the Leap . . . and Others Don’t (Collins, 2001a), translated into 35 languages, compares matched pairs of companies that started out in the same industries with similar profiles (market share, size, and profitability). One in each pair remained “good,” while the other went on to become “great” (market share dominance, sustained profit growth, and high prestige).

• How the Mighty Fall presents case studies of how some great companies self-destruct (Collins, 2009). • Great by Choice: Uncertainty, Chaos, and Luck—Why Some Thrive Despite Them All compares

companies that successfully adapted to complex dynamic environments with those that failed to adapt— and thus failed (Collins & Hansen, 2011).

The cornerstone of Collins’s research method is conducting in-depth case studies that allow drawing systematic contrasts between successful and unsuccessful companies.

The critical question is not “What do successes share in common?” or “What do failures share in common?” The critical question is “What do we learn by studying the contrast between success and failure?” Think of it this way: Suppose you wanted to study what makes gold medal winners in the Olympic Games. If you only studied the gold medal winners by themselves, you’d find that they all had coaches. But if you looked at the athletes that made the Olympic team, but never won a medal, you’d find that they also had coaches! The key question is, “What systemically distinguishes gold medal winners from those who never won a medal?”

Our comparison method has proven to be the key for calling into question powerfully entrenched myths and discerning fundamental principles that apply over long stretches of time and across a wide range of circumstances. (Collins, 2012, p. 1)

Qualitative Inquiry Contributions In this module, we have looked at seven kinds of knowledge-generating contributions that can flow from

qualitative inquiry:

1. Illuminating meaning

2. Studying how things work 3. Capturing stories to understand people’s perspectives and experiences 4. Elucidating how systems function and their consequences for people’s lives 5. Understanding context: how and why it matters 6. Identifying unanticipated consequences 7. Making case comparisons to discover important patterns and themes across cases

Exhibit 1.2 summarizes these contributions.

As the book unfolds, we will examine many more contributions of qualitative inquiry to knowledge, both theory and practice, and the interconnections between the two. But we’re just getting going. Let’s step back and more explicitly define the nature and niche of qualitative methods.

EXHIBIT 1.2 The Contributions of Qualitative Inquiry: Seven Examples

MODULE

2 What Makes Qualitative Data Qualitative

Not everything that counts can be counted, and not everything that can be counted counts. —William Bruce Cameron (1963)

Sociologist

Qualitative inquiry includes collecting quotes from people, verifying them, and contemplating what they mean. For an in-depth qualitative inquiry into the Cameron quote above, including its origin, varied versions over time, and diverse attributions, including Albert Einstein and many others, see Garson O’Toole (2010). My extensive use of quotations throughout this book treats them as both examples of qualitative data and sources of insight.

Qualitative reports describe and interpret something—whatever was studied. The data are words, stories, observations, and documents. Qualitative findings are based on three kinds of data: (1) in-depth, open-ended interviews; (2) direct observations; and (3) written communications. Interviews yield direct quotations from people about their experiences, opinions, feelings, and knowledge. Data from observations consist of detailed descriptions of people’s activities, behaviors, actions, and the full range of interpersonal interactions and organizational processes that are part of observable human experience. Written communications are a rich source of data. Finding, studying, and analyzing documents of all kinds are a part of qualitative inquiry. For example, qualitative data can include excerpts, quotations, or entire passages from organizational, clinical, or program records; memoranda and correspondence; social media postings; official publications and reports; personal diaries; and open-ended written responses to questionnaires and surveys (see Exhibit 1.3).

EXHIBIT 1.3 Three Kinds of Qualitative Data

1. Interviews

Open-ended questions and probes yield in-depth responses about people’s experiences, perceptions, opinions, feelings, and knowledge. Data consist of verbatim quotations with sufficient context to be interpretable.

2. Observations and fieldwork

Fieldwork descriptions of activities, behaviors, actions, conversations, interpersonal interactions, organizational or community processes, or any other aspect of observable human experience are documented. Data consist of field notes: rich, detailed descriptions, including the context within which the observations were made.

3. Documents

Written materials and documents from organizational, clinical, or program records; social media postings of all kinds; memoranda and correspondence; official publications and reports; personal diaries, letters, artistic works, photographs, and memorabilia; and written responses to open-ended surveys are collected. Data consist of excerpts from documents captured in a way that records and preserves the context.

The data for qualitative analysis typically come from fieldwork. During fieldwork the researcher spends time in the setting under study—a program, an organization, a community, or wherever situations of importance to a study can be observed, people interviewed, and documents analyzed. The researcher makes firsthand observations of activities and interactions, sometimes engaging personally in those activities as a “participant observer.” For example, an evaluator might participate in all or part of the program under study, participating as a regular program member, client, or student. The qualitative researcher talks with people about their experiences and perceptions. More formal individual or group interviews may be conducted. Relevant records and documents are examined. Extensive field notes are collected through these observations, interviews, and document reviews. The voluminous raw data in these field notes are organized into readable narrative descriptions, with major themes, categories, and illustrative case examples extracted through content analysis. The themes, patterns, understandings, and insights that emerge from fieldwork and subsequent analysis are the fruit of qualitative inquiry.

Mixed Methods Qualitative findings may be presented alone or in combination with quantitative data. Research and evaluation studies employing multiple methods, including combinations of qualitative and quantitative data, are common. At the simplest level, a questionnaire or interview that asks both fixed-choice (closed) questions and open-ended questions is an example of how quantitative measurement and qualitative inquiry are often combined. Here’s an example.

Quantitative survey question (fixed, scaled response categories):

How satisfied are you with the quality of public transportation in your area?

1. Very satisfied 2. More satisfied than dissatisfied 3. More dissatisfied than satisfied 4. Very dissatisfied

Qualitative follow-up question (open-ended):

1. What are you especially satisfied with? 2. What are you especially dissatisfied with?

Mixed methods yield both statistics and stories. Such studies report how many people fall into categories of interest and provide quotations and stories to elucidate what the numbers mean. See Exhibit 1.4 for an example.

The Quality of Qualitative Data The quality of qualitative data depends to a great extent on the methodological training, skill, sensitivity, and integrity of the researcher. Systematic and rigorous observation involves far more than just being present and looking around. Skillful interviewing requires much more than just asking questions. Credible content analysis demands considerably more than just reading to see what’s there. Generating meaningful and useful qualitative findings through observation, interviewing, and content analysis requires discipline, knowledge, training, practice, creativity, and hard work.

EXHIBIT 1.4 Mixed-Methods Example

Statistical data. Roughly 30% of entering freshmen in the United States are first-generation college students, and 24% (4.5 million) are both first-gens and low income. Nationally, 89% of low-income first- gens leave college within six years, without a degree. More than a quarter leave after their first year—four times the drop-out rate of higher-income second-generation students (Ramsey & Peale, 2010, p. 1).

Qualitative case quotation. “I did okay in high school, and everybody always said I’d go to college, but I didn’t even know you had to apply to go to college. Nobody in my family had ever been to college. I thought it was like going from primary school to high school, that the school authorities would just tell me where I’d be going. Looking back, it’s hard to believe that we were so ignorant, but sometimes you don’t know what you don’t know. We knew nothing about college at all. Me or my family. Nothing. Every bit of it is new to me and my family. It’s pretty scary, but I’m figuring it out” (Quote from Mike, a freshman at a large public university).

This chapter provides an overview of qualitative inquiry. Later chapters examine how to choose among the many options available within the broad range of qualitative methods, theoretical perspectives, and applications; how to design a qualitative study; how to use observational methods and conduct in-depth, open- ended interviews; and how to analyze qualitative data to generate findings. To set the stage for those more detailed methodological discussions, it will be helpful to have a better sense of the kinds of findings that emerge from qualitative studies. If you want to get somewhere, it’s helpful to have a sense of the destination. So let’s look at the findings from some classic qualitative studies.

Qualitative Findings: Themes, Patterns, Concepts, Insights, Understandings

Newton and the apple. Freud and anxiety. Jung and dreams. Piaget and his children. Darwin and Galapagos tortoises. Marx and England’s factories. Whyte and street corners. What are you obsessed with understanding?

—Halcolm

Mary Field Belenky and her colleagues set out to study women’s ways of knowing. They conducted extensive interviews with 135 women from diverse backgrounds probing how they thought about knowledge, authority, truth, themselves, life changes, and life in general. They worked as a team to group similar responses and stories together, informed partly by previous research but ultimately basing the analysis on their own collective sense of what categories best captured what they found in the narrative data. They argued with each other about which responses belonged in which categories. They created and abandoned categories. They looked for commonalities and differences. They worked hard to honor the diverse points of view they found, while also seeking patterns across stories, experiences, and perspectives. One theme emerged as particularly powerful: “Again and again women spoke of ‘gaining voice’” (Belenky, Clinchy, Goldberger, & Tarule, 1986, p. 16).

Voice versus silence emerged as a central metaphor for informing variations in ways of knowing. After painstaking analysis, they ended up with the five categories of knowing summarized in Exhibit 1.5, a framework that became very influential in women’s studies and represents one kind of fruit from qualitative inquiry.

One of the best-known and most influential books on organizational development and management is In Search of Excellence: Lessons From America’s Best-Run Companies by Peters and Waterman (1982). The

authors based the book on case studies of 62 highly regarded companies. They visited companies, conducted extensive interviews, and studied corporate documents. From that massive amount of data, they extracted eight attributes of excellence: (1) a bias for action; (2) being close to the customer; (3) autonomy and entrepreneurship; (4) productivity through people; (5) hands-on, value-driven work; (6) sticking to the knitting; (7) simple form and lean staff; and (8) simultaneous loose and tight properties. Their book devotes a chapter to each theme, with case examples and implications. Their research helped launch the quality movement, which has now moved from the business world to not-for-profit organizations and government. This study also illustrates a common qualitative sampling strategy: studying a relatively small number of special cases that are successful at something and therefore a good source of lessons learned.

EXHIBIT 1.5 Women’s Ways of Knowing: An Example of Qualitative Findings

Silence. A position in which women experience themselves as mindless and voiceless and subject to the whims of external authority.

Received knowledge. Women conceive of themselves as capable of receiving, even reproducing knowledge from external authorities but not capable of creating knowledge on their own.

Subjective knowledge. A perspective from which truth and knowledge are conceived as personal, private, and subjectively known or intuited.

Procedural knowledge. Women are invested in learning and apply objective procedures for obtaining and communicating knowledge.

Constructed knowledge. Women view all knowledge as contextual, experience themselves as creators of knowledge, and value both subjective and objective strategies for knowing.

SOURCE: Belenky et al. (1986, p. 15).

Stephen Covey (1989) used this same sampling approach in doing case studies of “highly effective people.” He identified seven habits these people practice: (1) being proactive; (2) beginning with the end in mind; (3) putting first things first; (4) thinking win–win; (5) seeking first to understand, then seeking to be understood; (6) synergizing, or engaging in creative cooperation; and (7) self-renewal. Kurtzman and Goldsmith (2010) studied highly effective leaders and identified patterns in how they get organizations to “achieve the extraordinary.”

What these influential books have in common is distilling a small number of important “lessons” from a huge amount of data based on outstanding exemplars. It is common in qualitative analysis for mounds of field notes and months of work to reduce to a small number of core themes. The quality of the insights generated is what matters, not the number of such insights. For example, in an evaluation of 34 programs aimed at people in poverty, we found a core theme that separated more effective from less effective programs: How people are treated affects how they treat others. If staff members are treated autocratically and insensitively by management, with suspicion and disrespect, staff will treat clients the same way. Contrariwise, responsiveness reinforces responsiveness, and empowerment breeds empowerment. These insights became the centerpiece of subsequent cross-project, collaborative organizational and staff development processes.

Jim Collins and his colleagues have continued and updated this methodological approach, conducting in- depth case studies of high-performing organizations. Their findings have been reported in a series of best- selling books mentioned earlier in this chapter in the section on making case comparisons to discover important patterns and themes. In their study of companies that thrive under conditions of uncertainty and chaos (Collins & Hansen, 2011), they began with an initial list of 20,400 companies and screened for small companies that achieved “spectacular results”: at least 10-fold growth over 15+ years through good times and bad. Only 7 companies made it into the final study sample. Their in-depth case studies revealed and explained three critical success factors: (1) fanatic discipline, (2) productive paranoia, and (3) empirical creativity. They also debunked “entrenched myths” such as the following:

• Successful leaders are not bold, risk-taking visionaries. • High performance is not distinguished by innovation. • Acting quickly and making fast, real-time decisions is not an effective way of dealing with rapid change. • Radical internal change is not an effective response to turbulent external environments (Collins & Hansen,

2011, pp. 9–10).

A different kind of qualitative finding is illustrated by Angela Browne’s book When Battered Women Kill (1987). Browne conducted in-depth interviews with 42 women from 15 states who were charged with a crime in the death or serious injury of their mates. She was often the first to hear these women’s stories. She used one couple’s history and vignettes from nine others, representative of the entire sample, to illuminate the progression of an abusive relationship from romantic courtship to the onset of abuse through its escalation until it was ongoing and eventually provoked a homicide. Her work helped lead to legal recognition of battered women’s syndrome as a legitimate defense, especially in offering insight into the common outsider’s question: Why doesn’t the woman just leave? Getting an insider perspective on the debilitating, destructive, and all-encompassing brutality of battering reveals that question for what it is: the facile judgment of one who hasn’t been there. The effectiveness of Browne’s careful, detailed, and straightforward descriptions and quotations lies in their capacity to take us inside the abusive relationship. Offering that inside perspective powers qualitative reporting.

SIDEBAR

QUALIA

Qualia refers to what we, as humans, subjectively add to our physical, sensory experience of the world through consciousness. Consciousness may fundamentally involve processing qualia as a neurological capacity (Ramachandran & Blakeslee, 1998). Qualia are studied, and debated, at the intersection between philosophy of mind and brain science (Tye, 2013).

Clarence Irving Lewis (1929) coined the term qualia in 1929 in his book Mind and the World Order:

There are recognizable qualitative characters of the given, which may be repeated in different experiences, and are thus a sort of universals; I call these “qualia.” But although such qualia are universals, in the sense of being recognized from one to another experience, they must be distinguished from the properties of objects . . . because it is purely subjective. (p. 3)

Qualia cannot be measured and standardized; they can only be experienced and reported. Qualia are fundamentally, inherently, neurologically, essentially, and epistemologically qualitative. If this intrigues you and you’re looking for a philosophical debate to get engaged with and mired in, see Qualia (2013).

This quick sampling of findings from classic qualitative studies is like a wine tasting, meant to introduce possibilities and support developing a more sophisticated palate or, like appetizers, as an opening to the fuller feast yet to come. Many important scholars have contributed to knowledge through qualitative inquiry. Many breakthroughs in our knowledge of how the world works and why it works as it does have emerged from qualitative studies. Exhibit 1.9, at the end of this chapter (p. 41), provides examples of distinguished and prestigious qualitative research pioneers throughout history and across a broad range of disciplines. The next section discusses some of the different purposes of and audiences for qualitative inquiry.

Different Purposes of and Audiences for Qualitative Studies: Research, Evaluation, Dissertations, and Personal Inquiry

As the title of this book indicates, qualitative methods are used in both research and evaluation. But because the purposes of research and evaluation are different, the criteria for judging qualitative studies can vary depending on a study’s purpose. This point is important. It means that one can’t judge the appropriateness of the methods in any study or the quality of the resulting findings without knowing the study’s purpose, agreed- on uses, and intended audiences. Evaluation and research typically have different purposes, expected uses, and intended users. Dissertations add yet another layer of complexity to this mix. Let’s begin with research.

Qualitative Research

Research aims to generate or test theory and contribute to knowledge. Research findings describe how the world works and why it works as it does. Such knowledge, and the theories that undergird knowledge, may subsequently inform action and evaluation, but action is not the primary purpose of fundamental research. Qualitative inquiry is especially powerful as a source of grounded theory, theory that is inductively generated from fieldwork, that is, theory that emerges from the researcher’s observations and interviews out in the real world rather than in the laboratory or the academy. The primary audiences for research are other researchers and scholars, as well as policymakers and others interested in understanding some phenomenon or problem of interest. The research training, methodological preferences, and scientific values of those who use research will affect how valuable and credible they find the empirical and theoretical findings of qualitative studies.

Qualitative Dissertations

Dissertations and graduate theses offer special insight into the importance of attention to audience. Savvy graduate students learn that to complete a degree program, the student’s committee must approve the work. The particular understandings, values, preferences, and biases of committee members come into play in that approval process. They will, in essence, evaluate the student’s contribution, including the quality of the methodological procedures followed and the analysis done. Qualitative dissertations, once quite rare, have become increasingly common as the criteria for judging qualitative contributions to knowledge have become better understood and accepted. But those criteria are not absolute or universally agreed on. As we shall see, there are many varieties of qualitative inquiry and multiple criteria for judging quality, many of which remain disputed.

Qualitative Evaluations

Program evaluation is the systematic collection of information about the activities, characteristics, and outcomes of programs to make judgments about the program, improve program effectiveness, and/or inform decisions about future programming. Policies, organizations, and personnel can also be evaluated. Evaluative research, quite broadly, can include any effort to judge or enhance human effectiveness through systematic data-based inquiry. Human beings are engaged in all kinds of efforts to make the world a better place. These efforts include assessing needs, formulating policies, passing laws, delivering programs, managing people and resources, providing therapy, developing communities, changing organizational culture, educating students, intervening in conflicts, and solving problems. In these and other efforts to make the world a better place, the question of whether the people involved are accomplishing what they want to accomplish arises. When one examines and judges accomplishments and effectiveness, one is engaged in evaluation.

Qualitative methods are often used in evaluations because they tell the program’s story by capturing and communicating the participants’ stories. Evaluation case studies have all the elements of a good story. They tell what happened when, to whom, and with what consequences. Many examples in this book are drawn from program evaluation, policy analysis, and organizational development. The purpose of such studies is to gather information and generate findings that are useful. Understanding the program’s and participants’ stories is useful to the extent that they illuminate the processes and outcomes of the program for those who must make decisions about the program. In Essentials of Utilization-Focused Evaluation (Patton, 2012a), I have presented a comprehensive approach to doing evaluations that are useful, practical, ethical, accurate, and accountable. The primary criterion for judging such evaluations is the extent to which the intended users

actually use the findings for decision making and program improvement. The methodological implication of this criterion is that the intended users must value the findings and find them credible. They must be interested in the stories, experiences, and perceptions of the program participants, beyond simply knowing how many came into the program, how many completed it, and how many did what afterward. Qualitative findings in evaluation illuminate the people behind the numbers and put faces on the statistics to deepen understanding and inform decision making.

Your Personal Interest and Passion as a Basis for Qualitative Inquiry

While the preceding discussion of evaluation, research, and dissertations has emphasized taking into account external audiences and consumers of qualitative studies, it is also important to acknowledge that you may study something because you want to understand it. As my children grew to adulthood, I found myself asking questions about coming of age in modern society, so I undertook a personal inquiry that became a book (Patton, 1999). But I didn’t start out to write a book. I started out trying to understand my own experience and the experiences of my children. That is a form of qualitative inquiry.

SIDEBAR

TOP TEN PIECES OF ADVICE TO A GRADUATE STUDENT CONSIDERING A QUALITATIVE DISSERTATION

The following query was posted on an Internet listserv devoted to discussing qualitative inquiry:

I am a new graduate student thinking about doing a qualitative dissertation. If you could give just one bit of advice to a student considering qualitative research for a dissertation, what would it be?

The responses below came from different people. I’ve combined some responses, edited them (while trying to maintain the flavor of the postings), and arranged them for coherence.

1. Be sure that a qualitative approach fits your research questions and interests. (Chapter 2 will help with this by presenting the primary themes of qualitative inquiry.)

2. Study qualitative inquiry. There are lots of different approaches and a lot to know. So it’s not a matter of just using qualitative methods but using a particular framework for undertaking a qualitative study. (Chapter 3 covers different qualitative theoretical orientations and approaches.)

3. Find a dissertation adviser who will support you in doing qualitative research. Otherwise, it can be a long, tough haul. A dissertation is a big commitment. There are other practical approaches to using qualitative methods that don’t involve all the constraints of doing a dissertation, things like program evaluation, action research, and organizational development. You can still do lots of great qualitative work without doing a dissertation. But if you can find a supportive adviser and committee, then, by all means, go for it. (Chapter 4 covers particularly appropriate practical applications of qualitative methods.)

4. Really work on design. Qualitative designs follow a different logic from quantitative research, especially with regard to purposeful sampling and conducting in-depth case studies. This is not the same as questionnaires and tests and experiments. You can combine designs, like quant and qual approaches, but integrating both kinds of data can be challenging. Either way, you have to understand what’s unique about qualitative designs. (Chapter 5 covers qualitative designs.)

5. Practice interviewing and observation skills. Practice! Practice! Practice! Do lots of interviews. Spend a lot of time doing practice fieldwork observations. Get feedback from someone who’s really good at interviewing and observations. There’s an amazing amount to learn. And it’s not just head stuff. Qualitative research takes skill. Don’t make the mistake of thinking it’s easy. The better I get at

it, the more I realize how bad I was when I started. (Chapters 6 and 7 cover the skills of qualitative inquiry.)

6. Figure out how to do qualitative analysis before you gather data. I’ve talked with lots of advanced grad students who rushed to collect data before they knew anything about analyzing it—and lived to regret it big time. This is true for statistical data, but somehow people seem to think that qualitative data are easy to analyze. No way. That’s a big-time NO WAY. And don’t think that the new software will solve the problem. Another big-time NO WAY. You, that is, YOU, still have to analyze the data. (Chapter 8 covers analysis.)

7. Be sure that you’re prepared to deal with the controversies of doing qualitative research. People on this listserv are constantly sharing stories about people who don’t “get” qualitative research and put it down. Don’t go into it naively. Understand the paradigms and politics. (Chapter 9 deals with the paradigms, politics, and ways of enhancing the credibility of qualitative inquiry.)

8. Do it because you want to and are convinced that it’s right for you. Don’t do it because someone told you it would be easier. It’s not. Try as hard as possible to pick/negotiate dissertation research questions that have to do with some passion/interest in your professional life. Qualitative research is time-consuming, intimate, and intense; you will need to find your questions interesting if you want to be at all sane during the process—and still sane at the end.

9. Find a good mentor or support group. Or both. In fact, find several of each. If you can, start a small group of peers in the same boat, so to speak, to talk about your research together on a regular basis— you can share knowledge, brainstorm, and problem solve, as well as share in each other’s successes, all in a more relaxed environment that helps take some of the edge off the stress (e.g., you might have potluck meals at different homes). This can be tremendously liberating (even on a less than regular basis). Take care of yourself.

10. Prepare to be changed. Looking deeply at other people’s lives will force you to look deeply at yourself.

Additional resources for graduate students:

• Completing Your Qualitative Dissertation: A Road Map From Beginning to End (Bloomberg & Volpe, 2012)

• The Qualitative Dissertation: A Guide for Students and Faculty (Piantanida & Garman, 2009) • A Practical Guide to the Qualitative Dissertation (Biklen & Casella, 2007) • Stretching Exercise for Qualitative Researchers (Janesick, 2011)

While doing interviews with recipients of MacArthur Foundation Fellowships (popularly called “Genius Awards”), I was told by a social scientist that her fieldwork was driven by her own search for understanding and that she disciplined herself to not even think about publication while engaged in interviewing and observing because she didn’t want to have her inquiry affected by attention to external audiences. She wanted to know because she wanted to know, and she had made a series of career and professional decisions that allowed her to focus on her personal inquiry without being driven by the traditional academic admonition to “publish or perish.” She didn’t want to subject herself to or have her work influenced by external criteria and judgment.

Thus far in this first chapter, we’ve looked at how qualitative inquiry contributes to our understanding of the world (Module 1) and what makes qualitative data qualitative (Module 2). We turn now to making methods decisions.

SIDEBAR

WHAT BRINGS YOU TO YOUR CHOSEN INQUIRY? THE INTERSECTION OF PERSONAL INTEREST AND SOCIETAL DYNAMICS

In his classic book The Sociological Imagination, C. Wright Mills (1959/2000) challenged scholars in the human disciplines to develop a point of view and a methodological attitude that would allow them to examine how the private troubles of individuals, which occur within the immediate world of experience, are connected to public issues and to public responses to these troubles. Mills’s sociological imagination was biographical, international, and historical.

Mills wanted [social science] to make a difference in the lives people lead. . . .

I want a critical methodology that enacts its own version of the sociological imagination. Like Mills, my version of the imagination is moral and methodological. And like Mills, I want a discourse that troubles the world, understanding that all inquiry is moral and political. . . . in the connection between critical inquiry and social justice.

—Norman K. Denzin (2010, p. 9) The Qualitative Manifesto: A Call to Arms

This is what undergirds and energizes the work of qualitative inquiry pioneer Norman K. Denzin, among the most prolific and eclectic of all qualitative methodologists. He took inspiration from the vision and imagination of C. Wright Mills. What is the source of your inspiration? What motivates you? What is your vision of the contribution you want to make? These questions are fundamental to inquiry because the answers get you through the long, hard slog of fieldwork, arduous data analysis, and disputes about conclusions.

Enter not through this portal of inquiry

if thee be languorous, lackadaisical,

timorous, or without vision.

—Halcolm

MODULE

3 Making Methods Decisions

You cannot successfully determine beforehand which side of the bread to butter. —The Law of the Perversity of Nature

The implication of thinking about purpose and audience in designing studies is that methods, no less than knowledge, are dependent on context. No rigid rules can prescribe what data to gather to investigate a particular interest or problem. There is no recipe or formula in making methods decisions. The widely respected social science methodologist and psychometrician Lee J. Cronbach observed that designing a study is as much art as science. It is “an exercise of the dramatic imagination” (Cronbach, 1982, p. 239). In research as in art there can be no single, ideal standard. Beauty no less than “truth” is in the eye of the beholder, and the beholders of research and evaluation can include a plethora of stakeholders: scholars, policymakers, funders, program managers, staff, program participants, journalists, critics, and the general public. Any given design inevitably reflects some imperfect interplay of resources, capabilities, purposes, possibilities, creativity, and personal judgments by the people involved. Research, like diplomacy, is the art of the possible. Exhibit 1.6 provides a set of questions to consider in the design process, regardless of type of inquiry. With that background, we can turn to consideration of the relative strengths and weaknesses of qualitative and quantitative methods.

EXHIBIT 1.6 Some Guiding Questions and Options for Making Methods Decisions

Methods Choices: Contrasting Qualitative and Quantitative Emphases

The key to making a good forecast is weighing quantitative and qualitative information appropriately.

—Nate Silver (2012, p. 100)

Thinking about design alternatives and methods choices leads directly to consideration of the relative strengths and weaknesses of qualitative and quantitative data. The approach here is pragmatic. Some questions lend themselves to numerical answers, and some don’t. If you want to know how much people weigh, use a scale. If you want to know if they’re obese, measure body fat in relation to height and weight and compare the results with population norms. If you want to know what their weight means to them, how it affects them, how they think about it, and what they do about it, you need to ask them questions, find out about their experiences, and hear their stories. A comprehensive and multifaceted understanding of weight in people’s lives requires both their numbers and their stories. Doctors who look only at test results and don’t also listen to their patients are making judgments with inadequate knowledge.

Qualitative methods facilitate study of issues in depth and detail. Approaching fieldwork without being constrained by predetermined categories of analysis contributes to the depth, openness, and detail of qualitative inquiry. Quantitative methods, on the other hand, require the use of standardized measures so that the varying perspectives and experiences of people can be fit into a limited number of predetermined response categories to which numbers are assigned.

The advantage of a quantitative approach is that it’s possible to measure the reactions of a great many people to a limited set of questions, thus facilitating comparison and statistical aggregation of the data. This gives a broad, generalizable set of findings presented succinctly and parsimoniously. By contrast, qualitative methods typically produce a wealth of detailed information about a much smaller number of people and cases. This increases the depth of understanding of the cases and situations studied but reduces generalizability.

Validity in quantitative research depends on careful instrument construction to ensure that the instrument measures what it is supposed to measure. The instrument must then be administered in an appropriate, standardized manner according to prescribed procedures. The focus is on the measuring instrument—the test items, survey questions, or other measurement tools. In qualitative inquiry, the researcher is the instrument. The credibility of qualitative methods, therefore, hinges to a great extent on the skill, competence, and rigor of the person doing the fieldwork—as well as the things going on in a person’s life that might prove to be a distraction. Qualitative methodology pioneers Egon Guba and Yvonna Lincoln (1981) have commented on this aspect of qualitative research:

Fatigue, shifts in knowledge, and cooptation, as well as variations resulting from differences in training, skill, and experience among different “instruments,” easily occur. But this loss in rigor is more than offset by the flexibility, insight, and ability to build on tacit knowledge that is the peculiar province of the human instrument. (p. 113)

Because qualitative and quantitative methods involve differing strengths and weaknesses, they constitute alternative, but not mutually exclusive, strategies for research. Both qualitative and quantitative data can be collected in the same study. To further illustrate these contrasting approaches and provide concrete examples of the fruit of qualitative inquiry, the rest of this chapter presents select excerpts from actual studies.

Comparing Two Kinds of Data: An Example

The Technology for Literacy Center was a computer-based adult literacy program in Saint Paul, Minnesota. It operated out of a storefront facility in a lower-socioeconomic area of the city. In 1988, after three years of pilot operation, a major funding decision had to be made about whether to continue the program. Anticipating

the funding decision, a year earlier, local foundations and public schools supported a “summative evaluation” to determine the overall outcomes and cost-effectiveness of the center. The evaluation design included both quantitative and qualitative data.

The quantitative testing data showed great variation. The statistics on average achievement gains masked the great differences among the participants. The report concluded that, while testing showed substantial achievement test gains for the treatment group versus the control group, the more important finding concerned the highly individualized nature of student progress. The report concluded that “the data on variation in achievement and instructional hours lead to a very dramatic, important and significant finding: there is no average student at TLC (Patton & Stockdill, 1987, p. 33).

This finding highlights the kind of program or treatment situation where qualitative data are particularly helpful and appropriate. The Technology for Literacy Center has a highly individualized program in which learners proceed at their own pace based on specific needs and interest. The students come in at very different levels, with a range of goals, participate in widely varying ways, and make very different gains. Average gain scores and average hours of instruction provide a parsimonious overview of aggregate progress. Adding case studies helps funders understand individual variation and what that variation means. To get at the meaning of the program for individual participants, the evaluation included case studies and qualitative data from interviews.

Individual Case Examples

One case was the story of Barbara, a 65-year-old black grandmother who came to Minnesota after a childhood in the Deep South. She worked as a custodian and house cleaner and was proud of never having been on welfare. She was the primary breadwinner for a home with five children spanning three generations, including her oldest daughter’s teenage children, for whom she had cared since her daughter’s unexpected death from hepatitis. During the week, she seldom got more than three hours of sleep each night. At the time of the case study, she had spent 15 months in the literacy program and progressed from not reading at all (second-grade level) to being a regular library user (and testing a grade level higher than where she began). She developed an interest in black history and reported being particularly pleased at being able to read the Bible on her own. She described what it was like not being able to read:

Where do you go for a job? You can’t make out an application. You go to a doctor, and you can’t fill out the forms, and it’s very embarrassing. You have to depend on other people to do things like this for you. Sometimes you don’t even want to ask your own kids because it’s just like you’re depending too much on people, and sometimes they do it willingly, and sometimes you have to beg people to help. . . .

All the progress has made me feel lots better about myself because I can do some of the things I’ve been wanting to do and I couldn’t do. It’s made me feel more independent to do things myself instead of depending on other people to do them for me.

A second contrasting case tells the story of Sara, a 42-year-old Caucasian woman who dropped out of school in the 10th grade. She worked as an office manager and tested at 12th-grade level on entry to the program. After 56 hours of study over 17 days, she passed the exam to receive a Graduate Equivalency Degree (GED), making her a high school graduate. She immediately entered college. She said that the decision to return for her GED was

an affirmation, as not having a diploma had really hurt me for a long time. . . . It was always scary wondering if somebody actually found out that I was not a graduate that they would fire me or they wouldn’t accept me because I hadn’t graduated. The hardest thing for me to do was tell my employer. He is very much into education, and our company is education oriented. So the hardest thing I ever had to do was tell him I was a high school dropout. I needed to tell because I needed time to go and take the test. He was just so understanding. I couldn’t believe it. It was just wonderful. I thought he was going to be disappointed in me, and he thought it was wonderful that I was going back. He came to graduation.

These short excerpts from two contrasting cases illustrate the value of detailed, descriptive data in deepening our understanding of individual variation. Knowing that each woman progressed about one grade level on a standardized reading test is only a small part of a larger, much more complex picture. Yet, with more than 500 people in the program, it would be overwhelming for funders and decision makers to attempt to make sense of 500 detailed case studies (about 5,000 double-spaced pages). Statistical data provide a succinct and parsimonious summary of major patterns, while select case studies provide depth, detail, and individual meaning.

Open-Ended Interview Responses

Another instructive contrast is to compare closed-ended questionnaire results with responses to open-ended group interviews. Questionnaire responses to quantitative, standardized items indicated that 77% of the adult literacy students were “very happy” with the Technology for Literacy Center program; 74% reported learning “a great deal.” These and similar results revealed a general pattern of satisfaction and progress. But what did the program mean to students in their own words?

To get the perspective of students, I conducted focus group interviews. I asked students to describe the program’s outcomes in personal terms. I asked, “What difference has what you are learning made in your lives?” Here are some of the responses:

• I love the newspaper now and actually read it. Yeah, I love to pick up the newspaper now. I used to hate it. Now I love the newspaper.

• I can follow sewing directions. I make a grocery list now, so I’m a better shopper. I don’t forget things. • Yeah, you don’t know how embarrassing it is to go shopping and not be able to read the wife’s grocery

list. It’s helped me out so much in the grocery store. • Helps me with my medicine. Now I can read the bottles and the directions! I was afraid to give the kids

medicine before because I wasn’t sure. • I don’t get lost anymore. I can find my way around. I can make out directions, read the map. I work in

construction, and we change locations a lot. Now I can find my way around. I don’t get lost anymore! • Just getting a driver’s license will be wonderful. I’m 50. If I don’t get the GED but if I can get a

license . . . ! I can drive well, but I’m scared to death of the written test. Just getting a driver’s license . . . , a driver’s license.

• Now I read outdoor magazines. I used to just read the titles of books—now I read the books! • I was always afraid to read at school and at church. I’m not afraid to read the Bible now at Bible class. It’s

really important to me to be able to read the Bible. • I can fill out applications now. You have to know how to fill out an application in this world. I can look in

the Yellow Pages. It used to be so embarrassing not to be able to fill out applications, not to be able to find things in the Yellow Pages. I feel so much better now. At least my application is filled out right, even if I don’t get the job, at least my application is filled out right.

• I’m learning just enough to keep ahead of my kids. My family is my motivation. Me and my family. Once you can read to your kids, it makes all the difference in the world. It helps you to want to read and to read more. When I can read myself, I can help them read so they can have a better life. The kids love it when I read to them.

These focus group interview excerpts provide some qualitative insights into the individual, personal experiences of adults learning to read. The questionnaire results (77% satisfied) provided data on statistically generalizable patterns, but the standardized questions only tap the surface of what it means for the program to have had “great perceived impact.” The much smaller sample of open-ended interviews adds depth, detail, and meaning at a very personal level of experience. The next example will show that qualitative data can yield not only deeper understanding but also political action as the depth of participants’ feelings is revealed.

SIDEBAR

DISCOVERING QUALITATIVE INQUIRY

Distinguished adult education scholar Malcolm Knowles created the discipline of andragogy (adult learning, in contrast to pedagogy, child learning). In his autobiography, The Making of an Adult Educator (1989), he listed his discovery of qualitative methods as an alternative way of studying and evaluating adult learning as one of the eight most important episodes of his life, right there alongside his marriage. He reported that it completely changed how he viewed the world and opened up critically important new ways of understanding how adults learn.

The Power of Qualitative Data In the early 1970s, the school system of Kalamazoo, Michigan, implemented a new accountability system. It was a complex system that included using standardized achievement tests administered in both fall and spring, criterion-referenced tests developed by teachers, performance objectives, teacher peer ratings, student ratings of teachers, parent ratings of teachers, principal ratings of teachers, and teacher self-ratings.

The Kalamazoo accountability system began to attract national attention. For example, the American School Board Journal reported in April 1974 that “Kalamazoo schools probably will have one of the most comprehensive computerized systems of personnel evaluation and accountability yet devised” (p. 40). In the first of a three-part series on Kalamazoo the American School Board Journal asserted, “Take it from Kalamazoo: a comprehensive, performance-based system of evaluation and accountability can work” (p. 32).

Not everyone agreed with that positive assessment, however. The Kalamazoo Education Association charged that teachers were being demoralized by the accountability system. Some school officials, on the other hand, argued that teachers did not want to be accountable. In the spring of 1976, the Kalamazoo Education Association, with assistance from the Michigan Education Association and the National Education Association, sponsored a survey of teachers to find out the teachers’ perspective on the accountability program (Perrone & Patton, 1976).

The education association officials were interested primarily in a questionnaire consisting of standardized items. One part of the closed-ended questionnaire provided the teachers with a set of statements with which they could agree or disagree. The questionnaire results showed that the teachers felt that the accountability system was largely ineffective and inadequate. For example, 90% of the teachers disagreed with the school administration’s published statement: “The Kalamazoo accountability system is designed to personalize and individualize education”; 88% reported that the system does not assist teachers to become more effective; 90% responded that the accountability system has not improved educational planning in Kalamazoo; 93% believed that “accountability as practiced in Kalamazoo creates an undesirable atmosphere of anxiety among teachers”; and 90% asserted, “The accountability system is mostly a public relations effort.” Nor did teachers feel that the accountability system fairly reflected what they did as teachers since 97% of them agreed that “accountability as practiced in Kalamazoo places too much emphasis on things that can be quantified, so that it misses the results of teaching that are not easily measured.”

It is relatively clear from these statements that most of the teachers who responded to the questionnaire were negative about the accountability system. When school officials and school board members reviewed the questionnaire results, however, many of them immediately dismissed those results by arguing that they had never expected teachers to like the system, teachers didn’t really want to be accountable, and the teachers’ unions had told their teachers to respond negatively anyway. In short, many school officials and school board members dismissed the questionnaire results as biased, inaccurate, and the result of teacher union leaders telling teachers how to respond in order to discredit the school authorities.

The same questionnaire included two open-ended questions. The first was placed midway through the questionnaire, and the second came at the end of the questionnaire.

1. Please use this space to make any further comments or recommendations concerning any component of the accountability system.

2. Finally, we’d like you to use this space to add any additional comments you’d like to make about any part of the Kalamazoo accountability system.

A total of 373 teachers (70% of those who responded to the questionnaire) took the time to respond to one of these open-ended questions. All of the comments made by the teachers were typed verbatim and included in the report. These open-ended data filled 101 pages. When the school officials and school board members rejected the questionnaire data, rather than argue with them about the meaningfulness of the teacher responses to the standardized items, we asked them to turn to the pages of open-ended teacher comments and simply read at random what the teachers said. Examples of the comments they read, and could read on virtually any page in the report, are reproduced below in six representative responses from the middle pages of the report.

Teacher Response No. 284: I don’t feel that fear is necessary in an accountability situation. The person at the head of a school system has to be human, not a machine. You just don’t treat people like they are machines!

The superintendent used fear in this system to get what he wanted. That’s very hard to explain in a short space. It’s something you have to live through to appreciate. He lied on many occasions and was very deceitful. Teachers need a situation where they feel comfortable. I’m not saying that accountability is not good. I am saying the one we have is lousy. It’s hurting the students—the very ones we’re supposed to be working for.

Teacher Response No. 257: This system is creating an atmosphere of fear and intimidation. I can only speak for the school I am in, but people are tense, hostile, and losing their humanity. Gone is the goodwill and team spirit of administration and staff, and I believe this all begins at the top. One can work in these conditions but why, if it is to “shape up” a few poor teachers. Instead, it’s having disastrous results on the whole faculty community.

Teacher Response No. 244: In order to fully understand the oppressive, stifling atmosphere in Kalamazoo, you have to “be in the trenches”—the classrooms. In 10 years of teaching, I have never ended a school year as depressed about “education” as I have this year. If things do not improve in the next 2 years, I will leave education. The Kalamazoo accountability system must be viewed in its totality and not just the individual component parts of it. In toto, it is oppressive and stifling.

In teaching government and history, students often asked what it was like to live in a dictatorship. I now know firsthand.

The superintendent, with his accountability model and his abrasive condescending manner, has managed in three short years to destroy teacher morale and effective creative classroom teaching.

Last evening, my wife and I went to an end of the school year party. The atmosphere there was strange—little exuberance, laughter, or release. People who in previous years laughed, sang, and danced were unnaturally quiet and somber. Most people went home early. The key topic was the superintendent, the school board election, and a millage campaign. People are still tense and uncertain.

While the school board does not “pay us to be happy,” it certainly must recognize that emotional stability is necessary for effective teaching to take place. The involuntary transfers, intimidation, coercion, and top to bottom “channelized” communication in Kalamazoo must qualify this school system for the list of “least desirable” school systems in the nation.

Teacher Response No. 233: I have taught in Kalamazoo for 15 years and under five superintendents. Until the present superintendent, I found working conditions to be enjoyable, and teachers and administration and the Board of Education all had a good working relationship. In the past 4 years—under the present superintendent—I find the atmosphere deteriorating to the point where teachers distrust each other and teachers do not trust administrators at all! We understand the position the administrators have been forced into and feel compassion for them—however, we still have no trust! Going to school each morning is no longer an enjoyable experience.

Teacher Response No. 261: A teacher needs some checks and balances to function effectively; it would be ridiculous to think otherwise—if you are a concerned teacher. But in teaching, you are not turning out neatly packaged little mechanical products, all alike and endowed with the same qualities. This nonsensical accountability program we have here makes the superintendent look good to the community. But someone who is in the classroom dealing with all types of kids, some who cannot read, some who hardly ever come to school, some who are in and out of jail, this teacher can see that and the rigid accountability model that neglects the above-mentioned problems is pure “BULLSHIT!”

Teacher Response No. 251: “Fear” is the word for “accountability” as applied in our system. My teaching before “Accountability” is the same as now. “Accountability” is a political ploy to maintain power. Whatever good there may have been in it in the beginning has been destroyed by the awareness that each new educational “system” has at its base a political motive. Students get screwed. . . . The bitterness and hatred in our system are incredible. What began as “noble” has been destroyed. You wouldn’t believe the new layers of administration that have been created just to keep this monster going.

Our finest compliment around our state is that the other school systems know what is going on and are having none of it. Lucky people. Come down and visit in hell sometime.

Face Validity and Credibility

What was the impact of the qualitative data collected from the teachers in Kalamazoo? You will recall that many of the school board members initially dismissed the standardized questionnaire responses as biased, rigged, and the predictable result of the union’s campaign to discredit school officials. However, after reading through a few pages of the teachers’ own personal comments, after hearing about the teachers’ experiences with the accountability system in their own words, the tenor of the discussion about the evaluation report changed. School board members could easily reject what they perceived as a “loaded” questionnaire. They could not so easily dismiss the anguish, fear, and depth of concern revealed in the teachers’ own reflections. The teachers’ words had face validity and credibility. Discussion of the evaluation results shifted from an attack on the measures used to the question “What do you think we should do?”

Not long after the evaluation report, the superintendent resigned. The new superintendent and school board used the evaluation report as a basis for starting afresh with the teachers. A year later, teacher association officials reported a new environment of teacher–administration cooperation in developing a mutually acceptable accountability system. The evaluation report did not directly cause these changes. Many other factors were involved in Kalamazoo at that time. However, the qualitative information in the evaluation report revealed the full scope and nature of teachers’ feelings about what it was like to work in the atmosphere created by the accountability system. The depth of those feelings as expressed in the teachers’ own words became part of the impetus for change in Kalamazoo.

The Purpose of Open-Ended Responses

Direct quotations are a basic source of raw data in qualitative inquiry, revealing respondents’ depth of emotion, the ways they have organized their world, their thoughts about what is happening, their experiences, and their basic perceptions. The task for the qualitative researcher is to provide a framework within which people can respond in a way that represents accurately and thoroughly their points of view about the world, or that part of the world about which they are talking, for example, their experience with a particular program being evaluated.

I have included the Kalamazoo evaluation findings as an illustration of qualitative inquiry because open- ended responses on questionnaires represent the most elementary form of qualitative data. There are severe limitations to open-ended data collected in writing on questionnaires, limitations related to the writing skills of respondents, the impossibility of probing or extending responses, and the effort required of the person completing the questionnaire. Yet, even at this elementary level of inquiry, the depth and detail of feelings revealed in the open-ended comments of the Kalamazoo teachers illustrate the fruit of qualitative methods.

The major way in which qualitative researchers seek to understand the perceptions, feelings, experiences, and knowledge of people is through in-depth, intensive interviewing, not just open-ended items on questionnaires. Chapter 7 on interviewing will present ways of gathering high-quality information from people. Effective interviewing techniques, skillful questioning, and the capacity to establish rapport are keys to obtaining credible and useful data through interviews. A particularly strong type of qualitative inquiry combines fieldwork observations with in-depth interviewing.

Combining Observations and Interviews

Qualitative data can include both direct observation and interview data. Sometimes a longitudinal study begins with observations and then continues with follow-up interviews. That was the design of a study of female college students. Sociologists Elizabeth A. Armstrong and Laura T. Hamilton, with a team of researchers, embedded themselves in a freshman dormitory at a major midwestern state university, observed the young women throughout their years in college, and then continued to gather data about their lives after college through in-depth interviews. They originally expected to learn a lot about romance and sex in college, but as often happens with open-ended qualitative fieldwork, what emerged as most important turned out to be different from what was expected. They ended up documenting the powerful effects of social class and socioeconomic status on college experiences and outcomes. They studied the culture of status seeking centered on sororities and found that the most well-resourced and seductive route to “success” was a “party pathway” anchored in the Greek sorority system, a system supported and encouraged by the university administration. This party pathway, they found, exerted influence over the academic and social experiences of all students but in different ways: It benefited the affluent, elite, and well-connected women but seriously disadvantaged the majority of women (Armstrong & Hamilton, 2013).

Inquiry by Observation What people say is a major source of qualitative data, whether what they say is obtained verbally through an interview or in written form through document analysis or survey responses. There are limitations, however, to how much can be learned from what people say. To understand fully the complexities of a situation, direct participation in and observation of the phenomenon of interest is a particularly fruitful method. The ideal observation captures context, the unfolding of events over time, and critical interactions, and it includes talking with those involved in the activities observed.

Observational data, especially participant observation, permit a program evaluator to understand a program or treatment to an extent not entirely possible using only the insights obtained through interviews. Of course, not everything can be directly observed or experienced—and participant observation is a highly labor- intensive and, therefore, a relatively expensive research strategy. Chapter 6 will present strategies and practices for high-quality fieldwork observations, including both participant and nonparticipant approaches.

© 2002 Michael Quinn Patton and Michael Cochran

Certain Really Discriminating People Like Nothing Better Than to Relax on the Beach With a Good, In-depth, and Detailed Qualitative Study in Hand

The purpose of qualitative observation is to take the reader into the setting observed. This means that observational data must have depth and detail. The data must be descriptive—sufficiently descriptive that the reader can understand what occurred and how it occurred. The observer’s notes become the eyes, ears, and perceptual senses for the reader. The descriptions must be thorough without being cluttered by irrelevant minutiae and trivia. The basic criterion to apply to a well-documented observation is the extent to which the observation permits the reader to enter and understand the situation under study.

SIDEBAR

THROUGH THE EYES OF A CHILD

Dr. Nancy Boxill, a child psychologist, studied homeless families in Atlanta.

I know of a small boy eight years old who sat alone on a park bench five or six hours every day for almost a week. He alternately played with the pigeons, watched the passing people, made patterns in the air with his feet and legs, or looked blankly into space. On the fourth day of his visit to this bench, a friend of mine asked this boy why he sat there every day. He replied that his mother brought him there in the mornings telling him to wait there while she looked for a job and a place for them to stay. There is no place else for him to go. When asked what he did all day he simply said that he watched and he waited. He watched the pigeons and the people. He made a game of guessing where each had to go. He said that mostly he just waited for his mother to come at the end of the day so they could wait together until the night shelter opened. (Boxill, 1990, p. 1)

I want to provide an observation example to illustrate what such a descriptive account is like. The observation I’ve selected describes a two-hour session of mothers participating in an early-childhood parent education program. The purpose of the program was to increase the skills, knowledge, and confidence of participants as well as provide a support group for first-time mothers. In funding the program, the legislators emphasized that they did not want parents to be told how to rear their children. Rather, the purpose of the parent education sessions was to increase the options available to parents so that they could make conscious choices about their own parenting styles and increase their confidence about the choices they make. Parents were also to be treated with respect and to be recognized as the primary educators of their children—in other words, the early-childhood educators were not to impose their expertise on parents but, instead, make clear that parents are the real experts regarding their own children.

We made site visits to see the programs in action and observe the parenting discussions. Descriptions of these sessions became the primary data of the evaluation. Our descriptions provided feedback to the legislature about whether their policy guidance was in fact being followed. In essence, in our role as evaluation observers, we became the eyes and ears of the legislature and the state program staff, permitting them to understand what was happening in various parent sessions throughout the state. Descriptive data about the sessions also provided a mirror for the staff who conducted those sessions, a way of looking at what they were doing to see if that was what they wanted to be doing.

Exhibit 1.7 provides a description from one such session. The criterion I invite you to apply in reading this observation is the extent to which sufficient data are provided to take you, the reader, into the setting and permit you to make your own judgment about the nature and quality of parent education being provided.

The Raw Data of Qualitative Inquiry

In Exhibit 1.7, the description of the parenting session is aimed at permitting the reader to understand what occurred in the session. The qualitative data are descriptive. Pure description and quotations are the raw data of qualitative inquiry. Description is meant to take the reader into the setting. The data do not include judgments about whether what occurred was good or bad, appropriate or inappropriate, or any other interpretive judgments. The data simply describe what occurred. State legislators, program staff, parents, and others used this description, and descriptions like this from other program sites, to discuss what they wanted the programs to be and do. The descriptions helped them make explicit their own judgmental criteria. So do anecdotes when systematically collected and analyzed. (See MQP Rumination # 1, page 31.)

Integrating Qualitative Inquiry Methods Thus far, the examples of observation and interviewing in this chapter have been presented as separate and distinct from each other. In practice, they are often fully integrated approaches. Becoming a skilled observer is essential even if you concentrate primarily on interviewing, because every face-to-face interview also involves and requires observation. The skilled interviewer is thus also a skilled observer, able to read nonverbal messages, sensitive to how the interview setting can affect what is said, and carefully attuned to the nuances of the interviewer–interviewee interaction and relationship.

EXHIBIT 1.7 Observation Description Illustrated: A Discussion for Mothers of Two-Year-Olds