U7D1-68 - **READ AND FOLLOW ALL INSTRUCTIONS AS OUTLINED BELOW.. Research Methodology Peer Review.

Politically Sensitive Sampling

This final analytically driven strategy involves a different kind of analysis: political analysis. Evaluation is inherently and inevitably political. A variation on the critical case sampling strategy involves selecting (or sometimes avoiding) a politically sensitive site or unit of analysis. For example, a statewide program may have a local site in the district of a state legislator who is particularly influential. By studying carefully the program in that district, evaluation data may be more likely to attract attention and be used. This does not mean that the evaluator then undertakes to make that site look either good or bad depending on the politics of the moment. That would clearly be unethical. Rather, sampling politically important cases is simply a strategy for trying to increase the usefulness and relevance of information where resources permit the study of only a limited number of cases.

The same political perspective (broadly speaking) may inform case sampling in applied or even basic research studies. A political scientist or historian might select the election year 2000 Florida vote-counting case, the Clinton impeachment effort, Nixon’s Watergate crisis, or Reagan’s Iran- Contra scandal for study, not only because of the insights they provide about the American system of government but also because of the likely attention such a study would attract. A sociologist’s study of a riot or a psychologist’s study of a famous suicide would likely involve some attention during sampling to the public and political importance of the case. Such political calculations may enter into initial case selection, but sometimes, the political importance of cases to include (or avoid) only becomes apparent during fieldwork. Thus, sampling politically important cases, interviews, observation sites, or documents may be an emergent sampling strategy. In any event, the analysis and interpretation of politically sensitive cases will have an explicitly political overlay from beginning to end, and the political attention garnered may or may not work out as intended. For, as political satirist and science fiction author George Orwell observed, “In our age there is no such thing as ‘keeping out of politics.’ All issues are political issues, and politics itself is a mass of lies, evasions, folly, hatred and schizophrenia.”

Summary of Analytically Focused Sampling

Saturation or redundancy sampling could have been included in this group. I chose to categorize it as an emergence-driven strategy, but it shares the defining characteristic that case selection and analysis can and do go on simultaneously in naturalistic, open-ended inquiry, as is true for these four purposeful strategies (numbers correspond to those in the summary in Exhibit 5.8, pp. 271 –272):

35. Confirming and disconfirming cases

36. Illumination and elaboration additions to the original sample

37. Qualitative research synthesis

38. Politically sensitive sampling

MODULE

38 Mixed, Stratified, and Nested Purposeful SamplingStrategies

Stratified or nested samples are samples within samples, a strategy within a strategy. Mixed strategies combine approaches. These combinations can serve to meet multiple inquiry interests and needs. They can deepen and narrow the focus of inquiry, like a funnel that channels the flow of a liquid more precisely, to increase relevance and credibility.

Stratified or Nested Samples

Purposeful samples can be stratified or nested by combining types of purposeful sampling. For example, you might combine typical case sampling with maximum heterogeneity sampling by taking a stratified purposeful sample of above-average, average, and below-average cases. This represents less than a full maximum variation sample but more than simple typical case sampling. The purpose of this nested purposeful sample is to capture major variations rather than to identify a common core, although the latter may also emerge in the analysis. Each of the strata would constitute a fairly homogeneous sample. This strategy differs from stratified random sampling in that the sample sizes are likely to be too small for generalization or statistical representativeness. Exhibit 5.13 illustrates an example of this stratifying or funneling process.

An outlier sample or maximum heterogeneity approach may yield an initial potential sample size that is still larger than the study can handle. The final selection, then, may be made randomly—a combination approach. Thus, purposeful strategies are not mutually exclusive. Each approach serves a somewhat different purpose. Because research and evaluations often serve multiple purposes, more than one qualitative sampling strategy may be necessary. In long-term fieldwork, several sampling strategies may be used at some point.

EXHIBIT 5.13 Example of Nesting Sampling Strategies

1. Begin with a maximum variation sampling based on known variations in the target population of interest, let’s say early-childhood Head Start sites in Minnesota, to document program diversity and analyze core themes.

2. Once in the field, add sites to the sample through snowball sampling, seeking additional diversity.

3. At the site level, identify and recruit key informants to provide in-depth understanding of the important characteristics and attributes of each site.

Mixing Probability and Purposeful Sampling

Mixed-methods designs can include combining probability and purposeful sampling strategies. Indeed, mixed sampling is at the core of mixed methods. The very first issue of the Journal of Mixed Methods Research featured a typology of mixed-methods sampling (Teddlie & Yu, 2007). Here are five examples of such mixed-sampling strategies:

1. Stratified mixed methods: Begin with a statistical distribution to stratify for purposeful sampling; for example, identify outliers, typical cases, or subgroups of interest. Exhibit 5.13 illustrates this combination. In selecting companies to compare using in-depth case studies, Collins (2001a) began with extensive statistical analysis of the financial performance of companies within sectors to select cases that his team could study to explain the difference between “great” and “good” companies.

2. Sequential mixed methods: Select cases from a probability sample for greater in-depth inquiry to illuminate and validate what the numbers mean. A random, representative sample of people with disabilities was surveyed to determine their priority needs. A small number of respondents in different categories of need were then interviewed to understand in depth their situations and priorities and to make the statistical findings more personal through stories. The qualitative sample was also used to validate the accuracy and meaningfulness of the survey responses.

3. Parallel mixed methods: Simultaneously conduct a survey using a probability sample for representativeness and generalizability, and at the same time conduct in-depth case studies purposefully chosen to provide depth of interpretation to elucidate what the survey results mean.

4. Triangulated mixed methods: Compare probability and purposeful samples, studied independently, to triangulate and examine consistency of findings with different methods and sampling strategies. An evaluation of an early-childhood home visitation program used both quantitative (pre–post measures) and qualitative (case studies) methods. The findings provided quite different—and conflicting—perspectives on the program, which supported the need for further inquiry (Sherwood, 2005).

5. Validity-focused mixed methods: Do observations and interviews to determine the validity of select statistical data. For example, have the procedures for gathering statistical data been rigorously followed? In 1851, French political theorist Pierre-Joseph Proudhon observed that “to be governed is to be noted, registered, enumerated, accounted for, stamped, measured, classified audited, patented, licensed, authorized, endorsed, reprimanded, prevented, reformed, rectified, and corrected in every operation, every transaction, every movement” (quoted by Schulz, 2014, p. 34). A hundred and sixty–plus years later, the ways in which we are enumerated, measured, and classified have expanded geometrically. One form of qualitative inquiry is to investigate where the numbers come from, how they are entered, and what they mean.

Consider death certificates in the United States. Death certificates list cause of death and are a critically important source of statistics depicting mortality and disease trends. But studies of how death certificates are completed have revealed substantial errors. Medical residents are often assigned to fill out death certificates in hospitals without adequate (or sometimes any) training in how to do so; they often rely on secondhand or thirdhand reports of cause of death, and they run into a variety of administrative and procedural problems in completing death certificates. A 2010 survey of 521 doctors in 38 residency programs across New York City found that “only a third believed death certificates to be accurate. Nearly half reported knowingly listing an inaccurate cause of death, and that number rose to sixty percent among residents with the most experience” (Schulz, 2014, p. 36). Qualitative fieldwork and interviews are necessary to get at the factors that lead to significant inaccuracies in this important data source. Chapter 4 (pp. 231–232) discussed conducting fieldwork to do such validity checks. I gave the example of entering guesses on a survey of subsistence farmers’ yields when I was working in agricultural extension in Burkina Faso in the 1960s, because we had no way of actually measuring yields; fieldwork on the data-gathering procedures would have revealed the numbers to be highly problematic.

Exhibit 5.11 (pp. 286–287) provides an example of combining probability and purposeful sampling, beginning with a population survey as the basis for outlier sampling (success case method) and then adding diversity criteria (urban–rural mix and size of program). This mixed- sampling approach increased the study’s validity and credibility significantly. Such mixed-methods sampling strategies have become more widely used as researchers and evaluators have gained experience and knowledge about how to combine approaches and as mixed-methods designs have become more common, a trend likely to accelerate with the historic launch in 2014 of the Mixed Methods International Research Association, “a momentous development in mixed methods research” (Mertens, 2014, p. 3). Especially common in mixed-methods sampling is the sequence of (a) a small purposeful sample to explore issues and generate hypotheses, followed by (b) a probability sample to answer questions of representativeness in a population of interest, followed by (c) a new purposeful sample to enhance interpretation of the quantitative (probability sample)

findings; this sequence was depicted in Exhibit 2.3 in Chapter 2 (p. 65). Another common mixed- methods sequence involves probability sampling first, for breadth of coverage, and then purposeful sampling as follow-up to add depth and enhance understanding and interpretation (Kemper, Stringfield, & Teddlie, 2003, pp. 284–285).

Summary of Mixed, Stratified, and Nested Sampling Strategies

Sampling is a means to an end. The end, or purpose, is generating knowledge and deepening understanding. In service of those purposes, we combine strategies and use multiple and mixed methods as much as is possible and appropriate. We’ve concluded this extensive and comprehensive discussion of purposeful sampling strategies with two flexible and integrating approaches (numbers correspond to those in the summary in Exhibit 5.8, p. 272):

39. Combined or stratified purposeful sampling strategies

40. Mixed probability and purposeful samples

MODULE

39 Information-Rich Cases

Sampling decisions are inherently practical. . . . It is in sampling, perhaps more than anywhere else in research, that theory meets the hard realities of time and resources

—Kemper et al. (2003, p. 273)

Exhibit 5.8 on page 266 provided an overview of the 40 purposeful sampling strategies we’ve now discussed. The underlying principle that is common to all these strategies is selecting information- rich cases—cases from which one can learn a great deal about the focus of inquiry and which therefore are worthy of in-depth study.

Maxwell (2012) provides an illuminative example of the considerations that go into identifying an information-rich and accessible sample. A graduate student proposed to study classroom discourse norms in a college department. She could only do in-depth interviews with a small sample of students, so she needed criteria for selecting interviewees. Her dissertation committee recommended that she interview sophomores and seniors to get diverse experiences and perspectives. When she proposed this to the department where the study would occur, faculty in the department told her that

sophomores were too new to the department to fully understand the norms of discourse, while seniors were too deeply involved in their theses and in planning for graduation to be good informants. Juniors turned out to be the choice that would best meet the criteria of having the desired information and being most likely to provide this in interviews. (p. 96)

As this example nicely illustrates, reasons for site selections or individual case sampling need to be thoughtfully deliberated, carefully articulated, and made explicit in the methods report. Credibility concerns will have to be taken into consideration. In the process of developing the research design, the evaluator or researcher is trying to consider and anticipate the kinds of arguments that will lend credibility to the study as well as the kinds of arguments that might be used to attack the findings. Moreover, it is important to be open and clear about a study’s limitations, that is, to anticipate and address criticisms that may be made of a particular sampling strategy, especially by people who think that the only high-quality samples are random ones.

Having weighed the evidence and considered the alternatives, evaluators and primary stakeholders make their sampling decisions, sometimes painfully but always with the recognition that there are no perfect designs. The sampling strategy must be selected to fit the purpose of the study, the resources available, the questions being asked, and the constraints being faced. This holds true for sampling strategy as well as sample size.

MQP Rumination # 5

Convenience Sampling Is Not Purposeful Sampling

I am offering one personal rumination per chapter. These are issues that have persistently engaged, sometimes annoyed, occasionally haunted, and often amused me over more than 40 years of research and evaluation practice. Here’s where I state my case on the issue and make my peace.

Convenience sampling is “defined as a sample in which research participants are selected based on their ease of availability” (Saumure & Given, 2008, p. 124). This means interviewing whoever happens to be at a place during a site visit, stopping people on the street and asking them a few questions, or studying a village because it is near the main road and easy to get to.

In the previous edition of this book, I wrote,

Sampling by convenience: doing what’s fast and convenient. This is probably the most common sampling strategy—and the least desirable. Too often evaluators using qualitative methods think that, because the sample size they can study will be too small to permit generalizations, it doesn’t matter how cases are picked, so they might as well pick ones that are easy to access and inexpensive to study. While convenience and cost are real considerations, they should be the last factors to be taken into account after strategically deliberating on how to get the most information of greatest utility from the limited number of cases to be sampled. Purposeful, strategic sampling can yield crucial information about critical cases. Convenience sampling is neither purposeful nor strategic. (Patton, 2002, pp. 241–242)

Having thus denigrated convenience sampling, or so I thought, I still made the mistake of including it in the summary table of sampling strategies as Item 15, where I wrote, “Do what’s easy to save time, money and effort. Poorest rationale; lowest credibility. Yields information-poor cases” (Patton, 2002, p. 244). Why was this a mistake? I have reviewed a number of research publications, evaluation proposals, and qualitative reports that state in the methods section that they have used “Patton’s purposeful sampling approach # 15.” My bad. Sigh!

So let me emphatically reiterate, convenience sampling is neither strategic nor purposeful. It is lazy and largely useless. And it is omitted from the summary table of purposeful sampling strategies in this book, Exhibit 5.8 (p. 266).

Five Problems With Convenience Sampling

1. Information-poor: Yin (2011) has succinctly stated the problem with convenience sampling: “It is likely to produce an unknown degree of incompleteness because the most readily available sources of data are not likely to be the most informative sources. Similarly, convenience samples are likely to produce an unwanted degree of bias” (p. 88).

2. Dangerous: Maxwell (2012) cautions that to make convenience the primary or sole criterion for sampling decisions is “dangerous” both because it diminishes or ignores the primary sampling criterion of being purposeful about finding the best information and understandings you seek “and because it exposes your conclusions to serious validity threats.” Unfortunately, he goes on to temper this warning by emphasizing that

the realities of access, cost, time, and difficulty necessarily influence every decision about what settings and participants to include in a study, and to dismiss these considerations as “unrigorous”

is to ignore the real conditions that will influence how data can be collected and the ability of these data to answer your research questions. (p. 95)

I would have preferred that he had ended with the conclusion that convenience sampling is “dangerous” and not offered practicality as a loophole and cop-out. The critical distinction, however, is that convenience should be, at best, a secondary or tertiary practical consideration and never the sole or primary criterion.

3. Limited utility: Morse (2010) also gets entangled in the convenience sample quagmire by arguing that convenience sampling allows researchers to go to accessible places where they are likely to see the social interactions that are the focus of their inquiry as a way of getting started. In effect, she is advocating convenience sampling as an open-ended, inductive way of early sampling to begin exploring the phenomenon of interest. Subsequent sampling will be purposefully theoretical to generate and deepen grounded theory. There are two points to be noted. (1) In this framing, convenience sampling does not stand alone as a purposeful strategy, but is simply an easy way to enter the field; inductive theoretical sampling is her real strategy. (2) I would much prefer to designate this approach as exploratory sampling or entry-into-the-field sampling or dipping-your- toe-in-the-inquiry-water sampling. But whatever the alternative labeling, she is not advocating basing the whole inquiry on finding easily accessible cases, which is what convenience sampling connotes.

4. Lazy, not opportunistic: Opportunity sampling is sometimes treated as the same as convenience sampling, and the terms are used interchangeably. Let me distinguish the two. Opportunity sampling is one tactic in a larger strategy of in-depth fieldwork. As I noted in defining and discussing opportunity sampling (p. 300), during fieldwork an opportunity may arise to interview someone or observe an activity, neither of which could have been planned in advance. For example, during an evaluation of an after-school youth drop-in center, a college student who used to participate in the center’s programs dropped by during a semester break to reconnect with former staff members. This offered an opportunity to learn about the history of the center and to see how a former participant viewed her experience. This differs from convenience sampling in that an unanticipated opportunity presents itself and is worth taking advantage of. But it is not the whole sampling strategy or even a major part of it.

5. Low credibility, easy to attack: Finally, those who are looking to attack the credibility of small qualitative samples love to highlight the worthlessness and laziness of convenience samples. Let’s stop giving them ammunition. Stop doing convenience sampling, and stop treating it as a viable purposeful option. As I have shouted in titling this rumination: Convenience sampling is not purposeful sampling. Make your qualitative sampling strategic and purposeful. That’s the criterion of qualitative excellence.

SOURCE: © Chris Lysy—freshspectrum.com

MODULE

40 Sample Size for Qualitative Designs

Qualitative inquiry is rife with ambiguities. There are purposeful strategies instead of methodological rules. There are inquiry approaches instead of statistical formulas. Qualitative inquiry seems to work best for people with a high tolerance for ambiguity. (And we’re still only discussing design. It gets worse when we get to analysis.)

Nowhere is this ambiguity clearer than in the matter of sample size. I get letters. I get calls. I get e-mails.

“Is 10 a large enough sample to achieve maximum variation?”

“I started out to interview 20 people for two hours each, but I’ve lost 2 people. Is 18 large enough, or do I have to find 2 more?”

“I want to study just one organization, but interview 20 people in the organization. Is my sample size 1 or 20 or both?”

My universal, certain, and confident reply to these questions is this: “It depends.”

There are no rules for sample size in qualitative inquiry. Sample size depends on what you want to know, the purpose of the inquiry, what’s at stake, what will be useful, what will have credibility, and what can be done with the available time and resources.

Earlier in this chapter, I discussed the trade-offs between breadth and depth. With the same fixed resources and limited time, a researcher could study a specific set of experiences for a larger number of people (seeking breadth) or a more open range of experiences for a smaller number of people (seeking depth). In-depth information from a small number of people can be very valuable, especially if the cases are information-rich. Less depth from a larger number of people can be especially helpful in exploring a phenomenon and trying to document diversity or understand variation. I repeat, the size of the sample depends on what you want to find out, why you want to find it out, how the findings will be used, and what resources (including time) you have for the study.

Janice Morse (2000), former editor of Qualitative Health Research, has explained insightfully how the nature of the data being collected and the theoretical tradition within which the study is positioned (see Chapter 3) affect sample size. She begins by articulating a principle to guide trade- offs in depth versus breadth:

The quality of the data and the number of interviews per participant determine the amount of useable data obtained. There is an inverse relationship between the amount of useable data obtained from each participant and the number of participants. The greater the amount of useable data obtained from each person (as number of interviews and so forth), the fewer the number of participants.

This principle links the number of participants with the research method used. If, when using semistructured interviews, one obtains a small amount of data per interview question (i.e., relatively shallow data), then to obtain the richness of data required for qualitative analysis, one needs a large number of participants (at least 30 to 60). If, on the other hand, one is doing a phenomenological study and interviewing each person many times, one has a large amount of data for each participant and

therefore needs fewer participants in the study (perhaps only 6 to 10). Grounded theory, with two to three unstructured interviews per person, may need 20 to 30 participants, adjusted according to the factors discussed above. (pp. 4–5)

To understand the credibility problem of small samples in qualitative inquiry, it’s necessary to place these small samples in the context of probability sampling. A qualitative inquiry sample only seems small in comparison with the sample size needed for representativeness when the purpose is generalizing from a sample to the population of which it is a part. Suppose there are 100 people in a program to be evaluated. It would be necessary to randomly sample 80 of those people (80%) to make a generalization at the 95% confidence level. If there are 1,000 people, 278 people must be sampled (28%); and if there are 5,000 people in the population of interest, 357 must be sampled (7%) to achieve a 95% confidence level in the generalization of findings. (See Fitz-Gibbon & Morris, 1987, p. 163, for a table on determining a representative sample size from a given population.)

The logic of purposeful sampling, seeking information-rich cases, is quite different. The problem is, however, that the utility and credibility of small purposeful samples are often judged on the basis of the logic, purpose, and recommended sample sizes of probability sampling. Instead, purposeful samples should be judged according to the purpose and rationale of the study: Does the sampling strategy support the study’s purpose? “Determining your final sample size is a matter of intellectual judgment based on the logic of making meaningful comparisons, developing and testing your explanations” (Mason, 2010, p. 139).

SIDEBAR

TO SAMPLE OR TO SELECT CASES

(With apologies to William Shakespeare’s Hamlet)

To sample or not to sample, but to select cases, that is the question:

Whether ’tis Nobler in the mind to suffer

The Slings and Arrows of outraged statisticians,

For whom sampling is only and ever will be

Random sampling to generalize to a Population,

Or to take Arms against the Purists,

And by opposing them: to sample purposefully.

Yea, to Dare use the word sample.

Or, to sample no more and by a phrase,

To say we end debate, and talk only of

Case selection, purposeful case selection,

Yea, selecting cases for a specified purpose,

But not to sample, foreswearing that word and the debates

That ensue, The Heart-ache, and the thousand Natural shocks

That Flesh is heir to? ‘Tis a resolution

Devoutly to be wished. To sample, to select cases,

To get on with data collection; Aye, there’s the rub,

For in whatever inquiry may come,

When we have shuffled off this language contention,

Must give us pause, and focus: There’s the respect,

For the people interviewed and observed,

Not subjects, nor cases, nor a sample,

But people with stories and lives,

That makes Calamity of so long life:

For who would bear the Whips and Scorns of time,

The Oppressor’s wrong, the proud man’s Contumely,

The pangs of ineffective programs, the Policy implementation’s delay,

The insolence of numbers only, devoid of storied life.

So call it a sample, or case selection,

It matters not to those whose worlds we seek to enter,

But make it purposeful, whatever it be called.

Serve that specified purpose with intention and forethought,

Fulfill that purpose with rigor and resolve,

To grunt and sweat under a worthy quest,

To discover what is not known,

To understand with depth and illumination,

The undiscovered Country, that Puzzles the will,

And makes us rather bear those ills we have,

For enterprises of great pitch and moment,

With this regard our debates turn awry,

And lose the name of Truth in lieu of Action.

Soft you now, into the fray of inquiry,

And name the thing you do, Sampling or case selection,

And make this choice the least of thy sins remembered.

The sample size, like all other aspects of qualitative inquiry, must be judged in context—the same principle that undergirds analysis and presentation of qualitative data. Random probability samples cannot accomplish what in-depth, purposeful samples accomplish, and vice versa.

Here is wise advice on sample size from eminent case study methodologist Robert Stake (2006):

The benefits of multicase study will be limited if fewer than, say, 4 cases are chosen, or more than 10. Two or three cases do not show enough of the interactivity between programs and their situations, whereas 15 or 30 cases provide more uniqueness of interactivity than the research team and readers can come to understand. But for good reason, many multicase studies have fewer than 4 or more than 15 cases. (p. 22)

Small samples that are truly in-depth have provided many of the most important breakthroughs in our understanding of the phenomenon under study. Piaget contributed a major breakthrough to our understanding of how children think by observing his own two children at length and in great depth. Freud established the field of psychoanalysis based originally on fewer than 10 client cases. Bandler and Grinder (1975) founded neurolinguistic programming by studying three renowned therapists: Milton Erickson, Fritz Perls, and Virginia Satir. Peters and Waterman (1982) formulated their widely followed eight principles for organizational excellence by studying 62 companies, a relatively small sample from the thousands of companies one might study. Sands (2000) did a fine dissertation studying a single school principal, describing the leadership of a female leader who entered a challenging school situation and brought about constructive change.

Clair Claiborne Park’s (2001) single-case study of her daughter’s autism reports 40 years of data on every stage of her development, language use, emotions, capacities, barriers, obsessions, communication patterns, emergent artistry, and challenges overcome and not overcome. Park and her husband made systematic observations throughout the years. Eminent medical anthropologist Oliver Sacks reviewed the data and determined in his preface to the book that more data are available on the woman in this extraordinary case study than on any other autistic human being who has ever lived. Here, then, is the epitome of n = 1 in-depth inquiry.

The validity, meaningfulness, and insights generated from qualitative inquiry have more to do with the information richness of the cases selected and the observational/analytical capabilities of the researcher than with sample size.

Large Qualitative Samples

Advances in qualitative software for data management and analysis have made larger sample sizes much more manageable and common. Increasingly, large-scale qualitative studies include samples of 60 to 100 (Mason, 2010). In Far From the Tree, Andrew Solomon (2012) reports interviewing more than 300 families in his inquiry into family experiences of deafness, dwarfism, autism, schizophrenia, disability, prodigies, transgender, crime, and children born of rape. Strong (1979) studied 1,120 pediatric consultations. GlobalGiving help nonprofits in 144 countries raise funds and

communicate their impacts. In an effort to allow people in local communities to tell their own stories, they created a process for collecting perspectives at the grassroots level from program staff and participants. In four years, they collected more than 57,000 stories for analysis and have used them for a variety of purposes, from needs assessment to program evaluation and trend analysis (Maxson, 2014). Such large samples capture vignettes and anecdotes, so they are not in-depth case studies, but they can and have been used to detect overall patterns in recipients’ experiences of development assistance.

Sample Size as Emergent and Flexible

As noted in the earlier sections on emergent sampling and quota sampling, the actual size of a sample can be flexible. In the beginning, when the initial design is formulated (and approved by others, e.g., funders or an IRB, if necessary), a desired or targeted sample size may be specified. That sample size can be a starting point or minimum, but it may not be the final number. The size and composition of the sample can be adjusted based on what is learned as fieldwork is conducted and the inquiry deepens. The emergent nature of qualitative inquiry applies especially powerfully to sample size. The sample can grow, or if saturation is achieved sooner than expected, the size can be reduced.

The final sample size might also involve a trade-off between greater depth versus more breadth. Suppose that the original design, given time and resources, called for conducting 20 two-hour interviews. Once the interviews began, however, it emerged that to do justice to the inquiry, interviews were taking four hours instead of two. This could mean reducing the sample size from the original 20 to 5 or 6. Or the opposite scenario might occur. The interviewees only needed 45 minutes to tell their stories instead of the planned two hours. This could mean increasing the sample size from 20 to 30.

Thus, the challenge of determining sample size becomes even more complicated when emergent strategies are used, like snowball sampling, RDS, and sampling to the point of redundancy. These purposeful strategies leave the question of sample size open, a prime example of the emergent nature of qualitative inquiry. There remains, however, the practical problem of how to negotiate an evaluation budget or get a dissertation committee to approve a design if you don’t have some idea of sample size. Sampling to the point of redundancy is an ideal, one that works best for basic research with unlimited timelines and unconstrained resources.

Bottom line: The strategic principle of design emergence rules (see Chapter 2, Exhibit 2.1, p. 46).

This issue of sample size is a lot like the problem students have when they are assigned an essay to write.

Student: How long does the paper have to be?

Instructor: Long enough to cover the assignment.

Student: But how many pages?

Instructor: Enough pages to do justice to the subject—no more, no less.

SIDEBAR

PURPOSEFUL SAMPLING SIZE IN DISSERTATIONS

Mason (2010) identified and analyzed 560 qualitative dissertations. The smallest sample was a single participant used in a life history study. The largest sample was 95. The median size was 28 (and mean was 31). However, the most common sample sizes he found were 20 and 30 (followed by 40, 10, and 25).

The significantly high proportion of studies utilizing multiples of 10 as their sample is the most important finding from this analysis. There is no logical (or theory driven) reason why samples ending in any one integer would be any more prevalent than any other in qualitative PhD studies using interviews. If saturation is the guiding principle of qualitative studies, it is likely to be achieved at any point, and is certainly no more likely to be achieved with a sample ending in a zero than with any other number.

He interpreted the finding that the most common sample size was a multiple of 10 as constituting a “pre-meditated approach,” in which the sample size is fixed in advance rather than emergent or saturation determined.

Practical Purposeful Sampling

The solution to determining a purposeful sample size is judgment and negotiation. I recommend that qualitative sampling designs specify minimum samples based on expected reasonable coverage of the phenomenon given the purpose of the study and stakeholder interests. You may add to the sample as fieldwork unfolds. You may change the sample if information emerges that indicates the value of a change. The design should be understood to be flexible and emergent. Yet, at the beginning, for planning and budgetary purposes, one specifies a minimum expected sample size and builds a rationale for that minimum, as well as criteria that would alert the researcher to inadequacies in the original sampling approach and/or size.

In the end, sample size adequacy, like all aspects of research, is subject to peer review, consensual validation, and judgment. What is crucial is that the sampling procedures and decisions be fully described, explained, and justified, so that information users and peer reviewers have the appropriate context for judging the sample. The researcher or evaluator is obligated to discuss how the sample affected the findings, the strengths and weaknesses of the sampling procedures, and any other design decisions that are relevant for interpreting and understanding the reported results. Exercising care not to overgeneralize from purposeful samples, while maximizing to the full the advantages of in-depth, purposeful sampling, will do much to alleviate concerns about small sample size.

Protection of Human Subjects, Sampling Issues, and Emergent Designs

Committees for the protection of human subjects (commonly known as IRBs in the United States) may not include members with expertise in qualitative methods. With or without such expertise, conflicts can arise between qualitative ideals and interpretation of procedures for protecting participants in research. For example, I was recently contacted about how to respond to a

committee that rejected purposeful sampling and insisted on random sampling, even for a small sample, to comply with the Belmont Principle of Justice that every individual in a program have equal access to participate in the study. The committee required that each participating agency collect the names of everyone who met the study criteria and then draw people at random so as to be fair.

The design solution involved a combination of purposeful and random sampling. First, agency records were used to establish a possible maximum variation pool of potential participants.

Those meeting the criteria were contacted and asked if they were interested in and willing to participate in an interview (with a $100 compensation) laying out the nature of the study. All those meeting the purposeful sampling criteria who responded positively within two weeks were considered in the potential pool. The target number for the study was 20. The first issue, then, was whether 20 were eligible and interested. At that stage, it would be possible to assess whether random sampling was the way to go or whether additional recruitment would be necessary to meet the target.

Let’s say 30 respond positively. You can then sample randomly, with stratification to ensure diversity, to get to 20. Then, you sample both randomly and purposefully (stratification) to get the final 12. If anyone doesn’t complete the interview among the sampled 12, you go to the next person on the random stratified list. This procedure meets the Belmont Justice standard of fairness because it permits any potential participants in the program meeting the minimum criteria to be considered and placed in the potential research pool.

Emergent sampling designs also pose special problems for IRBs. Such boards typically want to know, in advance of fieldwork, who will be interviewed and the precise questions that will be asked. If the topic is fairly innocuous and the general line of questioning relatively unobtrusive, an IRB may be willing to approve the framework of an emergent design with sample questions included but without full sample specification and a formal interview instrument.

Another approach is to ask for approval in stages. This means initially asking for approval for the general framework of the inquiry and specifically for the first exploratory stage of fieldwork, including procedures for assuring confidentiality and informed consent, and then returning periodically (e.g., quarterly or annually) to update the design and its approval. This is cumbersome for both the researcher and the IRB, but it is a way of meeting IRB mandates and still implementing an emergent design. This staged-approval approach can also be used when the evaluator is developing the design jointly with program staff and/or participants and therefore cannot specify the full design at the beginning of the participatory process.

These are just a couple of examples of the challenges involved in ensuring that qualitative designs meet ethical standards while maintaining as much as possible the creative and emergent elements of open, naturalistic inquiry.

How NOT to sample

SOURCE: © Chris Lysy—freshspectrum.com

MODULE

41 Mixed-Methods Designs

A study may employ more than one sampling strategy. It may also include multiple types of data. The chapters on interviewing, observation, and analysis will include information that will help in making design decisions. Before turning to those chapters, however, I want to discuss briefly the value of using multiple methods in research and evaluation.

Triangulation

The method must follow the question. Campbell, many decades ago, promoted the concept of triangulation—that every method has its limitations, and multiple methods are usually needed.

(Gene V. Glass eulogizing pioneering methodologist Donald T. Campbell, quoted in Tashakkori & Teddlie, 1998, p. 22)

Triangulation strengthens a study by combining methods. This can mean using several kinds of methods or data, including using both quantitative and qualitative approaches. Denzin (1978b) has identified four basic types of triangulation: (1) data triangulation—the use of a variety of data sources in a study, (2) investigator triangulation—the use of several different researchers or evaluators, (3) theory triangulation—the use of multiple perspectives to interpret a single set of data, and (4) methodological triangulation—the use of multiple methods to study a single problem or program.

The term triangulation is taken from land surveying. Knowing a single landmark only locates you somewhere along a line in a direction from the landmark, whereas with two landmarks (and your own position being the third point of the triangle), you can take bearings in two directions and locate yourself at their intersection. Triangulation also works metaphorically to call to mind the world’s strongest geometric shape—the triangle (e.g., the form used to construct geodesic domes). The logic of triangulation is based on the premise that

no single method ever adequately solves the problem of rival causal factors. Because each method reveals different aspects of empirical reality, multiple methods of observations must be employed. This is termed triangulation. I now offer as a final methodological rule the principle that multiple methods should be used in every investigation. (Denzin, 1978b, p. 28)

Triangulation is ideal. It can also be expensive. A study’s limited budget and time frame will affect the amount of triangulation that is practical, as will political constraints (stakeholder values) in an evaluation. Certainly, one important strategy for inquiry is to employ multiple methods, measures, researchers, and perspectives—but to do so reasonably and practically.

Most good researchers prefer addressing their research questions with any methodological tool available, using the pragmatist credo of “what works.” For most researchers committed to the thorough study of a research problem, method is secondary to the research question itself, and the underlying worldview hardy enters the picture, except in the most abstract sense. (Tashakkori & Teddlie, 1998, p. 22)

A rich variety of methodological combinations can be employed to illuminate an inquiry question. Some studies mix interviewing, observation, and document analysis. Others rely more on interviews than observations, and vice versa. Studies that use only one method are more vulnerable to errors linked to that particular method (e.g., loaded interview questions, biased or untrue responses), unlike studies that use multiple methods, in which different types of data provide cross- data validity checks. Using multiple methods allows inquiry into a research question with “an arsenal of methods that have non-overlapping weaknesses in addition to their complementary strengths” (Brewer & Hunter, 1989, p. 17). Mixed methods strengthen the credibility of evidence in evaluation (Mertens & Hesse-Biber, 2013).

©2002 Michael Quinn Patton and Michael Cochran

However, a common misunderstanding about triangulation is that the point is to demonstrate that different data sources or inquiry approaches yield essentially the same result. But the point is really to test for such consistency. Different kinds of data may yield somewhat different results because different types of inquiry are sensitive to different real-world nuances. Thus, understanding the inconsistencies in findings across different kinds of data can be illuminative. Finding such inconsistencies ought not be viewed as weakening the credibility of results but, rather, as offering opportunities for deeper insight into the relationship between inquiry approach and the phenomenon under study.

Triangulation within a qualitative inquiry strategy can be attained by combining both interviewing and observations, mixing different types of purposeful samples (e.g., both intensity and opportunity sampling), or examining how competing theoretical perspectives inform a particular analysis (e.g., the transcendental phenomenology of Husserl versus the hermeneutic phenomenology of Heidegger).

A study can also be designed to cut across inquiry approaches and achieve triangulation by combining qualitative and quantitative methods. Doing so well, in a truly integrated manner, involves, indeed may require, a “mixed methods way of thinking” (Greene, 2007; see sidebar). Mixed methods have become particularly important, even preferred, in program evaluation, including impact evaluation (Bamberger, 2013; Rogers, 2012).

Mixing Data, Design, and Analysis Approaches

Borrowing and combining distinct elements from pure or coherent methodological strategies can generate creative mixed inquiry strategies that illustrate variations on the theme of triangulation. We begin by distinguishing measurement, design, and analysis components of the hypothetico- deductive (quantitative/experimental) and holistic-inductive (qualitative/naturalistic) paradigms. The ideal-typical qualitative methods strategy consists of three core elements: (1) qualitative data, (2) a holistic-inductive design of naturalistic inquiry, and (3) content or case analysis. In the traditional hypothetico-deductive approach to research, the ideal study would include (a) quantitative data from (b) experimental (or quasi-experimental) designs and (c) statistical analysis.

SIDEBAR

MIXED-METHODS WAY OF THINKING

The core meaning of mixed methods social inquiry is to invite multiple mental models into the same inquiry space for purposes of respectful conversation, dialogue, and learning one from the other, toward a collective generation of better understanding of the phenomena being studied. By definition, then, mixed methods social inquiry involves a plurality of philosophical paradigms, theoretical assumptions, methodological traditions, data gathering and analysis techniques, and personalized understandings and value commitments—because these are the stuff of mental models. . . .

A mixed methods way of thinking involves an openness to multiple ways of seeing and hearing, multiple ways of making sense of the social world, and multiple standpoints on what is important and to be valued and cherished. A mixed methods way of thinking rests on assumptions that there are multiple legitimate approaches to social inquiry, that any given approach to social inquiry is inevitably partial, and that thereby multiple approaches can generate more complete and meaningful understanding of complex human phenomena. A mixed methods way of thinking means genuine acceptance of other ways of seeing and knowing as legitimate. A mixed methods way of thinking involves an active engagement with difference and diversity.

—Jennifer C. Greene (2007, p. xii)

Measurement, design, and analysis alternatives can be mixed to create eclectic designs, like customizing an architectural plan to tastefully integrate modern, postmodern, and traditional elements or preparing an elegant dinner with a French appetizer, a Chinese entrée, and an American dessert—not to everyone’s taste, to be sure, but the possibilities are endless. At least, that’s the concept. To make the idea of mixed elements more concrete and to illustrate the creative possibilities that can emerge out of a flexible approach to research, it will be helpful to examine alternative design possibilities for a single program evaluation. The examples that follow have been constructed under the artificial constraint that only one kind of measurement, design, and analysis could be used in each case. In practice, of course, the possible mixes are much more varied, because any given study could include several measurement approaches, varying design approaches, and different analytical approaches to achieve triangulation.

The Case of Operation Reach-Out: Variations in Program Evaluation Design

Let’s consider design alternatives for a comprehensive program aimed at high school students at high risk educationally (poor grades, poor attendance, poor attitudes toward school), with highly vulnerable health (poor nutrition, sedentary lifestyle, high drug use), and who are likely candidates for delinquency (alienation from dominant societal values, running with a “bad” crowd, angry). The program consists of experiential education internships through which these high-risk students get individual tutoring in basic skills, part-time job placements that permit them to earn income while gaining work exposure, and an opportunity to participate in peer-group discussions aimed at changing health values, establishing a positive peer culture, and increasing social integration. Several evaluation approaches are possible.

Pure Hypothetical-Deductive Approach to Evaluation: Experimental Design, Quantitative Data, and Statistical Analysis

The program does not have sufficient resources to serve all targeted youth in the population. A pool of eligible youth is established, with admission into the program on a random basis and the remaining group receiving no immediate treatment intervention. Before the program begins and one year later, all the youth, both those in the program and those in the control group, are administered standardized instruments measuring school achievement, self-esteem, anomie, alienation, and locus of control. Rates of school attendance, illness, drug use, and delinquency are obtained for each group. When all data have been collected by the end of the year, comparisons between the control and experimental groups are made using inferential statistics.

Pure Qualitative Strategy: Naturalistic Inquiry, Qualitative Data, and Content Analysis

Procedures for recruiting and selecting participants for the program are determined entirely by the staff. The evaluator finds a convenient time to conduct an in-depth interview with new participants as soon as they are admitted into the program, asking students to describe what school is like for them, what they do in school, how they typically spend their time, what their family life is like, how they approach academic tasks, their views about health, and their behaviors and attitudes with regard to delinquent and criminal activity. In brief, participants are asked to describe themselves and their social world. The evaluator observes the program activities, collecting detailed descriptive data about staff–participant interactions and conversations, staff intervention efforts, and youth reactions. The evaluator finds opportunities for additional in-depth interviews with the participants to find out how they view the program, what kinds of experiences they are having, and what they’re doing. Near the end of the program, in-depth interviews are conducted with the participants to learn what behaviors have changed, how they view things, and what their expectations are for the future. Interviews are also conducted with program staff and some parents. These data are content analyzed to identify the patterns of experiences participants bring to the program, what patterns characterize their participation in the program, and what patterns of change are reported by and observed in the participants.

Mixed Form: Experimental Design, Qualitative Data, and Content Analysis

As in the pure experimental design, potential participants are randomly assigned to treatment and control groups. In-depth interviews are conducted with all the youth, both those in the treatment group and those in the control group, both before the program begins and again at the end of the program. Content and thematic analyses are performed so that the control and experimental group patterns can be compared and contrasted.

Mixed Methods: Experimental Design, Qualitative Data, and Statistical Analysis

Participants are randomly assigned to treatment and control groups, and in-depth interviews are conducted both before the program and at the end. These interview data, in raw form, are then given to a panel of judges, who rate each interview along several outcome dimensions operationalized as a 10-point scale. For both the “pre” interview and the “post” interview, the judges assign ratings on dimensions such as likelihood of success in school (low = 1, high = 10), likelihood of committing criminal offenses (low = 1, high = 10), commitment to education, commitment to engaging in productive work, self-esteem, and manifestation of desired nutritional and health habits. Inferential statistics are then used to compare these two groups. Judges make the ratings without knowledge of which participants were in which group. Outcomes on the rated scales are also statistically related to background characteristics of the participants.

Mixed Methods: Naturalistic Inquiry, Qualitative Data, and Statistical Analysis

As in the pure qualitative form, students are selected for the program on the basis of whatever criteria staff members choose to apply. In-depth interviews are conducted with all students before and at the end of the program. These data are then submitted to a panel of judges, who rate them on a series of dimensions similar to those listed in the previous example. Change scores are computed for each individual, and changes are statistically related to background characteristics of the students to determine in a regression format which characteristics are likely to predict success in the program. In addition, observations of program activities are rated on a set of scales developed to quantify the climate attributes of activities: for example, the extent to which the activity involved active or passive participation, the extent to which student–teacher interaction was high or low, the extent to which interactions were formal or informal, and the extent to which participants had input into program activities. Quantitative ratings of activities based on qualitative descriptions are then aggregated to provide an overview of the treatment environment of the program.

Mixed Methods: Naturalistic Inquiry, Quantitative Data, and Statistical Analysis

Students are selected for the program according to staff criteria. The evaluator enters the program setting without any predetermined categories of analysis or presuppositions about important variables or variable relationships. The evaluator observes important activities and events in the program, looking for the types of behaviors and interactions that will emerge. For each significant type of behavior or interaction observed, the evaluator creates a category and then uses a time and

space sampling design to count the frequency with which those categories of behavior and interaction are exhibited. The frequency of the manifestation of the observed behaviors and interactions is then statistically related to characteristics such as group size, duration of the activity, staff–student ratios, and social/physical density.

Pure and Mixed Strategies

Exhibit 5.14 summarizes the six alternative design scenarios we’ve just reviewed for evaluation of “Operation Reach-Out.” As these alternative designs illustrate, purity of approach is only one option. Inquiry strategies, measurement approaches, and analysis procedures can be mixed and matched in the search for relevant and useful information. That said, it is worth considering the case for maintaining the integrity and purity of qualitative and quantitative paradigms. The 12 themes of qualitative inquiry described in the second chapter (Exhibit 2.1) do fit together as a coherent strategy. The openness and personal involvement of naturalistic inquiry mesh well with the openness and depth of qualitative data. Genuine openness flows naturally from an inductive approach to analysis, particularly an analysis grounded in the immediacy of direct fieldwork and sensitized to the desirability of holistic understanding of unique human settings.

Likewise, there is an internal consistency and logic to experimental designs that test deductive hypotheses derived from theoretical premises. These premises identify the key variables to consider in testing theory or measuring, controlling, and analyzing hypothesized relationships between program treatments and outcomes. The rules and procedures of the quantitative/experimental paradigm are aimed at producing internally valid, reliable, replicable, and generalizable findings.

Guba and Lincoln (1988) have argued that the internal consistency and logic of each approach, or paradigm, mitigates against methodological mixing of different inquiry modes and data collection strategies. Their cautions are not to be dismissed lightly. Mixing parts of different approaches is a matter of philosophical and methodological controversy. Yet the practical mandate in evaluation (Patton, 1981, 2012a) to gather the most relevant possible information for evaluation users outweighs concerns about methodological purity based on epistemological and philosophical arguments. The intellectual mandate to be open to what the world has to offer surely includes methodological openness. In practice, it is altogether possible, as we have seen, to combine approaches and to do so creatively (Patton, 1987); just as machines that were originally created for separate functions like printing, faxing, scanning, and copying have now been combined into a single integrated technological unit, so too methods that were originally created as distinct, stand- alone approaches can now be combined into more sophisticated and multifunctional designs.

Advocates of methodological purity argue that a single evaluator cannot be both deductive and inductive at the same time, or cannot be testing predetermined hypotheses and still remain open to whatever emerges from open-ended, phenomenological observation. Yet, in practice, human reasoning is sufficiently complex and flexible that it is possible to research predetermined questions and test hypotheses about certain aspects of a program while being quite open and naturalistic in pursuing other aspects of the program. In principle, this is not greatly different from a questionnaire that includes both fixed-choice and open-ended questions. The extent to which a qualitative approach is inductive or deductive varies along a continuum. As evaluation fieldwork begins, the evaluator may be open to whatever emerges from the data—a discovery or inductive approach. Then, as the inquiry reveals patterns and major dimensions of interest, the evaluator will begin to focus on verifying and elucidating what appears to be emerging—a more deductively oriented approach to data collection and analysis.

EXHIBIT 5.14 Data Collection, Design, and Analysis Combinations: Pure and Mixed Design Strategies

The extent to which a study is naturalistic in design is also a matter of degree. This applies particularly to the extent to which the investigator places conceptual constraints on or makes presuppositions about the program or phenomenon under study. In practice, the naturalistic approach may often involve moving back and forth from inductive, open-ended encounters to more hypothetical-deductive attempts to verify hypotheses or solidify ideas that emerged from those more open-ended experiences, sometimes even manipulating something to see what happens.

©2002 Michael Quinn Patton and Michael Cochran

Sophisticated Emergent Design Strategy

These examples of variations in qualitative approaches are somewhat like the differences between experimental and quasi-experimental designs. Pure experiments are the ideal; quasi- experimental designs often represent what is possible and practical. Likewise, full participant observation over an extended period of time is the qualitative ideal. In practice, many acceptable and meaningful variations to qualitative inquiry can be designed.

This spirit of adaptability and creativity in designing studies is aimed at being pragmatic and responsive to real-world conditions and, when doing evaluations, at meeting stakeholder information needs. Mixed methods and strategies allow creative research adaptations to particular settings and questions, though certain designs pose constraints that exclude other possibilities. It is not possible, for example, to operate a program as an experiment by assigning participants to treatment and control groups while at the same time operating the program under naturalistic inquiry conditions, in which all eligible participants enter the program (and thus there is no control group and no random assignment). Another incompatibility: Qualitative descriptions can be converted into quantitative scales for purposes of statistical analysis, but it is not possible to work the other way around and convert purely quantitative measures into detailed, qualitative descriptions.

MODULE

42 Qualitative Design Chapter Summary and Conclusion: Methods Choices and Decisions

Every path we take leads to fantasies about the paths not taken. —Halcolm

This chapter opened with a case study example (the story of Daniel in Babylon) to set the context for qualitative design and methods decision making. I then presented a typology of different research purposes (basic research, applied research, summative evaluation, formative evaluation, and action research) to emphasize that, just as form follows function in architecture, design follows purpose in research and evaluation. Next came seven points of guidance for framing qualitative inquiry questions. After dealing with formulation of questions, we turned to design and data collection options, emphasizing critical trade-offs between depth and breadth. This led to consideration of different perspectives on what constitutes a case study, or even a case, and a variety of possible units of analysis. That set the stage for an extended presentation of purposeful sampling (case selection) strategies in the search for information-rich cases, and discussion of sample size. Finally, we examined a variety of mixed-methods designs based on different combinations of design, measurement, and analysis options. Exhibit 5.15 summarizes the issues discussed in this chapter that must be addressed in designing a study, which can also serve as an outline for a qualitative inquiry design proposal.

Which research design is best? Which strategy will provide the most useful information to decision makers? No simple and universal answer to these questions is possible. The answer in each case will depend on the purpose of the study; the scholarly or evaluation audience for the study (what intended users want to know); the funds available; the political, organizational, and cultural context; and the interests/abilities/perspectives of the researchers.

The word “design” is both a noun and a verb, denoting both the product and the process of our planning. Issues of context and culture are especially important considerations in both the product and process of designing an evaluation.

—Nick Smith (2013) Evaluation scholar

In qualitative inquiry, the problem of design poses a paradox. The term design suggests a very specific blueprint, but “design in the naturalistic sense . . . means planning for certain broad contingencies without, however, indicating exactly what will be done in relation to each” (Lincoln & Guba, 1985, p. 226). Ideally, a qualitative design can remain sufficiently open and flexible to pursue whatever turns up during early interviewing and fieldwork. The degree of flexibility and openness is, however, a matter of considerable variation among designs.

What is certain is that different methods can produce quite different findings. The challenge is to figure out which design and methods are most appropriate, productive, and useful in a given

situation. It’s worth distinguishing arguments about which methods are most appropriate versus arguments about the intrinsic and universal superiority of one method over another.

Every cobbler thinks leather is the only thing. Most social scientists, including the present writer, have their favorite research methods with which they are familiar and have some skill in using. And I suspect we mostly choose to investigate problems that seem vulnerable to attack through these methods. But we should at least try to be less parochial than cobblers. Let us be done with the arguments of participant observation versus interviewing—as we have largely dispensed with the arguments for psychology versus sociology—and get on with the business of attacking our problems with the widest array of conceptual and methodological tools that we possess and they demand. This does not preclude discussion and debate regarding the relative usefulness of different methods for the study of specific problems or types of problems. But that is very different from the assertion of the general and inherent superiority of one method over another on the basis of some intrinsic qualities it presumably possesses. (Trow, 1970, p. 149)

Two overarching themes are at the core of this chapter and, indeed, this book. First, there are many design and data collection options—and varying perspectives about the relative strengths and weaknesses of each, though each has distinct strengths and weaknesses. Second, the variety of approaches gives you an opportunity—and responsibility—to design your study in support of your inquiry questions and within the context of your own field. In so doing, and in wrestling with methodological options and competing perspectives, it is worth keeping in mind the admonition of Nobel Prize–winning physicist Percy Bridgman: “There is no scientific method as such, but the vital feature of the scientist’s procedures has been merely to do his utmost with his mind, no holds barred” (quoted in Waller, 2004, p. 106).

EXHIBIT 5.15 Outline for a Qualitative Inquiry Design Proposal

But it all begins with what you choose to study. How do you decide that? Halcolm provides an answer in the cartoon that closes this chapter.

APPLICATION EXERCISES

1. Distinguishing research purposes: (a) Using Exhibits 5.1 and 5.2, identify an area of inquiry of interest to you. Generate inquiry questions to match the different kinds of research (as in Exhibit 5.2). (b) Discuss the connection between the five different purposes of inquiry (Exhibit

5.1) and purposeful sampling. How are these purposes connected conceptually and methodologically?

2. Depth versus breadth trade-offs: This chapter includes an extensive discussion of design trade- offs. Select a topic of interest to you, and illustrate the trade-offs between depth and breadth that you might face in designing a qualitative inquiry on that topic. Be as concrete and specific as possible about what a broad design would entail versus what a deep inquiry design would involve, with the same amount of money to support either one. Use Exhibit 5.5 to guide your analysis.

3. Purposeful sampling: Select a topic of interest to you. Among the 40 purposeful sampling strategies in Exhibit 5.8, choose 3 different strategies (in three different categories, A–H), and describe how you would use that purposeful sampling strategy to undertake your inquiry.

4. Mixed-methods inquiry: In your field of interest, find two studies that describe their approach as “mixed methods” or “multiple methods.” Describe the studies, comparing and contrasting them. What did each mean by mixed methods? What was the nature of the “mix”? What are the strengths and weaknesses of the different kinds of data included in the mix?