Final 6078-A1 - Scientific Merit Evaluation Paper

Purposeful Sampling in Qualitative Research Synthesis Harsh Suri THE UNIVERSITY OF MELBOURNE

ABSTRACT

Informed decisions about sampling are critical to improving the quality of research synthesis. Even though several qualitative research synthesists have recommended purposeful sampling for synthes- izing qualitative research, the published literature holds sparse discussion on how different strategies for purposeful sampling may be applied to a research synthesis. In primary research, Patton is fre- quently cited as an authority on the topic of purposeful sampling. In Patton’s original texts that are referred to in this article, Patton does not make any suggestion of using purposeful sampling for re- search synthesis. This article makes a unique contribution to the literature by examining the adapt- ability of each of Patton’s 16 purposeful sampling strategies to the process of qualitative research synthesis. It illuminates how different purposeful sampling strategies might be particularly suited to constructing multi-perspectival, emancipatory, participatory and deconstructive interpretations of published research.

Keywords: Purposeful sampling, qualitative research synthesis, meta-synthesis, meta-study, qualit- ative meta-analysis.

Research synthesis is a special type of research review that is not only descriptive, informative

and evaluative, but also connective (Mays, Pope, and Popay, 2005). ‘Synthesis refers to

making a whole into something more than the parts alone imply’ (Noblit & Hare, 1988,

p. 28). The purpose of research synthesis is to produce new knowledge by making explicit

connections and tensions between individual study reports that were not visible before. It

involves purposeful selection, review, analysis and synthesis of primary research reports on

a similar topic. In a rigorous synthesis, readers are provided with sufficient information

about the synthesis process so that they can make informed decisions about the extent to

which the synthesized findings may be adapted to their own contexts.

With the growth of research activity in recent years, each topic tends to be examined by

different researchers in diverse contexts, employing a wide range of methods, invariably

resulting in disparate findings on the same topic. Making useable sense of such complex

bodies of research can be an overwhelming experience for most stakeholders. These stake-

holders include policymakers, administrators, educators, health professionals, funding

agencies, researchers, students, patients, various advocacy groups and the wider community.

Research syntheses can play an important role in disseminating research knowledge and in

shaping further research, practice and public perception. Hence, issues of ethical represent-

ations (Suri, 2008) and methodological rigour in research syntheses are as crucial as they are

in primary research (Petticrew & Roberts, 2006).

Much of the growing body of literature on research synthesis methods has been dominated

by quantitative researchers. In the last two decades, however, an increasing number of re-

searchers from education and healthcare have recognised the importance of synthesizing

qualitative research. These scholars acknowledge that the emphasis of qualitative research

on particularities and complexities of individual contexts is at odds with any synthetic effort.

Yet they assert that some form of synthesis is essential to enhance the practical value of

© RMIT Publishing, http://www.informit.com.au/qrj.html

Su ri,

H ar

sh , 2

01 1,

'P ur

po se

fu l S

am pl

in g

in Q

ua lit

at iv

e Re

se ar

ch S

yn th

es is'

, Q ua

lit at

iv e R

es ea

rc h

Jo ur

na l,

vo l.

11 , n

o. 2

, p p.

6 3-

75 . D

OI 1

0. 33

16 /Q

RJ 11

02 06

3. T

hi s i

s a p

ee r-

re vi

ew ed

a rti

cle .

qualitative research in policymaking and informing practice at a broader level (Suri & Clarke,

2009). Stressing that any effort of synthesizing qualitative research should be essentially in-

terpretive rather than aggregative, they have proposed interpretive methods of research syn-

thesis under various names, such as meta-ethnography (Noblit & Hare, 1988), cross-case

analysis (Miles & Huberman, 1994), meta-analysis of qualitative research (Jensen & Allen,

1994), qualitative meta-synthesis (Sandelowski, Docherty, and Emden, 1997; Zimmer,

2006), qualitative systematic review (Booth, 2001), meta-study (Paterson, Thorne, Canam,

and Jillings, 2001), and qualitative research synthesis (Major & Savin-Baden, 2010).

Several other methodologists argue that a comprehensive synthesis of research should

include both quantitative and qualitative studies. They reason that quantitative methods are

inappropriate to synthesize methodologically diverse research as quantitative research syn-

thesis methods assume a certain degree of methodological and contextual homogeneity across

studies, which is impractical. They assert that only qualitative synthesis methods are suitable

for synthesizing methodologically diverse quantitative and qualitative research (Suri &

Clarke, 2009). Examples of qualitative methods proposed for synthesizing methodologically

diverse research include qualitative meta-analysis (Kasworm, 1990), exploratory case-study

oriented review (Ogawa & Malen, 1991), interpretivist-oriented review (Eisenhart, 1998),

meta-synthesis (Bair, 1999) and realist synthesis (Pawson, 2006).

There is yet another group of methodologists, for example Lather (1999) and Schwandt

(1998), who make a case for post-structural reviews of research which they insist are not

research syntheses per se as they focus on identifying the cracks, or the gaps, in a field rather

than producing a meta-narrative. Unlike Lather (1999) and Schwandt (1998), I have delib-

erately used the term ‘research synthesis’ as a blanket term, which includes critically oriented

post-structural reviews, to reclaim its usage for an inclusive context rather than being limited

to only those syntheses that produce meta-narratives. I have retained the term research syn-

thesis ‘to both circulate and break with the signs that code it’ (Lather, 1993, p. 674) by

rupturing the exclusive notion of research synthesis as an objective and reductionist aggreg-

ation of research findings.

I have used the term qualitative research synthesis as an umbrella term for all qualitative

methods of synthesizing research which are informed by interpretive, critical, emancipatory

and/or postmodern sensibilities. Primary research included in a qualitative research synthesis

may be qualitative and/or quantitative, depending on the purpose of the synthesis.

METHODOLOGICAL UNDERPINNINGS

This article is based on a larger project in which a methodologically inclusive research syn-

thesis (MIRS) framework was conceptualised (Suri, 2007) by distilling and synthesizing di-

verse ideas, theories, and strategies from the extensive bodies of literature on research syn-

thesis methods and primary research methods. The MIRS framework was developed to address

the following overarching question: 'Given that contemporary educational research is marked

by diversity, complexity, and richness of purposes, methods, and perspectives, how can such

variety and complexity be accommodated and reflected at the level of synthesizing educational

research?' In developing the MIRS framework, a combination of purposeful sampling

strategies were employed, some of which are described later in this article to illustrate the

applicability of various purposeful sampling strategies.

The goal of this article is to contest the notion that exhaustive sampling is the only legit-

imate form of sampling for research synthesis. The question that is addressed here is not

64 | Qualitative Research Journal, vol. 11, no. 2, 2011

‘what sampling strategies are typically employed by qualitative research synthesists?' Rather,

by drawing upon hypothetical examples, the question that is being addressed is ‘how might

different purposeful sampling strategies be adapted to expand possibilities within research

syntheses?’ It is hoped that the readers will use this discussion as a departure point to syn-

thesize research for a wide range of purposes, many of which are typically not attempted by

contemporary research synthesists.

I begin this article by building a case for purposeful sampling in research synthesis. Then

I draw on the concepts of data saturation and data sufficiency for guiding decisions related

to enacting closure when searching for relevant evidence in research synthesis. I conclude

this article by outlining key questions which must be considered in making strategic decisions

in relation to sampling in research synthesis. In the literature on primary research methods,

Patton (1990, 2002) has provided a comprehensive discussion of purposeful sampling and

is frequently cited as an authority on purposeful sampling:

The logic and power of purposeful sampling lie in selecting information-rich cases for study

in depth. Information-rich cases are those from which one can learn a great deal about issues

of central importance to the purpose of the inquiry, thus the term purposeful sampling.

Studying information-rich cases yields insights and in-depth understanding rather than em-

pirical generalizations (Patton, 2002, p. 230, emphasis in original).

Patton has suggested 16 strategies for purposeful sampling in qualitative research, each

of which is intended to serve a different purpose. In Patton’s original texts that I refer to,

Patton does not make any suggestion of using purposeful sampling in research syntheses.

Patton’s original texts exclusively discuss utilisation of purposeful sampling in primary re-

search. In this article, I explore the potential adaptation of Patton’s concept of purposeful

sampling to the process of a research synthesis by using three techniques. First, the terms

that I use here are the verbatim terms that Patton (2002) has used to distinguish between

the 16 strategies under the broad umbrella of purposeful sampling. Second, I seamlessly in-

tegrate Patton’s ideas in my discussion of purposeful sampling in research syntheses. In many

instances, I have adapted Patton’s quotes to the process of a research synthesis beyond their

originally intended context of primary research. Third, I illustrate with examples how each

of Patton’s 16 strategies may be utilised in syntheses with varied purposes.

BUILDING A CASE FOR PURPOSEFUL SAMPLING IN RESEARCH SYNTHESIS

Research syntheses on the same topic conducted for different purposes can have different

sampling strategies, each being equally legitimate but tailored to serve the different purposes.

Synthesists must carefully identify sampling strategies that are conceptually aligned with the

synthesis purpose, that credibly and sufficiently address the synthesis purpose, and that are

feasible, ethical and efficient (Kemper, Stringfield, and Teddlie, 2003). Synthesists must

also delineate the caveats associated with their sampling strategies and speculate on how

these caveats may impact upon the synthesis findings. In doing so, it is worth reflecting on

the politics of publishing: that is, who/what is more likely to get published. This involves

being aware of various publication biases, that is, outcome bias, confirmatory bias, funding

bias and methodological bias (for a methodologically inclusive discussion of these biases, see

Suri, 2008). Synthesists must be reflexive about how these potential biases might impact

Harsh Suri,'Purposeful Sampling in Qualitative Research Synthesis' | 65

upon their synthesized product as well as how their synthesis might reify/contest some of

these biases.

A rigorous research synthesis makes much more demand on time and resources when

compared with ad hoc reviews (Elmore, 1991; Ogawa & Malen, 1991; Stock, Benito, and

Lasa, 1996). Just as the perfect primary research study has never been conducted, neither

has the perfect synthesis. The issue that confronts a synthesist is often ‘how to maximise the

quality of the synthesis within the available resources’, rather than ‘how to do the most rig-

orous synthesis’. Several primary researchers discuss various pragmatic constraints which

must be kept in mind when identifying a suitable purpose for study (e.g. Blaxter, Hughes,

and Tight, 2001). Research synthesists are also bound by the pragmatic constraints of time,

resources and access to information and expertise (for a detailed discussion of these pragmatic

constraints, see Suri, 2007). Often decision-makers and stakeholders want relevant, under-

standable, and accurate information which they can use soon. In many practical situations,

synthesists may find that a highly rigorous approach is overly formalistic, too time consuming,

and unnecessarily expensive within the available resources and deadlines (Patton, 1991, pp.

287-289).

Many research syntheses which attract large funding, such as the systematic reviews of

the EPPI-centre, cost about ₤75,000 (Oakley, 2003, p. 28). Often these reviews tend to

address the questions of politicians or other decision-making bodies who can provide adequate

funds. Systematic reviewers typically aim for extensive sampling and assume sufficient access

to financial resources, information and expertise. Unless synthesists strategically design

syntheses within various pragmatic constraints, they may inadvertently contribute to the si-

lencing of concerns of certain groups of stakeholders who cannot commission syntheses that

require large scale funding.

Many qualitative research synthesists also question the viability of holistically synthesizing

a large number of qualitative reports in a way that adequately attends to the intersections

between the findings, contexts, epistemologies, ontologies and methodologies of individual

studies. These qualitative synthesists recommend that an in-depth synthesis of purposefully

selected studies is more desirable than a superficial synthesis of a large number of studies

(e.g. Bondas & Hall, 2007; Booth, 2001; Lloyd Jones, 2004; Noblit & Hare, 1988; Pawson,

Greenhalgh, Harvey, and Walshe, 2005). Even though several qualitative research synthesists

have recommended purposeful sampling in research syntheses, the published literature holds

sparse discussion on how different strategies for purposeful sampling might be applied to

research synthesis. In the next section, I attempt to partially fill this gap in the literature by

discussing how different purposeful sampling strategies may be adapted to synthesize research

for facilitating understanding, participation, emancipation and/or deconstruction.

EXAMINING THE ADAPTABILITY OF PATTON’S PURPOSEFUL SAMPLING STRATEGIES TO QUALITATIVE RESEARCH SYNTHESES

Purposeful sampling requires access to key informants in the field who can help in identifying

information-rich cases. Qualitative research synthesists can draw upon the ‘invisible college’

technique frequently utilised by quantitative researchers. A traditional invisible college involves

a group of central figures investigating a particular field along with the numerous researchers

who are in touch with any of these key researchers. Research retrieved through this channel

is likely to be biased towards the beliefs prevalent among these key researchers (Rosenthal,

1994). Electronic invisible colleges include listservs or newsgroups, some of which might

66 | Qualitative Research Journal, vol. 11, no. 2, 2011

focus on research while others might focus on contemporary practices (Cooper, 1998).

Listservs with a research focus can be useful in identifying primary research studies or previous

syntheses. Practice-focused listservs can be useful in identifying reports that are particularly

valued among practitioners or in identifying the synthesis questions that might be of partic-

ular interest to practitioners. Browsing through listservs can also help identify researchers

and practitioners who have expertise in the substantive domain of interest. The synthesist

can later contact these experts directly to request references to the specific studies on which

their claims and opinions are based.

Extreme or Deviant Case Sampling The extreme or deviant case sampling in a research synthesis would involve selecting ‘illu-

minative cases’ (Patton, 2002, p. 232) that exemplify contexts where an innovation was

perceived notably as a success or a failure. The main weakness of extreme case sampling is

its lack of generalisability through representativeness. This weakness is of less concern for

synthesists who focus on how things should be or could be rather than how things are. This

strategy would be particularly suitable for ‘realist syntheses’, proposed by Pawson (2006),

which investigate how a program is likely to work under particular circumstances by examin-

ing successful as well as unsuccessful implementations of the program.

Intensity Sampling Intensity sampling in a research synthesis would involve selecting studies that are ‘excellent

or rich examples of the phenomenon of interest, but not highly unusual cases… cases that

manifest sufficient intensity to illuminate the nature of success or failure, but not at the ex-

treme’ (Patton, 2002, p. 234). To develop a comprehensive understanding of many educa-

tional changes, it is crucial to examine cases where these changes have been embedded

thoroughly in the system over a sufficient period of time. However, such intense manifestation

of an innovation can be cost-intensive and/or associated with high risk-factors. As a result,

the innovation might be implemented with sufficient intensity in only a few studies. Many

other studies might examine the implementation of the innovation over short durations of

time with minimal interventions. As an example, a large number of studies have been con-

ducted to investigate how students learn in collaborative environments. Given the individu-

alistic nature of most high stake testing, most studies use collaborative learning strategies as

an add-on to regular teaching and learning activities. Many students in these studies might

engage with collaborative tasks superficially. In a small number of studies, the learning

activities as well as assessment tasks have been revised intensely to reward collaboration. An

in-depth synthesis of the latter type of studies would be particularly useful in illuminating

a range of opportunities, challenges, advantages and disadvantages associated with curricula

driven by an ethos of collaborative learning.

Maximum Variation (Heterogeneity) Sampling A maximum variation sample is constructed by identifying key dimensions of variations and

then finding cases that vary from each other as much as possible. This sampling yields: ‘(1)

high-quality, detailed descriptions of each case, which are useful for documenting uniqueness,

and (2) important shared patterns that cut across cases and derive their significance from

having emerged out of heterogeneity’ (Patton, 2002, p. 235). Employing maximum variation

sampling, research synthesists can identify essential features and variable features of a phe-

nomenon as experienced by diverse stakeholders among varied contexts to facilitate informed

Harsh Suri,'Purposeful Sampling in Qualitative Research Synthesis' | 67

global decision-making. Presuming that different study designs illuminate different aspects

of a phenomenon, maximum variation sampling can be utilised to construct an holistic

understanding of the phenomenon by synthesizing studies that differ in their study designs

on several dimensions. In conceptualising the MIRS framework, I used a combination of

sampling strategies, one of which is maximum variation sampling. I deliberately drew ideas

from methods of primary research and research synthesis that were markedly different from

each other on many dimensions.

Homogenous Sample ‘In direct contrast to maximum variation sampling is the strategy of picking a small, homo-

genous sample, the purpose of which is to describe some particular subgroup in depth’

(Patton, 2002, p. 235). Research synthesists are frequently criticised for ‘mixing apples and

oranges’. Research synthesists can overcome this problem to some extent by selecting studies

that are relatively homogenous in their study designs and conceptual scope. Homogenous

samples can facilitate meaningful comparisons across studies. Underscoring the epistemolo-

gical incommensurability of different qualitative methods, some qualitative research synthesists

recommend a certain level of methodological homogeneity among primary research studies

which are included in a qualitative research synthesis (e.g. Eastabrooks, Field, and Morse,

1994; Paterson et al., 2001). Homogenous samples are particularly suitable for participatory

syntheses in which the synthesist co-synthesizes research with practitioners about a phenomen-

on that has direct implications for their practice (for a detailed discussion of participatory

synthesis, see Suri, 2007). For instance, a group of secondary math teachers intending to

introduce collaborative learning activities into their classroom might benefit more from co-

synthesizing collaborative learning research in secondary math rather than collaborative

learning research across all grade-levels and different disciplines.

Typical Case Sampling The purpose of typical case sampling ‘is to describe and illustrate what is typical to those

unfamiliar with the setting’. Typical cases are selected ‘with the cooperation of key informants’

or using ‘statistical data… to identify “average-like” cases’. When employing typical case

sampling, it is crucial ‘to attempt to get broad consensus about which cases are typical–and

what criteria are being used to define typicality’ (Patton, 2002, p. 236). Research synthesists

can select typical primary research studies employed in the field with the cooperation of key

researchers in the field to describe typical methodologies and study designs employed to

examine the phenomenon. This would be particularly useful for studying how common

themes recurring in the published literature might be related to the relative strengths and

weaknesses of the typical methodologies or theories underpinning the typical studies.

Critical Case Sampling Critical case sampling can facilitate ‘logical generalizations’ with the reasoning ‘that “if it

happens there, it will happen anywhere,” or, vice versa, “if it doesn’t happen there, it won’t

happen anywhere”’ (Patton, 2002, p. 236). Critical case sampling in a research synthesis

might be employed to assist stakeholders in making informed decisions about the viability

of an educational program. For example, consider an innovation that produces desirable

outcomes, but is being rejected by many practitioners as they believe that its implementation

requires substantial resources. A synthesis of primary research studies which describe in detail

successful implementation of the innovation with minimal resources might be useful to al-

68 | Qualitative Research Journal, vol. 11, no. 2, 2011

leviate the practitioners’ resistance towards the innovation. Alternatively, consider an innov-

ation which requires substantial financial resources. However, the proponents of the innov-

ation assert that the innovation is cost-effective provided sufficient resources are invested in

its implementation. In such an area, a research synthesist can selectively synthesize cases re-

ported in primary research studies that were sufficiently endowed with resources to logically

verify, or challenge, the claims made by those advocating the innovation.

Snowball or Chain Sampling Snowball sampling involves seeking information from key informants about details of other

‘information-rich cases’ in the field. ‘The chain of recommended informants would typically

diverge initially as many possible sources are recommended, then converge as a few key

names get mentioned over and over’ (Patton, 2002, p. 237). Snowball sampling can be

utilised by seeking information from various listservs to identify primary research reports

that are frequently referred to by various stakeholders interested in the phenomenon. Even

though snowball sampling can introduce an ‘expert bias (e.g. preferences for large samples

or frequently cited studies)’ (Light & Pillemer, 1984, p. 40), it is particularly useful for

capitalising on expert wisdom, identifying studies that are highly valued by different stake-

holders and identifying studies outside the academic mainstream. Another way in which

snowball sampling may be utilised in a research synthesis is based on the assumption that

the most cited primary research reports are the most information-rich cases. The synthesist

might identify the most cited primary research reports by ‘footnote chasing’, which involves

searching the citation indices, or by browsing through the bibliographies of selected reports

on the relevant topic: previous research syntheses, primary research reports, policy documents,

papers written by practitioners and papers written for practitioners. Unlike footnote chasing

for exhaustive sampling, footnote chasing for snowball sampling would involve locating the

most cited papers. However, this approach can reinforce confirmatory bias where studies

that agree with the prevalent wisdom are more likely to be published and cited while studies

that contest the conventional wisdom are less likely to be published or cited (Suri, 2008).

To overcome this problem, Booth (2001) suggests that we regard a key article as one which

references a broad selection of key articles to demonstrate that the study is informed by a

wide range of perspectives. However, Booth’s suggestion can also be problematic as it may

exclude methodologically rigorous and relevant studies which focus deeply, rather than

broadly, on a certain aspect of the phenomenon studied.

Criterion Sampling Criterion sampling involves reviewing and studying ‘all cases that meet some predetermined

criterion of importance’ (Patton, 2002, p. 238). This approach is frequently employed by

research synthesists to construct a comprehensive understanding of all the studies that meet

certain pre-determined criteria. Most research synthesists employ criterion sampling by

stating explicit inclusion/exclusion criteria which includes specifications for methodological

rigour. It is crucial to reflect critically and realistically on the criteria being used, especially

the criteria for methodological rigour. Very strict criteria for methodological rigour can

result in inclusion of such a small number of studies that the transferability of synthesis

findings becomes questionable. At the same time, including methodologically weak studies

can also result in the synthesis findings being based on questionable evidence. Rather than

looking for methodologically perfect studies, the synthesist must reflect on how the method-

ological specifications of the study might have influenced its report.

Harsh Suri,'Purposeful Sampling in Qualitative Research Synthesis' | 69

Theory-Based Sampling, Operational Construct Sampling, and Theoretical Sampling Theory-based sampling involves selecting cases that represent important theoretical constructs

about the phenomenon of interest. This is similar to operational construct sampling in which

one selects cases that represent ‘real-world examples (i.e. operational examples) of the con-

structs in which one is interested’ (Patton, 2002, pp. 238-239, emphasis in original).

Grounded-theorists define theoretical sampling as the sampling that is based on the concepts

emerging from the data for the purpose of exploring ‘the dimensional range or varied condi-

tions along which the properties of concepts vary’ (Strauss & Corbin, 1998, p. 73). Research

synthesists who employ constant comparative methods or grounded-theory approaches can

fruitfully utilise theoretical sampling to systematically elucidate and refine the ‘variations in,

manifestations of, and meanings of a concept as it is found’ (Patton, 1978, p. 238) in the

selected primary research studies. Many qualitative synthesists recommend theoretical

sampling as a suitable option for research syntheses (Dixon-Woods, Agarwal, Jones, Young,

and Sutton, 2005; Mays et al., 2005). For example, in their meta-study, Paterson and her

colleagues (2001) draw on theory-based sampling or operational construct sampling by setting

out operational definitions of the key constructs about the phenomenon of interest. The

boundaries of these operational definitions are further articulated by explicitly stating inclu-

sion/exclusion criteria in relation to selecting primary research reports for the synthesis.

Confirming and Disconfirming Cases ‘Confirmatory cases are additional examples that fit already emergent patterns; these cases

confirm and elaborate the findings, adding richness, depth, and credibility’ (Patton, 2002,

p. 239). Confirmatory cases may be sought in an openly ideological synthesis when the

synthesist wishes to advocate a particular stance for ethical, moral and/or political reasons.

A synthesist may also seek confirming cases in order to validate the perceptions of a particular

group of marginalised stakeholders. Disconfirming cases ‘are the examples that don’t fit.

They are a source of rival interpretations as well as a way of placing boundaries around

confirmed findings’ (Patton, 2002, p. 239). To shake our complacent acceptance of popular

myths and/or generalisations in a field, synthesists can exclusively seek primary research

studies that disconfirm generalisations proposed in policy documents, previous syntheses or

primary research studies.

Stratified Purposeful Sampling ‘Stratified samples are samples within samples’ where each stratum is ‘fairly homogenous’.

The purpose of stratified purposeful sampling is ‘to capture major variations’ even though

‘a common core… may also emerge in the analysis’ (Patton, 2002, p. 240). Stratified pur-

poseful sampling is useful for examining the variations in the manifestation of a phenomenon

as any key factor associated with the phenomenon is varied. In a research synthesis, this

factor may be contextual, methodological, or conceptual. It is particularly useful to study

different models of implementing a particular teaching and learning strategy, such as distinct

models of cooperative learning that are commonly used by teachers. Often, traditional re-

viewers tacitly draw on stratified purposeful sampling by clustering studies according to a

key dimension of variation and then discussing each cluster in-depth. In developing the

MIRS framework, I employed stratified purposeful sampling to select key publications from

many distinct qualitative research traditions. By seeking input from qualitative researchers

with diverse methodological orientations and reading general qualitative research methods

70 | Qualitative Research Journal, vol. 11, no. 2, 2011

texts, I identified distinct strata of qualitative research methodologies and clusters of key

texts within each stratum.

Opportunistic or Emergent Sampling ‘Opportunistic, emergent sampling takes advantage of whatever unfolds as it unfolds’ by

utilising ‘the option of adding to a sample to take advantage of unforeseen opportunities

after fieldwork has begun’ (Patton, 2002, p. 240, emphasis in original). Opportunistic or

emergent sampling can be useful for synthesizing a research area which is at its exploratory

stage, such as mobile learning, or when the synthesist does not have an emic or insider status

in the relevant field of research. Emergent sampling is also suited to participatory syntheses

where the synthesis purpose evolves in response to the changing needs of the participant co-

synthesists (Suri, 2007). For instance, the purpose of a synthesis in the area of mobile

learning might be guided by the key questions or concerns of a group of professors who are

teaching with mobile technologies. The synthesist might then enter the field and search for

reports to address these questions. When the synthesist feeds this information back to the

professors, their questions might also change. In response to their changing questions, the

synthesist might seek further studies with a different set of criteria. While pursuing these

searches, the synthesist is also likely, serendipitously, to find primary research reports that

will provide useful insights into the phenomenon of mobile learning. Given the exploratory

nature of the process of developing the MIRS framework, I employed opportunistic sampling

at the broadest level.

Purposeful Random Sampling ‘For many audiences, random sampling, even of small samples, will substantially increase

the credibility of the results’ (Patton, 2002, pp. 240-241). In theory, research synthesists

can employ exhaustive searches to locate most of the primary research reported on a topic

and then randomly select a few reports from this pool of reports for in-depth discussion.

However, given the resources required for locating all primary research reports on a topic,

it would be cost-ineffective to randomly discard studies from further consideration. Hence,

this sampling has little appeal in practice.

Sampling Politically Important Cases Sampling politically important cases involves ‘selecting (or sometimes avoiding) a politically

sensitive site or unit of analysis’ (Patton, 2002, p. 241). Like most scholarly endeavours in

education, research syntheses are essentially political. A synthesist might consciously select

politically important reports so that the synthesis gains the attention of different stakeholders

and the synthesis findings get used. For instance, in a synthesis of key criticisms of educa-

tional research published in the 1990s, Oancea (2005) illustrated her key observations

through a detailed analysis of three politically important documents that were frequently

cited in the newspapers. Syntheses of hot topics, in which several stakeholders are interested,

are also likely to attract appropriate funding and more impact (Elmore, 1991).

Convenience Sampling Convenience sampling is ‘probably the most common sampling strategy–and the least desir-

able’. It would involve selecting reports that are ‘easy to access and inexpensive to study’.

This form of sampling is ‘neither purposeful nor strategic’ (Patton, 2002, pp. 241, emphasis

in original). As in many primary research studies, convenience sampling is also employed

Harsh Suri,'Purposeful Sampling in Qualitative Research Synthesis' | 71

in many research reviews. Both primary researchers and research synthesists must resort to

convenience sampling as the last option. First, synthesists must reflect on various other

purposeful sampling strategies to identify a strategy that is most suitable for their purpose

and is also viable within the pragmatic constraints associated with the synthesis. When

convenience sampling has been employed in a research synthesis, the nature of its use and

associated caveats must be clearly described.

Combination or Mixed Purposeful Sampling Synthesists often employ a combination of two or more sampling strategies to select evidence

that adequately addresses their purpose. Mixed purposeful sampling can facilitate triangulation

and flexibility in meeting the needs of multiple stakeholders (Patton, 2002). For example,

synthesists may strategically utilise extensive sampling to draw generalisations at a higher

level of abstraction. Then, they may employ typical case sampling to provide readers with

an immediacy of typical studies that contributed towards informing the more abstract gen-

eralisations. When selecting a combination of sampling strategies, synthesists must reflect

on how those strategies complement each other.

SAMPLE SIZE AND ENACTING CLOSURE TO FURTHER SEARCHES

Decisions associated with enacting closure to further searches for evidence must be guided

by the purpose of the synthesis, the overarching logic of sampling, and pragmatic constraints.

There are two main logics associated with these decisions in primary research as well as re-

search synthesis: data saturation and data sufficiency.

Data Saturation Data saturation may be associated with the stage when further collection of evidence provides

little in terms of further themes, insights, perspectives or information in a qualitative research

synthesis. The concept of data saturation is dependent on the nature of the data source as

well as the synthesis question. There is a higher likelihood of reaching data saturation if the

data collection is purposeful. The more precise a question, the quicker it tends to reach data

saturation. Progressive refinement of a synthesis question is likely to bring an earlier stage

of data saturation. With open ended questions, every new report is likely to offer additional

information. A broad question, like ‘what does research tell us about virtual classrooms?’, is

not likely to bring about a sense of closure or data saturation. On the other hand, the syn-

thesist is likely to reach the data saturation stage earlier with a focused question like ‘what

are the key methodologies being employed to examine gender differences in math achievement

on standardised tests among middle school students?’

Data Sufficiency Most research synthesists refrain from rigidly prescribing a minimum or maximum number

of primary research studies to be included in a synthesis. Some methodologists recommend

their methods are suitable for synthesizing even a small number of qualitative research

studies: for example, meta-ethnography for three studies (Noblit & Hare, 1988), aggregated

analysis for four studies (Eastabrooks et al., 1994) and meta-study for twelve studies (Paterson

et al., 2001, p. 38). Many qualitative research synthesists who synthesize methodologically

diverse research tend to conduct extensive searches and include a large number of studies.

For example, Wideen and colleagues (1998) included 93 studies and Kasworm (1990) in-

72 | Qualitative Research Journal, vol. 11, no. 2, 2011

cluded 96 documents. Paterson and her colleagues identify the following principles that

ought to guide the sample size of the synthesis: ‘the data should be sufficient to permit

comparisons among selected dimensions and constructs’; ‘the reports should reflect the work

of several distinct and independent investigators’; and ‘the data should be sufficient to answer

the research question’ (Paterson et al., 2001, p. 37). These principles can be usefully applied

to most qualitative research syntheses. A research synthesist, like a primary researcher, is often

confronted with various pragmatic constraints of time and resources as well as access to ex-

pertise and information. The stage of data saturation is not frequently reached in either

primary research or research synthesis projects. The logic of data sufficiency is guided by

the synthesist’s perception of what constitutes sufficient evidence for achieving the synthesis

purpose. The synthesist must repeatedly ensure that the claims made in the synthesis are

sufficiently grounded in the evidence collected for the synthesis.

USING THIS DISCUSSION AS A DEPARTURE POINT

In general, synthesists must leave an ‘interpretive trail’ of the different ways in which studies

have been used or omitted (Pawson et al., 2005, p. 31). In leaving an interpretive trail of

their searches, research synthesists must critically reflect and report on the following questions:

• What sampling logic is amenable to meet the synthesis purpose, within the available re-

sources and pragmatic constraints, efficiently and sufficiently?

• What logic will guide the decision to cease searching for further evidence?

• What are the justifications for these decisions?

• What are the caveats associated with these decisions?

In this article, I have illustrated how different, purposeful sampling strategies may be

suited for research syntheses conducted for diverse purposes. By drawing on a range of hy-

pothetical examples, I have illuminated how various purposeful sampling strategies might

be particularly suitable for syntheses geared to facilitate understanding, participation,

emancipation and deconstruction. I have discussed how research synthesists can draw upon

the concepts of data saturation and data sufficiency to inform their decisions to cease searches

for further studies. In my discussion of strategic sampling in research syntheses, I urge syn-

thesists to carefully identify sampling strategies which address the synthesis purpose efficiently,

credibly, sufficiently and ethically. I make no attempt to prescribe certain sampling strategies

for research synthesis. Rather, the intention here is to expand methodological possibilities

within research syntheses by proposing new ways of thinking about the methodology of

synthesis. I hope the users and producers of research synthesis will use this article as a depar-

ture point to think creatively and critically about purposes and amenable sampling strategies

for a research synthesis.

REFERENCES Bair, C. R. (1999, Nov 18). Meta-synthesis. Paper presented at the 24th annual meeting of the Asso-

ciation for the Study of Higher Education: San Antonio (ED437866).

Blaxter, L., Hughes, C. & Tight, M. (2001). How to research (2nd ed.). Buckingham: Open University

Press.

Harsh Suri,'Purposeful Sampling in Qualitative Research Synthesis' | 73

Bondas, T. & Hall, E. O. C. (2007). A decade of metasynthesis research in health sciences: A meta-

method study. International Journal of Qualitative Studies on Health and Well-Being, 2(2), 101-

113. [CrossRef]

Booth, A. (2001, May 14). Cochrane or cock-eyed? How should we conduct systematic reviews of

qualitative research? Paper presented at the Qualitative Evidence-based Practice Conference,

Taking a Critical Stance. Coventry University.

Cooper, H. M. (1998). Synthesizing research: A guide for literature reviews. (3rd ed.). Thousand

Oaks, CA: Sage.

Dixon-Woods, M., Agarwal, S., Jones, D., Young, B. & Sutton, A. J. (2005). Synthesising qualitative

and quantitative evidence: A review of possible methods. Journal of Health Services Research and

Policy, 10(1), 45-53. [CrossRef]

Eastabrooks, C. A., Field, P. A. & Morse, J. M. (1994). Aggregating qualitative findings: An approach

to theory development. Qualitative Health Research, 4(4), 503-511. [CrossRef]

Eisenhart, M. (1998). On the subject of interpretive reviews. Review of Educational Research, 68(4),

391-399.

Elmore, R. F. (1991). Comment on ‘Towards rigor in reviews of multivocal literatures: Applying the

exploratory case study method’. Review of Educational Research, 61(3), 293-297.

Jensen, L. A. & Allen, M. N. (1994). A synthesis of qualitative research on wellness-illness. Qualitative

Health Research, 4(4), 349-369. [CrossRef]

Kasworm, C. E. (1990). Adult undergraduates in higher education: A review of past research perspect-

ives. Review of Educational Research, 60(3), 345-372.

Kemper, E. A., Stringfield, S. & Teddlie, C. (2003). Mixed methods sampling strategies in social

science research. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social

& behavioral research, Thousand Oaks, CA: Sage, 273-296.

Lather, P. (1993). Fertile obsession: Validity after poststructuralism. The Sociological Quarterly,

34(4), 673-693. [CrossRef]

Lather, P. (1999). To be of use: The work of reviewing. Review of Educational Research, 69(1), 2-

7.

Light, R. J. & Pillemer, D. B. (1984). Summing up: The science of reviewing research. Cambridge:

Harvard University Press.

Lloyd Jones, M. (2004). Application of systematic review methods to qualitative research: Practical

issues. Journal of Advanced Nursing, 48(3), 271-278. [CrossRef]

Major, C. H. & Savin-Baden, M. (2010). An introduction to qualitative research synthesis: Managing

the information explosion in social science research. London: Routledge.

Mays, N., Pope, C. & Popay, J. (2005). Systematically reviewing qualitative and quantitative evidence

to inform management and policy making in the health field. Journal of Health Services Research

and Policy, 10(1), 6-20. [CrossRef]

Miles, M. B. & Huberman, A. M. (1994). Qualitative data analysis: An expanded sourcebook. (2nd

ed.). Thousand Oaks, CA: Sage.

Noblit, G. W. & Hare, R. D. (1988). Meta-ethnography: Synthesizing qualitative studies. Newbury

Park: Sage.

Oakley, A. (2003). Research evidence, knowledge management and educational practice: Early lessons

from a systematic approach. London Review of Education, 1(1), 21-33. [CrossRef]

Oancea, A. (2005). Criticisms of educational research: Key topics and levels of analysis. British

Educational Research Journal, 31(2), 57-183. [CrossRef]

Ogawa, R. T. & Malen, B. (1991). Towards rigor in reviews of multivocal literatures: Applying the

exploratory case study method. Review of Educational Research, 61(3), 265-286.

Paterson, B. L., Thorne, S. E., Canam, C. & Jillings, C. (2001). Meta-study of qualitative health

research: A practical guide to meta-analysis and meta-synthesis. Thousand Oaks, CA: Sage.

Patton, M. Q. (1978). Utilization-focused evaluation. Thousand Oaks, CA: Sage.

Patton, M. Q. (1990). Qualitative evaluation and research methods. Thousasnd Oaks, CA: Sage.

74 | Qualitative Research Journal, vol. 11, no. 2, 2011

Patton, M. Q. (1991). Towards utility in reviews of multivocal literatures. Review of Educational

Research, 61(3), 287-292.

Patton, M. Q. (2002). Qualitative research and evaluation methods (3rd ed.). Thousand Oaks, CA:

Sage.

Pawson, R. (2006). Evidence-based policy: A realist perspective. London: Sage.

Pawson, R., Greenhalgh, T., Harvey, G. & Walshe, K. (2005). Realist review: A new method of

systematic review designed for complex policy interventions. Journal of Health Services Research

and Policy, 10(1), 21-34. [CrossRef]

Petticrew, M. & Roberts, H. (2006). Systematic reviews in the social sciences: A practical guide.

Malden, MA: Blackwell. [CrossRef]

Rosenthal, M. C (1994). The fugitive literature. In H. M. Cooper & L. V. Hedges (Eds.), The

handbook of research synthesis, New York: Sage, 85-94.

Sandelowski, M., Docherty, S. & Emden, C. (1997). Qualitative metasynthesis: Issues and techniques.

Research in Nursing and Health, 20(4), 365-371. [CrossRef]

Schwandt, T. A. (1998). The interpretive review of educational matters: Is there any other kind?.

Review of Educational Research, 68(4), 409-412.

Stock, W. A., Benito, J. G. & Lasa, N. B. (1996). Research synthesis: Coding and conjectures.

Evaluation and the Health Professions, 19(1), 104-117. [CrossRef]

Strauss, A. & Corbin, J. (1998). Basics of qualitative research: Techniques and procedures for devel-

oping grounded theory. (2nd ed.). Thousand Oaks, CA: Sage.

Suri, H. (2007). Expanding possibilities within research syntheses: A methodologically inclusive re-

search synthesis framework. Unpublished PhD thesis. Melbourne: The University of Melbourne.

Suri, H. (2008). Ethical considerations in synthesising research: Whose representations?. Qualitative

Research Journal, 8(1), 62-73. [CrossRef]

Suri, H. & Clarke, D. J. (2009). Advancements in research synthesis methods: From a methodolo-

gically inclusive perspective. Review of Educational Research, 79(1), 395-430. [CrossRef]

Wideen, M. F., Mayer-Smith, J. A. & Moon, B. J. (1998). A critical analysis of the research on

learning to teach: Making the case for an ecological perspective on inquiry. Review of Educational

Research, 68(2), 130-178.

Zimmer, L. (2006). Qualitative meta-synthesis: A question of dialoguing with texts. Journal of Ad-

vanced Nursing, 53(3), 311-318. [CrossRef]

ABOUT THE AUTHOR Harsh Suri (PhD) is a Lecturer at the Centre for the Study of Higher Education, The University of Melbourne, Australia. She has developed a methodologically inclusive research synthesis (MIRS) framework for designing and evaluating research syntheses from distinct methodological orienta- tions. She has written several papers on contemporary methods of research synthesis. Two of her early papers were recognised as outstanding presentations: Early Career Researcher Award at the Mathematics Education Research Group of Australasia (MERGA) conference in 1997 and Best Graduate Presentation at the AQR conference in 1999. She is cited in most current publications on research synthesis methods which are inclusive of qualitative research.

Email: [email protected]

Harsh Suri,'Purposeful Sampling in Qualitative Research Synthesis' | 75

Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.