Week 5

Q R

Qualitative Research 13(6) 699 –716

© The Author(s) 2012 Reprints and permissions:

sagepub.co.uk/journalsPermissions.nav DOI: 10.1177/1468794112465634

qrj.sagepub.com

Participant selection as a conscious research method: thinking forward and the deliberation of ‘Emergent’ findings

L Earle Reybold George Mason University, USA

Jill D Lammert George Mason University, USA

Stacia M Stribling George Mason University, USA

Abstract Participant selection is one of the most invisible and least critiqued methods in qualitative circles. Researchers do not just collect and analyze neutral data; they decide who matters as data. Each choice repositions inquiry, closing down some opportunities while creating others. After reviewing the selection literature, we present critical vignettes of our selection choices in three separate studies, examining how those choices directed meaning making within and beyond the studies. Our analysis across these vignettes uncovered a constant interface—and often a struggle—between our personal situations and social agendas as qualitative researchers. Four aspects of this Reporting In/Reporting Out tension are discussed: trusting qualitative research, building the story, dealing with powerful others, and accepting unintended consequences. We encourage qualitative researchers to critically think forward their selection choices before and during the research process, to be mindful that selection is a constitutive method of the data collection and analysis process.

Keywords method, qualitative research, quality, sampling, selection

Corresponding author: L Earle Reybold, College of Education and Human Development, Division of Educational Psychology, George Mason University, MS 6D2, Fairfax, VA 22030, USA. Email: [email protected]

Article

700 Qualitative Research 13(6)

Qualitative research is a fusion of planning and discovery. Some research choices are deliberate and designed into the research process; others are spontaneous and provoked by circumstance. The result is a mosaic of interpretation that emerges through both the structure and the moment of inquiry. This defining essence of qualitative research is characterized by Denzin and Lincoln (2005) as a bricolage or montage, a piecing together and editing of parts into a whole with its own meaning and significance. Each research choice has the potential to reposition inquiry, to erase some possible options while creating others.

Nowhere is this dynamic more obvious than in the selection of who and what to study. In qualitative research, purposeful selection is legendary. In fact, Patton (2002) claimed ‘nothing better captures the difference between quantitative and qualitative methods than the different logics that undergird sampling approaches’ (p. 230). In qualitative research, the logic of selection is grounded in the value of information-rich cases and emergent, in-depth understanding not available through random sampling. From this perspective, purposeful selection is a strategy for accessing appropriate data that ‘fit the purpose of the study, the resources available, the questions being asked, and the constraints being faced’ (p. 242).

Purposeful selection, though, is more than a technique to access data; our selection choices frame who and what matters as data (Freeman, 2000). These choices interface the other methods in a study to ultimately become the stories that are told. Consider, for example, the intersection of participant selection and interview analysis. The par- ticipant’s story is embedded in a matrix of researcher choices: research questions, selection criteria, interview style, analysis technique, and countless other choices. Thus, purposeful selection is a mechanism for making meaning, not just uncovering it. From this perspective, purposeful selection is epistemological; researchers construct versions of reality grounded in their selection choices.

We argue that selection constitutes one of the most invisible and least critiqued meth- ods in qualitative scholarship. The rare in-depth discussion of selection usually offers a postinquiry critique of choices, a reflection on the outcome of those choices. Oakley (2003) criticized the ‘mythology of ‘hygienic’ research [and the] mystification of the researcher and the researched as objective instruments of data production’ (p. 260). Still, except for a few confessional tales of reflexivity, published methods—and the methods of selection—seem laid out for the audience as linear and uncomplicated.

But Bentz and Shapiro (1998) argued that mindful inquiry ‘stresses focus, intention, and awareness of whatever is present in a situation or experience’ (p. 171, emphasis added). To be mindful of selection choices requires accountability to those decisions in the moment of inquiry. We describe this as ‘thinking forward’ the selection process. Purposeful selection, from this perspective, is a conscious and deliberate method of considering alternative research trajectories created by our choices.

In this article, we critically examine our own selection choices, distinguishing between ‘reflecting back’ and ‘thinking forward’ about the selection process. First, we review the literature about the selection process as (a) subjective focus, (b) quality procedure, and (c) integrative method. We then discuss the method of analysis we applied to our own selection choices—a cyclic process of personal reflection, critical questioning, and group interview. Following our discussion of method, we present critical narratives of how we

Reybold et al. 701

as individual researchers have made sense of our selection choices, both in terms of ‘reflecting back’ and ‘thinking forward’. In that same section, we also present our analysis across these narratives. That analysis revealed a constant tension between personal situ- ations and researcher agendas. Our discussion examines this tension in relation to four subthemes: trusting qualitative research, dealing with powerful others, building the story, and accepting unintended consequences.

Literature: what is the nature of selection?

This section examines how the literature characterizes the nature and function of selection in qualitative research. We have chosen to focus on three primary approaches: selection as subjective focus, selection as quality procedure, and selection as integra- tive method.

Selection as subjective focus

Scholars across traditions of research note the subjective nature of qualitative inquiry, pointing out the human tendency to see reality as it is believed to be (Heshusius, 1994; Peshkin, 1988; Reinharz, 1997; Visweswaran, 2003). This accounting for subjectivity has been described as ‘reciprocal ethnography’ (Lawless, 1992: 311), ‘working the hyphen’ between self and other (Fine, 1994: 70), the ‘ethnographic self’ (Coffey, 1999: 5), and ‘a journey of interpretation’ (Peshkin, 2000: 5). Gubrium and Koro-Ljungberg (2007) examined interviews as an ‘inevitable border-making process of communication’ (p. 690).

Qualitative researchers often question the core assumptions of their research, both within and beyond the design. Why are certain participants selected? How are interview questions crafted? What observation notes are left unwritten? Who am I in this decision-making process? Subjectivity is the qualitative researcher’s obligation to situate the self in relation to inquiry, to elucidate research choices as a matter of position, without manipulating interpretation and representation of data (Fine, 1994). Thus, we socialize our students and novice researchers to understand that subjectivity is good; bias is bad. But where to draw the line between the two? Coffey (1999) described it as a tension ‘between the production process and the products of fieldwork’ (p. 136).

Subjectivity is a substantial feature of selection, but it is rarely discussed in terms of site and sample selection beyond choice of topic or participant selection criteria. Perhaps that limitation is more a function of how researchers communicate selection choices to our reading audience. Traditional publication outlets expect a conventional listing of methods and usually those are descriptive of the final set of choices rather than a map of the many we tried and even discarded. Final manuscripts often present the method of qualitative research as a continuity of choices, with each method dis- cussed separately and tangibly.

Of course, we know research does not unfold so neatly. Sometimes, qualitative researchers anticipate choices, such as selection of participants, to evolve across the study through theoretical sampling and sensitivity to ‘nuances and queues in the data’

702 Qualitative Research 13(6)

(Corbin and Strauss, 2008: 19). But we also must respond to methods choices in real time, and we must do so using resources at hand.

Selection as quality procedure

There is general agreement among qualitative researchers that purposeful selection is the best strategy to obtain ‘information-rich’ cases that can give in-depth insight into the subject of study (e.g. Creswell, 2002; Freeman et al., 2007; Glesne, 2006; Maxwell, 2005; Patton, 2002). Scholars also agree that researchers should clearly state the rationale for their selection decisions so consumers of research can consider sources of bias or other limitations to the study. Glesne (2006) suggested that researchers ‘discuss what is peculiar about your respondent selection that could show the phenomena of interest in some lights but not in others’ (p. 169). Patton even goes so far as to say that researchers should anticipate and address criticisms that may be made of a particular selection strategy.

Each study has its own ‘special demands’ (Brewer and Hunter, 2006) that necessarily direct ‘practitioners to study selectively certain universes of persons and groups while putting others beyond their reach’ (p. 98). Obviously, one would expect to get different findings by changing focus; nonetheless, these choices are ignored in many studies. The complexity of selection extends beyond identification of participants. Selection is an extension of one’s theoretical and conceptual framework. ‘If sampling is seen as a rational selection process that has implications for the truth claims of one’s research, then sampling is going on all the time’ (p. 102).

How researchers seek and gain access to participants reflects more than the research- er’s own assumptions, though. These choices also reflect the sociocultural milieu in which the research is conducted. Selection choices are slices of the reality around us, both in terms of tangibles (e.g. resources, physical access, participant availability) and intangibles (e.g. social and political relationships, rapport, historical timing).

Research is a social act that is itself shaped by social and cultural conventions. Although we seek access to information that we believe to be data and represent data in ways that we consider to be the telling of data, what we are really doing is making a social and historical statement of what are to be data. (Freeman, 2000: 367)

Selection as integrative method

Aside from textbook discussions of sampling procedures in qualitative research (e.g. Patton, 2002, being one of the most comprehensive and detailed), it is difficult to find a significant amount of data-based inquiry that explores the idea of selection as method. The articles that do highlight sampling and selection hone in on a range of issues related to this process. For example, Sandelowski (1999) examined the ways in which time impacts qualitative research design, particularly in the field of nursing and health. She distinguished between studies focused on a single event in time and on an event as it unfolds over time or on both issues simultaneously. Participants, she said, ‘enter quali- tative studies by virtue of their intimate knowledge of and ability to communicate these

Reybold et al. 703

events’ (p. 81). They are not merely sampling participants. Researchers are sampling events and time as a critical feature of those participants.

Sobal (2001), in a retrospective analysis of certain articles, concluded that researchers should explicitly discuss the ways in which researchers reached ‘saturation’ in data collection and should give more attention to issues of sample extensiveness in order to effectively transfer the knowledge gained. Still, his focus remains on quantity without addressing the issue of quality and what a given sample, no matter how large or small, has to offer in deepening our understanding of an issue or phenomenon.

Selection as method requires researchers to be aware that choosing sites and partici- pants for our research is more than a technical process. As Peshkin (2001) reminded us, these choices are the ‘selection and choice of what to perceive’ (p. 251). How we perceive the research issue impacts who we perceive to be at the core of that issue and thereby what we hope to learn from those whom we have identified.

Peshkin (2000), reflecting on his own ‘journey of interpretation’ (p. 5) in one particular study, offered ‘a counterpoint of problematics that reveals where alternative interpretive decisions could have been made’ (Abstract). Peshkin’s idea of interpretation includes selection choices across the study. ‘I select what will come into and affect my concep- tion. Such selection, together with ordering, associating, and meaning making, is an ele- ment of interpretation’ (p. 9). This broad application of the term ‘selection’ fits our argument that selection is a form of data collection, not just entrée to the field.

Method: a discussion of emergence

As suggested by the scholarship reviewed in this article, selection is not a simple or isolated decision. It is intricately connected to the ways in which a researcher perceives an issue, the events related to that issue, the purpose of exploring the issue, and the knowledge one hopes to gain about it. Selection must be considered as an essential element of method in the way that it shapes research questions, data collection, data analysis, and findings. In other words, conclusions emerge through choices.

We initiated this study following an advanced qualitative course taught by Earle that emphasized dialogue and critique of methods. Jill and Stacia were students in the class, but they were already established researchers in their own fields of study. (Both Jill and Stacia have since graduated and have taught a range of graduate courses, including research methods.) While the initial interest in selection was stimulated by a class semi- nar, we designed and implemented this project as peers. Early on we discussed how dif- ferent selection choices might have altered our research, so we mapped our selection choices for a specific study in which each of us had a major role. This led to a spontane- ous and thoughtful discussion of the obvious and not-so-obvious bonds between selec- tion and quality, and we decided to take up the challenge to explore the selection process more deeply and personally.

Reflecting back and thinking forward

During our initial meetings, we met informally to explore our own selection experiences and assumptions, particularly in relation to our definitions of selection and its value to

704 Qualitative Research 13(6)

qualitative research in general. We recorded these sessions for the sake of focusing future discussions. These meetings were invaluable in deciding our working relationship across our roles as faculty member and doctoral students. We chose a collaborative approach to intentionally avoid a hierarchical ‘mentoring’ relationship based on rank, thus, we shared fully in the process and product of our study.

Meanwhile, these discussions prompted us to share literature sources and read across various genres of qualitative design to focus our inquiry. We revisited methods texts and articles related specifically to selection, as well as critical examples of thoughtful selection choices. Through this exploration of the literature, we considered alternative frameworks to explore our ideas: heuristic and phenomenological inquiry, narrative and portraiture, self-study, and action research. Ultimately, we decided to blend elements of heuristic and narrative research because of our decision to focus on the story of our own practice as researchers.

Those first meetings resulted in the elaboration of an important distinction between how we see selection methods generally presented in most discussions of the topic (reflecting back) and our support for a more critical rendering of selection (thinking forward). The first concept is a useful but limited critique of site and participant choices, while the latter encourages communication of redirection of those methods ab initio, or from the beginning.

General selection discussions position selection critique as a reaction to what has been done; sometimes this results in revision or explanation of how and why certain partici- pants were selected. Alternately, we promote a deliberate and conscious accounting of selection; this requires forethought of how options might unfold and how we as research- ers choose to adjust selection across the study. Thinking forward requires us to counter our actual choices with possible outcomes of our choices.

Our readings in narrative and heuristic research, along with our decision to focus on the concept of thinking forward, led us to consider our own stories of selection for analy- sis. We prepared our narratives and shared them with one another for critique and ques- tioning. We also developed an analytic framework for these narratives and other data. Our goal was to engage in a critical action cycle that would be public and emergent but specifically targeted on Stacia’s emergent research and thinking forward her selection process (see Figure 1). This model served as our analytic design for sharing and inter- preting our selection experiences, allowing us to explore our decisions in relation to our professional roles and research situations, inform ongoing decisions in our own research, provide a scaffold experience to developing research, and situate selection as part of the data generation cycle.

Based on this framework and our discussions, we developed a set of open-ended questions for an interactive group discussion. Questions focused on how we learned about selection techniques (formally and practically), our experiences with selection, and critical reflection on our socialization and experiences. We audiotaped this final discussion session, during which we addressed the questions in a conversational mode that allowed for follow-up questions and tangential points we had not considered individually.

We each reviewed the data, drawing on our personal narratives of the selection pro- cess, our formal group dialogue, and our many follow-up discussions in person and by

Reybold et al. 705

F ig

u re

1 .

A na

ly ti c

fr am

ew o rk

fo r

th in

ki ng

fo rw

ar d

th e

se le

ct io

n pr

o ce

ss .

706 Qualitative Research 13(6)

email. As we shared our individual analyses, our conversations began to focus more on shared circumstances across our stories of selection.

A critical analysis of selection choices: so what?

The phrase ‘thinking forward’—coined during one of our brainstorming sessions—pulls together our interests in conscious deliberation of methods and consideration of alterna- tive research trajectories created by our selection choices. In the spirit of calling more attention to the ways in which selection choices affect—even define—results in qualita- tive research, we offer vignettes of our own selection decisions. First, Earle and Jill explore individual stories of how each one made past selection choices that could have been more conscious and deliberate. Stacia then explains how participation in this group challenged her own thinking about selection, particularly for her dissertation planning. We conclude with a collective analysis of our experiences and a discussion of how our understanding of selection choices might be used by other researchers to think forward their own selection choices.

Earle: regrouping and restructuring selection in a longitudinal study. I was a graduate of adult education and looking for a tenure-track faculty position. Meanwhile, I was working as an adjunct instructor and research associate. After months in Malaysia collecting data, I had plenty of transcripts to keep me busy writing for some time. However, I had self- financed much of my research, and my situation dictated a change of venue for continued study as a hopeful faculty member. I decided to focus those interests inward on the academy. The initial participants, doctoral students at one of the universities where I was teaching, were convenient, as was the topic.

At first, I did not intend the study to be longitudinal; I only wanted to see whether and how academic culture affected doctoral students’ thinking about the professorate. After I interviewed those students, I realized my deepening curiosity about faculty identity and epistemology. I wondered, though, whether the stories told by these four students were shaped by their experiences in a certain academic culture. So, I posted announcements in local college hallways and at education conferences (this latter choice extended the study beyond the bounds of the three original institutions). Initial participants told peers about the study, many of whom volunteered to be interviewed. They also recommended their mentor faculty members as participants, noting the significance of the relationship to the development of their identity as faculty. I realized the value of network sampling and intentionally contacted students and faculty members who were mentioned in previous interviews.

Then two crucial events changed the nature and direction of the study even further. One participant graduated and accepted her first faculty position; that same year I accepted my first tenure-track position in another state. The former student asked me to consider interviewing her as she transitioned into her faculty role. That was the impetus for the longitudinal design. I continued to add participants, but with the intention of fol- lowing their experiences across their careers (Reybold, 2003). Additionally, in my capacity as a new faculty member, I shared my research journey with other incoming faculty members; several volunteered for the study. I decided to include first-year

Reybold et al. 707

faculty members in the study if they fit the other criteria. Again, I spread word about the study parameters to area institutions and at national conferences. Without research assistance, I was conducting all interviews, usually on an annual basis.

When the number of participants reached 48, I decided to stop adding members to the study, that is, until I realized two flaws in my developmental network selection strategy. While my sample looked much like the education classrooms in which I taught—white and female—I did not want my research to recreate the bias of studies I critiqued. Consequently, rather than shape my selection choices as a response to emerging themes, I decided to shape future selection choices as a matter of equity. The study now numbers 55 participants. While I stopped collecting new data in 2009, I continue analysis through the lens of faculty ethics and discrimination against women and minority faculty mem- bers (Reybold, 2008; Reybold et al., 2012).

I fully believe that the developmental nature of the study’s design is its strength; it is also its fundamental weakness. Although the participants fit my selection criteria, how did convenience impact evolving criteria and later selection choices? How useful are findings collected at the beginning of the study? Finally, how do I ‘track back’ my selec- tion strategies to account for my findings as a result of those early choices?

Jill: critiquing the unseen impact of selection choices. I participated in a research project intended to document and evaluate the work of a development program being imple- mented in rural communities in a developing country. The project had two components: qualitative group interviews of program beneficiaries and a quantitative evaluation of the program’s impact.

I was tasked with carrying out the qualitative study, yet as a junior researcher, I had no say during participant selection. Interview participants were identified by the program team based on two criteria: communities that were easy to access and who were consid- ered ‘good to talk to’ (essentially a blend of convenience and purposeful selection). These criteria were considered justified given the time and resource constraints, but I strongly believe that underlying the choice of participants was a (possibly subconscious) desire to interview people who had positive experiences with (and therefore good things to say about) the program. Confounding the problem, due to the limited time frame and language issues, one of the program team members and two representatives of the local implementing agency—symbols of power and money for the participants—actually con- ducted the interviews while I listened and occasionally asked for clarification or addi- tional information. I then analyzed the transcribed and translated data for the final report.

I have no doubt that participant selection in this project limited our results. Even though the development program strives to empower participating communities, during the interviews, the different local actors actively maintained traditional patterns of social class interaction and patronage, calling into question the ability and/or willingness of respondents to be truthful and open. While participants appeared at ease during the inter- views and talked freely about difficulties they encountered—even criticizing certain aspects of the program—they did not stray too far from playing their ‘role’ as program beneficiaries. Our participants often spoke of themselves as ‘ignorant’ and needing out- side help. One time, they turned an interview on its head and started questioning the local representative about a negative side effect of the program. Yet, immediately recognizing

708 Qualitative Research 13(6)

from the representative’s response that they had ‘gone too far’ with their critique, they began telling her how nice and clean and organized one of the program investments was and expressed their thanks.

Despite the limitations of the qualitative study results, I am certain the project team did not purposefully set out to skew the findings. They simply believed in the program and wanted to share their beliefs with the world—and their choice of participants reflected that desire. Furthermore, the participant selection criteria were considered per- fectly appropriate given the existing financial and temporal constraints.

I do not mean to say that the results included in the final report do not reflect the ‘real- ity’ of the participants of our interviews—however, complex that ‘reality’ might be—but the richness and completeness of our findings definitely suffered due to limitations in participant selection. In the end, the qualitative report might be more appropriately con- sidered a public relations tool.

Stacia: short-term planning for a long-term research agenda. Engaging in conversations with Earle and Jill about qualitative research methods, and more specifically, selection in qualitative studies, encouraged me to carefully examine my own selection process as I planned my doctoral dissertation. The goal of my dissertation was to better understand the process of engaging young children in a critical literacy curriculum (Stribling, 2010). In the beginning, I had a general idea of the issue I wanted to study but soon realized that who I studied was intricately connected to what I might learn. I knew that I wanted to study teachers who were already incorporating critical literacy skills in their classroom practice, but it turned out they were quite difficult to find.

As I searched for participants, my proposal writing came to a halt. I was not sure how to frame the issue, as I did not have a context for the problem and therefore did not know what specific research questions I might ask. Not knowing the specific questions ham- pered my ability to pinpoint data sources and collection procedures and, therefore, the method of data analysis. Some colleagues worried that my failure to complete my pro- posal was a result of procrastination, so they encouraged me to just write my plans for the study and ‘plug in’ the details of the participants later; if needed, I could include a series of ‘if/then’ statements that might allow for possible participant scenarios. I really struggled with this idea since there seemed to be an infinite number of ‘if/then’ state- ments I could write: My research questions would differ based on how many participants I found, how committed they were to critical literacy practices, how they enacted critical literacy in their own classrooms, who their students were, and so on.

In thinking forward my selection procedures for my dissertation, I wrote memos doc- umenting my thinking throughout the selection process. I then incorporated these memos into my dissertation proposal in order to make my process more transparent for the reader. It was important for the readers to understand that my research evolved over time; like a camera coming into focus, the details about research questions, data collection, and data analysis became clearer as I found the teachers (and students) who would participate in my dissertation study.

Of course, once I identified these participants, selection remained a salient issue in the process that unfolded. Because the two teachers with whom I worked had different visions of critical literacy (something I could not have known until spending significant

Reybold et al. 709

amount of time in their classrooms), my researcher role shifted significantly in each setting. I entered as a participant-observer in both classrooms but ended up emphasizing the ‘observer’ role in one classroom while emphasizing the ‘participant’ role in the other. I feel extremely fortunate to have been engaged in deep discussions of ‘thinking forward’ my selection process from the start, not because it eliminated complications as my dis- sertation unfolded but because it better prepared me for the potential impacts of the choices I made and helped me articulate the ways in which selection processes drove my data collection and analysis and ultimately the conclusions that I was able to draw from the experience.

Analysis across stories

There are obvious differences in our researcher roles and how we entered into the critical dialogue cycle of planning, adjusting, and countering. Even so, analysis of our group dialogue and personal narratives identified persistent themes that bridged our individual experiences and cut across our socialization as scholars and development of researcher identities. At the core of our stories, we noticed a constant interface—and often a struggle— between our personal situations and social agendas as qualitative researchers. We refer to this interface as Reporting In/Reporting Out, recognizing that the two are not mutually exclusive. In the following, we examine this interface through four subthemes: trusting qualitative research, dealing with powerful others, building the story, and accepting unintended consequences.

Trusting qualitative research. In our discussions of selection experiences, we noticed two questions kept coming to mind as we made our decisions about who and what to include in our studies. The first was, ‘Do you trust my methods?’ The second, ‘Do I trust my methods?’ The first question is reminiscent of the paradigm wars cited by Oakley (1999), Alexander (2006), and others. The second question points to an internalized commitment to ‘reporting in’, to a combined consciousness and critical deliberation of methods.

In a thoughtful and distinctive critique of her selection choices as a novice researcher, Freeman (2000) admitted that the selection process was not as simple as she expected. ‘Before I could access what it is I wanted to know, I had to first figure out what I meant by know, who I thought should do the knowing, and where I thought this knowledge could be found’ (p. 360). Clearly, Freeman’s understanding of the selection process went far beyond the conventional discussions of access or technical validity. She was con- scious of selection, in and of itself, as meaning making.

Trust in one’s methods, personal or public, was conceptualized differently by each of us. Earle emphasized doctoral program socialization and training:

I was never taught [selection]. No one really dogged me on how I chose those people for my dissertation, [and] no-one has held me accountable for how I select. This had to come out of my teaching other people this is important, and me being frustrated that [others] weren’t giving me the answers that I needed. I had to force it ... . I’ve been thinking about it across my studies; I’ve had to attend to it in my study ... I reported out the characteristics, but I never thought forward

710 Qualitative Research 13(6)

about how that would affect my future research trajectory. I have this great data now, but it can’t be what it wasn’t.

Jill suggested that the question of trusting method is more about socialization to the method itself:

I was listening to that and I was just thinking [as a research methods person] sometimes more quantitatively-minded people get more caught up in the methods because [knowing the methods allows them to] replicate it later. Qualitatively-minded people are more interested in the discussion and the conclusions [than focusing so much on the ability to replicate the methods].

Stacia took the stance that all researchers ‘are concerned with producing quality research, and we are concerned with being reflective about our process in ways that help others understand and with being transparent ... We just go into comparison mode because that’s human nature’. Jill reminded the group that ‘we’re making an argument that’s valid for both qualitative and quantitative research. Maybe that’s why the comparison is coming out’. Trusting your methods is worthy of critique across ‘all types of research’. Stacia agreed, critiquing the concept of ‘capturing the participant’s true story’. Rather, ‘what people fail to realize is there’s selection in the quantitative research, as well, [in spite of] this assumption that when you present those results they’re “real”’.

Building the story. Qualitative researchers are reminded to design their methods around their research questions; on the other hand, those same researchers are prompted to adapt their methods to the unfolding inquiry. The resulting research story rarely accounts for these changes, emphasizing only the product of those changes. Reflecting on our own experiences, we had struggled to balance this ‘telling the story’ of the research process with ‘building the story’ of our research choices. Selection choices might be chosen by funding sources or lead researchers, for example, or initial choices might give way to emerging research questions or changing researcher interests. Finally, acknowledging serendipity might seem at cross purposes with planning quality research.

Stacia as a doctoral student preparing to begin her dissertation, experienced a near impasse between the need to complete a well-designed proposal within the time frame of the dissertation proposal course and her own desire to honor the qualitative research maxim of emergent choices. Pondering how she might resolve this dilemma, Stacia questioned whether research questions should always decide selection choices, espe- cially in teacher research:

What’s really interesting is that [in teacher research] your research question and methods are tied to who your participants are. For me it always started with the participants, and the questions emerged from that setting [which is why I’ve had such a hard time writing my dissertation proposal]. I can’t have a question and I can’t know how I’m gonna proceed if I don’t know who these people are.

Stacia’s dilemma sparked considerable discussion about the linearity of research propos- als. For example, Earle connected Stacia’s situation to how qualitative research is taught.

Reybold et al. 711

While Earle describes herself as a ‘critical constructivist’, she admitted that she has been ‘more linear’ in her teaching of qualitative research. ‘I’m really seeing this issue very differently and wondering how I can incorporate that into my own teaching of this, and my own thinking of the questions emerging from the context’. Earle was ‘surprised’ at her initial overly simplistic understanding of Stacia’s predicament, especially as a teacher who advocates qualitative research as iterative and creative endeavor. ‘You really didn’t know what your questions were’. Earle had understood Stacia’s predicament ‘cogni- tively’ as a matter of question refinement, not in practical terms. ‘It just sunk in’, she said, that Stacia’s dilemma was fundamental not just technical.

Jill also reacted to our dialogue about the disaggregation of selection and results. The reality of research, Jill said, is that research questions are embedded in the context of study.

There’s no way you [can study] that ‘what’ without that group, that context. ... You may [study] it differently with a different group in a different context and it might [be] similar, but not the same. But many people don’t really think that.

Rather, many assume an uncritical or even naïve position toward selection: ‘Oh well if you can’t do it with this group, you do it with that group and you’ll get the same idea— you’ll get roughly the same results’.

While this conversation documented our struggle to relate our process to others, we also uncovered yet another aspect of building the research story: public identity as a scholar. Ultimately, decisions regarding who is studied can potentially shape who we become as researchers and our expertise. For example, Earle said her selection choices had resulted in more attention to issues of sex and gender, ethnicity, and race—and bias, in general—in higher education. Midway through her longitudinal study of academic identity, she critiqued her selection choices as biased toward the voices of white women. Purposefully, she chose to change her selection criteria to listen more closely to stories of difference. ‘What I’m finding out is that I’m now being labeled [an expert] according to the selection choices I’ve made along the way’, she said. This labeling subsequently led to speaking engagements on these topics, leading Earle to focus even more research attention on these areas. ‘My entire research agenda is now developing around this selec- tion choice that I began making. It has shut down some opportunities and opened up others’.

Stacia questioned the irony of being considered an expert on any topic due to selec- tion choices: ‘Because I take the time to recognize my selection process, and what that means, that then becomes the center [of my expertise]’. Jill agreed, adding,

[It’s interesting that when you pay] attention to issues of race, class, gender in selection, there are those who would want to make that the research itself. That then goes back to the idea that selection doesn’t really matter. It’s like, If you pay attention to those issues, then all of a sudden those issues [become] the purpose of the research.

Still, Stacia acknowledged the relationship between selection choices and recognition of expertise, especially when marketing yourself as an academic:

712 Qualitative Research 13(6)

As a doc student—I’m not sure where I picked up this idea—I think that we do recognize that whatever direction we go will inevitably define who we are as researchers. As I was planning for this dissertation [I realized] if I choose one group [or] topic, it defines me.

Dealing with powerful others. That our discussion turned to power is not surprising. Each of us is trained in the literature of critical theory, and our academic lives capture the ten- sions across research design and powerful others. These tensions included faculty senior- ity, status as graduate assistants, the inordinate emphasis on ‘numbers’ in some disciplines, the impact of funding on results (especially when those results are expected in order for funding to continue), the inability of participants to shape their stories, and the influence of university structures and the faculty rewards. These controls on our selection choices mirrored those imposed in everyday professional life. While some readers might con- sider these issues as background noise to be controlled in research planning, we found the influence of powerful others to be part of the larger discourse related to our research.

Stacia’s struggle to choose a research question highlights the power of the dissertation committee to shape a research agenda, even beyond the dissertation itself. ‘I’m immedi- ately thinking about going on job interviews. I have to think about what I’m qualified to do’. Her selection choices have broadened her expertise, and ‘I can market myself in the literacy field, not just in the teacher professional development field. The ‘who’ is chang- ing my marketable skills, my knowledge’. This influence of institutional and relational structures on student and faculty research practice is well documented in higher educa- tion literature, especially in studies that focus on faculty socialization and management of faculty rewards (e.g. Lindholm, 2003, 2004).

Accepting unintended consequences. Reflexivity is the gold standard of quality for qualita- tive researchers en masse. The three of us have been quite socialized—at least in the abstract—to the ‘shoulds’ of data collection and analysis, and at the core of our reflexiv- ity is the awareness of fallibility. Researchers make mistakes, and many a confessional tale deals with the consequences of those errors.

As we explored the impact of these selection choices on our researcher agendas and identities, we came to realize the unintended consequences of these choices. For exam- ple, Jill’s frustration with the ‘research’ process she participated in led her to change her PhD concentration to research methods so that she could influence future research endeavors as an ‘expert’ in the field. ‘So I hadn’t really done a whole lot of thinking about it, other than I wanted to get more people to interview from different groups, rather than just the ones that they were picking’. Jill has spent considerable time reflecting back on her study and wonders ‘how our results would have changed had we made different selection choices’. There are specific selection choices she recognizes in hindsight:

What if we had simply conducted individual rather than group interviews? Did the group dynamic make people more or less likely to speak ‘openly’ and ‘honestly?’ Would those occasional critiques of the program have occurred, or might they have been even more strident? What if we had chosen communities in more remote areas? We would have learned about how the program works (or does not work) in areas lacking easy access to large cities. If our

Reybold et al. 713

participants had come from communities that struggled during program implementation, as well as those who did not, we could have identified certain elements that facilitated or hindered implementation success ...

The list could go on ... If the study had looked at how these successful communities achieved what they did—rather than simply asking them about their experience as if it were typical—the results might have been more robust.

In Earle’s case, challenging her early participant choices led to a more aggressive selec- tion strategy to include more diverse experiences in her study, and this led to speaking and writing opportunities about women and minority faculty experiences. These oppor- tunities, in turn, demanded more time and research energy. The narrowing of her scholar- ship was unexpected but satisfying. The participants and findings, she said, ‘kind of morphed over the years. The study became its own creature ... connecting with certain people at certain times in my own career’. She characterized that particular study as ‘organic and evolving’ in that ‘the findings were very much following the trajectory of my own development and my choices’.

Conclusion

Our analysis extends the tradition of critiquing ‘insider acts of how the research evolved and developed: how access and relationships were negotiated and managed; what went wrong; what was rewarding or challenging’ (Coffey, 1999: 122). We maintain that the product of a study must be consciously attentive to the relationship between selection choices and results. There must be an accounting for the relationship between the researcher and those chosen to represent a reality and we argue that begins with acknowl- edging the tension between ‘reporting in’ and ‘reporting out’ our methods.

The results of a study are not so much found as they are constructed through researcher choice and interpretation. Researchers, then, should examine and reveal the complexity of their selection process in order to satisfy their readers’—and their own— trust in qualitative research methods. To accomplish this, discussions of selection choices and changes need to go beyond a discrete listing of criteria or description of participants. While these constructs can help researchers think more clearly about their options and enable writers to communicate their choices to the reader, describing who your participants are is quite different from explaining how you chose those participants (Fine, 1994; Freeman, 2000).

Researchers also should acknowledge that the practice of research is subject to the expectations of ‘powerful others’ who exert structural control over the research process in various arenas. Our selection narratives in this study identified several of these ‘powerful others’ and their influence over the course of our research, including university professors, editorial boards, grant funders, uncritical readers, and even peers.

In too many research reports, one selection choice seems to fold into the next with such ease that, when all is said and done, the pattern of selection seems simple and concrete. Many authors offer a nod to criteria, but they rarely disclose the muddy and messy decisions they made as they were ‘building the story’.

714 Qualitative Research 13(6)

In all selection stories, there is a fragile balance between insight and apology, and researchers also must navigate the limitations of journal space, audience reception, and confidentiality. We must ‘accept unintended consequences’ of our inquiry, whether those consequences are opportunities or disappointments to our original research goals.

Our goal here is to remind readers—and ourselves—that selection is complex. It is a disservice to qualitative study when it is taken for granted or even ignored as a method of data collection and analysis. Instead, as Bentz and Shapiro (1998) said, ‘choosing a research method necessarily requires one to make a conscious choice about the assump- tions underlying the inquiry. One must, in other words, take responsibility for one’s approach and its consequences’ (p. 5).

Funding

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

References

Alexander HA (2006) A view from somewhere: explaining the paradigms of educational research. Journal of Philosophy of Education 40(20): 205–221.

Bentz VM and Shapiro JJ (1998) Mindful Inquiry in Social Research. London: Sage. Brewer J and Hunter A (2006) Foundations of Multimethod Research: Synthesizing Styles.

Thousand Oaks, CA: Sage. Coffey A (1999) The Ethnographic Self: Fieldwork and the Representation of Identity. London:

Sage. Corbin JM and Strauss AL (2008) Basics of Qualitative Research: Techniques and Procedures for

Developing Grounded Theory (3rd Edition). Thousand Oaks, CA: Sage. Creswell JW (2002) Educational Research: Planning, Conducting, and Evaluating Quantitative

and Qualitative Research. Upper Saddle River, NJ: Merrill. Denzin NK and Lincoln YS (2005) Introduction: the discipline and practice of qualitative research.

In: Denzin NK and Lincoln YS (eds) The SAGE Handbook of Qualitative Research. Thousand Oaks, CA: Sage, 1–32.

Fine M (1994) Working the hyphens: reinventing self and other in qualitative research. In: Denzin NK and Lincoln YS (eds) The SAGE Handbook of Qualitative Research. Thousand Oaks, CA: Sage, 70–82.

Freeman M (2000) Knocking on doors: on constructing culture. Qualitative Inquiry 6: 59–369. Freeman M, DeMarrais K, Preissle J, et al. (2007) Standards of evidence in qualitative research: an

incitement to discourse. Educational Researcher 36(1): 25–32. Glesne C (2006) Becoming Qualitative Researchers: An Introduction (3rd Edition). Boston:

Pearson-Allyn & Bacon. Gubrium E and Koro-Ljungberg M (2007) Contending with border making in the social

constructionist interview. Qualitative Inquiry 11: 689–715. Heshusius L (1994) Freeing ourselves from objectivity: managing subjectivity or turning toward a

participatory mode of consciousness? Educational Researcher 23(3): 15–22. Lawless EJ (1992) ‘I was afraid someone like you ... an outsider ... would misunderstand’:

negotiating interpretive differences between ethnographers and subjects. Journal of American Folklore 105: 302–314.

Lindholm JA (2003) Perceived organizational fit: nurturing the minds, hearts, and personal ambitions of university faculty. Review of Higher Education 27: 125–149.

Reybold et al. 715

Lindholm JA (2004) Pathways to the professoriate: the role of self, others, and environment in shaping academic career aspirations. Journal of Higher Education 75: 603–635.

Maxwell JA (2005) Qualitative Research Design: An Interactive Approach (2nd Edition). Thousand Oaks, CA: Sage.

Oakley A (1999) Paradigm wars: some thoughts on a personal and public trajectory. International Journal of Social Research Methodology 2(3): 247–254.

Oakley A (2003) Interviewing women: a contradiction in terms. In: Lincoln YS and Denzin NK (eds) Turning Points in Qualitative Research: Tying Knots in a Handkerchief. Walnut Creek, CA: AltaMira Press, 243–263.

Patton MQ (2002) Qualitative Evaluation and Research Methods (3rd Edition). Newbury Park, CA: Sage.

Peshkin A (1988) In search of subjectivity—one’s own. Educational Researcher 17(7): 17–21. Peshkin A (2000) The nature of interpretation in qualitative research. Educational Researcher

29(9): 5–9. Peshkin A (2001) Angles of vision: enhancing perception in qualitative research. Qualitative

Inquiry 7(2): 238–253. Reinharz S (1997) Who am I? The need for a variety of selves in the field. In: Hertz R (ed.)

Reflexivity and Voice. Thousand Oaks, CA: Sage, 3–20. Reybold LE (2003) Pathways to the professorate: The development of faculty identity in Edu-

cation. Journal of Innovative Higher Education 27(4): 235–252. Reybold LE (2008) The social and political structuring of faculty ethicality in education. The

Journal of Innovative Higher Education 32(5): 279–295. Reybold LE, Brazer SD, Schrum L, and Corda K (2012) The politics of dissertation advising:

How early career women faculty negotiate access and participation. Journal of Innovative Higher Education 37(3): 227–242

Sandelowski M (1999) Focus on qualitative methods: time and qualitative research. Research in Nursing & Health 22: 79–87.

Sobal J (2001) Sample extensiveness in qualitative nutrition education research. Journal of Nutrition Education 33(4): 184–192.

Stribling SM (2010) Supporting young children’s development as critically literate members of democratic communities: Stories of change and challenge. (Doctoral dissertation). Retrieved from ProQuest Dissertations & Theses (3421709).

Visweswaran K (2003) Defining feminist ethnography. In: Lincoln YS and Denzin NK (eds) Turning Points in Qualitative Research: Tying Knots in a Handkerchief. Walnut Creek, CA: AltaMira Press, 73–94.

Author biographies

L Earle Reybold is an Associate Professor of qualitative research methods in the Graduate School of Education at George Mason University. She received her PhD in adult education at the University of Georgia, where she studied adult learning and development. Her research interests include aca- demic identity and qualitative research methods, and her most recent studies explore faculty epis- temology, professional ethics in higher education, critical dimensions of academic culture, meth- odological quality, and the politics of `alternative’ research design.

Dr Lammert is a research methodologist with 13 years of experience designing and conducting research and evaluation projects in the field of education and human development. In her current position as Senior Study Director at Westat, she is providing technical assistance in program evaluation to US Department of Education (USED) Office of Special Education grantees and col- laborating on the development of studies to examine implementation of USED’s Elementary and

716 Qualitative Research 13(6)

Secondary Education Act Title I & II initiatives and to investigate the role of the American Recovery and Reinvestment Act in education reform efforts.

Stacia Stribling is an Assistant Professor in the Graduate School of Education at George Mason University. Her research interests include early childhood critical literacy development, culturally relevant pedagogy, and teacher professional development. As a former public school teacher, Stacia advocates for teacher research as an essential component of teacher professional develop- ment; she serves on the National Association for the Education of Young Children’s (NAEYC) Teacher Research Steering Committee and was the Secretary/Treasurer of the Teacher as Researcher Special Interest Group of the American Educational Research Association (AERA).