Week 8 - Assignment 1: Research Ethics and Article Critique and Week 8 - Assignment 2: Complete CITI Certification

QualitativeResearchQuality: ACollaborative Inquiry AcrossMultipleMethodological Perspectives

JoachimWalther,a NicolaW. Sochacka,a LisaC.Benson,b

AmyE.Bumbaco,c NadiaKellam,d

AliceL.Pawley,e and CanekM.L.Phillipse

aUniversityofGeorgia, bClemsonUniversity, cUniversityof Florida, dArizonaStateUniversity, ePurdueUniversity

Abstract Background The field of engineering education research is adopting an increasingly diverse range of qualitative methods. These developments necessitate a coherent language and con- ceptual framework to critically engage with questions of qualitative research quality.

Purpose/Hypothesis This article advances discussions of qualitative research quality through sharing and analyzing a methodologically diverse, practice-based exploration of research quality in the context of five engineering education research studies.

Design/Method As a group of seven engineering education researchers, we drew on the col- laborative inquiry method to systematically examine questions of qualitative research quality in our everyday research practice. We used a process-based, theoretical framework for research quality as the anchor for these explorations.

Results We constructed five practice explorations spanning grounded theory, interpretative phenomenological analysis, and various forms of narrative inquiry. Examining the individual contributions as a whole yielded four key insights: quality challenges require examination from multiple theoretical lenses; questions of research quality are implicitly infused in research practice; research quality extends beyond the objects, procedures, and products of research to concern the human context and local research setting; and research quality lies at the heart of introducing novices to interpretive research.

Conclusions This study demonstrates the potential and further need for the engineering edu- cation community to advance methodological theory through purposeful and reflective engage- ment in research practice across the diverse methodological approaches currently being adopted.

Keywords research methods; qualitative; research evaluation criteria; research quality; collaborative inquiry

Introduction The engineering education research community is embracing a diverse range of qualitative methods of inquiry (Case & Light, 2011; Douglas, Koro-Ljungberg, & Borrego, 2010). With their focus on rich descriptions of lived experiences and perspectives, these approaches

Journal of Engineering Education VC 2017 ASEE. http://wileyonlinelibrary.com/journal/jee July 2017, Vol. 106, No. 3, pp. 398–430 DOI 10.1002/jee.20170

are inherently suited to addressing key areas of the expanding research agenda (Johri & Olds, 2014), such as underrepresentation in engineering and the profoundly human aspects of engi- neering students’ professional formation (National Science Foundation, 2016). As more engi- neering education researchers adopt qualitative methods of inquiry, the field faces the interconnected challenges of how to conceptualize, address, and articulate issues of research quality across the varied approaches and their respective intellectual traditions. Building on prior efforts to advance engineering education research as a rigorous discipline (Borrego, 2007b; Lohmann, 2008; Streveler & Smith, 2006), Walther, Sochacka, and Kellam’s (2013a) theoretical framework for research quality offered a conceptual basis and shared language to foster a coherent discourse around qualitative research quality across a range of interpretive methods of inquiry.

The present article is situated in a continued process of adopting and adapting the pro- posed quality framework through collaborative engagement with the research community (Walther & Sochacka, 2014a, 2015). The project described here emerged from a small group of scholars that formed at a workshop on qualitative research quality (Walther, Sochacka, & Kellam, 2013b), where the co-authors were inspired to apply and collectively explore the quality framework across the diverse methodological traditions of their own research programs. Driven by what was experienced as an “important or compelling concern for all participants” (Bray, Lee, Smith, & Yorks, 2000, pp. 65–66), we framed our subsequent practice-based investigation of research quality as a collaborative inquiry (Alvesson & Sk€oldberg, 2009; Bray et al., 2000; Heron & Reason, 1997; Love, 2008; Reason, 1988; Reason & Bradbury, 2007; Yorks & Kasl, 2002), a method that focuses on generating transferable knowledge claims from the shared, systematic exploration of one’s own experience. In this article, we present five practice explorations that we constructed during our two-year collaborative encounter, each of which illustrates key features from our shared discussions. Looking across these individual contributions, we derive a number of broader insights concerning interpretive research quality. Our goals for this project were threefold.

To advance research practice Through the collaborative application and exploration of the quality framework in our own research projects, we sought to further the ways we engage with challenges and strategies around research quality in our work and, at the same time, advance a pragmatically grounded, working understanding of the prior quality framework.

To advance methodological theory Through our systematic focus on generating trans- ferable insights from a shared exploration of our own experiences, we sought to advance the theoretical discourse around interpretive methodology in engineering edu- cation research.

To share lived experience of research practice The above goals and the affordances of the collaborative inquiry allow us to share, through purposefully structured accounts, some of the complexity and richness that characterize everyday research, a practice that is uncommon for engineering education research and, at the same time, is under- stood to crucially complement, ground, and enrich theoretical advances of methodol- ogy (Barbour, 2003; Lather, 1986; Miller & Dingwall, 1997; Smeyers, 2005; St. Pierre, 2014).

Qualitative Research Quality: A Collaborative Inquiry 399

Prior Literature The following situates the present work in the context of the ongoing formative discourse con- cerning the emergence of engineering education research as a field and its accepted research methods and practices. Broader discussions of qualitative research quality in related disciplines provided another anchor point for our project and are represented below in a necessarily con- densed form that focuses on key features of the discussion. For a more thorough treatment of prior work on qualitative research quality in other fields, see Walther et al.’s (2013a) presenta- tion of the initial quality framework that emerged out of a synthesis of this body of thought.

Throughout the development of engineering education as a research discipline, scholars have adopted, and continue to explore, a wide range of qualitative research methods (Case & Light, 2011; Koro-Ljungberg & Douglas, 2008; Leydens, Moskal, & Pavelich, 2004). In parallel, explicit and intentional efforts to promote and shape the discipline (Borrego, 2007b; Fortenberry, 2006; Haghighi, 2005; Jamieson & Lohmann, 2009; Radcliffe, 2006; Wormley, 2006) have, among other aspects, focused on the issue of rigor in engineering education research (Borrego, 2007a; Streveler & Smith, 2006). Streveler, Borrego, and Smith (2007), for example, argued that “in order to increase the rigor of engineering education research, engineering practitioners need . . . to learn the literature, methods, and paradigms of educational research” (p. 142). Accord- ingly, other scholars have directed their attention to clarifying the epistemological bases of qualitative research (Douglas et al., 2010) with the aim of fostering a more coherent discourse across the different intellectual traditions that inform the discipline. Borrego (2007a), for example, examined and sought to clarify the “fundamental differences that prevent [the] application of traditional engineering standards of rigor directly to engineering education research” (p. 91). Building on these efforts, scholars have more recently called for inclusive consider- ations of research quality across various methodological traditions (Baillie & Douglas, 2014).

Across other fields that employ qualitative forms of inquiry, there is “considerable con- troversy” (Anfara, Brown, & Mangione, 2002, p. 28) regarding what defines rigorous or trustworthy qualitative research (Flick, 2006). In the area of educational research, in particu- lar, perspectives on qualitative research quality have been described as especially “fragmented” (Moss et al., 2009, p. 502). Addressing this challenge, scholars have proposed conceptions of research quality along the distinct lines of research quality guidelines (Eakin & Mykhalov- skiy, 2003; Elliott, Fischer, & Rennie, 1999; Lather, 1986; Malterud, 2001), quality criteria (Bernhard & Baillie, 2012; Lincoln, 1995; Morrow, 2005; O’Brien, Harris, Beckman, Reed, & Cook, 2014; Tong, Sainsbury, & Craig, 2007; Tracy, 2010), and lists of specific research procedures to ensure research quality (Mays & Pope, 2000). In parallel with these efforts, longstanding and repeated suggestions from within the qualitative community have pointed to the need for process-oriented approaches to research quality (Cho & Trent, 2006; Flick, 2006, 2007, 2008, 2009; Lather, 1986; Reason & Rowan, 1981). The research quality frame- work that formed the departure point for this collaborative inquiry synthesized prior work on quality criteria and proposed a process-oriented model that maps five fundamental validation constructs and the notion of process reliability across the progression of a qualitative inquiry (Walther, Pawley, & Sochacka, 2015; Walther et al., 2013a).

Theoretical Framework Through a focus on process, rather than guidelines, criteria, or specific procedures, the goal of Walther et al.’s (2013a) framework was to shift “the attention from defining standards of

400 Walther, Sochacka, Benson, Bumbaco, Kellam, Pawley, & Phillips

rigor applied to the research results to viewing, demonstrating, and assessing research quality throughout the entire research process” (p. 638). In other words, the framework aimed to provide a “flexible, inclusive, and contextual view of research quality” (p. 628) to enable engi- neering education researchers to systematically engage with questions of research quality from a project’s inception through to publication and beyond, locate and explicate the specific func- tion of quality considerations or strategies in their contribution to overall research quality, and discuss these considerations in a coherent manner with the wider engineering education research community. To this end, the theoretical proposal adopted the use of the term “validation” over “validity” to draw attention to a sustained, in-process commitment to research quality and, at the same time, discourage positivist inclinations to make definitive statements regarding the absolute and objective “validity” of specific findings (see also Angen, 2000; Koro-Ljungberg, 2008; Mishler, 1990).

Since its publication in the Journal of Engineering Education in 2013, Walther et al.’s quality framework has undergone two important expansions, both of which emerged in response to vari- ous and extended engagements with the engineering education research community (Walther & Sochacka, 2014a, 2014b; Walther, Sochacka, & Kellam, 2012; Walther et al., 2013b). One of these expansions entailed the addition of a sixth construct, ethical validation, which emerged in the context of a collaborative inquiry that took place in parallel with the one described in this arti- cle (see Walther et al., 2015). The second expansion to the original theoretical proposal emerged in response to requests from engineering education researchers for a practice-based version of the framework (Walther & Sochacka, 2014a, see critical probing questions in Figure 1). The

Figure 1 Practice-oriented version of the quality framework that illustrates the quality constructs through probing questions (from Walther & Sochacka, 2014a).

Qualitative Research Quality: A Collaborative Inquiry 401

remaining paragraphs of this section provide a brief overview of key features of the framework. Walther et al. (2015), Walther and Sochacka (2014a), and Walther et al. (2013a) provide further discussion.

The quality framework comprises two dimensions: a process model, which spans making and handling data as two broad stages of the qualitative inquiry; and a typology of theoretical, procedural, communicative, pragmatic, and ethical validation, as well as process reliability. The stages of making and handling data are intended to broadly represent the qualitative inquiry process across various methodological approaches. It is important to emphasize that this view is not intended to imply a sharp distinction between the two stages, nor to neglect either the iterative nature of interpretive studies, or the different focuses that various method- ologies place on various phases. Figure 1 presents the process model and validation and reli- ability constructs in the form of a practice-based model that operationalizes the above two dimensions through critical probing questions. These probing questions are designed to cap- ture facets of each validation and reliability construct for different stages of the inquiry and are thus intended to guide a reflective exploration of research quality throughout the entire process from making to handling data. In line with St. Pierre’s (2014) call to “engage the ontological” in qualitative research, this representation of the quality framework starts with the ontological question, “What is, in terms of scope and nature, the specific social reality we want to investigate?” In this way, the framework is designed to provide a methodologically unencumbered “way in” for novice qualitative engineering education researchers to begin to explore their question, or phenomenon of interest, while, at the same time, creating a space for researchers of all levels of expertise and backgrounds to uncover the epistemological, theo- retical, ethical, and other implications of their study.

The following paragraphs briefly describe the five validation constructs and process reli- ability, which compose the typology dimension of the quality framework.

Theoretical validation focuses on the ontological nature of the social reality under inves- tigation and the fit between this social reality (both in its complexity and coherence) and the theory, or findings, generated. It is important to point out here that this fit does not imply that there is one “right” way to interpret or represent a particular social reality. Some interpre- tations, however, may be more right than others. As Kirk and Miller (1986, p. 11) explained, “There is a world of empirical reality out there. The way we perceive and understand that world is largely up to us, but the world does not tolerate all understandings of it equally” (see also Amedeo, Golledge, & Stimson, 2009; Silverman, 2009).

Procedural validation concerns features or elements that researchers incorporate into the research design (e.g., the use of theoretical and methodological frameworks; decisions concerning data selection, collection, and analysis, etc.) to inherently improve the fit between the reality studied and the theory, or findings, generated. Such features are intended to contribute to the efforts of the researcher in making data to “see” an appropri- ate universe of participants’ experiences (e.g., extreme sampling) or, in handling data, to employ methods that explicitly maintain the grounding of interpretations in the data (e.g., the constant comparative method).

Communicative validation concerns the integrity of the interlocking processes of social construction, or shared meaning-making, between relevant communication communities (Apel, 1972) at different stages of the inquiry. For example, the making data phase focuses on the researchers’ and participants’ co-construction of the latter’s “experience-near” constructs (Geertz, 1974, p. 28), while in the handling data phase, this focus shifts to the systematic shared meaning-making processes that occur within research teams to generate the

402 Walther, Sochacka, Benson, Bumbaco, Kellam, Pawley, & Phillips

“experience-distant” constructs that are subsequently integrated into the meaning conventions of the research community.

Pragmatic validation concerns the compatibility of theoretical constructs with empirical reality in terms of resonance, understanding, explanation, or utility. These theoretical con- structs include both those that are brought into qualitative studies and those generated in the form of research findings. Accordingly, here the term “empirical reality” refers to both the original research setting and to similar settings in which the research findings might be trans- ferred or have implications.

Ethical validation focuses on the profoundly human aspects of qualitative research that are not captured in conventional considerations of ethics, such as institutional review board approvals that frame ethical considerations as external to the research process (Christians, 2011). The notion of ethical validation extends conceptions of non-dualistic ontology, such as bias and mutual influence between the researcher and the researched, to conceptualize research as entering a community with research participants, co-researchers, and, ultimately, with the “customers” of the research (Walther et al., 2015). Ethical validation can thus assist researchers in addressing “process ethics” (Ellis, 2016, p. 435) or “ethics in practice” (Guillemin & Gillam, 2004), an effort that is not only important from the perspective of ful- filling ethical responsibilities toward research participants but, as we have argued elsewhere (Walther et al., 2015), is also inextricably linked to producing high-quality interpretive research findings.

Process reliability concerns the consistency and mitigation of random influences on the research process. This quality construct does not imply striving for the replicability of research out- comes, but rather the conscious consideration of arbitrary influences that can be mitigated through purposeful and reflective procedures. In the making data phase, process reliability focuses on strate- gies to support the collecting and recording of the data in a dependable way; in the handling data phase, the focus shifts to the definition and documentation of interpretation procedures.

Methodology and Approach Although primarily set in the context of the above-described methodological discourse in the engineering education research community, this study also responds to longstanding and re-emerging calls for qualitative researchers to systematically examine their own practice (Barbour, 2003; Lather, 1986; Miller & Dingwall, 1997; Smeyers, 2005; St. Pierre, 2014). Beyond improving individual research practice, such efforts have also been proposed as a way to contribute to the theoretical understandings that underpin qualitative research, particularly with a view to advancing our collective understandings around qualitative research quality (Lather, 1986; Moss et al., 2009). As Barbour (2003) explained,

Our own everyday research practice, involving teaching and supervision, provides an opportunity to advance debates about rigor by allowing us [as qualitative researchers] to formulate a considered response that respects both the complexities and the unique contribution that qualitative research can make. (p. 1019)

An exploration of methodological theory that is grounded in everyday research practice also has the potential to overcome the problematic separation of methodology and research practice where, as Melia (1997) argued, “methods debates, [as they] have become more philo- sophical, or at least epistemological . . . have become less useful for the doing of research” (p. 35). Further elaborating on the benefits of a practice-based exploration of methodological

Qualitative Research Quality: A Collaborative Inquiry 403

theory, Alvesson (2009) suggested that such efforts can lead research to be more “reflective and creative due to a better interaction between the philosophical theoretical ideas and the empirical-practical sources of inspiration” (pp. 10–11). Critically examining and explicitly sharing the richness, or inevitable messiness, of our everyday research also serves to illustrate the considerations underlying methodology or research design decisions that contribute to overall research quality. In a trend that Barbour (2003) termed “technical essentialism,” such considerations often remain hidden behind the labels we apply to these complex and situated research strategies.

Leveraging these opportunities for systematically examining “our own backyard” (Barbour, 2003, p. 1025) of actual, implemented research practice, we used the collaborative inquiry method (Bray et al., 2000; Heron & Reason, 1997; Love, 2008; Reason & Bradbury, 2007; Yorks & Kasl, 2002) to frame our systematic process of shared practice, exploration, and meaning-making.

Collaborative Inquiry The collaborative inquiry approach combines elements from the intellectual traditions of action research and participatory forms of inquiry (Bray et al., 2000; Yorks & Kasl, 2002) in that it offers a “systematic structure for learning from experience” (Yorks & Kasl, 2002, p. 3). Like action research, collaborative inquiry combines efforts to improve a practice context with systematic ways of generating transferable knowledge claims. Unlike action research, collabo- rative inquiries typically take place in purposefully created professional learning communities, in contrast to action research, which is typically conducted in natural settings (Brydon-Miller, Greenwood, & Maguire, 2003). As Kasl (personal communication in Bray et al., 2000, p. 7) explained, while “an action research team goes out to collect data from someone else,” collab- orative inquiry promotes change in, and generates research findings from, a researcher’s own experience, thus “creating new knowledge drawn systematically from the life experiences of persons most centrally involved in the context of the inquiry” (Yorks & Kasl, 2002, p. 4). In the tradition of participatory forms of research, collaborative inquiry thus redefines the research “subjects” as active participants and co-investigators.

Conceiving of practice as situated in a community of practitioners, collaborative inquiry is based on shared engagement and explorations whereby “together, inquirers formulate a com- pelling question that they can answer by examining ‘data’ from their personal experience” (Yorks & Kasl, 2002, p. 5). More specifically, participants engage in a systematic, iterative process that emphasizes the productive interplay between the following elements: engaging in practice, reflection and exchange with others about significant experiences, the generation of abstract knowledge to inform and improve practice, and the subsequent implementation of the theoretical knowledge in their own practice (Bray et al., 2000). This focus on shared exploration and meaning-making in the context of action makes this approach particularly suited to “topics that are professionally developmental” (Yorks & Kasl, 2002, p. 3). In the fol- lowing Shared Exploration section, we describe how these repeated cycles of practice, reflec- tion, exchange, and the generation of abstract knowledge manifested in our study.

Shared Exploration The progression of our collaborative inquiry spanned a timeframe of two years and is described here with a view toward providing an authentic sense of our lived process of shared exploration. The collaborative inquiry group consisted of necessarily self-selected members who saw utility in exploring the topic of qualitative research quality. The group included the original authors of the framework, Walther, Sochacka, and Kellam, who assumed roles as

404 Walther, Sochacka, Benson, Bumbaco, Kellam, Pawley, & Phillips

both facilitators (Walther and Sochacka) and full participants in the collaborative inquiry in order to expose their research projects to the same degree of scrutiny that was being asked of the other participants. This configuration reflected the necessary commitment to the sus- tained engagement required for a collaborative inquiry but also posed challenges of bias in the process. To mitigate this risk, we designed our process to ensure that insights from the study were “well founded on . . . the experiences of [all of] the co-researchers as co-subjects” (Rea- son, 1988, p. 40). This quality principle of collaborative inquiry reflects the notion of theoret- ical validation discussed above and was achieved along two specific lines. First, the group purposefully engaged in robust discussions that were characterized by questioning, disagree- ment, and iteration, as described below – a way of engaging that reflects the notion of com- municative validation and lies at the core of quality in collaborative inquiry research. Second, the insights related to interpretive research quality were not only subjected to collective scru- tiny but also exposed to the research practice of the participants, where findings were “confirmed through agreement in action or use” (Reason, 1988, p. 42). In other words, as a group we critically sought resonance to indicate that the insight derived from the process actually resolved or illuminated the quality challenge that was the context for its genesis. Equivalent to the notion of pragmatic validation, Bray describes this aspect of collaborative inquiry as “repeated cycles of reflection and action . . . to represent valid knowledge and to have significant influence on their practice” (Bray et al., 2000, p. 1).

The shared practice explorations were facilitated in an iterative process of identifying criti- cal practice incidents that the participants judged to have a bearing on the quality of their research projects, individually and collectively analyzing the incidents through the lens of the quality framework, and exploring connections across the accounts with a view to identifying novel quality-related insights. The accounts and emerging analyses were created by the author teams and collaboratively edited in a shared file repository. At the second workshop (Walther & Sochacka, 2014b), one year after the formation of the group, each author team presented and discussed their practice incidents and preliminary analyses to the larger group of work- shop participants in a moderated “fishbowl” panel (Silberman, 1996). This process served to solicit feedback and communicatively validate emerging theoretical claims around, and in some cases beyond, the constructs in the quality framework.

Based on this feedback, the collaborative inquiry group further developed the five individual contributions through a structured peer review process whereby each draft was reviewed by two other members of the group. This effort resulted in substantial collective rewrites as the author teams and assigned reviewers worked together to achieve conceptual clarity and consen- sus, on both the local discussion level and with respect to the overarching methodological insights that emerged across the five contributions (see Discussion and Conclusions section).

Practice Explorations The five practice explorations we present in this section represent a range of methodologies, from a constructivist grounded theory study described by Bumbaco, to an interpretative phenomeno- logical analysis conducted by Benson, to variations of narrative studies shared by Pawley, Phillips, Kellam, Walther, and Sochacka. As illustrated in Figure 2, our shared explorations of the research quality challenges, questions, and strategies described in each of these contributions led us to examine numerous aspects of Walther et al.’s (2013a) quality framework.

Each practice exploration begins with a brief overview of the study context. Depending on the focus of the contribution, these introductory sections vary in their treatment of relevant

Qualitative Research Quality: A Collaborative Inquiry 405

theories and methods. The practice explorations then describe one to two field accounts, writ- ten by the researcher(s) responsible for each project, which are intended to provide the reader with a rich and authentic view of the lived research experience of the authors. The field accounts describe aspects or moments of the projects that the respective authors judged to be significant from a research quality perspective. Finally, in the analysis sections, we draw on language from the quality framework to articulate the quality insights that emerged in the course of our collaborative inquiry process.

In constructing the practice explorations, we endeavored to maintain a consistent style and structure across the contributions. However, given the diversity of studies in terms of meth- odological perspective, degree of completion, and involvement of others (e.g., undergraduate and graduate students or other faculty), some variation remains across the contributions. The use of “I” or “we” in the practice explorations applies to the author(s) of those sections. The use of “we” in all other sections of the article refers to the entire collaborative inquiry team.

The first practice exploration describes how Pawley and Phillips wrestled with the role of emotion and empathy in their qualitative study of underrepresented groups in engineering degree programs.

“Learning from Small Numbers”: The Role of Pragmatic Validation in a Narrative-Based Project

Alice L. Pawley and Canek M. L. Phillips

We are investigating research questions related to underrepresented engineering students’ experiences of engineering educational institutions as gendered, raced, and classed (Pawley, 2013; Pawley & Phillips, 2014). Our research is theoretically informed by D. E. Smith’s (1990, 2005) work on ruling relations and methodologically informed by decolonizing meth- odologies (L. T. Smith, 1999) and narrative analysis (Polkinghorne, 1995).

Figure 2 Practice exploration of the theoretical space presented by the quality framework.

406 Walther, Sochacka, Benson, Bumbaco, Kellam, Pawley, & Phillips

Over 20 years ago, Acker (1990) and others argued against claims that the idealized “abstract, bodiless” organizational worker is gender-neutral (p. 151). Instead, they claimed that the best person who can inhabit this role, and thus organizations, successfully “is actually a man, and it is the man’s body, its sexuality, minimal responsibility in procreation, and con- ventional control of emotions that pervades work and organizational processes” (Acker, 1990, p. 152). Dorothy Smith built on the concept of the idealized worker and gendered organiza- tions by theorizing about how the gendering of the organization comes about in actual day- to-day practice. Specifically, she articulated a theory of “ruling relations” (D. E. Smith, 1990, 2005), which are the operating procedures by which diverse social relations are coordinated. For example, students, staff, faculty, and different members of the nonuniversity community come together daily to buildings that we label as “university” buildings, to work in ways that we call “education”; this is the coordination of social relations, where people interact together to engage with a shared purpose. Ruling relations are all the “rules” that structure these social relations – from the expectation that meetings and classes occur primarily between 8 a.m. and 5 p.m. (for many schools), to how we know how to sit in a room for 50 minutes at a time and take notes on what the person at the front is saying, to the anticipated transaction between turning in assignments and receiving a grade, to the expectation that people designated as “engineering students” will work together in teams on gradable products but not “too closely” in ways that would raise concerns about “academic integrity,” to the exclusion of diaper- changing tables in bathrooms because it is assumed that the primary users are undergraduate students who are presumed not to have children. What makes these relations “ruling” is that they serve the interests of the organization over the people in the organization. We discuss this concept further elsewhere (Pawley, 2013; Pawley & Phillips, 2014).

While much work has subsequently been done in the context of gendered organizations, there are also parallel arguments regarding how organizations are raced (Acker, 2006). We understand engineering colleges to have been gendered and raced by virtue of white men hav- ing been the primary people defining the “operating procedures” within them for the length of their histories. In this research project, we seek to understand how these procedures oper- ate to (perhaps unintentionally) marginalize white women and people of color striving to learn and work within them.

In the following field accounts, we describe how Walther et al.’s (2013a) concept of prag- matic validation has informed the development of this project. In the following field accounts, the speaker’s pauses are represented as ellipses.

Experiencing “resonance” as a quality concern In the grant proposal that now funds this research, I (Pawley) spent a good deal of time theoretically grounding the research question and the interview-based methodology within the framework of rul- ing relations. However, after having collected some interviews, I started to second- guess myself. I had intentionally adopted the “ruling relations” theoretical framework because I found it to be persuasive, helpful, and largely absent from the engineering education research literature base. During the interviews, I used it to focus the direction and scope of my questioning. This approach was resulting in rich and emotionally powerful accounts of underrepresented students’ experiences.

For example, I vividly recall Moises, the second person I interviewed. Moises started his story with “four and a half years ago, I was a senior in high school and I washed dishes.” He talked about his dream of running cross country, how his parents structured his options after high school as either college or the military, how his sister

Qualitative Research Quality: A Collaborative Inquiry 407

helped him figure out that if he couldn’t run for a living he could do worse than major in engineering. He enrolled in college out of state, without really thinking too much about finances, and got himself into a pile of student debt. Academically, he also had a rocky start. While his grades ended up okay, he talked about how tough his junior year was, how he was working himself too hard. And then he told a story, in tears, about how he almost died – he hadn’t been sleeping enough, and he fell asleep driving through the mountains.

He said, “You try not to be religious and stuff but . . . The Man’s looking out for me . . . because through all the adversities and everything I went through . . . that was the factor, you know . . . just like . . . I would have been gone . . . that would have been it. You know . . . after everything I went through . . . and I survived that.”

Experiences such as these indicated to me that the project was “working,” in the sense of uncovering the range of challenges underrepresented students face in the course of pursuing their studies, but my doubts remained. I had wholeheartedly used the notion of ruling relations to elicit these stories. Was this a dishonest endeavor? Was I rigging the study to “succeed” no matter what, in explaining engineering education’s continuing homogeneity through the lens of ruling relations?

Making sense of emotion to improve research quality After having so deeply engaged with the participants in this study, I (Pawley) felt I needed to be sure to “do right” by them, to make sure that the explanatory power of their lived experiences was “used” as far as it could be, so that each participant’s effort and willingness to share with me, a white woman, was not wasted. I felt a sense of responsibility to do “justice” to their stories. When Phillips and I started the data analysis, though, we couldn’t help but notice that when we selected sections of the text as codeable “snippets” and then looked at the collection of snippets, we saw all the emotion and power of their stories drain away.

It was as if Moises had had no need for tissues, hadn’t expressed such fear and pain to a stranger. It felt like a dishonest representation of his story. We decided we could no longer persist in using this analytical method that now seemed both methodologically and ethically problematic. We needed to find a way to treat the story holistically that respected what the participant was trying to tell us and in what way, which did not snip it up into more digest- ible or dissectible “chunks.” This led us down an entirely different analytical path than what we had prepared to do, one which was grounded in narrative theory, which tried to respect how the participants told their stories, not just what they said in their stories.

Analysis These field accounts describe two key quality challenges that were connected by a common thread of the evocative emotional facets that characterized our participants’ lived realities and permeated our shared processes of sense-making. Here we draw on the notion of pragmatic validation, which Walther et al. (2013) described as the extent to which “concepts . . . with- stand exposure to the reality investigated” (p. 640), to reflect on the role of emotion in devel- oping and articulating the quality of our research process and findings.

In the first field account, Pawley reflected on the sense of “resonance” she experienced through conducting her interview with Moises: the questions she used to guide the participant’s story, and what Moises decided to say, resulted in the sharing of an emotional vulnerability. Through consul- tation with colleagues over the course of this collaborative inquiry, we have come to understand that the emotion being experienced by both Pawley and Moises was evidence of the authentically

408 Walther, Sochacka, Benson, Bumbaco, Kellam, Pawley, & Phillips

co-constructed nature of the interviewer-interviewee interactions. The fact that the framework of ruling relations that informed the interview design did withstand exposure to the reality under investigation, in that the questions asked yielded an authentic connection with the participants, was evidence of its suitability for the study. Considered in this way, the emotions that the partici- pants and Pawley experienced in the interviews serve not as evidence of her lack of objectivity (and therefore poor scientific practice), but as an indicator of the validation of the ruling relations framework that prompted the emotions in the first place. Therefore, our decision to persist with ruling relations as a theoretical framework seems validated.

In the context of the second account, it is clear that our initial analytical lens did not with- stand exposure to the reality under investigation. A thematic analysis is underpinned by cer- tain assumptions about the patterns within the social reality investigated – namely, that the participants’ meanings can be captured by, and condensed into, a hierarchical structure of organized themes. However, our approach of breaking up the participants’ stories into snip- pets and allocating them to themes did not result in an interpretive understanding that cap- tured the full extent and power of Moises’s lived experience.

It is this notion of emotionally evocative research results that we are currently exploring as we share our findings with the research community (Walther et al., 2015). In the context of handling data, Walther et al. (2013a) describe pragmatic validation as “concerned with the external customer . . . validation takes the form of checking the impacts or benefits from using the knowledge in practice” (p. 647). With this in mind, we contend that sharing our partici- pants’ stories with engineering education administrators would “work” if we succeed in evok- ing an empathic response that is reflective of the emotional essence of our participants’ experiences. If such an impact from using knowledge in practice persuades administrators to change the ruling relations of their institutions to be more inclusive, then this would be an additional expression of the pragmatic validation of our research process.

Communicative Validation in a Constructivist Grounded Theory Study

Amy E. Bumbaco

I framed my dissertation in the context of a larger research project on critical thinking in higher education. In my part of this project, I am using constructivist grounded theory (Charmaz, 2006) to understand how students and faculty from engineering and the humanities perceive and enact critical thinking. I am also examining how faculty incorporate critical thinking into their classes, and how faculty perceive students to enact critical thinking in their classwork. In addition to a deeper understanding of critical thinking, another goal of my study is to identify differences and similarities between engineering and the humanities in terms of how critical thinking is concep- tualized and performed. So far, I have conducted semistructured interviews with several faculty members and students from English and materials science and engineering. I am now in the pro- cess of analyzing these data using thematic analysis followed by constructivist grounded theory.

As I code the interview transcripts, I am using process journaling, a technique that com- bines aspects of a log or audit trail with traditional memos (Charmaz, 2006; Corbin & Strauss, 2008; Richards, 2005) to help me to maintain a detailed record of the data analysis and theory development. In this way, process journaling might be conceptualized as a strategy to promote process reliability, that is, to ensure a consistent process of interpretation and to document and demonstrate the dependability of this process. In the following two field

Qualitative Research Quality: A Collaborative Inquiry 409

accounts, however, I focus on how I am also using process journaling as a way to foster com- municative validation, that is, to support “the multiple, interlocking acts of social construction that span the entire process” (Walther et al., 2013a, p. 636). More specifically, the part of the process I focus on is meaning-making within the research team with a view to how to com- municate my interpretive findings to the wider research community.

Communicating within the research team During the process of focused coding, that is, after initial coding, I began grouping the initial codes to identify common themes across the data. This step of the data analysis involved identifying when the ini- tial codes represented similar concepts and creating focused codes that captured that concept. During this process, I met with my dissertation advisor on a weekly basis to seek advice on and discuss the development of my codes. At one of these meetings, I asked my advisor what to do with codes I was considering deleting. My advisor is familiar with the data, but is not immersed in it in the same way that I am, and so he could not offer advice simply by looking at the codes. For example, one code I considered deleting was “varying definitions of critical thinking.” To him, this code seemed highly rele- vant, and he questioned why I was considering deleting it. Through an examination of my process journal, he was able to follow the development of this code and see that there was a similar code called “defines critical thinking in multiple ways.” Through the discussion that followed, we agreed that one of these codes could be deleted, and that the second code better captured the underlying meaning of the concept. This advice helped me continue coding, sorting codes, and deleting codes with more confi- dence and clarity.

The process journal has become a key part of the conversations with my research advisor. I am able to record my thought process as it develops, which provides a means for my advisor to understand the ways in which I am working with the data as the analysis proceeds. Others have also pointed to the use of memos and audit trails as a means of communicating within a research team. For example, Corbin and Strauss (2008) state, “Without memos . . . it would be difficult to keep the lines of communication open between researchers” (p. 119). Docu- menting changes and decisions in the process journal allows fellow researchers to compare codes to data, codes to codes, and relationships that I have developed. In this way, both my advisor and I can understand why I have made various decisions in my analysis and what implications those decisions might have. Use of documentation in this way has also strength- ened our ability to co-construct meaning as a research team during discussions that involve not just the two of us but also other graduate students working on different projects in my group, by providing a transparent means to share insights, interpretations, and decisions.

Communicating to the research community As I begin to think about publish- ing my results, I am planning ways in which I can use the process journal in my publi- cations. One important element included in the process journal is the thought process I am engaging in as I organize connections among categories in order to develop the grounded theory. More specifically, I am looking for ways in which codes relate to each other in order to group them and then form relationships. As the relationships deepen, I am identifying bigger-picture connections to inform the development of the grounded theory. For example, the theoretical piece that focuses on “how to do critical thinking” comprises “how to get to the end goal” and “reaching the end goal” as part of the overall critical thinking process cycle. Even though these two categories

410 Walther, Sochacka, Benson, Bumbaco, Kellam, Pawley, & Phillips

are separated in this cycle, their relationship as recorded in the process journal has helped me to develop this part of the theory and the subsequent critical thinking process cycle.

This field account shows that when the communication community comprises readers of a jour- nal article or dissertation, the role of the process journal differs when it is used within the research team. Rather than delineating specific moves (e.g., moving an initial code from one cat- egory to another, or creating a new subcategory) it provides a means to understand the broader meanings of categories. Understanding these broader meanings is also important in research team discussions but is particularly important in communicating with external audiences.

Analysis As shown in the first field account, the process journal can serve to support processes of com- municative validation within the research team by providing the basis for negotiating a shared understanding of the analytical tools being used and the processes for moving through the data to create meaning. Thus, in this context, the focus of the communicative validation effort through the use of the process journal is on co-constructing shared understandings of my interpretations as they emerge in my hunches and sometimes nebulous, not-yet-grasped insights. A part of this process is to maintain a shared understanding of where I have been in the data and where I need to go next. It is important, in this stage of the study, to understand detailed moves and changes that constitute the gradual definition and delineation of the codes so that all members of the team understand how the analysis is unfolding.

The second field account illustrates a different form of communicative validation by allowing readers to understand the meanings of categories as constructed by those who conduct the data analysis. More specifically, I described how I am planning to draw on the coding journal to share longer-range trajectories of the interpretation process to provide a deeper understanding of the more fully developed constructs by means of their history of being defined and delineated. In the second field account, I described one example in which my interpretation can be made clear to readers by giving them what is, in essence, a replay of the interpretive process rather than a post hoc description. In doing so, I hope to deepen readers’ understanding of the overall theory and connections between concepts within that theory.

As a result of exploring my process journaling practice through the lens of communicative validation, I have come to appreciate that explicitly sharing details of the process is not merely a way of increasing the reader’s trust in a dependable process but, ideally, will add value and depth to the reader’s understanding of the findings. At the same time, I have observed that an expanded consideration of the purpose of keeping a process journal, in the sense of looking towards how it might serve to foster deeper levels of communication both within the research team and with the broader research community, has also deepened my own commitment to mitigating random influences on the research. In this way, I see process journaling as a research practice that contributes to both process reliability and communica- tive validation, with each effort mutually supporting the other and strengthening the overall quality of the research findings. Thus, I have come to regard the process journal (or audit trail or memo) not as a static document but rather as a coherent part of the process of inter- pretive research.

Qualitative Research Quality: A Collaborative Inquiry 411

The Role of Communicative Validation in the Enculturation of a New Researcher

to a Phenomenology Project

Lisa C. Benson

The process of a new team member joining a research project can be understood as encultura- tion: the socialization of the individual into a project such that he or she “acquires appropriate values and behaviors by learning what the culture deems to be necessary” (Perry, 2008, p. 547). The personal and social construction of knowledge requires systematic and deliberate immersion of an individual into the culture of the project team, typically in the absence of direct instruction (Perry, 2008). In graduate education, academic enculturation has been characterized by attrib- utes such as collegiality, mentoring, program structure, and learning to participate (Boyle & Boice, 1998; Casanave & Li, 2008). In this contribution, I examine the enculturation process through the lens of communicative validation, with a particular focus on ways to foster a pro- ductive and supportive communication community (Apel, 1972) in a research group setting.

The enculturation of the new graduate student I describe here took place in the context of a research project focused on interactions between student motivation and the development of their problem-solving skills. The main players in this example are two senior graduate stu- dent researchers, Danny and Brittany (all names are pseudonyms), who conducted two rounds of interviews with seven engineering undergraduate students, and Trina, a new graduate stu- dent, who analyzed the data from the second set of interviews, and who had limited prior experience in interpretive research.

The first round of interviews encompassed undergraduate students’ perceptions of their academic and professional futures and of solving engineering problems. In the second round of interviews, conducted after the participants had completed a problem that was new to them, the participants were asked to describe their approaches to solving the problem. Danny, the lead graduate student on the project, applied interpretative phenomenological analysis (IPA; J. Smith, Flowers, & Larkin, 2009) to the first interviews, in which the phenomenon of interest was connections between future-oriented motivations and present tasks (i.e., solv- ing engineering problems) as described by the students. IPA examines individuals’ interpreta- tions of experiences of a phenomenon as described, in this case, in in-depth interviews, to reveal tacit meanings embedded in personal descriptions (Lopez & Willis, 2004). Danny completed the IPA for the first set of interviews and initiated the IPA for the second set; Trina completed the latter. In the following field accounts, I describe how Trina navigated the process of enculturation by following research procedures, taking advantage of a shared space, and communicating within the community at several levels.

Joining a community of practice My research group is modeled as a community of practice, wherein more experienced members practice leadership and mentoring, and newer members are similar to apprentices to their more senior peers. When Trina joined my research group, I secured a desk for her in “the bullpen,” a large, open office space that is shared by eight to 10 graduate students, and invited her to attend the research group’s weekly meetings. I also met with Trina individually and provided her with background reading materials for the project, including relevant academic papers as well as examples from Danny and Brittany’s prior analyses. By initiating Trina into the research group and project in these ways, my goal was to help her to learn the ropes of doing qualitative engineering education research. I was happy to read in Trina’s

412 Walther, Sochacka, Benson, Bumbaco, Kellam, Pawley, & Phillips

weekly reflections that she was learning from these resources and also identifying gaps in her knowledge:

The extent to which I understand the project has come from reading Danny’s and Brittany’s papers. I know that the point is to look at problem solving in terms of [knowledge] transfer. I know that a series of interviews were conducted after students from BME and ME completed a problem. I don’t know the name of the project, nor am I sure how what I am working on will be used or tied into the bigger picture. – Trina, Week 1

Trina found answers to these questions and others – for example, what to read next and what approaches to data analysis she should take (including what software to use) by interacting with other graduate students in the bullpen. This close proximity to other, more experienced, researchers also provided her with an opportunity to listen in on coherent conversations about qualitative research. As a result, by Week 2 Trina was using domain-specific terms such as “coding” and confronting her lack of experience with qualitative, versus quantitative, methods:

My part in this project includes a qualitative analysis of the problem-solving inter- view transcripts. I have had ample experience with quantitative analysis, but I’m unfamiliar with qualitative analysis. So far I have installed R, and RQDA, figured out how to load a file, add codes, mark codes. – Trina, Week 2

Through a combination of being co-located in the bullpen, listening and interacting with others, and keeping a reflective journal, Trina began to explore specific strategies to increase the quality of her work as a novice qualitative researcher:

Brittany has suggested that I start bracketing, or listing out my understanding of different terms or areas in the field. I’m totally unfamiliar with this term, and an initial search came up with many different uses of this term, none of which fit the brief description I was given. – Trina, Week 3

Conducting data analysis independently and critically By Week 5, Trina was coding the data and immersing herself in the communication community (Apel, 1972) of the research group by developing her own codes and examining the appropriateness of the framework of knowledge transfer as a basis for her analysis. After reading through all the data at least once and feeling overwhelmed by the many different ways students described their experiences with the phenomenon, I suggested she take some time to “dwell in the data,” to step away and let the meaning of the data emerge. As part of this process, we also met with an external advisor on the project to discuss questions Trina had about the coding process:

The coding process got much easier as I proceeded . . . I started out trying to fit every- thing into the [a priori] framework . . . After speaking with [the external advisor], I started just marking everything that seemed significant and observing patterns. – Trina, Week 7

Having direct access to the external advisor was an important part of Trina’s devel- opment. It demonstrated to her the structure of the research community beyond those

Qualitative Research Quality: A Collaborative Inquiry 413

in her immediate proximity. Encouraged by the research team and the external advisor to research her topic independently and review the data again, Trina began to add new layers of meaning to the analysis and expand the theoretical framework. She also began to critically question the procedures she was following as she interpreted the data and consulted with group members in the bullpen:

As I was going through the transcripts, I noticed I was coding “knowledge as prop- agated” inappropriately. I have had to delete a lot of those codes. – Trina, Week 7

During Week 9, Trina made a small breakthrough as she recognized a limitation of our theoretical framework of knowledge transfer and proposed a new code to address an aspect of the phenomenon that she found missing in our existing codes that she felt was important for the analysis:

“During our interviews the idea of ‘knowledge of concept’ appeared and proved to be a unique phenomenon from the other metacognitive knowledge codes and in need of its own code.” – Trina, Week 13 (Project Summary)

At Week 12, after revising and clarifying the codebook, Trina started over, stripping out all of the old codes and recoding all of the data. She recognized the benefit of let- ting the meaning within the data emerge over time, the outcome of “dwelling” time:

To write the summary, for each category I reread the student’s transcript for the big picture in that category, and looked at individual codes for that category. – Trina, Week 13 (Project Summary)

Finally, as Trina’s analysis was integrated with Danny’s results, Trina contributed to the generation of new knowledge. After describing how some students discussed the problem they were asked to solve numerically and others focused on concepts, Trina observed:

This behavior demonstrates a range of conceptual understanding; students fell in the range of describing the problem mostly conceptually to mostly numerically [. . . .] This order is the same order in which the students define engineering prob- lems as [ranging from] open-ended [to close-ended]. From this phenomenon, we can see that students who view the problem as more open-ended are also the students who conceptualize the problem more. – Trina, Week 13 (Project Summary)

Analysis According to Walther et al. (2013), the key function of communicative validation “is to estab- lish a ‘community of interpretation’ (Apel, 1972) with both the internal and external custom- ers of the research” (p. 646). As illustrated in the above two field accounts, I used three main strategies to enculturate Trina into this community and ensure continuity of the research pro- ject in terms of the theoretical assumptions underpinning the method, the contextual proce- dures of analysis, and the emerging interpretations. These entailed:

Engagement with research procedures These included Trina’s reflections, the code book, and the formal and informal introductions to the research topic, data, software, and analytic methods by members of our research group.

414 Walther, Sochacka, Benson, Bumbaco, Kellam, Pawley, & Phillips

Use of shared space The physical co-location of, and communication between, the students within our group in an open work area (“the bullpen”) allowed students to easily and instantly query each other, check meanings and interpretations, and share resources. Trina’s reflections provided implicit evidence of this through her frequent references to observations and short conversations with other group members.

Development of a research community The communication community in this pro- ject involved several tiers, including the project team (direct mentoring), the larger research group (culture of communication and support), and the research community (knowledge of and access to experts and external advisors).

This analysis suggests that the pursuit of communicative validation as an aspect of research qual- ity goes beyond considerations of the robustness of socially constructing meaning about the “research object” and includes purposeful efforts to socially construct a shared understanding of the research process within the team that is at the same time anchored in broader communication communities around research methodology. In Trina’s case, this shared understanding concerned the procedures implemented in the particular project, the methodological commitments as understood by the group, and the understandings of the data and interpretations that emerged between the members of the research team. The purposeful effort to promote this aspect of com- municative validation included both explicit (e.g., the code book) and emergent (e.g., the bull pen) elements that spanned the three dimensions of procedures, space, and communities. This practice exploration thus suggests that quality considerations are not only located at or limited to the project level but extend to the collaborative structure of research units.

The Interplay of the Science and the Art of Doing Research: Using Process Reliability and Expanding

the View of Communicative Validation

Nadia Kellam and Joachim Walther

In this study we are seeking to develop an understanding of the role of emotion in engineer- ing student learning (Kellam, Walther, Wilson, Kerow, & Lande, 2015). We used a narrative interviewing technique (Hollway & Jefferson, 2000; Roulston, 2010; Seidman, 2006) to elicit stories of how undergraduate engineering students became interested in engineering and how they experienced engineering up to the current point in their education. After the narrative interviews were conducted, we analyzed data through multiple iterations of listening to audio files, identification and coding of goals and events in the narratives, constructions of narra- tives, thematic analysis uncovering the emotions present in the narrative constructions, and generation of theory related to the role of emotion in engineering student learning.

In this practice exploration, we discuss an early phase in this larger research project where the research team was engaged with making data, that is, conducting the narrative interviews. In a narrative interview, the goal is to elicit and co-construct the larger, oftentimes rehearsed story of the participant and then to dig deeper into each part of the story to get a more in- depth account of the events as the participant experienced them (Gubrium, 2012; Gubrium & Holstein, 2002; Holstein & Gubrium, 2012; Mishler, 1986). For this purpose, we initially developed a detailed interview protocol (Weiss, 1994) to dependably guide the research team to conduct interviews in a way that minimized random influences on the process (process reli- ability). This detailed interview protocol might be regarded as an example of the science of

Qualitative Research Quality: A Collaborative Inquiry 415

interviewing, as we tried to scaffold the interviews in a way that would allow novice inter- viewers in the research team to elicit high quality data across a wide range of possible interview scenarios. As we conducted more interviews and reflected on them in the research team, this detailed protocol evolved into a more fluid approach, as we sought to balance the interview pro- tocol with the emergent nature of students’ narratives as they were co-constructed in the inter- views – what we have come to refer to as the art of interviewing and what might be productively examined through the lens of communicative validation. We used the quality framework to purposefully navigate tensions between the science and art of interviewing that emerged between our efforts to pursue these sometimes conflicting aspects of research quality.

Developing a narrative interview protocol to capture participants’ experiences This field account describes our efforts to ensure that the quality of our narrative inter- views was consistent across multiple interviewers in a research team with varying levels of experience with conducting interviews. Early iterations of the interview protocol involved detailed steps to follow during the meeting and a list of specific questions to probe vari- ous aspects of the participants’ experiences. All of the members of the research team – three graduate research assistants and two faculty members – conducted interviews. As we started conducting interviews, at least three of us would listen to each interview and the interviewer would first reflect in their log trail and then the team would reflect during our meeting on how the interview went: ways that it went well, and ways that could be improved. This shared reflective process led us to more deeply appreciate how important it is to emphasize not only the science of interviewing as captured in our protocol, but also to focus on developing the art of interviewing. We observed that mastering this art involved listening deeply, engaging fully, and asking follow-up questions to help flesh out the narrative of the participant. In interviews conducted in this manner, we observed the flow of the conversation and the relative ease with which the interviewers elicited coherent narratives from the participant.

Below we use an example from an interview to illustrate our process of transitioning from focusing on the science of learning to interview to the art of interviewing. The excerpt below is from an early interview (conducted on February 2, 2014), with a gradu- ate research assistant as the interviewer (I) and an undergraduate engineering student as the participant (P).

I: Right. What have been your interactions with students so far? You mentioned hav- ing them to help figure things out. What are some other interactions you’ve had so far like inside the classroom, outside the classroom?

P: Everyone you talk to in the dining hall no matter who they are seems to be extremely friendly, so that’s good. That might just be a Southern culture thing or maybe college in general.

I: Right.

P: What was the question?

I: Just different interactions with students and what are some interactions you’ve had?

P: People are pretty cool. I haven’t met anyone who I just intensely dislike. There’s a competitive drive, which I like. If you don’t have someone to compete with then what’s the point of achieving yourself?

416 Walther, Sochacka, Benson, Bumbaco, Kellam, Pawley, & Phillips

I: Right.

P: That’s about it. No one’s problematic and stuff.

I: Tell me more about this competitive drive. What do you mean by that?

P: It’s not just “I want to pass and get through college.” I want to do the best I can to achieve. If someone makes something, another person will try to make it better. They’ll keep driving to see who gets it solved. It’s not just “stick with me” medioc- rity, it’s like “push us all.”

I: Right, okay. What have some of your interactions with faculty been like?

P: I would say it’s kind of back and forth. I mean, some of the faculty are really nice, friendly, helpful. Others are kind of doing the job to do the job. That’s kind of both sides of professors.

In this example, the interviewer focused on the questions listed in the interview protocol. At one point he strayed from the interview protocol to ask a follow-up ques- tion about the competitive drive. But for the most part the interview lacked flow and the participant spoke mostly in generalities instead of sharing his detailed stories – one of the primary goals of narrative interviewing. The interviewer also reinforced the partici- pant’s responses by saying “right” throughout the interview, which can influence the participant’s response and interrupt the flow of the elicited story (Seidman, 2006). In a log trail entry, the interviewer reflected “I think the interview went okay, but I had a hard time getting specific narratives from the participant. Prior to the interview I men- tioned wanting to get his stories, but he gave more general statements.”

During our team’s weekly reflection meetings, we began to notice that the inter- viewers who were more closely following the interview protocol tended to not listen as well and took a long pause when they were searching through the protocol to find the next question to ask. Our attempt at establishing process reliability through detailed interview protocols and a scripted set of questions seemed to negatively affect the quality of the inter- views. While the same questions were being asked of each participant, the responses were not as helpful because the cadence of the interview was utilitarian, with the interviewer ask- ing a question, the participant responding, and the interviewer asking another. The flow of the interview did not have a conversational tone and did not encourage participants to elaborate on their responses. The extensive list of questions seemed to limit the inter- viewer’s ability to deeply engage with the participant and co-construct authentic and coherent accounts of their experiences, a central aspect of communicative validation.

Based on this collective reflection process, we drastically simplified the interview protocol to encourage better listening by the interviewer and to better flesh out the sto- ries of the participant through asking follow-up questions based on what was said. The original interview protocol consisted of 13 questions. The new, more open-ended interview protocol consisted of one guiding question that focused on the participant’s overall story and three follow-up questions to flexibly elicit detailed narratives of dif- ferent parts of the larger story (see questions below). With the explicit emphasis on lis- tening to the participants and the flexible openness of the protocol, we hoped to provide space for the graduate research assistants to develop the art of interviewing – thus encourag- ing the participant to share experiences and explore the meanings of those experiences.

Qualitative Research Quality: A Collaborative Inquiry 417

For the guiding question, we asked, “I understand that you are a # year [type of engineering] engineering student. Could you tell me your story of how you got to where you are today?” The three follow-up questions each began with “You mentioned . . .” followed by “Tell me more about that,” “Can you give me an example of that?” or “How did you experience this moment?”

Revising the interview protocol significantly helped the narrative interview team to successfully elicit rich narratives from the participants. After a revision of our interview protocol, a much richer data set emerged. Without individual and group reflection prac- tices, we might have completed all of the interviews without noticing the lack of rich- ness in the data, only to discover it when we began our analysis of the data.

Analysis The above field account illustrates how we used the quality framework to navigate tensions between our efforts to promote process reliability through dependable procedures and our striving for communicative validation through authentically co-constructing participants’ nar- ratives. As a result of these quality considerations, we modified our detailed interview proto- col, which had initially been used to increase process reliability through eliminating random influences, to a simpler interview protocol that allowed for more flexibility during the inter- view, thus promoting the robustness and openness of the exchange between interviewer and participant (i.e., communicative validation). We also note that the progression of the research team, consisting of the use of log trails and individual and collective reflections, was itself a process of socially constructing our way of conducting interviews. This aspect of the commu- nicative validation of the contextual research process within the research team is similarly dis- cussed in Benson’s contribution above.

In this emergent process, we used the quality framework as a conceptual frame and language to identify, articulate, and navigate tensions or goal conflicts between different facets of quality that arose in the context of our work. This practice exploration thus suggests that efforts to pro- mote research quality in one validation category cannot be viewed in isolation. Rather, we con- tend that quality strategies need to be viewed across all validation categories with the quality framework providing the means to identify synergies and purposefully weigh potential tensions that may arise (see also contributions by Bumbaco and by Sochacka and Walther).

Productive Interpretive Dissonance: The Interplay of Theoretical and Communicative Validation

Processes in a Narrative Inquiry

Nicola W. Sochacka and Joachim Walther

In this study we are examining how engineering is portrayed in the media, with a particular focus on issues related to attracting diverse groups of students to the profession (Sochacka, Walther, Wilson, & Brewer, 2014). Our research approach is theoretically anchored in the concept of framing from the field of media studies, which highlights the role the media plays in privileging some aspects of a perceived reality over others (Entman, 1993). Methodologi- cally, we draw on the work of Roe (1992) in the field of applied narrative analysis, who has developed an approach to investigate complex, uncertain, and polarized policy problems. Like the problems Roe addresses in his work, we argue that diversity, recruitment, and retention efforts in engineering are complex, uncertain, and polarized (see Brewer, Sochacka, &

418 Walther, Sochacka, Benson, Bumbaco, Kellam, Pawley, & Phillips

Walther, 2015), and in need of a “different agenda” – one which will allow us “to move on issues that were dead in the water on . . . older agendas” (Roe, 1992, p. 52).

Narrative policy analysis (NPA) is based on the understanding that policy issues are under- pinned by stories that are “a force in themselves and must be considered explicitly in assessing policy options” (Roe, 1992, p. 2). The principal tasks of NPA are to uncover the dominant sto- ries, counterstories, and nonstories that characterize an issue; construct a metanarrative that is told by the comparison of these three different types of stories; and, in the last step, critically evaluate how the newly created metanarrative reframes the issue in such a way as to make it more amenable to effective decision making. We used this approach to analyze one year of First Bell daily electronic news briefings (August 2011–July 2012; Sochacka et al., 2014).

The data analysis for the project was undertaken over a one-year period as a collaborative effort between two senior researchers (Sochacka and Walther) and one undergraduate stu- dent. Through the following field accounts, we examine our research team’s process of co- constructing stories from the data, an endeavor that was characterized by contention and mul- tiple iterations, as features of communicative validation, and the parallel, gradual convergence toward resonance of the findings that indicated a developing degree of theoretical validation. Both quality constructs helped us navigate this process, and their productive interplay is illus- trated in the following accounts of two key turning points in the project as reconstructed from Sochacka’s log trail (Richards, 2005).

Recognizing patterns in the data During the first six months of the project, we had pursued a grounded theory approach to organize features of the public discourse into a thematic structure of codes. The analysis seemed to progress in a steady fashion as the undergraduate researcher generated codes that described distinct aspects of the data. At the same time, a slowly growing lack of enthusiasm over our emerging find- ings indicated that, as a team, we did not experience a sense of resonance between our analysis and the bigger picture that the data seemed to hold. More specifically, while the codes seemed to represent the complexity of perspectives on a micro level, on a macro level they did not make sense or teach us much new about the issues we were inves- tigating. A breakthrough in this frustrating phase of the work was the recognition of what appeared to be more and less dominant versions of stories about engineering that seemed to reach across the data. Two early examples of this were the juxta- positions of codes, such as “engineering as mainly math- and science-based” versus “engineering as more than just math and science” and “people not entering STEM fields” versus “problems getting a job with an engineering degree.”

This first turning point, the idea of storied patterns in the data, led us to the narra- tive policy analysis method and provided the team with a glimpse of resonance and thus renewed enthusiasm.

Striving for interpretive resonance In our subsequent interpretive efforts, we endeavored to fit the prior topic codes from the grounded theory analysis into two overarching stories, which we termed “materialistic competitiveness” as the dominant story, and “holistic innovations” as the counterstory. However, try as we did to make logical narrative connections between the topic codes, our theoretical understandings invariably remained complicated. The overall stories seemed to make sense but could not accom- modate the different aspects of the discourse in a coherent way without becoming overly and uselessly complex. For several months, we felt as if we were going around in circles. The weekly meetings were dynamic and driven by our sense of broader resonance but, at

Qualitative Research Quality: A Collaborative Inquiry 419

the same time, also frustrating. It seemed that each time we thought we were getting somewhere, the undergraduate student would share a different set of quotes, which would send us off on another interpretive trajectory. We were stuck among the trees and could not (for the life of us!) see how they fit into the forest.

As one of several explorations, we ultimately followed a promising hunch to look for differences, both within the codes we had previously configured into each of the two main narratives and between the two narratives themselves. It was this second turning point that gave us our first insight into the possibility that there were, perhaps, not one but several dominant stories being told about engineering in the media articles. The set of dominant stories that subsequently emerged from our ongoing process captured both the essence of the data and built on, simplified, and, thus, illuminated our prior interpretations (Sochacka et al., 2014). With the trees now rooted firmly in place, we could at last see the forest, a breakthrough that was accompanied by a discernible shift in the atmosphere of the meetings from one of dissonance to one of resonance. In fact, it was quite a reve- lation on the day the undergraduate researcher described recognizing the explanatory and illuminating power of the newest iteration of our model as fitting with and deepen- ing her own understanding of the data.

Analysis These field accounts of two main turning points in our project show how processes of theo- retical validation were intertwined with and supported by processes of communicative valida- tion. More specifically, we developed a deeper understanding of theoretical validation as the capacity of research to reflect the “coherence and complexity of the social reality under inves- tigation” (Walther et al., 2013a, p. 640). The iterative and nonlinear process of convergence towards this fit, or subjective agreement between the theory generated and the social reality under investigation, was mirrored by a struggle in the interpretation community (i.e., the research team) to construct shared meaning from the data.

At the first turning point, our interpretations captured some of the complexity of the data, in that our codes represented aspects of the discourse at the micro level (the trees). As a team we experienced a degree of agreement about the details but a lack of a sense of coherence in terms of larger, explanatory patterns in the data (the forest). This issue of theoretical validation is also related to Pawley and Phillips’s discussion of the assumptions underpinning the analytic lens being challenged in the sense of pragmatic validation. Here, we complement this insight by pro- viding another example of the role that an emergent research design (i.e., our change from the- matic analysis to narrative policy analysis) can play in seeking to achieve theoretical validation.

Leading up to the second turning point, our focus on constructing narratives brought us closer to a sense of coherence; however, this perceived interpretive progress stood in contrast to lively disagreements over the ways in which the interpretations could accommodate the com- plexity of the data. While we were able to glimpse the forest, we could not seem to fit in all of the trees. With our decision to examine a system of several dominant stories and one counter- story, we were finally able to achieve a coherent interpretation in which the complexity of the data appeared as organized richness that supported rather than challenged the larger patterns.

We propose that our process of co-constructing interpretations of the data that resonated with all members of the research team – an effort that was characterized by multiple struggles and disagreements – also offers an expanded view on communicative validation. More specifi- cally, we contend that our practice-based examination of this quality construct reveals that the

420 Walther, Sochacka, Benson, Bumbaco, Kellam, Pawley, & Phillips

process of co-constructing meaning is not always a linear development of consensus or inter- rater agreement. Rather, the process can be strengthened and its robustness demonstrated by embracing what we propose as “productive interpretive dissonance” in a research team.

Discussion and Conclusions

The above practice explorations each examined issues of interpretive research quality that emerged over the course of our collaborative inquiry. In the following sections, we look across these explorations to derive four overarching insights that speak to the theoretical assump- tions underpinning quality considerations in interpretive inquiries and offer departure points for advancing the theoretical discussion of qualitative methodology in the engineering educa- tion research community. These insights are also discussed in relation to the quality frame- work that provided the conceptual grounding for our shared process. As such, the discussion here contributes to the incremental and ongoing pragmatic validation of the framework while offering transferable methodological insights of broader relevance.

ResearchQuality as aMultidimensional Issue In a number of our discussions, we experienced individual team members’ quality challenges as intricate and multilayered issues. Bumbaco, for example, discussed the connected but dis- tinct functions of her process journal. At one level, she used the journal as a way to maintain consistency in her own development of the codes, and, at another level, it served a key role in supporting communication with her advisor and, ultimately, the research community. Identi- fying these functions as contributing to both process reliability and communicative validation lent additional intent and purpose to her continued use of the process journal. Similarly, Kel- lam and Walther explored the tensions between the usefulness of a detailed interview protocol in striving for consistency across multiple interviewers and the tendency of a detailed protocol to render the interview approaches of novice researchers somewhat mechanistic. In this exam- ple, the notions of process reliability and communicative validation served to identify and articulate goals conflicts between two fundamental facets of overall quality.

These examples and other similar experiences of the co-researchers led to our first over- arching insight, that issues of, or challenges to, research quality are multidimensional and need to be flexibly viewed from different angles, each of which may reveal unique perspectives about the underlying features of the situation. Applying the different theoretical lenses offered in the quality framework revealed distinct fundamental facets of research quality as well as their productive intersections or tensions.

In relation to broader discussions of research quality, this need to consider multiple, inter- secting, and context-dependent facets of a specific quality challenge provides an interesting perspective on existing approaches to developing quality in interpretive research. More specif- ically, we contend that the guidelines, criteria, or procedures discussed at the beginning of this article may, particularly for novice researchers, limit discussions of research quality to sin- gle issues or techniques and obscure a broader, contextual view of the interconnected richness of quality challenges in practice.

Qualitative Research Quality: A Collaborative Inquiry 421

Quality as Implicitly Infused in Research Practice A second insight that emerged across the analysis of the above practice explorations concerns the ways in which issues or challenges of research quality manifest in the course of a research project. We rarely experienced the quality considerations in the authors’ projects to be dis- tinct, clearly identifiable events or points in the research project. Instead, the issues arose gradually, often originating from dynamics that were not immediately visible, a recognition of which emerged from experiences of tensions or discomfort and were only gradually uncovered through intentional processes of reflection and analysis.

In the context of the narrative policy analysis project, Sochacka and Walther described a prolonged phase of frustration that was experienced by the members of the research team as they strove to wrestle coherent interpretations from the complex dataset. This process was supported by a robust discussion culture in the group (communicative validation) but only gradually emerged as also being grounded in an experienced lack of resonance and conceptual clarity from the results (theoretical validation). The ability to articulate this interplay between communicative and theoretical validation allowed the team to identify the source of their dis- comfort and ultimately persist in striving for resonance in their findings. In a similar way, Pawley described being prompted by feelings of doubt to critically engage the core premise of her study and her own motivations and biases (pragmatic validation of prior theory in making data), while weighing these doubts against the strong sense of resonance she experienced with her respondents (communicative validation). Being able to articulate and make the sources of her discomfort explicit through the aforementioned quality constructs contributed signifi- cantly to navigating this key turning point of her project in a purposeful way.

These examples demonstrate that quality challenges are often implicitly infused into our research practice and the specific circumstances of a project. As researchers, we thus need to pay attention to, and acknowledge, the sometimes subtle symptoms that indicate potential quality issues. The approach of quality criteria suggested in the literature (Bernhard & Baillie, 2012; Lincoln, 1995; Morrow, 2005; O’Brien et al., 2014; Tong et al., 2007; Tracy, 2010) is one possi- ble way to identify such quality issues. Depending on their use, however, criteria might risk being too separate from the research process to be able to capture research quality as the lived process we experienced in this collaborative inquiry. Once identified, we experienced it as crucial to make potential quality challenges explicit and articulate their impact on or role in overall research qual- ity. In this way, we can purposefully navigate the nuances and complexities of interpretive inqui- ries in naturalistic settings. In our collaborative inquiry, we found the constructs in the quality framework to offer both critical starting points and a coherent language for these explorations.

As the engineering education community further develops methodological expertise across multiple interpretive approaches, we contend that we must cultivate a coherent, conceptual language around research quality that we can use to integrate into our everyday practice, share and discuss methodological accounts, and advance our understanding of methodology in both practical and theoretical terms.

Beyond theObjects, Procedures, and Products Our discussions of quality challenges in our respective studies led us to a third insight, that people and context play a key role in achieving overall quality in a project. Pawley and Phillips’s and Sochacka and Walther’s explorations discussed above brought the role of the

422 Walther, Sochacka, Benson, Bumbaco, Kellam, Pawley, & Phillips

researchers as whole persons shaping the entirety of the project into focus. More specifically, we examined the complex interplay through which bias, motivation, and intent contribute to overall quality – a dynamic that suggests a more central role of the researcher and the research team throughout the process than established approaches such as a priori subjectivity statements (Preis- sle, 2008) seem to acknowledge. In a similar fashion, Benson explored the broader role that the research group, with its tacit as well as intentional structures, plays in the ways that research qual- ity is embedded into individual projects. In her example, the interaction of formal structures (communicative and procedural validation) and the intentional efforts to shape a robust culture and network of communication in the group powerfully highlighted the opportunities for devel- oping research quality that arise from considering the local context of research.

The analysis of these and similar examples suggests that, as researchers, we need to broaden our view of research quality beyond the phenomena under investigation, the applica- tion of procedures, and the trustworthiness of the findings themselves. Such a broader view would explicitly acknowledge us as individuals and research teams and the ways in which we socially construct specific, local versions of conducting research that play a key role in achiev- ing quality across our projects. This influence and role of a local research culture and shared capacity seems particularly relevant for engineering education research as a field that has inherited the structure of research groups or labs from its engineering science antecedents.

The analysis of these locally constructed ways of conducting interpretive research using the constructs and language of the quality framework was, in our experience, a powerful way to reveal these less examined aspects of methodology and make them accessible for purposeful decision making and planning. The suggestion to broaden our view of methodology to more genuinely and productively include researchers and context thus complements the above call to cultivate a coherent language around quality in interpretive engineering education research.

IntroducingNovices to Interpretive Research In all of the above practice explorations, aspects of engaging novice researchers in interpretive work were prominent features of the shared discussions. Benson described the integration of novice researchers in her group as an intentional and purposeful process of enculturation that was informed by specific fundamental aspects of research quality. Similarly, Kellam and Walther discussed the challenge of balancing procedural consistency against authentically engaging participants in interviews conducted by novice researchers. From the perspective of a developing researcher, Bumbaco, who was a graduate student at the time of the collabora- tive inquiry, discussed the importance of creating a productive communication community with her advisor through the purposeful use of her research journal.

These examples can be summarized in our fourth insight, that issues or challenges of research quality are critical sources for explicit teaching and tacit learning of methodology. These occasions are opportunities for students, or researchers from other fields, to acquire methodological competence, outside the abstract appreciation of principles and procedures, and engage the richness and messiness of lived research practice with a level of clarity and intentionality. As researchers and mentors, we can support this process by actively modeling quality thinking and analysis as infused throughout our locally constructed ways of conduct- ing interpretive research. Such modeling includes noticing and acknowledging potential qual- ity issues, and having the language to articulate and analyze them with a view to engaging these challenges actively and in context.

Qualitative Research Quality: A Collaborative Inquiry 423

Closure In the above insights, we recognize research quality issues as multidimensional, infused into practice rather than distinct from it, and connected to contextual factors beyond a particular project. The first three of the four insights came to the fore when we considered introducing novices to interpretive research. This connection between ensuring research quality and train- ing novices points to tangible opportunities to further advance our thinking about, and devel- opment of, methodological competence in our research groups, and in the broader research community. More specifically, as engineering education researchers we can continue to sharpen our sensitivity to issues of quality that may otherwise remain hidden behind the daily challenges of conducting research. Further developing a coherent conceptual language would allow us to make these quality issues explicit and amenable to purposeful decision making in our research process. On a broader level, we can cultivate broader discourses and structures to promote research quality beyond the scope of one project. Through these efforts, we can develop not only a new generation of capable engineering education researchers but also members of the research community who are able and committed to actively advancing our methodological discourse, an endeavor that serves to increase the acceptance of our diverse methods of inquiry and the impacts of our research findings.

Acknowledgments This material is based upon work supported by the National Science Foundation under grants 1055900, 1159016, 1055950, 1150668, and 1160350. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation. We would like to thank those who have contributed to our projects, either as research participants, members of our research groups, or external advisors. We are also grateful to the anonymous reviewers and editorial team at JEE for their careful reading of our manuscript and insightful and constructive suggestions.

References Acker, J. (1990). Hierarchies, jobs, bodies: A theory of gendered organizations. Gender &

Society, 4, 139–158. https://doi.org/10.1177/089124390004002002 Acker, J. (2006). Inequality regimes: Gender, class, and race in organizations. Gender & Soci-

ety, 20, 441–464. https://doi.org/10.1177/0891243206289499 Alvesson, M., & Sk€oldberg, K. (2009). Reflexive methodology: New vistas for qualitative

research. Thousand Oaks, CA: Sage. Amedeo, D., Golledge, R. G., & Stimson, R. J. (2009). Person-environment-behavior research:

Investigating activities and experiences in spaces and environments. New York, NY: Guilford. Anfara, V. A., Brown, K. M., & Mangione, T. L. (2002). Qualitative analysis on stage: Mak-

ing the research process more public. Educational Researcher, 31, 28–38. https://doi.org/ 10.3102/0013189x031007028

Angen, M. J. (2000). Evaluating interpretive inquiry: Reviewing the validity debate and opening the dialogue. Qualitative Health Research, 10, 378–395. https://doi.org/10.1177/ 104973230001000308

Apel, K. O. (1972). The a priori of communication and the foundation of the humanities. Man and World, 5, 3–37. https://doi.org/10.1007/bf01253016

424 Walther, Sochacka, Benson, Bumbaco, Kellam, Pawley, & Phillips

Baillie, C., & Douglas, E. P. (2014). Confusions and conventions: Qualitative research in engineering education. Journal of Engineering Education, 103(1), 1–7. https://doi.org/10. 1002/jee.20031

Barbour, R. S. (2003). The newfound credibility of qualitative research? Tales of technical essentialism and co-option. Qualitative Health Research, 13, 1019–1027. https://doi.org/ 10.1177/1049732303253331

Bernhard, J., & Baillie, C. (2012). Standards for quality of research in engineering education: A prologomenon. Thessaloniki, Greece: Aristotle University of Thessaloniki.

Borrego, M. (2007a). Conceptual difficulties experienced by trained engineers learning educational research methods. Journal of Engineering Education, 96, 91–102. https://doi. org/10.1002/j.2168-9830.2007.tb00920.x

Borrego, M. (2007b). Development of engineering education as a rigorous discipline: A study of the publication patterns of four coalitions. Journal of Engineering Education, 96, 5–18. https://doi.org/10.1002/j.2168-9830.2007.tb00911.x

Boyle, P., & Boice, B. (1998). Best practices for enculturation: Collegiality, mentoring, and structure. New Directions for Higher Education, 1998, 87–94. https://doi.org/10.1002/he.10108

Bray, J. N., Lee, J., Smith, L. L., & Yorks, L. (2000). Collaborative inquiry in practice: Action, reflection, and making meaning. Thousand Oaks, CA: Sage.

Brewer, M., Sochacka, N. W., & Walther, J. (2015). Into the pipeline: A freshman student’s experiences of stories ‘told’ about engineering. ASEE Annual Conference, Seattle, WA. https://doi.org/10.18260/p.24355

Brydon-Miller, M., Greenwood, D., & Maguire, P. (2003). Why action research? Action Research, 1, 9–28. https://doi.org/10.1177/14767503030011002

Casanave, C. P., & Li, X. (2008). Learning the literacy practices of graduate school: Insiders’ reflections on academic enculturation. Ann Arbor, MI: University of Michigan Press.

Case, J. M., & Light, G. (2011). Emerging research methodologies in engineering education research. Journal of Engineering Education, 100, 186–210. https://doi.org/10.1002/j.2168- 9830.2011.tb00008.x

Charmaz, K. (2006). Constructing grounded theory: A practical guide through qualitative analysis. Thousand Oaks, CA: Sage.

Cho, J., & Trent, A. (2006). Validity in qualitative research revisited. Qualitative Research, 6, 319–340. https://doi.org/10.1177/1468794106065006

Christians, C. G. (2011). Ethics and politics in qualitative research. In N. K. Denzin & Y. S. Lincoln (Eds.), The Sage handbook of qualitative research. Thousand Oaks, CA: Sage.

Corbin, J., & Strauss, A. (2008). Basics of qualitative research (3rd ed.). Thousand Oaks, CA: Sage. Douglas, E. P., Koro-Ljungberg, M., & Borrego, M. (2010). Challenges and promises of

overcoming epistemological partiality: Advancing engineering education through accep- tance of diverse ways of knowing. European Journal of Engineering Education, 35, 247–257. https://doi.org/10.1080/03043791003703177

Eakin, J. M., & Mykhalovskiy, E. (2003). Reframing the evaluation of qualitative health research: Reflections on a review of appraisal guidelines in the health sciences. Journal of Eval- uation in Clinical Practice, 9, 187–194. https://doi.org/10.1046/j.1365-2753.2003.00392.x

Elliott, R., Fischer, C. T., & Rennie, D. L. (1999). Evolving guidelines for publication of qualitative research studies in psychology and related fields. British Journal of Clinical Psy- chology, 38, 215–229. https://doi.org/10.1348/014466599162782

Qualitative Research Quality: A Collaborative Inquiry 425

Ellis, C. (2016). Compassionate research: Interviewing and storytelling from a relational ethics of care. In I. Goodson, M. Andrews, & A. Ari (Eds.), The Routledge international handbook on narrative and life history. New York, NY: Routledge.

Entman, R. M. (1993). Framing: Toward clarification of a fractured paradigm. Journal of Communication, 43, 51–58. https://doi.org/10.1111/j.1460-2466.1993.tb01304.x

Flick, U. (2006). An introduction to qualitative research. London: Sage. Flick, U. (2007). Managing quality in qualitative research. London: Sage. Flick, U. (2008). Designing qualitative research. Thousand Oaks, CA: Sage. Flick, U. (2009). An introduction to qualitative research. Thousand Oaks, CA: Sage. Fortenberry, N. L. (2006). An extensive agenda for engineering education research. Journal of

Engineering Education, 95, 3. https://doi.org/10.1002/j.2168-9830.2006.tb00872.x Geertz, C. (1974). From the native’s point of view: On the nature of anthropological under-

standing. Bulletin of the American Academy of Arts and Sciences, 28, 26–45. https://doi.org/ 10.2307/3822971

Gubrium, J. F. (2012). The Sage handbook of interview research: The complexity of the craft. Thousand Oaks, CA: Sage.

Gubrium, J. F., & Holstein, J. A. (2002). From the individual interview to the interview soci- ety. In J. F. Gubrium & J. A. Holstein (Eds.), Handbook of interview research: Context and method. Thousand Oaks, CA: Sage.

Guillemin, M., & Gillam, L. (2004). Ethics, reflexivity, and “ethically important moments” in research. Qualitative Inquiry, 10, 261–280. https://doi.org/10.1177/1077800403262360

Haghighi, K. (2005). Quiet no longer: Birth of a new discipline. Journal of Engineering Education, 94, 351. https://doi.org/10.1002/j.2168-9830.2005.tb00862.x

Heron, J., & Reason, P. (1997). A participatory inquiry paradigm. Qualitative Inquiry, 3, 274–294. https://doi.org/10.1177/107780049700300302

Hollway, W., & Jefferson, T. (2000). Doing qualitative research differently: Free association, narrative and the interview method. Thousand Oaks, CA: Sage.

Holstein, J. A., & Gubrium, J. F. (2012). Varieties of narrative analysis. Thousand Oaks, CA: Sage.

Jamieson, L. H., & Lohmann, J. R. (2009). Creating a culture for scholarly and systematic inno- vation in engineering education: Ensuring us engineering has the right people with the right talent for a global society. https://www.asee.org/member-resources/reports/CCSSIE/ CCSSIEE_Phase1Report_June2009.pdf

Johri, A., & Olds, B. M. (Eds.). (2014). Cambridge handbook of engineering education research. New York, NY: Cambridge University Press.

Kellam, N. N., Walther, J., Wilson, G., Kerow, K., & Lande, M. (2015). Uncovering the role of emotion in learning through first- and second-year engineering students’ narratives. Research in Engineering Education Symposium, Dublin, Ireland. https://www.scopus.com/record/ display.uri?eid52-s2.0-84962768485&origin5inward&txGid5B48A152193019A6DCF C559CF14BD023A.wsnAw8kcdt7IPYLO0V48gA%3a2

Kirk, J., & Miller, M. L. (1986). Reliability and validity in qualitative research. Newbury Park, CA: Sage.

Koro-Ljungberg, M. (2008). Validity and validation in the making in the context of qualita- tive research. Qualitative Health Research, 18, 983–989. https://doi.org/10.1177/ 1049732308318039

426 Walther, Sochacka, Benson, Bumbaco, Kellam, Pawley, & Phillips

Koro-Ljungberg, M., & Douglas, E. P. (2008). State of qualitative research in engineering education: Meta-analysis of JEE articles, 2005–2006. Journal of Engineering Education, 97, 163–174. https://doi.org/10.1002/j.2168-9830.2008.tb00965.x

Lather, P. A. (1986). Issues of validity in openly ideological research: Between a rock and a soft place. Interchange, 17, 63–84. https://doi.org/10.1007/bf01807017

Leydens, J. A., Moskal, B. M., & Pavelich, M. J. (2004). Qualitative methods used in the assessment of engineering education. Journal of Engineering Education, 93, 65–72. https:// doi.org/10.1002/j.2168-9830.2004.tb00789.x

Lincoln, Y. S. (1995). Emerging criteria for quality in qualitative and interpretive research. Qualitative Inquiry, 1, 275–289. https://doi.org/10.1177/107780049500100301

Lohmann, J. R. (2008). Refining our focus. Journal of Engineering Education, 97, 1. https:// doi.org/10.1002/j.2168-9830.2008.tb00947.x

Lopez, K. A., & Willis, D. G. (2004). Descriptive versus interpretive phenomenology: Their contributions to nursing knowledge. Qualitative Health Research, 14, 726–735. https://doi. org/10.1177/1049732304263638

Love, N. (2008). Using data to improve learning for all: A collaborative inquiry approach. Thou- sand Oaks, CA: Sage.

Malterud, K. (2001). Qualitative research: Standards, challenges, and guidelines. The Lancet, 358, 483–488. https://doi.org/10.1016/S0140-6736(01)05627-6

Mays, N., & Pope, C. (2000). Qualitative research in health care: Assessing quality in quali- tative research. BMJ, 320, 50–52. https://doi.org/10.1136/bmj.320.7226.50

Melia, K. M. (1997). Producing ‘plausible stories’: Interviewing student nurses. In G. Miller & R. Dingwall (Eds.), Context and method in qualitative research. Thousand Oaks, CA: Sage.

Miller, G., & Dingwall, R. (1997). Context and method in qualitative research. Thousand Oaks, CA: Sage.

Mishler, E. (1986). Research interviewing: Context and narrative. Cambridge, MA: Harvard University Press.

Mishler, E. (1990). Validation in inquiry-guided research: The role of exemplars in narrative studies. Harvard Educational Review, 60, 415–442. https://doi.org/10.17763/haer.60.4. n4405243p6635752

Morrow, S. L. (2005). Quality and trustworthiness in qualitative research in counseling psy- chology. Journal of Counseling Psychology, 52, 250–260. https://doi.org/10.1037/0022- 0167.52.2.250

Moss, P. A., Phillips, D. C., Erickson, F. D., Floden, R. E., Lather, P. A., & Schneider, B. L. (2009). Learning from our differences: A dialogue across perspectives on quality in education research. Educational Researcher, 38, 501–517. https://doi.org/10.3102/0013189x09348351

National Science Foundation. (2016). Research in the formation of engineers (RFE). Engineering Education and Centers. https://www.nsf.gov/funding/pgm_summ.jsp?pims_ id5503584

O’Brien, B. C., Harris, I. B., Beckman, T. J., Reed, D. A., & Cook, D. A. (2014). Standards for reporting qualitative research: A synthesis of recommendations. Academic Medicine, 89, 1245–1251. https://doi.org/10.1097/acm.0000000000000388

Pawley, A. L. (2013). Learning from small numbers of underrepresented students’ stories: Discus- sing a method to learn about institutional structure through narrative. ASEE Annual Confer- ence, Atlanta, GA. https://peer.asee.org/19030

Qualitative Research Quality: A Collaborative Inquiry 427

Pawley, A. L., & Phillips, C. M. L. (2014). From the mouths of students: Two cases of narrative analysis to understand engineering education’s ruling relations as gendered and raced. ASEE Annual Conference, Indianapolis, IN. https://doi.org/10.18260/p.24063

Perry, J. W. (2008). Acculturation. In J. E. Grusec & P. D. Hastings (Eds.), Handbook of socialization: Theory and research. New York, NY: Guilford.

Polkinghorne, D. E. (1995). Narrative configuration in qualitative analysis. International Journal of Qualitative Studies in Education, 8, 5–23. https://doi.org/10.1080/ 0951839950080103

Preissle, J. (2008). Subjectivity statement. In L. M. Given (Ed.), The Sage encyclopedia of qual- itative research methods (pp. 844–845). Thousand Oaks, CA: Sage.

Radcliffe, D. F. (2006). Shaping the discipline of engineering education. Journal of Engineer- ing Education, 95, 263. https://doi.org/10.1002/j.2168-9830.2006.tb00901.x

Reason, P. (1988). Human inquiry in action: Developments in new paradigm research. Thou- sand Oaks, CA: Sage.

Reason, P., & Bradbury, H. (2007). The Sage handbook of action research: Participative inquiry and practice. Thousand Oaks, CA: Sage.

Reason, P., & Rowan, J. (1981). Issues of validity in new paradigm research. In P. Reason & J. Rowan (Eds.), Human inquiry: A sourcebook of new paradigm research. New York, NY: John Wiley.

Richards, L. (2005). Handling qualitative data: A practical guide. London; Thousand Oaks, CA: Sage.

Roe, E. M. (1992). Applied narrative analysis: The tangency of literary criticism, social sci- ence and policy analysis. New Literary History, 23, 555–581. https://doi.org/10.2307/ 469220

Roulston, K. (2010). Reflective interviewing: A guide to theory and practice. Washington, D.C.: Sage.

Seidman, I. (2006). Interviewing as qualitative research: A guide for researchers in education and the social sciences (3rd ed.). New York, NY: Teachers College Press.

Silberman, M. L. (1996). Active learning: 101 strategies to teach any subject. Boston, MA: Allyn and Bacon.

Silverman, D. (2009). Doing qualitative research. Thousand Oaks, CA: Sage. Smeyers, P. (2005). Idle research, futile theory, and the risk for education: Reminders of irony

and commitment. Educational Theory, 55, 164–183. https://doi.org/10.1111/j.0013-2004. 2005.00005.x

Smith, D. E. (1990). Texts, facts and femininity: Exploring the relations of ruling. New York, NY: Routledge.

Smith, D. E. (2005). Institutional ethnography: A sociology for people. Lanham, MD: Rowman Altamira.

Smith, J., Flowers, P., & Larkin, M. (2009). Interpretative phenomenological analysis: Theory, method and research. London: Sage.

Smith, L. T. (1999). Decolonizing methodologies: Research and indigenous peoples. London: Zed Books.

Sochacka, N. W., Walther, J., Wilson, J., & Brewer, M. (2014). Stories ‘told’ about engineering in the media: Implications for attracting diverse groups to the profession. Frontiers in Engineering Education Conference, Madrid, Spain. https://doi.org/10.1109/fie.2014. 7044009

428 Walther, Sochacka, Benson, Bumbaco, Kellam, Pawley, & Phillips

St. Pierre, E. A. (2014). A brief and personal history of post qualitative research toward “post inquiry.” Journal of Curriculum Theorizing, 30, 2–19. https://journal.jctonline.org/index. php/jct/article/view/521/stpierre.pdf

Streveler, R. A., Borrego, M., & Smith, K. A. (2007). Moving from the scholarship of teach- ing and learning to educational research: An example from engineering. In D. R. Robert- son (Ed.), To improve the academy (Vol. 25, pp. 139–149). Bolton, MA: Anker.

Streveler, R. A., & Smith, K. A. (2006). Guest editorial: Conducting rigorous research in engineering education. Journal of Engineering Education, 95. https://doi.org/10.1002/j. 2168-9830.2006.tb00882.x

Tong, A., Sainsbury, P., & Craig, J. (2007). Consolidated criteria for reporting qualitative research (COREQ): A 32-item checklist for interviews and focus groups. International Journal for Quality in Health Care, 19, 349–357. https://intqhc.oxfordjournals.org/content/ intqhc/19/6/349.full.pdf

Tracy, S. J. (2010). Qualitative quality: Eight “big-tent” criteria for excellent qualitative research. Qualitative Inquiry, 16, 837–851. https://doi.org/10.1177/1077800410383121

Walther, J., Pawley, A. L., & Sochacka, N. W. (2015). Exploring ethical validation as a key consideration in interpretive research quality. ASEE Annual Conference, Seattle, WA. https://doi.org/10.18260/p.24063

Walther, J., & Sochacka, N. W. (2014a). Qualifying qualitative research quality (the Q3 pro- ject): An interactive discourse around research quality in interpretive approaches to engineering education research. Frontiers in Engineering Education Conference, Madrid, Spain. https://doi.org/10.1109/fie.2014.7043988

Walther, J., & Sochacka, N. W. (2014b). Research quality workshop. Workshop conducted at Purdue University, West Lafayette, IN.

Walther, J., & Sochacka, N. W. (2015). Building capacity and social capital around interpretive research quality. ASEE Annual Conference, Seattle, WA. https://doi.org/10.18260/p. 23642

Walther, J., Sochacka, N. W., & Kellam, N. N. (2012). Qualifying qualitative research quality: A conversation for engineering education researchers. Workshop conducted at Swinburne University, Melbourne, Victoria.

Walther, J., Sochacka, N. W., & Kellam, N. N. (2013a). Quality in interpretive engineering education research: Reflections on an example study. Journal of Engineering Education, 102, 626–659. https://doi.org/10.1002/jee.20029

Walther, J., Sochacka, N. W., & Kellam, N. N. (2013b). Research quality workshop. Work- shop conducted at the University of Georgia, Athens, GA.

Weiss, R. S. (1994). Learning from strangers: The art and method of qualitative interview stud- ies. New York, NY: The Free Press.

Wormley, D. N. (2006). A year of dialogue focused on engineering education research. Journal of Engineering Education, 95, 179. https://doi.org/10.1002/j.2168-9830.2006. tb00890.x

Yorks, L., & Kasl, E. (2002). Learning from the inquiries: Lessons for using collaborative inquiry as an adult learning strategy. New Directions for Adult & Continuing Education, 2002(94), 93–104. https://doi.org/10.1002/ace.63

Qualitative Research Quality: A Collaborative Inquiry 429

Authors Joachim Walther is an associate professor in the College of Engineering and founding

director of the Engineering Education Transformations Institute at the University of Geor- gia, 597 D. W. Brooks Drive, Athens, GA 30602; [email protected].

Nicola W. Sochacka is a research scientist in the College of Engineering and the associate director of the Engineering Education Transformations Institute at the University of Geor- gia, 597 D. W. Brooks Drive, Athens, GA 30602; [email protected].

Lisa C. Benson is an associate professor of engineering and science education at Clemson University, with a joint appointment in bioengineering, M12 Holtzendorff Hall, Clemson University, Clemson, SC 29634-0956; [email protected].

Amy E. Bumbaco is an academic advisor in the First Year Programs Department at Embry-Riddle Aeronautical University, 600 S. Clyde Morris Blvd, Daytona Beach, FL 32114; [email protected].

Nadia Kellam is an associate professor in the Polytechnic School within the Ira A. Fulton Schools of Engineering at Arizona State University, Peralta Hall 330F, 7171 E. Sonoran Arroyo Mall, Mesa, AZ 85212; [email protected].

Alice L. Pawley is an associate professor in the School of Engineering Education and affil- iate faculty with the Women, Gender, and Sexuality Studies Program and Division of Envi- ronmental and Ecological Engineering at Purdue University, 701 W. Stadium Ave, West Lafayette IN 47907; [email protected].

Canek M. L. Phillips is a postdoctoral researcher working remotely for the Polytechnic School at Arizona State University starting January 2017, 1007 Catalpa St., Austin, TX 78702; [email protected].

430 Walther, Sochacka, Benson, Bumbaco, Kellam, Pawley, & Phillips

Copyright of Journal of Engineering Education is the property of Wiley-Blackwell and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.