Research Methods Literature Review PLEASE SEE DOCUMENTS NEEDED ATTACHED!!!

See discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/284790946

Research Design for Mixed Methods: A Triangulation-based Framework and

Roadmap

Article  in  Organizational Research Methods · November 2015

DOI: 10.1177/1094428115610808

CITATIONS

98 READS

13,587

3 authors, including:

Some of the authors of this publication are also working on these related projects:

Justification shift and uncertainty: Why are low-probability near misses underrated against organizational routines? View project

Scott F. Turner

University of South Carolina

14 PUBLICATIONS   659 CITATIONS   

SEE PROFILE

Richard M. Burton

Duke University

175 PUBLICATIONS   3,261 CITATIONS   

SEE PROFILE

All content following this page was uploaded by Scott F. Turner on 21 December 2015.

The user has requested enhancement of the downloaded file.

Research Design for Mixed Methods: A Triangulation-Based Framework and Roadmap

Scott F. Turner Darla Moore School of Business

University of South Carolina Columbia SC 29208

803-777-5973 [email protected]

Laura B. Cardinal C. T. Bauer College of Business

University of Houston Houston, TX 77204

713-743-2559 [email protected]

Richard M. Burton Fuqua School of Business

Duke University Durham, NC 27708

919-660-7700 [email protected]

Organizational Research Methods November 25 2015

Published online before print

Research Design for Mixed Methods: A Triangulation-Based Framework and Roadmap

Abstract All methods individually are flawed, but these limitations can be mitigated through mixed methods research, which combines methodologies to provide better answers to our research questions. In this study, we develop a research design framework for mixed methods work that is based on the principles of triangulation. Core elements for the research design framework include theoretical purpose, i.e., theory development and/or theory testing; and methodological purpose, i.e., prioritizing generalizability, precision in control and measurement, and authenticity of context. From this foundation, we consider how the multiple methodologies are linked together to accomplish the theoretical purpose, focusing on three types of linking processes: convergent triangulation, holistic triangulation, and convergent and holistic triangulation. We then consider the implications of these linking processes for the theory at hand, taking into account the following theoretical attributes: generality/specificity, simplicity/complexity, and accuracy/inaccuracy. Based on this research design framework, we develop a roadmap that can serve as a design guide for organizational scholars conducting mixed methods research studies. Keywords research methods, mixed methods, triangulation, validity, research strategies, theory development, theory testing Acknowledgements The authors wish to thank Associate Editor David Ketchen and three anonymous reviewers for their detailed and thoughtful feedback. Portions of this research were presented at the 75th Annual Meeting of the Academy of Management and the 35th Annual International Conference of the Strategic Management Society.

1

Research Design for Mixed Methods: A Triangulation-Based Framework and Roadmap

As a basic concept in the social sciences, triangulation refers to using multiple, different approaches to generate better understanding of a given theory or phenomenon (Burton & Obel, 2011; Singleton & Straits, 1999). While early work in the social sciences focused on the role of triangulation in measurement, i.e., using different methods to assess construct validity (Campbell & Fiske, 1959), more recent attention has emphasized how triangulation can contribute to our understanding more broadly. Mixed methods research is based on the idea of heightened understanding through methodological triangulation (Denzin, 1970, 2012; Molina-Azorin, 2007; Torrance, 2012). For this form of triangulation, McGrath (1982) describes the use of multiple methodological strategies, which represent generic classes of research strategies for gaining knowledge about a research question; examples include laboratory experiments and field studies. McGrath (1982) discusses the importance of methodological triangulation from the view that individually “all research strategies and methods are seriously flawed” (p. 70), but in combination, the use of multiple methods that do not share the same failings can enhance what is known about a given research question. In similar fashion, Weick (1969) states, “we typically need multiple methods, or techniques which are imperfect in different ways. When multiple methods are applied, the imperfections in each method tend to cancel one another” (p. 21).

Although triangulation can take place within a given methodology, such as using different

subject populations for a lab experiment (e.g., Bateman & Zeithaml, 1989) or using different platforms for computer simulation (Axtell, Axelrod, Epstein, & Cohen, 1996), mixed methods research focuses on triangulation that spans multiple methodologies (Denzin, 1970, 2012; Singleton & Straits, 1999). Jick (1979) argues that methods-spanning forms of triangulation are on the complex end of a continuum of triangulation design while within-method forms are on the simple end. For triangulation that spans methodologies, much attention has been directed to convergent triangulation, which is based on the idea that knowledge develops by obtaining convergence in substantive findings across a diverse set of methodologies (McGrath, Martin, & Kulka, 1982). This reflects the view that using multiple methods produces more valid results as the strengths of one method can offset the limitations of another method (Jick, 1979; Scandura & Williams, 2000).

While scholarly advice has been offered for many forms of convergent triangulation

(McGrath et al., 1982; Pedhazur & Schmelkin, 1991), less attention has been directed to the holistic form of triangulation. This represents an important omission because holistic triangulation offers the promise of capturing “more complete, holistic, and contextual portrayal of the unit(s) under study” and enriching “our understanding by allowing for new or deeper dimensions to emerge” (Jick, 1979, pp. 603-604). Relative to convergent triangulation, which emphasizes knowledge development in the search for agreement, the holistic form asserts that divergence can be just as important as convergence in the accrual of knowledge. This is based in part on the idea that certain methods are uniquely capable of “seeing” particular aspects of a phenomenon or “judging” particular attributes of a theory.1 Thus, through holistic triangulation, 1 A parallel discussion of convergence versus divergence can be found with respect to performance management in the organizational behavior literature. For example, we might not expect to find convergence when performance of an individual’s work is evaluated by peers versus a supervisor because these sources have different vantage points and can provide qualitatively different types of information. We wish to thank a reviewer for highlighting this observation.

2

scholars can generate more insight for a theory or develop more complete understandings of a given phenomenon through the use of multiple methodologies (Denzin, 2012; Gibbert & Ruigrok, 2010; Shah & Corley, 2006).

While scholars have made the case for the benefits and promise of methodological

triangulation in the organizational sciences (e.g., Jick, 1979; McGrath et al., 1982), in a related review, Scandura and Williams (2000) found that triangulating across methods is still rare in the organizational sciences. There are a number of reasons for the scarcity of studies employing methodological triangulation. One core reason is that there is limited guidance available for organizational scholars regarding how to design mixed methods research studies and particularly for how to design holistic triangulation studies. Other important challenges include the time- consuming nature of the work, the need to assemble and coordinate researcher expertise across different methodological areas, and the demands associated with publishing mixed methods research.

Thus, the objective of this article is two-fold: the first objective is to develop a research

design framework for mixed methods work that is based on the principles of triangulation, and then based on that framework, the second objective is to provide a roadmap to guide organizational scholars interested in conducting mixed methods research studies. Our emphasis is on research that combines methods that stand more or less on equal footing, i.e., equivalent methods, rather than situations in which one method is dominant and another is in a supporting role (Molina-Azorin, 2007). While triangulation in research methodology can encompass a number of aspects, such as using multiple sources of data and examining multiple empirical settings (Scandura & Williams, 2000), as a methods foundation, our study draws from the research strategies presented in McGrath (1982). In this study, we focus on the following research strategies: archival records, 2 case studies, computer simulations, experimental simulations, field experiments, formal theory (mathematical), interviews, lab experiments, and surveys. In the article, we often use the research strategy and method terms interchangeably.

For the first objective, our research design framework is based on several core elements,

which are captured in Figure 1. The first two elements focus on theoretical and methodological purpose – that is, the theoretical purpose for the research, and the purposes for the methods that are utilized in that research. The third element focuses on how the multiple methodologies are linked together given theoretical and methodological purpose and considers the implications of the linking process for the theory at hand.3 For our second objective, we draw from this research design framework to develop a roadmap that provides guidance for organizational scholars regarding how to design and implement triangulation-based mixed methods research studies. We conclude with further examination of two key issues involved in the design and implementation

2 Some scholars have placed emphasis on archival records as a distinct research strategy that involves the use of archival sources of data (Austin, Scherbaum, & Mahlman, 2002; Scandura & Williams, 2000; Singleton & Straits, 1999). While Scandura and Williams (2000) incorporated this approach under the label of field studies that use secondary data, we have adopted the archival records label used in Singleton and Straits (1999), given its distinction from many field studies. 3 Given our emphasis on methods at the level of research strategies (McGrath, 1982) and their relationship with theoretical purpose, our framework focuses more on design and to a lesser extent on measurement and analysis among the core methodological topics of design, measurement, and analysis (e.g., Aguinis, Pierce, Bosco, & Muslin, 2009).

3

of mixed methods studies: how to select a set of methodologies that increases validity by offsetting the limitations inherent to the individual methodologies, and how to overcome practical challenges involved in conducting mixed methods research. This discussion considers important assumptions and challenges associated with triangulation-based mixed methods research that influence whether and how this research can produce heightened understanding.

*** Insert Figure 1 Here ***

The Foundation for Triangulation: Theoretical Purpose and Methodological Purpose

Theoretical purpose. To develop a research design framework for mixed methods research involving triangulation, we first need to consider the theoretical purpose for the research study, which often ties closely to the research question (Burton & Obel, 1995; Molina-Azorin, 2007). Theoretical purpose refers to what researchers aim to accomplish in the study as it pertains to theory; such aims include theory generation, theory elaboration, and theory testing (Lee, Mitchell, & Sablynski, 1999). For mixed methods studies that focus on convergent triangulation, the theoretical purpose may involve assessing the validity of an explanation by conducting multiple tests using methodologies that are flawed in different ways, or it may involve starting with one method for theory development,4 and concluding with another method for theory testing. Thus, theory testing often plays an important role in convergent triangulation research. By contrast, for studies focused on holistic triangulation, the theoretical purpose often centers on the development of theory.

Methodological purpose. Following theoretical purpose, scholars pursuing mixed methods

research need to give careful attention to the purposes for which they are utilizing particular methods. McGrath (1982, 1995) suggests that scholars select research strategies based on the extent to which a given method can accomplish several objectives: (1) its ability to maximize generalizability with respect to populations (e.g., actors), (2) its precision in control and measurement of variables related to behaviors of interest, and (3) its ability to provide authenticity of context for the observed behaviors. Accomplishment of these respective objectives enables researchers to contend with threats to different types of validity. For example, a method that offers precision in control and measurement of variables is well-suited for addressing threats to internal validity, which refers to the validity of inferences regarding whether covariation between an independent and dependent variable results from a causal relationship, as opposed to the possibility that other extraneous variables are responsible for the outcome. Alternatively, a method that can maximize generalizability with respect to populations can contend with threats to external validity, which refers to the validity of inferences about whether the causal relationship is generalizable to other subject populations, settings, and time periods (Shadish, Cook, & Campbell, 2002). As such, the selection of research methodologies is important because it influences the answers to our research questions in that the types of evidence, nature of claims, and confidence in inferences depend considerably on which methods are being employed.

4 For parsimony in presentation, we use “theory development” to encompass theory generation and theory elaboration. In the former case, researchers ideally begin with no theory, while in the latter case, the objective is to advance preexisting theory; in both cases, though, the objective is development of theory (Eisenhardt, 1989; Lee et al., 1999).

4

According to McGrath (1982, 1995), certain research strategies are often well-suited to one of these purposes; for example, lab experiments enable precision in control and measurement of variables, while field studies (i.e., case studies) provide an authentic context within which the behaviors of actors are observed. While this view emphasizes that individual methods tend to have particular strengths and weaknesses, it is also important to keep in mind that certain strategies can be utilized to accomplish different methodological purposes. For example, computer simulations can be used to maximize generalizability with respect to populations, or they can be used as laboratories for control and manipulation of variables (Burton, 2003; Davis, Eisenhardt, & Bingham, 2007; Harrison, Lin, Carroll, & Carley, 2007). As another example, archival records may be used to enhance precision in control and measurement of variables (e.g., focusing on measures well-suited for a particular group, firm, or industry), maximize generalizability with respect to populations (e.g., seeking breadth across groups, firms, or industries), or capture behaviors of interest that have taken place within an authentic context. Table 1 highlights which methodological purposes are commonly fulfilled by particular research strategies.

*** Insert Table 1 Here ***

Overall theoretical purpose (i.e., theory development, theory testing) and methodological

purpose (i.e., generalizability, precision in control/measurement, authenticity of context) provide an important foundation for triangulation-based research design.

Linking Methods in Light of Theoretical and Methodological Purposes

In the previous section, we described how theoretical and methodological purposes are at the foundation of triangulation-based research. In this section, we focus on the process of linking, which is a central way in which scholarly understanding in a given area can be extended through the use of mixed methods.

In the design of triangulation-based studies, linking refers to the process by which multiple

research strategies are brought together within a research study to realize the theoretical purpose for that work. We argue that there are three core processes for linking research strategies in triangulation-based research: (a) linking processes focused on convergent triangulation, (b) linking processes focused on holistic triangulation, and (c) linking processes focused on convergent and holistic triangulation. As described next, what distinguishes these processes is how the set of methodologies relate to the theoretical purpose for the study and how the researcher leverages methodological purpose in determining the complement of methodologies utilized in the study.

Linking processes focused on convergent triangulation are commonly of two basic types; one

type focuses on theory testing, with two research strategies being used for the purpose of testing the same theory, while the other type spans both categories of theoretical purpose, with one research strategy focused on theory development and one focused on theory testing.5 For both types of convergent triangulation, the researcher utilizes multiple methods to assess whether

5 For ease of explanation, we typically refer to linking processes as if two methods were being utilized, although researchers could clearly use more than two methods in a mixed methods research study.

5

convergent results are observed across the methods. Linking processes for holistic triangulation are focused on theory development and span at

least two categories of methodological purpose (e.g., prioritizing generalizability, prioritizing control/measurement, prioritizing authenticity of context). For holistic triangulation, the researcher utilizes multiple research strategies because he or she expects to learn different and unique things from them.6

Linking processes focused on convergent and holistic triangulation span both categories of

theoretical purpose (i.e., theory development, theory testing) and involve at least two categories of methodological purpose. For convergent and holistic triangulation, the researcher utilizes multiple research strategies because he or she expects to obtain better understanding through opportunities for the methods to show convergent results in certain areas and through unique perspectives/angles that one or more of the individual methods can provide in other areas. Figure 2 provides a number of examples of these linking processes.

*** Insert Figure 2 Here ***

Linking of methodologies in convergent triangulation studies. For studies focused on

convergent triangulation, the objective for the researcher is to demonstrate more valid results by finding agreement across different research strategies (Jick, 1979; McGrath, 1982). In Figure 2, linking processes focusing on convergent triangulation are reflected in horizontal arrows within theory testing (i.e., testing theory using two complementary methods) and in arrows that begin in theory development and conclude in theory testing (i.e., developing and testing theory).

Good examples of convergent triangulation studies that test theory using two complementary

methods can be found in Bardolet, Fox, and Lovallo (2011) and Morris and Moore (2000). Bardolet et al. begins with the observation that strategy scholars commonly explain subsidization across business units in multi-business firms with agency theory (e.g., principal-agent conflicts/negotiations). The researchers then propose that the cross-subsidization phenomenon may be explained by a simpler argument that corporate executives have cognitive biases in favor of even resource allocations. To test this argument, they initially turn to regressions using archival data, specifically using Compustat data to generate a large dataset that spans many industries. The researchers found support for their argument with regression analyses of the archival data, but they were also aware that the archival-based analyses did not allow for sufficient controls to rule out common explanations based on agency theory. Thus, they next turned to a series of lab experiments as a complementary method for theory testing, which offered greater capacity to control for such possibilities and provided consistent results, which offered greater support for their argument.

As another example of convergent triangulation, Morris and Moore (2000) examined how

learning was influenced by counterfactual thinking, and how organizational accountability influenced this process. The researchers distinguished between upward counterfactual 6 An alternative process could be one of holistic triangulation with the aim of theory testing. For example, multiple methods might be used for their unique ability to test different aspects of the same theory (e.g., Flynn & Staw, 2004). We do not consider such a process here because of the emphasis on convergence with theory testing in mixed methods research (i.e., multiple methods providing different tests for the same question).

6

comparisons, which compare actual events to better possible alternatives, and downward ones, which foster thinking relative to worse possible alternatives. They argue that upward counterfactual comparisons promote learning because they focus attention on factors preceding outcomes that might be changed in order to produce better outcomes, which provides a frame for learning how to act in similar situations in the future, and they argue that organizational accountability inhibits upward counterfactual comparisons because it restricts complex and self- critical thinking. In the study, the researchers tested these predictions through two complementary methods: (a) statistical analyses using archival data of experienced pilots’ assessments of near-miss aviation incidents and (b) an experimental simulation in which undergraduate students operate a flight simulator.7 The analyses initially focused on the archival data from experienced pilots because it offered an authentic context as described in Table 1, whereas the researchers described tests of similar processes in prior studies as limited to academic and lab settings. While these analyses supported their predictions, the regressions based on archival data could only rule out certain alternative explanations, so the researchers next turned to an experimental simulation using a flight simulator that afforded greater precision in control and measurement.

As a good example of the alternative form of convergent triangulation, i.e., developing and

testing theory, consider Cohen and Klepper (1996). The study examines the effect of organizational size on the allocation of R&D effort between process and product innovation. The theoretical purpose of the study was first to develop theory through a mathematical model that explains the influence of size on the allocation of R&D effort and then to test the predictions derived from the model with regression analyses using archival data. From the view of methodological purpose, both methods prioritize generalizability with respect to populations of organizations. Specifically, the researchers develop a theoretical model that they expect to hold for organizations in a variety of industries, and they test the predictions using archival data from the Federal Trade Commission that includes businesses in 36 different manufacturing industries.

As another example of this form of convergent triangulation, Grant, Berg, and Cable (2014)

examine how self-reflective job titles affect emotional exhaustion. Beginning with the purpose of theory development, the researchers conducted a case study at the Make-A-Wish Foundation. Based on their analyses of interviews, observations, and archival sources, the researchers discovered that employees’ selecting their own job titles contributed to less emotional exhaustion, and they proposed three mechanisms (self-verification, psychological safety, and external rapport) to explain this effect. While the case study suited the methodological purpose of authenticity of context, it was also considered to be an extreme case of employee burnout, such that the researchers described selecting an alternative research strategy to test their inductively-derived theory in a way that could support greater generalizability. Their choice of a second research strategy – a field experiment – also contributed by offering greater precision in control and measurement. Thus, they conducted a field experiment in a large health services organization, which included pre- and post-intervention surveys, and the results supported their proposition that choosing self-reflective titles reduces emotional exhaustion for employees. 7 McGrath (1995) describes the experimental simulation research strategy as one in which “the researcher attempts to achieve much of the precision and control of the laboratory experiment but to gain some of the realism (or apparent realism) of field studies. This is done by concocting a situation or behavior setting or context, as in the laboratory experiment, but making it as much like some class of actual behavior setting as possible.” (p. 157). McGrath (1995) also provides the use of flight simulators as a good example of this type of research.

7

However, the researchers found that only two of the three proposed mechanisms – self- verification and psychological safety – had a significant mediating effect in the field experiment.

Linking of methodologies in holistic triangulation studies. For studies focused on holistic

triangulation, the objective is to obtain fuller understanding through unique insights/perspectives gained from the different research strategies being utilized; according to Jick (1979), such triangulation allows for more complete and contextual portrayal of the units under study. In Figure 2, linking processes focusing on holistic triangulation are reflected in horizontal arrows, with theory development as the theoretical purpose.

Cardinal, Turner, Fern, and Burton (2011) provide a good example of holistic triangulation.

With the purpose of theory development, the researchers wanted to establish a more comprehensive understanding of organizing for innovation by exploring a traditional contingency-based model of product development. They argued that different technological environments have different features that require different information processing, such that better performance would be observed from project designs that fit the information-processing requirements of the environment. The researchers initially turned to computer simulation methodology given its precision and capacity for manipulation of design and environmental conditions as outlined in Table 1. The simulations produced unexpected results that only partly aligned with the propositions, calling into question a traditional contingency perspective of designing for product development; specifically, rather than finding superior performance for project designs that fit their respective technological environments, the simulation results suggested that organizations face substantial performance trade-offs (e.g., speed of development vs. cost) with project design. The researchers then turned to multiple case studies as a complementary methodology for authenticity of context that focused on six project development projects at three firms competing in the three focal technological environments and included 75 interviews with project leaders and team members across the six projects; this methodology offered authenticity of context for the study participants and enabled the researchers to take a deeper look into how technological environments affect performance priorities, project design, and performance.

As another example of holistic triangulation, Anteby (2010) utilized multiple methods for the

purpose of theory development, focusing on the broad question of what makes markets moral, with particular interest in understanding variations in legitimacy for trades involving the same class of goods. The empirical setting was trade in cadavers in the State of New York and involved multiple sources of data, including interviews with participants in the cadaver trade, direct observations at supplying and recipient organizations, and archival data documenting the exchange of cadavers in the state of New York. After focusing initially on the interviews and direct observations for the purpose of understanding how commerce in cadavers worked, Anteby then turned to the archival data to construct a map that identified which organizations were involved in supplying and receiving cadavers in New York and documented the flow of cadavers in the state. The interviews and observations facilitated authenticity of context for the study participants as well as for the researcher and offered a deep look into how the trade operated; in turn, the archival data provided a precise accounting of the involved organizations and the flow of cadavers among them. Through these methods, Anteby was able to develop a practice-based model of trade legitimacy, focusing the distinctions between proper and improper means of commerce made by professionals in the cadaver trade.

8

Lin, Zhao, Ismail, and Carley (2006) also provide a nice example of the holistic triangulation

linking process. The objective of this study was one of theory development, with a particular focus on how to design organizations for better performance during crises. Drawing from organizational theory, and specifically using a neo-information processing perspective, the researchers developed a computer simulation model for predicting the performance of organizations under crisis conditions. Next, turning to archival data, they collected data on the relative effectiveness of 80 organizations faced with actual crises; these archival data were used both to seed the contextual conditions for simulation predictions of organizational performance and as a source of data for the actual performance of the organizations. Comparisons of the simulation-predicted organizational performance with the actual-observed organizational performance pointed to areas where the theory needed to be extended, and the researchers then returned to the simulation model to conduct a number of “what if” experiments for that purpose.

Linking of methodologies in convergent and holistic triangulation studies. For studies that

encompass both convergent and holistic triangulation, the objective is to assess the validity of a theory or set of results by examining across the research strategies, including the extent of agreement across the research strategies (the convergent aspect) and their capacity to offer unique perspective that can provide a more complete understanding of a phenomenon or theory (the holistic aspect). On Figure 2, linking processes focusing on convergent and holistic triangulation are reflected in arrows that span cells vertically and horizontally.

As one example of convergent and holistic triangulation, in their study of knowledge sharing

within networks, Dyer and Hatch (2006) used regression analyses of survey data for theory testing, followed by theory development through interviews. Drawing on evolutionary economics, the researchers argued that a buyer can generate competitive advantage by sharing knowledge with its supplier network. Following development of their argument, the researchers’ initial objective was to test the following prediction: a buyer that transfers more knowledge to its supplier network will develop the capabilities of their suppliers, such that the operational relationship between the buyer and its suppliers will be more productive (i.e., creation of relative competitive advantage). This test was conducted using survey data obtained from US-based suppliers for Toyota, namely, a methodological approach offering precision in measurement as well as an authentic context as highlighted in Table 1. Consistent with the principle of convergent triangulation, the hypothesis was supported by the analyses of survey data as well as by interviews conducted with employees at US-based suppliers for Toyota. Next, and consistent with the principle of holistic triangulation, the researchers sought to develop theory to explain why the created advantage could be sustained given that suppliers could share the knowledge provided by a focal buyer with its other buyers such that the relative competitive advantage could dissipate. For this aspect, Dyer and Hatch used qualitative analysis of the interviews with employees at US-based suppliers for Toyota, which offered an authentic context; this method provided rich insight and nuance for understanding why the relative competitive advantage could be sustained.

In addition, Nguyen and Nguyen (2008) provide a good example of convergent and holistic

triangulation in their study of Vietnamese entrepreneurs. The researchers were interested in examining the psychological values and motivations for establishing a new business, with particular interest in understanding differences between Vietnamese and Western entrepreneurs.

9

Their initial research strategy was focused on theory testing. Specifically, the researchers used a survey approach with a sample of 117 Vietnamese entrepreneurs to test three core hypotheses. The survey was administered in person in an effort to maximize the response rate given cultural views of research. Only the first hypothesis was fully supported, which focused on entrepreneurs’ values reflecting both personal and collective orientations. For the second and third hypotheses, the researchers found that the entrepreneurs did not fully share the values of their American counterparts as predicted and that neither values nor motives affected venture success. Next, the researchers turned to in-depth interviews with female Vietnamese entrepreneurs to understand what motivates these entrepreneurs to start their businesses and also to learn how those motives change over time. For this part, they employed a grounded study approach using qualitative field data to develop a conceptual framework. From the view of convergent triangulation, the researchers found support that was consistent with the survey-based results, namely, support for non-economic motives such as continuous learning, promotion of Vietnamese culture, and helping the disabled. In addition, and consistent with the principles of holistic triangulation, they discovered that over time, economic motives for founding the business gave way to more altruistic motives after the business had stabilized or was successful.

Ely (1994) provides another example of convergent and holistic triangulation. This study

examined how the representation of women in upper management affects the hierarchical and peer relationships that form among women in organizations. Using the empirical context of law firms, the researcher created four matched pairs of law firms, where each pair of organizations included one firm with male-dominated upper management and one firm with more gender balance in upper management. Social identity theory and organizational demography research were drawn upon to develop the arguments and predictions. Ely then identified a sample of 30 women associates from the eight law firms and conducted two interviews with each of the 30 associates. For her objective of theory testing, Ely utilized two different methods focused on the same sample of associates. First, as a way to preserve authenticity of context for the participants, the interviews were content analyzed to identify core themes pertaining to the hypotheses, and regression analyses were performed on the themes related to the hypotheses. Second, Ely turned to a survey as a way to provide more precise measurement; using the themes emerging from the interview transcripts, she developed for a survey that was completed by each of the interviewed associates and then performed regression analysis on the survey data. Consistent with the principle of convergent triangulation, the results were largely supported across the statistical analyses of the content-analyzed interviews data and the survey data. In addition, Ely was able to revisit the original interview transcripts for further interpretation and insight, which is consistent with the principle of holistic triangulation and contributed to theory development by offering more depth and nuance for understanding the why behind the focal relationships.

As demonstrated by this set of mixed methods research examples, scholars can link multiple,

complementary methods in different ways to generate better understanding or obtain a more valid answer to a specific research question. By understanding the different strengths and weaknesses of particular research strategies as summarized in Table 1, scholars conducting mixed methods research can offset some of the limitations inherent in a single-method study.

Implications for Theory from Linking Processes

In the previous section, we examined different linking processes by which strategy and

10

organizational scholars triangulate across research strategies when conducting mixed methods research. In this section, we consider the implications that these processes have for the attributes of the theory at hand.

In particular, we draw from the work of Thorngate (1976) and Weick (1979), which

considers theoretical explanations in the social sciences with respect to the following attributes: (a) generality versus specificity in domain scope, (b) simplicity versus complexity in parameters, and (c) accuracy versus inaccuracy in prediction. The first attribute, generality versus specificity, refers to the range of phenomena for which the theoretical explanation holds, with general explanations being preferred to specific ones because they have application across a broader range of settings. The second attribute, simplicity versus complexity, refers to how parsimonious the theoretical explanation is, which often reflects how many variables are needed for the explanation; from this view, parsimonious or simple explanations are considered more elegant as fewer variables are required to explain a phenomenon. The third attribute, accuracy versus inaccuracy, refers to how well the theoretical explanation captures the actual state of a given phenomenon; accuracy is often assessed by how well the predictions derived from the theoretical explanation correspond with the actual state. Thorngate emphasizes that while it is desirable to maximize each of these attributes, there are trade-offs among them, which he captures in an impostulate for theory: “it is impossible for an explanation of social behavior to be simultaneously general, simple, and accurate” (p. 126). In recognition of these trade-offs, Weick encourages scholars to accept that “at most only two of those three virtues can be realized” (p. 36) and to focus attention on work that seeks to maximize at most two of the attributes (e.g., general and simple) while being particularly cognizant of research by others that addresses the foregone attribute (e.g., accuracy).

We emphasize the Thorngate/Weick work because it addresses important theoretical

attributes and highlights the presence of inherent trade-offs among them. In addition, the criteria are applicable for theory even in its earliest stages of development, which is often the case for mixed methods research studies with the objective of theory development. For evaluation of more established theory, we highlight that other theoretical frameworks (e.g., Bacharach, 1989) are also valuable.8 While Thorngate (1976), Weick (1979), and Bacharach (1989) emphasize many similar properties, such as generality, parsimony, and predictive accuracy, the latter is a more comprehensive framework for evaluating established theory according to a range of issues for variables, constructs, and relationships, and one in which testing plays a prominent role.

Next we examine how the processes that we discussed previously shape attributes of a theory

using several of our examples of linking processes for illustrative purposes. Convergent triangulation. Proponents of convergent triangulation have stressed that greater

validation for a theory is produced when it has been subject to multiple tests using different methods and yields consistent results (Denzin, 1970; Jick, 1979; McGrath, 1995). In this study, we have highlighted two different forms of linking processes focused on convergent triangulation. The first form focuses on testing theory using two complementary research 8 While the present study focuses on work by Thorngate (1976) and Weick (1979), we encourage those developing theory in mixed methods research studies to be mindful of the Bacharach (1989) framework; while the focal theory may not be developed enough for an evaluation based on such a framework, awareness of the corresponding elements can help in establishing a solid theoretical foundation.

11

strategies. For example, consider the two research strategies linked in Morris and Moore (2000). Recall

that the researchers were interested in understanding how learning is affected by counterfactual thinking and how this process is influenced by organizational accountability. For the first test, they assessed predictions derived from theory using regressions of archival data in which experienced pilots reflected on near-miss aviation incidents. This method offered an authentic context for the study participants and helped the researchers to rule out some alternative explanations and address certain potential confounds (e.g., incident severity). But there were other aspects that could not be accounted for with the archival data, such as the possibility of motivational differences between pilots with organizational accountability and those without such accountability. Thus, Morris and Moore turned to an experimental simulation involving undergraduate students operating a flight simulator, which enabled them to randomly assign participants to conditions, and helped them to eliminate plausible alternative explanations and address the potentially-confounding factors.

From the view of theory attributes, the Morris and Moore (2000) study was one focused on

theory with respect to accuracy (i.e., support for external validity from the archival method and internal validity from the experimental simulation). But little study evidence was provided to support the generality of the explanation for other settings as both methods focused on a specific and similar task: incidents for pilots during real or simulated flight experiences. So the two research strategies in this study support the accuracy of the theory, with one contributing to external validity and one to internal validity, but they offer little to support or extend understanding of the generality of the theoretical explanation.

We highlight that an alternative mix of methods might have supported different theory

attributes. For example, the researchers might have paired the initial archival-based research strategy with analyses of survey data that were less specific to the particular task and involved a broader range of participants. If consistent results were found across the two research strategies, the study would support the accuracy of predictions from the theory based upon greater external validity and would offer evidence that could support claims of a more general domain for the theoretical explanation.

The second form of convergent triangulation is one in which the first research strategy is

used for the purpose of theory development, followed by a second that tests predictions derived from the developed theory. Researchers often employ this triangulation approach as a way of demonstrating that the developed theory is strong enough to survive an initial round of empirical scrutiny. As an example, consider the linking approach in Cohen and Klepper (1996). In this study, the researchers initially developed theory using mathematical modeling, followed by testing a set of corresponding hypotheses using regressions of archival data that spanned a broad number of industries. From the view of theory attributes, the study supports a general and simple theoretical explanation; through the math model, the researchers developed a relatively simple explanation that they expect to hold for a broad number of industries, and their statistical analysis of industry-spanning archival data provided evidence to support that the explanation has breadth in domain scope. However, the study methods and results offered less support with respect to accuracy; as a research strategy for theory development, mathematical modeling often relies on strong assumptions (Adner, Polos, Ryall, & Sorenson, 2009; Debreu, 1986; Freese, 1980), and

12

the empirical test using archival data offers limited support for internal validity, including little in the way of control variables and attempts to account for alternative explanations.

Again, we note that an alternative mix of research strategies could have supported different

attributes for the theoretical explanation. For example, the authors might have followed theory development using mathematical modeling with a more focused empirical test, such as regression analyses with archival or survey data that focus on a smaller number of industries for longer periods of time with more control variables. Such an approach could offer more support in terms of accuracy through greater internal validity; but it would offer less support for the generality of the explanation and may suggest the need for more complexity in the explanation to provide accurate predictions.

Holistic triangulation. The next process, holistic triangulation, involves the use of two or

more research strategies for theory development. In particular, this form of mixed methods research aims to develop a more complete understanding of a given phenomenon, as when scholars use methods that are uniquely able to generate or elaborate certain ideas (i.e., increasing the breadth or depth of an explanation).

Consider Cardinal et al. (2011) as an example. In the study, the researchers set out to explore

a contingency-based model of product development, where project performance was a function of the fit between technological environment and project design. The researchers initially examined their propositions using computer simulation, the results of which provided some support and some contradictions with their initial theory. They then turned to multiple case studies (Eisenhardt, 1989) to try to better understand how project performance was a function of technological environment and project design. Rather than supporting a traditional interaction- based contingency model of product development, the two methods in combination suggested a mediation-based contingency model, whereby the technological environment influenced performance priorities that affected project design and project performance. Holistic understanding was the product of case studies contributing unique insight in terms of how managers prioritized different performance dimensions in different environments, which affected project design (i.e., the first part of the mediation model), and the computer simulation method contributing unique insight in terms of how project designs affected particular dimensions of project performance (i.e., the second part of the mediation model).

With respect to theory attributes, for the first part of the mediation model, the multiple case

study method contributed to accuracy on the basis of external validity (Eisenhardt & Graebner, 2007), and for the second part of the mediation model, the computer simulation method contributed to accuracy in terms of internal validity (i.e., precise control, manipulation of variables). So both methods contributed to accuracy in different ways for different parts of the model. But as is often the case with theory development from case studies, the resulting theoretical explanation was complex (Eisenhardt, 1989). Also, while the study offered limited evidence to support domain generality for the theory, it did offer some support for generality in that the simulation platform had been validated in a variety of project management settings (Jin & Levitt, 1996; Levitt, Cohen, Knuz, Nass, Christiansen, & Jin 1994) and the theory was derived from multiple case studies (Eisenhardt, 1989; Eisenhardt & Graebner, 2007).

As an alternative, the researchers could have linked the research strategies in a different way,

13

with different implications for theory attributes. Rather than using two methods to explore two different parts of the model, the researchers might have used two methods to consider the entire model. For example, they could have initially used multiple case study methodology to explore how technological environment affects performance priorities and project design and performance, and then as a complement, conducted interviews with project managers focused on product development across a broad array of industries. In so doing, their study would have provided greater support for generality in the domain for the explanation; at the same time, the resulting theory would still be complex (i.e., based on analyses of case studies and analyses of interviews) and the overall support for accuracy in predication likely unchanged (i.e., the interviews might provide additional support in terms of external validity but likely less internal validity without the simulation).

Convergent and holistic triangulation. The third process is convergent and holistic

triangulation, which focuses on using multiple research strategies for both developing and testing theory. Similar to convergent triangulation, the researchers aim to find areas with consistent results across the methods, but like holistic triangulation, they also expect that at least one method will be able to offer unique insight that can extend understanding of the phenomenon.

In thinking about how the linking process for convergent and holistic triangulation affects

theory attributes, recall the Dyer and Hatch (2006) study of how knowledge sharing by buyers affects the performance of buyer-supplier networks. The researchers initially used regression analyses of survey data to test their theoretical argument regarding how buyers can generate competitive advantage by sharing knowledge with their suppliers. Next, they turned to interviews, first in seeking corroborating evidence for their survey-based findings and second to explore the related question of how the created competitive advantage is sustained. Overall, the theoretical explanation for how knowledge sharing led to the creation of competitive advantage was supported by the survey results and the interviews (convergent triangulation), and qualitative analysis of the interviews also developed understanding of how that created advantage was sustained (holistic triangulation). From the perspective of theory attributes, through the combination of methods, the study supported the accuracy of predictions derived from the theory (i.e., support for internal validity from the surveys and interviews). But the study provided little evidence to support the generality of the theoretical explanation, given that both the surveys and interviews used the same small sample of actors.

As we have suggested earlier, an alternative linking of research strategies could have

different implications for theory attributes. For example, the researchers might have initially used the interviews to further develop the theoretical explanation, both for creating and sustaining competitive advantage, and then used regressions of survey data from a broader sample of participants to test the core elements of the explanation. If so, the study would have greater support in terms of accuracy in prediction (i.e., internal validity from the interviews and surveys) and more support for the domain generality of the explanation by testing the predictions with a wider range of actors (i.e., external validity from the surveys). Further, both research strategies would have contributed to both aspects of the explanation (i.e., creating and sustaining competitive advantage). While starting with the interviews may increase the complexity of the theory, there would also be an opportunity to revisit the interview transcripts following the survey results for further interpretation, insight, and theory development (similar to Ely, 1994).

14

A Roadmap for Triangulation-Based Mixed Methods Research

In this section, we present a roadmap that can serve a design guide for organizational scholars seeking to pursue triangulation-based mixed methods research. Figure 3 summarizes the key steps for this roadmap.

*** Insert Figure 3 Here ***

Step 1: Decide on the Research Question

Like any good research project, an early objective for scholars is to identify an intriguing research question that has the potential to yield interesting and important insights for theory or understanding of a phenomenon. Scholars should also have clear understanding of prior work that has considered the research question, as this can play a critical role in both recognizing and articulating why the question has good prospects for yielding important contributions for the field, and oftentimes, deep understanding of prior research enables fruitful evolution and refinement in the question. From the perspective of research design, particular attention should be given to the form of the question, e.g., why, what, how. This is an extension of the idea that particular methods are well-suited for addressing particular forms of research questions; for example, case study methodology is appropriate for how and why questions (Eisenhardt & Graebner, 2007; Yin, 1994); in similar fashion, for mixed methods research, particular combinations of methods are well-suited for producing insight for certain forms of research questions.

As examples, consider some of the research questions in the exemplar studies we presented

earlier. Cohen and Klepper (1996) highlighted how it is important for policymakers, managers, and academics to understand what drives the composition of firms’ R&D efforts and focused on the question of what is the effect of firm size on the composition of R&D across product and process innovation. In work by Cardinal et al. (2011), the researchers focused attention on the issue of organizing for product development, examining the specific question of how project performance is affected by technological environment and project design.

Step 2: Determine Theoretical Intentions

Once scholars have an important research question in mind, their attention should be directed

to determining their intentions with respect to theory. Theoretical intentions encompass both theoretical purpose and theoretical attributes. For theoretical purpose, the decision is whether the research at hand is primarily oriented to theory development, theory testing, or both developing and testing theory. For theoretical attributes, the decision involves determining priorities with respect to the generalizability, simplicity, and accuracy (GAS) of the theory, given trade-offs among these attributes (Thorngate, 1976; Weick, 1979). For theory development, this pertains to the theory that researchers are trying to create or elaborate, while for theory testing, it relates to whether the theory is being tested in such a way that claims of generality, simplicity, or accuracy can be supported.

While theoretical purpose has received considerable attention in the organizational sciences,

this is less the case for theoretical attributes, although it is no less important in the pursuit of

15

heightened understanding of a phenomenon or theory. What makes theoretical attributes particularly important is the presence of often unrecognized or unacknowledged trade-offs (Thorngate, 1976). Weick (1979, p. 36) states, “failure to accept the inevitable tradeoffs implied in the GAS formulation seems to be at the heart of many current research problems. Investigators act as if they can simultaneously accomplish all three aims in their explanations, and that delusion is at the heart of much trivial, inconclusive research.” Thus, researchers need to explicitly decide which one or two among the theoretical attributes of generality, accuracy, and simplicity is their intention for the study.

As examples of theoretical intentions, we return to our set of exemplars. Cohen and Klepper

(1996) stated the first aim clearly as they proposed to develop and test a theory for the effect of firm size on the allocation of R&D effort between process and product innovation; while their second aim was more implicit, it was evident that they were seeking to develop a general theory. Cardinal et al. (2011) stated their first aim as one of theory development, specifically theory elaboration (Eisenhardt & Graebner, 2007), and they prioritized accuracy in their efforts to extend and refine extant theory.

Step 3: Select the Process for Triangulation

After determining the theoretical intentions for the focal work, the next step for scholars is to decide on the process for triangulation, which makes clear which objective(s) they are trying to accomplish through the use of a mixed methods design. As we have discussed, the three core processes are convergent triangulation, holistic triangulation, and convergent and holistic triangulation. For scholars with the intention to develop theory, there are several options. If focusing solely on theory development, holistic triangulation is the appropriate process. If researchers are seeking to combine theory development with theory testing, one option is the form of convergent triangulation where one method is used to develop theory and another method is used to test the developed theory (e.g., Grant et al., 2014), while the other option is convergent and holistic triangulation, where one method can develop theory and another method tests and extends the theory (e.g., Ely, 1994), or one method can test theory while another method provides a complementary test and then extends the theory (e.g., Dyer & Hatch, 2006). For scholars focused solely on theory testing, the appropriate process is convergent triangulation, where multiple methods are used to test the same theory (e.g., Bardolet et al., 2011).

A related issue for researchers is deciding whether to pursue the selected process for

triangulation by linking multiple methods within one research study or by using different methods across multiple research studies. In the case of linking methods within a single research study, design decisions are often made in advance of initiating the work, while scholars engaged in using different methods across research studies frequently make corresponding decisions in later stages, namely, greater clarity on complementary methods emerges as a research program evolves and progresses. For scholars considering the question of whether to link methods within a single study, a key factor should be the theoretical intentions for the study; for example, if the intention is to develop comprehensive theory, it may require a holistic triangulation process that is based in linking particular methods within a single research study; similarly, linking within a study ensures that triangulation actually happens for a given question, recognizing that research attention on particular problems can be fleeting (Staw, 1982). Scholars should also keep in mind pragmatic issues with decisions regarding linking methods within a single study, recognizing that

16

there can be value in linking methods that are related in certain ways. For example, conducting interviews and surveys with a similar sample enables time/resource efficiencies (e.g., Ely, 1994), and focusing on a similar task for methods involving archival data and design of an experiment can enable a more efficient and coherent presentation of the ideas to an audience (e.g., Bardolet et al., 2011).

Step 4: Determine the Mix of Methodologies

As the last step in triangulation-based research design, scholars need to decide which combination of research strategies will best meet their objectives, taking into account the research question, theoretical intentions, and process of triangulation. In other words, scholars should select their methodologies following their identification of the research question(s), determination of theoretical purpose (i.e., development and/or testing), prioritization among theoretical attributes (i.e., generality, accuracy, simplicity), and selection of the process for triangulation (i.e., convergent, holistic, or both). To illustrate, we look at one or two exemplar studies for each of the triangulation processes.

Convergent triangulation. For processes of convergent triangulation, consider first the

situation where a researcher wants to use one research strategy to develop theory and another to test the developed theory and has the intention to develop a general theory. Cohen and Klepper (1996) is good example of such a study. With these objectives, the scholars used mathematical modeling as a method that is conducive to development of general theory. They then used a compatible method for testing the developed theory, specifically, conducting regression analysis on archival data spanning a wide range of industries as a way to provide evidence that can support the generality of the theory. Second, consider the form of convergent triangulation in which a researcher wants to use two research strategies for purposes of testing the same theory. As a good example for this process, recall Bardolet et al. (2011). As the initial research strategy, the researchers utilized regression analyses of archival data that spanned many industries, which contributed to external validity and supported the generality of the theory. Next, as a complementary methodology, they conducted a lab experiment, which supported the accuracy of the theory through internal validity based in greater precision in control and measurement.

Holistic triangulation. For the process of holistic triangulation, scholars are focusing

exclusively on theory development, but they similarly want to determine the set of research strategies based upon their research question and theoretical intentions. For example, consider Cardinal et al. (2011). In this study, the researchers implicitly selected accuracy as their focal priority and generality as a secondary priority. As one method, they used computer simulation, which is well suited for theory development and contributes to internal validity (Burton & Obel, 2011; Davis et al., 2007); in turn, their choice of simulation platform (VDT/SimVision) was one in which the model had been calibrated using a range of actual projects across various industries, such that the findings enabled some claims as to accuracy as well as generality for the theory. As the other method, the researchers used case studies, which is also appropriate for theory development, and because their approach involved multiple case studies, the method offers support for claims of generality and accuracy for the developed theory (Eisenhardt & Graebner, 2007).

A key point here is that both research strategies were appropriate for the purpose of theory

17

development, but they also were complementary to one another. Specifically, both were helpful in developing a broad/holistic understanding of how project performance is a function of technological environment and project design, but they also offered unique contributions, with the case studies contributing more insight for the first part of the mediation model (i.e., how managers prioritized performance differently in different technological environments) and the simulations contributing more for the second part of the model (i.e., how project designs influenced project performance).

Convergent and holistic triangulation. Similar to the other processes, when scholars are

engaged in convergent and holistic triangulation, they should determine the set of research strategies in light of their research question and theoretical intentions. With convergent and holistic triangulation, scholars are focused on both theory development and theory testing, although not necessarily in that order.

Recall the Dyer and Hatch (2006) study that was primarily oriented to accuracy. The

researchers began with the intention to test predictions from their theory of how buyers create relative competitive advantage by sharing knowledge with their suppliers, and they used regression analysis of survey data as a way of assessing the validity of the theory. Then, as a complement, the researchers turned to interviews. Initially they examined whether the survey results would be confirmed by the interviews; the researchers then used the interviews to develop theory regarding a related research question (why the created relative competitive advantage is sustained).

In sum, this section draws from the triangulation-based research design framework to

identify key steps for designing mixed methods studies. In the final section, we extend this discussion to consider issues related to the validity of these studies and the practical challenges involved in conducting them.

Discussion

While scholars have long argued for the benefits of triangulation-based research (Denzin, 1970; Jick, 1979), researchers have been slow to adopt mixed methods in the organizational sciences (Creswell, 2014; Scandura & Williams, 2000). In our view, a key factor in this slow rate of adoption is the limited guidance available for organizational scholars in terms of how they can design research studies using mixed methods to accomplish their goals. In this study, we develop a research design framework for mixed methods research that is based on the principles of triangulation. Specifically, this framework draws together two areas that are paramount for mixed methods research: (a) the methodologically-based work focused on triangulation (e.g., Jick, 1979; McGrath, 1982) and (b) the theoretically-oriented research that identifies associated trade-offs (e.g., Thorngate, 1976; Weick, 1979).9 Based on the research design framework, we develop a roadmap that can serve as a design guide for organizational scholars who are interested in conducting mixed methods research. We see these developments as timely and important as there are signs of growing interest in mixed methods research in areas like strategic management, along with evidence that this type of work is impactful (Molina-Azorin, 2012). In this section,

9 We thank a reviewer for highlighting that this integration of the methodological and theoretical represents a valuable contribution in the triangulation literature.

18

we extend these ideas in two important areas: (1) from a design perspective, we offer further insight into the core factors that constitute exemplary mixed methods research studies, with particular attention on selecting an effective mix of research strategies, and (2) from a practical perspective, we address some of the key challenges associated with conducting mixed methods research in the organizational sciences.

First, a central issue in the design of mixed methods research is determining the particular set

of methods to address the research question. From a triangulation perspective, scholars like McGrath (1982, 1995) have stressed the importance of selecting research strategies with maximum divergence in terms of prioritizing generalizability across populations, precision in control and measurement, and authenticity of context. But as Jick (1979, p. 604) highlights, the quality of triangulation-based mixed methods research design hinges on a “buried” assumption: “the effectiveness of triangulation rests on the premise that the weaknesses in each single method will be compensated by the counter-balancing strengths of another... although it has always been observed that each method has assets and liabilities, triangulation purports to exploit the assets and neutralize, rather than compound, the liabilities.” Thus, thoughtful deliberation is required when selecting the combination of research strategies for mixed methods work in order to ensure that the research strategies do not share the same liabilities and that the strengths of one method serve to offset the weaknesses of another. While guidance for researchers has emphasized maximum divergence for addressing these issues (McGrath, 1982, 1995), we argue that great mixed methods research studies strike a productive balance between divergence and commonality across the methods.

Divergence. From one perspective, researchers need to seek divergence in the set of research

strategies in order to offset vulnerabilities that are inherent to the individual strategies. For example, consider a scholar who is interested in pursuing a convergent triangulation study that uses two different methods to test a theoretical explanation for a research question. By choosing a laboratory experiment as one of the methods, the researcher is well-positioned with respect to inferences about causality and threats to internal validity, but he or she is subject to considerable threats in areas like reactive measurement, which refers to the potential influence that the process of observation may have on a subject, and external validity (McGrath, 1995; Shadish et al., 2002; Singleton & Straits, 1999). Given these threats, an effective research design would pair the lab experiment with a research strategy that is strong in the latter areas, such as archival records.10 We highlight that one of our exemplars – Bardolet et al. (2011) – followed this approach in their selection of archival records and a lab experiment for their mixed methods study.

As another example, consider a scholar who is interested in conducting a convergent and

holistic triangulation study for the purpose of theory development. If he or she were to select a case study for one of the research strategies, which offers a number of advantages with respect to inductive theory development (Gioia, Corley, & Hamilton, 2012; Glaser & Strauss, 1967), the researcher would be subject to criticism in areas like internal validity (e.g., lack of control), construct validity (e.g., measurement difficulty), and difficulty in replication (Shadish et al. 2002; Singleton & Straits, 1999). For an effective mixed methods design, the researcher would be well-served to select a research strategy that in strong in the latter areas and thus capable of

10 With respect to the threat of reactive measurement, a related factor for archival records can be the extent to which subjects are aware that the records are or are likely to be made public (McGrath, 1995, p. 164).

19

offsetting the key limitations of the case study method. Good examples would be research strategies based on a survey or experiment (Singleton & Straits, 1999). We highlight that one of our exemplars, Grant et al. (2014), followed this particular approach in selecting the set of research strategies for their study. Thus, we concur with scholars’ recommendations that effective mixed methods designs seek divergence across methods, such that a method that is subject to a particular validity threat is offset by another that is strong in that respect. But we also believe that there is considerable value in seeking commonality in order to facilitate interpretation across the methods.

Commonality. While commonality has received less attention in mixed methods design, it is

no less important. This aspect focuses on the value of making incremental advances in knowledge accrual by holding certain methodological elements constant. Specifically, while extant research emphasizes the value of maximizing divergence across research strategies to offset the limitations inherent to the individual strategies, it is also important to realize the challenges that this can pose for interpreting the set of results. For example, if a researcher is pursuing convergent triangulation and finds support for the theory using a particular research strategy, setting, and sample and finds no support for it using a very different research strategy, setting, and sample, it may be hard to fully understand the implications for theory. But if the latter involved a similar setting and sample and varied only the research strategy, then the researcher is in a better position to interpret potential mixed results and contribute to theory. We frequently observed this incremental approach to mixed methods research in our exemplar studies, including the use of the same sample with different research strategies (Dyer & Hatch, 2006; Ely, 1994), and use of the same basic task with different research strategies (Bardolet et al., 2011; Morris & Moore, 2000).

As a related point, we encourage scholars to keep closely in mind their theoretical intentions

(i.e., generality, simplicity, accuracy) as they determine empirical settings and samples. Rather than trying to find settings and samples that contribute to all three of the GAS attributes and running into trade-off challenges, we would encourage researchers to try to create samples that align with their theoretical intentions and to be explicit regarding these choices in presenting the research. This is also consistent with the idea that research strategies can have multiple purposes. For example, depending on the setting and sample, archival records could be drawn upon to support the generality of a theory (Cohen & Klepper, 1996) or the accuracy of a theory (Morris & Moore, 2000). Similarly, surveys can be conducted with a broad range of participants when generalizability of populations is a priority, as McGrath (1982, 1995) suggests. But in our exemplars, we found that surveys often involved a much narrower range of participants, frequently using the same or a very similar sample of participants in a pairing of research strategies (e.g., surveys, interviews); in these instances, while the surveys did not contribute much to supporting the generalizability of a theory, they offered greater precision in measurement and control over the field-based methods that they were paired with (Dyer & Hatch, 2006; Ely, 1994), as well as pragmatic value, for example, time/resource efficiencies.

In sum, we consider how divergence and commonality in the set of methods affect validity

and interpretability and emphasize that effective mixed methods research strikes a productive balance between divergence and commonality. On the one hand, validity is enhanced through divergence when scholars select research strategies that diverge from one another in ways that can offset the inherent vulnerabilities of the individual strategies. Thus, a cautionary note for

20

mixed methods researchers is to ensure they do not select research strategies that are too similar; otherwise, we end up “driving our triangulation posts into the [same] spot” (McGrath et al., 1982, p. 111). On the other hand, interpretation is enhanced through commonality when mixed methods researchers vary only certain methodological elements, such as the research strategy, while holding other methodological elements constant (e.g., empirical setting, sample). From this perspective, mixed methods researchers also need to be careful that they do not seek maximal divergence in their mix of methodological elements and thereby weaken their capacity to interpret the set of results. This balancing among divergence and commonality points to the need for future research to give more attention to the benefits and drawbacks of triangulation-based approaches to mixed methods research, particularly from the view of variability within and across multiple methodological elements, for example, research strategies, empirical settings, samples, observers (Denzin, 2012; Scandura & Williams, 2000).

In addition, we highlight that a related issue of importance involves generativity, namely,

how the mix of methods influences the generation of new theoretical insights and uncovering of novel research questions. One can imagine ways in which greater divergence provides the heterogeneity and breadth needed for creative insight and development of novel theory; alternatively, one can consider how greater commonality may enable sharper focus and more depth of understanding, which fosters the development of new theoretical insight. While outside the scope of the present study, we encourage future research to consider this important question of how methodological divergence and commonality influence the development of emergent theory and identification of important, unexplored research questions.

Second, conducting mixed methods research poses a number of practical constraints for

researchers in the organizational sciences. For one, mixed methods research can be very time- consuming. With respect to the time constraint, we advise prospective scholars to be intentional in their selection of research strategies, not just from the perspective of validity and interpretability but also with respect to time requirements. Fortunately, the latter often go hand in hand. While we previously advocated for commonality across research strategies for interpretation value, doing so can also be pragmatic with respect to time conservation. As examples, using different methods involving a similar or identical sample can save time with respect to data collection (Dyer & Hatch, 2006; Ely, 1994), and using methods involving the same type of activity (e.g., Morris & Moore, 2000) can conserve time with respect to background preparation for conducting the study.

Another practical challenge for mixed methods research is assembling the necessary

expertise for conducting the study. By its very nature, mixed methods research requires expertise in different methodologies, with well-designed studies frequently involving very different methods. Given that methodological expertise often involves specialization, a team of researchers can be required for conducting mixed methods research. Further, in much the same way as the mix of methods requires a balance of divergence and commonality, the team of researchers itself needs a mix of experts from different areas, disciplines or schools, but they also must have overlapping expertise as well to facilitate interpretation. In addition, a spirit of collaboration and learning is a must.

The publication process can also be challenging for mixed methods research. As Jick (1979,

p. 604) indicated, whether explicitly or implicitly, “journals tend to specialize by methodology

21

thus encouraging purity of method.” When targeting such an outlet for mixed methods research, scholars need to be cognizant of which methods are favored and capable of clearly articulating how the focal set of methods can lead to greater insight and understanding for the question at hand. Further, rather than having the methods stand on equal footing, a researcher may choose to emphasize the more accepted methodology, which is then complemented by a more novel method according to the standards of the outlet/field. Along similar lines, Creswell (2014) advises that scholars take into account the inclination of one’s field to particular methods and designs.

But publishing mixed methods research can be challenging even in journals that have

policies encouraging diverse and mixed methods. To constitute high-quality mixed methods research, each method must be done well, and the mix of methods must say something more; in other words, the methods as a set need to go beyond the separate methods. This can present practical challenges in multiple ways. For one, in the process of demonstrating that each method has been done well and saying something more, it is easy to run into page length constraints with journal articles. As another, while journals may have policies that advocate mixed methods, it is not always the case that individual reviewers do. In these instances, authors can often find considerable value from the use of exemplar mixed methods studies, both as guides for framing and as indicators of legitimacy given the early stage of mixed methods research (Creswell, 2014). In addition, editors clearly play a critical role, both with selecting reviewers and in guiding the subsequent review process.

Last, even after the studies have been published, mixed methods research can pose

challenges. As Jick (1979, p. 609) observes, triangulating across methodologies makes replication more challenging, particularly when qualitative data/analyses are involved. While mixed methods research clearly magnifies the challenges of replication, we note that the extent likely depends upon study design; for example, some of the challenges with replication are attenuated when mixed methods studies involve multiple research strategies that center on the same sample or empirical setting.

With all these challenges and difficulties with mixed methods research, one may ask why

bother. In fact, without an effective research design, it is likely not worth the effort. But if designed well, mixed methods research studies provide deeper insight and better understanding for our research questions, relative to single method studies.

Conclusion

While mixed methods offers a powerful way to broaden and deepen our understanding of phenomena and theories, there has been a slow rate of adoption in the organizational sciences. Our purpose in this article is two-fold. First, we aim to clarify how scholars use mixed methods to answer their research questions by developing a research design framework that is based on the principles of triangulation. Second, we offer guidance for scholars interested in mixed methods research by providing a corresponding roadmap that begins with the research question and theoretical intentions and culminates in the selection of methodologies. In doing so, we hope that this study can help to advance mixed methods research in the organizational sciences.

22

REFERENCES Adner, R., Polos, L., Ryall, M., & Sorenson, O. (2009). The case for formal theory. Academy of

Management Review, 34, 201-208. Aguinis, H., Pierce, C. A., Bosco, F. A., & Muslin, I. S. (2009). First decade of Organizational

Research Methods: Trends in design, measurement, and data-analysis topics. Organizational Research Methods, 12, 69-112.

Anteby, M. (2010). Markets, morals, and practices of trade: Jurisdictional disputes in the U.S. Commerce in cadavers. Administrative Science Quarterly, 55, 606–638.

Austin, J. T., Scherbaum, C. A., & Mahlman, R. A. (2002). History of research methods in industrial and organizational psychology: Measurement, design, analysis. In S. G. Rogelberg (Ed.), Handbook of research methods in industrial and organizational psychology (pp. 1-33). Oxford, UK: Blackwell.

Axtell, R., Axelrod, E., Epstein, J. M., & Cohen, M. D. (1996). Aligning simulation models: A case study and results. Computational and Mathematical Organization Theory, 1, 123-142.

Bacharach, S. B. (1989). Organizational theories: Some criteria for evaluation. Academy of Management Review, 14, 496-515.

Bardolet, D., Fox, C. R., & Lovallo, D. (2011). Corporate capital allocation: A behavioral perspective. Strategic Management Journal, 32, 1465–1483.

Bateman, T. S., & Zeithaml, C. P. (1989). The psychological context of strategic decisions: A model and convergent experimental findings. Strategic Management Journal, 10, 59-74.

Burton, R. M. (2003). Computational laboratories for organization science: Questions, validity and docking. Computational and Mathematical Organization Theory, 9, 91–108.

Burton, R. M., & Obel, B. (1995). The validity of computational models in organization science: From model realism to purpose of the model. Computational and Mathematical Organization Theory, 1, 57-72.

Burton, R. M., & Obel, B. (2011). Computational modeling for what-is, what-might-be, and what-should-be studies – and triangulation. Organization Science, 22, 1195-1202.

Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81-105.

Cardinal, L. B., Turner, S. F., Fern, M. J., & Burton, R. M. (2011). Organizing for product development across technological environments: Performance trade-offs and priorities. Organization Science, 22, 1000–1025.

Cohen, W. M., & Klepper, S. (1996). Firm size and the nature of innovation within industries: The case of process and product R&D. The Review of Economics and Statistics, 78, 232–243.

Creswell, J. W. (2014). Research design: Qualitative, quantitative, and mixed methods approaches. Thousand Oaks, CA: Sage Publications.

Davis, J. P., Eisenhardt, K. M., & Bingham, C. B. (2007). Developing theory through simulation methods. Academy of Management Review, 32, 480–499.

Debreu, G. (1986). Theoretic models: Mathematical form and economic content. Econometrica, 54, 1259–1270.

Denzin N. K. (1970). The research act: A theoretical introduction to sociological methods. Chicago, IL: Aldine Publishing Company.

23

Denzin, N. K. (2012). Triangulation 2.0. Journal of Mixed Methods Research, 6, 80-88. Dyer, J. H., & Hatch, N. W. (2006). Relation-specific capabilities and barriers to knowledge

transfers: Creating advantage through network relationships. Strategic Management Journal, 27, 701–719.

Eisenhardt, K. M. (1989). Building theories from case study research. Academy of Management Review, 14, 532-550.

Eisenhardt, K. M., & Graebner, M. E. (2007). Theory building from cases: Opportunities and challenges. Academy of Management Journal, 50, 25–32.

Ely, R. J. (1994). The effects of organizational demographics and social identity on relationships among professional women. Administrative Science Quarterly, 39, 203–238.

Flynn, F. J., & Staw, B. M. (2004). Lend me your wallets: The effect of charismatic leadership on external support for an organization. Strategic Management Journal, 25, 309-330.

Freese, L. (1980). Formal theorizing. Annual Review of Sociology, 6, 187–212. Gibbert, M., & Ruigrok, W. (2010). The “what” and “how” of case study rigor: Three strategies

based on published work. Organizational Research Methods, 13, 71-737. Gioia, D. A., Corley, K. G., & Hamilton, A. L. (2012). Seeking qualitative rigor in inductive

research: Notes on the Gioia methodology. Organizational Research Methods, 16, 15-31. Glaser, B. G., & Strauss, A. L. (1967). The discovery of grounded theory: Strategies for

qualitative research. New York: Aldine de Gruyter. Grant, A. M., Berg, J. M, & Cable, D. M. (2014). Job titles as identity badges: How self-

reflective titles can reduce emotional exhaustion. Academy of Management Journal, 37, 1201-1225.

Harrison, J. R., Lin, Z., Carroll, G. R., & Carley, K. M. (2007). Simulation modeling in organizational and management research. Academy of Management Review, 32, 1229–1245.

Jick, T. D. (1979). Mixing qualitative and quantitative methods: Triangulation in action. Administrative Science Quarterly, 24, 602-611.

Jin, Y., & Levitt, R. E. (1996). The virtual design team: A computational model of project organizations. Computational and Mathematical Organization Theory, 2, 171–195.

Lee, T. W., Mitchell, T. R., & Sablynski, C. J. (1999). Qualitative research in organizational and vocational psychology, 1979-1999. Journal of Vocational Behavior, 55, 161-187.

Levitt, R. E., Cohen, G. P., Knuz, J. C., Nass, C. I., Christiansen, T., & Jin, Y. (1994), The ‘virtual design team’: Simulating how organization structure and information processing tools affect team performance. In K. M., Carley & M. J., Prietula, (Eds.), Computational organization theory (pp. 1-18). New York: Lawrence Erlbaum Associates.

Lin, Z., Zhao, X., Ismail, K. M., & Carley, K. M. (2006). Organizational design and restructuring in response to crises: Lessons from computational modeling and real-world cases. Organization Science, 17, 598–618.

McGrath, J. E. (1982). Dilemmatic: The study of research choices and dilemmas. In J. E. McGrath, J. Martin, & R. A. Kulka (Eds.), Judgment calls in research (pp. 69-102). Beverly Hills, CA: Sage Publications.

McGrath, J. E. (1995). Methodology matters: Doing research in the behavioral and social sciences. In R. M. Backer (Eds.), Readings in human-computer interaction: Toward the year

24

2000 (pp. 152–169). Burlington, MA: Morgan Kaufman. McGrath, J. E., Martin, J., & Kulka R. A. (1982). Some quasi-rules for judgment calls in

research. In J. E. McGrath, J. Martin, R. A. Kulka (Eds.), Judgment calls in research (pp. 103-118). Beverly Hills, CA: Sage Publications.

Molina-Azorin, J. F. (2007). Mixed methods in strategy research: Applications and implications in the resource-based view. Research Methodology in Strategy and Management, 4, 37-73.

Molina-Azorin, J. F. (2012). Mixed methods research in strategic management: Impact and applications. Organizational Research Methods, 15, 33–56.

Morris, M. W., & Moore, P. C. (2000). The lessons we (don’t) learn: Counterfactual thinking and organizational accountability after a close call. Administrative Science Quarterly, 45, 737–765.

Nguyen, H. D., & Nguyen, N. T. (2008). Examining personal values and entrepreneurial motives of Vietnamese entrepreneurs in the 21st century: Two empirical studies. African and Asian Studies, 7, 141-171.

Pedhazur, E. J, & Schmelkin, L. (1991). Measurement, design and analysis: An integrated approach. New York, NY: Psychology Press.

Scandura, T. A., & Williams, E. A. (2000). Research methodology in management: Current practices, trends, and implications for future research. Academy of Management Journal, 43, 1248-1264.

Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton-Mifflin Company.

Shah, S. K., & Corley, K. G. (2006). Building better theory by bridging the quantitative- qualitative divide. Journal of Management Studies, 43, 1821-1835.

Singleton, R., & Straits, B. C. (1999). Approaches to social research. Oxford, UK: Oxford University Press.

Staw, B. M. (1982). Some judgments on the judgment calls approach. In J. E. McGrath, J. Martin, R. A. Ku, J. E. Martin, R. A. Kulka (Eds.), Judgment calls in research (pp. 119-127). Beverly Hills, CA: Sage Publications.

Thorngate, W. (1976). Possible limits on a science of social behavior. In J. H. Strickland, F. E. Aboud, & K. J. Gergen (Eds.), Social psychology in transition (pp. 121-139). New York, NY: Plenum Press.

Torrance, H. (2012). Triangulation, respondent validation, and democratic participation in mixed methods research. Journal of Mixed Methods Research, 6, 111-123.

Weick, K. E. (1969). The social psychology of organizing (first edition). Menlo Park, CA: Addison-Wesley Publishing Company.

Weick, K. E. (1979). The social psychology of organizing (second edition). Menlo Park, CA: Addison-Wesley Publishing Company.

Yin, R. K. (1994). Case study research: Design and methods (second edition). Beverly Hills, CA: Sage Publications.

25

Table 1. Methodological Purposes Fulfilled by Particular Research Strategies

Research Strategy Methodological Purpose within Mixed Methods Research Design

Archival Records

Archival records can be used in a variety of ways. As a strategy, it can be effective in maximizing generalizability with respect to populations, enhancing precision in control/measurement of variables, and/or capturing behaviors that have taken place in an authentic context

Case Study

Case studies are well-suited for capturing behaviors that have taken place in an authentic context. Although this method is typically not well-suited for maximizing generalizability with respect to populations, the use of multiple case studies allows for more claims regarding generalizability

Computer Simulation

Computational simulation is well-suited for enhancing precision in control/measurement of variables and can be effective in maximizing generalizability with respect to populations

Experimental Simulation

Experimental simulation can be effective in enhancing precision in control/measurement of variables and capturing behaviors that have taken place in an authentic context

Field Experiment

Field experiments can be effective in enhancing precision in control/measurement of variables and capturing behaviors that have taken place in an authentic context

Formal Theory (Mathematical)

Formal theory through mathematical models is well-suited for enhancing precision in control/measurement of variables and can be effective in maximizing generalizability with respect to populations

Interviews Interviews are well-suited for capturing behaviors that have taken place in an authentic context Lab

Experiment Lab experiments are well-suited for precision in control/measurement of variables

Survey Surveys can be effective in precision in control/measurement of variables and capturing behaviors that have taken place in an authentic context

26

27

28

View publication statsView publication stats