Quantitative and Qualitative Research
Combining Qualitative and Quantitative Approaches: Some Arguments for Mixed Methods Research
Thorleif Lund University of Oslo
One purpose of the present paper is to elaborate 4 general advantages of the mixed methods approach. Another purpose is to propose a 5-phase evaluation design, and to demonstrate its usefulness for mixed methods research. The account is limited to research on groups in need of treatment, i.e., vulnerable groups, and the advantages of mixed methods are illustrated by the help of the 5-phase evaluation design. The basic idea is that the total set of relevant attributes and changes for such a vulnerable group should be taken into consideration in all phases, and that the mixed methods approach will provide an optimal treatment, will give a more complete description and understanding of the treatment effects, and will facilitate generalization to professional work.
Keywords: mixed methods, qualitative-quantitative combination, evaluation design
The research methodology in the social and behavioral sciences has undergone radical changes over the past 50 years. One may speak of three methodological movements: (1) the quantitative movement, (2) the qualitative movement, and (3) the mixed methods movement (Polit & Beck, 2004; Teddlie & Tashakkori, 2003). Research in the twentieth century, especially in the first half of the century, was dominated by the quantitative move- ment. Its philosophical basis of positivism can be said to have been substituted by critical realism in the last half of the century (Cook & Campbell, 1979). The qualitative approach developed partly as a protest against the dominance of the quantitative tradition, and it attained its definitive breakthrough around 1970. Several philosophical assumptions have been proposed for the qualitative approach, mainly some variants of constructivism (Lincoln & Guba, 2000). The differences between the two approaches with respect to philo- sophical basis, scientific fruitfulness, and empirical methods have been extensively debated. The disagreement has been great, in particular with respect to philosophical positions, as illustrated by the “paradigm wars” (Gage, 1989), and the two approaches are still regarded by many researchers as incompatible means for knowledge construction (Teddlie & Tashak- kori, 2003). The mixed methods movement represents a blending of quantitative and quali- tative methods in research, and it can be said to have been evolved historically from the notion of “triangulating” information from different data sources (Campbell & Fiske, 1959; Denzin, 1978; Morse, 1991; Patton, 1990). The mixed methods approach can be con- sidered established as a formal discipline around 2000. This third movement is characterized by a practical/pragmatic attitude in that the research questions in empirical studies are given
ISSN 0031-3831 print/ISSN 1470-1170 online # 2012 Scandinavian Journal of Educational Research http://dx.doi.org/10.1080/00313831.2011.568674 http://www.tandfonline.com
Thorleif Lund, Department of Special Needs Education, University of Oslo. Correspondence concerning this article should be addressed to Thorleif Lund, Department
of Special Needs Education, University of Oslo, Box 1140, Blindern, N-0318 Oslo, Norway. E-mail: thorleif.lund@isp.uio.no or E-mail: solveig.lund@c2i.net.
Scandinavian Journal of Educational Research Vol. 56, No. 2, April 2012, 155 – 165
high priority, not philosophy of science, and in that qualitative and quantitative methods are used in combination for answering such questions. Mixed methods have been used in both basic and applied research, especially in the applied field of evaluation research.
The patterns of strengths and weaknesses of the qualitative approach are different from that of the quantitative approach (Polit & Beck, 2004). For example, qualitative methods are more appropriate for hypothesis generation than for hypothesis testing, whereas the oppo- site pattern can be said to hold for quantitative methods. Moreover, by qualitative methods we ordinarily obtain greater depth than by quantitative ones, while quantitative methods often result in better objectivity and generalizability than qualitative ones. The basic rationale of the mixed methods strategy is that by combining qualitative and quantitative methods one can utilize their respective strengths and escape their respective weaknesses (Tashakkori & Teddlie, 1998).
How should mixed methods research be defined more precisely? A representative defi- nition is given by Creswell, Clark, Gutmann, and Hanson (2003) as follows: “A mixed methods study involves the collection or analysis of both quantitative and qualitative data in a single study in which the data are collected concurrently or sequentially, are given a pri- ority, and involve the integration of the data at one or more stages in the process of research.” (p. 212, emphasis in original). Thus, qualitative and quantitative methods may be used concurrently or sequentially, one approach may be weighted stronger than the other, and the integration may be comprehensive or restricted. Whereas the definition is limited to a single study, mixed methods will sometimes be defined more broadly so as to include blend- ing of the two approaches within a coordinated cluster of individual studies, as well (Creswell & Clark, 2011; Polit & Beck, 2004).
In the mixed methods literature, several typologies of designs have been proposed and discussed (Creswell & Clark, 2011; Creswell, Clark, Gutmann, & Hanson, 2003; Greene & Caracelli, 1997; Maxwell & Loomis, 2003; Sandelowski, 2000; Tashakkori & Teddlie, 2003). Furthermore, the literature includes a discussion of which philosophical assumptions and validity criteria are appropriate for mixed methods research, and some variants of prag- matism are ordinarily proposed (Teddlie & Tashakkori, 2003).
Since the mixed methods approach is still young and probably relatively unknown to many researchers, one purpose of the present paper is to elaborate four general advantages of using this approach instead of qualitative or quantitative methods in isolation. Another purpose is to propose a five-phase evaluation design, and to illustrate its usefulness in mixed methods research. The design represents an extensive revision of the evaluation design of Borich (1985). The proposed five-phase design can be considered a new variant of the mixed methods multiphase design as defined by Creswell and Clark (2011). A multi- phase design is a flexible large-scale enterprise, where quantitative and qualitative methods are combined within and between several phases, and where the phases depend on each other and on an overall objective for the enterprise.
The elaboration of the general advantages is limited to research on groups in need of treatment—i.e., vulnerable groups—and is given in the context of the five-phase design. Persons with social anxiety problems are used as an (artificial) example. The overall research objective will be to develop an optimal treatment to be used effectively in professional work for helping the vulnerable group. The total set of subjective and objective attributes and changes of significance to possible treatments for the group is termed life space. The basic idea here is that the group’s life space should be taken into consideration in all phases of the evaluation, and that mixed methods in each phase are necessary for a successful solution
156 LUND
of this task. The account below is given in principal terms, while statistical and technical details are omitted.
Advantages of Mixed Methods Studies and the Five-Phase Design
Several authors have pointed out the utility of combining qualitative and quantitative methods (Adcock & Collier, 2001; Brewer & Hunter, 1989; Erzberger & Kelle, 2003; Maxwell & Loomis, 2003; Morse, 1991; Polit & Beck, 2004; Sandelowski, 1996, 2000; Tashakkori & Teddlie, 1998). The four general advantages below are meant to be in line with this literature:
(1) Mixed methods research is more able to answer certain complex research questions than qualitative or quantitative research in isolation. For example, given that quali- tative methods are more appropriate for hypothesis generation and quantitative methods for hypothesis testing, mixed methods enable the researcher better to sim- ultaneously answer a combination of exploratory and confirmatory questions. Theory may therefore be generated and verified in the same investigation. As another example, in an intervention study, a randomized experimental design can be used for describing causal effects and a qualitative interview for explaining how these effects were generated. Hence, in one study, quantitative and qualitative methods can answer complex research questions related to both causal description and causal explanation.
(2) Qualitative and quantitative results may relate to different objects or phenomena, but may be complementary to each other in mixed methods research. Hence, the combination of the different perspectives provided by qualitative and quantitative methods may produce a more complete picture of the domain under study.
(3) Mixed methods research may provide more valid inferences. If the results from quite different strategies such as qualitative and quantitative ones converge, the val- idity of the corresponding inferences and conclusions will increase more than with convergence within each strategy.
(4) In mixed methods research, qualitative and quantitative results may be divergent or contradictory, which can lead to extra reflection, revised hypothesis, and further research. Thus, given that data have been collected and analyzed correctly, such divergence can generate new theoretical insights.
The three first-mentioned general advantages are elaborated and illustrated below, whereas the fourth one is briefly commented upon. The five-phase evaluation design serves as a frame for the elaboration, and anxiety persons are used for illustration. A general descrip- tion of the design is given first, followed by an account of how mixed methods can be used in each phase, and of how the phases depend on each other. For simplicity, it is assumed that the same research team is involved in all phases.
The design is presented in Figure 1, and the five phases are as follows: (1) Need analysis, (2) Construction and choice, (3) Implementation and process analysis, (4) Effect assessment and interpretation, and (5) Generalization. The first phase consists in scrutinizing the field of interest in order to decide which interventions are needed. Based on this first-phase infor- mation, the second phase comprises construction or choice of methodological elements of relevance to later phases, i.e., appropriate program(s), effect and process variables, sampling, designs, and analyses. The program implementation and the causal process are analyzed in
MIXED METHODS 157
the third phase, the program effects are estimated and interpreted in the fourth phase, whereas the results are generalized to relevant targets in the fifth phase.
It follows that the five phases are related, and this dependence is indicated by the arrows between the phases from left to right. Note also that the intervention study proper is represented by phase 2, 3, 4, and 5, whereas the first phase provides information to the inter- vention study. By Knowledge space in the Figure is meant the relevant set of substantive and methodological knowledge, provided by earlier research, as well as methodological and ethical standards (Lund, 2005b). The arrows from knowledge space to the five phases illustrate that each phase depends on this space. Sometimes the sequence of phases is not as linear as indicated by the arrows between the phases from left to right, and the possibility of nonlinearity is illustrated by the three arrows from right to left below phase 3, 4, and 5. Finally, evaluation research presupposes criteria (Weiss, 1998), and the evaluation criteria are here represented by methodological standards (e. g. validity systems) in knowledge space. Evaluation research may be involved with each of the five phases or with the set of all phases combined.
Suppose we have a large group of adults seeking help for their social anxiety problems. For such persons, the research purpose in the first phase should be to describe, explore, and evaluate anxiety-related aspects of their life space, i.e., subjective and objective aspects in connection with family, job, friends, past events, plans for the future, self-image, sleep, and so on. The evaluation aims to generate information about which life-space aspects ought to be changed by interventions. Discovery of causal chains involving anxiety will be important, especially the detection of manipulable causes of anxiety, because the program construction in the second phase should take care of such causes.
A combination of quantitative and qualitative methods are useful for solving these first- phase tasks, e.g., quantitative surveys and other non-experimental designs, as well as quali- tative interviews on representative or atypical clinical samples. All three first-mentioned general advantages can be relevant here. For example, the first one is implied if interviews generate a hypothesis about which factors cause the anxiety, and if this hypothesis is then tested by some quantitative, non-experimental approach. As for the second advantage, if quantitative and qualitative results refer to partly different parts of the life space, but in a
Figure 1. A five-phase evaluation design.
158 LUND
complementary sense, the combined results yield a fuller picture of the life space for the anxiety group. Thirdly, the validity of inferences, e.g., inferences about causes and conse- quences of anxiety, will be more strengthened by convergent results with mixed methods than by convergence within quantitative or qualitative strategies. Finally, knowledge space provides substantive and methodological information of relevance for solving the first- phase tasks.
One research purpose in the second phase for the anxiety group is—on the basis of infor- mation from the first phase and knowledge space—to construct for later phases appropriate effect and process variables as well as a program expected to affect these variables. The vari- ables should correspond to the first-phase aspects in need of change, and the program should be related to causal information in the first phase. Mixed methods will be useful in the con- struction of the variables. First, in line with the third general advantage, the construct validity for some variables can be strengthened by a mixed methods strategy, e.g., by combining qualitative interviews and psychometric procedures. Second, some life-space aspects for the anxiety group may be better operationalized by quantitative methods and other aspects by qualitative methods. Quantitative variables will be the result in the former case, while the latter case yields some qualitative operationalizations, for instance in the form of inter- view guides. The integration of these two kinds of life-space representations will provide a more complete picture, thus illustrating the second general advantage. Similar arguments hold for constructing a suitable program.
The second phase also includes choice of sampling, situation, design, and analysis for use in the later phases, and these decisions should take mixed methods into consideration. As for sampling, mixed methods would normally require large, representative samples of anxiety clients as well as small and typical or atypical samples, the former selected for quantitative purposes and the latter for qualitative ones. The choice of experimental situation depends on the desired targets of generalization, i.e., the situation in the investigation should be repre- sentative for these target situations. With respect to design and analysis, a combination of quantitative and qualitative designs with their respective analyses will be useful for studying the program implementation and processes in the third phase, both experimental and quali- tative designs/analyses are relevant for assessing the effects in the fourth phase, while the generalizations in the fifth phase depend partly on the earlier choices of designs/analyses and on the respective results.
The research purpose in the third phase is to study and evaluate the implementation of the experimental variable as well as to analyse the causal process in order to understand how the program impact has been mediated to the effect variables. The solutions of these tasks are dependent on the second-phase choices and knowledge space. The results can be used to explain how the effects to be described in the fourth phase have been generated.
Mixed methods will be useful in the third stage for the anxiety group as follows. As for implementation, qualitative and quantitative methods (qualitative interviews and quantitative observations, say) will clarify whether the program and control conditions have been implemented as planned in the second phase. Possible obstacles to the planned implemen- tation, such as lack of time, financial resources, and status conflicts, may thereby be effec- tively detected and taken care of.
It can be argued that all three first-mentioned general advantages of mixed methods are relevant for exploring these obstacles, and the arguments will be similar to those given above for the first phase. Furthermore, the study of the causal mediation should be a central part of the third phase. In our anxiety example, the program impact on anxiety might be mediated by
MIXED METHODS 159
reality orientation. That is, the program has to increase reality orientation of the patients before anxiety reduction can take place. Mixed methods will be valuable for discovering and testing such causal chains, e.g., by a combination of exploratory interviews (Lincoln & Guba, 2000) and structural modeling (Bollen, 1989). The three first-mentioned general advantages are relevant here, according to similar arguments as given before.
The research purpose in the fourth phase is to estimate and interpret the program effects, and these endeavours depend on the choices made in the second phase and knowledge space. For our anxiety group, these effects correspond to all program-produced changes in their life space, and this set of changes is here termed effect space. Both qualitative and quantitative effect changes are included in the effect space, and the effects will all be related—directly or indirectly—to anxiety. The aim in the fourth phase is therefore to assess and interpret this effect space, and mixed methods will be suitable for solving these tasks.
Suppose a randomized control-group post-test design has been undertaken in our example, where the treatment group has received the program and the other group is an atten- tion-control group. Assume further that the same qualitative interviews and quantitative tests have been used for the two groups at post-test, and that text analysis has been used for the qualitative data and statistical analysis for test data. We therefore have two assessed life spaces of post-test scores/levels on quantitative and qualitative attributes, one space for each group. Due to the randomization, the difference between these two assessed post-test- scores life spaces (treatment-group space minus control-group space) will be an assessment of the patients’ effect space, i.e., the assessed effect space.
The second and third general advantages are relevant with such a mixed methods approach. The second advantage is involved in that qualitative and quantitative results rep- resent different regions of the patients’ effect space, and in that these two sets of results supplement each other. If some qualitative and quantitative results converge on some causal inferences, the validity of these inferences will be increased, which illustrates the third advantage. These two advantages are further demonstrated if the program comprises several components (lectures, group discussions, and coping exercises, say), and if the cor- responding component effects are estimated by program patients at post-test by qualitative interviews as well as by some quantitative rating-scale procedure.
In the fifth phase, the assessed effect space will be generalized to and across relevant targets of persons, settings, and times. For our anxiety study, such targets are similar groups in actual therapy settings or in need of therapy, and long-term generalizations will, of course, be important. The choice of targets of generalization depends on the general aim and research problem of the intervention study.
The validity of generalizations will be based on the mixed methods choices and results in the earlier phases, on information from knowledge space, as well as on the similarity between study and target. As a rule, the greater the similarity with respect to persons, settings, and times, the higher the validity of the corresponding generalizations of the assessed effect space to targets (Shadish, Cook, & Campbell, 2002). Empirical results are needed in the fifth phase in order to assess this study-target similarity. Thorough descriptions of persons, settings, and times within study and targets will indicate the degree of similarity, and both qualitative and quantitative procedures will be useful in this respect. The three former general advantages are relevant here, according to the same arguments as those given before. Thus, a successful solution of how to transfer the assessed effect space from study to targets in the fifth phase requires that mixed methods strategies have been used in all five phases in Figure 1.
160 LUND
The preceding account illustrates the three first-mentioned general advantages of mixed methods in the context of a five-phase evaluation model of relevance to vulnerable groups. As for the fourth advantage, divergent or contradictory results provided by qualitative and quan- titative methods may occur in all five phases. For example, suppose that the quantitative and qualitative analyses in the fourth phase yield opposite estimates of the program effects for our anxiety patients. Given that methodological errors can be eliminated, such a paradoxical case will naturally lead to an extra scrutiny of the patients’ life space, with new theoretical insight as a probable consequence. A real example of the fourth advantage is given by Trend (1979) in his evaluation of an experimental federal housing subsidy program, involving qualitative and quantitative data collection and analysis. Qualitative observation results directly contra- dicted the results of the quantitative analysis of the program outcomes, and this paradox generated new mixed methods research. Trend eventually proposed a coherent causal expla- nation for the original contradictory results that went beyond the initial incompatible quantitative and qualitative conclusions, and that revealed serious shortcomings in these conclusions.
The basic idea in this paper is that life space for a vulnerable group should be focused upon in all five phases, and that mixed methods strategies are necessary for successful need assessment, program and instrument development, causal explanation, causal descrip- tion, and generalizations. This focus on the life space and use of mixed methods will probably lead to that all critical aspects are taken care of in the evaluation study, that an optimal program is constructed for influencing these aspects, and that the effect space is more com- pletely described. Hence, to restrict the analysis to either quantitative or qualitative effects may result in that important parts of a multidimensional effect space are neglected, i.e., a kind of underestimation of the program impact. Note, in passing, that since the popular tech- nique of meta-analysis includes quantitative results only (Hunter & Schmidt, 1990; Lipsey & Wilson, 2000), use of this technique for vulnerable groups may yield an incomplete picture of program impacts. Also, this focus on life space will lead to a greater similarity between the evaluation study and relevant professional targets, e.g., therapies for anxiety patients, because life spaces are dealt with in such targets. Consequently, the focus results in more valid generalizations from the study to professional targets.
Several experimental designs are relevant for assessing the effect space in our anxiety example, and mixed methods strategies are useful with all of them. As pointed out above, if a randomized control-group posttest design is chosen, with post-test scores on quantitative and qualitative attributes in each group, the difference between these two assessed post-test- score life spaces constitutes the assessed effect space. Suppose the randomized design is supplied with pre-test measurements on the same quantitative and qualitative attributes as on the post-test occasion. For each group, we then have assessed post-test-score life space and assessed pre-test-score life space, and the difference between these two spaces (the former minus the latter) is the assessed descriptive (noncausal) change space for the group. The difference between the two groups’ descriptive change spaces yields the same estimate of effect space as that with the former design, apart from random errors. If, on the other hand, a quasi-experimental pre-test-post-test design without a control group is chosen, the assessed descriptive change space for the program group may be interpreted as an estimate of effect space. A similar reasoning applies to alternative quasi-experimental designs. Moreover, given that the program consists of several com- ponents, the effect space for these components can be estimated by mixed methods as men- tioned earlier.
MIXED METHODS 161
Since the primary research purpose for a vulnerable group will be to choose an optimal program (and its potential effect space) to be used in professional work for helping this group, generalization issues should have high priority. As suggested above, external validity can be strengthened in various ways, for instance by increasing the study-target similarity with respect to persons, settings, and times. As for times, long-term program effects should be investigated because the greatest impacts may take place over time. For example, the program may result in that our anxiety patients improve their relationships to other people, attain more attractive jobs, and get additional education. Such effect changes will probably occur some time after the program interval. It follows that appropriate follow-up life-space measurements should be included in the experimental design.
Also, as pointed out by Shadish et al. (2002), causal explanation will be useful for causal generalizations. For our anxiety example, suppose that mixed methods analyses in the third phase indicate that satisfactory program impacts on anxiety have been mediated by a reality- orientation variable. This causal-chain information may give hints about how professionals can obtain even greater anxiety reductions by strengthening the causal side of the chain. If, on the other hand, the effect estimates turn out to be trivial or zero, there are two alterna- tives: either the program may be ineffective or the program could be valuable but its implementation has been hindered by some practical circumstances. If the second alternative is correct, and if such obstacles can be eliminated in professional work, it will be wrong to reject the program. Without third-phase analyses one cannot decide between the two alterna- tives. Hence, for both positive and zero program effects, thorough third-phase analyses by mixed methods are needed for successful generalization to professional targets.
Knowledge space can also be helpful for solving the generalization problem. For our anxiety group, substantive theory and results from earlier empirical research on other patient groups may facilitate the transfer of program impacts to professional work. Quantitat- ive and qualitative results in knowledge space can be used in combination for this purpose, even if these two kinds of results are not generated from mixed methods studies.
The five-phase evaluation design proposed here as a variant of the multiphase design is flexible in that in each phase qualitative and quantitative methods may be used concurrently or sequentially, one approach may be weighted stronger than the other, and the integration may be extensive or restricted. Hence, as with other multiphase designs, the five-phase design represents combinations of simpler mixed methods designs (Creswell & Clark, 2011). If each phase corresponds to a mixed method study, the five-phase design corresponds to a coordinated cluster of five such individual studies.
Final Remarks
Although mixed methods can ordinarily be considered more effective for research on vul- nerable groups than quantitative or qualitative methods in isolation, such a combined approach has some logistic challenges. The approach encompasses often—especially in using a multi- phase design—large-scale research programs and team work, and tends therefore to require more resources than the two other approaches. This resource use might be counted as an argu- ment against mixed methods, but such an argument is invalid, because a satisfactory knowl- edge status for a vulnerable group will be more effectively attained by a coordinated and complex mixed methods investigation than by some unrelated simple studies. As for team work, since typically no team members are experts in both quantitative and qualitative methods, one challenge is how to develop a needed common mixed-methods insight in the
162 LUND
team. Moreover, different values, interests, and personality traits among the team members may lead to collaboration conflicts, and such conflicts have to be resolved. Various models for professional competency and collaboration have been proposed and studied empirically (Newman & Benz, 1998; Shulha & Wilson, 2003; Teddlie & Tashakkori, 2003). Another logistic challenge concerns pedagogical issues. The possibilities of the mixed methods approach should be clarified to graduate and post-graduate students in separate mixed method courses. This is not the usual case at the present time, however. Typically, students take research courses in quantitative and qualitative methods, but they are not given a systema- tic demonstration of how to combine these two kinds of methods. Creswell et al. (2003) have elaborated alternative models for teaching mixed methods research.
Which validity system and philosophical paradigm are appropriate for mixed methods? These issues have been extensively debated (Teddlie & Tashakkori, 2003). As for validity system, there has been no clear favorite. For example, Teddlie and Tashakkori (2003) are sceptical about the concept of validity, and propose instead an alternative set of quality cri- teria related to inferences in mixed methods research. The position taken in the present paper is that, since it can be argued that the Campbellian validity system for quantitative research (Shadish et al., 2002) is relevant also for the qualitative approach (Lund, 2005a), this system is applicable in mixed methods research as well. However, the validity system should be revised on some points, especially concerning the definition of causal inferences and the related internal validity, as argued by Cronbach (1982), Kruglanski and Kroy (1976), Lund (2010), and Reichardt (2008). The Campbellian system is based on critical realism (Cook & Campbell, 1979). Since critical realism can be considered a sound philosophical paradigm in both quantitative and qualitative cases (Lund, 2005a), this paradigm is regarded here as adequate for mixed methods research, too. Pragmatism has often been proposed as the best paradigm, primarily because mixed methods studies are typically characterized by a strong focus on research questions and practical use of results (Tashakkori & Teddlie, 1998), but this focus is not incompatible with critical realism.
How to weight qualitative and quantitative methods in a mixed methods study is an impor- tant and complicated methodological problem, and its solution depends on many factors, e.g., research purpose, kind of phenomenon, and knowledge status of the research domain. Hence, mixed methods studies vary with respect to this priority issue. In some studies qualitative and quantitative methods are considered of equal importance, whereas in other cases one approach is weighted stronger than the other, and the degree of this differential weighting may vary con- siderably across studies. This variation may take place within a study, as well. For example, for our anxiety patients, quantitative results may be considered more important for causal description than qualitative results, while the opposite weighting may be relevant for causal explanation. The high prestige associated with use of modern advanced statistical-mathemat- ical models in social science can be problematic with respect to the priority issue. That is, this prestige may lead to that the related quantitative results are given undue weight in many cases, and hence to that important aspects of life space are more or less neglected.
The elaboration of the advantages of mixed methods in this paper has focused on evalu- ation research on vulnerable groups, but similar arguments can be given for other kinds of applied research, and also for basic research (Maxwell & Loomis, 2003; Morse, 1991; Sande- lowski, 2000). Though the third methodological movement of mixed methods is still a young discipline, and several issues need to be clarified (Teddlie & Tashakkori, 2003), this approach should be considered a valuable contribution to the social and behavioral sciences, for example to educational and psychological research.
MIXED METHODS 163
References
Adcock, R., & Collier, D. (2001). Measurement validity: A shared standard for qualitative and quan- titative research. American Political Science Review, 95, 529–545.
Bollen, K.A. (1989). Structural equations with latent variables. New York: Wiley. Borich, G.D. (1985). Needs assessment and the self-evaluating organization. Studies in Educational
Evaluation, 11, 205–215. Brewer, J., & Hunter, A. (1989). Multimethod research: A synthesis of styles. Newbury Park, CA:
Sage. Campbell, D., & Fiske, D.W. (1959). Convergent and discriminant validation by the multitrait-multi-
method matrix. Psychological Bulletin, 56, 81–105. Cook, T.D., & Campbell, D.T. (1979). Quasi-experimentation: Design and analysis issues for field
settings. Chicago, IL: Rand-McNally. Creswell, J.W., & Clark, V.L.P. (2011). Designing and conducting mixed methods research. London:
Sage. Creswell, J.W., Clark, V.L.P., Gutmann, M.L., & Hanson, W.E. ( 2003). Advanced mixed methods
research designs. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (pp. 209–240). London: Sage.
Creswell, J.W., Tashakkori, A., Jensen, K.D., & Shapley, K.L. (2003). Teaching mixed methods research: Practices, dilemmas, and challenges. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (pp. 619–637). London: Sage.
Cronbach, L.J. (1982). Designing evaluations of educational and social programs. San Francisco, CA: Jossey-Bass.
Denzin, N.K. (1978). The research act: A theoretical introduction to sociological methods. New York: McGraw-Hill.
Erzberger, C., & Kelle, U. (2003). Making inferences in mixed methods: The rules of integration. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (pp. 453–488). London: Sage.
Gage, N. (1989). The paradigm wars and their aftermath: A “historical” sketch of research and teaching since 1989. Educational Researcher, 18, 4–10.
Greene, J.C., & Caracelli, V.J. (1997). Advances in mixed-method evaluation: The challenges and benefits of integrating diverse paradigms. San Francisco, CA: Jossey-Bass.
Hunter, J.E., & Schmidt, F.L. (1990). Methods of meta-analysis: Correcting error and bias in research findings. Newbury Park, CA: Sage.
Kruglanski, A.W., & Kroy, M. (1976). Outcome validity in experimental research: A reconceptualiza- tion. Journal of Representative Research in Social Psychology, 7, 168–178.
Lincoln, Y.S., & Guba, E.G. (2000). Paradigmatic controversies, contradictions, and emerging con- fluences. In N.K. Denzin & Y.S. Lincoln (Eds.), Handbook of qualitative research (pp. 163–188). Thousand Oaks, CA: Sage.
Lipsey, M.W., & Wilson, D.B. (2000). Practical meta-analysis. Newbury Park, CA: Sage. Lund, T. (2005a). The qualitative-quantitative distinction: Some comments. Scandinavian Journal of
Educational Research, 49, 115–132. Lund, T. (2005b). A metamodel of central inferences in empirical research. Scandinavian Journal of
Educational Research, 49, 385–398. Lund, T. (2010). Causal inferences in the Campbellian validity system. Scandinavian Journal of
Educational Research, 54, 205–220. Maxwell, J.A., & Loomis, D.M. (2003). Mixed methods design: An alternative approach. In A.
Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (pp. 241–271). London: Sage.
Morse, J.M. (1991). Approaches to qualitative-quantitative methodological triangulation. Nursing Research, 40, 120–122.
164 LUND
Newman, I., & Benz, C.R. (1998). Qualitative-quantitative research methodology: Exploring the interactive continuum. Carbondale: University of Illinois Press.
Patton, M.Q. (1990). Qualitative evaluation and research methods. Newbury Park, CA: Sage. Polit, D.F., & Beck, C.T. (2004). Integration of qualitative and quantitative designs. In D.F. Polit &
C.T. Beck (Eds.), Nursing research: Principles and methods (pp. 273–288). London: Lippincott Williams & Wilkins.
Reichardt, C.S. (2008, November). An alternative to the Campbellian conceptualization of validity, Paper presented at the Evaluation 2008 Conference, Denver, CO.
Sandelowski, M. (1996). Using qualitative methods in intervention studies. Research in Nursing & Health, 19, 359–364.
Sandelowski, M. (2000). Combining qualitative and quantitative sampling, data collection, and analy- sis techniques in mixed-methods studies. Research in Nursing & Health, 23, 246–255.
Shadish, W.R., Cook, T.D., & Campbell, D.T. (2002). Experimental and quasi-experimental designs for generalized causal inference. New York: Houghton Mifflin.
Shulha, L.M., & Wilson, R.J. (2003). Collaborative mixed methods research. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (pp. 639–669). London: Sage.
Tashakkori, A., & Teddlie, C. (1998). Mixed methodology: Combining qualitative and quantitative approaches. Thousand Oaks, CA: Sage.
Tashakkori, A., & Teddlie, C. (Eds.). (2003). The past and future of mixed methods research: From data triangulation to mixed methods designs. Handbook of mixed methods in social and behav- ioral research (pp. 671–701). London: Sage.
Teddlie, C., & Tashakkori, A. (2003). Major issues and controversies in the use of mixed methods in the social and behavioral sciences. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (pp. 3–50). London: Sage.
Trend, M.G. (1979). On the reconciliation of qualitative and quantitative analysis: A case study. In T.D. Cook & C.S. Reichardt (Eds.), Qualitative and quantitative methods in evaluation research (pp. 68–86). Newbury Park, CA: Sage.
Weiss, C.H. (1998). Evaluation. Upper Saddle River, NJ: Prentice Hall.
MIXED METHODS 165
Copyright of Scandinavian Journal of Educational Research is the property of Routledge and its content may
not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written
permission. However, users may print, download, or email articles for individual use.