DUE: 4/17/17 @ 3 pm est.

CHAPTER TEN

H

Mixed Methods Procedures

ow would you write a mixed methods procedure section for your proposal? Up until this point, we have considered collected quantitative data and qualitative data. We have not discussed “mixing” or combining the two forms of data in a study. We can start with the assumption that

both forms of data provide different types of information (open-ended data in the case of qualitative and closed-ended data in the case of quantitative). If we further assume that each type of data collection has both limitations and strengths, we can consider how the strengths can be combined to develop a stronger understanding of the research problem or questions (and, as well, overcome the limitations of each). This “mixing” or blending of data, it can be argued, provides a stronger understanding of the problem or question than either by itself. This idea is at the core of a reasonably new method called “mixed methods research.”

Conveying the nature of mixed methods research and what it is begins a good mixed methods procedure. Begin with the assumption that mixed methods is a new methodology in research and that the readers need to be educated as to the basic intent and definition of the design, the reasons for choosing the procedure, and the value it will lend to a study. Then, decide on a mixed methods design to use. There are many from which to choose and consider the different possibilities and decide which one is best for the proposed study. With this choice in hand, discuss the data collection, the data analysis, and the data interpretation and validation procedures within the context of the design. Finally, end with a discussion of potential ethical issues that need to be anticipated in the study, and suggest an outline for writing the final study. These are all standard methods procedures, but they are framed in this chapter as they apply to mixed methods research. Table 10.1 shows a checklist of the mixed methods procedures addressed in this chapter.

Table 10.1 A Checklist of Questions for Designing a Mixed Methods Procedure

_____________ Is a basic definition of mixed methods research provided? _____________ Are the reasons given for using both quantitative and qualitative data? _____________ Does the reader have a sense for the potential use of mixed methods research? _____________ Are the criteria identified for choosing a mixed methods design? _____________ Is the mixed methods design identified? _____________ Is a visual model presented that illustrates the research strategy? _____________ Is the proper notation used in presenting the visual model? _____________ Are procedures of data collection and analysis mentioned as they relate to the chosen design? _____________ Are the sampling strategies for both quantitative and qualitative data collection mentioned for the design? _____________ Are specific data analysis procedures indicated for the design? _____________ Are the procedures for validation mentioned for the design and for the quantitative and qualitative research?

_____________ Is the narrative structure of the final dissertation or thesis mentioned, and does it relate to the type of mixed methods design being used?

COMPONENTS OF MIXED METHODS PROCEDURES

Mixed methods research has evolved into a set of procedures that proposal developers can use in planning a mixed methods study. In 2003, the Handbook of Mixed Methods in the Social and Behavior Sciences (Tashakkori & Teddlie, 2003) was published (and later added to in a second edition, see Tashakkori & Teddlie, 2010), providing a comprehensive overview of this approach. Now several journals emphasize mixed methods research, such as the Journal of Mixed Methods Research, Quality and Quantity, Field Methods, and the International Journal of Multiple Research Approaches while numerous others actively encourage this form of inquiry (e.g., International Journal of Social Research Methodology, Qualitative Health Research, Annals of Family Medicine). Numerous published research studies have incorporated mixed methods research in the social and human sciences in diverse fields such as occupational therapy (Lysack & Krefting, 1994), interpersonal communication (Boneva, Kraut, & Frohlich, 2001), AIDS prevention (Janz et al., 1996), dementia caregiving (Weitzman & Levkoff, 2000), occupational health (Ames, Duke, Moore, & Cunradi, 2009), mental health (Rogers, Day Randall, & Bentall, 2003), and in middle school science (Houtz, 1995). New books arrive each year solely devoted to mixed methods research (Bryman, 2006; Creswell & Plano Clark, 2011; Greene, 2007; Morse & Niehaus, 2009; Plano Clark & Creswell, 2008; Tashakkori & Teddlie, 1998, 2010; Teddlie & Tashakkori, 2009).

Describe Mixed Methods Research

Because mixed methods research is relatively new in the social and human sciences as a distinct research approach, it is useful to convey a basic definition and description of the approach in a method section of a proposal. This might include the following:

• Begin by defining mixed methods. Recall the definition provided in Chapter 1. Elements in this definition can now be enumerated so that a reader has a complete set of core characteristics that describe mixed methods (see a more expanded view of defining mixed methods research in Johnson, Onwuegbuzie, & Turner, 2007): It involves the collection of both qualitative (open-ended) and quantitative (closed-ended) data

in response to research questions or hypotheses. It includes the analysis of both forms of data. The procedures for both qualitative and quantitative data collection and analysis need to be

conducted rigorously (e.g., adequate sampling, sources of information, data analysis steps). The two forms of data are integrated in the design analysis through merging the data, connecting

the data, or embedding the data. These procedures are incorporated into a distinct mixed methods design that also includes the

timing of the data collection (concurrent or sequential) as well as the emphasis (equal or unequal) for each database.

These procedures can also be informed by a philosophical worldview or a theory (see Chapter 3).

• Discuss that many different terms are used for this approach, such as integrating, synthesis, quantitative and qualitative methods, multimethod, and mixed methodology but that recent writings tend to use the term mixed methods (Bryman, 2006; Tashakkori & Teddlie, 2010).

• Educate the reader about the background of mixed methods by reviewing briefly this history of this approach to research. It can be seen as a new methodology originating around the late 1980s and early 1990s based on work from individuals in diverse fields such as evaluation, education, management, sociology, and health sciences. It has gone through several periods of development including the formative stage, the philosophical debates, the procedural developments, and more recently reflective positions (noting controversies and debates) and expansion into different disciplines and into many countries throughout the world. Several texts outline these developmental phases (e.g., Creswell & Plano Clark, 2011; Teddlie & Tashakkori, 2009). This section could also include a brief discussion about the importance or rise of mixed methods today through such indicators of federal funding initiatives, through dissertations, and through the discipline-specific discussions about mixed methods found in journals across the social and health sciences (see Creswell, 2010, 2011).

• Follow this section with statements about the value and rationale for the choice of mixed methods as an approach for the dissertation or thesis project. At a general level, mixed methods is chosen because of its strength of drawing on both qualitative and quantitative research and minimizing the limitations of both approaches. At a practical level, mixed methods provides a sophisticated, complex approach to research that appeals to those on the forefront of new research procedures. It

also can be an ideal approach if the researcher has access to both quantitative and qualitative data. At a procedural level, it is a useful strategy to have a more complete understanding of research problems/questions, such as the following: Comparing different perspectives drawn from quantitative and qualitative data Explaining quantitative results with a qualitative follow-up data collection and analysis Developing better measurement instruments by first collecting and analyzing qualitative data

and then administrating the instruments to a sample Understanding experimental results by incorporating the perspectives of individuals Developing a more complete understanding of changes needed for a marginalized group through

the combination of qualitative and quantitative data Having a better understanding the need for and impact of an intervention program through

collecting both quantitative and qualitative data over time

• Indicate the type of mixed methods design that will be used in the study and the rationale for choosing it. A detailed discussion of the primary strategies available will be discussed shortly. Include a figure or diagram of these procedures.

• Note the challenges this form of research poses for the inquirer. These include the need for extensive data collection, the time-intensive nature of analyzing both qualitative and quantitative data, and the requirement for the researcher to be familiar with both quantitative and qualitative forms of research. The complexity of the design also calls for clear, visual models to understand the details and the flow of research activities in this design.

TYPES OF MIXED METHODS DESIGNS

There have been several typologies for classifying and identifying types of mixed methods strategies that proposal developers might use in their proposed mixed methods study. Creswell and Plano Clark (2011) identified several classification systems drawn from the fields of evaluation, nursing, public health, education policy and research, and social and behavioral research. In these classifications, authors used diverse terms for their types of designs, and a substantial amount of overlap of types existed in the typologies. For purposes of this discussion, I will identify and discuss the three basic mixed methods designs (as shown in Figures 10.1 and 10.2) and then briefly mention more advanced strategies that incorporate these three basic forms. Each approach will be discussed in terms of a description of the design, the forms of data collection and data analysis, interpretation, and validity challenges.

Convergent Parallel Mixed Methods Design

Description of the design. The convergent mixed methods approach is probably the most familiar of the basic and advanced mixed methods strategies. Researchers new to mixed methods typically first think of this approach because they feel that mixed methods only consists of combining the quantitative and qualitative data. In this approach, a researcher collects both quantitative and qualitative data, analyzes them separately, and then compares the results to see if the findings confirm or disconfirm each other (see Figure 10.1). The key assumption of this approach is that both qualitative and quantitative data provide different types of information—often detailed views of participants qualitatively and scores on instruments quantitatively—and together they yield results that should be the same. It builds off the historic concept of the multimethod, multitrait idea from Campbell and Fiske (1959), who felt that a psychological trait could best be understood by gathering different forms of data. Although the Campbell and Fiske conceptualization included only quantitative data, the mixed methods researchers extended the idea to include the collection of both quantitative and qualitative data.

Data collection. The qualitative data can assume any of the forms discussed in Chapter 8, such as interviews, observations, documents, and records. The qualitative data can be instrument data, observational checklists, or numeric records, such as census data, as discussed in Chapter 7. The key idea with this design is to collect both forms of data using the same or parallel variables, constructs, or concepts. In other words, if the concept of self-esteem is being measured quantitatively, the same concept is asked during the qualitative data collection process, such as in an open-ended interview. Another data collection issue is the sample size for both the qualitative and quantitative data collection process. Unquestionably, the data for the qualitative data collection will be smaller than that for the quantitative data collection. This is because the intent of data collection for qualitative data is to locate and obtain information from a small sample but to gather extensive information from this sample; whereas, in quantitative research, a large N is needed in order to conduct meaningful statistical tests. How is this inequality resolved in a convergent mixed methods design? Sometimes mixed methods researches will collect information from the same number of individuals on both the qualitative and quantitative database. This means that the qualitative sample will be increased, and it will limit the amount of data collected from any one individual. Another approach would be to weight the qualitative cases so that they equal the N in the quantitative database. One other approach taken by some mixed methods researchers is not to consider the unequal sample sizes a problem. They would argue that the intent of qualitative and quantitative research differ (one to gain an in-depth perspective and the other, to generalize to a population) and that each provides an adequate count. Another issue in sampling is whether the individuals for the sample of qualitative participants should also be individuals in the quantitative sample. Typically, mixed methods researchers would include the sample of qualitative participants in the larger quantitative sample, because ultimately researchers make a comparison between the two databases and the more they are similar, the better the comparison.

Figure 10.1 Three Basic Mixed Methods Designs

Figure 10.2 Advanced Mixed Methods Designs

Data analysis. The challenge in a convergent mixed methods design is how to actually converge or to

merge the data. We know from a description of this design that the two databases are analyzed separately and then brought together. There are several ways to merge the two databases. The first approach is called a side-by-side comparison. These comparisons can be seen in the discussion sections of mixed methods studies. The researcher will first report the quantitative statistical results and then discuss the qualitative findings (e.g., themes) that either confirm or disconfirm the statistical results. Alternatively, the researcher might start with the qualitative findings and then compare them to the quantitative results. Mixed methods writers call this a side-by-side approach because the researcher makes the comparison within a discussion, presenting first one set of findings and then the other. A good example of this can be seen in the Classen and colleagues’ (2007) study.

Researchers can also merge the two databases by changing qualitative codes or themes into quantitative variables and then combining the two quantitative databases—a procedure called data transformation. The researcher takes the qualitative themes or codes and counts them (and possibly groups them) to form quantitative measures. Some useful procedures that mixed methods researchers have used can be found in Onwuegbuzie and Leech (2006). A final procedure is to merge the two forms of data in a table or a graph. This is called a joint display of data, and it can take many different forms. It might be a table that arrays the themes on the horizontal axis and a categorical variable (e.g., different types of providers such as nurses, physician assistants, and doctors) on the vertical axis. It might be a table with key questions or concepts on the vertical axis and then two columns on the horizontal axis indicating qualitative responses and quantitative results to the concepts (Li, Marquart, & Zercher, 2000). The basic idea is for the researcher to jointly display both forms of data— effectively merging them—in a single visual.

Interpretation. The interpretation in the convergent approach is typically written into a discussion section of the study. Whereas the results section report on the findings from the analysis of both the quantitative and qualitative databases, the discussion section includes a report comparing the results from the two databases and notes whether there is convergence or divergence between the two sources of information. Typically the comparison does not yield a clean convergent or divergent situation, and the differences exist on a few concepts, themes, or scales. When divergence occurs, steps for follow-up exist. The researcher can state divergence as a limitation in the study without further follow-up. This approach represents a weak solution. Alternatively, mixed methods researchers can return to the analyses and further explore the databases, collect additional information to resolve the differences, or discuss the results from one of the databases as possibly limited (e.g., the constructs were not valid quantitatively or the qualitative themes did not match the open-ended questions).

Validity. Validity using the convergent approach should be based on establishing both quantitative validity (e.g., construct) and qualitative validity (e.g., triangulation) for each database. Is there a special form of mixed methods validity that needs to be addressed? There are certainly some potential threats to validity in using the convergent approach, and several of these have already been mentioned. Unequal sample sizes may provide less of a picture on the qualitative side than the larger N on the quantitative side. The use of different concepts or variables on both sides, quantitative and qualitative, may yield incomparable and difficult to merge findings. A lack of follow-up on conclusions when the scores and themes diverge also represents an invalid strategy of inquiry.

Explanatory Sequential Mixed Methods Design

Description of the design. The explanatory sequential mixed methods approach is a design in mixed methods that appeals to individuals with a strong quantitative background or from fields relatively new to qualitative approaches. It involves a two-phase project in which the researcher collects quantitative data in the first phase, analyzes the results, and then uses the results to plan (or build on to) the second, qualitative phase. The quantitative results typically inform the types of participants to be purposefully selected for the qualitative phase and the types of questions that will be asked of the participants. The overall intent of this design is to have the qualitative data help explain in more detail the initial quantitative results. A typical procedure might involve collecting survey data in the first phase, analyzing the data, and then following up with qualitative interviews to help explain the survey responses.

Data collection. The data collection proceeds in two distinct phases with rigorous quantitative sampling in the first phase and with purposeful sampling in the second, qualitative phase. One challenge in this strategy is to plan adequately what quantitative results to follow up on and what participants to gather qualitative data from in the second phase. The key idea is that the qualitative data collection builds directly on the quantitative results. The quantitative results that then are built on may be extreme or outlier cases, significant predictors, significant results relating variables, insignificant results, or even demographics. For example, when using demographics, the researcher could find in the initial quantitative phase that individuals in different socioeconomic levels respond differently to the dependent variables. Thus, the follow-up qualitatively may group respondents to the quantitative phase into different categories and conduct qualitative data collection with individuals representing each of the categories. Another challenge is whether the qualitative sample should be individuals that are in the initial quantitative sample. The answer to this question should be that they are the same individuals, because the intent of the design is to follow up the quantitative results and explore the results in more depth. The idea of explaining the mechanism—how the variables interact —in more depth through the qualitative follow-up is a key strength of this design.

Data analysis. The quantitative and the qualitative databases are analyzed separately in this approach. The quantitative results are then used to plan the qualitative follow-up. One important area is that the quantitative results cannot only inform the sampling procedure but it can also point toward the types of qualitative questions to ask participants in the second phase. These questions, like all good qualitative research questions, are general and open-ended. Because analysis proceeds independently for each phase, this design is useful for student research and perhaps easier to accomplish (than the convergent design) because one database builds on the other and the data collection can be spaced out over time.

Interpretation. The mixed methods researcher interprets the follow up results in a discussion section of the study. This interpretation follows the form of first reporting the quantitative, first-phase results and then the qualitative, second phase results. However, this design then employs a third form of interpretation: how the qualitative findings help to explain the quantitative results. A common misstep at this point by beginning researchers is to merge the two databases. While this approach may be helpful, the intent of the design is to have the qualitative data help to provide more depth, more insight

into the quantitative results. Accordingly, in the interpretation section, after the researcher presents the general quantitative and then qualitative results, a discussion should follow that specifies how the qualitative results help to expand or explain the quantitative results. Because the qualitative database questions narrows the scope of the quantitative questions, a direct comparison of the two databases (as in the convergent design) means an inadequate comparison of variables or concepts.

Validity. As with all mixed methods studies, the researcher needs to establish the validity of the scores from the quantitative measures and to discuss the validity of the qualitative findings. In the explanatory sequential mixed methods approach, additional validity concerns arise. The accuracy of the overall findings may be compromised because the researcher does not consider and weigh all of the options for following up on the quantitative results. Attention may focus only on personal demographics and overlook important explanations that need further understanding. The researcher may also contribute to invalidate results by drawing on different samples for each phase of the study. This minimizes the important of one phase building on the other. The sample size may also be inadequate on either quantitative side of the study or the qualitative side. These are a few of the challenges than need to be built into the planning process for a good explanatory sequential mixed methods study.

Exploratory Sequential Mixed Methods Design

Description of the design. If we reverse the explanatory sequential approach and start with a qualitative phase first followed by a quantitative phase, we have an exploratory sequential approach. An exploratory sequential mixed methods is a design in which the researcher first begins by exploring with qualitative data and analysis and then uses the findings in a second quantitative phase. Like the explanatory sequential approach, the second database builds on the results of the initial database. The intent of the strategy is to develop better measurements with specific samples of populations and to see if data from a few individuals (in qualitative phase) can be generalized to a large sample of a population (in quantitative phase). For example, the researcher would first collect focus group data, analyze the results, develop an instrument based on the results, and then administer it to a sample of a population. In this case, there may not be adequate instruments measuring the concepts with the sample that the investigator wishes to study. In effect, the researcher employs a three-phase procedure with the first phase as exploratory, the second as instrument development, and the third as administering the instrument to a sample of a population.

Data collection. In this strategy, the data collection would occur in two phases with the initial qualitative data collection followed by the second quantitative data collection. The challenge is how to use the information from the initial phase in the second phase. Several options exist. The qualitative data analysis can be used to develop an instrument with good psychometric properties (i.e., validity, reliability). The qualitative data analysis will yield quotes, codes, and themes (see Chapter 8). The development of an instrument can proceed by using the quotes to write items for an instrument, the codes to develop variables that group the items, and themes that that group the codes into scales. This is a useful procedure for moving from the qualitative data analysis to scale development. Scale development also needs to follow good procedures for instrument design, and the steps for this including such ideas as item discrimination, construct validity, and reliability estimates (see DeVellis, 2012). Developing a good psychometric instrument that fits the sample and population under study is not the only use of this design. A researcher can analyze the qualitative data to develop new variables, to identify the types of scales that might exist in current instruments or to form categories of information that will be explored further in a quantitative phase. The question arises if the sample for the qualitative phase is the same for the quantitative phase. This cannot be, because the qualitative sample is typically much smaller than a quantitative sample needed to generalize from a sample to a population. Sometimes mixed methods researchers will use entirely different samples for the qualitative and quantitative components of the study (unlike the explanatory sequential design). However, a good procedure is to draw both samples from the same population but make sure that the individuals for both samples are not the same. To have individuals help develop an instrument and then to survey them in the quantitative phase would introduce confounding factors into the study.

Data analysis. In this strategy the researcher analyzes the two databases separately and uses the findings from the initial exploratory database to build into quantitative measures. This means that the researcher needs to pay careful attention to the qualitative data analysis steps and determine what findings to build on. If, for example, the researcher uses grounded theory (see Chapter 9), the theoretical model generated may provide a model to be tested in the second, quantitative phase. A qualitative case study can yield different cases that become the focus on important variables in the

second quantitative phase.

Interpretation. Researchers interpret the mixed methods results in a discussion section of a study. The order of interpretation is to first report the qualitative findings, the use of the qualitative results (e.g., the development of an instrument, the development of new quantitative measures), and then the quantitative results of the final phase of the study. It does not make sense to compare the two databases, because they are typically drawn from different samples (as noted above in the data collection discussion) and the intent of the strategy is to determine if the qualitative themes can be generalized to a larger sample.

Validity. Researchers using this strategy need to check for the validity of the qualitative data as well as the validity of the quantitative scores. Special validity concerns arise, however, in using this design that need to be anticipated by the proposal developer. One concern is that the researcher may not use appropriate steps to develop a good psychometric instrument. Developing a good instrument is not easy, and adequate steps need to be put in place. Another concern is that a researcher may develop an instrument or measures that do not take advantage of the richness of the qualitative findings. This occurs when the qualitative data lacks rigor or occurs simply at the theme level without the further data analysis steps associated with using one of the qualitative design-types, such as ethnography, grounded theory, or case study procedures. Finally, as previously mentioned, the sample in the qualitative phase should not be included in the quantitative phase as this will introduce undue duplication of responses. It is best to have the qualitative participants provide information for scale, instrument, or variable design but not to also be the individuals completing the follow-up instruments.

Several Advanced Mixed Methods Designs

Three advanced mixed methods designs incorporate the elements of the convergent, explanatory sequential, and exploratory sequential approaches. Once one has the foundation of the three basic approaches, they can be included into more advanced strategies that add further elements into the overall procedures (see Figure 10.2) (see also Creswell & Plano Clark, 2011). The embedded mixed methods design nests one or more forms of data (quantitative or qualitative or both) within a larger design (e.g., a narrative study, an ethnography, an experiment). For example, within an experiment, the researcher can collect qualitative data and collect it during the experiment (convergent) or before the experiment begins (sequentially) or after the experiment ends (sequentially). It is a popular design within the health sciences and when investigators test an intervention or program in an applied setting (e.g., in a school). A second strategy is to incorporate elements of the convergent, explanatory sequential, or exploratory sequential approaches within a social justice framework to help a marginalized group. This design is call transformative mixed methods, and the researcher uses a social justice theory as a framework for a mixed methods study. This theory (e.g., feminist, racial) frames many aspects of the mixed methods study, such as the research problem, the questions, the data collection and analysis, interpretation, and the call for action. It is popular in studying marginalized groups in various countries, especially third world countries, throughout the world and in the United States (e.g., indigenous populations, females, racial and ethnic groups, disabled individuals). Finally, another advanced design is the multiphase mixed methods in which researchers conduct several mixed methods projects, sometimes including mixed methods convergent or sequential approaches, sometimes including only quantitative or qualitative studies in a longitudinal study with a focus on a common objective for the multiple projects. This form of research is popular in the evaluation or program implementation fields in which multiple phases of the project stretch over time. These projects may go back and forth between quantitative, qualitative, and mixed methods studies, but they build on each other to address a common program objective.

Mixed Methods Notation in the Figures

In Figures 10.1 and 10.2, we see that some notation and labels are used to convey the procedures in mixed methods strategies. Over the years, these shorthand labels have become popular in the mixed methods field. Mixed methods notation provides shorthand labels and symbols that convey important aspects of mixed methods research, and they provide a way that mixed methods researchers can easily communicate their procedures (see Table 10.2). Morse (1991) first developed the notation, and it has been added to by writers such as Tashakkori and Teddlie (1998) and Plano Clark (2005) who suggest the following:

• QUAL and QUAN capitalization indicates an emphasis or priority on the quantitative or qualitative data, analysis, and interpretation in the study. In a mixed methods study, the qualitative and quantitative data may be equally emphasized, or one may be more emphasized than the other. Capitalization indicates that an approach or method is emphasized. Lowercase indicates lesser priority or emphasis on the method.

• Quan and Qual stand for quantitative and qualitative, respectively, and they use the same number of letters to indicate equality between the forms of data.

• A plus sign—+—indicates a convergent or merging integration of data collection—with both quantitative and qualitative data collected at same time.

• An arrow—→—indicates a sequential form of data collection; one form (e.g., qualitative data) builds or connects with the other (e.g., quantitative data).

• Parentheses—( )—indicate that one form of data collection is embedded within another or embedded within a larger design.

• Double arrows—→←—mean that the flow of activities can go both ways.

• Also in the figures we see boxes that highlight important major components of the design—such as data collection or data analysis.

Table 10.2 Notation Used in Mixed Methods Research

Factors Important in Choosing a Mixed Methods Design

The choice of a particular mixed methods design is based on several factors that relate to the intent of the procedures as well as practical considerations. I will begin with the procedural reasons for choosing a particular mixed methods strategy. It should be recognized that many variations exist in mixed methods designs, and the particular approach an investigator has in mind may not conform exactly to the approaches specified here. However, these designs represent the common underlying features of many designs, and, with modification, researchers can find their own strategy. Consider the amount of time you have to collect data. Concurrent approaches are less time consuming because both qualitative and quantitative data are collected at the same time in the same visit to the field.

Choice Based on Outcomes Expected Earlier in this chapter, I reviewed the reasons for choosing mixed methods research. In Table 10.3 ,

I repeat the reasons but this time link them to expected outcomes of a mixed methods project and the type of mixed methods strategy. This thinking calls for the researcher to determine the outcome anticipated at the end of the mixed methods study and then to link it to the types.

Choice Based on How the Data Will Be Used Together (or Integrated) To choose a mixed methods strategy beyond considering the outcome anticipated, the researcher

needs to consider whether mixed methods integration of the two databases will be merged, connected, or embedded. Merging the data involves combining the quantitative and qualitative data through the procedures of a side-by-side comparison, data transformation, or a joint display. Connecting the data means that the analysis of one data set is used to lead into or build into the second data set. In short, the data analysis of one data set informs the data collection of the other data set. In embedding, one data set—involving quantitative, qualitative, or combined data—is embedded within a larger design.

For example, in a convergent design the two are considered to be independent and the data collection and analysis proceeds for each database separately. In an embedded experimental design, the qualitative data may be collected independently of the experiment and used to support or augment the larger design, the experiment. Alternatively, the two databases may be connected with one building on the other. This is a sequential type of design (explanatory sequential strategy or an exploratory sequential strategy), and one database does not stand in isolation of the other database. In these sequential designs, the data collection in the second phase cannot be conducted until the first phase results are in. In short, the follow-up data collection builds directly on the results of the initial data collection.

Table 10.3 Choosing a Mixed Methods Project, Expected Outcomes, Type of Design

Re as ons for Choos ing Mixe d Me thods Expe cte d Outcome s Re comme nde d Mixe d Me thods De s ign

Comparing different perspectives drawn from quantitative and qualitative data

Merging the two databases to show how the data convergent or diverge

Convergent parallel mixed methods design

Explaining quantitative results with qualitative A more in-depth understanding of the quantitative Explanatory sequential

data results (often cultural relevance) mixed methods design

Developing better measurement instruments A test of better measures for a sample of a population Exploratory sequential mixed methods design

Understanding experimental results by incorporating perspectives of individuals

An understanding of participant views within the context of an experimental intervention

Embedded mixed methods design

Developing an understanding of needed changes for a marginalized group

A call for action Transformative mixed methods design

Understanding the need for an impact of an intervention program

A formative and summative evaluation Multiphase mixed methods design

Choice Based on the Timing of the Data Collection A related factor is timing in mixed methods data collection—whether the two databases are

collected concurrently, at roughly the same time, or with one following the other, sequentially. A convergent strategy typically involves collecting data concurrently while the explanatory sequential and the exploratory sequential strategies means that the data will be collected in sequence. Sometimes this criterion is difficult to identify in published mixed methods studies, but it should go into the thinking about choosing a mixed methods strategy.

Choice Based on the Emphasis Placed on Each Database Like timing, the emphasis placed on each database in mixed methods research is also somewhat

difficult to determine and to apply to the question of choice. A mixed methods study can illustrate an equal emphasis (or priority or weight) on both databases, or an unequal emphasis. For example, a mixed methods project can stress the qualitative phase of the research and give minimal attention to the quantitative phase. How can we tell? We can look at the number of pages in a study to determine emphasis, how the study begins (e.g., with a strong quantitative theory orientation or personal qualitative stories), the amount of depth and sophistication given to the qualitative and quantitative data collection and analysis, or even the background training of the investigator. As mentioned earlier in the section on notation, capital letters are used in the notation for greater emphasis (e.g., QUAN) and lowercase letters for less emphasis (e.g., quan). The emphasis can help determine the choice of a mixed methods strategy. Typically if the researcher seeks to emphasize both databases, a convergent approach is best. Alternatively, if a stronger emphasis is sought for the quantitative approach, then an explanatory sequential strategy is used because it began with the quantitative component of the study. If a qualitative approach is to be emphasized, then an exploratory sequential strategy is chosen. These are not firm guidelines, but they may play into the overall decision about a choice of strategy.

Choice Based on Type of Design Most Suited for a Field On a practical level, the choice of a strategy depends on the inclination of fields toward certain

designs. For quantitatively oriented fields, the explanatory sequential approach seems to work well because the study begins (and perhaps is driven) by the quantitative phase of the research. In qualitatively oriented fields, the exploratory sequential approach may be more appealing because it begins with an exploration using qualitative research. However, in this approach, an outcome may be a measurement instrument that is tested so that the outcome, a quantitative outcome, outweighs in

importance how the study began. In some fields, the choice of approach may be dependent on collecting data efficiently, and this would argue for a convergent mixed methods study in which both quantitative and qualitative data are typically collected at roughly the same time rather than at different times that require more visits to the research site.

Choice Based on a Single Researcher or Team A final practical reason for a choice of a strategy depends on whether a single researcher (e.g.,

graduate student) conducts the study or a team of researchers (e.g., funded long-term investigation). If the investigator is a single researcher, the sequential strategies of an explanatory sequential or exploratory sequential approach are best because the investigation can be divided into two manageable tasks rather than multiple data collection and analysis procedures. The study can be projected out over a period of time rather than collecting multiple forms of data at the same time as in a convergent approach. When time is a problem, I encourage students to think about an embedded model of design. This model emphasizes the use of a recognized research design (e.g., an experiment), and it includes within it a minor, secondary form of data collection (e.g., a few interviews with some of the participants). The fact that both forms of data are not equal in size and rigor enables the study to be reduced in scope and manageable for the time and resources available. For students, I recommend that they find a published mixed methods journal article that uses their design and introduce it to advisers and faculty committees so that they have a working model to understand the design. Since we are at the early stage of adopting mixed methods research in many fields, a published example of research in a field will help to create both legitimacy for mixed methods research and the idea that it is a feasible approach to research for graduate committees or other audiences. If a research team is conducting the study, multiple forms of data collection at the same time or over a long period of time are possible, such as in an embedded or a multiphase design. Although a single researcher can conduct a transformative study, the labor-intensive nature of collecting data in the field involving participants as collaborators typically suggests more of a team approach than the inquiry by a single investigator.

Examples of Mixed Methods Procedures

Illustrations follow of mixed methods studies that use both the sequential and concurrent strategies and procedures.

Example 10.1 A Convergent Parallel Mixed Methods Design

Classen et al. (2007) studied older driver safety in order to develop a health promotion intervention based on modifiable factors influencing motor vehicle crashes with older drivers (age 65 and older). It was a good example of a convergent mixed methods study. The central purpose of the study was identified in the abstract:

This study provided an explicit socio-ecological view explaining the interrelation of possible causative factors, an integrated summary of these causative factors, and empirical guidelines for developing public health interventions to promote older driver safety. Using a mixed methods approach, we were able to compare and integrate main findings from a national crash dataset with perspectives of stakeholders. (p. 677)

This purpose statement identified the use of both quantitative (i.e., a national crash data set) and a qualitative (i.e., stakeholders’ perspectives) data. From one of the research questions in the study, we learned that the authors compared the qualitative stakeholder perspectives, needs, and goals for safe and unsafe driving with the quantitative results of the factors that influenced driving injuries. So the expected outcome was to compare the findings. The method section commented on the quantitative national data set, the statistical analysis of this data set, and then the qualitative data set and its analysis. Although not stated explicitly, the data were used together to form results, not used for one database to build on another, and the timing was to look at both databases concurrently. A diagram illustrated the procedures involved in both collecting and analyzing the information. A results section first reported the quantitative results and then the qualitative results. More emphasis was given to the quantitative results, leading to the conclusion that this study favored the quantitative research. However, these reports on the results from the two databases were followed by an analysis of key findings in which the quantitative and qualitative results were compared for supportive and nonsupportive findings. In this discussion section, the researchers merged the two databases in a side- by-side comparison. Looking more broadly at the topic and the authors, we saw that the quantitative emphasis would probably be better accepted in the field of occupational therapy than qualitative research. Also, a scan of the authors’ biographical sketches showed that this mixed methods study was completed by a team of researchers drawn from individuals with quantitative and qualitative expertise.

Example 10.2 An Explanatory Sequential Mixed Methods Design

In 2007, Banyard and Williams conducted an explanatory sequential mixed methods study examining how women recover from childhood sexual abuse. The quantitative component of the study

consisted of structured interviews (with 136 girls in 1990 and a subset of 61 girls in 1997) looking at resilience, correlates of resilience, over time across 7 years of early adulthood. The qualitative aspect consisted of interviews with a subset of 21 girls about their life events, coping, recovery, and resilience. The intent of the mixed methods study was to use the qualitative interviews to “explore and make sense” of the quantitative findings (p. 277). Here was the purpose statement:

Multiple methods are used to examine aspects of resilience and recovery in the lives of female survivors of child sexual abuse (CSA) across 7 years of early adulthood. First quantitative changes in measures of resilience over time were examined. To what extent did women stay the same, increase, or decrease in functioning in a variety of sphere across 7 years during early adulthood? Next, the role of re-traumatization as an impediment to ongoing resilience and correlates of growth or increased well-being over time were examined. Finally, because resilient processes in adulthood have not been the focus of much research and require further description, qualitative data from a subset of participants was used to examine survivors’ own narratives about recovery and healing to learn about key aspects of resilience in women’s own words. (p. 278)

As suggested by this statement, the expected outcome of the study was projected to be a detailed picture of resilience and the personal perspectives of the survivors as learned through qualitative data. Also, the authors intended to probe the quantitative findings, to explain them in more detail through the qualitative data. With this intent, the study set up as a sequential approach, with the two databases connected and one building on the other. Also, with this approach, the timing illustrated the qualitative data collection followed the quantitative results. It was difficult to discern whether this study placed greater emphasis on the quantitative or qualitative component of the project. The project began with a quantitative longitudinal phase with extensive discussions of the measures used to gather data. The authors detailed the quantitative results. However, the qualitative findings illustrated many themes that emerged from the interviews with the women. These themes pointed toward new issues that helped to develop the concept of resilience, such as the turning points in the women’s lives, the ongoing nature of recovery, and the role of spirituality in recovery. The study was conducted by a team of researchers from psychology and criminal justice and supported by the National Institutes of Health (NIH).

Example 10.3 An Exploratory Sequential Mixed Methods Design

A good example of an exploratory sequential study with an experimental test outcome is found in Betancourt et al. (2011). This study used mixed methods research to adapt and evaluate a family strengthening intervention in Rwanda. The investigators sought to examine the mental health problems facing HIV-affected children in Rwanda. They first began with an exploratory, qualitative first phase of interviews with children and their caregivers. From a qualitative thematic analysis of the data, they then performed an extensive review of the literature to locate standardized measures that matched their qualitative findings. They found some measures and added some new ones to develop a survey instrument. This instrument went through several refinements following rigorous procedures of instrument-scale development (e.g., backward and forward translations, a discussion

of items, reliability and validity) to develop good construct validity for the measures. These measures (e.g., family communication, good parenting, and others) then became the pretest and posttest assessments in an experimental (intervention) study. For the intervention in the study, the researchers were led to a strengths-based, family-based prevention program that was hypothesized to be related to the measures. The final step in the mixed methods process was to use the validated measures within a program that featured the prevention program. At various points in this study, the researchers also collaborated with stakeholders to help to develop good measures. Thus, this study illustrated a good, complex mixed methods project with an initial qualitative phase, an instrument development phase, and an experimental phase. It shows how an initial exploration qualitatively can be used to support a later quantitative testing phase. They stated the purpose of the study as follows:

In the multi-step process used in this mental health services research, we aimed to (1) carefully unpack locally-relevant indicators of mental health problems and protective resources using qualitative methods; (2) apply qualitative findings to the adaptation of mental health measures and the development of a locally-informed intervention; (3) validate the selected mental health measures; and (4) apply the measures to rigorous evaluation research on the effectiveness of the intervention chosen through the mixed methods process. (p. 34)

In this mixed methods study, the expected outcome was clearly to develop good psychometric measures and then to use the measures as outcomes in an experimental project. It was also to use the qualitative data to develop hypotheses that might be tested using the intervention in the experiment. The initial phase qualitative data collection was connected to the subsequent quantitative measures and their rigorous testing for scores on validity and reliability. The entire project was timed for the quantitative phase to follow the qualitative phase, and the quantitative phase could be stated as the development of the measures (and survey) and the experimental intervention study. If I were to diagram this project, it would be qual QUAN QUAN. As this notation shows, the emphasis in the project favored quantitative research, and the project could be seen as pointing toward the program intervention test at the end of the article. Recognizing that the researchers came from public health, an organization called Partners in Health, and a children’s hospital, the strong quantitative orientation of the project makes sense. Overall, this mixed methods study illustrated both the core exploratory sequential design and the more advanced embedded experimental design with a sequential focus. To conduct such a complex project, the study involved a team of researchers both in the United States and in Rwanda.

Example 10.4 A Transformative Design

The final example is a feminist study using a transformative explanatory sequential mixed methods study by Hodgkin (2008). This study investigated the concept of social capital for men and women in households in a regional city in Australia. Social capital described norms and networks that enabled people to work collectively together to address and resolve common problems (e.g., through social activities, the community, and civic participation). The basic mixed methods approach was an explanatory sequential design with an initial survey, a quantitative phase, followed

by an interview, qualitative phase. As stated by the author, “the qualitative study elaborated on and enhanced some of the results from the quantitative study” (p. 301). In addition, the author declared that this was a feminist mixed methods project. This means that Hodgkin used a feminist framework (see Chapter 3) to encase the entire mixed methods project. She also referred to the Merten’s transformative research paradigm (Mertens, 2007) that gave voice to women, used a range of data collection methods, and bridged the subjective and objective ways of knowing (see the epistemology discussion in Chapter 3). The purpose of the study was this:

The author will provide examples of quantitative data to demonstrate the existence of different social capital profiles for men and women. Stories will also be presented to provide a picture of gender inequality and expectation. The author will conclude by arguing that despite reluctance on the part of feminists to embrace quantitative methods, the big picture accompanied by the personal story can bring both depth and texture to a study. (p. 297)

Thus, in this mixed methods study, the expected outcome for the study was to help explain the initial survey results in more depth with qualitative interview data. Added to this would be the transformative perspective of seeking to provide a picture of gender inequality and expectations. The databases were used sequentially with the qualitative interviews following and expanding on the quantitative surveys. While the surveys were sent to both men and women in households (N =1431), the interviews included only women in the survey sample (N = 12). The women interviewed were of different ages, they varied in terms of their work activities (inside and outside the home), they were mothers, and they varied in their educational level of attainment. The timing of the data collection was in two phases with the second-phase qualitative interviews building on the results from the first- phase quantitative surveys. In fact, the survey data indicated that men and women differed in terms of their level of social participation in groups, and in community group participation. The emphasis in this study seemed to be equal between the quantitative and qualitative components, and clearly the sole author of the study sought to provide a good example of mixed methods research that used a feminist framework. How was this framework used? The author announced at the beginning of the study that “the aim of this article is to demonstrate the use of mixed methods in feminist research” (p. 296). Then the author discussed the lack of qualitative research in the empirical studies of social capital and noted the White, middle-class notion of community that dominated the discussions of social capital. Further, the author talked about lifting up the voices of those disenfranchised by gender and engaged in a study that first pointed out gender differences in social, community, and civic participation within a large sample of men and women, and then focused a qualitative follow-up on only women to understand the women’s role in more depth. The qualitative findings then addressed themes that influence womens’ participation, such as wanting to be a “good mother,” wanting to avoid isolation, and wanting to be a good citizen. A summary of the qualitative findings indicates specifically how the qualitative data helped to enhance the findings of the initial survey results. Unlike many feminist mixed methods studies, the conclusion did not indicate a strong call for action to change the inequality. It only mentioned in passing that the mixed methods study provided a powerful voice to gender inequality.

SUMMARY

In designing the procedures for a mixed methods discussion, begin by defining mixed methods research and its core characteristics, briefly mentioning its historical evolution; discuss your chosen mixed methods design; and note the challenges in using the design. Convey a diagram of your procedures that includes good notation to help the reader understand the flow of activities. As you discuss your design, convey the elements that go into it, such as the procedures used in a convergent parallel, an explanatory sequential, or an exploratory sequential mixed methods study. Also consider whether you will overlay your project with a more advanced procedure that embeds the data or mixed methods within a larger design; uses a transformative framework that advocates for social justice; or strings together multiple quantitative, qualitative, or mixed methods studies into a longitudinal line of investigation, all targeted toward a single objective. Finally, consider factors that play into your choice of a mixed methods design. These involve considering what outcomes you expect from the study, the integration of the databases, the timing of them, the emphasis placed on each database, the choice of design that matches your field, and the conduct of the project either by yourself or a team of researchers.

Writing Exercises 1. Design a combined qualitative and quantitative study that employs two phases sequentially.

Discuss and provide rationales for why the phases are ordered in the sequence you propose. 2. Design a combined qualitative and quantitative study that gives emphasis to qualitative data

collection and less emphasis to quantitative data collection. Discuss the approach to be taken in writing the introduction, the purpose statement, the research questions, and the specific forms of data collection.

3. Develop a figure and specific procedures that illustrate the use of a theoretical lens, such as a feminist perspective. Use the procedures of either an explanatory or exploratory design for conducting the study. Use appropriate notation in the figure.

ADDITIONAL READINGS

Greene, J. C., Caracelli, V. J., & Graham, W. F. (1989). Toward a conceptual framework for mixed- method evaluation designs. Educational Evaluation and Policy Analysis, 11(3), 255–274.

Jennifer Greene and associates undertook a study of 57 mixed methods evaluation studies reported from 1980 to 1988. From this analysis, they developed five different mixed methods purposes and seven design characteristics. They found the purposes of mixed methods studies to be based on seeking convergence (triangulation), examining different facets of a phenomenon (complementarity), using the methods sequentially (development), discovering paradox and fresh perspectives (initiation), and adding breadth and scope to a project (expansion). They also found that the studies varied in terms of the assumptions, strengths, and limitations of the method and whether they addressed different phenomena or the same phenomena; were implemented within the same or different paradigms; were given equal or different weight in the study; and were implemented independently, concurrently, or sequentially. Using the purposes and the design characteristics, the authors have recommended several mixed methods designs.

Morse, J. M. (1991). Approaches to qualitative-quantitative methodological triangulation. Nursing Research, 40(1), 120–123.

Janice Morse suggests that using qualitative and quantitative methods to address the same research problem leads to issues of weighing each method and their sequence in a study. Based on these ideas, she then advances two forms of methodological triangulation: (a) simultaneous, using both methods at the same time, and (b) sequential, using the results of one method for planning the next method. These two forms are described using a notation of capital and lowercase letters that signify relative weight as well as sequence. The different approaches to triangulation are then discussed in the light of their purpose, limitations, and approaches.

Plano Clark, V. L. & Creswell, J. W. (2008). The mixed methods reader. Thousand Oaks, CA: Sage.

Creswell and Plano Clark have developed a comprehensive guide to designing the steps in conducting mixed methods research. This design theme is carried forward in specific examples of published mixed methods studies in this reader. Examples are provided of the convergent design, the explanatory sequential design, and the exploratory sequential design. Also, the book contains key articles throughout the years that have informed the development of the field of mixed methods.

Tashakkori, A., & Teddlie, C. (Eds.). (2010). SAGE handbook of mixed methods in social and behavioral research (2nd ed.). Thousand Oaks, CA: Sage.

This handbook, edited by Abbas Tashakkori and Charles Teddlie, represents a major effort to map the field of mixed methods research. The chapters provide an introduction to mixed methods, illustrates methodological and analytic issues in its use, identifies applications in the social and human sciences, and plots future directions. For example, separate chapters illustrate the use of mixed methods research in evaluation, management and organization, health sciences, nursing, psychology, sociology, and education.