Research 22

5/28/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252337/View?ou=122307 1/12

T

CHAPTER 10

Mixed Methods Research

Some evaluators employ randomized controlled trials (RCTs) as the gold standard of evidence-based practice (EBP). Critics of RCT designs argue that RCTs do not include the complexity of program

participants’ experiences. . . . I argue that weaving in a subjectivist methodology and shifting methodological perspectives and methods into RCT-based evaluations prior to, during, and after the

RCT design serves to enhance the credibility and social justice [of] RCT praxis. —Hess-Biber, 2013, p. 49

In This Chapter • Mixed methods research is defined, and typical characteristics are discussed.

• The importance of mixed methods in educational and psychological research is explored.

• The philosophical assumptions of this approach are explained, along with the methodological implications of those assumptions.

• Specific designs for mixed methods research are presented, along with examples of these approaches.

• Questions for critically analyzing mixed methods research are provided.

Youngs and Piggot-Irvine (2012) conducted a pragmatic mixed methods evaluation of a program designed to prepare people who aspire to become school principals in New Zealand. They collected national-level quantitative data from questionnaires and qualitative data from document analysis, semistructured phone interviews, and focus groups, as well as data from regional case studies.

Ungar and Liebenberg (2011) wondered what resilience looked like in youth who thrive despite living with adversity. They used a transformative mixed methods approach, rooted in local understandings of the concept in 11 countries, to develop an instrument to capture cross-cultural aspects of resilience and to test the quality of that instrument.

wo sample studies that used mixed methods are included in Chapter 1. Berliner, Barrat, Fong, and Shirk’s (2008) study of school dropouts and reenrollment, summarized in Sample Study 1.5, illustrates research that used mixed methods within the pragmatic paradigm. Boddy (2009)

study of the effect of a mentoring program for poor women in Sample Study 1.4 is an evaluation study that used mixed methods in the transformative paradigm. The vocabulary and methodological designs and their implications are discussed in this chapter.

Definition and Characteristics Mixed methods designs include both qualitative and quantitative features in the design, data collection, and analysis (Teddlie & Tashakkori, 2009).1 In the first issue of Journal of Mixed Methods Research, Tashakkori and Creswell (2007) define mixed methods as “research in which the investigator collects and analyzes data, integrates the findings, and draws inferences using both qualitative and quantitative approaches or methods in a single study or program of inquiry” (p. 4). Hence, mixed methods can refer to the use of both quantitative and qualitative methods to answer research questions in a single study, as well as those studies that are part of a larger research program and are designed as complementary to

p. 304

p. 303

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 05/28/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

5/28/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252337/View?ou=122307 2/12

provide information related to several research questions, each answered with a different methodological approach. While mixed methods have an intuitive appeal, they also demand that the researcher be expert in both approaches to research or work with a team that has such expertise. Issues related to the design of studies, as well as to ensuring the quality of a mixed methods approach, are explored in this chapter.

Teddlie and Tashakkori (2010) described a truly mixed approach methodology as methodologically eclectic, meaning that the researcher selects and synergistically integrates “the most appropriate techniques from a myriad of QUAL, QUAN, and mixed methods to more thoroughly investigate a phenomenon of interest” (p. 8). The intent may be to seek a common understanding through triangulating data from multiple methods or to use multiple lenses simultaneously to achieve alternative perspectives that are not reduced to a single understanding. Educational and psychological researchers mix methods to a varying degree at various points in their research, although they may not use mixed methods at every stage of their studies. Researchers can insert multiple mixed options into their work at various points in the research process, including the definition of purpose, overall design, methods, sampling, data recording, analysis, and interpretation.

Importance in Educational and Psychological Research Mixed methods have particular value when a researcher is trying to solve a problem that is present in a complex educational or social context (Teddlie & Tashakkori, 2009). Because mixed methods designs incorporate techniques from both the quantitative and qualitative research traditions, they can be used to answer questions that could not be answered in any other way. Many researchers have used mixed methods because it seemed intuitively obvious to them that this would enrich their ability to draw conclusions about the problem under study. Onwuegbuzie and Combs (2010) cite Green et al.’s (1989) typology of mixed methods research purposes to provide more nuanced reasons to use mixed methods:

triangulation (i.e., compare findings from the qualitative data with the quantitative results); complementarity (i.e. seek elaboration, illustration, enhancement, and clarification of the findings from one analytical strand [e.g., qualitative] with results from the other analytical strand [e.g., quantitative]); development (i.e., use the results from one analytical strand to help inform the other analytical strand); initiation (i.e., discover paradoxes and contradictions that emerge when findings from the two analytical strands are compared that might lead to a reframing of the research question); and expansion (i.e., expand breadth and range of a study by using multiple analytical strands for different study phases. (p. 411, bold added)

When the purpose of the research is complex, the researcher often needs to ask multiple questions, which frequently necessitates the use of mixed methods. Mixed methods have the potential to contribute to addressing multiple purposes and thus to meeting the needs of multiple audiences for the results. Youngs and Piggot-Irvine (2012) answered several questions in this mixed methods evaluation of a school principal preparation program. The questions focused on the quality of the program design, delivery, recruitment and retention, and outcomes. They described their desire to answer multiple questions using the most appropriate type of data needed to answer those questions,

Hesse-Biber (2013) and White (2013) both argue that mixed methods designs can enhance research that uses randomized control trials (RCTs) described in Chapter 4 of this book. Hesse-Biber (2013) argues for the inclusion of qualitative methodologies before, during, and after the RCT in order to enhance the appropriateness of the intervention for the targeted population, explore the dynamics associated with changes in outcomes, and increase the credibility of the results. White (2013) claims that RCTs are the most appropriate design for establishing cause and effect in the sense of determining the effectiveness of an intervention. However, he encourages researchers and evaluators to use qualitative approaches in conjunction with the RCT to answer questions related to the quality of the intervention design, the targeting of audiences, barriers to participation, and adoption of the

p. 305

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 05/28/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

5/28/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252337/View?ou=122307 3/12

intervention by intended beneficiaries.

EXTENDING YOUR THINKING

Mixed Methods Advantages and Disadvantages

1. In your opinion, what are the advantages and disadvantages of using mixed methods in psychological or educational research?

2. Talk to potential users of research information or think of yourself as a consumer. What are the strengths and weaknesses of a mixed methods approach?

Philosophical Assumptions: Pragmatism, Mixing Paradigms, and Transformation On the basis of a review of social research that claimed to use mixed methods, Greene and Caracelli (2003) concluded that inquiry decisions are rarely, if ever, consciously rooted in philosophical assumptions or beliefs. Rather, researchers based their choice of mixed methods on the nature of the phenomena being investigated, the contexts in which the study is conducted, or the funding agencies’ requirements. Nevertheless, they did not conclude that paradigms and their associated philosophical assumptions were irrelevant—merely unexamined. They further suggested that by attending too little to philosophical ideas and traditions, many mixed method inquirers are insufficiently reflective and their practice insufficiently unproblematized. Examining the philosophical assumptions underlying research, mixed methods or not, can offer a better understanding of the complex social world in which educators and psychologists operate. (See a list of options concerning the relationship between paradigmatic stance and choice of mixed methods in Box 10.1.)

BOX 10.1 Possible Reactions to Paradigms

• Incompatibility Thesis: Mixed methods are impossible due to the incompatibility of the paradigms underlying the methods.

• Complementary Strengths: Adhere to the tenets of rigor as defined within each paradigm.

• Multiple Paradigms Drive Mixed Methods: Mixed methods can be approached from a pragmatic or transformative paradigm.

• Dialectical Pragmatism: The research question is the dictator of method; qualitative and quantitative methods are taken seriously and results of both methods are synthesized for each study.

• Transformative Mixed Methods: Mixed methods enhance the ability of researchers to capture the complexity of issues of human rights and social transformation.

• Dialectical Thesis: All paradigms have something to offer; research will advance through the deliberate and critical dialogues that occur among scholars who situate themselves in different paradigms (Greene, 2007).

• A-Paradigmatic: Ignore philosophical assumptions (at your own risk).

SOURCE: Greene (2007); Mertens (2009); Teddlie and Tashakkori (2009).

Teddlie and Tashakkori (2009) propose the use of pragmatism as one philosophical orientation to guide mixed methods researchers. Simply put, pragmatists consider the research question to be more important than either the method they use or the worldview that is supposed to underlie the method. These researchers use the criterion “what works?” to determine which method to use to answer a

p. 306

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 05/28/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

5/28/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252337/View?ou=122307 4/12

ese esea c e s use t e c te o w at wo s? to dete e w c et od to use to a swe a specific research question. Thus, the pragmatic orientation rejects the either/or mentality suggested by a choice between the postpositive and the constructivist paradigms. Here is how Teddlie and Tashakkori (2009) describe the pragmatic researcher:

Pragmatists decide what they want to study based on what is important within their personal value systems. They then study the topic in a way that is congruent with their value system, including units of analysis and variables that they feel are most likely to yield interesting responses. . . . This description of pragmatists’ behaviors is consistent with the way that many researchers actually conduct their studies, especially research that has important social consequences. (pp. 90–91)

This pragmatic basis can be contrasted with that of the transformative paradigm in which the emphasis is on the inclusion of values and viewpoints, especially of marginalized groups, as the driving force for all aspects of the research. Mertens (2009) questions the notion of the centrality of the researcher’s values in the pragmatic paradigm. She raises questions in terms of which values, whose values, and the role of the researcher within the context of values. While Teddlie and Tashakkori (2009) claim that practicality should serve as the value basis for a researcher’s choices, researchers within the transformative paradigm would ask, Practical for what? What are the intentions of the researcher, client, or policymakers? Should the researcher/evaluator be guided by a higher social purpose than what is found to be useful to those in power?

Mertens (2009) noted that a mixed methods design could also fit within a transformative framework if it was designed to reflect, in both perspective and outcomes, a dedication to social change at levels ranging from the personal to the political. Furthermore, it is possible to conduct almost any mixed methods study with a transformative or advocacy purpose.

EXTENDING YOUR THINKING

Paradigms and Mixed Methods What are the philosophical assumptions that underlie the mixed methods approach? How can you integrate the assumptions of the three major paradigms (postpositivist, constructivist, and transformative) to guide your thinking in the use of mixed methods? What are the assumptions of the pragmatic paradigm and their implications for educational and psychological research?

Mixed Methods Design Options The specific mixed methods approaches are defined by the ordering of the application of the quantitative and qualitative methods (simultaneously or sequentially), as well as at what point the mixing of methods occurs. Qualitative and quantitative data collection can occur in parallel form or sequential form.

• Parallel form: Concurrent mixed methods/model designs in which two types of data are collected and analyzed.

• Sequential form: One type of data provides a basis for collection of another type of data.

Creswell (2009) uses these two basic design types to organize possible variations in mixed methods designs, although he uses the term concurrent in place of parallel. I agree with Teddlie and Tashakkori (2009) that parallel is a more inclusive term than concurrent because the latter term implies the two methods must be used at the same time rather than the inference from the term parallel, which is that they occur in proximity to each other. Creswell (2009) describes possible twists, such as parallel (concurrent) designs that are explanatory (qualitative data are collected after quantitative data to try to explain the quantitative results) or exploratory (qualitative data collection is followed by quantitative data collection to explore a phenomenon), and sequential designs that are triangulated (use of both quantitative and qualitative data to see if there is convergence in findings) or embedded (one type of

p. 308

p. 307

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 05/28/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

5/28/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252337/View?ou=122307 5/12

quantitative and qualitative data to see if there is convergence in findings) or embedded (one type of data plays a dominant role while the other type of data plays a supportive role).

Mixed methods can also involve the conversion of qualitative data to a quantitative form or vice versa; Teddlie and Tashakkori (2009) call this a conversion design. They also add another design called multilevel design in which quantitative data are collected at one level of an organization (e.g., student level) and qualitative data are collected at another level (e.g., administrator level). The mixed methods community also talks about designation of approaches as dominant or less dominant (or supportive), depending on the extent to which the overall study uses either quantitative or qualitative approaches— to the same extent or one or the other to a greater or lesser extent (Teddlie & Tashakkori, 2009). Mertens (2009) and Mertens and Wilson (2012) added a transformative cyclical design in which different methods are used at different times during the life cycle of a study, with the data collected at each stage informing decisions about the data needed in the next stage.

As I believe that paradigmatic stances are the beginning point for decision making in research, I separate the main design options into the two paradigms that are salient in the mixed methods research community: pragmatic and transformative. Table 10.1 provides a summary of four design options for a mixed methods approach.2 Following a discussion of these approaches, the transformative cyclical design is discussed.

Pragmatic Parallel Mixed Methods Design The pragmatic parallel mixed methods design is one in which qualitative and quantitative data are collected and analyzed to answer a single study’s research questions (Onwuegbuzie & Teddlie, 2002). The final inferences are based on both data analysis results. The two types of data are collected independently at the same time or with a short time lag. (If the research is designed with two relatively independent phases, one with qualitative questions and data collection and analysis techniques and the other with quantitative questions and data collection and analysis techniques, then Creswell, 2009, calls this an embedded design. The inferences made on the basis of the results of each strand are pulled together to form meta-inferences at the end of the study.)

Youngs and Piggot-Irvine (2012) provide a detailed example of a pragmatic mixed methods design in their evaluation of a preparation program for people who aspired to become school principals (hereafter called aspirants). Here is how they described their approach and methods: “Our pragmatic position was informed by the intent of our contract with the Ministry of Education. . . . We took a ‘what works’ approach” (p. 188). At the national level, they used three quantitative questionnaires with all participants; analyzed documents, including regional provider plans, curricula outlines, and communications with participants; and conducted semistructured interviews and focus groups with the regional providers and facilitators.

The inclusion of two regional case studies into the design meant that we created the conditions for multilevel triangulation to occur so that the national data generated through aspirant questionnaires and regional provider interviews could be compared and contrasted with specific examples from the two regions. This enabled us to report not just on the more generalized national findings of the evaluation research but also provide examples of regional practice and contextualized aspirant perceptions of their development and learning. (p. 191)

Table 10.1 Mixed Methods Design Options and Examples

p. 309

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 05/28/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

5/28/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252337/View?ou=122307 6/12

Pragmatic Sequential Mixed Methods Design In the pragmatic sequential mixed methods design, one type of data (e.g., quantitative) provides a basis for the collection of another type of data (e.g., qualitative) (Creswell, 2009; Teddlie & Tashakkori, 2009). It answers one type of question by collecting and analyzing two types of data. Inferences are based on the analysis of both types of data. A sequential design is one in which the conclusions that are made on the basis of the first strand lead to formulation of questions, data collection, and data analysis for the next strand. The final inferences are based on the results of both strands of the study. In some cases, the second strand/phase of the study is used to confirm or disconfirm the inferences of the first strand or to provide further explanation for unexpected findings in the first strand. This approach can be used to generate hypotheses to be explored in more depth or to develop surveys that use correct language for the population.

Berliner et al. (2008) provide an example of a sequential mixed methods design in their study of high school dropouts. The researchers had access to a longitudinal student-level database that contained information about the date that dropouts left school and if they reenrolled in the same district. They used data from a cohort of students who started high school in 2001/2002 and were expected to graduate in the 2005/2006 school year. They conducted quantitative statistical analyses on dates of withdrawal and return, graduation rates, student courses, and demographic characteristics, as well as students’ reasons for dropping out. The quantitative data revealed that dropping out is not necessarily a permanent decision. While only 45% of the students graduated within the expected 4 years, of those who did drop out, almost one third of the students reenrolled during the 5 years of the study, and 18% of those graduated from high school in 2005/2006. (Twenty percent of the students transferred to other school districts, and researchers did not have access to their school outcomes.) The authors describe their sequential mixed methods design as follows:

Drawing on the preliminary analysis of the district dataset, interviews were conducted to clarify, affirm, or challenge the study findings and to explore state and district policies and practices that affect reenrollment and students’ experiences dropping out and reenrolling in San Bernardino City Unified School District high schools. In fall 2007, interview data were collected from 20 district contacts during a weeklong, in-person site visit. (p. ii)

Christ (2007, p. 228) provides another example of sequential design that began with a longitudinal quantitative survey of the supports needed by college students with disabilities. The researcher compared the results of a national survey that was conducted in 1999 and 2001 to see if changes occurred over this 2-year period. The results were used to raise additional questions, frame criteria for purposeful selection of postsecondary institutions, and guide development of interview questions at the selected sites. Christ then conducted semistructured interviews at three postsecondary sites that were considered by experts in the field to be exemplary. The resulting data were analyzed to determine

p. 310

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 05/28/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

5/28/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252337/View?ou=122307 7/12

critical themes related to effectively supporting college students with disabilities. These data were then used as a basis for designing a third stage of the study: an intensive case study of a single institution that was facing a severe budget cut.

Transformative Parallel Mixed Methods Design The transformative parallel mixed methods design is based on the use of both quantitative and qualitative methods in a study that has a goal of social change at levels ranging from the personal to the political (Mertens, 2009; Mertens & Wilson, 2012). The design gives primacy to the value-based and action-oriented dimensions of different inquiry traditions (Greene, 2007).

The Shared Reading Project (SRP) provides an example of a transformative mixed methods design that used parallel qualitative and quantitative data collection and analysis strategies (Delk & Weidekamp, 2001; Mertens, Delk, & Weidekamp, 2003). The SRP was designed to provide hearing parents and caregivers with visually based strategies to read books to their deaf and hard-of-hearing children from birth through age 8. The SRP was designed to meet the needs of five groups of traditionally underserved deaf and hard-of-hearing students, including members of diverse ethnocultural groups, those who have secondary disabilities, people who live in rural areas, people who come from homes in which a spoken language other than English is used, and/or people who are lower- achieving academically. Tutors, most of whom are deaf, visited the families in their homes to teach them signing and reading strategies and answer their questions about the process of teaching their children to read.

The SRP collected both quantitative and qualitative data to address multiple evaluation questions. The quantitative measures included closed-ended surveys and logs that the families kept that indicated such things as the demographic characteristics and the number of times families used the books that the tutors brought them between visits. The qualitative data were collected through in-person, on-site interviews with the families, tutors, and site coordinators. Foreign language interpreters were used in homes where the spoken language was other than English. Deaf interviewers conducted the interviews with the deaf tutors; these interviews were videotaped and then transcribed. Participants were asked to describe such things as their role in the project, their emotional experiences during the project, and successes and challenges.

The quantitative and qualitative data were analyzed with a specific attempt to identify the impact of the project on the traditionally underserved groups. Quantitative data indicated that 104 of the 116 children belonged to at least one of the traditionally underserved groups. The quantitative and qualitative data were disaggregated in the analysis in order to provide a picture of the unique successes and challenges faced by members of these groups in teaching their deaf and hard-of-hearing children to read. Because the focus was on traditionally underserved groups, the results were shared with the people in their communities to determine lessons learned and options for additional action. Because of the demonstrable results in terms of improved literacy, several of the sites have moved to institutionalize the SRP, incorporating it into their regular school program. In some sites, the skill of the deaf tutors has been increasingly recognized and they have been given responsibilities to work in the classrooms as well as at the students’ homes. One site has made a decision to expand the program to include Hmong and Latino families of children enrolled in the school.

Transformative Sequential Mixed Methods Design The transformative sequential mixed methods approach shares the transformative goal described in the previous section, as well as the methodological sequencing of the pragmatic sequential option (Mertens, 2009).

Meadow-Orlans et al. (2003) provide an example of a transformative sequential mixed methods design in their study of parents’ experiences with their young deaf and hard-of-hearing children. The study proceeded in three phases: first, a quantitative national survey; then, individual parent interviews; and third, focus group interviews. The initial phase of the study was designed not only to provide a broad picture of parents’ early experiences but also to investigate differences in experiences based on such characteristics as race or ethnicity, parent hearing status, presence of additional disabilities beyond hearing loss, level of parent education, and the socioeconomic status of the family. The National Parent

p. 311

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 05/28/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

5/28/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252337/View?ou=122307 8/12

g , p , y Project survey was designed to obtain information directly from families by eliciting descriptions of their experiences, evaluations of effectiveness of services, and recommendations for improvement. Parents provided important insights throughout the project from the design of the survey and interview protocol to the analyses and implications of the findings.

The quantitative data analysis indicated great diversity in the characteristics of the families and their responses to the survey questions. For example, one third of the participating parents had children with a disability in addition to a hearing loss, and many of these families encountered difficulties obtaining needed services. Overall, families generally expressed satisfaction with the services their children received; however, some families, particularly non-White families, were less satisfied with their services.

These findings led to the qualitative portion of the study, which used in-depth telephone interviews and a purposive sampling strategy to investigate in more depth the experiences of families with children with co-occurring disabilities, parents who were deaf or hard of hearing, parents of children with cochlear implants, and families of color. In the final phase, three focus groups of parents were conducted in large urban areas with the intention of expanding the number of Hispanic and African American families represented in the sample. In replying to very open-ended questions, participants provided insights into their experiences, feelings, and concerns during the time of suspicion and confirmation of their child’s hearing loss; discussed the people or services that were most helpful and needs that were not addressed by early intervention services; described their communication decision- making process; and offered the advice they had for professionals and for other parents.

The data were analyzed and reported within the larger national context, while still preserving the subgroup analyses of traditionally underrepresented groups. Special attention was given to interrogating bodies of knowledge that have become institutionalized as established concepts and practices that were determined by groups that traditionally have power in our society. Groups in power (generally professionals) largely determine for those with less power (parents) what are accepted practices. These “cultural facts” are accepted without challenge until the voices of individuals affected have the opportunity to articulate their experiences and express their perspectives. Thus, the parents’ comments served as a basis to give insights not only to other parents who might be starting down the road of life with their deaf or hard-of-hearing child but also to professionals who serve this population.

Transformative Cyclical Mixed Methods Design Bledsoe (2008) provides an example of a transformative cyclical mixed methods design in her evaluation of an obesity reduction program in a school system in a disadvantaged neighborhood in New Jersey (see Figure 10.1). She entered the context after program developers had decided that the reason students were obese was because of poor self-concept. Rather than accepting that version of reality, she asked them if they had checked with the students to understand their version of reality— that is, their understanding of the meaning of obesity and reasons for reducing their weight. As the program developers had not yet done that, she collected qualitative data to do just that. What did obesity mean to the students? Did they overeat because they had poor self-concepts? Her qualitative data revealed that the students did not have poor self-concepts. Indeed, they felt that being “big” was not such a bad thing. However, they did express concern about the health consequences associated with obesity (e.g., heart disease, diabetes) and they wanted to know how to avoid the likelihood that they would be afflicted with those conditions. They wanted to eat healthy foods and exercise more, but they saw barriers such as lack of access to fresh fruits and vegetables and a lack of safe places to get exercise. Thus, a more culturally responsive program was developed that included food fairs to demonstrate how to cook favorite foods in a more healthy way and dance contests in which the number of steps taken were counted by a pedometer. Students also used geographic information systems (GIS) to map food outlets, kept food diaries, and had physical measurements of weight and blood tests for diabetes. In this way, the cyclical use of data, both qualitative and quantitative, allowed for the design, implementation, and successful outcomes in this project.

Figure 10.1 Obesity Reduction Program Transformative Cyclical Mixed Methods Design

p. 313

p. 312 00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 05/28/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

5/28/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252337/View?ou=122307 9/12

SOURCE: Bledsoe (2008).

EXTENDING YOUR THINKING

Mixed Methods Design Options Explain the main design options for mixed methods research. Give an example of how each of these could be applied in educational and psychological research.

Select a research problem in education or psychology. Find a research study related to the problem you selected that uses a mixed methods approach. Use the questions for critically analyzing mixed methods research to identify its strengths and weaknesses. What improvements would you suggest for that study?

Questions for Critically Analyzing Mixed Methods Research

Teddlie and Tashakkori (2009) suggest that there is a unique character to mixed methods that transcends a simplistic combination of methods. Therefore, they suggest the term inference quality to refer to issues that would be termed internal validity in quantitative terms or trustworthiness in qualitative terms. Greene (2007) also discusses this concept as warranting the quality of inferences in mixed methods research. Based on the dialectical stance with regard to mixed methods research, she offers the following considerations for quality:

• Focuses on the available data support for the inferences, using data of multiple and diverse kinds; • Could include criteria or stances from different methodological traditions; • Considers warrants for inquiry inferences a matter of persuasive argument, in addition to a matter

of fulfilling established criteria; and • Attends to the nature and extent of the better understanding that is reached with this mixed

methods design. (p. 169)

Table 10 2 provides a framework for considering criteria to determine quality in mixed methods

p. 314

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 05/28/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

5/28/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252337/View?ou=122307 10/12

Table 10.2 provides a framework for considering criteria to determine quality in mixed methods research.

One approach to critically analyzing a mixed methods study would be to use the criteria that are presented in previous chapters on specific quantitative methods for the quantitative portion of the study and those in the chapters on qualitative methods for the qualitative portion. This is not a bad strategy and could be a useful way to assess the quality of the individual parts of the study. This allows researchers to begin their critique of the quality of a mixed methods design by looking at the integrity of the methods as they are derived from the assumptions of each paradigm. So using the questions in the appropriate methodology chapter to critically analyze the methods for particular approaches in a project is appropriate.

However, tensions can arise because conflicting demands can be present when more than one paradigm is operationalized. For example, if the researcher is working inductively within the qualitative part of the study, the sample is small and purposively selected, and therefore would not meet the criteria for a quantitative sample that needs to be larger and have been randomly selected. If a quantitative component is being used, then a separate, randomized sample might be added for that portion. Small samples do raise issues of potential bias. Researchers should also be aware of the issues related to the use of a qualitative or quantitative data collection method to supplement the findings from the main study. For example, a quantitative study might include one or two focus groups to add a qualitative dimension to the main study. The researcher in such circumstances needs to acknowledge the limitations of the qualitative data, in that they cannot stand alone. They are intelligible and interpretable only when they are linked to the data from the main quantitative part of the study.

Table 10.2 Rigor: Criteria for Judging Quality in Quantitative, Qualitative, and Mixed Methods Research

When looking at the conclusions of a mixed methods design, it is possible that the results from both methods will agree with each other and thus confirm the conclusions reached. It is also possible that they will not agree with each other. Then, the researcher must explore plausible explanations for the disagreement. It may be due to the difference in approach or because of changes in the context over time that could influence performance on the dependent measures.

The selection of different methods in a study may be necessary to accommodate differences based on disability, culture, language, reading or writing levels, gender, class, and race or ethnicity for specific

p. 315

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 05/28/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

5/28/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252337/View?ou=122307 11/12

y, , g g , g g , g , , y p subgroups in a population (Mertens, 2009). Practically, individuals with different types of disabilities may be better able to provide accurate information about themselves if a qualitative or quantitative method is used. For example, in the Delk and Weidekamp (2001) study, telephone surveys were not appropriate for deaf individuals without adequate English writing skills. Other human dimensions, such as race or ethnicity, also need to be considered in the design of a study. As Stanfield (2011) noted, an atmosphere of distrust has developed between researchers and many members of racial or ethnic minority communities because of historical events such as the Tuskegee experiments, in which Black men with syphilis were left untreated so that researchers could study the progress of the disease. Mixed methods may be necessary to provide an opportunity to build a sense of trust between the researcher and the community. A researcher could define the problem to be studied through a qualitative phase of interacting with members of the community, using observation and interviewing. If trust can be developed, then it might be appropriate to introduce a quantitative phase to the project.

These issues and concerns then give rise to the following questions that can be used to critically analyze mixed methods research studies:

1. What are the multiple purposes and questions that justify the use of a mixed methods design? 2. Has the researcher matched the purposes and questions to appropriate methods? 3. To what extent has the researcher adhered to the criteria that define quality for the quantitative

portion of the study? 4. To what extent has the researcher adhered to the criteria that define quality for the qualitative

portion of the study? 5. How has the researcher addressed the tension between potentially conflicting demands of

paradigms in the design and implementation of the study? 6. Has the researcher appropriately acknowledged the limitations associated with data that were

collected to supplement the main data collection of the study? 7. How has the researcher integrated the results from the mixed methods? If necessary, how has the

researcher explained conflicting findings that resulted from different methods? 8. What evidence is there that the researcher developed the design to be responsive to the practical

and cultural needs of specific subgroups on the basis of such dimensions as disability, culture, language, reading levels, gender, class, and race or ethnicity?

EXTENDING YOUR THINKING

Rigor in Mixed Methods Research Design a study using a mixed methods design. Explain how your design respects the integrity of the qualitative and quantitative methods. Explain how you would address the criteria implicit in the questions for critically analyzing mixed methods research.

Using one or more of the studies from the quantitative or qualitative chapters, rethink the study using mixed methods. How would the study change? What would you gain? What would you lose?

Summary of Chapter 10: Mixed Methods Research

In some ways, mixed methods is not a new way of doing research, as many researchers have engaged in the collection of both quantitative and qualitative data. However, the research community now reflects an increased interest in mixed methods and how to explore more systematically the advantages and disadvantages of their use. Both pragmatic and transformative paradigms are used as a basis for hi l i d i h l f i d h d di h k h i

p. 316

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods 05/28/2019 - RS000000000000000 Integrating D

iversity W ith Q

uantit

5/28/2019 Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

https://ncuone.ncu.edu/d2l/le/content/122307/viewContent/1252337/View?ou=122307 12/12

this exploration and to examine those elements of mixed methods studies that make them unique compared to studies that use a single method. Designs for mixed methods include consideration of the temporal relation between the use of each type of method (quantitative and qualitative) and the philosophical belief systems that underlie the research decisions. Criteria for critically analyzing the strengths and weaknesses of mixed methods research include the criteria for the individual approaches (e.g., case studies or surveys), as well as consideration of the rationale for and implementation of the mixed methods themselves.

Notes

1. Multiple methods studies are considered those that use more than one method, but the methods chosen are singularly either quantitative or qualitative. Teddlie and Tashakkori (2009) label this monomethod “multi-strand research.”

2. In Teddlie and Tashakkori (2010), several typologies of mixed methods designs are presented in the various chapters. They note that the formal field of mixed methods research is in its adolescence, and therefore it is to be expected that various typologies would be present in the literature. Main points of definition appear to be the temporal relation between the quantitative and qualitative data collections (parallel vs. sequential), the philosophical paradigm underlying the work (e.g., pragmatic or transformative), the priority given to either qualitative or quantitative approaches (Creswell, 2009 described this as dominant, subdominant relations), and the purpose of the research (confirmatory or exploratory). If design options were presented for each possible combination of these dimensions, then there would be a large number of possible designs for mixed methods. For the sake of parsimony, I elected to include five major categories for the major types of mixed methods designs.

p. 317

00000001583532 - Research and Evaluation in Education and Psychology: tative, Q

ualitative, and M ixed M

ethods