U10A1-68 - Qualitative Research Plan - ***TUTOR FOLLOW ALL INSTRUCTIONS AS OUTLINED TO COMPLETE THIS WORK. READ ATTACHMENTS.

PSYCHIATRIC SERVICES � ps.psychiatryonline.org � March 2011 Vol. 62 No. 3 225555

In the past decade, mental healthservices researchers have increas-ingly used qualitative methods in combination with quantitative meth- ods (1,2). This use of mixed methods has been partly driven by theoretical models that encourage assessment of consumer perspectives and of contex-

tual influences on disparities in the delivery of mental health services and the dissemination and implementa- tion of evidence-based practices (3,4). These models call for research designs that use quantitative and qualitative data collection and analy- sis for a better understanding of a re-

search problem than might be possi- ble with use of either methodological approach alone (5,6). Numerous ty- pologies and guidelines for the use of mixed-methods designs exist in the fields of nursing (7,8), evaluation (9,10), public health (11,12), primary care (13), education (14), and the so- cial and behavioral sciences (5,15).

As Robins and colleagues (1) have observed, however, there has been lit- tle guidance in mental health services research on how to blend quantitative and qualitative methods to build upon the strengths of their respective epistemologies. Such guidance has been limited by the lack of consensus on the criteria that might be used to evaluate the quality of such research (5). From a policy perspective, the impact of the efforts of the National Institute of Mental Health (NIMH) (3,4) and other institutes of the Na- tional Institutes of Health (NIH) and funding agencies in encouraging the use of mixed methods in mental health services research also remains poorly understood.

To address these issues, we exam- ined the application of mixed-meth- ods designs in a sample of mental health services research studies pub- lished in peer-reviewed journals and in NIMH-funded research projects over five years. Our aim was to deter- mine how and why such methods were being used and whether there are any consistent patterns that might indicate a consensus among re- searchers as to how such methods can and should be used. This aim is viewed as an initial step toward the development of standards for effec-

Mixed-Methods Designs in Mental Health Services Research: A Review LLaawwrreennccee AA.. PPaalliinnkkaass,, PPhh..DD.. SSaarraahh MM.. HHoorrwwiittzz,, PPhh..DD.. PPaattrriicciiaa CChhaammbbeerrllaaiinn,, PPhh..DD.. MMiicchhaaeell SS.. HHuurrllbbuurrtt,, PPhh..DD.. JJoohhnn LLaannddssvveerrkk,, PPhh..DD..

The authors are affliliated with the Child and Adolescent Services Research Center, Rady Children’s Hospital, San Diego. Dr. Palinkas and Dr. Hurlburt are also with the School of Social Work, University of Southern California, 669 W. 34th St., Los Angeles, CA 90089-0411 (e-mail: [email protected]). Dr. Horwitz is also with the Department of Pe- diatrics, Stanford University, Palo Alto, California. Dr. Chamberlain is also with the Cen- ter for Research to Practice, Eugene, Oregon.

Objective: Despite increased calls for use of mixed-methods designs in mental health services research, how and why such methods are being used and whether there are any consistent patterns that might indicate a consensus about how such methods can and should be used are un- clear. Methods: Use of mixed methods was examined in 50 peer-re- viewed journal articles found by searching PubMed Central and 60 Na- tional Institutes of Health (NIH)–funded projects found by searching the CRISP database over five years (2005–2009). Studies were coded for aims and the rationale, structure, function, and process for using mixed methods. Results: A notable increase was observed in articles published and grants funded over the study period. However, most did not provide an explicit rationale for using mixed methods, and 74% gave priority to use of quantitative methods. Mixed methods were used to accomplish five distinct types of study aims (assess needs for services, examine ex- isting services, develop new or adapt existing services, evaluate servic- es in randomized controlled trials, and examine service implementa- tion), with three categories of rationale, seven structural arrangements based on timing and weighting of methods, five functions of mixed methods, and three ways of linking quantitative and qualitative data. Each study aim was associated with a specific pattern of use of mixed methods, and four common patterns were identified. Conclusions: These studies offer guidance for continued progress in integrating qual- itative and quantitative methods in mental health services research con- sistent with efforts by NIH and other funding agencies to promote their use. (Psychiatric Services 62:255–263, 2011)

tive uses of mixed methods in mental health services research and articula- tion of criteria for evaluating the qual- ity and impact of this research.

Methods We conducted a literature review of mental health services research publi- cations over a five-year period (Janu- ary 2005 to September 2009), using the PubMed Central database and the following search terms: mental health services, mixed methods, and qualitative methods. Data were taken from the full text of each research ar- ticle. Articles identified as potential candidates for inclusion had to report empirical research and meet one of the following selection criteria: a study specifically identified as a mixed-methods study in the title or abstract or through keywords; a qual- itative study conducted as part of a larger project, including a random- ized controlled trial, that also includ- ed use of quantitative methods; or a study that “quantitized” qualitative data (16) or “qualitized” quantitative data (17). On the basis of criteria used by McKibbon and Gadd (18) and Cresswell and Plano Clark (5), the analysis had to be fairly substantial; for example, a simple descriptive analysis of baseline demographic characteristics of participants was not sufficient to be included as a mixed- methods study. Further, qualitative studies that were not clearly linked to quantitative studies or methods were excluded from our review.

Using the same criteria and search terms, we also reviewed the NIH CRISP database (Computer Re- trieval of Information on Scientific Projects) of projects funded over the same five-year period. Projects were limited to R series (independent re- search awards), F series (predisserta- tion research awards), and K series (career development awards) grants. Data were taken from only the proj- ect descriptions provided by the ap- plicant and contained in the database.

Using typologies employed in other fields of inquiry (5–7,9), we next as- sessed the use of mixed methods in each study to determine the study aims, rationale, structure, function, and process. Study aims referred to the objectives of the overall project

that included both quantitative and qualitative studies or methods. The rationale for using mixed methods in- cluded conceptual reasons, such as exploration and confirmation (5), breadth and depth of understanding (19), and inductive and deductive theoretical drive (20). Pragmatic rea- sons for using mixed methods, such as addressing the weaknesses of one method by use of the other, and suit- ability to address research questions were also examined. Assessment of the structure of the research design was based on Morse’s (7) taxonomy, which gives emphasis to timing (for example, using methods in sequence [represented by a → symbol] versus using them simultaneously [repre- sented by a + symbol]) and to weight- ing (for example, primary method [represented by capital letters such as QUAN] versus secondary method [represented in lowercase letters such as qual]).

Assessment of the function of mixed methods was based on whether the two methods were being used to answer the same question or to an- swer related questions and whether they were used to achieve conver- gence, complementarity, expansion, development, or sampling (9). Final- ly, the process or strategies for com- bining qualitative and quantitative methods were assessed with the ty- pology proposed by Cresswell and Plano Clark (5): merging or converg- ing the two methods by actually bringing them together in the analysis or interpretation phase, connecting the two methods by having one build upon the results obtained by the oth- er, or embedding one data set within the other so that one type of method provides a supportive role for the oth- er method.

Results Our search identified 50 articles and 67 NIH-funded research projects published or funded between 2005 and 2009 that met our criteria for analysis. Seven of the NIH projects were excluded from further review because of missing data on the use of mixed methods. Three of the publi- cations were based on one of the NIH-funded projects, and two other publications were based on one fund-

ed project each. Any redundant aims or strategies for combining qualita- tive and quantitative methods identi- fied in linked publications and proj- ects were counted only once in our analysis.

A list of the 26 journals in which the articles were published and the jour- nals’ impact factors (IFs) is presented in Table 1. One-fifth of the articles were published in Psychiatric Ser- vices. The 2008 IFs of the journals for which information was available ranged from .74 (Psychiatric Rehabil- itation Journal) to 4.84 (Journal of the American Academy of Child and Adolescent Psychiatry). Twenty-one of the 50 articles (42%) had an IF of 2.0 or greater. Of the funded grants, three were predissertation research grants (F31s), 28 were career devel- opment awards (K01, K08, K23, K24, and K99), and 29 were independent research awards (R01, R03, R18, R21, R24, and R34).

Table 2 presents the year of publi- cation for the 50 articles and the start date of the 60 funded projects. Six- teen of the projects funded during this period had a start date before 2005. The smaller numbers of publi- cations and of projects in 2009 reflect the shorter period of observation (nine months) for that year. There was an exponential increase in the number of publications between 2005 and 2008, and the number of grants from 2005 to 2009 was more than twice that of the previous five- year period (2000–2004).

Table 3 summarizes for comparison the use of mixed-methods designs on the basis of study aims. Our analyses revealed the use of mixed methods to accomplish five distinct types of study aims and three categories of ration- ale. We further identified seven struc- tural arrangements, five uses or func- tions of mixed methods, and three ways of linking quantitative and qual- itative data together. Some papers and projects included more than one objective, structure, or function; hence the raw numbers may occa- sionally sum to more than the total number of studies examined. Twelve of the 50 articles presented qualita- tive data only but were part of larger studies that included the use of quan- titative measures. Further, we identi-

PSYCHIATRIC SERVICES � ps.psychiatryonline.org � March 2011 Vol. 62 No. 3225566

fied four commonly used designs, with each design associated with a specific aim or set of aims (Figure 1).

Study aims As shown in Table 3, the largest number of publications and projects (41 of 110, 37%) used mixed meth- ods in observational or quasi-experi- mental studies of existing services. Almost one-quarter (24%) used mixed methods to study the imple- mentation and dissemination of evi- dence-based practices. Mixed meth- ods were also used to develop evi- dence-based practices, treatment, and interventions (17%); to conduct randomized controlled trials of in- terventions (14%); or to assess the needs of populations for mental health services (14%). Six studies had more than one aim (for example, two studies conducted a needs as- sessment before developing new in- terventions, and two studies exam- ined implementation of an evidence- based practice within the context of a randomized controlled trial exam- ining the practice’s effectiveness.

Mixed-methods rationale Forty-one of the 60 project abstracts (68%) and 25 of the 50 published ar- ticles (50%) did not provide an explic- it rationale for the use of mixed meth- ods; consequently, the rationale was inferred from statements found in project objectives. Of the 25 pub- lished articles that did provide an ex- plicit rationale, only 11 provided one or more citations to justify use of mixed methods. The most common reason (93% of all articles and proj- ects) for using mixed methods was based on the specific objectives of the study (for example, qualitative meth- ods were needed for exploration or depth of understanding or quantita- tive methods were needed to test hy- potheses). In other instances, use of mixed methods was dictated by the nature of the data; studies that in- cluded a focus on variables related to values and beliefs, the process of service delivery, or the context in which services are delivered relied on qualitative methods to describe and examine these phenomena. In 9% of articles and projects, investigators specifically indicated that both meth-

ods were used so that the strengths of one method could offset the weak- nesses of the other (Table 3).

Mixed-methods structure The majority (58%) of the publica- tions and projects used the methods in sequence, with qualitative methods more often preceding quantitative

methods. Quantitative methods were the primary or dominant method in 74% of the publications and projects reviewed, and in 16 studies, qualita- tive and quantitative methods were given equal weight. In seven of the published studies, qualitative analy- ses were conducted on one or two open-ended questions attached to a

PSYCHIATRIC SERVICES � ps.psychiatryonline.org � March 2011 Vol. 62 No. 3 225577

TTaabbllee 11

Journals in which the 50 articles reviewed were published, with number published and 2008 impact factor

Number Impact Journal of articles factor

Psychiatric Services 10 2.48 Administration and Policy in Mental Health and

Mental Health Services 5 1.78 BMC Health Services Research 4 1.68 Implementation Science 3 2.87a Community Mental Health Journal 3 1.51 Family Practice 3 1.63 American Journal of Geriatric Psychiatry 2 4.02 Child: Care, Health and Development 2 1.15 American Journal of Community Psychology 1 2.39 American Journal of Public Health 1 4.24 Annals of Family Medicine 1 3.54 BMC Psychiatry 1 1.69a Comprehensive Psychiatry 1 2.05 Family Relations 1 1.32 International Journal of Child Health and Development 1 nab International Journal of Psychiatry in Medicine 1 .88 Journal of Child and Adolescent Psychiatric Nursing 1 nab Journal of Clinical Nursing 1 1.38 Journal of Cultural Diversity 1 nab Journal of Mental Health Policy and Economics 1 1.81 Journal of Psychiatric and Mental Health Nursing 1 1.08 Journal of the American Academy of Child and

Adolescent Psychiatry 1 4.84 Psychiatric Rehabilitation Journal 1 .74 Scandinavian Journal of Occupational Therapy 1 nab Social Psychiatry and Psychiatric Epidemiology 1 1.96 Tropical Medicine and International Health 1 2.31

a The unofficial impact score was provided by the journal because it is not rated by ISI. b na, not available

TTaabbllee 22

Year of publication or of project initiation of articles and projects reviewed

Articles Projects Total (N=50) (N=60) (N=110)

Year N % N % N %

2000–2004 0 0 16 28 16 14 2005 3 6 12 18 15 14 2006 5 10 7 12 12 11 2007 9 18 14 23 23 21 2008 23 46 7 12 30 27 2009a 10 20 4 7 14 13

a January through September 2009 only

survey, and 17 of the 50 published studies (34%) provided no references justifying their procedures for quali- tative data collection or analysis. Only one published study (21) provided a figure that illustrated the timing and weighting of qualitative and quantita- tive data collection and analysis, and none used terms like QUAN and qual to describe this structure.

In studies that aimed to assess needs for mental health services, ex- amine existing services, or develop new services or adapt existing servic- es to new populations, sequential de- signs were used two to four times more frequently than simultaneous designs. The latter type of design was more commonly used in randomized controlled trials and in implementa- tion studies.

Mixed-methods functions Our review of the publications and projects revealed five distinct func-

tions of mixing methods (Table 3). The first function was convergence, in which qualitative and quantitative methods were used sequentially or si- multaneously to answer the same question, either through triangulation (that is, the simultaneous use of one type of data to validate or confirm conclusions reached from analysis of the other type of data) or transforma- tion (that is, the sequential quantifi- cation of qualitative data or use of qualitative techniques to transform quantitative data). For instance, Gris- wold and colleagues (22) triangulated quantitative trends in functional and health outcomes of psychiatric emer- gency department patients with qual- itative findings of perceived benefits of care management and the value of integrated medical and mental health care to determine whether both types of data provided support for the ef- fectiveness of a care management in- tervention (QUAN + QUAL). Using

the technique of concept mapping (23), Aarons and colleagues (24) col- lected qualitative data on factors like- ly to have an impact on implementa- tion of evidence-based practices in public-sector mental health settings. These data were then entered in a software program that uses multidi- mensional scaling and hierarchical cluster analysis to generate a visual display of statement clusters (QUAL → quan).

A second function of integrating quantitative and qualitative methods was complementarity, in which each method was used to answer related questions for the purpose of evalua- tion or elaboration. This function was evident in a majority (65%) of the published studies and projects exam- ined. In evaluative designs, quantita- tive data were used to evaluate out- comes, whereas qualitative data were used to evaluate process. For in- stance, Bearsley-Smith and col-

PSYCHIATRIC SERVICES � ps.psychiatryonline.org � March 2011 Vol. 62 No. 3225588

TTaabbllee 33

Characteristics of 50 published studies and 60 funded projects that used mixed-methods designs, by study aimsa

Evaluate new Examine practices in Examine

Conduct needs existing Develop randomized implementation assessment services new practices controlled trials of new practices Total (N=15) (N=41) (N=19) (N=15) (N=26) (N=110)

Characteristic N % N % N % N % N % N %

Rationale for method Dictated by data 0 — 2 5 3 16 4 27 4 15 13 12 Dictated by objectives 15 100 37 90 18 95 11 73 21 81 102 93 Complement strengths 1 7 5 12 0 — 1 7 3 11 10 9

Structureb Sequential

QUAL → quan 2 13 3 7 4 21 1 7 1 4 11 10 qual → QUAN 5 33 9 22 13 68 0 — 2 8 29 26 quan → QUAL 1 7 1 2 0 — 0 — 0 — 2 2 QUAN → qual 2 13 14 34 0 — 5 33 1 4 22 20

Simultaneous qual + QUAN 4 27 6 15 0 — 6 40 14 54 30 27 QUAL + quan 1 7 4 10 0 — 0 — 1 4 6 5 QUAL + QUAN 1 7 5 12 2 10 3 20 5 19 16 15

Function Convergence 3 20 7 17 1 5 3 20 7 27 21 19 Complementarity 9 60 29 71 6 32 10 67 17 65 71 65 Expansion 0 — 8 20 0 — 6 40 12 46 26 24 Development 4 27 9 22 16 84 1 7 7 27 37 34 Sampling 1 7 2 5 0 — 0 — 5 19 8 7

Process Merge the data 6 40 19 46 2 10 3 20 11 42 41 37 Connect the data 8 53 17 41 16 84 2 13 9 35 52 47 Embed (nest) the data 1 7 9 22 3 16 11 73 15 58 39 35

a Six published articles or projects had two aims (total number of aims=116). b QUAL or qual, qualitative; QUAN or quan, quantitative. Upper- or lowercase indicates whether the method was primary or dominant versus second-

ary or subservient, respectively.

leagues (25) described the use of quantitative methods to investigate the impact on clinical care of imple- menting interpersonal psychotherapy for adolescents within a rural mental health service and the use of qualita- tive methods to record the process and challenges (that is, feasibility, ac- ceptability, and sustainability) associ- ated with implementation and evalu- ation (QUAN + qual). In elaborative designs, qualitative methods were used to provide depth of understand- ing and quantitative methods were used to provide breadth of under- standing. For instance, in a longitudi- nal study of mental health con- sumer–run organizations, Janzen and colleagues (26) used a quantitative tracking log for breadth of informa- tion about system-level activities and outcomes and key informant inter- views and focus groups for greater in- sight into the impacts of these activi- ties (QUAL + quan).

A third function of integrating qualitative and quantitative methods was expansion, in which one method was used in sequence to answer ques- tions raised by the other method. This function was evident in 24% of the published studies and projects exam- ined. In each instance, qualitative data were used to explain findings from the analyses of quantitative data. Brunette and colleagues (27) inter- viewed key informants and conduct- ed ethnographic observations of im- plementation efforts to understand why some agencies adhered to estab- lished principles for integrated dual disorders treatment and others did not (QUAN + qual).

A fourth function of mixed meth- ods was development, in which qual- itative methods were used sequen- tially to identify form and content of items to be used in a quantitative study (for example, survey ques- tions), to create a conceptual frame- work for generating hypotheses to be tested by using quantitative meth- ods, or to develop new interventions or adapt existing interventions to new populations (qual → QUAN). This function was used in 34% of the published studies and projects. Blasinsky and colleagues (28) used qualitative findings from site visits to develop quantitative rating scales to

construct predictors of outcomes and sustainability of a collaborative care intervention for older adults who had major depressive disorder or dysthymia. Green and colleagues (29) used qualitative data to generate a theoretical model of how relation- ships with clinics and clinicians’ ap- proach affect quality of life and re- covery from serious mental illness and then tested the model using questionnaire data and health-plan and interview-based data in a covari- ance structure model. Several of the research projects funded through the R34 mechanism (for example, MH074509-01, Kilbourne, principal investigator [PI]; MH078583-01, Druss, PI; and MH073087-01, Lew- is-Fernandez, PI) used qualitative data obtained from focus groups of consumers and providers to develop or adapt interventions for clients with specific conditions (for exam- ple, bipolar disorder, chronic med- ical conditions, and depressive disor- ders) (qual → QUAN).

The final function of mixed meth- ods was sampling, the sequential use of one method to identify a sample of participants for research that uses the other method. This technique was

used in only 7% of all studies. One form of sampling was the sequential use of quantitative data to identify po- tential participants for a qualitative study (quan → QUAL). For instance, Aarons and Palinkas (30) purposeful- ly sampled candidates for qualitative interviews who had the most positive or most negative views of an evi- dence-based practice on the basis of a Web-based quantitative survey. The other form of sampling used qualita- tive data to identify samples of partic- ipants for quantitative analysis (qual → QUAN). Woltmann and colleagues (31) created categories of low, medi- um, and high staff turnover on the ba- sis of staff perceptions of relevance of turnover obtained from qualitative in- terviews and then quantitatively ex- amined the relationship between these turnover categories and imple- mentation outcomes (qual + QUAN).

Only six of the published studies and none of the project abstracts ex- plicitly referred to the function of mixed methods by using terms such as triangulation (four published stud- ies) or complementarity (two pub- lished studies). As expected, the de- velopment function was used in a ma- jority (84%) of studies that aimed to

PSYCHIATRIC SERVICES � ps.psychiatryonline.org � March 2011 Vol. 62 No. 3 225599

FFiigguurree 11

Common mixed-methods designs used in mental health services researcha

a QUAL or qual, qualitative; QUAN or quan, quantitative. Upper- or lowercase indicates whether the method was primary or dominant versus secondary or subservient, respectively.

Needs assessment Service use

qual or QUAL Results QUAN

Needs assessment Service use

qual or QUAL QUAN or quan Results

Practice development

qual QUAN Results

Randomized controlled trials Implementation studies

QUAN Results

qual

Development

Complementarity

Complementarity

Complementarity Convergence Expansion

develop new practices or adapt exist- ing practices to new populations. A majority of observational and quasi- experimental studies of existing serv- ices (71%), randomized controlled trials (67%), implementation studies (65%), and needs assessment studies (60%) utilized mixed methods for the purposes of answering related ques- tions in complementary fashion. The use of one set of methods to explain the results of a study using another set of methods appears to have been limited to implementation studies (46%), randomized controlled trial evaluations (40%), and studies of ex- isting services (20%).

Process of mixing methods The final characteristic of mixed- methods designs that we examined was the process of mixing the quan- titative and qualitative methods. The largest percentage (47%) of ar- ticles and projects sought to connect the data sets (Table 3). This occurs when the analysis of one data set leads to (and thereby connects to) the need for the other data set, such as when quantitative results lead to the subsequent collection and analy- sis of qualitative data (that is, expan- sion) or when qualitative results are used to build to the subsequent col- lection and analysis of quantitative data, (for example, development) (5). For instance, Frueh and col- leagues (32) conducted focus groups to obtain information on the target population, their providers, and state-funded mental health systems that would enable the researchers to further adapt and improve a cogni- tive-behavioral therapy–based inter- vention for treatment of posttrau- matic stress disorder before imple- menting it (qual → QUAN). This type of mixing was found in almost all of the studies with aims to devel- op new practices or adapt existing practices to new populations; it was also more likely to be found in needs assessment and studies of existing services than in randomized con- trolled trials or implementation studies.

Over one-third (37%) of the studies merged the knowledge gained from the quantitative and qualitative data, either during the interpretation phase

when two sets of results that had been analyzed separately were brought to- gether or during the analysis phase when one type of data was trans- formed into the other type by consol- idating the data into new variables (5). This type of mixing was found in slightly less than half of the needs as- sessment, observational, and imple- mentation studies. For instance, Lucksted and colleagues (33) report- ed that a qualitative analysis of re- sponses to an open-ended postinter- vention question supported the quan- titative findings of the benefits of a relapse prevention and wellness pro- gram (QUAN + qual).

The embedding of small qualita- tive or qualitative-quantitative stud- ies within larger quantitative studies was observed in 35% of the pub- lished studies and projects reviewed and described as “nested designs” in six of the studies. This type of mixing was more commonly found in ran- domized controlled trials and in im- plementation studies, where qualita- tive studies of treatment or imple- mentation process or context were embedded within larger quantitative studies of treatment or implementa- tion outcome. For instance, to better understand the essential compo- nents of the patient-provider rela- tionship in a public health setting, Sajatovic and colleagues (34) con- ducted a qualitative investigation of patients’ attitudes toward a collabo- rative care model and how individu- als with bipolar disorder perceive treatment adherence within the con- text of a randomized controlled trial evaluating a collaborative practice model (QUAN + qual).

In 20% of published studies, more than one process was evident. For in- stance, Proctor and colleagues (35) connected the data by generating fre- quencies and rankings of qualitative data on perceptions of competing psychosocial problems collected from a community sample of 49 clients with a history of depression. These data were then merged with quantita- tive measures of depression status ob- tained through administration of the Patient Health Questionnaire–9 to explore the relationship of depression severity to problem categories and ranks.

Discussion The results of our analysis indicate that there has been substantial progress in using mixed-methods de- signs in mental health services re- search in response to efforts by NIMH (2,3) and other funding agen- cies to promote their use. Evidence for this progress is found in the in- creasing number of research projects that use mixed methods. The number of projects with mixed-methods de- signs funded over the five-year study period was more than twice the num- ber that began in the previous five- year period (2000–2004). Further- more, a majority (52%) of these fund- ed projects were predissertation or career development awards used by junior and midlevel investigators to acquire expertise in mixed-methods research.

We also observed a notable in- crease in the number of studies based on mixed-methods designs published each year during this five-year period. The number of published mental health services research studies with mixed-methods designs increased by 67% between 2005 and 2006, by 80% between 2006 and 2007, and by 155% between 2007 and 2008. Further- more, 21 of the 50 published studies (42%) that we reviewed appeared in journals with 2008 IFs of 2.0 or high- er, including ten articles published in Psychiatric Services; four articles ap- peared in a journal with an IF of 4.0 or higher. In contrast, McKibbon and Gadd (18) reported that only 11 of 37 (30%) mixed-methods studies of health services appeared in a journal with an IF of 2.0 or higher in the year 2000.

Despite this progress, however, our review also suggests that there is room for improvement in use of mixed-methods designs. Most studies did not make explicit or provide sup- port for the reasons for choosing a mixed-methods design; rather, we were forced to infer the rationale based on statements explaining what the methods were used for. Re- searchers may have felt that such ex- plicit statements were as unnecessary as statements explaining the rationale for using certain quantitative meth- ods, such as analysis of variance or survival analysis. However, the ab-

PSYCHIATRIC SERVICES � ps.psychiatryonline.org � March 2011 Vol. 62 No. 3226600

sence of an explicit rationale may also reflect a lack of understanding or ap- preciation of mixed-methods designs or a decision to use them without necessarily integrating or “mixing” them (5,6).

Most studies failed to provide ex- plicit descriptions of the design struc- ture or function that used terminolo- gy found in the mixed-methods liter- ature; use of such terminology is con- sistent with the general standards for high-quality mixed-methods research recommended by Cresswell and Plano Clark (5). Further, three- fourths of the 50 published studies re- viewed assigned priority to the use of quantitative methods, seven of the studies performed qualitative analy- ses of one or two open-ended ques- tions attached to a survey, and 17 of the studies provided no references justifying their procedures for quali- tative data collection or analysis. This may reflect an underappreciation of qualitative methods, as Robins and colleagues (1) have argued, or it may reflect a greater need for quantitative methods at the present time.

Although it was beyond the scope of this review to determine whether each study used mixed methods in ef- fective ways, we note that each study was subjected to rigorous peer review before being published or funded, and each was judged by this process to make a valuable contribution to the field of mental health services re- search. These studies also provide ev- idence of meaningful and sensible variations in mixed-methods ap- proaches to achieving various kinds of study aims and offer some guidance for integrating quantitative and quali- tative methods in mental health serv- ices research. For instance, the choice of a mixed-methods design ap- pears to be dictated by the nature of the questions being asked by mental health services researchers. Qualita- tive methods were used to explore a phenomenon when there was little or no previous research or to examine that phenomenon in depth, whereas quantitative methods were used to confirm hypotheses or examine the generalizability of the phenomenon and its associated predictors.

A majority of studies aiming to de- velop new practices or adapt existing

practices to new populations had the same structure (beginning with a small qualitative study before devel- oping or adapting the practice that was to be evaluated by using quantita- tive methods, which was found in 84% of the studies and projects) and the same process (connecting the findings of one set of methods with those of another set, which was found in 90% of the studies and projects). These studies reflect a growing awareness of the need to incorporate the preferences and perspectives of both service consumers and providers to ensure that new practices will be acceptable as well as feasible (32, 36–39).

Studies of existing services also tended to be sequential in structure, with qualitative methods used to elaborate or explain the findings of quantitative studies. In the majority of these studies, the process of mixing methods involved either merging two sets of data to achieve convergence or connecting them to achieve expan- sion (5). A similar pattern was ob- served in studies that aimed to ex- plore issues related to the needs for mental health services or provide more depth to our understanding of those needs. Such studies also ap- peared more likely to transform or “quantitize” qualitative data (24,35).

Randomized controlled trials and studies of implementation also shared similar patterns in use of mixed meth- ods, including simultaneous use of both methods to achieve complemen- tarity by embedding a qualitative or qualitative-quantitative study within a larger quantitative study, such as a randomized controlled trial. In the randomized controlled trials, qualita- tive methods were usually used to evaluate the process of providing the practice or intervention, whereas quantitative methods were used to evaluate the outcomes (25,40). In im- plementation research studies, quali- tative methods were used to explore or provide depth to understanding barriers and facilitators of interven- tion implementation, whereas quanti- tative methods were used to confirm hypotheses and provide breadth to understanding by assessing the gen- eralizability of findings (41,42).

The choice of mixed-methods de-

signs was also dictated by how the in- dividual questions being addressed by each method were related to one an- other. Studies that used different types of data to answer the same question reflected the function of convergence in a simultaneous struc- ture, where data were merged for the purpose of triangulation, or a sequen- tial structure, where qualitative data were transformed into quantitative data. Studies that used different types of data to answer related questions reflected the function of complemen- tarity, in which quantitative methods were used to measure outcomes, de- scribe content (for example, fidelity of services used and the nature of the mental health problem), and provide breadth (generalizability) of under- standing, whereas qualitative meth- ods were used to evaluate the process of service delivery (43–45), describe context (for example, setting) (26,34, 46), describe consumer values or atti- tudes (35,42,47), and provide depth (meaning) of understanding (28,48) in a simultaneous structure and em- bedded data process. Expansion, de- velopment, and sampling were also used to provide answers to related questions that could not be answered by one method alone, usually in a se- quential structure in which data sets were merged or connected together (24,30,37).

Finally, the choice of design ap- pears to be based on the strengths of one method relative to the weakness- es of the other. For instance, expan- sion was used to explain findings based on quantitative data with quali- tative data because explanation was not possible with the quantitative methods alone (25,27,40). In conver- gence, both sets of methods were used to confirm or validate one an- other, especially in instances where limited samples precluded testing of hypotheses with sufficient statistical power (30,49) and where limitations to qualitative data collection raised concerns about objectivity and trans- ferability of results. In studies devel- oping new methods, conceptual mod- els, and interventions, qualitative methods also served to enhance quantitative analysis by laying the groundwork essential for more valid measurement and theory and more

PSYCHIATRIC SERVICES � ps.psychiatryonline.org � March 2011 Vol. 62 No. 3 226611

effective, usable, and sustainable in- terventions (37). Sampling also worked to enhance validity by using qualitative methods to enhance quan- titative methods by developing tar- geted comparisons or by using quan- titative methods to enhance qualita- tive methods by establishing criteria for purposeful sampling (36).

In summary, the choice of a mixed- methods design appears to be associ- ated with three considerations: the nature of the question being asked (inductive-exploratory or deductive- confirmatory), how the questions be- ing addressed by each method are related to one another, and the strengths of each method relative to the weaknesses of the other.

Caution should be exercised in in- terpreting these findings given limita- tions in our study design and analysis. Despite our efforts to be comprehen- sive in the search process and to se- lect studies and projects on the basis of criteria with face validity, we un- doubtedly excluded several articles or projects that used mixed methods. For example, we may have excluded mixed-methods projects listed in the CRISP database that did not specify use of qualitative or mixed methods in the abstracts. We may have also ex- cluded published articles with quali- tative data that were part of larger, primarily quantitative studies if the articles did not reference the larger studies, or we may have excluded ar- ticles not listed in PubMed Central. In the absence of explicit informa- tion, we were often forced to infer the structure, rationale, and function of the design based on statements con- tained in the available material. Simi- larly, the CRISP abstracts describe only what the investigators proposed to do with mixed methods and do not indicate what was actually done. Our use of existing typologies of structure, function, and process were intended to serve as a starting point in our analysis rather than an attempt to “pi- geon-hole” each study into a specific typology. Our assessment of the progress made in the application of mixed-methods designs in response to calls for their use by funding agen- cies did not include indicators of whether these efforts had produced more useful, incisive, or insightful

knowledge for the purpose of ad- dressing mental health services ques- tions and problems. Such an assess- ment would require comparisons with the products of studies based on monomethod designs, which was be- yond the scope of this study.

Finally, it should be noted that the typology of mixed-methods use does not represent a set of standards for using mixed methods per se but is an important first step toward the devel- opment of such standards. Typologies by themselves do not explain why a particular method should be used and how to use a method appropriately. However, as Teddlie and Tashakkori (6) observed, there are five reasons or benefits to developing such a typolo- gy: typologies help to provide the field with an organizational structure, they provide examples of research de- signs that are clearly distinct from ei- ther qualitative or quantitative re- search designs, they help to establish a common language for the field, they help researchers decide how to pro- ceed when designing their studies, and they are useful as a pedagogical tool. A consensus conference or workshop bringing together experts in mixed methods and mental health services research to evaluate the em- pirically generated typology found in current patterns of mixed-methods use would appear to be the next logi- cal step in developing a set of stan- dards. Such standards would also be required to adhere to the epistemo- logical foundations of each method when used separately (for example, whether appropriate considerations are made to ensure the generalizabil- ity of quantitative results or theoreti- cal saturation of qualitative data and whether each method is appropriate- ly matched to the inductive or deduc- tive theoretical drive of the study) and when combined (for example, whether the knowledge gained when using the two methods together is more insightful and of greater value than the knowledge gained when us- ing them separately).

Conclusions Despite the limitations described above, the findings suggest an in- creasing use of mixed-methods de- signs to address changing priorities in

mental health services research and a consensus as to how such methods should be applied. The lack of explic- it statements explaining the rationale for using mixed methods and the evi- dent priority assigned to quantitative methods suggest that there is room for improvement. However, these studies appear to utilize a common set of designs and provide guidance for using mixed methods, with vary- ing approaches based on the nature of the question being asked (exploratory or confirmatory), how questions be- ing addressed by each method are re- lated to one another, and the strengths of each method relative to the weaknesses of the other.

Acknowledgments and disclosures

This study was funded through NIMH grant P50-MH50313-07.

The authors report no competing interests.

References

1. Robins CS, Ware NC, dosReis S, et al: Di- alogues on mixed methods and mental health services research: anticipating chal- lenges, building solutions. Psychiatric Ser- vices 59:727–731, 2008

2. Hopper K: Qualitative and quantitative re- search: two cultures. Psychiatric Services 59:711, 2008

3. Hoagwood K, Jensen PS, Petti T, et al: Outcomes of mental health care for chil- dren and adolescents: I. a comprehensive conceptual model. Journal of the American Academy of Child and Adolescent Psychi- atry 35:1055–1063, 1996

4. Hohmann AA: A contextual model for clin- ical mental health effectiveness research. Mental Health Services Research 1:83–91, 1999

5. Cresswell JW, Plano Clark VL: Designing and Conducting Mixed Method Research. Thousand Oaks, Calif, Sage, 2007

6. Teddlie C, Tashakkori A: Major issues and controversies in the use of mixed methods in the social and behavioral sciences; in Handbook of Mixed Methods in the Social and Behavioral Sciences. Edited by Tashakkori A, Teddlie C. Thousand Oaks, Calif, Sage, 2003

7. Morse JM: Approaches to qualitative- quantitative methodological triangulation. Nursing Research 40:120–123, 1991

8. Sandelowski M: Combining qualitative and quantitative sampling, data collection, and analysis techniques in mixed method studies. Research in Nursing and Health 23:246–255, 2000

9. Greene JC, Caracelli VJ, Graham WF: To- ward a conceptual framework for mixed method evaluation designs. Educational Evaluation and Policy Analysis 11:255– 274, 1989

PSYCHIATRIC SERVICES � ps.psychiatryonline.org � March 2011 Vol. 62 No. 3226622

10. Patton MQ: Qualitative Evaluation and Research Methods, 3rd ed. Newbury Park, Calif, Sage, 2002

11. Morgan DL: Practical strategies for com- bining qualitative and quantitative meth- ods: applications to health research. Quali- tative Health Research 8:262–276, 1998

12. Steckler A, McLeroy KR, Goodman RM, et al: Toward integrating qualitative and quantitative methods: an introduction. Health Education Quarterly 19:1–8, 1992

13. Cresswell JW, Fetters MD, Ivankova NV: Designing a mixed methods study in pri- mary care. Annals of Family Medicine 2:7–12, 2004

14. Cresswell JW: Mixed method research: in- troduction and application; in Handbook of Educational Policy. Edited by Cizek GJ. San Diego, Academic Press, 1999

15. Waszak C, Sines MC: Mixed methods in psychological research; in Handbook of Mixed Methods in the Social and Behav- ioral Sciences. Edited by Tashakkori A, Teddlie C. Thousand Oaks, Calif, Sage 2003

16. Miles MB, Huberman AM: Qualitative Data Analysis: An Expanded Sourcebook, 2nd ed. Thousand Oaks, Calif, Sage, 1994

17. Tashakkori A, Teddlie C: Mixed Methodol- ogy: Combining the Qualitative and Quan- titative Approaches. Thousand Oaks, Calif, Sage, 1998

18. McKibbon KA, Gadd CS: A quantitative analysis of qualitative studies in clinical journals for the 2000 publishing year. BMC Medical Informatics and Decision Making 4:11, 2004. DOI:10.1186/1472- 6947-4-11

19. Miller WL, Crabtree BF. Clinical research; in Handbook of Qualitative Research, 2nd ed. Edited by Denzin NK, Lincoln YS. Thousand Oaks, Calif, Sage, 2001

20. Morse JM: Principles of mixed methods and multimethod research design; in Handbook of Mixed Methods in the Social and Behavioral Sciences. Edited by Tashakkori A, Teddlie C. Thousand Oaks, Calif, Sage, 2003

21. Montgomery P, Forchuk C, Duncan C, et al: Supported housing programs for per- sons with serious mental illness in rural northern communities: a mixed method evaluation. BMC Health Services Re- search 8:156, 2008. DOI:10.1186/1472- 6963-8-156

22. Griswold KS, Zayas LE, Pastore PA, et al: Primary care after psychiatric crisis: a qual- itative analysis. Annals of Family Medicine 6:38–43, 2008

23. Trochim WM: An introduction to concept mapping for planning and evaluation. Evaluation and Program Planning 12:1–16, 1989

24. Aarons GA, Wells R, Zagursky K, et al: Ad- vancing a conceptual model of evidence- based practice implementation in child welfare. American Journal of Public

Health 99:2087–2095, 2009

25. Bearsley-Smith C, Brown MO, Sellick K, et al: Does interpersonal psychotherapy improve clinical care for adolescents with depression attending a rural child and ado- lescent mental health service? Study pro- tocol for a cluster randomized feasibility trial. BMC Psychiatry 7:53, 2004. DOI:10.1186/1471-244X-7-53

26. Janzen R, Nelson G, Hausfather N, et al: Capturing system level activities and im- pacts of mental health consumer-run or- ganizations. American Journal of Commu- nity Health 39:287–299, 2007

27. Brunette MF, Asher D, Whitley R, et al: Implementation of integrated dual disor- ders treatment: a qualitative analysis of fa- cilitators and barriers. Psychiatric Services 59:989–995, 2008

28. Blasinsky M, Goldman HH, Unützer J: Project IMPACT: a report on barriers and facilitators to sustainability. Administration and Policy in Mental Health and Mental Health Services 33:718–729, 2006

29. Green CA, Polen MR, Janoff SL, et al: Un- derstanding how clinician-patient relation- ships and relational continuity of care af- fect recovery from serious mental illness: STARS Study results. Psychiatric Rehabili- tation Journal 32:9–22, 2008

30. Aarons GA, Palinkas LA: Implementation of evidence-based practice in child wel- fare: service provider perspectives. Admin- istration and Policy in Mental Health and Mental Health Services Research 34:411– 419, 2007

31. Woltmann EM, Whitley R, McHugo GJ, et al: The role of staff turnover in the imple- mentation of evidence-based practices in health care. Psychiatric Services 59:732– 737, 2008

32. Frueh BC, Cusack KJ, Grubaugh AL, et al: Clinicians’ perspectives on cognitive-be- havioral treatment for PTSD among per- sons with severe mental illness. Psychiatric Services 57:1027–1031, 2006

33. Lucksted A, McNulty K, Brayboy L, et al: Initial evaluation of the Peer-to-Peer Pro- gram. Psychiatric Services 60:250–253, 2009

34. Sajatovic M, Davies M, Bauer MS, et al: Attitudes regarding the collaborative prac- tice model and treatment adherence among individuals with bipolar disorder. Comprehensive Psychiatry 46:272–277, 2005

35. Proctor EK, Hascke L, Morrow-Howell N, et al: Perceptions about competing psy- chosocial problems and treatment priori- ties among older adults with depression. Psychiatric Services 59:670–675, 2008

36. Woolfson R, Woolfson L, Mooney L, et al: Young people’s views of mental health ed- ucation in secondary schools: a Scottish study. Child: Care, Health and Develop- ment 35:790–798, 2009

37. Proctor EK, Knudsen KL, Fedoravicius N,

et al: Implementation of evidence-based practice in community behavioral health: agency director perspectives. Administra- tion and Policy in Mental Health and Men- tal Health Services Research 34:479–488, 2007

38. Institute of Medicine: Crossing the Quali- ty Chasm: A New Health System for the 21st Century. Washington, DC, National Academies Press, 2000

39. Mental Health: A Report of the Surgeon General. Rockville, Md, US Department of Health and Human Services, US Public Health Service, 1999

40. Byng R, Norman I, Redfern S, et al: Ex- posing the key functions of a complex in- tervention for shared care in mental health: case study of a process evaluation. BMC Health Services Research 8:274, 2008. DOI:10.1186/1472-6963-8-274

41. Gioia D, Dziadosz G: Adoption of evi- dence-based practices in community men- tal health: a mixed method study of practi- tioner experience. Community Mental Health Journal 44:347–357, 2008

42. Ostler T, Haight W, Black J, et al: Case se- ries: mental health needs and perspectives of rural children raised by parents who abuse methamphetamine. Journal of the American Academy of Child and Adoles- cent Psychiatry 46:500–507, 2007

43. Kramer TF, Burns BJ: Implementing cog- nitive behavioral therapy in the real world: a case study of two mental health centers. Implementation Science 3:14, 2008. DOI: 10.1186/1748-3-14

44. Chew-Graham C, Slade M, Montana C, et al: A qualitative study of referral to com- munity mental health teams in the UK: ex- ploring the rhetoric and the reality. BMC Health Services Research 7:117, 2007. DOI:10.1186/1472-6963-7-117

45. Young AT, Green CA, Estroff SE: New en- deavors, risk taking, and personal growth in the recovery process: findings from the STARS study. Psychiatric Services 59: 1430–1436, 2008

46. Sharkey S, MacIver S, Cameron D, et al: An exploration of factors affecting the im- plementation of a randomized controlled trial of a transitional discharge model for people with serious mental illness. Journal of Psychiatric and Mental Health Nursing 12:51–56, 2005

47. Lee BR, Munson MR, Ware NR, et al: Ex- periences of and attitudes toward mental health services among older youths in fos- ter care. Psychiatric Services 57:487–492, 2006

48. Becker D, Whitley R, Bailey EL, et al: Long-term employment trajectories among participants with severe mental illness in supported employment. Psychiatric Ser- vices 58:922–928, 2007

49. Wetherell JL, Ayers CR, Sorrell JT, et al: Modular psychotherapy for anxiety in old- er primary care patients. American Journal of Geriatric Psychiatry 17:483–492, 2009

PSYCHIATRIC SERVICES � ps.psychiatryonline.org � March 2011 Vol. 62 No. 3 226633

Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.