Week 10 disc

A Scaled Mixed-Methods Approach to Contextualizing Health-Related Measures

Marie Chesaniuk1 and Gertraud Stadler2 1 Department of Psychology, University of Illinois at Chicago

2 Health & Human Sciences, Institute Gender in Medicine (GiM), Prevention Research Group, Charité-Universitätsmedizin Berlin

A major goal of stigma and health research is to elucidate the link between patients’ lived experiences of stigma and measurable aspects of health (e.g., quality of life and health outcomes). However, “lived experience” and “measurable health outcomes” are often at methodological odds, with the former typically characterized qualitatively and the latter typically characterized using quantified measures. One way to bridge this methodological gap is to collect and report quantitative measures as part of a qualitative study, pool both the quantitative and qualitative data, and scale the combined qualitative and quantitative data for meta-analysis. This article will discuss the advantages and disadvantages of this approach, applications in healthcare, and offer suggestions for putting this method into practice.

Keywords: research methods, multimethod, patient-centered care, contextual factors

A major goal of stigma and health research is to elucidate the link between patients’ lived experiences of stigma and measurable aspects of health (e.g., quality of life and health outcomes). Perceived stigma in particular is a subjective experience. However, “lived experience” and “measurable health outcomes” are often at methodological odds, with the former typically characterized qualitatively and the latter typically characterized using quantified measures. One way to bridge this methodological gap is to use a mixed-methods approach. This article will discuss the pros and cons of this approach, give some examples, and offer suggestions for its practical use.

Advantages of a Mixed-Methods Approach

We will discuss three advantages of a mixed-methods approach: (a) contextualizing stigma and health outcome measures, (b) improving patient representation, and (c) increased traffic across quantitative and qualitative literatures. Here, we use Link and Phelan (2001) definition of stigma as a dynamic four-step social process involving (a) labeling, (b) negative stereotyping, (c) sepa- ration of “us” from “them,” and (d) status loss and discrimination. However, to avoid perpetuating “us” versus “them,” in this article, we use Stangl and colleagues (Stangl et al., 2019) language of stigma experiences and stigma practices. In line with widely used inter- vention development frameworks (e.g., Eldredge et al., 2016; PRECEDE-PROCEDE, Green & Kreuter, 2005), we assume a logic model where stigma is a determinant of health, with health outcomes broadly defined, including physical health and quality of life.

Contextualizing Health Outcome Measures via Process Information

The first advantage of a mixed-methods approach is to better describe the context in which measures of health and stigma occur. Purely quantitative measures of health outcomes (e.g., quality-of-life measure scores) or stigma do not decipher how—or the process by which—someone arrived at that outcome or self-report score.

Franklin et al. (2019) showed the importance of contextualizing measures. They used a qualitative approach to explain why self- management interventions that typically use quantitative goal set- ting may lack effectiveness in improving patient health outcomes. They interviewed patient–professional dyads and identified three patient goal typologies to better characterize sources of tension between treatment goals set by professionals versus the goals patients had in mind for themselves. Patient goals were heavily influenced by patients’ socioeconomic status. Franklin and colleagues identified patients with three different goal typologies: Typology 1 patients view “goals as opportunities” and are “ideal” patient whose goals happen to align best with desired study outcomes. This ideal patient goal typology was overwhelmingly found among high-SES men. Patients in the two other typologies are stigmatized for not living up to this ideal and not achieving desired study outcomes. Franklin and col- leagues’ qualitative study is thus transformative as well as informative and could serve as a model for how interventionists might follow-up with qualitative interviews on quantitative outcomes in trials.

“Back translation” of quantitative scale scores in qualitative interviews is one way to better contextualize and understand these quantities. Many psychometric measure validation studies begin with qualitative research phases in which patients are interviewed (Creswell & Creswell, 2017). These interviews are used to generate items for the measure, which are then quantified to produce a scale score (and also qualified into themes via factor analysis). Scale scores on a new measure may be compared against scale scores of other measures to establish convergent and/or discriminant validity. What is missing from the validation process is relating the scale

Marie Chesaniuk https://orcid.org/0000-0003-0883-7596 Declarations of interest: None. Correspondence concerning this article should be addressed to

Marie Chesaniuk, Department of Psychology, University of Illinois at Chicago, 1007 W Harrison St, Chicago, IL 60607, United States. Email: mchesa2@uic.edu

27

Stigma and Health

© 2021 American Psychological Association ISSN: 2376-6972 https://doi.org/10.1037/sah0000284

T h is d o cu m en t is co p y ri g ht ed

b y th e A m er ic an

P sy ch o lo g ic al

A ss o ci at io n o r o n e o f it s al li ed

p u b li sh er s.

T h is ar ti cl e is in te nd ed

so le ly

fo r th e p er so n al

u se

o f th e in di v id u al

u se r an d is n o t to

b e d is se m in at ed

b ro ad ly .

2021, Vol. 6, No. 1, 27– 31

scores back to patient interviews and lived experience, that is, “back translation.” For example, do people with higher scores on experienced stigma report more frequent and/or more severe stigma experiences in interviews? Mustanski et al. (2011) took a similar approach to establishing validity using predictors of sexual risk-taking behaviors as a health outcome rather than scale scores. They conducted quantitative methods to estimate predictors of spe- cific sexual risk behaviors among young men who have sex with men and used qualitative methods to describe the meaning of these behaviors to participants. Data prompted interviews are another good example for a mixed-

methods approach that uses participants’ personal quantitative data gathered before the interview to stimulate discussion during a qualitative interview; participants view their own data, interpret it, and comment on it (Kwasnicka et al., 2015). These interviews offer one path to addressing the validity problem, but are not regularly employed as part of psychometric validation studies. Data prompted interviews can also be used in interventions. For example, an adherence intervention where patients repeatedly reviewed charts of their quantitative adherence scores over time together with a nurse who helped them identify adherence barriers and tailored plans was effective for improving health outcomes in HIV patients (de Bruin et al., 2010, 2017). This approach could be applied to stigma and health scale scores as well. If a mismatch emerges between patients’ subjective experiences of stigma or health and validated measures, this should be explored further and might inform efficacy and effectiveness studies as well as clinical practice. Chronic pain is an example of how contextualizing quantitative

health data minimized opportunities for stigma practices and ex- periences and improved health outcomes. Chronic pain is a case where relating patients’ subjective experiences to quantitative health measures could decrease stigma and improve health outcomes. Patients experiencing chronic pain often experience mismatches across various measures of pain and providers’ judgments of their pain (Hirsh et al., 2015). Often, these patients are stigmatized specifically due to the inconsistency among their expressed pain, provider misconceptions about the biological bases of pain, and provider expectations of what a reported level of pain “ought” to look like. Patients reporting 9 or 10 out of 10 levels of pain who are not actively grimacing or crying out in pain are often doubted by providers who see them as not demonstrating the level of pain reported. This has led to undertreatment of pain, especially among lower SES and other vulnerable, typically minority, patients (Motov & Khan, 2009). Research connecting self-report scores with pa- tients’ lived experiences contributed to providers reducing discount- ing patient lived experience of their pain and, ultimately, better pain management (Adam et al., 2017, 2018; Hirsh et al., 2019). In sum, a mixed-methods approach could help to develop a better understand- ing of quantitative measures and help to contextualize them and transform research and clinical practice, from informing the devel- opment and validation of measures, more accurate diagnoses and better treatments.

Improving Patient Representation

The second advantage of a mixed-methods approach is a stronger representation of patients in their own terms. Using direct patient quotes alongside quantitative measures of stigma and health out- comes more closely represent patient perspectives than descriptive

or interpretive statistics alone. Boardman et al. (2011) are an example of scaling up qualitative interview methods, blending in quantitative variables, and prioritizing representation of patient voices. They blended large-scale public opinion poll methods with qualitative interview methods to gain insight into how people with depression use stigma experiences to increase resilience. Quantitative studies should more commonly include patients’ own voices in articles aiming to mitigate adverse health conse- quences and stigma in order to take patients’ perspectives seriously and for patient voices to appear in stronger articles with higher impact and resonance with stakeholders (Clancy, 2011).

Scholarly literature about stigma poses a philosophical and social issue: It is meant to be a tool to mitigate stigma experienced by its own research participants, and thus should maximize the humanity and empowerment of those patients. The roles of the reader and subject pose a gap in power that could favor the reader. The reader is active, in a position to choose what content to consume, and how they interpret that content and manage bias. Similarly, the roles of writer and subject pose an even starker power gap that favors the writer. The writer/researcher is in a position to direct discourse with subjects, censor or choose what quotes for reports, and sometimes does so without further input from subjects once data collection has ceased. The writer’s voice, for a combination of practical and professional reasons, tends to get far more air time than patient voices. Minimizing power gaps may also minimize the impression that limited patient-generated content is an indication of a lack of respect for or devaluation of patient perspectives.

While, for practical reasons, much of this literature needs to stay primarily in the narrative of the researcher/writer, it is instructive to consider the impact on stigma if patients were given every oppor- tunity to be heard in their own words in scholarly literature and what articles would look like if they featured equal or more patient- generated content than writer/researcher generated content. On the journal level, Stigma and Health feature first-person essays doc- umenting stigma experiences. Some ideas for dealing with power imbalances could come from other professional practices. In some models of psychological supervision, supervisees are encouraged to self-direct, examine, and minimize the power gap between patient and provider within the treatment relationship and setting. How can we as researchers, writers, and providers minimize the power gap between our subjects/patients and ourselves? Asking, “What would a patient-led literature look like?” and taking methodological and stylistic steps toward that vision is itself a stand against stigma and toward empowering patients in a literature in which they typically have less power. Including representative direct quotes is a standard of qualitative research and one step toward featuring the patient’s own voice in literature and more strongly regarding patients as experts in the lived experience of their health conditions.

Arguably more subversive and impactful would be featuring direct patient quotes in the quantitative research literature, where research subjects are far more likely to be represented as numbers rather than individuals with rich histories and social contexts. Quantitative research could benefit from incorporating more aspects of qualitative research, particularly the use of direct quotes. While cherry-picking quotes that support one’s hypothesis or findings could be a pitfall, quantitative researchers could use qualitative validity and bias check measures to protect against this. For exam- ple, researchers can write memos to better understand and correct for their own bias, they can seek quotes that counter their hypotheses

CHESANIUK AND STADLER

T h is d o cu m en t is co p y ri g ht ed

b y th e A m er ic an

P sy ch o lo g ic al

A ss o ci at io n o r o n e o f it s al li ed

p u b li sh er s.

T h is ar ti cl e is in te nd ed

so le ly

fo r th e p er so n al

u se

o f th e in di v id u al

u se r an d is n o t to

b e d is se m in at ed

b ro ad ly .

28

and findings, and use quotes to explore outliers. We can also contextualize quotes within the full sample and range of result (e.g., both treatment responders and nonresponders). Furthermore, quantitative researchers can use patient quotes to represent ambiva- lence, such as Dyson et al.’s (2010) representation of ambivalence about disclosure among youth with sickle cell disease.

Increased Traffic Across Quantitative and Qualitative Literatures

The third advantage of a mixed-methods approach is increased traffic across quantitative and qualitative literatures in a field. Including quantitative measures in qualitative articles increases the odds that qualitative work appears in literature searches con- ducted by quantitative researchers who may not have otherwise sought out qualitative studies to consider or cite in their own work. Juxtaposing qualitative and quantitative methods would cumulatively weaken the dichotomous schema some researchers and providers maintain about these two methods (Committee on Quality of Health Care in America, Institute of Medicine, 2001; Cooper et al., 2012; Povee & Roberts, 2014). Qualitative research carries some stigma itself: Some researchers and providers see it as less scientifically rigorous and/or applicable than quantitative approaches (Maher & Dertadian, 2018). This view disadvantages the patients whom quali- tative researchers hope to help with their work and limits the impact of qualitative research on further research and clinical practice. Adding quantitative measures to qualitative research within a mixed-methods approach could reduce the stigma of qualitative research through drawing on the familiarity bias and could benefit patients by not disadvantaging them with researchers’ own stigma in addition to any stigma they may face in their own lives. Similarly, including qualitative methods in primarily quantitative

studies would increase the visibility and usefulness of this research for those more familiar with the qualitative literature on stigma and health. Increasing mixed-methods approaches, in general, may serve to minimize an “us versus them” aspect of our own evidence base. This could involve more open-ended questions in self-report mea- sures and/or incorporating a case study or interview follow-up to more deeply represent patients and participants experiencing vari- ous health outcomes or stigma practices and experiences. Increasing traffic across qualitative and quantitative literatures

poses potential benefits not only to patients themselves, but also to the more efficient advancement of the science of stigma and health. Conducting research on the same topics in parallel maximizes inefficiency in science. By increasing traffic between qualitative and quantitative approaches, both are seen and used more efficiently in advancing research on the same or similar topics (Spellman et al., 2001).

Disadvantages of a Mixed-Methods Approach

In the next section, we will discuss two disadvantages of a mixed- methods approach: (a) the challenges of mixing methods well and (b) challenges of fitting more methods and analyses into reports.

The Challenges of Mixing Methods Well

A mixed-methods approach could give the impression that purely qualitative research is insufficient in and of itself. That is not the

intention nor is this approach a suggestion that all qualitative research employ this method. Purely qualitative studies are essential to a full and comprehensive body of research. In fact, using quantitative measures could impinge upon carefully developed relationships with qualitative research participants. While all parti- cipants should be made aware of all study components, so that quantitative measures should not come as a surprise to participants, quantitative measures are a change in dynamic from more intimate interviewing methods.

To employ a mixed-methods approach well requires skills in both qualitative and quantitative methods. Both qualitative and quantita- tive researchers need to either collaborate with skilled experts or acquire mixed-methods skills. Quantitative researchers may be tempted to pool survey or health outcome data collected across qualitative studies without fully integrating the rich qualitative data into their own secondary analyses (whether qualitative researchers ought to make data available to other researchers for secondary analysis is another ongoing debate).

Researchers may have concerns about the generalizability of convenience or niche qualitative samples. Although these samples may require some caveats and disclosure of sampling methods, Mullinix et al. (2016) find that such samples are largely generalizable. Wong et al. (2011) are an example of taking on the question of whether a theory is generalizable to a specific population. With a population of Asian college students, they conducted a quantitative analysis that helps compare this population with others previously tested using Joiner’s model of interpersonal–psychological theory of suicidal behavior. They then conducted a qualitative analysis of open- ended question responses to identify possible limits and caveats for this population within Joiner’s model. In this study, stigma and racism were shown to be caveats for the health outcome of suicide ideation.

Qualitative researchers may need to additionally consider what quantitative measures to include to facilitate pooling smaller research samples into meta-analyses. Using database building and core outcome models may facilitate consensus, the meta-analysis of qualitative samples, and streamline mixed-methods design pro- cesses (Williamson et al., 2012, 2017).

Challenges of Fitting More Methods and Analyses Into Reports

While some journals have loosened word counts to address a number of methodological concerns, most still employ some limit on report length (see Bussing et al. (2012), e.g., report on treatment willingness using integrated multimethods, including variable con- struction). This poses a challenge when a study design involves multiple methods, analyses, results, and results summaries. Making use of online supplementary materials and appendices may offer space to include qualitative interview schedules, lists of measures, example items, coding schemes, and/or figures. Additionally, it is common for researchers to make this information available upon request. Another approach would be to write a separate methods article for a study first and then cite this in subsequent articles reporting on that study.

Putting It Into Practice

Putting a mixed-methods approach into practice will take more than suitable collaborators and a general familiarity with both mixed

A SCALED MIXED-METHODS APPROACH

T h is d o cu m en t is co p y ri g ht ed

b y th e A m er ic an

P sy ch o lo g ic al

A ss o ci at io n o r o n e o f it s al li ed

p u b li sh er s.

T h is ar ti cl e is in te nd ed

so le ly

fo r th e p er so n al

u se

o f th e in di v id u al

u se r an d is n o t to

b e d is se m in at ed

b ro ad ly .

29

research methods. We would recommend the following steps. Starting with a literature search including existing systematic re- views across both qualitative and quantitative work is useful and an important first step guiding all efforts within the research project. This initial literature review informs the study design, including the interview questions and the choice of psychometrically validated measures or other appropriate quantifiable health outcomes relevant to the study topic. Appropriate study designs will administer quantitative measures in a way that maintains qualitative research relationships while also minimizing reporting bias, reporting mea- sure statistics, interpreting, and integrating these into the larger qualitative analysis and findings. A unique area of application is in the social validation of

psychometrically validated measures. Psychometric validation of measures virtually never includes social validation, or how much patients scoring in different ranges actually have different lived experiences of this construct. This is especially relevant in stigma and health. Qualitative research is uniquely equipped to explore how intersectional identities lead to similar or different stigma self- reports and health outcomes. Qualitative research methods using smaller sample sizes than quantitative research studies may also be uniquely equipped to represent multiple identities within the same individual. Dedicating the same or more space to comprehensive representations of each participant allows for the juxtaposition of multiple identities. More space can be dedicated to showing how multiple identities work together within the individual and impact experiences of stigma and health. A major drawback of quantitative research methods’ ability to

represent patients with multiple identities is the high number of individuals one would need per (niche) group. For example, it is very difficult to recruit patients with specific multiple identities in the quantities needed to power statistical analysis (e.g., it is easier to get enough patients with diabetes to reach statistical power than it is to recruit sufficient patients with diabetes who also identify as black, female, and queer as each new descriptor narrows the population from which researchers can draw). Pooling the diverse and inter- sectional samples of qualitative studies could support both qualita- tive and quantitative research on multiple identities. It could also make it possible to include qualitative studies in meta-analyses. Meta-analyses typically pool smaller samples into large samples.

However, these studies do not typically provide a rich account of who is represented in their samples, using descriptive statistics instead. By including qualitative studies, meta-analyses could include direct patient quotes and provide rich examples of their samples and feature patient voices. Qualitative studies are system- atically excluded from meta-analyses and therefore systematically excluded from some of the most impactful research in behavioral health and medicine. By creating traffic and pulling qualitative studies into meta-analytic study pools, qualitative studies could be considered alongside quantitative studies in field-defining meta- analytic research. Meta-analyses using qualitative studies could synthesize across qualitative studies in the narrative review portion of the meta-analysis. Then, in the statistical analysis, they could control for the type of study and/or population. The inclusion of qualitative samples may offer explanations for statistical phenomena like outliers, unexpected findings, study attrition, and low interven- tion effectiveness (e.g., Franklin et al., 2019). Burden et al. (2016) are an example of how to methodologically

integrate both qualitative and quantitative findings into a

metasummary (i.e., a mixed-methods meta-analysis). They per- formed an extensive literature search across both qualitative and quantitative studies and, in addition to more typical meta-analytic methods, produced frequency effect sizes for themes identified by qualitative studies. There is great potential for quasi-statistical method development in meta-summary.

Conclusion

Using a mixed-method approach to better bridge the gap between qualitative and quantitative research could benefit patients by contextualizing quantitative health measures, allowing more oppor- tunities to feature patient voices in scholarly literature via direct quotes, and facilitate intersectional identity representation in both qualitative and quantitative research. This approach could also benefit the science of behavioral health by increasing traffic across methodologies, increasing exposure to different approaches to similar topics, and may result in a more efficient way to build an evidence base. Although not without potential pitfalls, the novel approach of incorporating quantitative measures into what would otherwise be a purely qualitative study directly confronts the process-related shortcomings of quantitative measure development and quantified health outcomes commonly used in behavioral health research. By reporting research with an emphasis on patient empow- erment and representation, the literature can address stigma and health at the level of the study itself, but also by modeling in its use of “air time” dedicated to rich representations of patients’ lived experiences what it looks like to close a gap in power that maintains the stigma we seek to alleviate.

References

Adam, R., Burton, C. D., Bond, C. M., De Bruin, M., & Murchie, P. (2017). Can patient-reported measurements of pain be used to improve cancer pain management? A systematic review and meta-analysis. BMJ Supportive & Palliative Care, 7(4), 373–382.

Adam, R., de Bruin, M., Burton, C. D., Bond, C. M., Clausen, M. G., & Murchie, P. (2018). What are the current challenges of managing cancer pain and could digital technologies help? BMJ Supportive & Palliative Care, 8(2), 204–212. https://doi.org/10.1136/bmjspcare-2016-001232

Boardman, F., Griffiths, F., Kokanovic, R., Potiriadis, M., Dowrick, C., & Gunn, J. (2011). Resilience as a response to the stigma of depression: A mixed methods analysis. Journal of Affective Disorders, 135(1–3), 267– 276. https://doi.org/10.1016/j.jad.2011.08.007

Burden, C., Bradley, S., Storey, C., Ellis, A., Heazell, A. E., Downe, S., Cacciatore, J., Siassakos, D. (2016). From grief, guilt pain and stigma to hope and pride—a systematic review and meta-analysis of mixed-method research of the psychosocial impact of stillbirth. BMC Pregnancy and Childbirth, 16(1), Article 9. https://doi.org/10.1186/s12884-016-0800-8

Bussing, R., Koro-Ljungberg, M., Noguchi, K., Mason, D., Mayerson, G., & Garvan, C. W. (2012). Willingness to use ADHD treatments: A mixed methods study of perceptions by adolescents, parents, health professionals and teachers. Social Science & Medicine, 74(1), 92–100. https://doi.org/10 .1016/j.socscimed.2011.10.009

Clancy, C. M. (2011). Patient engagement in health care. Health Services Research, 46(2), 389–393. https://doi.org/10.1111/j.1475-6773.2011 .01254.x

Committee on Quality of Health Care in America, Institute of Medicine. (2001). Crossing the quality chasm: a new health system for the 21st century. National Academy Press.

CHESANIUK AND STADLER

T h is d o cu m en t is co p y ri g ht ed

b y th e A m er ic an

P sy ch o lo g ic al

A ss o ci at io n o r o n e o f it s al li ed

p u b li sh er s.

T h is ar ti cl e is in te nd ed

so le ly

fo r th e p er so n al

u se

o f th e in di v id u al

u se r an d is n o t to

b e d is se m in at ed

b ro ad ly .

30

Cooper, R., Chenail, R. J., & Fleming, S. (2012). A grounded theory of inductive qualitative research education: Results of a meta-data-analysis. Qualitative Report, 17, 8.

Creswell, J. W., & Creswell, J. D. (2017). Research design: Qualitative, quantitative, and mixed methods approaches. Sage.

de Bruin, M., Hospers, H. J., van Breukelen, G. J. P., Kok, G., Koevoets, W. M., & Prins, J. M. (2010). Electronic monitoring-based counseling to enhance adherence among HIV-infected patients: A randomized controlled trial. Health Psychology, 29(4), 421–428. https://doi.org/10.1037/a0020335

de Bruin, M., Oberjé, E. J., Viechtbauer, W., Nobel, H. E., Hiligsmann, M., van Nieuwkoop, C., Veenstra, J., Pijnappel, F. J., Kroon, F. P., van Zonneveld, L., & Groeneveld, P. H. (2017). Effectiveness and cost- effectiveness of a nurse-delivered intervention to improve adherence to treatment for HIV: A pragmatic, multicentre, open-label, randomised clinical trial. The Lancet. Infectious Diseases, 17(6), 595–604. https:// doi.org/10.1016/S1473-3099(16)30534-5

Dyson, S. M., Atkin, K., Culley, L. A., Dyson, S. E., Evans, H., & Rowley, D. T. (2010). Disclosure and sickle cell disorder: A mixed methods study of the young person with sickle cell at school. Social Science & Medicine, 70(12), 2036–2044. https://doi.org/10.1016/j.socscimed.2010.03.010

Eldredge, L. K. B., Markham, C. M., Ruiter, R. A., Fernández, M. E., Kok, G., & Parcel, G. S. (2016). Planning health promotion programs: An intervention mapping approach. John Wiley.

Franklin, M., Lewis, S., Willis, K., Rogers, A., Venville, A., & Smith, L. (2019). Goals for living with a chronic condition: The relevance of temporalities, dispositions, and resources. Social Science & Medicine, 233, 13–20. https://doi.org/10.1016/j.socscimed.2019.05.031

Green, L. W., & Kreuter, M. W. (2005). Health promotion planning: An educational and ecological approach (4th ed.). McGraw-Hill.

Hirsh, A. T., Hollingshead, N. A., Ashburn-Nardo, L., & Kroenke, K. (2015). The interaction of patient race, provider bias, and clinical ambi- guity on pain management decisions. The Journal of Pain, 16(6), 558– 568. https://doi.org/10.1016/j.jpain.2015.03.003

Hirsh, A. T., Miller, M. M., Hollingshead, N. A., Anastas, T., Carnell, S. T., Lok, B. C., : : : Ashburn-Nardo, L. (2019). A randomized controlled trial testing a virtual perspective-taking intervention to reduce race and socio- economic status disparities in pain care. Pain, 160(10), 2229–2240. https://doi.org/10.1097/j.pain.0000000000001634

Kwasnicka, D., Dombrowski, S. U., White, M., & Sniehotta, F. F. (2015). Data-prompted interviews: Using individual ecological data to stimulate narratives and explore meanings. Health Psychology, 34(12), 1191–1194. https://doi.org/10.1037/hea0000234

Link, B. G., & Phelan, J. C. (2001). Conceptualizing stigma. Annual Review of Sociology, 27(1), 363–385.

Maher, L., & Dertadian, G. (2018). Qualitative research. Addiction, 113(1), 167–172. https://doi.org/10.1111/add.13931

Motov, S. M., & Khan, A. N. (2009). Problems and barriers of pain management in the emergency department: Are we ever going to get better? Journal of Pain Research, 2, 5.

Mullinix, K. J., Leeper, T. J., Druckman, J. N., & Freese, J. (2016) Journal of Experimental Political Science. Political Science (Wellington, N.Z.), 2, 109–138.

Mustanski, B., Lyons, T., & Garcia, S. C. (2011). Internet use and sexual health of young men who have sex with men: A mixed-methods study. Archives of Sexual Behavior, 40(2), 289–300. https://doi.org/10.1007/ s10508-009-9596-1

Povee, K., & Roberts, L. (2014). Qualitative research in psychology: Attitudes of psychology students and academic staff. Australian Journal of Psychology, 66(1), 28–37. https://doi.org/10.1111/ajpy.12031

Spellman, B. A., Holyoak, K. J., & Morrison, R. G. (2001). Analogical priming via semantic relations. Memory & Cognition, 29(3), 383–393. https://doi.org/10.3758/BF03196389

Stangl, A. L., Earnshaw, V. A., Logie, C. H., van Brakel, W., Simbayi, L. C., Barré, I., & Dovidio, J. F. (2019). The health stigma and discrimination framework: A global, crosscutting framework to inform research, inter- vention development, and policy on health-related stigmas. BMC medi- cine, 17(1), Article 31.

Williamson, P., Altman, D., Blazeby, J., Clarke, M., & Gargon, E. (2012). Driving up the quality and relevance of research through the use of agreed core outcomes. Journal of Health Services Research & Policy. https:// doi.org/10.1258/jhsrp.2011.011131

Williamson, P. R., Altman, D. G., Bagley, H., Barnes, K. L., Blazeby, J. M., Brookes, S. T., : : : Kirkham, J. J. (2017). The COMET handbook: Version 1.0. Trials, 18(3), 280. https://doi.org/10.1186/s13063-017- 1978-4

Wong, Y. J., Koo, K., Tran, K. K., Chiu, Y. C., & Mok, Y. (2011). Asian American college students’ suicide ideation: A mixed-methods study. Journal of Counseling Psychology, 58(2), 197–209. https://doi.org/10 .1037/a0023040

Received February 1, 2020 Revision received July 21, 2020

Accepted October 1, 2020 ▪

A SCALED MIXED-METHODS APPROACH

T h is d o cu m en t is co p y ri g ht ed

b y th e A m er ic an

P sy ch o lo g ic al

A ss o ci at io n o r o n e o f it s al li ed

p u b li sh er s.

T h is ar ti cl e is in te nd ed

so le ly

fo r th e p er so n al

u se

o f th e in di v id u al

u se r an d is n o t to

b e d is se m in at ed

b ro ad ly .

31