Healthcare Spending
RESEARCH ARTICLE Open Access
A new instrument to measure high value, cost-conscious care attitudes among healthcare stakeholders: development of the MHAQ Serge B. R. Mordang1* , Karen D. Könings1, Andrea N. Leep Hunderfund2, Aggie T. G. Paulus3, Frank W. J. M. Smeenk1,4 and Laurents P. S. Stassen1,5
Abstract
Background: Residents have to learn to provide high value, cost-conscious care (HVCCC) to counter the trend of excessive healthcare costs. Their learning is impacted by individuals from different stakeholder groups within the workplace environment. These individuals’ attitudes toward HVCCC may influence how and what residents learn. This study was carried out to develop an instrument to reliably measure HVCCC attitudes among residents, staff physicians, administrators, and patients. The instrument can be used to assess the residency-training environment.
Method: The Maastricht HVCCC Attitude Questionnaire (MHAQ) was developed in four phases. First, we conducted exploratory factor analyses using original data from a previously published survey. Next, we added nine items to strengthen subscales and tested the new questionnaire among the four stakeholder groups. We used exploratory factor analysis and Cronbach’s alphas to define subscales, after which the final version of the MHAQ was constructed. Finally, we used generalizability theory to determine the number of respondents (residents or staff physicians) needed to reliably measure a specialty attitude score.
Results: Initial factor analysis identified three subscales. Thereafter, 301 residents, 297 staff physicians, 53 administrators and 792 patients completed the new questionnaire between June 2017 and July 2018. The best fitting subscale composition was a three-factor model. Subscales were defined as high-value care, cost incorporation, and perceived drawbacks. Cronbach’s alphas were between 0.61 and 0.82 for all stakeholders on all subscales. Sufficient reliability for assessing national specialty attitude (G-coefficient > 0.6) could be achieved from 14 respondents.
Conclusions: The MHAQ reliably measures individual attitudes toward HVCCC in different stakeholders in health care contexts. It addresses key dimensions of HVCCC, providing content validity evidence. The MHAQ can be used to identify frontrunners of HVCCC, pinpoint aspects of residency training that need improvement, and benchmark and compare across specialties, hospitals and regions.
Keywords: High-value cost-conscious care, Attitudes, Instrument development, Learning environment, Post- graduate medical training
© The Author(s). 2020 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
* Correspondence: s.mordang@maastrichtuniversity.nl 1Department of Educational Development and Research, School of Health Professions Education, Maastricht University, P. O. Box 616, 6200 MD, Universiteitssingel 60, 6229, ER, Maastricht, the Netherlands Full list of author information is available at the end of the article
Mordang et al. BMC Health Services Research (2020) 20:156 https://doi.org/10.1186/s12913-020-4979-z
Background Providing high value, cost-conscious care (HVCCC) is critical to improve the value of health care and at the same time counter rising costs, eliminate wasted spending, and reduce overuse (provision of health- care services with no medical basis or for which harms equal or exceed benefit) [1–5]. Value in this context can be understood as quality divided by cost over time [6]. Cost-conscious refers to the aware- ness an individual has on the specific expenses and cost-effectiveness of an intervention, as well as negative consequences as a result of providing – or not providing - an intervention, like patient dissatis- faction [7, 8]. Providing HVCCC requires physicians to balance the potential benefits and harms of a test or treatment, while simultaneously considering costs and possible drawbacks [7]. Physician practice pat- terns influence the number and type of healthcare services patients receive [9]. The post-graduate training appears to be particularly formative in shaping residents’ current and future behaviors re- lated to high-value care, such as during exposure to faculty discussions on patient care [10]. Medical education thus has an obligation to ensure that stakeholders within the post-graduate learning en- vironment support the development of HVCCC practice patterns [11–17]. Learning environments are complex, involving per-
sonal, social, organizational, physical, and virtual compo- nents [18]. Multiple individuals from different stakeholder groups contribute to the creation of work- place environments, and the attitudes of these individ- uals may influence an organizations’ culture regarding how and what residents learn [19–23]. Attitudes are also important (albeit imperfect) predictors of individual be- havior [24], as evidenced by multiple studies showing as- sociations between physician attitudes and beliefs and their utilization of healthcare services [25–28]. Under- standing the attitudes of key stakeholders thus has the potential to offer valuable insights into the post-graduate training environment [29], but there is a scarcity of reli- able tools to measure individual attitudes on all dimen- sions of HVCCC. In post-graduate medical training, staff physicians, ad-
ministrators and patients shape residents’ recognition and understanding of HVCCC’s necessity [15, 17, 30– 32]. While different stakeholders can have different pref- erences regarding the provision of HVCCC, measuring all stakeholders’ attitudes can give insight in the resi- dent’s workplace environment regarding the different di- mensions of providing HVCCC. Prior studies have tried to measure the attitudes of particular stakeholder groups with respect to specific dimensions of HVCCC [8, 10, 23, 32–39]. However, a single reliable instrument to
measure the individual attitudes of all these stakeholder groups toward multiple dimensions of providing HVCCC has not yet been developed. Such an instrument could both assess attitudes at the individual level and compare attitudes between stakeholders on distinct di- mensions. It also enables comparisons among different units, organizations, and specialties on the dimensions of providing HVCCC. This study aims to a) develop an instrument, the
Maastricht HVCCC-Attitudes Questionnaire (MHAQ), to measure resident, staff physician, administrator and patient attitudes toward HVCCC and b) determine, using generalizability (G) theory [40], how many respon- dents are needed to reliably measure a specialty attitude score on a national level.
Method We reviewed the literature to identify existing instru- ments for assessing individual attitudes toward HVCCC. From these, we selected items from the questionnaire used by Leep Hunderfund et al. [36] in their study of medical student attitudes toward cost-conscious care. These items were based on previously published surveys of practicing physicians and focus groups interviews with physicians, who gave input and suggestions on the items, as well as on reviews of the literature on cost-conscious care with input from various field experts [8, 33–35], supporting its content validity [41]. For more details on the development of the items, see the study by Leep Hunderfund et al. [36]. However, the concept of HVCCC consists of three key dimensions. Next to cost- conscious care and potential drawbacks, containing both the direct cost-effectiveness and downstream conse- quences of including cost-effectiveness, also the provision of value needs to be addressed [7]. Further- more, because results were reported on an item level, underlying constructs needed to be explored in order to methodologically interpret and compare results of differ- ent stakeholders. We developed the MHAQ through a four-phase
process (Fig. 1): 1) Investigating subscales of cost-conscious care, using
items and original data from the survey conducted by Leep Hunderfund, et al. [36].
2) Adding items, which include the value dimension, to strengthen subscales, and adapting items for use by residents, staff physicians, administrators, and patients.
3) Testing items among four samples of these stakeholders and developing the final version of the MHAQ.
4) Assessing the number of respondents per specialty on a national level needed to reliably measure a specialty attitude score through generalizability analysis.
Mordang et al. BMC Health Services Research (2020) 20:156 Page 2 of 10
Phase 1: investigating subscales Questionnaire and data We used items from the aforementioned published sur- vey of U.S. medical students as the starting point for questionnaire development, as this survey derived their 21 items assessing individual attitudes toward cost- conscious care, on recently published surveys for prac- ticing physicians [36]. The authors used a four-point Likert scale (1 = strongly disagree to 4 = strongly agree).
Analysis Since we developed a new scale without having a priori hypotheses about the structure of the variables, we used exploratory factor analysis (principle component ana- lysis, PCA) to examine the structure of these 21 survey items and to define subscales. PCA maximizes explained variance of the items [42] and is considered suitable when examining new constructs [43, 44]. Varimax rota- tion was performed to maximize spread of all factors,
resulting in better interpretable factors [42]. We used a parallel analysis, the Kaiser Guttman criterion (eigen- values > 1) and inspection of the scree plot, to identify the optimal number of factors [45]. We tested internal- consistency reliability of constructs using Cronbach’s alpha [46].
Phase 2: preparing the MHAQ Additional items Based on the internal-consistency reliability of identified subscales (which were around 0.6) and to tailor the MHAQ to new stakeholders and a new context, we added nine items to the original questionnaire. Because the initial 21 items focused primarily on costs, new items focused on value (e.g., risks and benefits of treatment, consideration of patient values) given the importance of value in HVCCC. These items were based on items de- scribed in the context of validated surveys on high-value originating from experts in the field [10, 23, 39, 47].
Fig. 1 Overview of the four-phase process to develop the MHAQ
Mordang et al. BMC Health Services Research (2020) 20:156 Page 3 of 10
Different stakeholders We developed a parallel questionnaire for medical resi- dents, staff physicians and administrators. Items for pa- tients were identical in content, but formulated for a lay audience. Additionally, we added a fifth answering op- tion (‘I don’t know’) for patients, to prevent random an- swering when questions were not well understood. These items were pilot-tested with 56 patients in 4 cy- cles to refine formulations.
Different context For usage in a Dutch context, we translated all items into Dutch. A professional translator translated all items back into English to evaluate similarity between the ori- ginal source and translated items [48].
Phase 3: administering the MHAQ and developing the final version Data collection To recruit respondents, we approached hospital educa- tional committees from all academic training regions (n = 8) in the Netherlands. Willing members of the hos- pital educational committees recruited medical residents and staff physicians to participate in the study. Addition- ally, we approached residents and staff physicians through the periodic newsletter of the ‘Bewustzijnspro- ject’, a Dutch project promoting HVCCC on a national level. The last authors (F.S. and L.S.) approached admin- istrators (policy and/or financial) in several hospitals. We approached patients before and after patient con- sults, after gaining (ethical) approval by the relevant hos- pital and the physician in charge of the department, and via several patient platforms. We sent all invitations to complete the MHAQ between June 2017 and July 2018. Participants received an information letter, after which they signed an informed consent form before answering the questionnaire. Medical residents, staff physicians and administrators filled out the questionnaire online via Qualtrics, a survey software program. Patients also had the option to answer the questionnaire on hardcopy.
Analysis We analyzed data following the same procedure as in Phase 1. We analyzed data from all stakeholder groups separately, after which an optimal solution was deter- mined through a parallel analysis, as well as examination of each of the scree-plots and the Kaiser-Guttman criter- ion, followed by an inspection of the factor loadings. We calculated internal consistency reliability of constructs separately for all subscales and all stakeholders using Cronbach’s alpha. Since we developed new scales, a Cronbach’s alpha > 0.6 was considered acceptable [49].
Phase 4: generalizability analysis We conducted a generalizability analysis [50] to assess the number of respondents needed to reliably measure a shared attitude score toward HVCCC of residents and staff physicians by specialty on a national level. We used Levene’s homogeneity tests to determine equal variances between specialties of different hospitals. In terms of generalizability theory, we performed a single facet ana- lysis with attitude scores nested within specialties. We carried out a variance component analysis, using spe- cialty as random factor and attitude score as dependent factor. We estimated the variance associated with spe- cialties and the variance of attitude scores nested within specialties using the following formula:
G ¼ Vs
Vsþ Vp : s Np
in which Vs is the associated variance of specialties, Vp:s is the associated variance of a participants’ attitude score within specialties, and Np is the number of participants attitude scores. We used results from G-study variance components to estimate SEM and conduct D-studies to project reliability estimates for varying numbers of re- spondents. For feasibility, we accepted a G-coefficient greater than 0.6 [50]. All data were analyzed using IBM SPSS statistics for Windows, version 25.0 (Armonk, NY: IBM Corp.).
Results Phase 1 The dataset from the published study on cost-conscious care included responses from students at 10 medical schools geographically distributed across the U.S.. Nine of these schools granted permission to use de-identified data from their students for the purposes of this study (3195 responses of 5992 total students surveyed). No student identifiers were collected and we removed school identifiers prior to sharing. Results of PCA indi- cated a three subscale-model. All factors had eigenvalues above 1.5. The first subscale contained five items about the responsibility of physicians to provide/promote HVCCC (Table 1); the second subscale contained five items about the relationship of physicians and patients when implementing HVCCC; the final subscale con- tained four items about considering costs in clinical de- cision making. Cronbach’s alphas of the subscales were between 0.64 and 0.66. Seven items had factor loadings < .4, representing a low communality for these items, and were not included in these subscales. These items, however, were still included in phases 2 and 3.
Mordang et al. BMC Health Services Research (2020) 20:156 Page 4 of 10
Phase 2 Table 3 shows the nine new items we added in phase 2, indicated with an asterisk. After translation into Dutch language, content of the original source items and the translated items was identical. The resulting question- naires for all stakeholder groups contained 30 items, in- cluding 21 items from the original questionnaire and nine newly added items.
Phase 3 In total, 301 residents and 297 staff-physicians com- pleted the MHAQ. Residents and staff physicians worked in 31 different specialties and 32 hospitals, geo- graphically distributed across the Netherlands. Fifty- three administrators and 521 patients completed the
MHAQ. Administrators and patients came from five hospitals in the South of the Netherlands (Table 2).
Data analyses To develop a questionnaire that is applicable to mul- tiple stakeholders in postgraduate medical education and enables reliable comparisons between stake- holders, grouping of items per subscale has to be the same for all stakeholders. S.M. and K.K. determined a best-fitting subscale composition for all stakeholders, based on the inspection of factor structures for each of the stakeholders. When compromises were neces- sary, factor analyses of residents and staff-physicians were prioritized when creating optimal subscales for all stakeholders, since these groups are most central
Table 1 Original items per subscale
Survey item Cronbach’s alpha
Subscale 1 α = .65
Physician clinical practices (e.g., ordering, prescribing) are key drivers of high health care costs. Cost to society should be important in physician decisions to use or not to use an intervention. Cost-effectiveness data should be used to determine what treatments are offered to patients. Trying to contain costs is the responsibility of every physician. Managing health care resources for all patients is compatible with physicians’ obligation to serve individual patients.
Subscale 2 α = .64
Patients will be less satisfied with the care they receive from physicians who discuss costs when choosing tests and treatments. Doctors are too busy to worry about the costs of tests and procedures. It is easier to order a test than to explain to the patient why a particular test is unnecessary. Practicing cost-conscious care will undermine patients’ trust in physicians. Ordering fewer tests and procedures will increase physicians’ risk of medical malpractice litigation.
Subscale 3 α = .66
Physicians should take a more prominent role in limiting use of unnecessary tests. Physicians should be aware of the costs of the tests or treatments they recommend. Physicians should talk to patients about the costs of care when discussing treatment options. Physicians should change their clinical practices (eg, ordering, prescribing) if the cost of care they provide is higher than colleagues who care for similar patients.
Table 2 Demographics of each stakeholder group
Characteristics Residents Staff physicians Administrators Patients
N respondents 301 297 53 521
N female respondents (%) 191 (65) 151 (51) 27 (51) 241 (46)
Age in years, Mean 30.6 45.9 51.7 59
Medical specialty (%) 296 (98.3) 295 (99.3) - -
Non-Surgical 172 (57.1) 166 (55.9) - -
Surgical 89 (29.6) 70 (23.6) - -
Supportive 35 (11.6) 59 (19.9) - -
Type of administrator (%)
Department administrator - - 17 (32.1) -
Division administrator - - 13 (24.5) -
Hospital administrator - Board level - - 7 (13.2) -
Other Administrator - - 16 (30.2) -
Mordang et al. BMC Health Services Research (2020) 20:156 Page 5 of 10
in post-graduate medical training. The best-fitting subscale composition for all stakeholders was a three- factor model. All factors had eigenvalues above 1. Four of five items of subscale 1 in phase 1 again clus- tered on the same factor, together with three add- itional items from the original subscale 3, as well as two items that had a low factor loading in phase 1 and one new item. The four items of subscale 2 in phase 1 again loaded all on the same factor. Three new items also loaded on this factor. The remaining item from subscale 3 loaded on a third factor, which also included one item from subscale 1, two items with low factor loadings in phase 1, and four new items. Thus, eight of the nine items added in phase 2 strengthened the subscales. All items in phase 1 fo- cused on cost-conscious care, but in phase 3 some of these items loaded on high value care. This is due to the content of these items, which do contain a cost component, but are in essence statements on high value care. Because in phase 1 high value care was not evaluated, these items loaded in this phase on a different subscale. For the final subscale composition, we optimized Cronbach’s al- phas for each stakeholder group, considering all subscales had to fit every stakeholder.
Final MHAQ The aforementioned analyses resulted in 25 items distrib- uted among three subscales, each covering an important di- mension of HVCCC in clinical environments. We defined the labels of subscales in our team of experts, based on the main focus of the consisting items. Subscale 1, defined as high-value care, contained eight items about physicians’ provision of high value care (Cronbach’s alphas ranging from 0.61 for staff physicians to 0.77 for administrators). Subscale 2, defined as cost incorporation, contained 10 items about the integration of healthcare costs in physi- cians’ daily practice (Cronbach’s alphas ranging from 0.69 for staff physicians to 0.80 for patients). Subscale 3, defined as perceived drawbacks, contained seven items about per- ceived drawbacks of practicing HVCCC (Cronbach’s alphas ranging from 0.67 for residents to 0.82 for patients). Table 3 presents the final version of the MHAQ. (The survey instrument is available as supplementary file.)
Phase 4 Generalizability This reliability estimation was performed separately for medical residents and staff physicians and for each subscale. Levene’s homogeneity tests indicated equal
Table 3 An overview of the MHAQ, viewing all items per subscale. (R) Reversed items.
Survey item Cronbach’s alpha
Residents Staff-physicians Administrators Patients
(1) High-value care α = .65 α = .61 α = .77 α = .67
Physicians should take a more prominent role in limiting use of unnecessary tests. The cost of a test or medication is only important if the patient has to pay for it out of pocket. (R) Managing health care resources for all patients is compatible with physicians’ obligation to serve individual patients. Eliminating unnecessary tests and procedures will improve patient safety. Physicians should consider a patient’s doubts and values in their clinical decisions.a
Physicians should offer patients choices of care, taking advantages, disadvantages and costs into account.a
Physicians should limit waste of care in their own hospital/clinic.a
Physicians should have sufficient knowledge of the interplay between advantages/disadvantages and costs of common tests.a
(2) Cost incorporation α = .71 α = .69 α = .74 α = .80
Physicians should try not to think about the cost to the health care system when making treatment decisions. (R) Physicians should be aware of the costs of the tests or treatments they recommend. Physicians should talk to patients about the costs of care when discussing treatment options. Physicians should change their clinical practices (e.g., ordering, prescribing) if the costs of care they provide is higher than colleagues who care for similar patients. Physician clinical practices (e.g., ordering, prescribing) are key drivers of high health care costs. Costs to society should be important in physician decisions to use or not to use an intervention. It is unfair to ask physicians to be cost-conscious and still keep the welfare of their patients foremost in their minds. (R) Cost-effectiveness data should be used to determine what treatments are offered to patients. Trying to contain costs is the responsibility of every physician. Physicians should discuss cost efficiency of care with their patients.a
(3) Perceived drawbacks α = .67 α = .70 α = .79 α = .82
Patients will be less satisfied with the care they receive from physicians who discuss costs when choosing tests and treatments. Doctors are too busy to worry about the costs of tests and procedures. Practicing cost-conscious care will undermine patients’ trust in physicians. Ordering fewer tests and procedures will increase physicians’ risk of medical malpractice litigation. Ordering more tests reduces a physicians’ diagnostic uncertainty.a
Ordering fewer tests and procedures will lead to more complications.a
Patients find it unpleasant to talk about costs of tests or treatments.a
aNew items that were added in phase 2. The item “if a physicians’ medical practices have a direct influence on a physicians’ salary, it will obstruct a physicians’ cost-conscious care approach” did not cluster on any of the subscales
Mordang et al. BMC Health Services Research (2020) 20:156 Page 6 of 10
variances between specialties (e.g., cardiology, internal medicine) across different hospitals. Results from D- studies indicated the number of respondents needed to reliably measure (G-score ≥ 0.6) residents’ attitude score per specialty on a national level is 28 for the subscale high value care, 52 for the subscale cost in- corporation, and 15 for the subscale perceived draw- backs. For staff physicians, the number of respondents needed was respectively 14 for the subscale high value care, 21 for the subscale cost incorporation, and 32 for the subscale perceived drawbacks. Figures 2 and 3
display an overview of the G-score per subscale for residents and staff physicians.
Discussion This study describes the development of the MHAQ and provides reliability evidence supporting its use to measure attitudes toward HVCCC among important stakeholders in the post-graduate clinical learning environment. The MHAQ assesses three key dimensions of HVCCC and may be used to identify frontrunners who endorse and prioritize HVCCC, to pinpoint aspects of HVCCC
Fig. 2 D-study projecting MHAQ reliability of resident respondents. Note: value of 0.6 is considered reliable
Fig. 3 D-study projecting MHAQ reliability of staff physician respondents. Note: value of 0.6 is considered reliable
Mordang et al. BMC Health Services Research (2020) 20:156 Page 7 of 10
that need to be improved or changed to better sup- port HVCCC in the post-graduate learning environ- ment, and to facilitate comparisons among different stakeholder groups, specialties, regions, and potentially hospitals or departments. The MHAQ includes three subscales relating to provision of high-value care (8 items), integration of costs (10 items), and perceived drawbacks of HVCCC (7 items). These subscales en- compass all key dimensions of providing HVCCC in clinical practice [7], hence supporting the content val- idity of MHAQ scores. Scores on high-value care reflect the degree to
which individuals believe physicians should be respon- sible for limiting unnecessary testing, reducing waste, considering risks, benefits, and patient preferences when making diagnostic or therapeutic intervention decisions. High scores on this subscale can identify proponents of HVCCC who believe physicians should be frontrunners in the provision of high-value care. When key individuals within the clinical learning en- vironment advocate high-value care, corresponding role modelling can help to shape future physicians’ HVCCC practice patterns [17, 30, 51]. Scores on cost incorporation reflect individual beliefs
about the degree to which physicians should integrate costs in their daily clinical practice, for example when making treatment decisions or when discussing options with patients. Although physicians assume they contrib- ute minimally to healthcare costs [35], they actually dir- ect up to 87% of all healthcare spending [52]. Knowing physicians’ view on the incorporation of costs in their daily practice, together with patients’ view on the incorp- oration of costs, can be important starting points for transformation efforts to educate future physicians about providing HVCCC [14]. Scores on perceived drawbacks reflect individual be-
liefs about potential drawbacks of HVCCC, like patient dissatisfaction or risks of malpractice. Perceptions like these are known barriers to the implementation of HVCCC in practice [53] and drivers of unnecessary test- ing [54]. When individuals within the same organization have different perceptions of the drawbacks, incorpor- ation of HVCCC in daily clinical practices is unsustain- able. Pinpointing organizations as such could initiate aligned education programs for all stakeholders in that organization on the benefits of HVCCC, to create a common understanding and support of the delivery of HVCCC [17, 55]. Internal consistency reliability was sufficient for all
stakeholders on all subscales. The internal consistency re- liability for subscale scores was lower for residents and staff physicians than for patients and administrators. This could suggest that residents and physicians have more nu- anced views on the provision of high-value care,
integration of costs into clinical practice, and potential drawbacks of HVCCC. Alternatively, items formulated for a lay audience may be more evident in meaning and there- fore clearer to answer than items used in the question- naires for residents, staff physicians, and administrators. The patient version of the MHAQ thus has the potential to inform future improvement of subscale reliability for other stakeholders when developing the MHAQ further. The MHAQ can not only be used to measure attitudes
toward HVCCC at the individual level, but also to com- pare attitudes among larger groups, e.g. specialties, hos- pitals, regions. Our D-study results predict 14 to 52 respondents would be required to reliably assess HVCCC attitudes among resident or staff physicians, supporting the feasibility of group comparisons at the national, specialty level.
Strengths and limitations This study has certain strengths and limitations. First, the MHAQ is based on a previously published questionnaire informed by a literature review on HVCCC, which was further enhanced through the addition of items (also based on the literature) that emphasized value as an im- portant dimension in addition to cost and drawbacks. Fu- ture studies could provide additional content validity evidence for MHAQ scores by presenting items to subject matter experts, for example in a Delphi-study [56]. Sec- ond, while we are the first, to our knowledge, to simultan- eously survey resident, staff physician, administrator, and patient attitudes toward HVCCC, our study did not in- clude all potential stakeholders. Future studies could ex- tend our work by including other relevant groups, such as nurses and other allied health professionals, who contrib- ute to the clinical learning environment. Third, we used the same items in the U.S. and the Netherlands, which strengthens the broad usability of the MHAQ. However, healthcare delivery systems vary by country and MHAQ items may not be equally applicable in all settings. Fourth, while the final version of the MHAQ showed promising reliabilities, and D-studies support the feasibility of reliable assessments at the specialty level, there were too few re- sults from a single department within a single hospital to calculate a reliable G-score at the department level. Fur- ther studies are needed to assess the number of respon- dents needed for a reliable department-level attitude score, which may most closely approximate the clinical learning environment experience by residents.
Conclusion The MHAQ is a new instrument capable of reliably measuring attitudes toward HVCCC among individuals within multiple relevant stakeholder groups - residents, staff physicians, administrators, and patients - with sub- scales that address key dimensions of HVCCC. The
Mordang et al. BMC Health Services Research (2020) 20:156 Page 8 of 10
MHAQ can be used to identify frontrunners who en- dorse and prioritize HVCCC, to pinpoint aspects of HVCCC that need to improved or changed to better support HVCCC in the post-graduate learning environ- ment, and to facilitate comparisons among different stakeholder groups, specialties, regions, and potentially hospitals or departments.
Supplementary information Supplementary information accompanies this paper at https://doi.org/10. 1186/s12913-020-4979-z.
Additional file 1. The Maastricht HVCCC Attitude Questionnaire (MHAQ).
Abbreviations D-studies: Decision studies; G-coefficient: Generalizability coefficient; G- score: Generalizability score; G-studies: Generalizability studies; HVCCC: High- Value, Cost Conscious Care; MHAQ: Maastricht HVCCC Attitude Questionnaire; PCA: Principal Component Analysis; SEM: Standard Error of Measurement; U.S.: United States
Acknowledgements The authors would like to thank Angelique van Bijsterveld and Corry den Rooyen for their support and assistance with connecting to key individuals across the Netherlands, who were able to help with data collection. In addition, the authors are grateful for the help of all project leaders in different Dutch educational regions in administering the MHAQ to potential participants. This research study was presented as a research paper at the 2019 Annual AMEE (Association for Medical Education in Europe) Conference, Vienna, Austria, August 26, 2019 [57].
Authors’ contributions S.M. contributed to conception and design of the study, to the acquisition, analysis and interpretation of the data in this study, and drafting and revising of the paper. He approves submission and publication of the paper and agrees with being accountable for all aspects thereof. K.K. contributed to conception and design of the study, to the acquisition, analysis and interpretation of the data in this study, and substantial revising of different versions of the paper. She approves submission and publication of the paper and agrees with being accountable for all aspects thereof. A.L.H. contributed to the acquisition of the data in this study, and substantintial revising of different versions (of parts) of the paper. She approves submission and publication of the paper and agrees with being accountable for all aspects thereof. A.P. contributed to the analysis used in the paper and reviewing of different versions (of parts) of the paper. She approves submission and publication of the paper and agrees with being accountable for all aspects thereof. F.S. contributed to conception and design of the study, to the acquisition and interpretation of the data in this study, and substantial revising of different versions of the paper. He approves submission and publication of the paper and agrees with being accountable for all aspects thereof. L.S. contributed to conception and design of the study, to the acquisition and interpretation of the data in this study, and substantial revising of different versions of the paper. He approves submission and publication of the paper and agrees with being accountable for all aspects thereof. The author(s) read and approved the final manuscript.
Funding Not applicable.
Availability of data and materials The Dutch dataset collected during the current study is available from the corresponding author on reasonable request.
Ethics approval and consent to participate The Ethical Review Board (ERB) of the Netherlands Association for Medical Education (NVMO) approved this study (no. NERB814 and amendment no. NERB817) before launch.
Informed consent was asked from all participants in this study and all were given the opportunity to withdraw from participating in the study.
Consent for publication All participants consented to their data being used anonymously.
Competing interests The authors declare that they have no competing interests.
Author details 1Department of Educational Development and Research, School of Health Professions Education, Maastricht University, P. O. Box 616, 6200 MD, Universiteitssingel 60, 6229, ER, Maastricht, the Netherlands. 2Department of Neurology, Mayo Clinic, Rochester, MN, USA. 3Department of Health Services Research, Care and Public Health Research Institute, Maastricht University, Maastricht, the Netherlands. 4Department of Pulmonary Medicine, Catharina Hospital, Eindhoven, the Netherlands. 5Department of Surgery, Maastricht University Medical Center, Maastricht, the Netherlands.
Received: 4 November 2019 Accepted: 11 February 2020
References 1. American College of Physicians. Controlling health care costs while promoting
the best possible health outcomes. Philadelphia: American College of Physicians; 2009: Policy Monograph. (Available from American College of Physicians, 190 N. Independence Mall West, Philadelphia, PA 19106.).
2. Fischer ES, Welch GH. Avoiding the unintended consequences of growth in medical care: how might more be worse? JAMA. 1999;281(5):446–53.
3. Sociaal Cultureel Planbureau. Meebetalen aan de zorg. In: Het ministerie van Volksgezondheid WeS, editor. Den Haag: Textcetera; 2012.
4. College Geneeskundige Specialismen. Bewustzijnsproject: Bewust kosteneffectief kwaliteit van zorg leveren in geneeskundig-specialistische vervolgopleidingen. 2015.
5. Morgan DJ, Brownlee S, Leppin AL, Kressin N, Dhruva SS, Levin L, et al. Setting a research agenda for medical overuse. BMJ. 2015;351:h4534.
6. Smoldt RK, Cortese DA. Pay-for-performance or pay for value? Mayo Clin Proc. 2007;82:210–3.
7. Owens DK, Qaseem A, Chou R, Shekelle P. High-value, cost-conscious health care: concepts for clinicians to evaluate the benefits, harms, and costs of medical interventions. Ann Intern Med. 2011;154:174–80.
8. Goold SD, Hofer T, Zimmerman M, Hayward RA. Measuring physician attitudes toward cost, uncertainty, malpractice, and utilization review. J Gen Intern Med. 1994;9:544–9.
9. Sirovich BE, Gallagher PM, Wennberg DE, Fisher ES. Discretionary decision making by primary care physicians and the cost of U.S. health care. Health Aff. 2008;27:813–23.
10. Ryskina KL, Smith CD, Weissman A, Post J, Dine JC, Bollman K, et al. U.S. internal medicine residents’ knowledge and practice of high-value care: a national survey. Acad Med. 2015;90(10):1373–9.
11. Cooke M. Cost consciousness in patient care-what is medical education's responsibility? N Engl J Med. 2010;362:1253–5.
12. King BC, Abramson E, DiPace J, Gerber L, Hammad H, Naifeh M. High value, cost-conscious care: perspective of pediatric faculty and residents. Acad Pediatr. 2016;16(6):e7.
13. Korenstein D. Charting the route to high-value care the role of medical education. JAMA. 2015;314(22):2359–61.
14. Weinberger SE. Educating trainees about appropriate and cost-conscious diagnostic testing. Acad Med. 2011;86(11):1352.
15. Weinberger SE. Providing high value cost-conscious care: A critical seventh general competency for physicians. Ann Intern Med. 2011;155:386–8.
16. Ryskina KL, Korenstein D, Weissman A, Masters P, Alguire P, Smith CD. Development of a high-value care subscore on the internal medicine in- training examination. Ann Intern Med. 2014;161(10):733–9.
17. Stammen LA, Stalmeijer RE, Paternotte E, Oudkerk Pool A, Driessen EW, Scheele F, et al. Training physicians to provide high-value, cost-conscious care: a systematic review. JAMA. 2015;314(22):2384–400.
18. Foundation JMJ. Improving environments for learning in the health professions. Atlanta: Macy Foundation Conference; 2018.
19. Cruess RL, Cruess SR, Boudreau JD, Snell L, Steinert Y. A schematic representation of the professional identity formation and socialization of
Mordang et al. BMC Health Services Research (2020) 20:156 Page 9 of 10
medical students and residents: a guide for medical educators. Acad Med. 2015;90(6):718–25.
20. Jarvis-Selinger S, Pratt DD, Regehr G. Competency is not enough: integrating identity formation into the medical education discourse. Acad Med. 2012;87(9):1185–90.
21. Ajzen I. The theory of planned behavior. Organ Behav Hum Decis Process. 1991;50:179–211.
22. Ajzen I. The theory of planned behaviour: reactions and reflections. Psychol Health. 2011;26(9):1113–27.
23. Gupta R, Moriates C, Harrison JD, Valencia V, Ong M, Clarke R, et al. Development of a high-value care culture survey: a modified delphi process and psychometric evaluation. BMJ Qual Saf. 2016;0:1–9.
24. Armitage CJ, Conner M. Efficacy of the theory of planned behaviour: a meta-analytic review. Br J Soc Psychol. 2001;40:471–99.
25. Cutler D, Skinner JS, Stern AD, Wennberg DE. Physician beliefs and patient preferences: a new look at regional variation in health care spending. Am Econ J: Econ Policy. 2019;11(1):192–221.
26. Pearson SD, Goldman L, Orav EJ, Guadagnoli E, Garcia TB, Johnson PA, et al. Triage decisions for emergency department patients with chest pain: do physicians’ risk attitudes make the difference? J Gen Intern Med. 1995; 10(10):557–64.
27. Tubbs EP, Broeckel Elrod JA, Flum DA. Risk taking and tolerance of uncertainty: implications for surgeons. J Surg Res. 2006;131:1–6.
28. Zaat JOM, van Eijk JTM. General Practitioners’ uncertainty, risk preference, and use of laboratory tests. Med Care. 1992;30(9):846–54.
29. Bhatia RS, Levinson W, Shortt S, Pendrith C, Fric-Shamji E, Kallewaard M, et al. Measuring the effect of choosing wisely: an integrated framework to assess campaign impact on low-value care. BMJ Qual Saf. 2015;24(8):523–31.
30. Jochemsen-van der Leeuw HG, van Dijk N, van Etten-Jamaludin FS, Wieringa-de Waard M. The attributes of the clinical trainer as a role model: a systematic review. Acad Med. 2013;88(1):26–34.
31. Finset A. Patient participation, engagement and activation: increased emphasis on the role of patients in healthcare. Patient Educ Couns. 2017;100(7):1245–6.
32. Parand A, Dopson S, Renz A, Vincent C. The role of hospital managers in quality and patient safety: a systematic review. BMJ. 2014;4(9):e005055.
33. Hurst S, Slowther AM, Forde R, Pegoraro R, Reiter-Theil S, Perrier A, et al. Prevalence and determinants of physician bedside rationing. J Gen Intern Med. 2006;21:1138–43.
34. Kirchhoff AC, Hart G, Campbell EG. Rural and urban primary care physician professional beliefs and quality improvement behaviors. J Rural Health. 2014;30:235–43.
35. Tilburt JC, Wynia MK, Sheeler RD, Thorsteinsdottir B, James KM, Egginton JS, et al. Views of US physicians about controlling health care costs. JAMA. 2013;310(4):380–8.
36. Leep Hunderfund AN, Dyrbye LN, Starr SR, Mandrekar J, Naessens JM, Tilburt JC, et al. Role modeling and regional health care intensity: U.S. medical student attitudes toward and experiences with cost-conscious care. Acad Med. 2017;92(5):694–702.
37. Detsky AS. What patients really want from health care. JAMA. 2011;306(22): 2500–1.
38. Weiner BJ, Shortell SM, Alexander J. Promoting clinical involvement in hospital quality improvement efforts: the effects of top management, board, and physician leadership. Health Serv Res. 1997;32(4):491–510.
39. Colla CH, Kinsella EA, Morden NE, Meyers DJ, Rosenthal MB, Sequist TD. Physician perceptions of choosing wisely and drivers of overuse. Am J Manag Care. 2016;22(5):337–43.
40. Hall C. Comment: generalizability theory and assessment in medical training. Neurology. 2015;85:1628.
41. Association AER, Association AP, Education NCoMi. Standards for educational and psychological testing. Washington, United States of America: American Educational Research Association; 2014.
42. Field A. Discovering statistics using IBM SPSS statistics: sage; 2013. 43. McCoach DB, Gable RK, Madura JP. Instrument development in the affective
domain: school and corporate applications. 3rd ed. New York: Springer; 2013. 44. Boateng GO, Neilands TB, Frongillo EA, Melgar-Quinonez HR, Young SL. Best
practices for developing and validating scales for health, social, and behavioral research: a primer. Front Public Health. 2018;6:149.
45. O'Connor BP. SPSS and SAS programs for determining the number of components using parallel analysis and Velicer’s MAP test. Behav Res Methods Instrum Comput. 2000;32(3):396–402.
46. Etchegaray JM, Fischer W. Survey research: be careful where you step. BMJ Qual Saf. 2006;15(3):154–5.
47. American Board Of Internal Medicine Foundation. National physician survey. 2014.
48. Brislin RW. Back-translation for cross-cultural research. J Cross-Cult Psychol. 1970;1(3):185–216.
49. Loewenthal K, Lewis CA. An introduction to psychological tests and scales. Philadelphia: Psychology press; 2015.
50. Bloch R, Norman G. Generalizability theory for the perplexed: a practical introduction and guide: AMEE Guide No. 68. Med Teach. 2012;34(11):960–92.
51. Colla CH, Sequist TD, Rosenthal MB, Schpero WL, Gottlieb DJ, Morden NE. Use of non-indicated cardiac testing in low-risk patients: choosing wisely. BMJ Qual Saf. 2014;24(2):149–53.
52. Tartaglia KM, Kman N, Ledford C. Medical student perceptions of cost- conscious care in an internal medicine clerkship: a thematic analysis. J Gen Intern Med. 2015;30(10):1491–6.
53. Kanzaria HK, Hoffman JR, Probst MA, Caloyeras JP, Berry SH, Brook RH. Emergency physician perceptions of medically unnecessary advanced diagnostic imaging. Acad Emerg Med. 2015;22(4):390–8.
54. Bishop TF, Federman AD, Keyhani S. Physician views on defensive medicine a national survey. Arch Intern Med. 2010;170(12):1081–3.
55. Gupta R, Sehgal N, Arora VM. Aligning Delivery System and Training Missions in Academic Medical Centers to Promote High-Value Care. Acad Med. 2018.
56. Michels NRM, Denekens J, Driessen EW, Van Gaal LF, Bossaert LL, De Winter YD. A Delphi study to construct a CanMEDS competence based inventory applicable for workplace assessment. BMC Med Educ. 2012;12(1):86.
57. Mordang SBR, Könings KD, Leep Hunderfund AN, Paulus ATG, Smeenk FWJM, Stassen LPS. A new instrument to measure attitudes regarding high value, cost-conscious care of healthcare stakeholders: development of the MHAQ. Vienna: Oral presentation at Association for Medical Education in Europe Annual Conference; 2019.
Publisher’s Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Mordang et al. BMC Health Services Research (2020) 20:156 Page 10 of 10
BioMed Central publishes under the Creative Commons Attribution License (CCAL). Under the CCAL, authors retain copyright to the article but users are allowed to download, reprint, distribute and /or copy articles in BioMed Central journals, as long as the original work is properly cited.