Continuous Quality Improvement Initiative

Impact of Teaching Intensity and Sociodemographic Characteristics on CMS Hospital Compare Quality Ratings Wenshuai Wan, MD1, C. Jason Liang, PhD2, Richard Duszak, Jr., MD3, and Christoph I. Lee, MD, MS4,5,6

1Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, PA, USA; 2Biostatistics Research Branch, National Institute of Allergy and Infectious Diseases, National Institutes of Health, Bethesda, Maryland, USA; 3Department of Radiology and Imaging Sciences, Emory University School of Medicine, Atlanta, Georgia, USA; 4Department of Radiology, University of Washington School of Medicine, Seattle, Washington, USA; 5Department of Health Services, University of Washington School of Public Health, Seattle, Washington, USA; 6Hutchinson Institute for Cancer Outcomes Research, Seattle, Washington, USA.

KEY WORDS: health services research; quality assessment; Medicare; sociodemographic factors; risk adjustment.

J Gen Intern Med 33(8):1221–3

DOI: 10.1007/s11606-018-4442-6

© Society of General Internal Medicine 2018

INTRODUCTION

In July 2016, the Centers for Medicare & Medicaid Services (CMS) released BHospital Compare^ ratings for 3576 U.S. hospitals to help consumers compare hospitals’ quality of care, which was distilled into a single overall rating between 1 and 5 stars based on 57 process- and outcomes-based clinical mea- sures.1,2 A recent publication found differences in overall rat- ings by characteristics including a hospital’s teaching status.3

Professional groups (e.g., the American Hospital Association) have expressed concern that the current ratings scheme unfairly penalizes teaching hospitals, especially those serving low- income and otherwise disadvantaged patient populations.4

Our objective was to build a statistical model that adjusted for a more complete set of covariates than previous studies to analyze discrepancies in Hospital Compare ratings between teaching and non-teaching hospitals. Informed by previ- ous findings, we attempted to more fully capture key aspects of an academic mission-driven hospital, including trainee development, research, and uncompensated care.3,5 We based our model on U.S. geographic healthcare markets to assess the potential impact of adjusting overall quality ratings for hospi- tal characteristics, including their locations and the sociodemographic characteristics of their surrounding patient populations.

METHODS

Publicly available data sets were merged from CMS Hospital Compare, CMS Inpatient Prospective Payment System, and the U.S. Census. Children’s hospitals were excluded due to exemption from data reporting. Sociodemographic indicators were mapped to hospital location by health service area (HSA). Our study was granted exempt status by the

Institutional Review Board of the Hospital of the University of Pennsylvania. The relationships between Hospital Compare ratings and

various predictors were analyzed by univariate and multivar- iate linear regression models using R, version 3.4.0. We per- formed sensitivity analysis using two propensity score matching methods (multivariate logistic regression and ran- dom forests) to account for non-linear and non-additive associations.

RESULTS

In a multivariate linear regression model adjusting for hospital characteristics, geography, and sociodemographic factors, each increase of 1 resident (trainee) per 10 beds, a continuous measure of teaching intensity, was associated with a 0.09 Hospital Compare star rating decrease (p < 0.01, Table 1). Figure 1 also shows the correlation between increasing teach- ing intensity, as captured by resident-to-bed ratio, and overall mean Hospital Compare rating. Two propensity score sensitivity analyses, which matched

teaching and non-teaching hospitals on all geographic and sociodemographic variables in Table 1 (except resident-to- bed ratio) produced similar results. With multivariate logistic regression and random forest propensity matching, teaching hospitals had 0.32 (95% CI, 0.24, 0.39) and 0.23 (95% CI, 0.16, 0.31) lower Hospital Compare ratings, respectively (out of a possible 5 stars). Other non-teaching-related measures were also indepen-

dently associated with significant differences in Hospital Compare ratings when all hospital and sociodemographic factors were included in a single, multivariate linear regression (Table 1). Hospitals offering emergency services were associ- ated with lower ratings by 0.24 stars (p < 0.01). Location by U.S. region was highly associated with differences in hospital ratings. Compared to the largest cohort of hospitals (South Atlantic, n = 568), cohorts in the East North Central or Mid- Atlantic received up to 0.40 higher or 0.20 lower ratings, respectively (p < 0.01 for both).Published online April 16, 2018

1221

Professional groups (e.g., the American Hospital Association) have expressed concern that the current ratings scheme unfairly penalizes teaching hospitals, especially those serving low- income and otherwise disadvantaged patient populations. (See reference 4)

also shows the correlation between increasing teaching intensity, as captured by resident-to-bed ratio, and overall mean Hospital Compare rating.

In addition, HSA-level sociodemographic factors were inde- pendently associated with differences in hospital ratings (Table 1). Age, race, unemployment, and education (bachelor’s degree) of populations within a hospital’s HSA all contributed to variations in quality rating (p < 0.05 for race, others p < 0.01).

DISCUSSION

Our analysis suggests that an institution’s Hospital Compare score is strongly influenced by differences in hospital character- istics and sociodemographic factors. Even after adjusting for their location and surrounding patient populations, Hospital Compare ratings are lower for academic hospitals, and this rating penalty increases with higher teaching intensity. In addition, hospitals providing emergency services and those surrounded by a higher density of patients with sociodemographic risk factors also re- ceive lower overall ratings.

Table 1 Association Between Hospital, Location, and Sociodemographic Characteristics and Hospital Compare Ratings

Univariate linear regressions Multivariate linear regressiona

Coefficientb Std Error p value Coefficient Std Error p value

Hospital level predictors Resident-to-bed ratio − 1.390 0.089 < 0.001* − 0.899 0.103 < 0.001* Emergency services − 0.318 0.101 0.002* − 0.243 0.089 0.006* Ownership

Nonprofit – Reference – Reference Government − 0.223 0.043 < 0.001* − 0.116 0.040 0.004* Proprietary − 0.235 0.039 < 0.001* − 0.207 0.037 < 0.001*

# beds − 0.001 < 0.001 < 0.001* < 0.001 < 0.001 0.477 % occupied (inpatient) − 0.008 0.001 < 0.001* − 0.003 0.001 0.002* Locationc

South Atlantic – Reference – Reference Mid Atlantic − 0.251 0.057 < 0.001* − 0.201 0.061 0.001* New England 0.315 0.081 < 0.001* 0.198 0.080 0.014* East North Central 0.435 0.052 < 0.001* 0.400 0.055 < 0.001* East South Central 0.063 0.061 0.300* 0.086 0.059 0.140 West North Central 0.424 0.065 < 0.001* 0.138 0.064 0.032* West South Central 0.292 0.053 < 0.001* 0.203 0.055 < 0.001* Mountain 0.221 0.069 0.001* 0.116 0.075 0.124 Pacific 0.034 0.056 0.544 0.157 0.072 0.029*

Sociodemographic predictorsd

Race Non-Hispanic White – Reference – Reference African-American − 0.012 0.001 < 0.001* − 0.004 0.001 0.013* Hispanic − 0.007 0.001 < 0.001* − 0.005 0.006 0.408 Asian − 0.006 0.003 0.014* 0.006 0.003 0.042*

Non-English speakinge − 0.009 0.001 < 0.001* 0.007 0.006 0.287 Age > 65 − 0.001 0.004 0.848 − 0.025 0.005 < 0.001* Education

GED − 0.003 0.002 0.177 < 0.001 0.005 0.925 Bachelor’s 0.017 0.002 < 0.001* 0.026 0.006 < 0.001*

Unemployment − 0.126 0.011 < 0.001* − 0.044 0.015 0.003* Poverty

Individual − 0.022 0.003 < 0.001* − 0.019 0.014 0.190 Household − 0.021 0.003 < 0.001* 0.013 0.015 0.384

Abandoned Homesf − 0.003 0.003 0.327 0.005 0.003 0.098

*Statistically significant to p < 0.05 aAdjusted for all hospital characteristics, geography, and sociodemographic factors in Table 1 bCoefficients measured on Hospital Compare scale between 1 and 5 stars, i.e., 0.1 correlates with 0.1 higher overall quality star rating cRegion categorized by Medicare Hospital Inpatient Prospective Payment System dFrom 2010 U.S. Census (Summary File 1) and 2006–2010 American Community Survey by percent of population per health service area (HSA) eMeasured based on Spanish only fluency in U.S. Census fPercent of residential units without ownership, occupancy, or management for > 2 months annually

Fig. 1 Hospital compare ratings based on resident-to-bed ratio. Increased teaching intensity, represented by higher resident-to-bed ratios, was associated with a decreasing mean overall Hospital

Compare rating (between 1 and 5 stars) at U.S. teaching hospitals.

1222 Wan et al.: Adjustments for Teaching Hospital Quality Ratings JGIM

Univariate linear regressions Multivariate linear regression (see footnote a)

Coefficient (see footnote b)

Std Error p value Coefficient Std Error p value

Hospital level predictors Resident-to-bed ratio � 1.390 0.089 < 0.001* � 0.899 0.103 < 0.001* Emergency services Ownership � 0.318 0.101 0.002* � 0.243 0.089 0.006* Ownership Nonprofit � Reference � Reference Government � 0.223 0.043 < 0.001* � 0.116 0.040 0.004* Proprietary � 0.235 0.039 < 0.001* � 0.207 0.037 < 0.001* # beds � 0.001 < 0.001 < 0.001* < 0.001 < 0.001 0.477 % occupied (inpatient) � 0.008 0.001 < 0.001* � 0.003 0.001 0.002* Location (see footnote c) South Atlantic � Reference � Reference Mid Atlantic � 0.251 0.057 < 0.001* � 0.201 0.061 0.001* New England 0.315 0.081 < 0.001* 0.198 0.080 0.014* East North Central 0.435 0.052 < 0.001* 0.400 0.055 < 0.001* East South Central 0.063 0.061 0.300* 0.086 0.059 0.140 West North Central 0.424 0.065 < 0.001* 0.138 0.064 0.032* West South Central 0.292 0.053 < 0.001* 0.203 0.055 < 0.001* Mountain 0.221 0.069 0.001* 0.116 0.075 0.124 Pacific 0.034 0.056 0.544 0.157 0.072 0.029*

Sociodemographic predictors see footnote d Race Non-Hispanic White � Reference � Reference African-American � 0.012 0.001 < 0.001* � 0.004 0.001 0.013* Hispanic � 0.007 0.001 < 0.001* � 0.005 0.006 0.408 Asian � 0.006 0.003 0.014* 0.006 0.003 0.042* Non-English speaking (see footnote e)

� 0.009 0.001 < 0.001* 0.007 0.006 0.287 Age > 65 � 0.001 0.004 0.848 � 0.025 0.005 < 0.001* Education GED -0.003 0.002 0.177 < 0.001 0.005 0.925 Bachelor�s 0.017 0.002 < 0.001* 0.026 0.006 < 0.001* Unemployment � 0.126 0.011 < 0.001* � 0.044 0.015 0.003* Poverty Individual � 0.022 0.003 < 0.001* � 0.019 0.014 0.190 Household � 0.021 0.003 < 0.001* 0.013 0.015 0.384 Abandoned Homes (see footnote f)

-0.003 0.003 0.327 0.005 0.003 0.098 *Statistically significant to p < 0.05 a Adjusted for all hospital characteristics. geography. and sociodemographic factors in Table 1 b Coefficients measured on Hospital Compare scale between 1 and 5 stars. i.e. 0.1 correlates with 0.1 higher overall quality star rating c Region categorized by Medicare Hospital Inpatient Prospective Payment System d From 2010 U.S. Census (Summary File 1) and 2006-2010 American Community Survey by percent of population per health service area (HSA) e Measured based on Spanish only fluency in U.S. Census f Percent of residential units without ownership, occupancy, or management for > 2 months annually

In addition, HSA-level sociodemographic factors were independently associated with differences in hospital ratings (Table 1). Age, race, unemployment, and education (bachelor�s degree) of populations within a hospital�s HSA all contributed to variations in quality rating (p <0.05 for race, others p<0.01).

Our analysis suggests that an institution�s Hospital Compare score is strongly influenced by differences in hospital characteristics and sociodemographic factors. Even after adjusting for their location and surrounding patient populations, Hospital Compare ratings are lower for academic hospitals, and this rating penalty increases with higher teaching intensity. In addition, hospitals providing emergency services and those surrounded by a higher density of patients with sociodemographic risk factors also receive lower overall ratings. Fig. 1 Hospital compare ratings based on resident-to-bed ratio. Increased teaching

intensity, represented by higher resident-to-bed ratios, was associated with a decreasing mean overall Hospital Compare rating (between 1 and 5 stars) at U.S. teaching hospitals.

While our study does not fully explain how higher teaching intensity, as measured by resident-to-bed ratio, is associated with poorer Hospital Compare ratings, it may be that academic hospitals are called upon to provide a larger spectrum of services or are receiving incomplete adjustment for clinical factors (e.g., disease severity or comorbidities). Policymakers must ensure that quality measurement, as embodied in Hospi- tal Compare ratings, does not penalize academic hospitals for treating vulnerable and challenging patients, or for providing postgraduate medical education.

Corresponding Author: Wenshuai Wan, MD; Department of Radiol- ogy Hospital of the University of Pennsylvania, Philadelphia, PA, USA (e-mail: [email protected]).

Funding Information Wenshuai Wan, MD received funding during the research project from a Ruth L. Kirschstein National Research Service Award (T32).

Compliance with Ethical Standards:

Conflict of Interest: The authors declare that they do not have a conflict of interest.

REFERENCES 1. Centers for Medicare & Medicaid Services. CMS Releases First Ever

Hospital Compare Star Ratings. 2015 Apr 16. Available at: http://www. cms.gov/newsroom/mediareleasedatabase/press-releases/2015-press-re- leases-items/2015-04-16.html. Accessed 21 March 2018.

2. Centers for Medicare & Medicaid Services. First Release of the Overall Hospital Quality Star Rating on Hospital Compare. 2016 Jul 27. Available at: http://www.cms.gov/Newsroom/MediaReleaseDatabase/Fact-sheets/ 2016-Fact-sheets-items/2016-07-27.html. Accessed 21 March 2018.

3. DeLancey JO, Softcheck J, Chung JW. Associations Between Hospital Characteristics, Measure Reporting, and the Centers for Medicare & Medicaid Services Overall Hospital Quality Star Ratings. JAMA. 2017;317(19):2015–2017.

4. Pollack R. Statement on Centers for Medicare and Medicaid Services (CMS) overall hospital star ratings. 2016 Jul 27. Available at: http://www. aha.org/presscenter/pressrel/2016/160727-pr-starratings2.shtml. Accessed 21 March 2018.

5. Fiscella K, Burstin HR, Nerenz DR. Quality measures and sociodemographic risk factors: to adjust or not to adjust. JAMA. 2014;312(24):2615–6.

1223Wan et al.: Adjustments for Teaching Hospital Quality RatingsJGIM

While our study does not fully explain how higher teaching intensity, as measured by resident-to-bed ratio, is associated with poorer Hospital Compare ratings, it may be that academic hospitals are called upon to provide a larger spectrum of services or are receiving incomplete adjustment for clinical factors (e.g., disease severity or comorbidities). Policymakers must ensure that quality measurement, as embodied in Hospi- tal Compare ratings, does not penalize academic hospitals for treating vulnerable and challenging patients, or for providing postgraduate medical education.

Corresponding Author: Wenshuai Wan, MD: Department of Radiology Hospital of the University of Pennsylvania. Philadelphia, PA, USA (e-mail:

Funding Information Wenshuai Wan, MD received funding during the research project from a Ruth L. Kirschstein National Research Service Award (T32).

Compliance with Ethical Standards:

Conflict of Interest: The authors declare that they do not have a conflict of interest.

1. Centers for Medicare & Medicaid Services. CMS Releases First Ever Hospital Compare Star Ratings. 2015 Apr 16. Available at:

http://www.ms.gov/newsroom/mediareleasedatabase/press-releases/2015-pressreleases-items/2015-04-16.html Accessed 21 March 2018.

2. Centers for Medicare & Medicald Services, First Release of the Overall Hospital Quality Star Rating on Hospital Compare, 2016 Jul 27. Available at: http://www.cms.gov/Newsroom/MediaReleaseDatabase /Fact-sheets/ '2016-Fact-sheets-items/2016-07-27.html. Accessed 21 March 2018.

http://www.cms.gov/Newsroom/MediaReleaseDatabase /Fact-sheets/ '2016-Fact-sheets-items/2016-07-27.html.

4. Pollack R. Statement on Centers for Medicare and Medicaid Services (CMS) overall hospital star ratings. 2016 Jul 27. Available at: http://www. aha.org/presscenter/pressrel/2016/160727-pr-starratings2.shtml. Accessed 21 March 2018,http://www.aha.org/presscenter/pressrel/2016/160727-pr-starratings2.shtm

5. Fiscella K, Burstin HR, Nerenz DR. Quality measures and sociodemographic risk factors: to adjust or not to adjust. JAMA. 2014:312(24):2615-6.

Reproduced with permission of copyright owner. Further reproduction prohibited without permission.

Reproduced with permission of copyright owner. Further reproduction prohibited without permission.