Discussion 8 health policy

ORIGINAL RESEARCH

Hospital Performance Trends on National Quality Measures and the Association With Joint Commission Accreditation

Stephen P. Schmaltz, MPH, PhD1*, Scott C. Williams, PsyD1, Mark R. Chassin, MD, MPP, MPH1, Jerod M. Loeb, PhD1, Robert M. Wachter, MD2

1The Joint Commission, Oakbrook Terrace, Illinois; 2University of California, San Francisco, San Francisco, California.

BACKGROUND: Evaluations of the impact of hospital accreditation have been previously hampered by the lack of nationally standardized data. One way to assess this impact is to compare accreditation status with other evidence- based measures of quality, such as the process measures now publicly reported by The Joint Commission and the Centers for Medicare and Medicaid Services (CMS).

OBJECTIVES: To examine the association between Joint Commission accreditation status and both absolute measures of, and trends in, hospital performance on publicly reported quality measures for common diseases.

DESIGN, SETTING, AND PATIENTS: Performance data for 2004 and 2008 from U.S. acute care and critical access hospitals were obtained using publicly available CMS Hospital Compare data augmented with Joint Commission performance data.

MEASUREMENTS: Changes in hospital performance between 2004 and 2008, and percent of hospitals with 2008

performance exceeding 90% for 16 measures of quality-of- care and 4 summary scores.

RESULTS: Hospitals accredited by The Joint Commission tended to have better baseline performance in 2004 than non-accredited hospitals. Accredited hospitals had larger gains over time, and were significantly more likely to have high performance in 2008 on 13 out of 16 standardized clinical performance measures and all summary scores.

CONCLUSIONS: While Joint Commission-accredited hospitals already outperformed non-accredited hospitals on publicly reported quality measures in the early days of public reporting, these differences became significantly more pronounced over 5 years of observation. Future research should examine whether accreditation actually promotes improved performance or is a marker for other hospital characteristics associated with such performance. Journal of Hospital Medicine 2011;6:454–461. VC 2011 Society of Hospital Medicine

The Joint Commission (TJC) currently accredits approximately 4546 acute care, critical access, and specialty hospitals,1 accounting for approximately 82% of U.S. hospitals (representing 92% of hospital beds). Hospitals seeking to earn and maintain ac- creditation undergo unannounced on-site visits by a team of Joint Commission surveyors at least once every 3 years. These surveys address a variety of domains, including the environment of care, infec- tion prevention and control, information manage- ment, adherence to a series of national patient safety goals, and leadership.1

The survey process has changed markedly in recent years. Since 2002, accredited hospitals have been required to continuously collect and submit selected performance measure data to The Joint Commission throughout the three-year accreditation cycle. The tracer methodology, an evaluation method in which

surveyors select a patient to follow through the orga- nization in order to assess compliance with selected standards, was instituted in 2004. Soon thereafter, on-site surveys went from announced to unannounced in 2006. Despite the 50þ year history of hospital accredita-

tion in the United States, there has been surprisingly little research on the link between accreditation status and measures of hospital quality (both processes and outcomes). It is only recently that a growing number of studies have attempted to examine this relationship. Empirical support for the relationship between accred- itation and other quality measures is emerging. Accredited hospitals have been shown to provide bet- ter emergency response planning2 and training3 com- pared to non-accredited hospitals. Accreditation has been observed to be a key predictor of patient safety system implementation4 and the primary driver of hospitals’ patient-safety initiatives.5 Accredited trauma centers have been associated with significant reduc- tions in patient mortality,6 and accreditation has been linked to better compliance with evidence-based meth- adone and substance abuse treatment.7,8 Accredited hospitals have been shown to perform better on meas- ures of hospital quality in acute myocardial infarction (AMI), heart failure, and pneumonia care.9,10 Simi- larly, accreditation has been associated with lower risk-adjusted in-hospital mortality rates for congestive

Received: July 29, 2010; Revised: August 25, 2010; Accepted: January 10, 2011 2011 Society of Hospital Medicine DOI 10.1002/jhm.905 Published online in Wiley Online Library (Wileyonlinelibrary.com).

Additional Supporting Information may be found in the online version of this article.

*Address for correspondence and reprint requests: Stephen P. Schmaltz, MPH, PhD, The Joint Commission, One Renaissance Blvd., Oakbrook Terrace, IL 60181; Tel.: 630-792-5243; E-mail: [email protected]

454 An Official Publication of the Society of Hospital Medicine Journal of Hospital Medicine Vol 6 | No 8 | October 2011

heart failure (CHF), stroke, and pneumonia.11,12 The results of such research, however, have not always been consistent. Several studies have been unable to demonstrate a relationship between accreditation and quality measures. A study of financial and cost-related outcome measures found no relationship to accredita- tion,13 and a study comparing medication error rates across different types of organizations found no rela- tionship to accreditation status.14 Similarly, a compar- ison of accredited versus non-accredited ambulatory surgical organizations found that patients were less likely to be hospitalized when treated at an accredited facility for colonoscopy procedures, but no such rela- tionship was observed for the other 4 procedures studied.15

While the research to date has been generally sup- portive of the link between accreditation and other measures of health care quality, the studies were typi- cally limited to only a few measures and/or involved relatively small samples of accredited and non-accred- ited organizations. Over the last decade, however, changes in the performance measurement landscape have created previously unavailable opportunities to more robustly examine the relationship between ac- creditation and other indicators of hospital quality. At about the same time that The Joint Commis-

sion’s accreditation process was becoming more vig- orous, the Centers for Medicare and Medicaid Serv- ices (CMS) began a program of publicly reporting quality data (http://www. hospitalcompare.hhs.gov). The alignment of Joint Commission and CMS quality measures establishes a mechanism through which accredited and non- accredited hospitals can be compared using the same nationally standardized quality measures. Therefore, we took advantage of this unique circumstance—a new and more robust TJC accreditation program and the launching of public quality reporting—to exam- ine the relationship between Joint Commission ac- creditation status and publicly reported hospital quality measures. Moreover, by examining trends in these publicly reported measures over five years and incorporating performance data not found in the Hospital Compare Database, we assessed whether ac- creditation status was also linked to the pace of per- formance improvement over time. By using a population of hospitals and a range of

standardized quality measures greater than those used in previous studies, we seek to address the following questions: Is Joint Commission accreditation status truly associated with higher quality care? And does ac- creditation status help identify hospitals that are more likely to improve their quality and safety over time?

METHODS Performance Measures

Since July 2002, U.S. hospitals have been collecting data on standardized measures of quality developed by

The Joint Commission and CMS. These measures have been endorsed by the National Quality Forum16 and adopted by the Hospital Quality Alliance.17 The first peer-reviewed reports using The Joint Commission/ CMS measure data confirmed that the measures could successfully monitor and track hospital improvement and identify disparities in performance,18,19 as called for by the Institute of Medicine’s (IOM) landmark 2001 report, Crossing the Quality Chasm.20

In order to promote transparency in health care, both CMS—through the efforts of the Hospital Quality Alliance—and The Joint Commission began publicly reporting measure rates in 2004 using identical mea- sure and data element specifications. It is important to note that during the five-year span covered by this study, both The Joint Commission and CMS empha- sized the reporting of performance measure data. While performance improvement has been the clear objective of these efforts, neither organization estab- lished targets for measure rates or set benchmarks for performance improvement. Similarly, while Joint Commission-accredited hospitals were required to submit performance measure data as a condition of accreditation, their actual performance on the mea- sure rates did not factor into the accreditation deci- sion. In the absence of such direct leverage, it is inter- esting to note that several studies have demonstrated the positive impact of public reporting on hospital performance,21 and on providing useful information to the general public and health care professionals regarding hospital quality.22

The 16 measures used in this study address hospital compliance with evidence-based processes of care rec- ommended by the clinical treatment guidelines of respected professional societies.23 Process of care measures are particularly well suited for quality improvement purposes, as they can identify deficien- cies which can be immediately addressed by hospitals and do not require risk-adjustment, as opposed to out- come measures, which do not necessarily directly iden- tify obvious performance improvement opportuni- ties.24–26 The measures were also implemented in sets in order to provide hospitals with a more complete portrayal of quality than might be provided using unrelated individual measures. Research has demon- strated that greater collective performance on these process measures is associated with improved one- year survival after heart failure hospitalization27 and inpatient mortality for those Medicare patients dis- charged with acute myocardial infarction, heart fail- ure, and pneumonia,28 while other research has shown little association with short-term outcomes.29

Using the Specifications Manual for National Hospi- tal Inpatient Quality Measures,16 hospitals identify the initial measure populations through International Clas- sification of Diseases (ICD-CM-9) codes and patient age obtained through administrative data. Trained

An Official Publication of the Society of Hospital Medicine Journal of Hospital Medicine Vol 6 | No 8 | October 2011 455

Hospital Performance Trends | Schmaltz et al.

abstractors then collect the data for measure-specific data elements through medical record review on the identified measure population or a sample of this popu- lation. Measure algorithms then identify patients in the numerator and denominator of each measure. Process measure rates reflect the number of times a

hospital treated a patient in a manner consistent with specific evidence-based clinical practice guidelines (nu- merator cases), divided by the number of patients who were eligible to receive such care (denominator cases). Because precise measure specifications permit the exclusion of patients contraindicated to receive the specific process of care for the measure, ideal perform- ance should be characterized by measure rates that approach 100% (although rare or unpredictable situa- tions, and the reality that no measure is perfect in its design, make consistent performance at 100% improbable). Accuracy of the measure data, as meas- ured by data element agreement rates on reabstrac- tion, has been reported to exceed 90%.30

In addition to the individual performance measures, hospital performance was assessed using 3 condition- specific summary scores, one for each of the 3 clinical areas: acute myocardial infarction, heart failure, and pneumonia. The summary scores are a weighted av- erage of the individual measure rates in the clinical area, where the weights are the sample sizes for each of the measures.31 A summary score was also calcu- lated based on all 16 measures as a summary mea- sure of overall compliance with recommended care. One way of studying performance measurement in a

way that relates to standards is to evaluate whether a hospital achieves a high rate of performance, where high is defined as a performance rate of 90% or more. In this context, measures were created from each of the 2004 and 2008 hospital performance rates by dichotomizing them as being either less than 90%, or greater than or equal to 90%.32

Data Sources

The data for the measures included in the study are available on the CMS Hospital Compare public data- bases or The Joint Commission for discharges in 2004 and 2008.33 These 16 measures, active for all 5 years of the study period, include: 7 measures related to acute myocardial infarction care; 4 measures related to heart failure care; and 5 measures related to pneu- monia care. The majority of the performance data for the study were obtained from the yearly CMS Hospi- tal Compare public download databases (http:// www.medicare.gov/Download/DownloadDB.asp). When hospitals only reported to The Joint Commis- sion (154 hospitals; of which 118 are Veterans Administration and 30 are Department of Defense hospitals), data were obtained from The Joint Com- mission’s ORYX database, which is available for pub- lic download on The Joint Commission’s Quality Check web site.23 Most accredited hospitals partici-

pated in Hospital Compare (95.5% of accredited hos- pitals in 2004 and 93.3% in 2008).

Hospital Characteristics

We then linked the CMS performance data, aug- mented by The Joint Commission performance data when necessary, to hospital characteristics data in the American Hospital Association (AHA) Annual Survey with respect to profit status, number of beds (<100 beds, 100–299 beds, 300þ beds), rural status, geo- graphic region, and whether or not the hospital was a critical access hospital. (Teaching status, although available in the AHA database, was not used in the analysis, as almost all teaching hospitals are Joint Commission accredited.) These characteristics were chosen since previous research has identified them as being associated with hospital quality.9,19,34–37 Data on accreditation status were obtained from The Joint Commission’s hospital accreditation database. Hospi- tals were grouped into 3 hospital accreditation strata based on longitudinal hospital accreditation status between 2004 and 2008: 1) hospitals not accredited in the study period; 2) hospitals accredited between one to four years; and 3) hospitals accredited for the entire study period. Analyses of this middle group (those hospitals accredited for part of the study period; n ¼ 212, 5.4% of the whole sample) led to no significant change in our findings (their performance tended to be midway between always accredited and never-accred- ited hospitals) and are thus omitted from our results. Instead, we present only hospitals who were never accredited (n ¼ 762) and those who were accredited through the entire study period (n ¼ 2917).

Statistical Analysis

We assessed the relationship between hospital charac- teristics and 2004 performance of Joint Commission- accredited hospitals with hospitals that were not Joint Commission accredited using v2 tests for categorical variables and t tests for continuous variables. Linear regression was used to estimate the five-year change in performance at each hospital as a function of accredi- tation group, controlling for hospital characteristics. Baseline hospital performance was also included in the regression models to control for ceiling effects for those hospitals with high baseline performance. To summarize the results, we used the regression models to calculate adjusted change in performance for each accreditation group, and calculated a 95% confidence interval and P value for the difference between the adjusted change scores, using bootstrap methods.38

Next we analyzed the association between accredita- tion and the likelihood of high 2008 hospital perform- ance by dichotomizing the hospital rates, using a 90% cut point, and using logistic regression to estimate the probability of high performance as a function of ac- creditation group, controlling for hospital characteris- tics and baseline hospital performance. The logistic

456 An Official Publication of the Society of Hospital Medicine Journal of Hospital Medicine Vol 6 | No 8 | October 2011

Schmaltz et al. | Hospital Performance Trends

models were then used to calculate adjusted rates of high performance for each accreditation group in pre- senting the results. We used two-sided tests for significance; P < 0.05

was considered statistically significant. This study had no external funding source.

RESULTS For the 16 individual measures used in this study, a total of 4798 hospitals participated in Hospital Com- pare or reported data to The Joint Commission in 2004 or 2008. Of these, 907 were excluded because the performance data were not available for either 2004 (576 hospitals) or 2008 (331 hospitals) resulting in a missing value for the change in performance score. Therefore, 3891 hospitals (81%) were included in the final analyses. The 907 excluded hospitals were more likely to be rural (50.8% vs 17.5%), be critical access hospitals (53.9% vs 13.9%), have less than 100 beds (77.4% vs 37.6%), be government owned (34.6% vs 22.1%), be for profit (61.4% vs 49.5%), or be unaccredited (79.8% vs 45.8% in 2004; 75.6% vs 12.8% in 2008), compared with the included hos- pitals (P < 0.001 for all comparisons).

Hospital Performance at Baseline

Joint Commission-accredited hospitals were more likely to be large, for profit, or urban, and less likely to be government owned, from the Midwest, or criti- cal access (Table 1). Non-accredited hospitals per- formed more poorly than accredited hospitals on most of the publicly reported measures in 2004; the only exception is the timing of initial antibiotic therapy measure for pneumonia (Table 2).

TABLE 1. Hospital Characteristics in 2004 Stratified by Joint Commission Accreditation Status

Characteristic

Non-Accredited

(n ¼ 786) Accredited

(n ¼ 3105) P

Value*

Profit status, No. (%) <0.001 For profit 60 (7.6) 586 (18.9) Government 289 (36.8) 569 (18.3) Not for profit 437 (55.6) 1,950 (62.8)

Census region, No. (%) <0.001 Northeast 72 (9.2) 497 (16.0) Midwest 345 (43.9) 716 (23.1) South 248 (31.6) 1,291 (41.6) West 121 (15.4) 601 (19.4)

Rural setting, No. (%) <0.001 Rural 495 (63.0) 833 (26.8) Urban 291 (37.0) 2,272 (73.2)

Bed size <0.001 <100 beds 603 (76.7) 861 (27.7) 100–299 beds 158 (20.1) 1,444 (46.5) 300þ beds 25 (3.2) 800 (25.8)

Critical access hospital status, No. (%) <0.001 Critical access hospital 376 (47.8) 164 (5.3) Acute care hospital 410 (52.2) 2,941 (94.7)

* P values based on v2 for categorical variables.

TABLE 2. Hospital Raw Performance in 2004 and 2008, Stratified by Joint Commission Accreditation Status

Quality Measure, Mean (SD)*

2004 2008

Non-Accredited Accredited

P Value†

Non-Accredited Accredited

P Value†(n ¼ 786) (n ¼ 3105) (n ¼ 950) (n ¼ 2,941)

AMI Aspirin at admission 87.1 (20.0) 92.6 (9.4) <0.001 88.6 (22.1) 96.0 (8.6) <0.001 Aspirin at discharge 81.2 (26.1) 88.5 (14.9) <0.001 87.8 (22.7) 94.8 (10.1) <0.001 ACE inhibitor for LV dysfunction 72.1 (33.4) 76.7 (22.9) 0.010 83.2 (30.5) 92.1 (14.8) <0.001 Beta blocker at discharge 78.2 (27.9) 87.0 (16.2) <0.001 87.4 (23.4) 95.5 (9.9) <0.001 Smoking cessation advice 59.6 (40.8) 74.5 (29.9) <0.001 87.2 (29.5) 97.2 (11.3) <0.001 PCI received within 90 min 60.3 (26.2) 60.6 (23.8) 0.946 70.1 (24.8) 77.7 (19.2) 0.006 Thrombolytic agent within 30 min 27.9 (35.5) 32.1 (32.8) 0.152 31.4 (40.7) 43.7 (40.2) 0.008 Composite AMI score 80.6 (20.3) 87.7 (10.4) <0.001 85.8 (20.0) 94.6 (8.1) <0.001

Heart failure Discharge instructions 36.8 (32.3) 49.7 (28.2) <0.001 67.4 (29.6) 82.3 (16.4) <0.001 Assessment of LV function 63.3 (27.6) 83.6 (14.9) <0.001 79.6 (24.4) 95.6 (8.1) <0.001 ACE inhibitor for LV dysfunction 70.8 (27.6) 75.7 (16.3) <0.001 82.5 (22.7) 91.5 (9.7) <0.001 Smoking cessation advice 57.1 (36.4) 68.6 (26.2) <0.001 81.5 (29.9) 96.1 (10.7) <0.001 Composite heart failure score 56.3 (24.1) 71.2 (15.6) <0.001 75.4 (22.3) 90.4 (9.4) <0.001

Pneumonia Oxygenation assessment 97.4 (7.3) 98.4 (4.0) <0.001 99.0 (3.2) 99.7 (1.2) <0.001 Pneumococcal vaccination 45.5 (29.0) 48.7 (26.2) 0.007 79.9 (21.3) 87.9 (12.9) <0.001 Timing of initial antibiotic therapy 80.6 (13.1) 70.9 (14.0) <0.001 93.4 (9.2) 93.6 (6.1) 0.525 Smoking cessation advice 56.6 (33.1) 65.7 (24.8) <0.001 81.6 (25.1) 94.4 (11.4) <0.001 Initial antibiotic selection 73.6 (19.6) 74.1 (13.4) 0.508 86.1 (13.8) 88.6 (8.7) <0.001 Composite pneumonia score 77.2 (10.2) 76.6 (8.2) 0.119 90.0 (9.6) 93.6 (4.9) <0.001

Overall composite 73.7 (10.6) 78.0 (8.7) <0.001 86.8 (11.1) 93.3 (5.0) <0.001

Abbreviations: ACE, angiotensin-converting enzyme; AMI, acute myocardial infarction; LV, left ventricular; PCI, percutaneous coronary intervention. * Calculated as the proportion of all eligible patients who received the indicated care. † P values based on t tests.

An Official Publication of the Society of Hospital Medicine Journal of Hospital Medicine Vol 6 | No 8 | October 2011 457

Hospital Performance Trends | Schmaltz et al.

Five-Year Changes in Hospital Performance

Between 2004 and 2008, Joint Commission-accredited hospitals improved their performance more than did non-accredited hospitals (Table 3). After adjustment for baseline characteristics previously shown to be associated with performance, the overall relative (absolute) difference in improvement was 26% (4.2%) (AMI score difference 67% [3.9%], CHF 48% [10.1%], and pneumonia 21% [3.7%]). Accredited hospitals improved their performance significantly more than non-accredited for 13 of the 16 individual performance measures.

High Performing Hospitals in 2008

The likelihood that a hospital was a high performer in 2008 was significantly associated with Joint Commis- sion accreditation status, with a higher proportion of accredited hospitals reaching the 90% threshold com- pared to never-accredited hospitals (Table 4). Accred- ited hospitals attained the 90% threshold significantly more often for 13 of the 16 performance measures and all four summary scores, compared to non-accred- ited hospitals. In 2008, 82% of Joint Commission- accredited hospitals demonstrated greater than 90% on the overall summary score, compared to 48% of never-accredited hospitals. Even after adjusting for differences among hospitals, including performance at baseline, Joint Commission-accredited hospitals were

more likely than never-accredited hospitals to exceed 90% performance in 2008 (84% vs 69%).

DISCUSSION While accreditation has face validity and is desired by key stakeholders, it is expensive and time consuming. Stakeholders thus are justified in seeking evidence that accreditation is associated with better quality and safety. Ideally, not only would it be associated with better performance at a single point in time, it would also be associated with the pace of improvement over time. Our study is the first, to our knowledge, to show

the association of accreditation status with improve- ment in the trajectory of performance over a five-year period. Taking advantage of the fact that the accredi- tation process changed substantially at about the same time that TJC and CMS began requiring public reporting of evidence-based quality measures, we found that hospitals accredited by The Joint Commis- sion had had larger improvements in hospital per- formance from 2004 to 2008 than non-accredited hos- pitals, even though the former started with higher baseline performance levels. This accelerated improve- ment was broad-based: Accredited hospitals were more likely to achieve superior performance (greater than 90% adherence to quality measures) in 2008 on 13 of 16 nationally standardized quality-of-care meas- ures, three clinical area summary scores, and an

TABLE 3. Performance Change and Difference in Performance Change From 2004 to 2008 by Joint Commission Accreditation Status

Characteristic

Change in Performance*

Absolute Difference,

Always vs Never (95% CI)† Relative Difference,

% Always vs Never P Value† Never Accredited

(n ¼ 762) Always Accredited

(n ¼ 2,917)

AMI Aspirin at admission �1.1 2.0 3.2 (1.2–5.2) 160 0.001 Aspirin at discharge 4.7 8.0 3.2 (1.4–5.1) 40 0.008 ACE inhibitor for LV dysfunction 8.5 15.9 7.4 (3.7–11.5) 47 <0.001 Beta blocker at discharge 4.4 8.4 4.0 (2.0–6.0) 48 <0.001 Smoking cessation advice 18.6 22.4 3.7 (1.1–6.9) 17 0.012 PCI received within 90 min 6.3 13.0 6.7 (0.3–14.2) 52 0.070 Thrombolytic agent within 30 min �0.6 5.4 6.1 (�9.5–20.4) 113 0.421 Composite AMI score 2.0 5.8 3.9 (2.2–5.5) 67 <0.001

Heart failure Discharge instructions 24.2 35.6 11.4 (8.7–14.0) 32 <0.001 Assessment of LV function 4.6 12.8 8.3 (6.6–10.0) 65 <0.001 ACE inhibitor for LV dysfunction 10.1 15.2 5.1 (3.5–6.8) 34 <0.001 Smoking cessation advice 20.5 26.4 6.0 (3.3–8.7) 23 <0.001 Composite heart failure score 10.8 20.9 10.1 (8.3–12.0) 48 <0.001

Pneumonia Oxygenation assessment 0.9 1.4 0.6 (0.3–0.9) 43 <0.001 Pneumococcal vaccination 33.4 40.9 7.5 (5.6–9.4) 18 <0.001 Timing of initial antibiotic therapy 19.2 21.1 1.9 (1.1–2.7) 9 <0.001 Smoking cessation advice 21.8 27.9 6.0 (3.8–8.3) 22 <0.001 Initial antibiotic selection 13.6 14.3 0.7 (�0.5–1.9) 5 0.293 Composite pneumonia score 13.7 17.5 3.7 (2.8–4.6) 21 <0.001

Overall composite 12.0 16.1 4.2 (3.2–5.1) 26 <0.001

Abbreviations: ACE angiotensin-converting enzyme; AMI, acute myocardial infarction; CI, confidence interval; LV, left ventricular; PCI, percutaneous coronary intervention. * Performance calculated as the proportion of all eligible patients who received the indicated care. Change in performance estimated based on multivariate regression adjusting for baseline performance, profit status, bed size, rural setting, critical access hospital status, and region except for PCI received within 90 minutes and thrombolytic agent within 30 minutes which did not adjust for critical access hospital status. † P values and CIs calculated based on bootstrapped standard errors.

458 An Official Publication of the Society of Hospital Medicine Journal of Hospital Medicine Vol 6 | No 8 | October 2011

Schmaltz et al. | Hospital Performance Trends

overall score compared to hospitals that were not accredited. These results are consistent with other studies that have looked at both process and outcome measures and accreditation.9–12

It is important to note that the observed ‘‘accredita- tion effect’’ reflects a difference between hospitals that have elected to seek one particular ‘‘self-regulatory al- ternative to the more restrictive and extensive public regulatory or licensure requirements’’ with those that have not.39 The non-accredited hospitals that were included in this study are not considered to be ‘‘sub- standard hospitals.’’ In fact, hospitals not accredited by The Joint Commission have also met the standards set by Medicare in the Conditions of Participation, and our study demonstrates that these hospitals achieved reasonably strong performance on publicly reported quality measures (86.8% adherence on the composite measure in 2008) and considerable improvement over the 5 years of public reporting (av- erage improvement on composite measure from 2004 to 2008 of 11.8%). Moreover, there are many paths to improvement, and some non-accredited hospitals achieve stellar performance on quality measures, per- haps by embracing other methods to catalyze improvement. That said, our data demonstrate that, on average,

accredited hospitals achieve superior performance on these evidence-based quality measures, and their per- formance improved more strikingly over time. In

interpreting these results, it is important to recognize that, while Joint Commission-accredited hospitals must report quality data, performance on these meas- ures is not directly factored into the accreditation de- cision; if this were not so, one could argue that this association is a statistical tautology. As it is, we believe that the 2 measures (accreditation and publicly reported quality measures) are two independent assessments of the quality of an organization, and, while the performance measures may not be a ‘‘gold standard,’’ a measure of their association does provide useful information about the degree to which accredi- tation is linked to organizational quality. There are several potential limitations of the current

study. First, while we adjusted for most of the known hospital demographic and organizational factors asso- ciated with performance, there may be unidentified factors that are associated with both accreditation and performance. This may not be relevant to a patient or payer choosing a hospital based on accreditation sta- tus (who may not care whether accreditation is simply associated with higher quality or actually helps pro- duce such quality), but it is relevant to policy-makers, who may weigh the value of embracing accreditation versus other maneuvers (such as pay for performance or new educational requirements) as a vehicle to pro- mote high-quality care. A second limitation is that the specification of the

measures can change over time due to the acquisition

TABLE 4. Percent of Hospitals With High Performance* in 2008 by Joint Commission Accreditation Status

Characteristic

Percent of Hospitals with Performance

Over 90%† Adjusted (Actual) Odds Ratio, Always vs

Never (95% CI)† P Value†Never Accredited (n ¼ 762) Always Accredited (n ¼ 2,917)

AMI Aspirin at admission 91.8 (71.8) 93.9 (90.7) 1.38 (1.00–1.89) 0.049 Aspirin at discharge 83.7 (69.2) 88.2 (85.1) 1.45 (1.08–1.94) 0.013 ACE inhibitor for LV dysfunction 65.1 (65.8) 77.2 (76.5) 1.81 (1.32–2.50) <0.001 Beta blocker at discharge 84.7 (69.4) 90.9 (88.4) 1.80 (1.33–2.44) <0.001 Smoking cessation advice 91.1 (81.3) 95.9 (94.1) 2.29 (1.31–4.01) 0.004 PCI received within 90 min 21.5 (16.2) 29.9 (29.8) 1.56 (0.71–3.40) 0.265 Thrombolytic agent within 30 min 21.4 (21.3) 22.7 (23.6) 1.08 (0.42–2.74) 0.879 Composite AMI score 80.5 (56.6) 88.2 (85.9) 1.82 (1.37–2.41) <0.001

Heart failure Discharge instructions 27.0 (26.3) 38.9 (39.3) 1.72 (1.30–2.27) <0.001 Assessment of LV function 76.2 (45.0) 89.1 (88.8) 2.54 (1.95–3.31) <0.001 ACE inhibitor for LV dysfunction 58.0 (51.4) 67.8 (68.5) 1.52 (1.21–1.92) <0.001 Smoking cessation advice 84.2 (62.3) 90.3 (89.2) 1.76 (1.28–2.43) <0.001 Composite heart failure score 38.2 (27.6) 61.5 (64.6) 2.57 (2.03–3.26) <0.001

Pneumonia Oxygenation assessment 100 (98.2) 100 (99.8) 4.38 (1.20–1.32) 0.025 Pneumococcal vaccination 44.1 (40.3) 57.3 (58.2) 1.70 (1.36–2.12) <0.001 Timing of initial antibiotic therapy 74.3 (79.1) 84.2 (82.7) 1.85 (1.40–2.46) <0.001 Smoking cessation advice 76.2 (54.6) 85.8 (84.2) 1.89 (1.42–2.51) <0.001 Initial antibiotic selection 51.8 (47.4) 51.0 (51.8) 0.97 (0.76–1.25) 0.826 Composite pneumonia score 69.3 (59.4) 85.3 (83.9) 2.58 (2.01–3.31) <0.001

Overall composite 69.0 (47.5) 83.8 (82.0) 2.32 (1.76–3.06) <0.001

Abbreviations: ACE angiotensin-converting enzyme; AMI, acute myocardial infarction; CI, confidence interval; LV, left ventricular; PCI, percutaneous coronary intervention. * High performance defined as performance rates of 90% or more. † Performance calculated as the proportion of all eligible patients who received the indicated care. Percent of hospitals with performance over 90% estimated based on multivariate logistic regression adjusting for baseline performance, profit status, bed size, rural setting, critical access hospital status, and region except for PCI received within 90 minutes and thrombolytic agent within 30 minutes which did not adjust for critical access hos- pital status. Odds ratios, CIs, and P values based on the logistic regression analysis.

An Official Publication of the Society of Hospital Medicine Journal of Hospital Medicine Vol 6 | No 8 | October 2011 459

Hospital Performance Trends | Schmaltz et al.

of new clinical knowledge, which makes longitudinal comparison and tracking of results over time difficult. There were two measures that had definitional changes that had noticeable impact on longitudinal trends: the AMI measure ‘‘Primary Percutaneous Cor- onary Intervention (PCI) Received within 90 Minutes of Hospital Arrival’’ (which in 2004 and 2005 used 120 minutes as the threshold), and the pneumonia measure ‘‘Antibiotic Within 4 Hours of Arrival’’ (which in 2007 changed the threshold to six hours). Other changes included adding angiotensin-receptor blocker therapy (ARB) as an alternative to angioten- sin-converting enzyme inhibitor (ACEI) therapy in 2005 to the AMI and heart failure measures ACEI or ARB for left ventricular dysfunction. Other less signif- icant changes have been made to the data collection methods for other measures, which could impact the interpretation of changes in performance over time. That said, these changes influenced both accredited and non-accredited hospitals equally, and we cannot think of reasons that they would have created differ- ential impacts. Another limitation is that the 16 process measures

provide a limited picture of hospital performance.

Although the three conditions in the study account for over 15% of Medicare admissions,19 it is possible that non-accredited hospitals performed as well as accred-

ited hospitals on other measures of quality that were not captured by the 16 measures. As more standar-

dized measures are added to The Joint Commission and CMS databases, it will be possible to use the

same study methodology to incorporate these addi- tional domains. From the original cohort of 4798 hospitals reporting

in 2004 or 2008, 19% were not included in the study due to missing data in either 2004 or 2008. Almost two-thirds of the hospitals excluded from the study were missing 2004 data and, of these, 77% were criti- cal access hospitals. The majority of these critical access hospitals (97%) were non-accredited. This is in contrast to the hospitals missing 2008 data, of which only 13% were critical access. Since reporting of data to Hospital Compare was voluntary in 2004, it appears that critical access hospitals chose to wait later to report data to Hospital Compare, compared to acute care hospitals. Since critical access hospitals tended to have lower rates, smaller sample sizes, and be non-accredited, the results of the study would be expected to slightly underestimate the difference between accredited and non-accredited hospitals. Finally, while we have argued that the publicly

reported quality measures and TJC accreditation deci- sions provide different lenses into the quality of a given hospital, we cannot entirely exclude the possibil- ity that there are subtle relationships between these two methods that might be partly responsible for our findings. For example, while performance measure

rates do not factor directly into the accreditation deci- sion, it is possible that Joint Commission surveyors may be influenced by their knowledge of these rates and biased in their scoring of unrelated standards dur- ing the survey process. While we cannot rule out such biases, we are aware of no research on the subject, and have no reason to believe that such biases may have confounded the analysis. In summary, we found that Joint Commission-

accredited hospitals outperformed non-accredited hos- pitals on nationally standardized quality measures of AMI, heart failure, and pneumonia. The performance gap between Joint Commission-accredited and non- accredited hospitals increased over the five years of the study. Future studies should incorporate more ro- bust and varied measures of quality as outcomes, and seek to examine the nature of the observed relation- ship (ie, whether accreditation is simply a marker of higher quality and more rapid improvement, or the accreditation process actually helps create these salu- tary outcomes).

Acknowledgements The authors thank Barbara Braun, PhD and Nicole Wineman, MPH, MBA for their literature review on the impact of accreditation, and Bar- bara Braun, PhD for her thoughtful review of the manuscript.

Disclosure: Nothing to report.

References 1. The Joint Commission. Facts About Hospital Accreditation. Avail-

able at: http://www.jointcommission.org/assets/1/18/Hospital_Ac- creditation_1_ 31_11.pdf. Accessed on Feb 16, 2011.

2. Niska RW, Burt CW. Emergency Response Planning in Hospitals, United States: 2003–2004. Advance Data from Vital and Health Sta- tistics; No. 391. Hattsville, MD: National Center for Health Statis- tics; 2007.

3. Niska RW, Burt CW. Training for Terrorism-Related Conditions in Hospitals, United States: 2003–2004. Advance Data from Vital and Health Statistics; No. 380. Hattsville, MD: National Center for Health Statistics; 2006.

4. Longo DR, Hewett JE, Ge B, Shubert S. Hospital patient safety: char- acteristics of best-performing hospitals. J Healthcare Manag. 2007;52 (3):188–205.

5. Devers KJ, Pham HH, Liu G. What is driving hospitals’ patient-safety efforts? Health Aff. 2004;23(2):103–115.

6. DeBritz JN, Pollak AN. The impact of trauma centre accreditation on patient outcome. Injury. 2006;37(12):1166–1171.

7. Lemak CH, Alexander JA. Factors that influence staffing of outpa- tient substance abuse treatment programs. Psychiatr Serv. 2005;56(8) 934–939.

8. D’Aunno T, Pollack HA. Changes in methadone treatment practices. Results from a national panel study, 1988–2000. JAMA. 2002;288: 850–856.

9. Landon BE, Normand ST, Lesser A, et al. Quality of care for the treatment of acute medical conditions in US hospitals. Arch Intern Med. 2006;166:2511–2517.

10. Chen J, Rathore S, Radford M, Krumholz H. JCAHO accreditation and quality of care for acute myocardial infarction. Health Aff. 2003; 22(2):243–254.

11. Morlock L, Pronovost P, Engineer L, et al. Is JCAHO Accreditation Associated with Better Patient Outcomes in Rural Hospitals? Acad- emy Health Annual Meeting; Boston, MA; June 2005.

12. Joshi MS. Hospital quality of care: the link between accreditation and mortality. J Clin Outcomes Manag. 2003;10(9):473–480.

13. Griffith JR, Knutzen SR, Alexander JA. Structural versus outcome measures in hospitals: A comparison of Joint Commission and medi- care outcome scores in hospitals. Qual Manage Health Care. 2002;10(2): 29–38.

14. Barker KN, Flynn EA, Pepper GA, Bates D, Mikeal RL. Medication errors observed in 36 health care facilities. Arch Intern Med. 2002; 162:1897–1903.

15. Menachemi N, Chukmaitov A, Brown LS, Saunders C, Brooks RG. Quality of care in accredited and non-accredited ambulatory surgical centers. Jt Comm J Qual Patient Saf. 2008;34(9):546–551.

460 An Official Publication of the Society of Hospital Medicine Journal of Hospital Medicine Vol 6 | No 8 | October 2011

Schmaltz et al. | Hospital Performance Trends

16. Joint Commission on Accreditation of Healthcare Organizations. Specification Manual for National Hospital Quality Measures 2009. Available at: http://www.jointcommission.org/PerformanceMeasure- ment/PerformanceMeasurement/CurrentþNHQMþManual.htm. Accessed May 21, 2009.

17. Hospital Quality Alliance Homepage. Available at: http://www.hospi- talqualityalliance.org/hospitalqualityalliance/index.html. Accessed May 6, 2010

18. Williams SC, Schmaltz SP, Morton DJ, Koss RG, Loeb JM. Quality of care in U.S. hospitals as reflected by standardized measures, 2002– 2004. N Engl J Med. 2005;353(3):255–264.

19. Jha AK, Li Z, Orav EJ, Epstein AM. Care in U.S. hospitals—the Hos- pital Quality Alliance Program. N Engl J Med. 2005;353:265–274.

20. Institute of Medicine, Committee on Quality Health Care in America. Crossing the Quality Chasm: A New Health System for the 21st Cen- tury. Washington, DC: The National Academy Press; 2001.

21. Hibbard JH, Stockard J, Tusler M. Does publicizing hospital per- formance stimulate quality improvement efforts? Health Aff. 2003; 22(2):84–94.

22. Williams SC, Morton DJ, Koss RG, Loeb JM. Performance of top ranked heart care hospitals on evidence-based process measures. Cir- culation. 2006;114:558–564.

23. The Joint Commission Performance Measure Initiatives Homepage. Available at: http://www.jointcommission.org/PerformanceMeasure- ment/PerformanceMeasurement/default.htm. Accessed on July 27, 2010.

24. Palmer RH. Using health outcomes data to compare plans, networks and providers. Int J Qual Health Care. 1998;10(6):477–483.

25. Mant J. Process versus outcome indicators in the assessment of qual- ity of health care. Int J Qual Health Care. 2001;13:475–480.

26. Chassin MR. Does paying for performance improve the quality of health care? Med Care Res Rev. 2006;63(1):122S–125S.

27. Kfoury AG, French TK, Horne BD, et al. Incremental survival benefit with adherence to standardized health failure core measures: a per- formance evaluation study of 2958 patients. J Card Fail. 2008;14(2): 95–102.

28. Jha AK, Orav EJ, Li Z, Epstein AM. The inverse relationship between mortality rates and performance in the hospital quality alliance meas- ures. Health Aff. 2007;26(4):1104–1110.

29. Bradley EH, Herrin J, Elbel B, et al. Hospital quality for acute myo- cardial infarction: correlation among process measures and relation- ship with short-term mortality. JAMA. 2006:296(1):72–78.

30. Williams SC, Watt A, Schmaltz SP, Koss RG, Loeb JM. Assessing the reliability of standardized performance measures. Int J Qual Health Care. 2006;18:246–255.

31. Centers for Medicare and Medicaid Services (CMS). CMS HQI De- monstration Project-Composite Quality Score Methodology Over- view. Available at: http://www.cms.hhs.gov/HospitalQualityInits/ downloads/HospitalCompositeQualityScoreMethodologyOver- view.pdf. Accessed March 8, 2010.

32. Normand SLT, Wolf RE, Ayanian JZ, McNeil BJ. Assessing the accu- racy of hospital performance measures. Med Decis Making. 2007;27: 9–20.

33. Quality Check Data Download Website. Available at: http:// www.healthcarequalitydata.org. Accessed May 21, 2009.

34. Hartz AJ, Krakauer H, Kuhn EM. Hospital characteristics and mor- tality rates. N Engl J Med. 1989;321(25):1720–1725.

35. Goldman LE, Dudley RA. United States rural hospital quality in the Hospital Compare Database—accounting for hospital characteristics. Health Policy. 2008;87:112–127.

36. Lehrman WG, Elliott MN, Goldstein E, Beckett MK, Klein DJ, Gior- dano LA. Characteristics of hospitals demonstrating superior per- formance in patient experience and clinical process measures of care. Med Care Res Rev. 2010;67(1):38–55.

37. Werner RM, Goldman LE, Dudley RA. Comparison of change in quality of care between safety-net and non-safety-net hospitals. JAMA. 2008;299(18):2180–2187.

38. Davison AC, Hinkley DV. Bootstrap Methods and Their Application. New York: Cambridge; 1997: chap 6.

39. Pawlson LF, Torda P, Roski J, O’Kane ME. The role of accreditation in an era of market-driven accountability. Am J Manag Care. 2005; 11(5):290–293.

An Official Publication of the Society of Hospital Medicine Journal of Hospital Medicine Vol 6 | No 8 | October 2011 461

Hospital Performance Trends | Schmaltz et al.