HCA 448 Case 1

International Journal for Quality in Health Care 2001; Volume 13, Number 6: pp. 475–480

Process versus outcome indicators in the assessment of quality of health care JONATHAN MANT

Department of Primary Care & General Practice, Medical School, University of Birmingham, Birmingham, UK

Abstract

This paper reviews the relative strengths and weaknesses of outcome and process measures as performance indicators in health care. Differences in outcome may be due to case mix, how the data were collected, chance, or quality of care. Health care is only one determinant of health and other factors have important effects on health outcomes, such as nutrition, environment, lifestyle and poverty. The advantages of process measures are that they are more sensitive to differences in the quality of care and they are direct measures of quality. However, outcome measures are of greater intrinsic interest and can reflect all aspects of care, including those that are otherwise difficult to measure such as technical expertise and operator skill. Outcome indicators can be improved if efforts are made to standardize data collection and case mix adjustment systems are developed and validated. It is argued that this is worth doing only where it is likely that variations in health care might lead to significant variations in health outcome and where the occurrence of the outcome is sufficiently common that the outcome indicator will have the power to detect real differences in quality. If these conditions are not met, then alternative strategies such as process measurement and risk management techniques may be more effective at protecting the public from poor quality care.

Key words: clinical competence, medical audit, outcome and process measurement (health care), quality assurance

to consider in which circumstances it might be appropriateHigh-profile failures of health care in the UK have kept the to use outcome measures to monitor health care. In orderspotlight on methods to monitor the quality of health services. to do this, it is important to consider the different contextsHot on the heels of the ‘Bristol case’, a ‘once in a lifetime in which this debate is taking place.drama’ [1] in which three doctors were found guilty by the

General Medical Council of ‘serious professional misconduct’ A question of perspectivein relation to the deaths of 29 babies and young children

undergoing paediatric cardiac surgery, came the ‘Shipman An important contextual issue is the purpose for which the case’ [2]. Shipman was a British General Practitioner who performance indicator is to be used and by whom. The aim was convicted of murdering 15 of his patients, although he of using performance indicators may be to: is suspected of murdering many more. A recent analysis suggests that monitoring outcome could have identified sig- inform policy making or strategy at a regional or national nificant deviations from expected mortality rates in both level, these examples [3]. Others have argued that taking such an improve the quality of care of a health care facility, approach might distract from more mundane methods that monitor performance of health care funders such as HMOs, nevertheless may be more effective at protecting patients identify poor performers to protect public safety against poor care [4]. The debate over what are the most provide consumer information to facilitate choice of health useful types of performance indicator to monitor the quality care provider of health care is of international concern [5]. It is simplistic to view process and outcome measures as being in com- At the national level, performance indicators can be used

in two ways; to compare performance between countries andpetition with each other, but there are circumstances where one type of measure is likely to be more useful than the to inform policy within a country. The summary measure of

population health developed by the World Health Or-other. The aim of this paper is to review the relative strengths and weaknesses of outcome measures compared with process ganization, disability-adjusted life expectancy (DALE), allows

direct comparison of the health of countries, which can raisemeasures as performance indicators for health care and so

Address reprint requests to J. Mant, Department of Primary Care & General Practice, Medical School, University of Birmingham, Birmingham, B15 2TT, UK, E-mail: [email protected]

 2001 International Society for Quality in Health Care and Oxford University Press 475

J. Mant

important health policy questions, though methodological the type of patient that are cared for by the different providers. concerns currently limit the usability of the data [6,7]. Within Factors such as age, gender, co-morbidity, severity of disease a country, use of population outcome data can help formulate and socio-economic status would come under consideration and monitor health policy. Health care is only one determinant here. The importance of this cause of variation is illustrated of health. Historically, other factors, such as nutrition, en- by studies where differences in crude outcome rates disappear vironment, lifestyle, poverty and social structure of society when the rates are adjusted to take account of these con- have been demonstrated to have powerful effects on health founding factors. For example, the introduction of a stroke as measured by mortality rates [8,9]. Contemporary data unit in Edinburgh, Scotland, was associated with a 40% suggest that these factors continue to play an important role. reduction in mortality of stroke patients admitted to hospital. For example, changes in lifestyle, in particular diet and However, this difference disappeared once the results were smoking, appear to account for two-thirds of the 31% decline adjusted for case mix [16]. in coronary heart disease that occurred between 1980 and A second cause of variation in outcome is differences in 1992 in US women [10]. Similarly, changes in the prevalence the way data are collected. An outcome indicator used as a of risk factors appear to have accounted for half of the performance indicator has different component parts. For reduction in coronary heart disease mortality that has been example, if the outcome measure is a dichotomous variable observed in recent years in countries such as Scotland and such as death, these comprise a numerator, a denominator New Zealand [11,12]. Setting mortality targets can encourage and case-mix data used to adjust for potential confounding. policy makers and government to consider policies and Differences in the measurement of any of these will lead to actions outside the direct confines of the health care system apparent differences in outcome. For example, an important that might have an impact on health. For example, the factor in explaining an observed decline in case-mix adjusted mortality targets set by the UK government for diseases such mortality from cardiac surgery in New York State is an as heart disease, stroke and cancer were accompanied by an apparent increase in the prevalence of risk factors in the explicit inter-sectoral strategy for health. This emphasized patients upon whom surgery was performed [17]. Between the importance of non-health care factors as diverse as the 1989 and 1991, there was an increase in the reported pre- restriction of sale of cigarettes to underage smokers, the cost valence of renal failure, congestive heart failure, chronic of fruit and vegetables and unemployment [13]. obstructive pulmonary disease and unstable angina in patients

A narrower perspective is obtained if one takes as one’s undergoing cardiac surgery. These increases are likely to have population of interest those who are admitted to hospital. reflected changes in how these risk factors were ascertained, For example, 70% of deaths from acute myocardial infarction rather than genuine changes in case mix. If this is the case, occur in people who never reach hospital [14]. This moves then this would account for 40% of the observed reduction the agenda from disease prevention (whether by agencies in case-mix adjusted mortality. within or outside the health service) to disease treatment. Thirdly, observed differences may be due to chance. For This is relevant to bodies such as Health Maintenance Or- example, in the European Carotid Surgery Trial, the risk of ganizations (HMOs) in the US or primary care groups and major stroke or death within 30 days of carotid en- trusts in the UK who, as part of their remit, need to monitor

darterectomy was 7% (95% confidence interval, 5.8–8.3). the quality of the hospital services that they procure. This is

One hundred and forty-seven surgeons took part in this trial also an appropriate breadth of perspective for health care

and surgeon-specific mortality was in the range of 0–50%. providers seeking to improve the quality of their own care

However, because of the small number of patients included through such mechanisms as clinical audit. This perspective

in the trial by each surgeon, one cannot conclude either that is also of relevance to the public, whether to promote market

the surgeons with no operative strokes or deaths were better competition in privately funded health services or public

than the rest, or that the surgeon with a 50% complication accountability of state-financed health care [15].

rate was worse, as these differences might have arisen by In general, the broader the perspective required, the greater

chance [18]. Random variation is influenced both by number the relevance of outcome measures, since they reflect the

of cases included and the frequency with which the outcome inter-play of a wide variety of factors, some directly related

occurs. For example, to detect a 30% difference in outcometo health care, others not. As the perspective narrows, to between two units performing carotid endarterectomy withhospitals, or to departments, or indeed to individual doctors, 80% power at a significance level of 5%, with one unitoutcome measures become relatively less and process meas- achieving a 7% death and complication rate and anotherures relatively more useful. In order to consider why this is unit a 10% rate, would require the audit of 1422 carotidthe case, it is necessary to explore the possible causes of endarterectomies in each unit. Given that hospitals in Newvariation in outcome between health care providers. York State each performed on average 50 carotid en- darterectomies a year in 1995 [19], such a difference is unlikely

What causes variation in outcome between to be detected. Conversely, to detect a 30% difference in

health care providers? outcome between units treating patients with myocardial infarction, with one unit experiencing a 30% mortality andThere are four major categories of explanation that need to the other unit a 21% mortality, would only require the auditbe considered, as shown in Table 1. The first of these is

whether observed differences might be due to differences in of 389 patients in each unit [20]. Because this number of

476

Process or outcome indicators

Table 1 Reasons for variations in outcome

Category of explanation Sources of variation ........................................................................................................................................

Differences in type of patient Confounding by patient characteristics

Differences in measurement Ascertainment and definition of cases, outcome and risk factors

Chance Random variation, influenced by number of cases and frequency with which outcome occurs

Differences in quality of care Use of proven interventions

patients would be admitted in less than a year to the typical to obtain data on the care of 48 patients in each hospital to detect a significant difference in the process of care [20].hospital, detecting such a difference is feasible.

Finally, differences in outcome may reflect genuine dif- Similarly, one cannot conclude that the lack of difference in mortality of patients admitted to hospital with stroke inferences in quality of care. This may be due to variations in

the use of interventions, such as use of aspirin and beta- Edinburgh after the introduction of a stroke unit reflects a failure of the stroke unit [16]. To be confident that the strokeblockers in acute myocardial infarction [21], or less measurable

aspects such as the skill of a surgeon or of a cardiac arrest unit had failed in bringing about a reduction in mortality that would be consistent with the trial evidence would haveteam. It is these possible causes of variation that are of

relevance if an outcome measure is to be used as a per- required a 10 year study [22]. A second advantage of process measures is that they areformance indicator. In this circumstance, the conclusion that

a variation in outcome is due to a difference in quality of easy to interpret. A process measure such as use of aspirin in acute myocardial infarction is a direct measure of quality,care is essentially a diagnosis of exclusion. If one cannot

explain the variation in terms of differences in the type of whereas hospital-specific mortality from myocardial infarction is only an indirect measure. As discussed above, if differencespatient, in how the data were collected, or in terms of chance,

then quality of care becomes a possible explanation. However, in outcome are observed, then alternative explanations need to be considered before one can conclude that the differencethe conclusion that differences in outcome are due to dif-

ferences in quality of care will always be tentative and open reflects true variations in the quality of care. Conversely, a process measure is straightforward to interpret: the moreto the possibility that the apparent association between a

given unit and poor outcome is due to confounding by some people without contra-indications who receive a proven therapy, the better. A consequence of this is that the necessaryother factor that was not measured, or measured inadequately.

These alternative explanations of variation limit the value remedial action is clearer (use the therapy more often). However, if one does conclude that a higher mortality rateof outcome measures as performance indicators of health

care. Process measures have two important advantages over is due to poor quality of care, it is not immediately obvious what action needs to be taken, unless perhaps audits ofoutcome measures, as discussed below. the process of care have also been undertaken in parallel. Depending upon perspective (see above), this may not be aAdvantages of process measures problem. If the information is to be used by purchasers or

An important consideration is the extent to which an indicator consumers of health care to influence choice of provider

will detect a genuine difference in quality. An intrinsic ad- (and so improve market efficiency), then the underlying cause

vantage of process measures is that they are more sensitive of the differences in quality of care is arguably of less

than outcome measures to differences in the quality of care. importance.

For example, if real differences in the quality of care arising Outcome measures do, however, have a role in the moni-

from differential use of proven interventions between two toring of the quality of health care, in that they do have

hospitals treating patients with myocardial infarction resulted some intrinsic advantages.

in a 10% relative difference in hospital-specific mortality rates (30% versus 27%), it would require following up 3619 patients

Advantages of outcome measures in each hospital to detect such a mortality difference and be confident that it was not due to chance (with 80% power at One attraction of outcome measurement is that it is a measure

of something that is important in its own right. It is interestinga significance level of 5%) [20]. In a typical UK hospital, this would need 8 years of data collection. However, if one to know that the death rate from myocardial infarction

varies from hospital to hospital, even if the reasons for themeasured the uptake of the proven interventions that would lead to such a mortality difference, it would only be necessary differences have nothing to do with the quality of care. A

477

J. Mant

process measure is only of value if it is assumed to have a case mix data to allow risk-adjusted rates to be derived? (Given that process measures offer an easier to interpret andlink to outcome. By itself, the process measure is of little less costly alternative).intrinsic interest.

The first issue to consider is the extent to whichSecondly, outcome measurement will reflect all aspects of variations in quality of care might lead to significantthe processes of care and not simply those that are measurable variations in outcome. Taking stroke care as an example,or measured. Attributes such as technical expertise and op- a reasonable estimate of the impact that good quality careerator skill, while likely to be important determinants of might have on outcome is provided by the overview ofoutcome in some situations, cannot easily be captured as trials looking at the effect of organized inpatient care forperformance indicators. Where how you do something is as stroke [26]. This concluded that stroke unit care wasimportant as what you do, process measures will be unable associated with a 17% reduction in the odds of deathto capture the distinction [23]. Thus, where technical skill is after 1 year. Observed variations in outcome followingrelatively unimportant (e.g. giving an aspirin tablet), then the hospital admission with stroke are much greater than thisprocess measure is satisfactory. Where the technical skill is [27]. This suggests that the bulk of the observed variationimportant (e.g. performing coronary angioplasty or carotid in mortality is likely to be due to differences in case mixendarterectomy), it not only matters that the procedure is than to differences in quality of care. If large differencesperformed on the correct patients (a process measure), but in mortality persist after case mix adjustment, then thealso how well the procedure was carried out. likeliest explanation is that the case mix adjustment wasAnother possible reason why outcome indicators are often inadequate. In such circumstances, it is probably not worthused in some countries is that the data, at least to construct spending resource on developing case mix adjustmentsimple rates, are available from routine information systems. systems (and standardising the collection of all the additionalHospitals will routinely record the admission diagnosis and data that this would entail) to refine the outcome indicator,other data that can be used for simple case mix adjustment, since it is unlikely to be able to detect real differences insuch as age, gender and place of residence. In the UK, quality. However, in circumstances where one might an-process data has been less readily available in the past and ticipate that quality of care was a substantial determinanthas required retrospective case note reviews. This is not an of outcome, then outcome measures are likely to be ofissue in countries such as the US where process data are more value as an indicator of quality. For example, mortalityrequired to support the construction of charges for health care. following surgery is likely to depend to a significant extentAs the importance of process indicators becomes recognized, upon the skill and technique of the surgeon. Indeed, forprospective process data are more likely to be recorded in some operative procedures, the benefits of the surgery areother countries. For example, in the UK, the government’s outweighed by the potential harms if the complication rateNational Service Framework for Coronary Heart Disease, is high: a review of the evidence on carotid endarterectomyspecifies process measures that health care providers need concluded that it is effective only if performed by surgeonsto record and on which their performance will be who have complication rates of less than 6% [28]. In suchmonitored [24]. instances, it is likely to be of value to use case mix adjustment systems [29], since residual variation is moreWhen to use outcome indicators? likely to reflect quality of care.

The major problem with outcome measures as performance The second issue to take into account is whether the indicators are that they are not a direct measure of quality outcome indicator is likely to have the statistical power to of health care in the same way that process measures are. A detect differences in quality. Statistical power depends upon patient admitted to hospital with a heart attack may receive how common is the occurrence of the outcome. Since the atrocious care, yet despite this, is likely to survive [25]. target complication rate in carotid endarterectomy is low However, steps can be taken to minimize the possibility of (6%), monitoring outcome will only have limited ability to a false conclusion being drawn about the quality of care based detect whether an individual surgeon’s true complication on outcome measurement. Including sufficient numbers of rate is greater than 6%. For example, if a surgeon’s ‘true’ patients will reduce the possibility of random variation mask- complication rate was 8%, one would need to monitor the ing real differences or making spurious differences appear. outcome of 1200 operations to detect that this surgeon’s rate Statistical significance testing can also minimize the possibility was greater than 6% with 80% power at a significance level of the latter type of error. Standardising how data are collected of 5%. Thus, in circumstances where the expected outcome can reduce the extent to which differences in measurement is relatively rare, outcome indicators will only have limited can be the cause of observed variation. Development of power to detect real differences in quality. sophisticated case mix adjustment systems can reduce the Finally, one needs to consider whether there are any possibility that observed differences are due to differences practical alternatives to using outcome indicators in a given in the types of patient that are treated. The question is in area. If there is no evidence that a measure of process is what circumstances is it worth going to the effort and linked to outcome, then there is little justification for using expense of setting up an outcome monitoring system covering such a measure, unless perhaps if it is endorsed by a nationally adequate numbers of patients using consistent methods of credible consensus guideline. In such circumstances, outcome

measures may be most appropriate.case definition and ascertainment and including sufficient

478

Process or outcome indicators

Shipman and clinical governance: Shewhart’s forgotten lessons.Conclusion Lancet 2001; 357: 463–467.

Outcome measures are an indicator of health. They are valid 4. Frankel S, Sterne J, Smith GD. Mortality variations as a measure of general practitioner performance: implications of the Shipmanas performance indicators in as much as the quality of health case. Br Med J 2000; 320: 489.services has an impact on health [30]. In some circumstances,

the quality of health services has a relatively minor role in 5. Jencks SF, Cuerdon T, Burwen DR et al. Quality of medical determining health outcome, in other circumstances, a major care delivered to Medicare beneficiaries. J Am Med Assoc 2000; role. Where health services have major effects on outcome, use 284: 1670–1676. of outcome measures as performance indicators is appropriate

6. Mathers CD, Sadana R, Salomon JA et al. Healthy life expectancyand efforts should be taken to ensure that the data can be in 191 countries, 1999. Lancet 2001; 357: 1685–1691.

interpreted reliably (e.g. standardization of data collection and development of robust case mix adjustment systems). 7. Almeida C, Braveman P, Gold MR et al. Methodological concerns

and recommendations on policy consequences of the WorldWhere factors such as lifestyle and socio-economic cir- Health Report 2000. Lancet 2001; 357: 1692–1697.cumstance rather than health care are the major determinants

of outcome, it would be a misnomer to refer to an outcome 8. McKeown T. The Role of Medicine, 2nd edn, Oxford: Blackwell, measure as a performance indicator, since it would be acting 1979. as a broader barometer of the health of that population [31].

9. Wilkinson RG. Unhealthy Societies: the Afflictions of Inequality. Lon-This is not to say that health outcome data should not be don: Routledge, 1996.obtained (and indeed published) in such circumstances. The

purpose of collecting and publishing such data should not 10. Hu FB, Stampfer MJ, Manson JE et al. Trends in the incidence of coronary heart disease and changes in diet and lifestyle inbe to identify poorly performing hospitals, but rather to women. New Engl J Med 2000; 343: 530–537.inform upon wider aspects of health policy. It would be

inappropriate to develop complex case mix adjustment sys- 11. Capewell S, Morrison CE, McMurray JJ. Contribution of modern tems, since such adjustments will remove those very factors cardiovascular treatment and risk factor changes to the decline (e.g. smoking rates; socio-economic status) that ought to be in coronary heart disease mortality in Scotland between 1975 under scrutiny. and 1994. Heart 1999; 81: 380–386.

Process measures are direct measures of the quality of 12. Capewell S, Beaglehole R, Seddon M, McMurray J. Explanation

health care, provided that a link has been demonstrated for the decline in coronary heart disease mortality rates in

between a given process and outcome. Therefore, where such Auckland, New Zealand, between 1982 and 1993. Circulation measures are available and they are relevant and practical, 2000; 102: 1511–1516. then they should be used in preference to outcome measures

13. Secretary of State for Health. Saving lives: Our Healthier Nation.since they are much easier to interpret and are much more Cm 4386. London: The Stationery Office, 1999.sensitive to differences in the quality of care.

While it is tempting to use mortality variations as a way 14. Volmink JA, Newton JN, Hicks NR et al. Coronary event and of identifying poor performers, especially in the light of case fatality rates in an English population: results of the Oxford

myocardial infarction incidence study. Heart 1998; 80: 40–44.the Bristol case and the Shipman case in the UK, the appropriateness of such a strategy needs to be considered

15. Davies HTO, Marshall MN. Public disclosure of performance for each specific disease and operation. Alternative strategies data: does the public get what the public wants? Lancet 1999; such as process measurement or risk management techniques 353: 1639–1640. that focus on the analysis and investigation of individual

16. Davenport RJ, Dennis MS, Warlow CP. Effect of correctingincidents rather than seeking to interpret statistical variations outcome data for case mix: an example from stroke medicine.may be more effective at protecting the public from poor Br Med J 1996; 312: 1503–1505.

quality care [32]. 17. Green J, Wintfield N. Report cards on cardiac surgeons –

assessing New York State’s approach. N Engl J Med 1995; 332: 1229–1232.Acknowledgements

18. Rothwell PM, Warlow CP for the European Carotid Surgery Trialists’ Group. Interpretation of operative risks of individualI would like to acknowledge the helpful comments of Richard surgeons. Lancet 1999; 353: 1325.McManus on drafts of this paper.

19. Hannan EL, Popp AJ, Tranmer B, Fuestel P, Waldman J, Shah D. Relationship between provider volume and mortality for carotid endarterectomies in New York State. Stroke 1998; 29:References 2292–2297.

1. Smith R. All changed, changed utterly. Br Med J 1998; 316: 20. Mant J, Hicks N. Detecting differences in quality of care: the

1917–1918. sensitivity of measures of process and outcome in treating acute myocardial infarction. Br Med J 1995; 311: 793–796.2. O’Neill B. Doctor as murderer. Br Med J 2000; 320: 329–330.

3. Mohammed MA, Cheng KK, Rouse A, Marshall T. Bristol, 21. Marcinial TA, Ellerbeck EF, Radford MJ et al. Improving the

479

J. Mant

quality of care for Medicare patients with acute myocardial 27. Clinical Outcomes Working Group. Clinical Outcome Indicators. The Scottish Executive, 1999.infarction. Results from the cooperative cardiovascular project.

J Am Med Assoc 1998; 279: 1351–1357. 28. Cina CS, Clase CM, Haynes RB. Carotid endarterectomy for

symptomatic carotid stenosis (Cochrane Review). In the22. Mant J, Hicks NR, Fletcher J. Study should have had more Cochrane Library, 1, 2001. Update Software: Oxford.patients or longer time scale. Br Med J 1996; 313: 1006.

29. Rothwell PM, Slattery J, Warlow CP. A systematic review of 23. Mant J, Hicks NR. Assessing quality of care: what are the clinical and angiographic predictors of stroke and death due to

implications of the potential lack of sensitivity of outcome carotid endarterectomy. Br Med J 1997; 315: 1571–1577. measures to differences in quality? J Eval Clin Pract 1996; 2:

30. Bunker JP. Medicine matters after all. J R Coll Physicians Lond243–248. 1995; 29: 105–112.

24. Department of Health. National Service Framework for Coronary 31. Giuffrida A, Gravelle H, Roland M. Measuring quality of care

Heart Disease. London: Stationery Office, 2000. with routine data: avoiding confusion between performance indicators and health outcomes. Br Med J 1999; 319: 94–98.25. Brook RH, McGlynn EA, Shekelle PG. Defining and measuring

quality of care: a perspective form US researchers. Int J Qual 32. Mant J, Gatherer A. Managing clinical risk. Br Med J 1994; 308: Health Care 2000; 12: 281–295. 1522–1523.

26. Stroke Unit Trialists’ Collaboration. Organized inpatient (stroke unit) care for stroke (Cochrane Review). In The Cochrane Library: 1, 2001. Update Software: Oxford. Accepted for publication 3 July 2001

480