Week 3 Presentation

: • < ; • • • . n

From Outcomes Research to Disease Management: A Guide for the Perplexed Robert S. Epstein, MD, MS, and Louis M. Sherwood, MD

Outcomes research is a rapidly evolving field that incorpo- rates epidemiology, health services research, health eco- nomics, and psychometrics. Measurement of clinical and other outcomes has become increasingly important to the stakeholders in a rapidly changing health care environ- ment. The desire to improve outcomes and control costs has stimulated greater interest in cost-effectiveness stud- ies, which determine how well effective therapies work in the usual practice setting and how much they cost. The application of outcomes principles to the practices of health care providers has resulted in efforts to implement disease management programs. Unlike traditional pro- grams carried out by physicians, these new efforts are based on systematic population-based approaches to iden- tifying persons at risk, intervening with specific programs of care, and measuring clinical and other outcomes. The new efforts depend heavily on modern information systems.

AnnlntemMed 1996;124;832-837.

From Merck & Co., West Point, Pennsylvania. For current au- thor addresses, see end of text.

The well-being of the soul can be obtained onlyafter that of the body has been secured," said Maimonides in his classical treatise (1). This early comment predates the current appreciation of the effect of physical health on quality of life. Out- comes research has expanded dramatically in re- cent years (2-5) through multidisciplinary efforts involving health services researchers, epidemiolo- gists, economists, sociologists, statisticians, and eth- icists. Driving this interest are concerns about rap- idly increasing health care costs, growing consumer involvement in medical decision making, and incor- poration of information systems into clinical medi- cine. The health care industry has also increased its focus on efficiency and productivity, with particular emphasis on outcomes.

The latest concept in this field, disease manage- ment, refers to the use of an explicit systematic population-based approach to identify persons at risk, intervene with specific programs of care, and measure clinical and other outcomes. Disease man- agement is not new; for centuries, clinicians have been identifying and treating patients using infor- mation gleaned from their training, the medical lit- erature, and personal experience. What is new about the current model of disease management is that it

does much more than evaluate known practices and suggest guidelines. It is a systematic information- driven approach in which clinical encounters are computerized, summarized, and shared; opportuni- ties for intervention are not always detected solely at the bedside, but sometimes from a central data- base; practice guidelines may be implemented by computer screens in the practice setting; and initi- atives are measured and compared in terms of health outcomes.

At the core of disease management are principles described as outcomes research, an evolving and somewhat embryonic field. Confusion has arisen over terminology and the application of principles to the management of patients. We identify core concepts and highlight some issues that become ap- parent as outcomes principles are applied to practice.

Measurement of Outcomes

An important aspect of disease management is the abilify to measure and report health outcomes. The movement to collect this information has re- cently escalated, stimulated by federal initiatives to compare mortalify among hospitals (6, 7), by com- parative outcomes studies (8), and by payers seeking to improve qualify and reduce costs (9). Donabe- dian (10-12) described qualify health care as an optimal triad of structure, process, and outcomes. Anderson and colleagues (13) extended this model by suggesting that relations among these factors should guide decision making. The outcome mea- sures have evolved from simple dichotomous ones such as survival or occurrence of a clinical event to patient-oriented measures such as satisfaction, qual- ify of life, and functional status.

The term "outcomes" has been linked to differ- ent measures, ranging from physiologic values to qualify of life assessments. One approach to taxon- omy is shown in Table 1. The reporting of outcomes may need to differ, depending on the priorities of

See the related article on pp 838-842.

832 ©1996 American College of Physicians

Table 1. Types of Outcome Measures and Selected Examples

Outcomes

Clinical

Economic

Humanistic

lypes

Clinical events Physiologic and metabolic measures Mortality

Direct medical Indirect medical

Symptoms Quality of life Functional status Patient satisfaction

Examples

Myocardial infarctions, cerebrovascular accidents Blood pressure, measurement of cholesterol levels Death—specific cause (such as cardiovascular) or all causes (total)

Hospitalizations, outpatient visits, emergency department visits Work loss, restricted activity days

American Urological Association symptonn score SF-36 Questionnaire, Nottingham Health Profile Katz Activities of Daily Living scale, Karnofsky index Group Health Association of America survey

those examining the data. A elinieian is likely to be primarily interested in clinieal or humanistic out- eomes, whereas a health plan administrator may be interested in eeonomie ones. Patients are interested in humanistic outcomes and have trouble interpret- ing clinical outeomes. As elinieians' finaneial risk inereases, they are also developing greater interest in eeonomie outeomes. Current disease manage- ment programs inelude mixtures of all measures.

Several points should be emphasized as these measures gain broader application. Selection of spe- cific measures is currently a somewhat arbitrary pro- cess. For asthma, for example, several disease-spe- cific measures of quality of life are available (14- 17). One way to distinguish among these measures is to examine the evidence behind the development of each measure: Has each one been validated? This is now a standardized process of item genera- tion, item reduction, reproducibility, responsiveness, and validation (18, 19); however, multiple question- naires can meet these criteria and still leave the clinician confused.

Many outcomes measures were developed for use in population surveys or clinical trials and not for use in monitoring individual patients. The within- person variability ean be great, but it is often offset with aggregate data; thus, in the eontext of disease management, some measures must be used with care. Even when measures are aggregated, statistical power may be insuffieient to deteet a signifieant im- provement in outcomes. For example, in a modestly sized health plan with 25 000 members, one might estimate that 150 persons have diabetes and use insulin. Even if all 150 participated in a program that reduced the number of diabetes-related com- plications by 50% through improved diet, exercise, and appropriate use of insulin, the statistical power would be too low to document the program value compared with complication rates with a previous program. Thus, not all outeomes ean be collected on all populations; if outcomes are measured, sta- tistical issues need sufficient attention.

Finally, the interpretation of the change in out- comes is not yet clear for all measures. For exam-

ple, what is the meaning of a 4-point change on a 100-point quality-of-life scale? Is a $0.50 inerease in the per-member per-month charge enough to war- rant a disease management program? Researchers attempting to define elinical significance are begin- ning to address these issues by anchoring degree of change to other events (20-24). For example. Brook and colleagues (25) noted that a three-point differ- ence on a standardized mental health scale was analogous to being laid off from a job. Considerable additional researeh into the meaning and appliea- tion of outcomes is needed.

Effectiveness Studies

The distinction between effectiveness and efficacy is critical to disease management; treatments with proven effieaey do not always perform as well under eonditions of typieal clinical practice (effectiveness). In most efficacy studies, highly specialized praetitio- ners treat seleeted patients on tight protocols in a setting in which both patients and practitioners have economic incentives to comply. In effectiveness stud- ies, on the other hand, treatments can be viewed as they would be in usual settings (Table 2) (26-28).

Most therapies require evaluation of efficacy be- fore being implemented. This evaluation generally requires a priori hypotheses, randomization (to elitni- nate selection bias and confounding), homogeneous patients at high risk for the outcome, experienced investigators who follow a protocol, a comparative measure sueh as plaeebo (if ethical), and intensive follow-up to ensure eomplianee. Under these cir- cumstances, if a treatment proves to be better than placebo (or a comparative measure), one can be reassured that the treatment can work.

However, questions may remain about the ability of the treatment to work adequately in a broader range of patients and in usual practice settings in which both patients and providers face natural bar- riers to eare. These issues are of eentral importanee in assessing relative cost-effectiveness in the com- munity. To address these questions, most effective-

1 May 1996 • Annals of Internal Medicine • Volume 124 • Number 9 833

Table 2. Effectiveness Studies Compared with Efficacy Studies*

Criterion Efficacy Studies Effectiveness Studies

Study design Patient population Patient entry Informed consent Treatment Provider Outconnes Generalizability

Randomized protocol with well-defined control group Homogeneous Tightly controlled Obtained Comparison with placebo or other measure Experienced investigator, academic medical center Efficacy through surrogate markerst Limited

Observational or protocol-driven Heterogeneous Consecutive Not obtained in past Comparison with "usual" care "Usual" caregivers Patient outcomes Broad

* Modified from Perfetto and Epstein (27). t Longer-term intervention studies may have more definitive end points (2, 3).

ness studies have been done as observational studies, wherein observed groups are not randomly assigned to therapy and neither patients nor providers gen- erally know that they are being studied (usually retrospective studies). This eliminates issues related to inclusion of special patients and providers and allows the treatment to be observed as it is typically used. Selection bias, which occurs when patients are not randomly assigned to alternative treatments, be- comes potentially problematic, and adjustment for case mix and severity of illness becomes important.

In response to this problem, prospective effec- tiveness trials are beginning to emerge (29-31). As in typical clinical trials, these trials include the de- velopment of a protocol with a priori hypotheses, recruitment of participants with informed consent, randomization to eliminate selection bias, and rou- tine collection of data at prespecified time periods. Prospective effectiveness trials differ from typical clinical trials in that they enroll heterogeneous par- ticipants, use providers more similar to those who treat the disease, impose few protocol-driven inter- ventions, and incorporate outcome measures rele- vant to the disease and delivery system. Such trials attempt to provide unbiased information in relation to selection factors while still addressing outeomes and effectiveness.

In disease management, clinical trials are not always possible. When a health plan has developed a new clinical program, members may not feel com- fortable having an intervention withheld, and em- ployers may be unwilling to conduct a trial before disseminating the program. However, if the program is implemented without a concurrent comparison group, what remains is a before-and-after comparison; within a rapidly changing health care environment, the aging of the population and other factors can con- found results. This issue is an especially important methodologic challenge of program evaluation.

Health Economic Evaluation

The types of studies that are considered to be economic evaluations are shown in Table 3. Of

these, cost-effectiveness and cost-utility analyses overlap with outcomes research because economic measures are frequently related to outcomes such as mortality, quality of life, or value assigned to years of life saved (or quality-adjusted life-years).

Suppose the intervention is designed for middle- aged persons with a disease that leads to death and only requires treatment for 3 years. If the therapy costs $3/d, the total cost of therapy per 100 persons is 100 X 3 X 365 X 3, or $328 500. If the treatment does not reduce any other component of health care, one way of viewing the program would be in terms of its net expenditure of $328 500 per 100 persons, or net use of plan revenue.

However, a cost-effectiveness argument could in- corporate a noneconomic outcome such as mortality offset and be represented as the cost per year of life saved. For example, if mortality decreased from 6% to 3% over 3 years and if the average life expectancy at this age was 25 years, the number of life-years saved per 100 persons would be (0.06 - 0.03) X 100 X 25, or 75 per 100 persons. Because the net cost of therapy was $328 500, the cost would be $328 500 per 75 life-years saved or $4380 per year of life saved; this is highly cost-effective compared with other life-saving interventions (32).

The other health economic method that includes outcomes is cost-utility assessment, which is similar to cost-effectiveness but values each year of life saved as something other than 1 = alive and 0 = dead. In the above example, if other events (such as

Table 3. Health Economic Evaluations

Types

Cost minimization

Cost-benefit analysis

Cost-effectiveness analysis

Cost-utility analysis

Question To Address or Definition

When two or more interventions have similar effectiveness, which is least expensive overall?

When effectiveness and outcomes measured both differ, what is the overall economic tradeoff between programs (when both costs and benefits are measured in dollars)?

When the effectiveness of two programs differs (using the same outcome measure), what is the comparable cost per outcome?

Same as cost-effectiveness analysis except that the effectiveness measure can be assessed and valued by the patient

834 1 May 1996 • Annals of Internal Medicine • Volume 124 • Number 9

myocardial infarctions) are avoided with therapy and if we assume that life after each event is worth half a healthy year of life, we would add the 75 years of life to the half-years of "qualify-adjusted" life to calculate the total number of qualify-adjusted years. The numerator would not change, but a cost- utility ratio would be computed, reflecting the cost per number of qualify-adjusted life years.

Selection of methods depends on the research question and the perspective of the target audience; this approach can lead to disparate answers. If a health care benefits manager notices a $328 500 ex- penditure, the fact that this amounts to a high ratio of cost-effectiveness would not necessarily be com- pelling enough to warrant widespread adoption of a treatment that leads to unplanned spending. On the other hand, practitioners familiar with economic evaluation may be pleased to see so low a ratio and insist on the therapy. Patients may demand treat- ment if they do not pay for it and understand only the difference in mortality. Thus, the approach to economic evaluation of disease management varies widely with the stakeholder.

Another important issue is inference. Many eco- nomic evaluations rely on inferences being made, because information on long-term cost and potential outcomes is not always readily available. In these instances, inferences must be made on expected event rates; even with sensitivify analyses and dis- closure of all assumptions, evaluations are only as good as the models.

Outcomes Management

Outcomes management assumes that by system- atically measuring outcomes and reviewing the treat- ment that preceded the outcomes, optimal therapy can be determined (33, 34). This process should lead to management recommendations, followed by reevaluation of outcomes and a continuous oppor- tunify for improved delivery of care. This is a dy- namic management paradigm; it is not the same as outcomes research, which seeks to define the range of outcomes produced by alternative interventions. Outcomes management seeks to produce desirable outcomes in a clinical setting and is therefore the application of outcomes research to practice.

Outcomes management requires an infrastructure in which population-based outcomes can be readily assessed. First, a common set of outcomes measures should be endorsed and used; this is not current practice. In some cases, numerous alternative out- comes measures exist for the same disease, with no overseeing body to endorse one or another. Second, uniform collection and encoding of outcomes data in the population are necessary; this process can be

Table 4. Examples of Components of Disease Management Programs

Component

Health risk assessment Chart audit protocols Database analysis Psychometrics (such as

quality of life) Satisfaction indices Practice surveys Health fairs Clinical guidelines Clinical pathways Centers of excellence Clinical trials Effectiveness studies Economic analyses Professional education Patient education Automated telephone

systems Compliance programs

Identification

y y

y

y

y

Implementation

y y

y y y y y y

y y

y y

Measurement (Outcomes)

y y

y y y

y y y

y

difficult when care is rendered across numerous pro- viders who are not necessarily linked through a shared (that is, computerized) information system. Finally, to effect a change, these data must be linked to information on the process and structure of health care previously delivered.

The limited training and orientation of health care professionals to this fype of analysis and pro- gram development adds to the difficulfy in imple- menting outcomes management. Although simple in theory, incorporation of outcomes management prin- ciples into medical practice requires that a complex, expensive set of processes become seamlessly inte- grated into health care delivery. It also requires uni- form collection of data and transmission and, ulti- mately, consent from patients to provide the data.

Disease Management

Disease management incorporates outcomes re- search technology into outcomes measurement and management programs. No clear distinction can be made between outcomes management and disease management. The only distinctions may be tech- nical; disease management does not necessarily require outcomes assessment, but outcomes man- agement requires outcomes assessment before im- plementation.

The basic premise behind disease management is that there is a more optimal way to manage pa- tients, which results in lowered costs and improved health outcomes. This premise is predicated on three basic assumptions: 1) Medical practice varies (if not, why impose "best practices" or optimal care?); 2) variation is related to different outcomes (if not, why bother to intervene?); and 3) it is

1 May 1996 • Annals of Internal Medicine • Volume 124 • Number 9 835

Outcomes Research

I Epidemioiogic

Studies Clinical Trials

Effectiveness Studies

I

Psychometrics (e.g., Quality of

Life); Functional

Status

I O

Review of Drug and Nondnig

Interventions

Pharmaco- Economics

(Cost-Effectiveness and Cost-Benefit

Analyses)

Disease Management Strategy (Standards of Care/Guidelines)

Continuous Quality Improvement I Application

Outcomes Management

Process Changes Dmg Utilization Review/

Drug Utilization Evaluation

Compliance/Persistence Clinical/Laboratory Markers

Improved Quality of Care and Patient Outcomes

Figure 1. The multidisciplinary scientific basis of outcomes research and the manner in which it fiows into disease management programs. Continuous quality Improvement of outcomes is the ultimate goal.

possible to develop and implement a system of care that improves outcomes.

Sufficient research supports the first two points, but there is limited evidence that disease manage- ment systems can improve health outcomes. This is due, in part, to the recent emergence of the con- cept; few systems have been in place long enough to measure outcomes. Additionally, a successful dis- ease management program is generally based on information acquired by comparing the program with other systems of care. Providers are currently implementing community-wide programs without the benefit of such comparative data.

Disease management involves a shift from the classic paradigm of an individual physician providing comprehensive, continuous, and affordable health care to patients as they present in the clinical set- ting to a population-based systematic approach that identifies persons at risk, intervenes, measures the outcomes, and provides continuous quality improve- ment. At the core is the knowledge base on which to determine how best to identify patients, inter- vene, and measure outcomes. The information that drives this process involves a shift from consensus- based medicine to evidence-based medicine (35-37), the emerging science of integrating information from credible clinical trials to make decisions.

Comprehensive disease management requires a deep understanding of the natural history of disease to determine where in the life cycle of disease an intervention should be implemented. Because most chronic diseases have a long natural history, the

same intervention (for example, lipid lowering to benefit atherosclerosis) could be implemented for patients who have preexisting disease (secondary prevention) or for those who do not (primary pre- vention). The economic consequences and time frame will differ because of the frequency of inci- dent events or ability of the intervention to work. This level of detail helps to frame the program in terms of the characteristics of persons being sought for enrollment, expected duration and type of inter- vention, and selection of relevant outcomes. One must have information on the optimal method for managing disease in order to implement a successful disease management program. Some of the impor- tant issues about implementation include an under- standing of the inefficiencies in health care delivery (benefits design as it drives delivery), disincentives for the patient or provider to receive or deliver the highest quality care (such as access and cost issues), relative cost-effectiveness of alternative treatments, and the success of different interventions in modi- fying behavior (such as compliance).

For planning purposes, the intervention itself can be encapsulated in the form of clinical protocols or pathways (based on guidelines) that must be dissem- inated and applied. Many efforts are currently avail- able to deliver these messages at the point of care, but we have much to learn about the best ways to implement such programs and change provider and patient behavior.

Although overall disease management programs have not yet been evaluated, components of such

836 1 May 1996 • Annals of Internal Medicine • Volume 124 • Number 9

programs have been studied (Table 4). For example, recent research has shown that automated tele- phone reminder systems can increase clinic atten- dance by nearly 35% (38) and that the use of home blood pressure monitoring can reduce physician of- fice visits by 44% (39).

The old paradigm for the management of disease was a somewhat fragmented system involving the treatment of disease by individual practitioners. Treatment was often based on anecdotal or consen- sus information in a fee-for-service environment in which manual systems of record keeping and infor- mation storage were used. The new paradigm is population-based risk and disease assessment, sys- tems of disease prevention and health promotion, community-based intervention and provider contacts within a framework of automated information, evi- dence-based medicine, and defined protocols of care, with explicit collection of outcomes informa- tion. Figure 1 shows the spectrum of disease man- agement concepts reviewed in our paper and the manner in which they relate to each other.

In the future, patient health will be improved by maximizing functionality; minimizing disease, dis- ability, and death; and improving the efficiency and cost-effectiveness of health care. This improvement involves the use of effective outcomes tools, includ- ing linked databases and collaborative research ef- forts. It is also increasingly apparent that health care providers will need to be hands-on computer users when clinical decision making occurs with data on individual patients. Although the field of out- comes assessment and research has not been widely understood, its application to the practice of medi- cine through disease management has already begun.

Requests for Reprints: Louis M. Sherwood, MD, Merck & Co., PO Box 4, Sumneytown Pike, West Point, PA 19486-0004.

Current Author Addresses: Dr. Epstein: Merck-Medco Managed Care, 100 Summit Avenue, Montvale, NJ 07645. Dr, Sherwood: Merck & Co., PO Box 4, Sumneytown Pike, West Point, PA 19486-0004.

References 1. Maimonides MB. Guide for the Perplexed. Friedlander M, trans. 2d ed

London: Rautledge & Sons; 1904. 2. Rogers WJ, Johnstone DE, Yusuf S, Weiner DH, Gallagher P, Bittner

VA, et al. Quality of life among 5,025 patients with left ventricular dysfunc- tion randomized hetween placebo and enalapril: the Studies of Left Ventric- ular Dysfunction. The SOLVD Investigators. J Am Coll Cardiol 1994'23-393- 400. • ' •

3. Randomized trial of cholesterol lowering in 4444 patients with coronary heart disease: the Scandinavian Simvastatin Survival Study (4S) Lancet 1994-344' 1383-9. • ' •

4. Eaker E. Hahn RA. Women's health initiative (Letter). N Engl J Med. 1994;

5. Croog SH, Levine S, Testa MA, Brown B, Bulpitt a , Jenkins CD, et al. The effects of antihypertensive therapy on the quality of life. N Engl J Med 1986;314:1657-64.

6. Roper WL, Winkenwerder W, Hackbarth GM, Krakauer H. Effectiveness in health care. An initiative to evaluate and improve medical practice N Enol J Med. 1988;319:1197-202. '

7. Krakauer H, Bailey RC. Epidemiological oversight of the medical care pro- vided to Medicare beneficiaries. Stat Med. 199l;1D:52l-40.

8. Maklan CW, Greene R, Cummings MA. Methodological challenges and innovations in patient outcomes research. Med Care. 1994;32:JS13-21.

9. Corrigan JM, Nielsen DM. Toward the development of uniform reporting standards for managed care organizations: the Health Plan Employer Data and Information Set (Version 2.0). Jt Comm J Qual Improv. 1993; 19:566-75.

10. Donabedian A. Evaluating the quality of medical care. Milbank Mem Fund Q. 1966;44:166-206.

11. Donabedian A. The quality of medical care. Science. 1978;200:856-64. 12. Donabedian A. Explorations in Quality Assessment and Monitoring. 1. The

Definition of Quality and Approaches to its Assessment. Ann Arbor, Ml: Health Administration Pr; 1980:1-31.

13. Anderson GM, Spitzer WO, Weinstein MC, Wang E, Blackburn JL, Bergman U. Benefits, risks, and costs of prescription drugs: a scientific basis for evaluating policy options. Clin Pharmacoi Ther. 1990;48:111-9.

14. Jones PW. Quality of life, symptoms and pulmonary function in asthma: long-term treatment with nedocromil sodium examined in a controlled mul- ticentre trial. Nedocromil Sodium Quality of Life Study Group. Eur Respir J. 1994;7:55-62.

15. Marks GB, Dunn SM, Wooicock AJ. A scale for the measurement of quality of life in adults with asthma. J Clin Epidemiol. 1992,45:461-72.

16. Juniper EF, Guyatt GH, Epstein RS, Ferrie PJ, Jaeschke R, Hiller TK. Evaluation of impairment of health related quality of life in asthma: develop- ment of a questionnaire for use in clinical trials. Thorax. 1992;47:76-83.

17. Hyland ME. The Living with Asthma Questionnaire. Respir Med. 1991; 85(Suppl B):33-7.

18. Kirshner B, Guyatt G. A methodological framework for assessing health indices. J Chronic Dis. 1985;38:27-36.

19. Revicki DA, Rothman M, Luce B. Health-related quality of life assessment and the pharmaceutical industry. Pharmacoeconomics. 1992; 1:394-408.

20. Jaeschke R, Singer J, Guyatt GH. Measurement of health status. Ascer- taining the minimal clinically important difference. Controlled Clin Trials. 1989; 10:407-15.

21. Lydick E, Epstein RS. Interpretation of quality of life changes. Qual Life Res. 1993;2:221-6.

22. Cleeland CS. Assessment of pain in cancer measurement issues. Advances in Pain Research Therapy. 1990; 16:47-55.

23. Guess HA, Jacobsen SJ, Girman CJ, Oesterling JE, Chute CG, Panser LA, et al. The role of community-based longitudinal studies in evaluating treatment effects. Example: benign prostatic hyperplasia. Med Care 1996' 33(4 Suppl):AS26-35.

24. Deyo RA, Patrick DL. The significance of treatment effects: the clinical perspective. Med Care. 1995;33(4 Suppl):AS286-91.

25. Brook RH, Ware JE Jr, Rogers WH, Keeler EB, Davies AR, Donald CA, et al. Does free care improve adults' health? Results from a randomized controlled trial. N Engl J Med. 1983;309:1426-34.

26. Brook RH, Lohr KN. Efficacy, effectiveness, variations, and quality. Bound- ary-crossing research. Med Care. 1985;23:710-22.

27. Perfetto EM, Epstein RS. Assessing patient outcomes of drug therapy: the role of pharmacoepidemiology. In: Hartzeman AG, Porta MS, Tilson HH, eds. Pharmacoepidemiology: An Introduction. 3d ed. Cincinnati: Han/ey Whitney Books; [In press].

28. Diamond GA, Denton TA. Alternative perspectives on the biased founda- tions of medical technology assessment. Ann Intern Med. 1993; 118:455-64.

29. Oster G, Borok GM, Menzin J, Heyse JF, Epstein RS, Quinn V, et al. A randomized trial to assess effectiveness and cost in clinical practice: rationale and design of the Cholesterol Reduction Intervention Study (CRIS) Control Clin Trials. 1995; 16:3-16.

30. Oster G, Borok GM, Menzin J, Heyse JF, Epstein RS, Quinn V, et al. Cholesterol Reduction Intervention Study (CRIS): a randomized trial to assess effectiveness and costs in clinical practice. Arch Intern Med. [In press].

31. Simon G, Wagner E, Vonkorff M. Cost-effectiveness comparisons using real world randomized trials: the case of new antidepressant drugs J Clin Epidemiol. 1995;48:363-73.

32. Tengs TO, Adams ME, Pliskin JS, Safran DG, Siegel JE, Weinstein MC, et al. Five-hundred life-saving interventions and their cost-effectiveness Risk Anal. 1995;15:369-89.

33. Ellwood PM. Shattuck lecture—outcomes management. A technology of patient experiences. N Engl J Med. 1988;3ia:1549-56.

34. Steinwachs DM, Wu AW, Skinner EA. How will outcomes management work? Health Aff (Millwood). 1994;13:153-62.

35. Oxman AD, Sackett DL, Guyatt GH. Users' guides to the medical literature. I. How to get started. The Evidence-Based Medicine Working Group JAMA 1993;270:2093-5.

36. Guyatt GH, Sackett DL, Cook DJ. Users' guides to the medical literature. II. How to use an article about therapy or prevention. A. Are the results of the study valid? Evidence-Based Medicine Working Group. JAMA. 1993;270:2598-601

37. Guyatt GH, Sackett DU Cook DJ. Users' guides to the medical literature. II. How to use an article about therapy or prevention. B. What were the results and will they help me in caring for my patients? Evidence-Based Medicine Working Group. JAMA. 1994;271:59-63.

38. Tanke ED, Leirer VO. Automated telephone reminders in tuberculosis care Med Care. 1994;30:380-9.

39. Soghikian K, Casper SM, Fireman BH, Hunkeler EM, Hurley LB, Tekawa IS, et al. Home blood pressure monitoring. Effect on use of medical services and medical care costs. Med Care. 1992;30:855-65.

1 May 1996 • Annals of Internal Medicine • Voiume 124 • Number 9 837