case study

1995-davis.pdf

Home >English homework help >Article writing homework help >case study

Full Text

Ask A Librarian

Changing Physician Performance: A Systematic Review of the Effect of

Continuing Medical Education Strategies [Reviews]

Davis, David A. MD; Thomson, Mary Ann BHSc; Oxman, Andrew D. MD; Haynes R. Brian, PhD MD

From the Faculty of Medicine, University of Toronto (Ontario) (Dr Davis); the Faculty of Health Sciences, McMaster University, Hamilton, Ontario (Ms Thomson and Dr Haynes); and the National Institute of Public Health, Oslo, Norway (Dr Oxman).

Abstract

Objective: To review the literature relating to the effectiveness of education strategies designed to change physician performance and health care outcomes.

Data Sources: We searched MEDLINE, ERIC, NTIS, the Research and Development Resource Base in Continuing Medical Education, and other relevant data sources from 1975 to 1994, using continuing medical education (CME) and related terms as keywords. We manually searched journals and the bibliographies of other review articles and called on the opinions of recognized experts.

Study Selection.We reviewed studies that met the following criteria: randomized controlled trials of education strategies or interventions that objectively assessed physician performance and/or health care outcomes. These intervention strategies included (alone and in combination) educational materials, formal CME activities, outreach visits such as academic detailing, opinion leaders, patient- medicated strategies, audit with feedback, and reminders. Studies were selected only if more than 50 percent of the

60610. Volume 274(9), 6 September 1995, pp 700-705

| | |

Links

Article Review

Outline

Abstract METHODS

Data Sources Selection Criteria Educational Interventions Data Extraction

RESULTS Types of Physician Studies Settings Interventions Single Interventions Two-Method Interventions Multifaceted Interventions Effects of Needs Analysis Analysis by Domain of Behavior Change Targeted

COMMENT Study Limitations Comparisons With the Health Care Environment The Science and Practice of CME

REFERENCES AND NOTES

Recent History

Changing Physician Perfor...

Page 1 of 17Ovid: Davis: JAMA, Volume 274(9).September 6, 1995.700-705

8/23/2005http://gateway.ut.ovid.com/gw2/ovidweb.cgi

subjects were either practicing physicians or medical residents.

Data Extraction: We extracted the specialty of the physicians targeted by the interventions and the clinical domain and setting of the trial. We also determined the details of the educational intervention, the extent to which needs or barriers to change had been ascertained prior to the intervention, and the main outcome measure(s).

Data Synthesis: We found 99 trials, containing 160 interventions, that met our criteria. Almost two thirds of the interventions (101 of 160) displayed an improvement in at least one major outcome measure: 70 percent demonstrated a change in physician performance, and 48 percent of interventions aimed at health care outcomes produced a positive change. Effective change strategies included reminders, patient-mediated interventions, outreach visits, opinion leaders, and multifaceted activities. Audit with feedback and educational materials were less effective, and formal CME conferences or activities, without enabling or practice-reinforcing strategies, had relatively little impact.

Conclusion: Widely used CME delivery methods such as conferences have little direct impact on improving professional practice. More effective methods such as systematic practice-based interventions and outreach visits are seldom used by CME providers.

(JAMA. 1995;274:700-705)

IN MOST developed nations, health care systems are facing important changes, driven by political and economic forces and by the general concern for competent and consistent quality care. [1] Responses to these forces include altering the training of physicians across the continuum of medical education. Undergraduate medical education reform, stressing interdisciplinary care, problem-based learning, increased critical appraisal of research evidence, and community-based training, is part of the response to health care reform. [2] Graduate education reforms have also begun to focus on increased community-based experiences, with an emphasis on generalist education. These changes in medical education are aimed at producing a new generation of clinicians whose practice more completely reflects the goals of health care reform. [3]

Even if effective, changes aimed at undergraduates and residents necessarily demand a lengthy incubation period because of the slow percolation of graduates into the health care system. The contributions of continuing medical education (CME), a major facilitator of change in practitioner behavior, are undermined by difficulties in CME delivery, which is often disjointed, driven by the largesse of drug companies and the requests of physicians, and plagued by doubts about and relatively little evidence for its own effectiveness and efficacy. [4,5] Similar deficiencies can be found in the adoption of new information and techniques. For example, despite convincing evidence about the effectiveness of thrombolytic therapy in the management of myocardial infarction, even current texts and review articles frequently fail to reflect this treatment. [6] Thus, the effectiveness of educational methods aimed at improving physician performance and the health status of their patients appear inadequate to respond to the urgent demands of health care reform. [7]

Keeping in mind the context of larger reform issues, this review of educational interventions

Page 2 of 17Ovid: Davis: JAMA, Volume 274(9).September 6, 1995.700-705

8/23/2005http://gateway.ut.ovid.com/gw2/ovidweb.cgi

as tools in a CME system has the following objectives. First, it attempts to increase the knowledge base about the effectiveness of broadly defined educational interventions targeted at changing the performance of practicing physicians and the health status of their patients, to build on earlier work in this area [4,5] and a recent publication [8] that includes nonphysician health professional educational interventions and refines methodological practices in the search and review process, to extend others' similar syntheses, [9-11] and to describe which educational interventions have been tested in what populations of physicians in which settings and practice domains. Second, we analyze the extent to which preintervention design elements affect outcomes (for example, assessment of practice needs or analysis of barriers to change). Third, we attempt to further the theoretical knowledge base of changing physician behavior. Finally, we outline areas in which knowledge is as yet weak or inconclusive, thus shaping a research and development agenda.

METHODS

We used search and data extraction strategies similar to those of our earlier reviews [4,5,8] and briefly outlined here.

Data Sources

From 1975 to 1994, we searched the computer bibliographic databases of MEDLINE on a monthly basis, and those of ERIC, NTIS, CINAHL, HEALTHLINE, and EMBASE semiannually to select trials, studies, and descriptions in the broad domain of continuing health professional education, using keywords described in an earlier publication. [4] These keywords were combined with a sensitive search strategy [12] to identify randomized controlled trials (RCTs). In addition, manual searches of journals and the bibliographies of retrieved articles and the input of recognized experts in the field were used to retrieve articles. These articles were then reviewed by a health sciences librarian and/or a member of the study team, additional keywords or descriptors were added, and a summary was entered into a computerized and retrievable format using Reference Manager (Research Information Systems, Carlsbad, Calif). This final common source is called the Research and Development Resource Base in CME (RDRB-CME). [13]

Selection Criteria

To study the effects of the CME interventions that had been most rigorously tested and to gain as much comparability across studies as possible, we retrieved from the RDRB-CME those studies that met the following criteria: (1) RCTs or trials that used alternatives for randomization, such as alternating allocation to either control or experimental arms; (2) replicable educational interventions directed at changing clinical behavior or health outcomes; (3) study populations of health professionals in which the majority were either practicing physicians or medical residents; and (4) objective measurements of physician performance or health care outcomes. Inclusion criteria were pilot tested by all authors on five studies and then applied by at least two of three of us (M.A.T.,A.D.O.,D.A.D.) independently.

Educational Interventions

Page 3 of 17Ovid: Davis: JAMA, Volume 274(9).September 6, 1995.700-705

8/23/2005http://gateway.ut.ovid.com/gw2/ovidweb.cgi

We defined an educational intervention as any attempt to persuade physicians to modify their practice performance by communicating clinical information. We excluded strategies that were purely administrative (such as the reorganization of charts in a computerized or problem-based format) or applied incentives or coercion (financial rewards or sanctions). The details of the educational interventions were modified from a recent study [8] and included the following: (1) educational materials (including noninteractive printed, audiovisual, and computer-produced information); (2) formal CME programs such as conferences, seminars, workshops, small group sessions, traineeships (in-depth, mini-residencies, or preceptorships), and teleconferences; (3) outreach visits, including academic detailing (visits by physician educators such as pharmacists) or counter-detailing directly to physicians by pharmacists and others, or practice facilitation by nurses and other professionals; (4) local opinion leaders or educational influentials; (5) patient- mediated interventions such as patient education materials; (6) audit with feedback, including chart review with peers or supervisors; and (7) reminders. We also included combinations of these activities.

Data Extraction

Several categories of data were extracted: (1) the discipline of the targeted physician population, (2) the intended behavioral change, (3) the setting of the trial, (4) the degree to which a learning or performance gap had been identified by the authors, (5) the intervention, as outlined previously, and (6) the clinical setting of the trial. Further, (7) we attempted to determine the degree to which each intervention was planned or based on an analysis of practice deficiency or need. Specifically, we assessed whether national policy or guidelines had been developed and used; if local consensus had been sought, using the target audience or representatives of the target audience in planning; whether barriers to change were identified and targeted by the intervention; and finally, if gaps in performance were determined by the study authors, followed by interventions targeted at practitioners whose performance was judged to be suboptimal. These last data extraction criteria were developed by one of us (D.A.D.) and pilot tested by another (M.A.T.) on five studies, then applied in all trials.

Finally, the outcome measures used by the study authors were extracted, and a determination was made regarding the degree to which each study was positive, negative, inconclusive, or mixed. Positive studies were those that demonstrated a statistically significant change in at least one major outcome measure at the level of P<.05; negative studies showed no important change, despite sufficient statistical power to do so; inconclusive studies failed to demonstrate a change but lacked the statistical power to demonstrate or exclude a clinically important difference; and mixed studies were those that improved physician performance but not health care outcomes. These data were extracted by one of the authors (D.A.D. or M.A.T.) and checked by at least one of the other authors.

RESULTS

At the time of preparation of this review, the RDRB-CME contained more than 6000 articles and monographs on the broadly defined topic of CME. Ninetynine studies met our criteria for inclusion and form the substrate for this review. [14-112] Within these 99 studies, there were 160

Page 4 of 17Ovid: Davis: JAMA, Volume 274(9).September 6, 1995.700-705

8/23/2005http://gateway.ut.ovid.com/gw2/ovidweb.cgi

separate interventions apart from placebo or nonintervention controls.

Types of Physician Studies

Fifty-one of the 99 studies included internists, 35 focused on family physicians or general practitioners, and 15 trials did not specify physician type or background. Smaller numbers of trials studied obstetrician/gynecologists (n=5), pediatricians (n=4), emergency physicians (n=3), surgeons (n=2), and anesthesiologists (n=2). Many studies included more than one category of physician. Residents were a part of the study physician population in one third (n=34) of the studies, either in conjunction with practicing physician-supervisors or by themselves.

Settings

Three quarters (n=75) of the studies took place in outpatient settings: 27 in private office practices and 48 in ambulatory care centers or clinics. Of the latter, 35 authors described their settings as teaching centers, nine as managed care settings, [17,19,35,71,74,84,86,110,111] and three as veterans centers [54,70,92]; one study failed to characterize its location precisely. Twenty-six trials took place either in hospitals (20 on wards, [14,20,27,31,37,41,56,57,80,85,89,91,93- 96,105,106,108,112] three in emergency departments, [51,78,79] two on labor floors, [28,53] and one in an intensive care unit [77]) or in the long-term care setting. [16] Two of the trials each took place in two settings. [85,91]

Interventions

Of the 160 noncontrol interventions in these studies, 99 (62 percent) showed an improvement in at least one major outcome measure in physician performance or health care; 53 (33 percent) failed to demonstrate a change in either of these areas. Eight interventions (5 percent) displayed mixed results. [16,25,35,36,49,74,109] Many interventions analyzed both physician performance and health care outcomes. Of the 145 interventions that focused on physician performance, 101 (70 percent) demonstrated change in at least one major measure; 44 (30 percent) did not. Of the 46 interventions targeted at changing health care outcomes, 22 (48 percent) succeeded in doing so. The size of the effects in positive studies was small to moderate. The impact of specific interventions is described by single, two-method, and multifaceted interventions.

Single Interventions

Eighty-one single-intervention strategies were used in these trials, 49 (60 percent) of which demonstrated change in at least one major outcome measure, 30 (37 percent) failed to do so, and two studies [74,109] effected change in physician performance without altering health care outcomes.

Outreach visits, including academic detailing, demonstrated positive changes in interventions in which they were used alone. [15,92] Academic detailers in single-intervention and other studies included pharmacists, [15,84] standardized patients, [83] study investigators, [81] or nurse- facilitators. [39] Opinion leader strategies displayed positive effects in a few of the outcomes measured in the three trials in which they were used as a single intervention. [93-95] Patient-

Page 5 of 17Ovid: Davis: JAMA, Volume 274(9).September 6, 1995.700-705

8/23/2005http://gateway.ut.ovid.com/gw2/ovidweb.cgi

mediated methods, such as patient educational materials or patient reminders, showed a positive change in seven interventions [21,54,68,75,87,103] and negative effects in two. [67] Physician reminders was an effective single-method intervention: 22 of 26 interventions effected change, [17,22,26,29,34,52,63-66,71,75,78,79,86,87,98-102,104] and four interventions in two studies failed to do so. [67,108]

More variable results were displayed by audit with feedback: here, we found 10 positive outcomes [19,31,42,46,58,59,70,72,85,99] and 14 negative outcomes. [19,38,41,80,85,92,97,111,112] Outcomes from audit with feedback were more consistently effective when feedback was delivered in the form of chart review (eg, in studies by Everett et al, [41] Martin et al, [57] Pinkerton et al, [76] and Restuccia [85]).

Relatively short (1 day or less) formal CME events such as conferences generally effected no change; six interventions demonstrated negative or inconclusive effects. [23,38,44,45,50,96] One study, [74] using an in-depth traineeship or mini-residency, demonstrated changes in physician performance but not health care outcomes. Educational materials (eg, printed monographs or audiovisual programs) demonstrated a positive effect in four interventions, [55,58,77] but failed to demonstrate an effect in seven instances. [15,30,32,41,82,83,90]

Two-Method Interventions

Thirty-nine interventions used two educational methods, and 25 (64 percent) of these were positive, [27,30,33,37,39,41,48,55,57,58,65,68,70,83,84,89,96,102,106,110] 12 (31 percent) were negative or inconclusive, [19,33,38,40,44,47,60,76,82,97,105,107] and three (<1 percent) yielded mixed results. [16,49] When effective single methods, as described previously, were used in pairs, positive changes were generally noted. Examples included patient-mediated strategies combined with reminders, which demonstrated positive changes in three studies [27,68,102] and negative results in one. [105] In contrast, the combination of two less effective interventions, formal CME activities and educational materials, resulted in seven positive [33,37,39,48,55,106] and five negative or inconclusive studies. [19,33,60,76,82]

Multifaceted Interventions

Thirty-nine interventions used three or more educational strategies. Of these, 31 (79 percent) were positive, [14,18,20,26,28,33,39,44,53,56,60-62,69,73,77,81,91,103,109] including positive changes in health care outcomes. [103,109] A further five (13 percent) were negative or inconclusive,

[23,43,53,88] and three (8 percent) [25,35,36] displayed mixed results.

Effects of Needs Analysis

Several levels of needs analysis were reported in these studies, indicating in progressive fashion the degree to which study authors considered or defined a clinical need, performance gap, or barrier to performance change. First, 12 studies (<1 percent) did not mention the clinical need on which the intervention was based; five (42 percent) of these were positive studies. Second, about one fifth (n=34) of the interventions provided one to several references in clinical care and identified a general clinical area requiring change; 18 (53 percent) of these showed a positive

Page 6 of 17Ovid: Davis: JAMA, Volume 274(9).September 6, 1995.700-705

8/23/2005http://gateway.ut.ovid.com/gw2/ovidweb.cgi

change, 14 (41 percent) did not, and two (6 percent) displayed mixed results. Third, one quarter (n=41) of the interventions were based on already-developed clinical guidelines generally approved by a national body, producing a positive change in 25 (61 percent) of these 41 interventions, negative results in 15 (37 percent), and mixed results in one instance. Fourth, consensus processes to achieve agreement on the part of local health professionals was used by roughly one fourth (n=45) of the interventions, effecting a positive change in 26 (58 percent), no change in 16 (36 percent), and positive physician performance but no corresponding health care outcome change in three instances (6 percent). Finally, less than one fifth (n=28) of the interventions were targeted at specific behaviors identified by a gap analysis technique (such as an audit to determine suboptimal performers) or addressed specific barriers to change; 25 (89 percent) of these interventions exhibited a positive change, three (11 percent) did not.

Analysis by Domain of Behavior Change Targeted

Interventions in these studies targeted two broad domains of behavioral change: roughly one third (n=54) in the area of health promotion and disease prevention and two thirds (n=107) in clinical disease management. One study [48] involved two areas of behavioral change.

The category of disease prevention and health promotion yielded two subcategories, prevention and screening, in which there were 36 interventions, and lifestyle education and counseling (including smoking cessation), which accounted for 18 interventions. The former subcategory yielded 30 positive results [21,26,29,34,39,48,52,59,66,68,69,73,75,87,98,99,102,104,110] and six negative or inconclusive results. [67,97] Examples of effective interventions here included patient-mediated strategies, [21,87] reminders, [26,29,34,52,66,67] and multifaceted interventions.

[69] Of the 18 lifestyle or educational counseling interventions, 10 produced positive results,

[30,33,49,55,83,109] five were negative, [30,33,50,83] and three were mixed. [35,36,109] Among the effective interventions, the use of peer discussion (eg, Davidoff et al [37]) and "practice rehearsal" (eg, Kottke et al [49]), allowing physicians to rehearse certain communication skills in the setting of a formal CME event, produced positive results.

Under the category of clinical management, four interventions were found in the area of diagnosis. Of these, two were positive [78,79] and two negative. [88,89] Reminders appeared as effective change agents. [78,79] We found 24 interventions in the area of resource utilization, principally radiology and laboratory use, 17 of which produced significant change in outcomes

[19,20,28,31,37,41,56-58,77,85,100,101] and seven did not. [19,41,85,112] Reminders [100,101] and audit with feedback (either alone [31,58] or with another intervention such as a conference [57]) appeared to be useful measures here, as did multifaceted interventions. [20,28,56,77] In the category of prescribing, there were 14 interventions, three quarters (n=11) of which were positive. [15,16,22,42,46,62,70,84,91,92] An effective change agent in this category was the academic detail visit. [15,16,62] Finally, interventions to produce enhanced procedural skills demonstrated negative results in three of four interventions. [24,74] Audit with feedback was ineffective in one trial [24] using three interventions, while a clinical traineeship produced a change in the rate of sigmoidoscopy in one study. [74]

We found 58 interventions that attempted to alter outcomes in general medical management,

Page 7 of 17Ovid: Davis: JAMA, Volume 274(9).September 6, 1995.700-705

8/23/2005http://gateway.ut.ovid.com/gw2/ovidweb.cgi

32 (55 percent) of which achieved their goal and 26 (45 percent) did not. Examples of effective strategies include reminders, [63,64,71] two-method interventions (eg, reminders and patient-mediated strategies [27]), or intensive conferencing methods and educational materials

[48] and multifaceted interventions. [14,23,51,60,61] In the subcategory of hypertension, many interventions yielded negative results. [38,40,43,82,11]

COMMENT

The evidence provided by these trials supports some clear conclusions. First, there is a robust body of research assessing the outcomes of physicians' clinical education, as evidenced by several thousand extant articles, including increasing numbers of the most scientifically rigorous variety, the RCT. In addition, this research has grown substantially over the years: in 1984, seven articles were found matching our criteria [4]; in 1992, a further 43 trials [5]; and, 3 years later, another 49. Second, while definitional problems still exist, there is a growing convergence about what constitutes educational interventions, including a broadening operational definition of the term CME. It is apparent that ongoing physician education may now comprise a variety of measures well beyond the traditional short-course model; academic detailing, opinion leaders, audit with feedback, reminders, and other systems may be components of broadly defined CME. Third, this review confirms the major finding of our 1992 review, [5] namely, physician performance may be altered by many of these CME interventions and, to a lesser extent, so may health care outcomes. These alterations are most often small, less often moderate, and rarely large.

The reason changes in health care outcomes lag behind those in physician performance have been explored elsewhere [5,8] and include patients' not accepting physician recommendations, the socioeconomic and educational status of patients, and, frequently, the limited effectiveness of the clinical interventions themselves.

Study Limitations

In interpreting this review, readers should note that some publications may have escaped our search strategy and the literature may not contain important findings derived from negative trials. Further, when reporting is incomplete or vague, data extraction becomes problematic, as was frequently the case in these studies. For example, many studies did not indicate the method of randomization and most did not describe in replicable detail the exact nature of the educational intervention. Because the interventions themselves were of such variability and their participants, settings, targeted behaviors, and outcomes were so varied, estimates of the average effect from meta-analysis would have little practical meaning. Similarly, between-trial comparisons of different interventions are of limited value. [113] Importantly, among "positive" studies are many in which only one or two outcome measures of several are positive. This review made no attempt to summarize the size of effects. Our focus on RCTs systematically excluded the results of qualitative research methods that would afford additional valuable insights into physician behavioral change.

Comparisons With the Health Care Environment

Page 8 of 17Ovid: Davis: JAMA, Volume 274(9).September 6, 1995.700-705

8/23/2005http://gateway.ut.ovid.com/gw2/ovidweb.cgi

Do these trials represent actual physician educational experiences and settings? To a great extent they do not. Neither the complexity and method of the educational intervention nor the optimal circumstances provided in teaching centers or wards in which many of these studies were conducted match the learning and practice circumstances of most physicians. In addition, many of these trials experience a ceiling effect in studying volunteer physicians who are often performing at or near optimum levels. In this group, there may be only room for small changes that are difficult to detect. Moreover, these studies often use end points or outcome markers that may be selected more for their ease of determination (eg, blood pressure) than their clinical imperative, such as heart failure or stroke. Lastly, the types of physicians selected as the focus of these studies do not match the spectrum of today's US physicians. [114] Few if any studies focused on the skills of surgeons, the clinical work of psychiatrists, the function of nephrologists, or other subspecialists. In contrast, the trials did represent prevention and screening, the prime initiatives of health care reform. [7]

The Science and Practice of CME

In a manner analogous to the evidence-based care movement, [115] this review begins to apply scientific and theoretical principles to the area of health professional performance change, and it begins to indicate a movement toward "evidence-based CME."

Particularly promising among the effective interventions are reminders, academic detailing, and patient-mediated strategies. This effect may support the PRECEED model of Green et al, [116] which suggests that the interventions that best succeed in changing performance and health care outcomes are those using practice-enabling strategies (office facilitators or patient educational methods, for example) or reinforcing methods (feedback or reminders) in addition to predisposing or disseminating strategies. To some extent, the effectiveness of opinion leaders may also support the extensive work of Rogers [117] showing that the presence of an opinion leader permits colleagues to observe the outcomes of a particular innovation in an atmosphere conducive to its adoption on a community basis. Formal CME activities that include peer discussion and role- playing would appear to support the theories of Nowlen, [118] who stresses the role of the group in adopting new information, and Bandura, [119] whose Social Learning Theory stresses the importance of personal, environmental/situational, and behavioral factors. The sequenced provision of multifaceted interventions may mimic the process outlined by Fox et al [120] in which learning and change take place through a series of "impactors" or learning resources.

The effect of the interventions is not, however, consistent across practitioners, settings, or behaviors. At least part of the variation may be attributable to an individual readiness to change: physician-learners may progress at their own rates, depending on their motivation, their knowledge of a problem, or the perception of a gap between current knowledge and skills and those needed. [120] This was borne out by our needs analysis: when barriers to change were addressed or gaps were demonstrated and resources deployed to help the learner, change appeared to occur relatively frequently.

Whatever the theoretical basis of CME, there is clear evidence that its current provision in North America consists mostly of the less effective change strategies such as conferences, [121]

Page 9 of 17Ovid: Davis: JAMA, Volume 274(9).September 6, 1995.700-705

8/23/2005http://gateway.ut.ovid.com/gw2/ovidweb.cgi

compared with vanishingly small but more effective methods such as academic detailing or reminder systems. To base CME interventions on identified clinical needs, however, new linkages for CME providers will need to be found--in health services research, in hospitals, in provincial or state-generated data sources, from insurance carriers, and within managed care systems.

A full research agenda awaits the expanding interdisciplinary field of CME. This agenda needs to consider further intervention innovations and attend more closely to rigorous evaluations combined with more qualitative methods to elucidate physicians' perceptions regarding learning and change. To advance the process of describing research findings related to CME, more consistent reporting and design strategies and up-to-date systematic reviews of trials that address specific questions are required, along with collaboration between CME researchers and providers such as the Cochrane Collaboration on Effective Professional Practice. [122] Additionally, there is a need for more coherent research approaches that build on prior work and a focused effort to disentangle the interplay of change, setting, type of health professional, and intervention, perhaps following the model of multicenter trials. Such cooperative evidence-based approaches will advance the effort to improve the performance of physicians and the health outcomes of their patients.

The RDRB-CME is supported in part by the Alliance for Continuing Medical Education, Northbrook, Ill, the Society of Medical College Directors of Continuing Medical Education, Washington, DC, and the University of Toronto Faculty of Medicine.

We are indebted to Anne Taylor-Vaisey, MLS, continuing education librarian at the University of Toronto Faculty of Medicine, for her preparation of this review's database.

Corresponding author: Office of Continuing Education, Faculty of Medicine, University of Toronto, 150 College St, Toronto, Ontario, Canada M5S 1AB (Dr Davis).

REFERENCES AND NOTES

1. Amara R, Morrison JI, Schmid G. Looking Ahead at American Health Care. Institute for the Future, Healthcare Information Center, Washington, DC: McGraw-Hill International Book Co; 1988. [Context Link]

2. Association of American Medical Colleges. Roles for medical education in health care reform. Acad Med. 1994;69:512-515. Bibliographic Links [Context Link]

3. Pories WJ, Smout JC, Morris A, Lewkow VE. US health care reform: will it change postgraduate surgical education? World J Surg. 1994;18:745-752. Bibliographic Links [Context Link]

4. Haynes RB, Davis DA, McKibbon A, Tugwell P. A critical appraisal of the efficacy of continuing medical education. JAMA. 1984;251:61-64. Bibliographic Links [Context Link]

5. Davis DA, Thomson MA, Oxman AD, Haynes RB. Evidence for the effectiveness of CME: a review of 50 randomized controlled trials. JAMA. 1992;268:1111-1117. Bibliographic Links [Context Link]

6. Lau J, Antman EM, Jimenez-Silva J, Kupelnick B, Mosteller F, Chalmers TC. Cumulative meta-analysis of therapeutic trials for myocardial infarction. N Engl J Med. 1992;327:248-254. Bibliographic Links [Context Link]

Page 10 of 17Ovid: Davis: JAMA, Volume 274(9).September 6, 1995.700-705

8/23/2005http://gateway.ut.ovid.com/gw2/ovidweb.cgi

7. Jay SJ, Anderson JG. Continuing medical education and public policy in an era of health care reform. J Contin Educ Health Profess. 1993;13:195-209. [Context Link]

8. Oxman AD, Thomson MA, Haynes RB, Davis DA. No magic bullets: a systematic review of 102 trials of interventions to help health care professionals deliver services more effectively and efficiently. Can Med Assoc J. In press. [Context Link]

9. Bertram DA, Brooks-Bertram PA. The evaluation of continuing medical education: a literature review. Health Educ Monogr. 1977;5:330-362. Bibliographic Links [Context Link]

10. Beaudry JS. The effectiveness of continuing medical education: a quantitative synthesis. J Contin Educ Health Profess. 1989;9:285-307. [Context Link]

11. McLaughlin PJ, Donaldson JF. Evaluation of continuing medical education programs: a selected literature, 1984-1988. J Contin Educ Health Profess. 1991;11:65-84. [Context Link]

12. Cochrane Collaboration. Optimal MEDLINE Search Strategy for Identifying Randomized Clinical Trials (RCTs): Cochrane Collaboration Handbook. Oxford, England: UK Cochrane Centre; 1994: 29. [Context Link]

13. Davis DA, Fox RD, eds. The Physician as Learner: Linking Research to Practice. Chicago, Ill: American Medical Association Press; 1994:xiii. [Context Link]

14. Anderson FA Jr, Wheeler HB, Goldberg RJ, Hosmer DW, Forcier A, Patwardhan NA. Changing clinical practice: prospective study of the impact of continuing medical education and quality assurance programs on use of prophylaxis for venous thromboembolism. Arch Intern Med. 1994; 154:669-677. Bibliographic Links [Context Link]

15. Avorn J, Soumerai SB. Improving drug-therapy decisions through educational outreach: a randomized controlled trial of academically based 'detailing.' N Engl J Med. 1983;308:1457-1463. [Context Link]

16. Avorn J, Soumerai SB, Everitt DE, et al. A randomized trial of a program to reduce the use of psychoactive drugs in nursing homes. N Engl J Med. 1992;327:168-173. Bibliographic Links [Context Link]

17. Barnett GO. A computer based monitoring system for follow-up of elevated blood pressure. Med Care. 1983;21:400-409. Bibliographic Links [Context Link]

18. Bass MJ, McWhinney IR, Donner A. Do family physicians need medical assistants to detect and manage hypertension? Can Med Assoc J. 1986;134: 1247-1255. Bibliographic Links [Context Link]

19. Berwick DM, Coltin KL. Feedback reduces test use in a health maintenance organization. JAMA. 1986;255:1450-1454. Bibliographic Links [Context Link]

20. Billi JE, Hejna GF, Wolf FM, Shapiro LR, Stross JK. The effects of a cost-education program on hospital charges. J Gen Intern Med. 1987;2:306-311. Bibliographic Links [Context Link]

21. Brimberry R. Vaccination of high-risk patients for influenza: a comparison of telephone and mail reminder methods. J Fam Pract. 1988;26:397-400. Bibliographic Links [Context Link]

22. Britton ML, Lurvey PL. Impact of medication profile review on prescribing in a general medicine clinic. Am J Public Health. 1991;48:265-270. [Context Link]

23. Browner WS, Baron RB, Solkowitz S, Adler LJ, Gullion DS. Physician management of hypercholesterolemia: a randomized trial of continuing medical education. West J Med. 1994;161:572-578. Bibliographic Links [Context Link]

24. Buntinx F. Does feedback improve the quality of cervical smears? a randomized controlled trial. Br J

Page 11 of 17Ovid: Davis: JAMA, Volume 274(9).September 6, 1995.700-705

8/23/2005http://gateway.ut.ovid.com/gw2/ovidweb.cgi

Gen Pract. 1993;43:194-198. [Context Link]

25. Callahan CM, Hendrie HC, Dittus RS, Brater DC, Hui SL, Tierney WM. Improving treatment of late life depression in primary care: a randomized clinical trial. J Am Geriatr Soc. 1994;42:839-846. Bibliographic Links [Context Link]

26. Chambers CV, Balaban DJ, Carlson BL. Microcomputer-generated reminders: improving the compliance of primary care physicians with mammography screening guidelines. J Fam Pract. 1989; 29:273-280. [Context Link]

27. Chase CR. Medical information management: improving the transfer of research results to presurgical evaluation. Med Care. 1983;21:410-424. Bibliographic Links [Context Link]

28. Chassin MR, McCue SM. A randomized trial of medical quality assurance: improving physicians' use of pelvimetry. JAMA. 1986;256:1012-1016. Bibliographic Links [Context Link]

29. Cheney C, Ramsdell JW. Effect of medical records' checklists on implementation of periodic health measures. Am J Med. 1987;83:129-136. Bibliographic Links [Context Link]

30. Cockburn J, Ruth D, Silagy C, et al. Randomized trial of three approaches for marketing smoking cessation programmes to Australian general practitioners. BMJ. 1992;304:691-694. [Context Link]

31. Cohen DI, Jones P, Littenberg B, Neuhauser D. Does cost information availability reduce physician test usage? a randomized clinical trial with unexpected findings. Med Care. 1982;20:286-292. Bibliographic Links [Context Link]

32. Cohen SJ, Christen AG, Katz BP, et al. Counseling medical and dental patients about cigarette smoking: the impact of nicotine gum and chart reminders. Am J Public Health. 1987;77:313-316. Bibliographic Links [Context Link]

33. Cohen SJ, Weinberger M, Hui SL, Tierney WM, McDonald CJ. The impact of reading on physicians' nonadherence to recommended standards of medical care. Soc Sci Med. 1985;21:909-914. [Context Link]

34. Cowan JA. Effect of a fact sheet reminder on performance of the periodic health examination: a randomized controlled trial. Am J Prev Med. 1992; 8:104-109. Bibliographic Links [Context Link]

35. Cummings SR, Coates TJ, Richard RJ, et al. Training physicians in counselling about smoking cessation: a randomized trial of the 'Quit for Life' program. Ann Intern Med. 1989;110:640-647. Bibliographic Links [Context Link]

36. Cummings SR, Richard RJ, Duncan CL, et al. Training physicians about smoking cessation: a controlled trial in private practice. J Gen Intern Med. 1989;4:482-489. Bibliographic Links [Context Link]

37. Davidoff F, Goodspeed R, Clive J. Changing test ordering behavior: a randomized controlled trial comparing probabilistic reasoning with cost-containment education. Med Care. 1989;27:45-58. Bibliographic Links [Context Link]

38. Dickinson JC. Improving hypertension control: impact of computer feedback and physician education. Med Care. 1981;19:843-854. Bibliographic Links [Context Link]

39. Dietrich AJ. Cancer: improving early detection and prevention, a community practice randomised trial. BMJ. 1992;304:687-691. [Context Link]

40. Evans CE, Haynes RB, Birkett NJ, et al. Does a mailed continuing education program improve physician performance? results of a randomized trial in antihypertensive care. JAMA. 1986;255:501-504. Bibliographic Links [Context Link]

Page 12 of 17Ovid: Davis: JAMA, Volume 274(9).September 6, 1995.700-705

8/23/2005http://gateway.ut.ovid.com/gw2/ovidweb.cgi

41. Everett GD, deBlois CS, Chang PF, Holets T. Effect of cost education, cost audits, and faculty chart review on the use of laboratory services. Arch Intern Med. 1983;143:942-944. Bibliographic Links [Context Link]

42. Gehlbach SH, Wilkinson WE, Hammond WE, et al. Improving drug prescribing in a primary care practice. Med Care. 1984;22:193-201. Bibliographic Links [Context Link]

43. Gullion DS, Tschann JM, Adamson TE, Coates TJ. Management of hypertension in private practice: a randomized controlled trial in continuing medical education. J Contin Educ Health Profess. 1988;8:239- 255. [Context Link]

44. Headrick LA, Speroff T, Pelecanos HI, Cebul RD. Efforts to improve compliance with the National Cholesterol Education Program guidelines: results of a randomized controlled trial. Arch Intern Med. 1992;152:2490-2496. Bibliographic Links [Context Link]

45. Heale J, Davis DA, Norman GR, Woodward C, Neufeld VR, Dodd P. A randomized controlled trial assessing the impact of problem based versus didactic teaching methods in CME. Proc Annu Conf Res Med Educ. 1988;27:72-77. Bibliographic Links [Context Link]

46. Hershey CO, Goldberg HI, Cohen DI. The effect of computerized feedback coupled with a newsletter upon outpatient prescribing charges. Med Care. 1988;26:88-93. Bibliographic Links [Context Link]

47. Hershey CO, Porter DK, Breslau D, Cohen DI. Influence of simple computerized feedback on prescription charges in an ambulatory clinic. Med Care. 1986;24:472-481. Bibliographic Links [Context Link]

48. Jennett PA, Laxdal OE, Hayton RC, et al. The effects of continuing medical education on family doctor performance in office practice: a randomized control study. Med Educ. 1988;22:139-145. Bibliographic Links [Context Link]

49. Kottke TE, Brekke ML, Solberg LI, Hughes JR. A randomized trial to increase smoking intervention by physicians: doctors helping smokers, round I. JAMA. 1989;261:2101-2106. Bibliographic Links [Context Link]

50. Levinson W. The effects of two continuing medical education programs on communication skills of practicing primary care physicians. J Gen Intern Med. 1993;8:318-324. Bibliographic Links [Context Link]

51. Linn BS. Continuing medical education: impact on emergency room burn care. JAMA. 1980;244: 565- 570. Bibliographic Links [Context Link]

52. Litzelman DK, Dittus RS, Miller ME, Tierney WM. Requiring physicians to respond to computerized reminders improves their compliance with preventive care protocols. J Gen Intern Med. 1993; 8:311-317. [Context Link]

53. Lomas J, Enkin M, Anderson GM, Hannah WJ, Vayda E, Singer J. Opinion leaders vs audit and feedback to implement practice guidelines: delivery after previous cesarean section. JAMA. 1991; 265:2202-2207. Bibliographic Links [Context Link]

54. Magruder-Habib K, Zung WWK, Feussner JR. Improving physicians' recognition and treatment of depression in general medical care: result from a randomized clinical trial. Med Care. 1990;28:239-250. Bibliographic Links [Context Link]

55. Maiman LA, Becker MH, Liptak GS, Nazarian LF, Rounds KA. Improving pediatricians' compliance- enhancing practices. AJDC. 1988;142:773-779. Bibliographic Links [Context Link]

56. Manheim LM, Feinglass J, Hughes R, Martin GJ, Conrad K, Hughes EF. Training house officers to be cost conscious: effects of an educational intervention on charges and length of stay. Med Care. 1990;28:29-42.

Page 13 of 17Ovid: Davis: JAMA, Volume 274(9).September 6, 1995.700-705

8/23/2005http://gateway.ut.ovid.com/gw2/ovidweb.cgi

Bibliographic Links [Context Link]

57. Martin AR, Wolf MA, Thibodeau LA, Dau V, Braunweld E. A trial of two strategies to modify the test- ordering behavior of medical residents. N Engl J Med. 1980;303:1330-1336. Bibliographic Links [Context Link]

58. Marton KI, Tul V, Sox HC Jr. Modifying test-ordering behavior in the outpatient medical clinic: a controlled trial of two educational interventions. Arch Intern Med. 1985;145:816-821. Bibliographic Links [Context Link]

59. Mayefsky JH, Foye HR. Use of a chart audit: teaching well child care to pediatric house officers. Med Educ. 1993;27:170-174. Bibliographic Links [Context Link]

60. Mazzuca SA, Vinicor F, Einterz RM, Tierney WM, Norton JA, Kalasinski LA. Effects of the clinical environment on physicians' response to post-graduate medical education. Am Educ Res J. 1990; 27:473- 488. [Context Link]

61. McAlister NH, Covvey HD, Tong C, Lee A, Wigle ED. Randomised controlled trial of computer assisted management of hypertension in primary care. BMJ (Clin Res Ed). 1986;293:670-674. [Context Link]

62. McConnell TS, Cushing AH, Bankhurst AD, Healy JL, McIlvenna PA, Skipper BJ. Physician behavior modification using claims data: tetracycline for upper respiratory infection. West J Med. 1982;137:448- 450. Bibliographic Links [Context Link]

63. McDonald CJ. Use of a computer to detect and respond to clinical events: its effect on clinician behaviour. Ann Intern Med. 1976;84:162-167. Bibliographic Links [Context Link]

64. McDonald CJ. Protocol-based computer reminders, the quality of care and the non-perfectability of man. N Engl J Med. 1976;295:1351-1355. [Context Link]

65. McDonald CJ, Hui SL, Smith DM, et al. Reminders to physicians from an introspective computer medical record. Ann Intern Med. 1984;100: 130-138. Bibliographic Links [Context Link]

66. McDonald CJ, Wilson GA, McCabe GP Jr. Physician response to computer reminders. JAMA. 1980; 244:1579-1581. Bibliographic Links [Context Link]

67. McDowell I, Newell C, Rosser W. Computerized reminders to encourage cervical screening in family practice. J Fam Pract. 1989;28:420-424. Bibliographic Links [Context Link]

68. McPhee SJ, Bird JA, Fordham D, Rodnick JE, Osborn EH. Promoting cancer prevention activities by primary care physicians. JAMA. 1991;266: 538-544. Bibliographic Links [Context Link]

69. McPhee SJ, Bird JA, Jenkins CN, Fordham D. Promoting cancer screening: a randomized, controlled trial of three interventions. Arch Intern Med. 1989;149:1866-1872. Bibliographic Links [Context Link]

70. Meyer TJ, Van Kooten D, Marsh S, Prochazka AV. Reduction of polypharmacy by feedback to clinicians. J Gen Intern Med. 1991;6:133-135. [Context Link]

71. Morgan M. Computerized concurrent review of prenatal care. Q Rev Bull. 1978;4:33-36. [Context Link]

72. Norton PG, Dempsey LJ. Self-audit: its effect on quality of care. J Fam Pract. 1985;21:289-291. Bibliographic Links [Context Link]

73. Palmer RH, Louis TA, Hsu L, et al. A randomized controlled trial of quality assurance in sixteen ambulatory care practices. Med Care. 1985; 23:751-770. Bibliographic Links [Context Link]

Page 14 of 17Ovid: Davis: JAMA, Volume 274(9).September 6, 1995.700-705

8/23/2005http://gateway.ut.ovid.com/gw2/ovidweb.cgi

74. Perera DR, LoGerfo JP, Shulenberger E, Ylvisaker JT, Kirz HL. Teaching sigmoidoscopy to primary care physicians: a controlled study of continuing medical education. J Fam Pract. 1983;16:785-788. Bibliographic Links [Context Link]

75. Pierce M, Lundy S, Palanisamy A, Winning S, King J. Prospective randomised controlled trial of methods of call and recall for cervical cytology screening. BMJ. 1989;299:160-162. [Context Link]

76. Pinkerton RE, Tinanoff N, Willms JL, Tapp JT. Resident physician performance in a continuing education format: does newly acquired knowledge improve patient care? JAMA. 1980;244:2183-2185. Bibliographic Links [Context Link]

77. Pollack MM, Getson PR. Pediatric critical care cost containment: combined actuarial and clinical program. Crit Care Med. 1991;19:12-20. Bibliographic Links [Context Link]

78. Pozen MW, D'Agostino RB, Mitchell JB, et al. The usefulness of a predictive instrument to reduce inappropriate admissions to the coronary care unit. Ann Intern Med. 1980;92:238-242. Bibliographic Links [Context Link]

79. Pozen MW, D'Agostino RB, Selker HP, Sytkowski PA, Hood WB Jr. A predictive instrument to improve coronary-care-unit admission practices in acute ischemic heart disease: a prospective multicenter clinical trial. N Engl J Med. 1984;310:1273-1278. Bibliographic Links [Context Link]

80. Pugh JA, Frazier LM, DeLong E, Wallace AG, Ellenbogen P, Linfors E. Effect of daily charge feedback on inpatient charges and physician knowledge and behavior. Arch Intern Med. 1989;149:426-429. Bibliographic Links [Context Link]

81. Putnam RW, Curry L. Impact of patient care appraisal on physician behaviour in the office setting. Can Med Assoc J. 1985;132:1025-1029. Bibliographic Links [Context Link]

82. Putnam RW, Curry L. Physicians' participation in establishing criteria for hypertension management in the office: will patient outcomes be improved? Can Med Assoc J. 1989;140:806-809. Bibliographic Links [Context Link]

83. Rabin DL, Boekeloo BO, Marx ES, Bowman MA, Russell NK, Willis AG. Improving office-based physicians' prevention practices for sexually transmitted diseases. Ann Intern Med. 1994;121:513-519. Ovid Full Text Bibliographic Links [Context Link]

84. Raisch DW, Bootman JL, Larson LN, McGhan WF. Improving antiulcer agent prescribing in a health maintenance organization. Am J Hosp Pharm. 1990;47:1766-1773. Bibliographic Links [Context Link]

85. Restuccia JD. The effect of concurrent feedback in reducing inappropriate hospital utilization. Med Care. 1982;20:46-62. Bibliographic Links [Context Link]

86. Rogers JL. Medical information systems: assessing impact in areas of hypertension, obesity, and renal disease. Med Care. 1982;20:63-74. Bibliographic Links [Context Link]

87. Rosser WW, McDowell I, Newell C. Use of reminders for preventive procedures in family medicine. Can Med Assoc J. 1991;145:807-814. Bibliographic Links [Context Link]

88. Rubenstein LV, Calkins DR, Young RT, et al. Improving patient function: a randomized trial of functional disability screening. Ann Intern Med. 1989;111:836-842. [Context Link]

89. Sanazaro PJ, Worth RM. Concurrent quality assurance in hospital care: report of a study by private initiative in PSRO. N Engl J Med. 1978; 298:1171-1177. Bibliographic Links [Context Link]

90. Sibley JC, Sackett DL, Neufeld V, Gerrard B, Rudnick KV, Fraser W. A randomized trial of continuing

Page 15 of 17Ovid: Davis: JAMA, Volume 274(9).September 6, 1995.700-705

8/23/2005http://gateway.ut.ovid.com/gw2/ovidweb.cgi

medical education. N Engl J Med. 1982; 306:511-515. Bibliographic Links [Context Link]

91. Soumerai SB, Salem-Schatz S, Avorn J, Casteris CS, Ross-Degnan D, Popovsky MA. A controlled trial of educational outreach to improve blood transfusion practice. JAMA. 1993;270:961-966. Ovid Full Text Bibliographic Links [Context Link]

92. Steele MA, Bess DT, Franse VL, Graber SE. Cost effectiveness of two interventions for reducing outpatient prescribing costs. DICP. 1989;23: 497-500. Bibliographic Links [Context Link]

93. Stross JK. Evaluation of a continuing education program in rheumatoid arthritis. Arthritis Rheum. 1980;23:846-849. Bibliographic Links [Context Link]

94. Stross JK, Bole GG. Evaluation of an educational program for primary care practitioners, on the management of osteoarthritis. Arthritis Rheum. 1985;28:108-111. [Context Link]

95. Stross JK, Hiss RG, Watts CM, Davis WK, MacDonald R. Continuing education in pulmonary disease for primary-care physicians. Am Rev Respir Dis. 1983;127:739-746. Bibliographic Links [Context Link]

96. Sulmasy DP, Geller G, Faden R, Levine DM. The quality of mercy: caring for patients with 'do not resuscitate' orders. JAMA. 1992;267:682-686. Bibliographic Links [Context Link]

97. Szczepura A, Wilmot J, Davies C, Fletcher J. Effectiveness and cost of different strategies for information feedback in general practice. Br J Gen Pract. 1994;44:19-24. Bibliographic Links [Context Link]

98. Tape TG, Campbell JR. Computerized medical records and preventive health care: success depends on many factors. Am J Med. 1993;94:619-625. Bibliographic Links [Context Link]

99. Tierney WM, Hui SL, McDonald CJ. Delayed feedback of physician performance versus immediate reminders to perform preventive care. Med Care. 1986;24:659-666. Bibliographic Links [Context Link]

100. Tierney WM, McDonald CJ, Hui SL, Martin DK. Computer predictions of abnormal test results: effects on outpatient testing. JAMA. 1988;259:1194-1198. Bibliographic Links [Context Link]

101. Tierney WM, Miller ME, McDonald CJ. The effect on test ordering of informing physicians of the charges for outpatient diagnostic tests. N Engl J Med. 1990;322:1499-1504. Bibliographic Links [Context Link]

102. Turner RC, Waivers LE, O'Brien K. The effect of patient-carried reminder cards on the performance of health maintenance measures. Arch Intern Med. 1990;150:645-647. Bibliographic Links [Context Link]

103. Vinicor F, Cohen SJ, Mazzuca SA, et al. Diabetes: a randomized trial of the effects of physician and/or patient education on diabetes patient outcomes. J Chronic Dis. 1987;40:345-356. Bibliographic Links [Context Link]

104. Weingarten MA, Bazel D, Shannon HS. Computerized protocol for preventive medicine: a controlled self-audit in family practice. Fam Pract. 1989;6:120-124. Bibliographic Links [Context Link]

105. Wexler JR. Impact of a system of computer-assisted diagnosis. AJDC. 1975;129:203-205. Bibliographic Links [Context Link]

106. White CW, Albanese MA, Brown DD, Caplan RM. The effectiveness of continuing medical education in changing the behaviour of physicians caring for patients with acute myocardial infarction. Ann Intern Med. 1985;102:686-692. Bibliographic Links [Context Link]

107. White PT, Pharoah CA, Anderson HR, Freeling P. Randomized controlled trial of small group education on the outcome of chronic asthma in general practice. J R Coll Gen Pract. 1989;39:182-186. [Context Link]

Page 16 of 17Ovid: Davis: JAMA, Volume 274(9).September 6, 1995.700-705

8/23/2005http://gateway.ut.ovid.com/gw2/ovidweb.cgi

108. Wigton RS, Zimmer JL, Wigton JH, Patil KD. Chart reminders in the diagnosis of anemia. JAMA. 1981;245:1745-1747. Bibliographic Links [Context Link]

109. Wilson DM, Taylor DW, Gilbert JR, et al. A randomized trial of a family physician intervention for smoking cessation. JAMA. 1988;260:1570-1574. Bibliographic Links [Context Link]

110. Winickoff RN, Coltin KL, Morgan MM, Buxbaum RC, Barnett GO. Improving physician performance through peer comparison feedback. Med Care. 1984;22:527-534. Bibliographic Links [Context Link]

111. Winickoff RN, Wilner S, Neisuler R, Barnett GO. Limitation of provider interventions in hypertension quality assurance. Am J Public Health. 1985;75:43-46. Bibliographic Links [Context Link]

112. Wones RG. Failure of low-cost audits with feedback to reduce laboratory test utilization. Med Care. 1987;25:78-82. Bibliographic Links [Context Link]

113. Oxman AD, Guyatt GH. A consumers guide to subgroup analyses. Ann Intern Med. 1992;116:78-84. Bibliographic Links [Context Link]

114. American Medical Association. Physician Characteristics and Distribution in the USA. Chicago, Ill: American Medical Association; January 1, 1995. [Context Link]

115. Evidence-Based Medicine Working Group. Evidence-based medicine: a new approach to teaching the practice of medicine. JAMA. 1992;268:2420-2425. Bibliographic Links [Context Link]

116. Green L, Kreuter M, Deeds S, Partridge K. Health Education Planning: A Diagnostic Approach. Palo Alto, Calif: Mayfield Press; 1980. [Context Link]

117. Rogers EM. Diffusion of Innovation. New York, NY: The Free Press; 1983. [Context Link]

118. Nowlen PM. A New Approach to Continuing Education for Business and the Professions. New York, NY: Macmillan; 1988. [Context Link]

119. Bandura A. Social Foundations of Thought and Action: A Social Cognitive Theory. Englewood Cliffs, NJ: Prentice-Hall; 1986. [Context Link]

120. Fox RD, Mazmanian PE, Putnam RW, Changing and Learning in the Lives of Physicians. New York, NY: Praeger; 1989. [Context Link]

121. American Medical Association. The CME Directory. Chicago, Ill: American Medical Association; 1994. [Context Link]

122. Freemantle N, Grilli R, Grimshaw J, Oxman AD. The Cochrane Collaboration on Effective Professional Practice. Q Health Care. 1995;4:45-47. [Context Link]

Accession Number: 00005407-199509060-00017

Page 17 of 17Ovid: Davis: JAMA, Volume 274(9).September 6, 1995.700-705

8/23/2005http://gateway.ut.ovid.com/gw2/ovidweb.cgi