Wrap Up

EffectivenessofComputerizedDecisionSupportSystemsLinkedtoElectronicHealthRecords_ASystematicReviewandMeta-Analysis.pdf

Home >Information Systems homework help >Wrap Up

Effectiveness of Computerized Decision Support Systems LinkedtoElectronicHealthRecords:ASystematicReview and Meta-Analysis

We systematically reviewed

randomized controlled tri-

als (RCTs) assessing the ef-

fectivenessofcomputerized

decision support systems

(CDSSs) featuring rule- or

algorithm-based software

integrated with electronic

health records (EHRs) and

evidence-based knowledge.

We searched MEDLINE,

EMBASE, Cochrane Central

als, and Cochrane Database

of Abstracts of Reviews of

Effects. Information on sys-

tem design, capabilities, ac-

quisition, implementation

context, and effects on mortal-

ity, morbidity, and economic

outcomes were extracted.

Twenty-eight RCTs were

included. CDSS use did not

affect mortality (16 trials,

37395 patients; 2282 deaths;

risk ratio [RR] = 0.96; 95% con-

fidence interval [CI] = 0.85,

1.08; I2 = 41%).A statistically

significant effect was evi-

dent in the prevention of mor-

bidity, any disease (9 RCTs;

13868patients;RR = 0.82;95%

CI = 0.68, 0.99; I2 = 64%), but

selectiveoutcomereporting

or publication bias cannot

be excluded. We observed

differences for costs and

health service utilization, al-

though these were often

small in magnitude.

Across clinical settings,

new generation CDSSs in-

tegrated with EHRs do not

affect mortality and might

moderately improve morbid-

ity outcomes. (Am J Public

Health.2014;104:e12–e22.doi:

10.2105/AJPH.2014.302164)

Lorenzo Moja, MD, MSc, PhD, Koren H. Kwag, BSc, MSc, Theodore Lytras, MD, MPH, Lorenzo Bertizzolo, MD, Linn Brandt, MD, Valentina Pecoraro, BSc, Giulio Rigon, MD, MSc, Alberto Vaona, MD, MSc, Francesca Ruggiero, BA, MA, Massimo Mangia, Alfonso Iorio, MD, PhD, Ilkka Kunnamo, MD, PhD, and Stefanos Bonovas, MD, MSc, PhD

THE QUALITY OF MEDICAL

care is variable and often subop- timal across health care systems.1

Despite the growing availability of knowledge from randomized con- trolled trials (RCTs) and system- atic reviews to guide clinical prac- tice, there remains a discrepancy in the application of evidence into health care services.2 Current re- search demonstrates the potential of computerized decision support systems (CDSSs) to assist with problems raised in clinical prac- tice, increase clinician adherence to guideline- or protocol-based care, and, ultimately, improve the overall efficiency and quality of health care delivery systems.1,3,4

CDSSs have been additionally shown to increase the use of pre- ventive care in hospitalized pa- tients, facilitate communication between providers and patients, enable faster and more accurate access to medical record data, improve the quality and safety of medication prescribing, and decrease the rate of prescription errors.5---9 A recent study esti- mated that the adoption of Com- puterized Physician Order Entry and Clinical Decision Support could prevent 100 000 inpatient adverse drug events (ADEs) per year, resulting in increased inpa- tient bed availability by more than 700 000 bed-days and opportu- nity savings approaching €300 million in the studied European Union member states (i.e., the Czech Republic, France, the Netherlands,

Sweden, Spain, and the United Kingdom).10

Electronic Health Records (EHRs) represent another innova- tion that is gaining momentum in health care systems. In the United States, the use of EHRs is encour- aged by the $27 billion allocated in reimbursement incentives by the 2009 Health Information Technology for Economic and Clinical Health (HITECH) Act. Under the Act, clinicians and hos- pitals must demonstrate “mean- ingful use” of EHRs by adhering to a set of criteria, which includes the implementation of clinical de- cision support rules relevant to a specialty or high priority hospital condition such as diagnostic test ordering.11 The integration of CDSSs with EHRs through the delivery of guidance messages to health care professionals at the point of care may maximize the impact of both innovations.

A primary barrier to successful CDSS evaluation is its broad defini- tion adopted by the research com- munity, which encompasses a di- verse range of interventions and functions (see the box on page e2). The inclusion of studies with var- iable interventions across diverse health care settings precluded systematic reviews from reaching a decisive understanding of the impact of CDSSs.9,12---14 To address this issue, we conducted a system- atic review to rigorously evaluate the impact of CDSSs linked to EHRs on critical outcomes—mortality,

morbidity, and costs—and adopted a narrow definition of the inter- vention to facilitate its coherent and accurate evaluation.

METHODS

Our study protocol18 is regis- tered on PROSPERO: the inter- national prospective register of systematic reviews (ID: 2014: CRD42014007177). This work was performed in accordance with the PRISMA statement for report- ing systematic reviews and meta- analyses of studies that evaluate health care interventions.19

Eligibility Criteria

Population. Postgraduate health professionals (medical, nursing, and allied health) in primary, sec- ondary, and tertiary care settings. Only interventions that were im- plemented in real, nonsimulated, clinical settings were considered. Types of interventions. We adap-

ted the definition of a CDSS by Haynes et al.20 and Eberhardt et al.21 We defined a CDSS as an information system aimed to support clinical decision-making, linking patient-specific informa- tion in EHRs with evidence-based knowledge to generate case-specific guidance messages through a rule- or algorithm-based software. Our inclusion criteria emphasize the implementation of evidence- based medicine, meaning that computer-generated guidance messages had to be based on

SYSTEMATIC REVIEW

e12 | Systematic Review | Peer Reviewed | Moja et al. American Journal of Public Health | December 2014, Vol 104, No. 12

literature or a priori evidence (e.g., guidelines or point-of-care ser- vices) and not on expert opinions. This knowledge had to then be delivered to medical doctors or allied health care professionals through electronic media (e.g., computer, smartphone, or tablet). We did not exclude a CDSS, however, based on the degree of literature it covered in the litera- ture surveillance system. In other words, we included a CDSS if it integrated a single evidence-based

guideline or incorporated multiple evidence-based guidelines. We also included CDSSs irrespective of the level of patient information archived in the EHR.

Systems that alter the guidance based on previous experience or average behaviors were excluded.

We included software guidance messages, irrespective of the form (e.g., recommendations, alerts, prompts, or reminders), as well as guidance messages, regardless of the target assistance (e.g., diagnostic

test ordering and interpretation, treatment planning, therapy rec- ommendations, primary pre- ventive care, therapeutic drug monitoring and dosing, drug pre- scribing, or chronic disease man- agement). Patient-specific infor- mation had to derive from EHRs. Our operational definitions for considering a study “compliant” with the EHR were inclusive: from clinical data repository and health data repository (CDHR), to electronic medical---patient

record (EMR and EPR), and EHR.22

Our inclusion criteria match the “6S” Haynes’ model for evidence- based literature products23 and the evolution of online point- of-care services.24 The box below describes, in detail, the character- istics of the CDSSs we evaluated. Types of comparison groups. To

address our objectives, we con- sidered the following compari- sons: access to CDSSs according to our definition compared with (1)

Definitions of Computerized Decision Support Systems (CDSSs) Adopted by Authors of Other Systematic Reviews

Bates et al. 15(p524)

(and later adopted by Ash et al. 4(p980)

) defined a CDSS as a computer-based system providing “passive and active referential information as well as reminders, alerts, and guidelines.”

Kawamoto et al.16(p1) (and later adopted by Bright et al.9(p29)) identified a CDSS as “any electronic system designed to aid directly in clinical decision making, in which characteristics of individual

patients are used to generate patient-specific assessments or recommendations that are then presented to clinicians for consideration.”

Payne17(p47S) classified CDSSs as “computer applications designed to aid clinicians in making diagnostic and therapeutic decisions in patient care.”

Characteristics of Computerized Decision Support Systems (CDSSs)

Implementation strategy

Channel Electronic-based

Sharing Local application, networked, or Web applications

Type of device Local personal computer or handheld device

Computational architecture CDSS built into local EHR, knowledge available from central repository, entire system housed outside local site, clouding system

Information

Nature Knowledge-based

Provider Contents provided by national/international publisher, professional society, health care organization, or governmental agency

EBM methodology General references, specific guidelines for a given clinical condition, suggestions considering a patient’s unique clinical data, list of possible diagnoses, drug

interaction alerts, or preventive care reminders

Format: delivery form Messages reminders, prompts, alerts, algorithms, recommendations, rules, order sets, warnings, data reports, and dashboards

Target

Targeted setting Primary, secondary, or tertiary

Target expertise Preventive care (e.g., immunization, screening, or disease management guidelines for secondary prevention)

Diagnosis (e.g., suggestions for possible diagnoses that match a patient’s signs and symptoms)

Planning or implementing treatment (e.g., guidelines for specific diagnoses, drug dosage recommendations, or warnings for drug interactions).

Follow-up management (e.g., corollary orders, reminders for ADE monitoring)

Hospital, provider efficiency (e.g., care plans to minimize length of stay)

Cost reductions and improved patient convenience (e.g., duplicate testing alerts or drug formulary guidelines)

Overall goals Improved overall efficiency, early disease identification, accurate diagnosis, adherence of treatment to protocols, or prevention of ADEs

Time

Timing Immediately at the point of care, before the patient encounter, after the patient encounter, or at any time

Type of presentation “Automatic” (key issues: timing, autonomy and user control over response)

“On demand” (key issues: speed, ease of access, autonomy and user control over response)

Person: health professional Physicians, nurses, or allied health professionals

Note. ADE = adverse drug event; EBM = evidence-based medicine; EHR = electronic health record.

SYSTEMATIC REVIEW

December 2014, Vol 104, No. 12 | American Journal of Public Health Moja et al. | Peer Reviewed | Systematic Review | e13

standard care with no access to CDSSs, (2) CDSSs that do not generate advice, or (3) CDSSs that are not based on evidence. Trials comparing arms accessing the same CDSS at different intensities (e.g., one arm having guidance messages pushed to the health professional vs another arm hav- ing guidance message statically available in a folder) were not pooled together with the other trials in the quantitative analyses. Types of outcomes and assessment

measures. We identified a priori the following (primary) outcome measures for included studies:

1. Mortality: We selected mor- tality as it is the most relevant and objective outcome, al- though there may exist vari- ability across studies with regards to the time frame during which mortality is captured.

2. Morbidity: We selected and grouped objective patient outcomes such as occurrence of illness (e.g., pneumonia, myocardial infarction, stroke), progression of diseases and hospitalizations.

3. Economic outcomes: Infor- mation about health care uti- lization (e.g., length of stay, emergency department visits, and primary care consulta- tions), and costs.

We did not consider the fol- lowing outcomes: patient satisfac- tion, measures of process, and health care professional activity or performance (e.g., adherence to guidelines, rates of screening and other preventive measures, provi- sion of counseling, rates of appro- priate drug administration, and identification of at-risk behaviors). Types of studies. To be eligible,

studies had to be randomized controlled trials (RCTs). Ran- domization was allowed to be

either at the individual- or at the cluster-level.

Data Sources

We systematically searched the English-language literature indexed in MEDLINE, EMBASE, Cochrane Central Register of Controlled Trials, and Cochrane Database of Abstracts of Reviews of Effects. Studies found in the bibliographies of Systematic Reviews on CDSSs, as well as those identified by experts, were also considered. The full search strategies for MEDLINE and EMBASE are included in the Appendix.

Study Selection and Data

Extraction

We identified RCTs of the CDSSs fulfilling the aforementioned eligi- bility criteria. We combined the results into a reference manage- ment software program (EndNote X5 for Windows, Thomson Reu- ters, Philadelphia, PA). The data- base was filtered for duplications to derive a unique set of records. Investigators (K. H. K., T. L., L. B., L. B., V. P., G. R., A. V., and S. B.) independently examined the search results and screened the titles and abstracts; the full text reports of all potentially relevant trials were subsequently screened. Investiga- tors (K. H. K., T. L., L. B., L. B., V. P., G. R., A. V., and S. B.) indepen- dently abstracted information on CDSS characteristics and effect estimates from all included trials using a modified version of The Cochrane Effective Practice and Organisation of Care Review Group (EPOC) data collection checklist: study setting and methods (design), comparators, computer- ized CDSS characteristics, patient or provider characteristics, and outcomes. We performed all steps in the study selection and data extraction processes in duplicate.

When necessary, we attempted to contact the study authors to clarify uncertainties in the study design or results.

Risk of Bias Assessment

Two investigators (K. H. K., L. M.) assessed the potential risk for bias in included studies using the criteria outlined in the Cochrane Handbook for Systematic Reviews of Interventions.25 The assess- ment involved the following key domains: sequence generation, al- location concealment, blinding of outcome assessors, incomplete outcome data, selective outcome reporting, and other sources of bias (e.g., extreme baseline imbal- ance or failure to disclose source of funding for the study). We did not assess the blinding of person- nel and participants given the na- ture of the intervention. In fact, the use of masking procedures to prevent personnel and partici- pants from knowing the allocation to the intervention or control arms was impractical. Furthermore, blinding does not affect mortality, an outcome of this review. Our assessment referred only to stud- ies reporting mortality or morbid- ity outcomes. Any disagreement was resolved by discussion or by the involvement of a third inves- tigator (S. B.).

Data Synthesis

Risk ratios and 95% confidence intervals (CIs) were calculated for each trial by reconstructing con- tingency tables based on the num- ber of patients randomly assigned and the number of patients with the outcome of interest (analysis in accordance with the intention- to-treat principle). For the cluster- randomized trials, to calculate adjusted (inflated) CIs that ac- count for the clustering, we per- formed an approximate analysis as recommended in the Cochrane

Handbook.25 Our approach was to multiply the standard error of the effect estimate (from the anal- ysis ignoring the clustering) by the square root of the design effect.25 For this, we used an intracluster correlation coefficient (ICC = 0.027) borrowed from an external source.26 Then, each meta-analysis was performed twice, assuming either a fixed- effects27 or a random-effects model.28 In the absence of het- erogeneity, the fixed-effects and the random-effects models pro- vide similar results. When hetero- geneity is found, the random- effects model is considered to be more appropriate, although both models may be biased.29

For all statistical analyses we used the R software environ- ment,30 version 3.0.1, and the “meta” package for R,31 version 2.3---0. Selective outcome reporting or publication bias was assessed using the Begg and Mazumdar adjusted rank correlation test32

and the Egger regression asym- metry test.33 To evaluate whether the results of the studies were homogeneous, we used the Cochran Q test with a 0.10 level of signifi- cance.34 We also calculated the I2 statistic35 that describes the percentage variation across stud- ies that is attributed to heteroge- neity rather than chance. We regarded an I2 value less than 40% as indicative of “not impor- tant heterogeneity” and a value higher than 75% as indicative of “considerable heterogeneity.”25

To evaluate the stability of the results, we also performed a “leave-one-out” sensitivity analy- sis. The scope of this approach was to evaluate the influence of indi- vidual studies by estimating the summary relative risk in the ab- sence of each study.36 All P values are 2-tailed. For all tests (except for heterogeneity), a probability

SYSTEMATIC REVIEW

e14 | Systematic Review | Peer Reviewed | Moja et al. American Journal of Public Health | December 2014, Vol 104, No. 12

level less than .05 was considered statistically significant.

RESULTS

The results of our search and selection process are presented in Figure 1. We identified 28 RCTs, which met the predefined inclu- sion criteria.37---64 Eighteen studies reported mortality or morbidity data37---54 and were included in the meta-analyses, while 10 more studies reported only economic outcomes.55---64 A description of the RCTs is provided in the Appendix (available as a supplement to this article at http://www.ajph.org).

Risk of Bias in Studies

Included in the Meta-Analyses

Overall, the assessment of the 18 studies incorporated in the meta-analyses indicated high risk of bias across 7 (39%) and unclear risk for 10 studies (56%). Only 1 study44 (5%) was judged to be at low risk for bias. We noticed that the majority of trials did not measure mortality as an outcome, but reported it as additional in- formation, often as a reason for loss to follow-up. Readers should be aware that our risk of bias as- sessment did not evaluate studies based on their intended outcomes, but according to 2 outcomes of

our systematic review: mortality and morbidity. Quality assessment items are summarized in Figure 2.

Meta-Analysis of Mortality

Outcomes

Sixteen RCTs contributed to this analysis.37---52 A total of 37 395 individuals participated in these trials: 18 848 in the inter- vention groups and 18 547 in the control groups. Seven tri- als37,41,42,44,47,48,50 reported a lower mortality in the intervention group, while 8 trials38---40,42,46,49,51,52

reported a higher mortality. Only 3 were statistically signifi- cant.44,46,47 The overall mortality

rate on all 16 RCTs was 6.2% in the intervention groups (1171 deaths) and 6.0% in the control groups (1111 deaths). The pooled effect estimate was not statistically significant assuming either a fixed effects model (RR =1.00; 95% CI = 0.92, 1.08), or a random effects model (RR = 0.96; 95% CI = 0.85, 1.08). Figure 3 shows the forest plot of the RR estimates and 95% CIs from the individual trials and the pooled results. The Cochran Q test had a P value of .047 and the corresponding I2

statistic was 41%, both indicating moderate variability between studies. Visual inspection of the funnel plot (Figure 4a) indicated that pooled data did not appear to be heavily influenced by publi- cation bias, although it is also possible that few studies are “missing” from the area of non- significance. The P values for the tests of Begg and Egger were P = .96 and P = .29, respectively, also suggesting a low probability of publication bias. The “leave- one-out” sensitivity analysis, re- moving a study at a time (Figure 5), confirmed the stability of our results.

Meta-Analysis of Morbidity

Outcomes

Nine RCTs contributed to this analysis.40,42,44,45,49,51---54 A total of 13 868 individuals participated in these trials. The analysis re- vealed a weak inverse association between CDSS use and morbidity from any disease. The difference between the CDSS and control groups in the occurrence of mor- bidity outcomes was marginally significant assuming a random- effects model (RR = 0.82; 95% CI = 0.68, 0.99), but not signifi- cant assuming a fixed-effects model (RR = 0.91; 95% CI = 0.83, 1.00). Figure 3 shows the forest plot of the RR estimates and 95%

Additional records identified through bibliographies of systematic reviews

(n = 78)

Records identified through MEDLINE, EMBASE, CENTRAL, and DARE database searches

(n = 10 100)

(n = 9484) Records after duplicates removed

(n = 9484) Records screened

Records excluded based on title/abstract (n = 9238)

(n = 246) assessed for eligibility

Full-text articles

Id e

n ti

fi c

a ti

o n

S c

re e

n in

g E

li g

ib il

it y

In c

lu d

e d

Studies included in quantitative synthesis Total RCTs (n = 28)

- Morbidity (n = 9) - Mortality (n = 16) - Economic outcomes (n = 17)

- No access to full text (n = 10)

- Not RCT (n = 68) - Simulation study (n = 7) - Not health professional (n = 3) - Not CDSS assessment (n = 3)

- No acceptable comparator (n = 7) - Overlapping studies (n = 7) - Systematic reviews (n = 2)

- Not CDSS according to our criteria (n = 68) - Complex intervention where effect of CDSS could not be separated out (n = 6)

Studies excluded (n = 171)

Eligible studies evaluated for outcomes of interest

(n = 75)

Note. CDSS = computerized decision support systems; RCT = randomized controlled trial

FIGURE 1—Summary of evidence search and selection.

SYSTEMATIC REVIEW

December 2014, Vol 104, No. 12 | American Journal of Public Health Moja et al. | Peer Reviewed | Systematic Review | e15

CIs from the individual trials and the pooled results. The Cochran’s Q test had a P value of .005 and the corresponding I2 statistic was 64%, both indicating substantial variability between studies. Visual inspection of the funnel plot (Fig- ure 4b) indicated slight asymme- try, with relatively few studies existing midway in the area of nonsignificance. The P values for the Begg and the Egger’s tests were P = .18 and P = .07, respec- tively, suggesting the possible ex- istence of selective outcome re- porting bias or small study effects. The sensitivity analysis confirmed that the pooled estimates were fairly unstable (Figure 5).

Qualitative Assessment of

Economic Outcomes

Seventeen RCTs reported eco- nomic outcomes.41---43,45,46,50,53,55---64

Three of these46,50,59 presented the economic data in separate publications.65---67 Differences were seen for costs and health service utilization (e.g., drug or test orders), but these were often small in magnitude. Across economic outcomes, interventions equipped with CDSSs did not consistently perform better than nonequipped ones. Data regarding the impact of CDSSs on cost and health services utilization are given in Table 1.

DISCUSSION

This systematic review of 28 RCTs revealed little evidence for a difference in mortality when pooling results from comparisons of adoption of a CDSS integrated with an EHR versus health care settings without a CDSS. Our re- view indicates that differences in mortality outcomes, if they exist, appear small across studies and health care services, and may exist only in particular settings with specific diseases and circumstances.

However, most of the studies were underpowered and too short to prove or exclude an effect on mortality, and effects as large as a 25% increase or reduction could still be possible. We found weak evidence that an active CDSS is associated with a lower risk for morbidity. All morbidity outcomes selected were relevant from a clin- ical and health services perspec- tive. Again, results on morbidity outcomes were very diverse, lim- iting quantitative inferences; how- ever, the summary RR morbidity decrease of 10% to 18% places CDSSs linked to EHRs at the top of the spectrum of quality improve- ment interventions for their po- tential impact on health outcomes. The beneficial effects of CDSSs might still be greater than that suggested by the current analysis given the limited number of actual studies providing results on hard outcomes. Finally, we observed differences for costs and health service utilization, but these were often small in magnitude.

Several other systematic re- views provided pooled estimates of the RRs for CDSSs. All reviews observed large between-study heterogeneity. This is expected given the variability in interven- tion, settings, diseases, and study designs. Despite this limitation, they concluded in favor of CDSSs. Our review exhibits several differ- ences. We adopted stricter inclu- sion criteria, selecting only CDSSs featuring a rule- or algorithm- based software integrated with EHRs and evidence-based knowl- edge. The CDSSs we included can be viewed as a second generation in terms of their technology, in- formation management, and link- age to EHRs. Furthermore, we did not include process and laboratory outcomes such as adherence to guideline recommendations or change in blood values. Analyzing

Hetlevik et al.37

Montgomery et al.38

Hetlevik et al.39

McCowan et al.53

Kucher et al.40

Paul et al.41

McGregor et al.42

Rothschild et al.43

Gurwitz et al.54

Roy et al.44

Graumlich et al.45

MacLean et al.46

Bosworth et al.47

Cleveringa et al.48

Holbrook et al.49

O’Connor et al.50

Fitzgerald et al.51

Robbins et al.52

R an

d o

m s

eq u

en ce

g en

er at

io n

A llo

ca ti

o n

c o

n ce

al m

en t

In co

m p

le te

o u

tc o

m e

d at

Fr ee

o f s

el ec

ti ve

o u

tc o

m e

re p

o rt

in g

Fr ee

o f o

th er

s o

u rc

es o

f b ia

B lin

d in

g o

f p ar

ti ci

p an

ts a

n d

p er

so n

n el

Note. Green (+) = low risk of bias; Yellow (?) = unclear risk of bias; Red (–) = high risk of

bias.

FIGURE 2—Summary of risk-of-bias assessments of the

randomized controlled trials included in the meta-analyses.

SYSTEMATIC REVIEW

e16 | Systematic Review | Peer Reviewed | Moja et al. American Journal of Public Health | December 2014, Vol 104, No. 12

estimates from process outcomes is problematic. Their relevance is questionable and the quality of the data may have been less than optimal, particularly when the data sources were administrative rather than clinical. The overlap between our review and others is

limited, as there exists approxi- mately 50% in terms of the studies and less in terms of the rough data. The results of our review comple- ment previous analyses showing that CDSSs are best oriented to directly affect process outcomes (recommendation adherence) and,

with decreasing impact, morbidity and mortality.

Several included studies were cluster-RCTs that did not report if they accounted for clustering ef- fects. Trials randomizing at the group level should not be ana- lyzed at the individual participant

level. If the clustering is ignored, P values will be artificially small. This problem might result in false positive conclusions that the CDSS has an effect when it does not. Thus, we adjusted estimates of the RRs for our data synthesis using a method that inflates variances.

Heterogeneity: I2 = 63.6%

Relative Risk (Log Scale)

RR (95% CI)

W (Fixed)

W (Random)

Hetlevik et al.37 Montgomery et al.38 Hetlevik et al.39 Kucher et al.40 Paul et al.41 McGregor et al.42 Rothschild et al.43 Roy et al.44 Graumlich et al.45 MacLean et al.46 Bosworth et al.47 Cleveringa et al.48 Holbrook et al.49 O’Connor et al.50 Fitzgerald et al.51 Robbins et al.52

Risk Ratio

0.85 (0.59, 1.23)

0.47 (0.26, 0.85) 0.59 (0.43, 0.80) 0.86 (0.68, 1.08) 1.06 (0.92, 1.23) 0.71 (0.38, 1.31) 0.98 (0.80, 1.20) 0.92 (0.55, 1.53) 0.48 (0.24, 0.97) 1.17 (0.68, 2.02)

0.82 (0.68, 0.99)

2.5% 6.7% 13.5% 16.2% 19.0%

6.4% 17.1%

8.3% 5.2% 7.6%

9.1% 16.4% 40.7%

2.3% 21.1%

3.3% 1.7% 2.9%

100%0.91 (0.83, 1.00)

0.90 (0.73, 1.12) 1.11 (0.80, 1.53) 0.94 (0.74, 1.20) 0.47 (0.29, 0.78) 1.00 (0.42, 2.36) 1.21 (1.03, 1.43) 0.50 (0.25, 0.98) 0.81 (0.49, 1.35) 1.53 (0.26, 9.14) 0.60 (0.30, 1.20) 1.15 (0.65, 2.03) 2.00 (0.50, 7.94)

1.00 (0.92, 1.08) 0.96 (0.85, 1.08)

100% 100%

100%

. . .

Fixed effect model

Random effects model

Heterogeneity: I2 = 40.5%

0.2

0.5

1.24 (0.34, 4.58) 1.18 (0.82, 1.69) 1.01 (0.87, 1.17)

4.4% 7.1% 0.9% 7.4%

15.8% 12.6%

8.4% 11.4%

4.7% 1.8%

14.8% 2.8% 4.6% 0.5% 2.7% 3.8% 0.8%

0.4% 4.7%

28.6% 13.4%

5.7% 10.4%

2.4% 0.8%

21.9% 1.3% 2.4% 0.2% 1.2% 1.9% 0.3%

McCowan et al.53 Kucher et al.40 McGregor et al.42 Gurwitz et al.54 Roy et al.44 Graumlich et al.45 Holbrook et al.49 Fitzgerald et al.51 Robbins et al.52

Note. CI = confidence interval; RR = risk ratio; W = weight. The RR and 95% CI for each study are displayed on a logarithmic scale.

FIGURE 3—Forest plots from individual studies and meta-analysis for (a) mortality, all follow-up, and (b) morbidity, any disease.

SYSTEMATIC REVIEW

December 2014, Vol 104, No. 12 | American Journal of Public Health Moja et al. | Peer Reviewed | Systematic Review | e17

However, such adjusted results should be interpreted cautiously; if the clustering effect is limited across studies, the analysis may be too conservative.

Our meta-analysis has additional limitations. We did not evaluate the quality of the evidence-based information supporting the CDSS

recommendations. We accepted study authors’ description of a CDSS as evidence-based at face value, even if the authors did not explain the source of evidence or knowledge in detail. Furthermore, the limited number of trials, espe- cially regarding the meta-analysis for the morbidity outcomes,

increases the uncertainty of the findings and conclusions. The tri- als included were conceptually heterogeneous in terms of their design, setting, participants, and interventions, as well as the defi- nition and measurement of out- comes. In addition, although our literature search was as inclusive as possible without the exclusion of studies based on methodologi- cal characteristics, the search was restricted to studies published in indexed journals. We did not search for unpublished studies or for source data. Moreover, the tri- als included in this meta-analysis were not designed to specifically analyze the relationship between mortality and CDSS use. In fact, mortality was additional informa- tion provided often as a reason behind loss to follow-up. Addi- tionally, the follow-up was too short to detect a sufficient number of deaths to show potentially rel- evant differences. Finally, we can- not exclude that pooling the mor- tality outcome across different settings (e.g., intensive care units versus primary care) could have influenced the overall result to- ward a null effect with primary care studies bearing larger weight in the meta-analysis.

The results of this review may provide sufficient evidence to fuel the debate on the prospects of CDSSs linked to EHRs. For those perceiving CDSSs as an autocratic command to doctors, our system- atic review may be interpreted as evidence that they do not affect patient mortality, on average, and should be abandoned. For those interested in CDSS dissemination, our results, which show a decrease in morbidity across all settings by one fifth, may be used as an argument to increase CDSS adop- tion within health care services. Both interpretations might be ex- aggerated as the evidence is still in

its infancy along with the technol- ogy and implementation. Many of the trials adopted locally devel- oped CDSS interventions, which may have compromised their level of integration into clinicians’ workflow. The next generation of CDSS trials should focus on sys- tems with a more global outlook featuring authoritative point-of- care services68 and full integration with EHRs. The conclusion of a landmark article by Sim et al.,69

published almost 15 years ago, still reflects the current scenario:

Although the promise of clinical decision support system-facilitated evidence-based medicine is strong, substantial work remains to be done to realize the potential benefits.69(p527)

In conclusion, our results on health care services equipped with versus health care not equipped with CDSSs suggest, in broad terms, that this technology does not result in substantial benefits or risks for patients in terms of mor- tality. This effect, when it occurs, is largely dependent on the disease and setting characteristics. Focus- ing on subgroup analyses, how- ever, can lead to misleading claims when the overall data are limited and unavoidably weak because of inherent design problems. Ef- fects on morbidity might exist and the magnitude of the effect, in the order of 10% to 20%, could be large enough to impact mor- tality if appropriate follow-up is ensured. The results of this study may provide enough evidence to advance the debate on the pros- pects of CDSSs. j

About the Authors Lorenzo Moja is with the Department of Biomedical Sciences for Health, Univer- sity of Milan, and the Unit of Clinical Epidemiology, IRCCS Orthopedic Institute Galeazzi, Milan, Italy. Koren H. Kwag is with the Unit of Clinical Epidemiology, IRCCS Orthopedic Institute Galeazzi, Milan. Theodore Lytras is with the

Kucher et al.40

Relative Risk

Kucher et al.40

0.2 0.5 1.0 2.0 5.0

St an

d ar

d E

rr o

0.8

0.6

0.4

0.2

0.0

Paul et al.41

McGregor et al.42 Hetlevik et al.37

Roy et al.44

Bosworth et al.47

MacLean et al.46

Rothschild et al.43

Hetlevik et al.39

Cleveringa et al.48 Fitzgerald et al.51

O’Connor et al.50

Graumlich et al.45

Montgomery et al.38

Robbins et al.52

Holbrook et al.49

Gurwitz et al.54

Graumlich et al.45 McGregor et al.42

Kucher et al.40

Holbrook et al.49

Robbins et al.52

McCowan et al.53 Roy et al.44

Fitzgerald et al.51

0.5 1.0 1.5

Relative Risk

0.3

0.2

0.1

0.0

St an

d ar

d E

rr o

FIGURE 4—Funnel plots of observed relative risk against standard

error for (a) mortality and (b) morbidity.

SYSTEMATIC REVIEW

e18 | Systematic Review | Peer Reviewed | Moja et al. American Journal of Public Health | December 2014, Vol 104, No. 12

Department of Epidemiological Surveil- lance and Intervention, Hellenic Centre for Disease Control and Prevention, Athens, Greece, the Centre for Research in Environ- mental Epidemiology (CREAL), Barcelona, Spain, and the Department of Experimental and Health Sciences, Universitat Pompeu Fabra, Barcelona. Lorenzo Bertizzolo and Francesca Ruggiero are with the Depart- ment of Biomedical Sciences for Health, University of Milan. Linn Brandt is with the Department of Internal Medicine, Inland Hospital Trust, Oslo, Norway, the Department of Internal Medicine, Diakonhjemmet Hospital, Oslo, and HELSAM, University of Oslo. Valentina Pecoraro is with the University of Milan.

Giulio Rigon and Alberto Vaona are with Azienda ULSS 20, Verona, Italy. Massimo Mangia is with Medilogy SRL, Milan. Alfonso Iorio is with the Department of Clinical Epidemiology and Biostatistics, McMaster University, Hamilton, Ontario. Ilkka Kunnamo is with Duodecim Medi- cal Publications Ltd, Helsinki, Finland. Stefanos Bonovas is with the Laboratory of Drug Regulatory Policies, IRCCS Mario Negri Institute for Pharmacological Re- search, Milan, and the Department of Pharmacology, School of Medicine, Univer- sity of Athens, Athens. Correspondence should be sent to Lorenzo

Moja, MD, MSc, PhD, Dr Public Health, IRCCS Orthopedic Institute Galeazzi, Via

R. Galeazzi 4, 20161 Milan, Italy (e-mail: [email protected]). Reprints can be ordered at http://www.ajph.org by clicking the “Reprints” link.

This article was accepted June 28, 2014.

Contributors All authors participated in conceptualization and design of the study, and critical revision and final approval of the article. L. Moja obtained funding. L. Moja, K. H. Kwag, T. Lytras, L. Bertizzolo, L. Brandt, V. Pecoraro, G. Rigon, A. Vaona, and S. Bonovas collected and assembled the data. L. Moja, K. H. Kwag, T. Lytras, and S. Bonovas analyzed and interpreted the

data. L. Moja, K. H. Kwag, and S. Bonovas drafted the article. L. Moja, T. Lytras, and S. Bonovas provided statistical expertise.

Acknowledgments This work was supported by the Italian Ministry of Health (GR-2009-1606736) and by Regione Lombardia (D.R.G. IX/ 4340 26/10/2012). L. Moja is employed by the IRCCS Galeazzi and Università degli Studi di Milano, which have nonexclusive contracts with commercial publishers to develop or adapt CDSSs based on criti- cally appraised studies and systematic reviews. I. Kunnamo is the Editor-in-Chief at Duodecim Medical Publications, a Finnish company owned by the Finnish Medical Society Duodecim, which de- velops the Evidence-Based Medicine electronic Decision Support (EBMeDS) service and publishes EBM Guidelines. Massimo Mangia is the Chief Executive Officer of Medilogy, an Italian company that develops and supplies MediDSS, a CDSS.

The authors would like to thank Vanna Pistotti for her support with de- veloping the search strategy.

Note. Funding sources had no role in the writing of this article or the decision to submit it for publication.

Human Participant Protection An institutional review board approval was not needed for this systematic review because data were obtained from sec- ondary sources.

References 1. Roshanov PS, Fernandes N, Wilczynski JM, et al. Features of effective computerised clinical decision support systems: meta-regression of 162 rando- mised trials. BMJ. 2013;346:f657.

2. National Research Council. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: The National Academies Press; 2001.

3. Chaudhry B, Wang J, Wu S, et al. Systematic review: impact of health in- formation technology on quality, effi- ciency, and costs of medical care. Ann Intern Med. 2006;144(10):742---752.

4. Ash JS, McCormack JL, Sittig DF, et al. Standard practices for computerized clinical decision support in community hospitals: a national survey. J Am Med Inform Assoc. 2012;19(6):980---987.

5. Kaushal R, Shojania KG, Bates DW. Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review. Arch Intern Med. 2003;163(12):1409---1416.

6. Bonnabry P, Despont-Gros C, Crauser D, et al. A risk analysis method to evaluate the impact of a computerized provider order entry system on patient safety. J Am Med Inform Assoc. 2008;15(4):453---460.

RR (95% CI)

0.96 (0.85, 1.10)

0.94 (0.82, 1.07) 0.95 (0.83, 1.09) 1.01 (0.92, 1.11) 0.95 (0.84, 1.08) 0.92 (0.82, 1.04) 0.98 (0.87, 1.10) 0.96 (0.85, 1.09) 0.95 (0.84, 1.08) 0.97 (0.86, 1.10) 0.95 (0.83, 1.08) 0.95 (0.84, 1.08)

0.86 (0.72, 1.03) 0.89 (0.75, 1.05) 0.81 (0.65, 1.01) 0.78 (0.63, 0.95) 0.83 (0.68, 1.01) 0.78 (0.62, 0.99) 0.81 (0.66, 0.99) 0.85 (0.71, 1.02) 0.80 (0.66, 0.97)

0.82 (0.68, 0.99)

0.96 (0.85, 1.08)

0.6 0.8 1 1.2

Risk Ratio

Omitting Hetlevik et al.37 Omitting Montgomery et al.38 Omitting Hetlevik et al.39 Omitting Kucher et al.40 Omitting Paul et al.41 Omitting McGregor et al.42 Omitting Rothschild et al.43 Omitting Roy et al.44 Omitting Graumlich et al.45 Omitting MacLean et al.46 Omitting Bosworth et al.47 Omitting Cleveringa et al.48 Omitting Holbrook et al.49 Omitting O’Connor et al.50 Omitting Fitzgerald et al.51 Omitting Robbins et al.52

0.95 (0.84, 1.08) 0.94 (0.82, 1.07) 0.94 (0.81, 1.09) 0.96 (0.84, 1.10)

Random effects model

Omitting McCowan et al.53

0.6 0.8 1 1.2

Omitting Kucher et al.40 Omitting McGregor et al.42 Omitting Gurwitz et al.54 Omitting Roy et al.44 Omitting Graumlich et al.45 Omitting Holbrook et al.49 Omitting Fitzgerald et al.51 Omitting Robbins et al.52

Note. CI = confidence interval; RR = risk ratio. Pooled estimates are from random-effects models with 1 study omitted at a time.

FIGURE 5—“Leave-one-out” sensitivity analyses for studies with (a) mortality outcomes and (b) morbidity

outcomes.

SYSTEMATIC REVIEW

December 2014, Vol 104, No. 12 | American Journal of Public Health Moja et al. | Peer Reviewed | Systematic Review | e19

TA B LE

1 — Im p a c t o f C o m p u te ri ze d D e c is io n S u p p o rt S ys te m s (C D S S s)

o n C o st s a n d H e a lt h S e rv ic e s U ti li za ti o n

St ud y

Im pa ct D im en si on

O ut co m es Re po rt ed

Re fe re nc e Ti m e,

M on th s

CD SS

Co st sa

Co st D iff er en ce b

Va ria bl ec

Im pa ct d

O ’C on no r et al .5 0 ,e

Co st -e ffe ct iv en es s

O ut pa tie nt an d ph ar m ac y co st s

12 Ye ar 1: 12 0;

fo llo w in g ye ar s: 76

21 69 0

$/ Q AL Y

↑

Ph ar m ac y co st s

12 D om in an t

$/ Q AL Y

↓

Ph ar m ac y co st s

12 30 17

$/ Q AL Y

↑

H ol br oo k et al .5 9 ,f

Co st -e ffe ct iv en es s

D is ea se

m an ag em en t co st s

12 18 21

15 3 16 9

$/ Q AL Y

↑

Su br am an ia n et al .5 6

Co st al lo ca tio n

D ru g or de r co st s

12 . . .

–7 .3 1

Co st /r es id en t- y

$ D ru g or de r an d la bo ra to ry te st co st s

12 . . .

–4 .7 1

Co st /r es id en t- y

$ Pa ul et al .4 1

Co st al lo ca tio n

An tib io tic

co st s

6 . . .

–6 8

Co st pe r pa tie nt

$ H ea lth

se rv ic es ut ili za tio n

Le ng th of ho sp ita l st ay

6 . . .

–0 .6 2

D ay s/ pa tie nt

↓

M cG re go r et al .4 2

Co st al lo ca tio n

An tim

ic ro bi al co st s

3 . . .

–3 7. 64

Co st /p at ie nt

$ H ea lth

se rv ic es ut ili za tio n

Le ng th of ho sp ita l st ay

3 . . .

–1 .0

D ay s/ pa tie nt

↓

Ti m e sp en t to pe rf or m in te rv en tio ns

. . .

–0 .9

Pe rs on -h ou rs /d

↓

Ab de l-K ad er et al .5 7

H ea lth

se rv ic es ut ili za tio n

Re na l re fe rr al s

12 . . .

–6 .8 %

Ab so lu te ch an ge

in re fe rr al s

↓

Re na l te st or de rs

12 . . .

9. 2%

Ab so lu te ch an ge

in re fe rr al s

↑

Re na l dr ug

or de rs

12 . . .

N o di ffe re nc e

$ Ta m bl yn et al .6 0

H ea lth

se rv ic es ut ili za tio n

Ps yc ho tr op ic dr ug

or de rs

22 . . .

–0 .0 5

D ru g un it/ pa tie nt

$ Be ll et al .5 8

H ea lth

se rv ic es ut ili za tio n

H os pi ta l vi si ts

24 . . .

0. 15

N o. /p at ie nt

↑

D ru g or de r an d la bo ra to ry /d ia gn os tic

te st s

24 . . .

M ix ed

re su lts

. . .

↑ ↓

M ac Le an

et al .4 6 ,g

Co st al lo ca tio n

H os pi ta l an d em er ge nc y ro om

co st s

1 4

–1 0. 94

Co st /p at ie nt /m o

$ M ac Le an

et al .4 6

Co st al lo ca tio n

Co st of ho sp ita l da ys an d pr im ar y/ se co nd ar y ca re vi si ts

12 . . .

–2 42 6

Co st fo r al l pa tie nt s/ y

$ Ja vi tt et al .6 1

Co st al lo ca tio n

Co st of vi si ts an d dr ug

or de rs

1 . . .

–2 4. 8

Co st pe r pa tie nt /m o

$ Ap ko n et al .6 2

Co st al lo ca tio n

Co st of vi si ts , dr ug

or de rs , an d te st s

2 . . .

91 Co st /p at ie nt

↑

Ti er ne y et al .6 3

Co st al lo ca tio n

O ut pa tie nt an d in pa tie nt ch ar ge s

12 . . .

M ix ed

re su lts

Co st /p at ie nt

↑ ↓

G ra um lic h et al .4 5

H ea lth

se rv ic es ut ili za tio n

Em er ge nc y ro om

vi si ts

6 . . .

–5 .2 %

Ab so lu te ch an ge

in vi si ts

↓

M cC ow an

et al .5 3

H ea lth

se rv ic es ut ili za tio n

H os pi ta l vi si ts

6 . . .

–3 %

Ab so lu te ch an ge

in no . pa tie nt s

↓

Pr im ar y ca re vi si ts

6 . . .

–1 9%

Ab so lu te ch an ge

in no . pa tie nt s

↓

D ru g or de rs

6 . . .

–1 9%

Ab so lu te ch an ge

in no . pa tie nt s

↓

Ro th sc hi ld et al .4 3

H ea lth

se rv ic es ut ili za tio n

Le ng th of st ay

5– 6

. . .

–1 .4 (t ot al ), –1 .8 (I CU )

D ay s/ pa tie nt

↓

Co bo s et al .6 4

Co st al lo ca tio n

Co st s fo r di se as e m an ag em en t

12 . . .

–8 2

Co st /p at ie nt

↓

D ru g or de r co st s

12 . . .

–8 0

Co st /p at ie nt

↓

G on za le s et al .5 5

H ea lth

se rv ic es ut ili za tio n

An tib io tic

dr ug

or de r

6 . . .

–1 5. 1%

% of pa tie nt s pr es cr ib ed

an tib io tic s

↓

N ot e. IC U = in te ns iv e ca re un it; Q AL Y = qu al ity -a dj us te d lif e ye ar .

a Co st s ar e re po rt ed

in U S$ , pe r pa tie nt . Fo r co st s no t or ig in al ly re po rt ed

in U S$ , th ey w er e co nv er te d to U S$

at th e ra te of Ja nu ar y 1 of th e st ud y’ s pu bl ic at io n ye ar .

b W he n th e co st to im pl em en t th e CD SS

w as

no t re po rt ed , it w as

no t ac co un te d fo r in th e co st di ffe re nc e be tw ee n th e in te rv en tio n an d co nt ro l.

c I f th e tim

e pe rio d co rr es po nd in g to th e va ria bl e w as

no t re po rt ed , it w as

as su m ed

it to be

th e du ra tio n of th e fo llo w -u p (p ro vi de d in tim

e ho riz on ).

d ↑ , in cr ea se ; ↓ , de cr ea se ; $ , no

ch an ge ; ↑ ↓ , m ix ed

im pa ct s fo r th is ou tc om e.

e E co no m ic an al ys is in G ilm er et al .6 5

f E co no m ic an al ys is in O ’R ei lly et al .6 6

g Ec on om ic an al ys is in Kh an

et al .6 7

SYSTEMATIC REVIEW

e20 | Systematic Review | Peer Reviewed | Moja et al. American Journal of Public Health | December 2014, Vol 104, No. 12

7. de Lusignan S, Chan T. The de- velopment of primary care information technology in the United Kingdom. J Ambul Care Manage. 2008;31(3):201---210.

8. Romano MJ, Stafford RS. Electronic health records and clinical decision sup- port systems: impact on national ambu- latory care quality. Arch Intern Med. 2011;171(10):897---903.

9. Bright TJ, Wong A, Dhurjati R, et al. Effect of clinical decision-support systems: a systematic review. Ann Intern Med. 2012;157(1):29---43.

10. Swedish Presidency of the EU, e-Health for a Healthier Europe! Oppor- tunities for a better use of healthcare re- sources, 2009. Available at: http://www. government.se/content/1/c6/12/98/ 15/5b63bacb.pdf. Accessed July 1, 2014.

11. Miriovsky BJ, Shulman LN, Abernethy AP. Importance of health information technology, electronic health records, and continuously aggregating data to comparative effectiveness research and learning health care. J Clin Oncol. 2012;30(34):4243---4248.

12. Jeffery R, Iserman E, Haynes RB. Can computerized clinical decision support systems improve diabetes management? A systematic review and meta-analysis. Diabet Med. 2013;30(6):739---745.

13. Roshanov PS, Misra S, Gerstein HC, et al. Computerized clinical decision sup- port systems for chronic disease manage- ment: a decision-maker-researcher part- nership systematic review. Implement Sci. 2011;6:92.

14. Souza NM, Sebaldt RJ, Mackay JA, et al. Computerized clinical decision sup- port systems for primary preventive care: a decision-maker-researcher partnership systematic review of effects on process of care and patient outcomes. Implement Sci. 2011;6:87.

15. Bates DW, Kuperman GJ, Wang S, et al. Ten commandments for effective clinical decision support: making the prac- tice of evidence-based medicine a reality. J Am Med Inform Assoc. 2003;10(6): 523---530.

16. Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical prac- tice using clinical decision support sys- tems: a systematic review of trials to identify features critical to success. BMJ. 2005;330(7494):765.

17. Payne TH. Computer decision sup- port systems. Chest. 2000;118(2 suppl): 47S---52S.

18. Effectiveness of computerized decision support systems linked to patient records: a systematic review and meta-analysis. PROSPERO 2014: CRD42014007177. Available at: http://www.crd.york.ac.uk/PROSPERO/ display_record.asp?ID=CRD42014007177. Accessed July 1, 2014.

19. Moher D, Liberati A, Tetzlaff J, Altman DG; PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med. 2009;151:264---269, W64.

20. Haynes RB, Wilczynski NL. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: methods of a decision- maker-researcher partnership systematic review. Implement Sci. 2010;5:12.

21. Eberhardt J, Bilchik A, Stojadinovic A. Clinical decision support systems: po- tential with pitfalls. J Surg Oncol. 2012; 105(5):502---510.

22. Gunter TD, Terry NP. The emer- gence of national electronic health record architectures in the United States and Australia: models, costs, and questions. J Med Internet Res. 2005;7(1):e3.

23. Dicenso A, Bayley L, Haynes RB. Accessing pre-appraised evidence: fine- tuning the 5S model into a 6S model. Evid Based Nurs. 2009;12(4):99---101.

24. Moja L, Banzi R. Navigators for medicine: evolution of online point-of- care evidence-based services. Int J Clin Pract. 2011;65(1):6---11.

25. Higgins JPT, Green S, eds. Cochrane Handbook for Systematic Reviews of In- terventions, version 5.0.1. The Cochrane Collaboration. 2011. Available at: www. cochrane-handbook.org. Accessed October 6, 2014.

26. Health Services Research Unit. Da- tabase of ICCs: Spreadsheet (empirical estimates of ICCs from changing profes- sional practice studies). Available at: http://www.abdn.ac.uk/hsru/research/ delivery/behaviour/methodological- research. Accessed July 1, 2014.

27. Mantel N, Haenszel W. Statistical aspects of the analysis of data from ret- rospective studies of disease. J Natl Cancer Inst. 1959;22:719---748.

28. DerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trials. 1986; 7:177---188.

29. Petitti DB. Statistical methods in meta-analysis. In: Petitti DB, ed. Meta- Analysis, Decision Analysis, and Cost- Effectiveness Analysis. New York: Oxford University Press; 1999.

30. R Core Team. R: a language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing; 2013. Available at: http:// www.R-project.org. Accessed July 1, 2014.

31. Schwarzer G. Meta: an R package for meta-analysis. 2013. Available at: http:// cran.r-project.org/package=meta. Accessed July 1, 2014.

32. Begg CB, Mazumdar M. Operating characteristics of a rank correlation test for publication bias. Biometrics. 1994; 50:1088---1101.

33. Egger M, Davey Smith G, Schneider M, Minder C. Bias in meta-analysis de- tected by a simple, graphical test. BMJ. 1997;315:629---634.

34. Cochran WG. The combination of estimates from different experiments. Biometrics. 1954;10:101---129.

35. Higgins JPT, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in meta-analyses. BMJ. 2003;327:557---560.

36. Tobias A. Assessing the influence of a single study in the meta-analysis esti- mate. Stata Tech Bull. 1999;8:15---17.

37. Hetlevik I, Holmen J, Krüger O. Implementing clinical guidelines in the treatment of hypertension in general practice. Evaluation of patient outcome related to implementation of a computer- based clinical decision support system. Scand J Prim Health Care. 1999;17(1): 35---40.

38. Montgomery AA, Fahey T, Peters TJ, MacIntosh C, Sharp DJ. Evaluation of computer based clinical decision support system and risk chart for management of hypertension in primary care: rando- mised controlled trial. BMJ. 2000; 320(7236):686---690.

39. Hetlevik I, Holmen J, Krüger O, Kristensen P, Iversen H, Furuseth K. Implementing clinical guidelines in the treatment of diabetes mellitus in general practice. Evaluation of effort, process, and patient outcome related to implementa- tion of a computer-based decision support system. Int J Technol Assess Health Care. 2000;16(1):210---227.

40. Kucher N, Koo S, Quiroz R, et al. Electronic alerts to prevent venous throm- boembolism among hospitalized patients. N Engl J Med. 2005;352(10):969---977.

41. Paul M, Andreassen S, Tacconelli E, et al. Improving empirical antibiotic treatment using TREAT, a computerized decision support system: cluster random- ized trial. J Antimicrob Chemother. 2006; 58(6):1238---1245.

42. McGregor JC, Weekes E, Forrest GN, et al. Impact of a computerized clinical decision support system on reducing in- appropriate antimicrobial use: a random- ized controlled trial. J Am Med Inform Assoc. 2006;13(4):378---384.

43. Rothschild JM, McGurk S, Honour M, et al. Assessment of education and computerized decision support interven- tions for improving transfusion practice. Transfusion. 2007;47(2):228---239.

44. Roy PM, Durieux P, Gillaizeau F, et al. A computerized handheld decision- support system to improve pulmonary embolism diagnosis: a randomized trial. Ann Intern Med. 2009;151(10):677---686.

45. Graumlich JF, Novotny NL, Nace GS, Aldag JC. Patient and physician percep- tions after software-assisted hospital

discharge: cluster randomized trial. J Hosp Med. 2009;4(6):356---363.

46. Maclean CD, Gagnon M, Callas P, Littenberg B. The Vermont diabetes informa- tion system: a cluster randomized trial of a population based decision support system. J Gen Intern Med. 2009;24(12):1303---1310.

47. Bosworth HB, Olsen MK, Dudley T, et al. Patient education and provider de- cision support to control blood pressure in primary care: a cluster randomized trial. Am Heart J. 2009;157(3):450---456.

48. Cleveringa FG, Welsing PM, van den Donk M, et al. Cost-effectiveness of the diabetes care protocol, a multifaceted computerized decision support diabetes management intervention that reduces cardiovascular risk. Diabetes Care. 2010; 33:258---263.

49. Holbrook A, Pullenayegum E, Thabane L, et al. Shared electronic vascu- lar risk decision support in primary care: Computerization of Medical Practices for the Enhancement of Therapeutic Effective- ness (COMPETE III) randomized trial. Arch Intern Med. 2011;171(19):1736---1744.

50. O’Connor PJ, Sperl-Hillen JM, Rush WA, et al. Impact of electronic health record clinical decision support on di- abetes care: a randomized trial. Ann Fam Med. 2011;9(1):12---21.

51. Fitzgerald M, Cameron P, Mackenzie C, et al. Trauma resuscitation errors and computer-assisted decision support. Arch Surg. 2011;146(2):218---225.

52. Robbins GK, Lester W, Johnson KL, et al. Efficacy of a clinical decision-support system in an HIV practice: a randomized trial. Ann Intern Med. 2012;157(11): 757---766.

53. McCowan C, Neville RG, Ricketts IW, et al. Lessons from a randomized controlled trial designed to evaluate com- puter decision support software to im- prove the management of asthma. Med Inform Internet Med. 2001;26(3):191---201.

54. Gurwitz JH, Field TS, Rochon P, et al. Effect of computerized provider order entry with clinical decision support on adverse drug events in the long-term care setting. J Am Geriatr Soc. 2008;56(12):2225---2233.

55. Gonzales R, Anderer T, McCulloch CE, et al. A cluster randomized trial of decision support strategies for reducing antibiotic use in acute bronchitis. JAMA Intern Med. 2013;173(4):267---273.

56. Subramanian S, Hoover S, Wagner JL, et al. Immediate financial impact of computerized clinical decision support for long-term care residents with renal in- sufficiency: a case study. J Am Med Inform Assoc. 2012;19(3):439---442.

57. Abdel-Kader K, Fischer GS, Li J, Moore CG, Hess R, Unruh ML. Auto- mated clinical reminders for primary care providers in the care of CKD: a small

SYSTEMATIC REVIEW

December 2014, Vol 104, No. 12 | American Journal of Public Health Moja et al. | Peer Reviewed | Systematic Review | e21

cluster-randomized controlled trial. Am J Kidney Dis. 2011;58(6):894---902.

58. Bell LM, Grundmeier R, Localio R, et al. Electronic health record-based de- cision support to improve asthma care: a cluster-randomized trial. Pediatrics. 2010;125(4):e770---e777.

59. Holbrook A, Thabane L, Keshavjee K, et al. Individualized electronic decision support and reminders to improve di- abetes care in the community: COMPETE II randomized trial. CMAJ. 2009;181(1-2): 37---44.

60. Tamblyn R, Eguale T, Buckeridge DL, et al. The effectiveness of a new generation of computerized drug alerts in reducing the risk of injury from drug side effects: a cluster randomized trial. J Am Med Inform Assoc. 2012;19(4):635---643.

61. Javitt JC, Rebitzer JB, Reisman L. Information technology and medical mis- steps: evidence from a randomized trial. J Health Econ. 2008;27(3):585---602.

62. Apkon M, Mattera JA, Lin Z, et al. A randomized outpatient trial of a decision- support information technology tool. Arch Intern Med. 2005;165(20):2388---2394.

63. Tierney WM, Overhage JM, Murray MD, et al. Effects of computerized guide- lines for managing heart disease in pri- mary care. J Gen Intern Med. 2003; 18(12):967---976.

64. Cobos A, Vilaseca J, Asenjo C, et al. Cost effectiveness of a clinical decision support system based on the recommen- dations of the European Society of Car- diology and other societies for the man- agement of hypercholesterolemia: report of a cluster-randomized trial. Dis Manag Health Outcomes. 2005;13(6):421---432.

65. Gilmer TP, O’Connor PJ, Sperl-Hillen JM, et al. Cost-effectiveness of an elec- tronic medical record based clinical de- cision support system. Health Serv Res. 2012;47(6):2137---2158.

66. O’Reilly D, Holbrook A, Blackhouse G, Troyan S, Goeree R. Cost-effectiveness of a shared computerized decision sup- port system for diabetes linked to elec- tronic medical records. J Am Med Inform Assoc. 2012;19(3):341---345.

67. Khan S, Maclean CD, Littenberg B. The effect of the Vermont Diabetes In- formation System on inpatient and emer- gency room use: results from a random- ized trial. Health Outcomes Res Med. 2010; 1(1):e61---e66.

68. Banzi R, Liberati A, Moschetti I, Tagliabue L, Moja L. A review of online evidence-based practice point-of-care in- formation summary providers. J Med In- ternet Res. 2010;12(3):e26.

69. Sim I, Gorman P, Greenes RA, et al. Clinical decision support systems for the practice of evidence-based medicine. J Am Med Inform Assoc. 2001;8(6):527---534.

SYSTEMATIC REVIEW

e22 | Systematic Review | Peer Reviewed | Moja et al. American Journal of Public Health | December 2014, Vol 104, No. 12

Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.