Identify a health information technology system, explain how it improves healthcare outcomes. Identify the organization you work for uses this system.
MParra
Healthinfo3.pdf
RECEIVED 10 March 2015 REVISED 28 July 2015
ACCEPTED 29 July 2015 PUBLISHED ONLINE FIRST 13 November 2015
Effects of health information technology on patient outcomes: a systematic review
Samantha K Brenner,1,2,3,4 Rainu Kaushal,1,2,4,5,6 Zachary Grinspan,1,2,5,6
Christine Joyce,5,6 Inho Kim,6,7 Rhonda J Allard,9 Diana Delgado,8 and Erika L Abramson1,2,5,6
ABSTRACT ....................................................................................................................................................
Objective To systematically review studies assessing the effects of health information technology (health IT) on patient safety outcomes. Materials and Methods The authors employed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement methods. MEDLINE, Cumulative Index to Nursing Allied Health (CINAHL), EMBASE, and Cochrane Library databases, from 2001 to June 2012, were searched. Descriptive and comparative studies were included that involved use of health IT in a clinical setting and measured effects on patient safety outcomes. Results Data on setting, subjects, information technology implemented, and type of patient safety outcomes were all abstracted. The quality of the studies was evaluated by 2 independent reviewers (scored from 0 to 10). A total of 69 studies met inclusion criteria. Quality scores ranged from 1 to 9. There were 25 (36%) studies that found benefit of health IT on direct patient safety outcomes for the primary outcome measured, 43 (62%) studies that either had non-significant or mixed findings, and 1 (1%) study for which health IT had a detrimental effect. Neither the quality of the studies nor the rate of randomized control trials performed changed over time. Most studies that demonstrated a positive benefit of health IT on di- rect patient safety outcomes were inpatient, single-center, and either cohort or observational trials studying clinical decision support or computer- ized provider order entry. Discussion and Conclusion Many areas of health IT application remain understudied and the majority of studies have non-significant or mixed findings. Our study suggests that larger, higher quality studies need to be conducted, particularly in the long-term care and ambulatory care settings.
....................................................................................................................................................
Keywords: health information technology, adverse events, patient outcomes, systematic review
Effectively harnessing the potential of health information technology (health IT) to improve patient safety, reduce harm, and improve patient outcomes remains a unifying national goal among healthcare pro- viders, patients, and regulators. For several decades, the use of com- puter systems has been considered a potential mechanism to support and improve clinical care.1 Through the Medicare and Medicaid Electronic Health Record (EHR) Incentive Program, known as the meaningful use program, the federal government is investing billions of dollars to promote the adoption of health IT in order to improve pa- tient outcomes.2 Rates of health IT adoption in the inpatient and outpa- tient settings are increasing, and the range of available technology remains vast and varied.3 An important barrier to health IT adoption has been the uncertain effect on patient outcomes, particularly given the costliness of implementation of computerized infrastructures.4 In order to evaluate the current state of the literature, we conducted a systematic review to determine the effect of multiple health IT tools on patient safety outcomes.
While 31 systematic reviews have been conducted with a focus on health IT interventions and patient safety outcomes, this systematic review is different for several reasons. First, many of the systematic reviews focused upon one specific health IT,5–10 most commonly clini- cal decision support (CDS).11–26 Second, prior reviews often focused upon one area of clinical care such as outpatient,13,15,27,28
inpatient,6,16,29 intensive care,30 pediatrics,5,30 or geriatrics.23 Other papers targeted very specific outcomes, such as the effects of health IT as it relates to antibiotic medications,22 anticoagulant therapy,20 lab testing,7 or treatment of hypertension.13 Finally, many prior reviews looked specifically at effects of health IT on one safety outcome— adverse drug events (ADEs).5,6,19,24,26,29,31–33
Prior studies generally included both non-randomized and random- ized trials.4–9,15–19,21–23,27–29,32–34 Eleven of the prior systematic re- views included only the highest level of evidence studies, randomized controlled trials (RCT).10–14,20,24–26,31,35 Findings from these system- atic reviews were mixed. Three of the previously mentioned 11 studies conducted a meta-analysis: 1 found improvement in patient safety outcomes,24 1 stated insufficient studies to conclude,26 and the final paper was equivocal.10
Therefore, this systematic review serves to provide a cumulative picture of the effects of multiple types of health IT on an array of direct patient safety outcomes in all clinical areas. This is an important time to be studying health IT as adoption rates continue to rise, policymakers continue to support and promote its use, and the determination of how and when to begin regulation of health IT remains under debate. To our knowledge, no prior systematic review has evaluated a comprehensive set of health IT tools while also exclusively focusing on determining the effects of those technologies on direct patient safety outcomes.
Correspondence to Erika Abramson, MD MS, Assistant Professor of Pediatrics and Healthcare Policy and Research, Weill Cornell Medical College of Cornell University,
525 East 68th Street, Rm M-610A, New York, City, NY 10065, USA; [email protected]; Tel: 212-746-3929; Fax: 212-746-3140 VC The Author 2015. Published by Oxford University Press on behalf of the American Medical Informatics Association. All rights reserved. For Permissions, please
email: [email protected] For numbered affiliations see end of article.
R EVIEW
1016
Brenner S.K, et al. J Am Med Inform Assoc 2016;23:1016–1036. doi:10.1093/jamia/ocv138, Review
MATERIALS AND METHODS Study Identification and Selection Health IT was broadly defined as any automated or computerized sys- tem implemented to aid in the management of health information. We focused on the following health information technologies: computer- ized physician order entry (CPOE), e-prescribing, CDS, order entry alerts, EHR, health information exchange (HIE), patient portals, auto- mated error detection software to detect medication errors (AED), electronic medication administration records (eMAR), medication ad- ministration barcodes, electronic medication reconciliation software (eMedRec), automated medication dispensing systems (AutoDisp), and electronic clinical pathways. Medication administration barcodes in- cluded barcode systems that dispense medication from an automated machine, as well as barcode systems that are used to ensure correct patient identification during the process of medication administration. Automated error detection systems referred to systems that look back to find the orders that may have led to an ADE or a pADE, in contrast to CPOE, which is designed to help aid the provider in correct prescrib- ing at the point of care. We chose these tools through a combination of a priori knowledge of the literature, as well as health IT tools identi- fied as part of the systematic review search process. Other patient- centered interventions such as health IT phone applications or home automated blood pressure cuff monitoring were not actively excluded; however, we did not identify any studies that assessed the impact of these technologies on direct patient outcomes. In cases in which au- thors did not identify the type of health IT employed using commonly known acronyms or terminology, reviewers used the description of the intervention to determine which type of health IT was being employed.
The authors also identified the clinicians under study. For cases in which the clinicians employing a particular health IT intervention were not identified, the authors reported “NR,” not reported. In cases where a clinician type was not applicable—for example, patient centered tools—those studies were denoted as N/A.
The patient outcomes chosen were identified from the studies in- cluded in the review, as well as from author knowledge of outcomes likely to be affected by health IT. After the analysis was completed, outcomes were then grouped on the basis of similar types of out- comes. In the articles for which more than one patient safety outcome was studied, reviewers included in the summary table only the primary outcome numerical effect size. However, for all outcomes, whether or not statistical significance was reached, the positive, negative, or non- significant effect on patient outcomes was considered and recorded (Table 1).
We performed searches in bibliographic databases, Ovid Medline, Ovid EMBASE, the Cumulative Index to Nursing Allied Health (CINAHL) via Ebscohost, and Cochrane Library from January 2001 to June 2012. Conference proceedings were reviewed as well as bibliogra- phies of selected articles. Citations of all identified prior systematic re- views were also reviewed. The search strategy included combinations of keywords and controlled vocabulary. A validated filter to represent patient safety was applied.105 Appendix A illustrates the detailed search strategy for the four databases.
All citations, index terms, and abstracts (if available) were re- viewed and rated as “potentially relevant” or “not relevant.” In accor- dance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement for reporting of systematic re- views, one reviewer reviewed the entire set first of titles, followed by the abstracts. Articles that were potentially relevant were included in the set reviewed by 2 independent reviewers (see Figure 1).106
Articles were reviewed independently, and studies were included in the review if 1) the study participants were health professionals in
clinical practice or postgraduate training, 2) the intervention was health IT studied in a clinical setting, and 3) the outcomes (even if secondary and not primary) that were assessed included at least one direct patient safety outcome (including any aspect of patient well- being, with process measures considered insufficient). Only English- language studies were included. All disagreements were resolved by consensus.
Study Evaluation Two authors independently assessed all selected studies for methodo- logical quality. A previously described 10-point Methodological Quality Assessment was adapted to the purposes of this study.9,11,17 This methodological rating scale assesses for 5 potential sources of bias, each scored either 0, 1, or 2, including (A) the method of allocation to study groups (random vs selected concurrent controls vs non-concur- rent controls), (B) the unit of allocation (ward or clinic vs physician vs patient), (C) baseline differences between groups which could poten- tially be linked to the study outcome (no baseline differences and/or appropriate statistical adjustments made for differences vs baseline differences apparent without statistical adjustment vs unable to as- sess), (D) the type of safety outcome measure (objective outcome or subjective outcome with blinded assessment vs objective outcome with no blinding vs subjective outcome without blinding of assessors), and (E) completeness of follow-up (>90% vs 80%-90% vs <80% and/or unable to assess).11 As such, a score of 10 represents studies whose design had the lowest amount of bias (Table 2). Disagreements were resolved by discussion to reach consensus. Reviewer agreement and inter-rater reliability was analyzed by the kappa statistical method. Since one reviewer reviewed all of the articles, and multiple reviewers were paired with the principal reviewer, a quadratic-weighted kappa was chosen.107
Adopting the methodology employed by a prior systematic review (Chaudhry et al.)4, quantitative reports were considered “hypothesis- testing” if the investigators compared data between groups or across time periods, using statistical tests to assess differences. We further categorized hypothesis-testing studies into 5 study types. RCTs were defined as studies that had a control and experimental arm for which the intervention (health IT) was randomly assigned. Cohort trials were defined as non-randomized studies for which a concurrent control arm was included. Observational studies were most often before-and-after studies in which the “before” group served as the only control. Time series analyses were studies for which time-series statistical analyses were conducted. Lastly, case-control studies were studies for which cases and controls were picked retrospectively, based on exposure to health IT.4
Data Extraction and Analysis For each article included, both reviewers extracted information regard- ing patients, clinicians involved, setting, intervention, and outcomes for each of the studies. The safety outcomes evaluated were catego- rized into the following groups: 1) ADEs or adverse events (AEs); 2) mortality; 3) thrombosis or bleed; 4) length of stay (LOS); 5) infection rates; 6) readmission, admission, or emergency department (ED) vis- its; 7) fall rates or pressure ulcer; 8) hemodynamic instability or inten- sive care unit (ICU) transfer; 9) myocardial infarction (MI) or cardiac events; 10) chronic disease exacerbations; and 11) altered mental sta- tus (AMS) or stroke incidence.
Adapting methodology used in prior reviews, positive studies were those in which the primary outcome studied showed statistically sig- nificant improvement. Negative studies were those for which there were statistically significant worse patient safety outcomes. Mixed or
R EVIEW
Brenner S.K, et al. J Am Med Inform Assoc 2016;23:1016–1036. doi:10.1093/jamia/ocv138, Review
1017
Ta bl
e 1:
Pu bl
is he
d Ar
tic le
s St
ud yi
ng H
ea lth
IT Ef
fe ct
s on
Pa tie
nt O
ut co
m es
.
Au th
or s
(y ea
r) St
ud y
Pe rio
d (m
on th
s)
St ud
y D
es ig
n (P
ro sp
or Re
tr os
p)
Q ua
lit y
Sc or
e (0
–1 0)
Co un
tr y
St ud
y Si
te (n
)
Se tti
ng Cl
in ic
ia n
Af fe
ct ed
N o.
Cl in
ic ia
ns H
ea lth
IT In
te rv
en tio
n N
o. Pa
tie nt
s Pa
tie nt
O ut
co m
ea O
ut co
m e
Ef fe
ct Si
ze b
P- va
lu e
Ke y
Fi nd
in gs
Al vi
s et
al .
(1 98
5) .3
6 N
R RC
T (P
ro sp
) 6
US H os
p (1
) O
R M
D s
– Sm
ar t
Pu m
p 30
H em
od yn
am ic
St ab
ili ty
(h yp
o- an
d hy
pe rt
en si
on )
P AR
R 14
% <
.0 5
Pa tie
nt s
w ho
ha d
co m
pu te
r- as
si st
ed ,
sm ar
tp um
p- m
ed ia
te d
fe nt
an yl
in fu
si on
ha d
gr ea
te r
he m
od yn
am ic
st ab
ili ty
.F ew
er ad
ju va
nt dr
ug s
w er
e us
ed w
ith sm
ar t-
pu m
p te
ch no
lo gy
th an
w ith
m an
ua lb
ol us
of fe
nt an
yl ap
pr oa
ch .
Ba re
nf an
ge r,
Sh or
t, an
d G
ro es
ch (2
00 1)
.3 7
19 98
–1 99
9 (5
) Co
ho rt
(P ro
sp )
3 US H
os p
(1 )
Al lw
ar ds
M D
s –
CD S
37 8
M or
ta lit
ya LO
S P
AR R
1. 4%
– D
ec re
as ed
m or
ta lit
y an
d LO
S w
er e
ob se
rv ed
w he
n ad
ju st
ed fo
r se
ve rit
y of
ca se
s w
he n
us in
g CD
S su
rr ou
nd in
g an
ti- bi
ot ic
us ag
e. Co
st s
w er
e al
so de
cr ea
se d.
Bu is
in g
et al
. (2
00 8)
.3 8
20 00
–2 00
7 (8
4) Ti
m e
Se rie
s (P
ro sp
) 5
Au st
ra lia
H os
p (1
) Al
lw ar
ds M
D s
– CD
S 18
9 G
N ba
ct er
em ia
a In
- ho
sp m
or ta
lit y
LO S
M N
S –
30 -d
ay m
or ta
lit y,
ra te
s of
G N
ba ct
er em
ia ,
an d
LO S
re m
ai ne
d un
ch an
ge d
w he
n CD
S w
as em
pl oy
ed to
en co
ur ag
e ap
pr op
ria te
an tib
io tic
us ag
e.
Ch ap
ui s
et al
. (2
01 0)
.3 9
Co ho
rt (P
ro sp
) 4
Fr an
ce H
os p
(1 )
M ed
ic al
IC U
RN s
– Au
to D
is p
11 5
AD E
M N
S –
W hi
le au
to m
at ed
di sp
en si
ng sy
st em
ca us
ed fe
w er
m ed
ic at
io n
er ro
rs as
co m
- pa
re d
to ba
se lin
e an
d co
nc ur
re nt
co nt
ro l
un its
,i td
id no
tr ed
uc e
er ro
rs ca
us in
g pa
tie nt
ha rm
.S ub
je ct
iv el
y, nu
rs es
re po
rt ed
im pr
ov ed
w or
ki ng
co nd
iti on
s w
ith au
to m
at ed
di sp
en si
ng sy
st em
.
Ch er
to w
et al
. (2
00 1)
.4 0
19 97
–1 99
8 (8
) Co
ho rt
(P ro
sp )
8 US H
os p
(1 )
Al la
du lt
w ar
ds M
D s
– CD
S CP
O E
17 82
8 ad
m is
si on
s Re
na lI
m pa
irm en
ta
LO S
M N
S –
Th er
e w
as no
si gn
ifi ca
nt de
cr ea
se in
th ei
r pr
im ar
y ou
tc om
e of
th e
nu m
be r
of pa
tie nt
s w
ho ex
pe rie
nc ed
a de
cl in
e in
fu nc
tio n
du rin
g ho
sp ita
liz at
io n.
H ow
ev er
, th
er e
w as
a si
gn ifi
ca nt
re du
ct io
n in
le ng
th of
st ay
du rin
g in
te rv
en tio
n pe
rio ds
w ith
ph ys
ic ia
n al
er ts
to ad
ju st
m ed
ic at
io ns
ba se
d on
re na
lf un
ct io
n.
Ch o
et al
. (2
01 2)
.4 1
20 09
–2 01
0 (1
1) O
bs er
v (P
ro sp
) 2
Ko re
a H
os p
(1 )
Al la
du lt
w ar
ds M
D s
– CD
S 28
5 Co
nt ra
st -i
nd uc
ed AK
I P
AR R
7% –
Ra te
s of
co nt
ra st
-i nd
uc ed
AK Id
ro pp
ed af
te r
im pl
em en
ta tio
n of
a co
m pu
te r
al er
t pr
og ra
m re
m in
di ng
pr ov
id er
s to
or de
r pr
op hy
la xi
s fo
r pa
tie nt
s w
ith an
el ev
at ed
G FR
un de
rg oi
ng a
CT .
Co ok
et al
. (2
01 1)
.4 2
20 05
–2 00
9 (6
0) O
bs er
v (R
et ro
) 3
US H os
p (1
) Al
la du
lt w
ar ds
M D
s –
CD S
EH R
CP O
E
36 12
ch ar
ts N
os oc
om ia
lC .d
iff a
N os
oc om
ia lM
RS A
M N
S –
Ra te
s of
no so
co m
ia lC
.d iff
ic ile
in fe
ct io
ns di
d no
tc ha
ng e,
bu tr
at es
of M
RS A
in fe
c- tio
ns de
cr ea
se d
si gn
ifi ca
nt ly
af te
r im
pl e-
m en
ta tio
n of
a EH
R w
ith CD
S. (c on
tin ue
d)
R EVIEW
Brenner S.K, et al. J Am Med Inform Assoc 2016;23:1016–1036. doi:10.1093/jamia/ocv138, Review
1018
Ta bl
e 1:
Co nt
in ue
d
Au th
or s
(y ea
r) St
ud y
Pe rio
d (m
on th
s)
St ud
y D
es ig
n (P
ro sp
or Re
tr os
p)
Q ua
lit y
Sc or
e (0
–1 0)
Co un
tr y
St ud
y Si
te (n
)
Se tti
ng Cl
in ic
ia n
Af fe
ct ed
N o.
Cl in
ic ia
ns H
ea lth
IT In
te rv
en tio
n N
o. Pa
tie nt
s Pa
tie nt
O ut
co m
ea O
ut co
m e
Ef fe
ct Si
ze b
P- va
lu e
Ke y
Fi nd
in gs
Co lp
ae rt
et al
. (2
00 6)
.4 3
20 04
(1 .2
5) RC
T (P
ro sp
) 6
Be lg
iu m
H os
p (1
) Su
rg ic
al IC
U In
te ns
iv is
ts Re
si de
nt s
5 3
8 To
ta l
CP O
E CD
S 90
N on
-i nt
er ce
pt po
te n-
tia lA
D Es
a Ra
te of
N on
-i nt
er ce
pt ed
po te
nt ia
lA D
Es To
ta l
AD Es
Ra te
of AD
Es
P AR
R 1.
4 ra
te s
of N
on -i
nt er
ce pt
ed AD
Es
< .0
01 Th
e po
te nt
ia lA
D Es
an d
ac tu
al AD
Es w
er e
re du
ce d
in bo
th ab
so lu
te nu
m be
r an
d ra
te in
un its
w hi
ch CP
O E
w ith
CD S
w as
us ed
as op
po se
d to
pa pe
r- ba
se d
un its
.C PO
E w
ith CD
S w
as fo
un d
to ha
ve th
e m
os t
im pa
ct on
m ed
ic at
io n
pr es
cr ib
in g
er ro
rs th
at ha
d th
e le
as tp
ot en
tia ln
eg at
iv e
pa tie
nt im
pa ct
.
D el
Be cc
ar o
et al
.( 20
06 ).4
4 20
02 –2
00 4
(2 6)
O bs
er v
(P ro
sp )
6 US H
os p
(1 )
Pe di
at ric
IC U
M D
s –
CP O
E 25
33 M
or ta
lit y
M N
S –
N o
si gn
ifi ca
nt ch
an ge
in m
or ta
lit y
po st
CP O
E im
pl em
en ta
tio n
in a
PI CU
w as
fo un
d w
he n
m or
al ity
w as
tr ac
ke d
fo r
13 m
on th
s pr
e- an
d po
st -
im pl
em en
ta tio
n.
D ow
di ng
,T ur
le y,
an d
G ar
rid o.
(2 01
2) .4
5
20 03
–2 00
9 (8
4) Ti
m e
se rie
s (P
ro sp
) 4
US H os
p (2
9) Al
lw ar
ds RN
s –
EH R
eM AR
CP O
E CD
S
N R
H os
pi ta
l- ac
qu ire
d pr
es su
re ul
ce rs
a Fa
ll in
ju rie
s
M N
S –
EH R
im pl
em en
ta tio
n w
ith ris
k as
se ss
m en
t to
ol s
w as
as so
ci at
ed w
ith de
cr ea
se d
ra te
s in
H AP
U, bu
tn ot
in fa
ll ra
te s,
bu tf
in di
ng w
as no
ts ig
ni fic
an tw
he n
co nt
ro lle
d fo
r ov
er tim
e.
D yk
es et
al .
(2 01
0) .4
6 20
09 (6
) RC
T (P
ro sp
) 8
US H os
p (4
) M
ed ic
in e
w ar
ds RN
s –
CD S
10 26
4 Fa
ll Ra
te s
P AR
R 1.
03 fa
lls pe
r 10
00 pa
tie nt
da ys
– Us
e of
CD S
fa ll
pr ev
en tio
n to
ol w
as as
so -
ci at
ed w
ith de
cr ea
se d
fa ll
ra te
in in
te rv
en -
tio n
un its
.P at
ie nt
s ov
er 65
ye ar
s ol
d ha
d gr
ea te
st de
cr ea
se in
fa ll
ra te
s. Th
er e
w as
no si
gn ifi
ca nt
ch an
ge in
fa lls
re su
lti ng
in pa
tie nt
in ju
ry .
Et ch
el ls
et al
. (2
01 1)
.4 7
20 08
(6 )
Co ho
rt (P
ro sp
) 7
Ca na
da H
os p
(2 )
M ed
ic in
e w
ar ds
M D
s –
CD S
AE D
27 1
Ra te
s AE
s M
N S
– Re
al -t
im e
al er
tin g
sy st
em w
ith si
m ul
ta ne
- ou
s de
ci si
on su
pp or
t, bu
th ad
no si
gn ifi
- ca
nt im
pa ct
on ra
te s
of ad
ve rs
e ev
en ts
.
Ev an
s et
al .
(1 99
8) .4
8 19
92 –1
99 5
(3 6)
O bs
er v
(P ro
sp )
6 US H
os p
(1 )
IC U
M D
s –
CD S
16 81
AD Es
a
LO S
P 70
% re
du ct
io n
.0 18
Si gn
ifi ca
nt re
du ct
io n
in th
e ra
te of
AD E
fr om
an tib
io tic
s w
as re
du ce
d by
an an
ti- in
fe ct
iv e
CD S
to ol
.L O
S w
as al
so de
cr ea
se d.
Ev an
s et
al .
(1 99
5) .4
9 19
94 –1
99 5
(1 9)
O bs
er v
(R et
ro )
6 US H
os p
(1 )
Ad ul
tI CU
M D
s –
CD S
96 2
AD Es
a M
N S
– Us
in g
a CD
S sy
st em
fo r
an tib
io tic
se le
c- tio
n di
d no
td ec
re as
e ra
te s
of AD
Es .
Ev an
s et
al .
(1 99
3) .5
0 19
89 –1
99 2
(4 8)
O bs
er v
(R et
ro )
7 US H
os p
(1 )
Al la
du lt
w ar
ds M
D s
– AE
D CD
S 92
64 9
AD Es
P AR
R 13
.6 %
< .0
01 Co
m pu
te riz
ed su
rv ei
lla nc
e of
AD Es
re du
ce d
th e
nu m
be r
of se
ve re
AD Es
. Ad
di tio
na lly
,t he
ir su
rv ei
lla nc
e sy
st em
w as
us ed
to cr
ea te
co m
pu te
r al
er ts
to ph
ar m
ac is
ts w
he n
ph ys
ic ia
ns pr
es cr
ib ed
m ed
ic at
io ns
to pa
tie nt
s w
ith pr
ev io
us ly
kn ow
n dr
ug al
le rg
ie s,
w hi
ch al
so si
gn ifi
- ca
nt ly
re du
ce d
th e
ra te
of AD
Es .
(c on
tin ue
d)
R EVIEW
Brenner S.K, et al. J Am Med Inform Assoc 2016;23:1016–1036. doi:10.1093/jamia/ocv138, Review
1019
Ta bl
e 1:
Co nt
in ue
d
Au th
or s
(y ea
r) St
ud y
Pe rio
d (m
on th
s)
St ud
y D
es ig
n (P
ro sp
or Re
tr os
p)
Q ua
lit y
Sc or
e (0
–1 0)
Co un
tr y
St ud
y Si
te (n
)
Se tti
ng Cl
in ic
ia n
Af fe
ct ed
N o.
Cl in
ic ia
ns H
ea lth
IT In
te rv
en tio
n N
o. Pa
tie nt
s Pa
tie nt
O ut
co m
ea O
ut co
m e
Ef fe
ct Si
ze b
P- va
lu e
Ke y
Fi nd
in gs
Fi tz
m au
ric e
et al
. (1
99 6)
.5 1
19 93
–1 99
4 (1
2) RC
T (P
ro sp
) 5
US Am b
(2 )
Pr im
ar y
Ca re
M D
s –
CD S
49 M
or ta
lit ya
Th ro
m bo
tic ev
en ts
H em
or rh
ag ic
ev en
ts
M N
S –
Th er
e w
as no
di ffe
re nc
e in
ad ve
rs e
ev en
t ra
te s
fo r
pa tie
nt s
w ho
w er
e ra
nd om
iz ed
to CD
S do
si ng
of th
ei r
w ar
fa rin
.I N
R w
as be
tte r
co nt
ro lle
d in
in te
rv en
tio n
gr ou
p.
Fr an
ce s
et al
. (2
00 1)
.5 2
19 97
(4 )
RC T
(P ro
sp )
4 US Am
b (2
) Pr
im ar
y Ca
re M
D s
63 CD
S 73
0 M
or ta
lit ya
M I
M N
S –
Ra te
s of
M Ia
nd m
or ta
lit y
w er
e no
ts ig
ni fi-
ca nt
ly di
ffe re
nt fo
r pa
tie nt
s w
ho se
ph ys
i- ci
an s
w er
e ra
nd om
iz ed
to ha
ve CD
S re
m in
di ng
th em
to or
de r
ap pr
op ria
te po
st -
M Im
ed ic
al m
an ag
em en
t.
Fi um
ar a
et al
. (2
01 0)
.5 3
20 06
–2 00
8 (2
3) Co
ho rt
(P ro
sp )
8 US H
os p
(1 )
M ed
ic al
an d
Su rg
ic al
w ar
ds M
D s
42 5
CD S
88 0
90 -d
ay in
ci de
nc e
of VT
E M
N S
– CD
S to
en co
ur ag
e D
VT pr
op hy
la xi
s di
d no
t de
cr ea
se ra
te s
of VT
E at
90 -d
ay s.
Fl em
in g
et al
. (2
00 9)
.5 4
20 06
–2 00
8 (3
0) Co
ho rt
(P ro
sp )
2 US H
os p
(8 )
M ed
ic al
w ar
ds M
D s
– CD
S Al
g 44
54 In
pa tie
nt M
or ta
lit ya
30 -d
ay m
or ta
lit y
M 2.
9% AR
R <
.0 1
Un ad
ju st
ed re
su lts
sh ow
ed re
du ct
io n
in bo
th in
-h os
pi ta
lm or
ta lit
y an
d 30
-d ay
m or
ta lit
y w
he n
or de
r se
tw as
us ed
. Re
su lts
ad ju
st ed
fo r
co va
ria te
s w
er e
no t
fo un
d to
be si
gn ifi
ca nt
w ith
us e
of CD
S. Ad
ju st
ed re
su lts
w er
e of
bo rd
er lin
e si
gn ifi
ca nc
e.
G an
dh ie
ta l.
(2 00
5) .5
5 19
99 –2
00 0
Co ho
rt (P
ro sp
) 6
US Am b
(4 )
In te
rn al
M ed
ic in
e M
D s
34 CP
O E
66 1
AD Es
M N
S –
Ra te
s of
AD E
w er
e th
e sa
m e
in th
e CP
O E
vs .h
an dw
rit te
n m
ed ic
at io
n or
de rs
in th
e am
bu la
to ry
ca re
se tti
ng .N
on -s
ig ni
fic an
t tr
en d
to w
ar ds
in cr
ea se
in AD
E w
as fo
un d.
G la
ss m
an et
al .
(2 00
7) .5
6 20
01 –2
00 2
(8 )
Co ho
rt (P
ro sp
) 5
US Am b
(N R)
In te
rn al
M ed
ic in
e M
D s
– AE
D CP
O E
91 3
AD Es
M N
S –
Ra te
s of
AD Es
w er
e th
e sa
m e
in th
e re
tr o-
sp ec
tiv e
er ro
r de
te ct
io n
sy st
em fo
r a
CP O
E sy
st em
.T he
AE D
fo cu
se d
pr im
ar ily
on dr
ug -d
ru g
in te
ra ct
io ns
an d
dr ug
-d is
- ea
se in
te ra
ct io
ns .
G ra
um lic
h et
al .
(2 00
9) .5
7 20
04 –2
00 7
(3 9)
RC T
(P ro
sp )
8 US H
os p
(1 )
M ed
ic in
e w
ar ds
M D
s 69
eM ed
Re cC
PO E
63 1
Re ad
m is
si on
ra te
a ED
vi si
tr at
e AD
Es po
st -
di sc
ha rg
e
M N
S –
D is
ch ar
ge so
ftw ar
e di
d no
ta ffe
ct re
ad -
m is
si on
ra te
s, ED
vi si
tr at
es or
ad ve
rs e
ev en
ts po
st -d
is ch
ar ge
.
G ur
w itz
et al
. (2
00 8)
.5 8
(1 2)
RC T
(P ro
sp )
5 US
an d
Ca na
da LT
C (2
)
Al lw
ar ds
M D
s PA
N Ps
37 CP
O E
CD S
11 18
AD E
ra te
a Pr
ev en
ta bl
e AD
E ra
te M
N S
– CP
O E
w ith
CD S
di d
no tr
ed uc
e ad
ve rs
e dr
ug ev
en tr
at e
or pr
ev en
ta bl
e ad
ve rs
e dr
ug ev
en tr
at e.
H an
et al
. (2
00 5)
.5 9
20 01
–2 00
3 (1
8) O
bs er
v (R
et ro
) 6
US H os
p (N
R) Pe
di at
ric IC
U M
D s
– CP
O E
19 42
M or
ta lit
y N
eg O
R 3.
71 –
Th er
e w
as an
un ex
pe ct
ed in
cr ea
se in
m or
ta lit
y co
in ci
de nt
w ith
CP O
E im
pl em
en -
ta tio
n fr
om 13
m on
th s
be fo
re im
pl em
en -
ta tio
n an
d 5
m on
th s
af te
r CP
O E
im pl
em en
ta tio
n.
(c on
tin ue
d)
R EVIEW
Brenner S.K, et al. J Am Med Inform Assoc 2016;23:1016–1036. doi:10.1093/jamia/ocv138, Review
1020
Ta bl
e 1:
Co nt
in ue
d
Au th
or s
(y ea
r) St
ud y
Pe rio
d (m
on th
s)
St ud
y D
es ig
n (P
ro sp
or Re
tr os
p)
Q ua
lit y
Sc or
e (0
–1 0)
Co un
tr y
St ud
y Si
te (n
)
Se tti
ng Cl
in ic
ia n
Af fe
ct ed
N o.
Cl in
ic ia
ns H
ea lth
IT In
te rv
en tio
n N
o. Pa
tie nt
s Pa
tie nt
O ut
co m
ea O
ut co
m e
Ef fe
ct Si
ze b
P- va
lu e
Ke y
Fi nd
in gs
H ol
ds w
or th
et al
. (2
00 7)
.6 0
20 00
–2 00
1, 20
04 (9 þ
7) Co
ho rt
(P ro
sp )
5 US H
os p
(1 )
Pe di
at ric
IC U
N R
– CP
O E
CD S
Ba se
lin e:
12 10
po st
- CP
O E:
11 97
Ra te
s of
AD Ea
To ta
l AD
Es Ra
te s
of po
te n-
tia la
nd pr
ev en
ta bl
e AD
Es
P O
R 0.
76 –
Th er
e w
as si
gn ifi
ca nt
re du
ct io
n in
th e
to ta
lA D
Es ,p
re ve
nt ab
le AD
Es ,a
nd po
te n-
tia lA
D Es
af te
r im
pl em
en ta
tio n
of CP
O E
sy st
em .S
ub -g
ro up
an al
ys es
fo un
d th
at th
er e
w er
e si
gn ifi
ca nt
re du
ct io
ns in
ad ve
rs e
ev en
ts as
so ci
at ed
w ith
ce rt
ai n
an tib
io tic
dr ug
cl as
se s.
LO S
re m
ai ne
d ne
ar ly
id en
tic al
be tw
ee n
th e
AD E
an d
po te
nt ia
lA D
E gr
ou ps
bo th
be fo
re an
d af
te r
CP O
E im
pl em
en ta
tio n.
Ja ni
,B ar
be r,
an d
W on
g. (2
01 0)
.6 1
20 05
–2 00
6 (1
3) Co
ho rt
(P ro
sp )
7 UK H
os p
(1 )
Pe di
at ric
w ar
ds M
D s
– CP
O E
15 90
M ed
do si
ng er
ro rs
va ry
in g
se ve
rit y
P AR
R 1%
< .0
01 El
ec tr
on ic
pr es
cr ib
in g
ca n
re du
ce do
si ng
er ro
rs w
ith ou
tC D
S.
Jh a
et al
. (2
00 8)
.6 2
20 04
–2 00
5 (2
4) Co
ho rt
(P ro
sp )
4 US H
os p
(1 ,6
72 )
M ed
ic in
e w
ar ds
N /A
– CP
O E
N R
Ad ju
st ed
30 -d
ay M
or ta
lit y
Ra te
s fo
r: (1
)A M
Ia (2
)C H
F (3
) PN
A
M AR
R 1.
5% <
.0 02
H os
pi ta
ls th
at ha
d im
pl em
en te
d CP
O E
an d
pa rt
ic ip
at ed
in re
po rt
in g
w er
e fo
un d
to ha
ve lo
w er
ra te
s of
ad ju
st ed
30 -d
ay m
or -
ta lit
y in
bo th
AM Ia
nd PN
A pa
tie nt
s. Th
er e
w as
no di
ffe re
nc e
in m
or ta
lit y
in CH
F pa
tie nt
s.
Ji m
en ez
-M un
oz et
al .(
20 11
).6 3
20 06
–2 00
7 (4
) Co
ho rt
(P ro
sp )
7 Sp
ai n
H os
p (1
) M
ed ic
in e
w ar
ds M
D s
RN s
– Au
to D
is p
CP O
E N
R AE
s P
AR R
2. 73
% <
.0 03
CP O
E an
d au
to m
at ed
di sp
en si
ng sy
st em
s re
du ce
d m
ed ic
at io
n er
ro rs
af fe
ct in
g pa
tie nt
in tw
o ph
as es
dr ug
ad m
in is
te rin
g ph
as es
:t ra
ns cr
ip tio
n an
d ad
m in
is tr
at io
n w
he n
co m
pa re
d w
ith th
e pa
pe r-
ba se
d gr
ou p.
Er ro
r pr
ev al
en ce
ra te
s re
qu iri
ng m
on ito
rin g
or no
t, w
er e
no ta
ffe ct
ed in
th e
pr es
cr ip
tio n
ph as
e.
Ke en
e et
al .
(2 00
7) .6
4 19
95 –1
99 7
(3 9)
O bs
er v
(P ro
sp )
4 US H
os p
(1 )
Pe di
at ric
an d
N eo
na ta
lI CU
s M
D s
– CP
O E
12 91
M or
ta lit
y M
N S
– Ra
te s
of m
or ta
lit y
di d
no tc
ha ng
e w
ith CP
O E
im pl
em en
ta tio
n.
Ki ng
et al
. (2
00 3)
.6 5
19 93
–1 99
6 19
97 –1
99 9
(7 2)
Co ho
rt (R
et ro
) 3
Ca na
da H
os p
(1 )
Pe di
at ric
: M
ed ic
al an
d Su
rg ic
al w
ar ds
M D
s Re
si de
nt s
– CP
O E
36 10
3 AE
s M
N S
– Th
er e
w as
no ef
fe ct
of ra
te ra
tio s
of AD
Es in
th e
pr e-
an d
po st
-C PO
E im
pl em
en ta
tio n
pe rio
ds .T
he re
w as
a si
gn ifi
ca nt
ly de
cr ea
se d
ra te
of po
te nt
ia lA
D Es
in th
e co
nt ro
lw ar
ds (w
ith ha
nd w
rit te
n or
de rs
). By
co nt
ra st
,m ed
ic at
io n
er ro
r ra
te s
w er
e al
so si
gn ifi
ca nt
ly re
du ce
d in
th e
CP O
E w
ar ds
.
Ku ch
er et
al .
(2 00
5) .6
6 20
00 –2
00 4
(4 1)
RC T
(P ro
sp )
4 US H
os p
(1 )
M ed
ic al
an d
Su rg
ic al
w ar
ds M
D s
12 0
CD S
25 06
VT E
at 90
da ys
a
M or
ta lit
y at
30 an
d 90
da ys
P 3.
3% AR
R <
.0 01
Fo r
pa tie
nt s
fo r
w ho
m th
ro m
bo em
bo lic
pr op
hy la
xi s
ha d
no tb
ee n
or de
re d,
a co
m -
pu te
r ge
ne ra
te d
al er
tt o
ph ys
ic ia
ns re
du ce
d VT
E ra
te s
at 90
da ys
w ith
no di
f- fe
re nc
e in
m or
ta lit
y be
tw ee
n gr
ou ps
.
(c on
tin ue
d)
R EVIEW
Brenner S.K, et al. J Am Med Inform Assoc 2016;23:1016–1036. doi:10.1093/jamia/ocv138, Review
1021
Ta bl
e 1:
Co nt
in ue
d
Au th
or s
(y ea
r) St
ud y
Pe rio
d (m
on th
s)
St ud
y D
es ig
n (P
ro sp
or Re
tr os
p)
Q ua
lit y
Sc or
e (0
–1 0)
Co un
tr y
St ud
y Si
te (n
)
Se tti
ng Cl
in ic
ia n
Af fe
ct ed
N o.
Cl in
ic ia
ns H
ea lth
IT In
te rv
en tio
n N
o. Pa
tie nt
s Pa
tie nt
O ut
co m
ea O
ut co
m e
Ef fe
ct Si
ze b
P- va
lu e
Ke y
Fi nd
in gs
Ku pe
rm an
et al
. (1
99 9)
.6 7
19 94
–1 99
5 (4
) RC
T (P
ro sp
) 8
US H os
p (1
) Ad
ul tM
ed ic
al an
d Su
rg ic
al w
ar ds
M D
s –
CD S
N R
M or
ta lit
ya Ar
re st
Tr an
sf er
to IC
U M
I D
el iri
um St
ro ke
Re na
l In
su ffi
ci en
cy AK
I D
ia ly
si s
Re tu
rn to
O R
Al lo
ut co
m es
M N
S –
Th er
e w
as no
di ffe
re nc
e in
ad ve
rs e
ev en
t ra
te s
fo r
pa tie
nt s
w ho
w er
e ra
nd om
iz ed
to ha
vi ng
th ei
r ph
ys ic
ia ns
re ce
iv e
al er
ts re
ga rd
in g
co nc
er ni
ng la
bo ra
to ry
va lu
es in
th ei
r m
ed ic
al re
co rd
s. Th
e in
te rv
en tio
n gr
ou p
ha d
sh or
te r
m ed
ia n
tim e
in te
rv al
s be
fo re
an ap
pr op
ria te
tr ea
tm en
tw as
or de
re d.
Le cu
m be
rr ie
ta l.
(2 00
8) .6
8 20
05 –2
00 7
(1 8)
O bs
er v
(P ro
sp )
6 Sp
ai n
H os
p (1
) M
ed ic
al an
d Su
rg ic
al W
ar ds
M D
s –
CD S
19 33
8 VT
E M
O R
0. 53
– Ra
te s
of VT
E w
er e
no ts
ig ni
fic an
tly re
du ce
d by
al er
ts to
ph ys
ic ia
ns ov
er al
li n
th e
po st
-i nt
er ve
nt io
n pe
rio d,
bu ti
n su
b- gr
ou p
an al
ys is
of su
rg ic
al pa
tie nt
s, a
si g-
ni fic
an tr
ed uc
tio n
in VT
E ev
en ts
w as
id en
- tif
ie d,
w hi
ch w
as st
ab le
ov er
tim e.
Le so
ur d
et al
. (2
00 2)
.6 9
20 01
(N R)
RC T
(b ot
h Re
tr o
an d
Pr os
p)
3 Fr
an ce
H os
p (1
) RE
I M
D s
– CD
S 53
re tr
os p.
16 4
pr os
p. Pr
eg na
nc y
P N
S –
CD S
w as
as ef
fe ct
iv e
as cl
in ic
ia ns
in us
in g
ov ar
ia n
st im
ul at
io n
w ith
FS H
in re
su lti
ng in
pr eg
na nc
y.
Li na
re s
et al
. (2
01 1)
.7 0
Co ho
rt (P
ro sp
) 3
US H os
p (1
) M
ed ic
al w
ar ds
M D
s –
CD S
25 1
An tim
ic ro
bi al
-A Ea
C. di
ff fr
om tr
ea tm
en t
P N
N T
10 –
As ym
pt om
at ic
ba ct
iu ria
an d
cu ltu
re -n
eg a-
tiv e
py ur
ia ha
d de
cr ea
se d
co m
pl ic
at io
ns w
he n
a co
m pu
te riz
ed al
er tr
em in
de d
pr o-
vi de
rs th
at th
os e
U/ A
an d
cu ltu
re re
su lts
di d
no tr
eq ui
re tr
ea tm
en t.
M ac
Iv or
et al
. (2
00 9)
.7 1
20 05
–2 00
7 (3
1) O
bs er
v (R
et ro
) 7
US H os
p (1
6) M
ed ic
al w
ar ds
N R
N /A
H IE
16 M
is -t
ra ns
fu si
on s
P 38
% in
cr ea
se –
Ce nt
ra liz
ed pa
tie nt
da ta
ba se
de te
ct ed
38 %
m or
e AB
O ty
pi ng
er ro
rs an
d pr
e- ve
nt ed
6 m
is -t
ra ns
fu si
on s
M ad
ar as
-K el
ly et
al .(
20 06
).7 2
20 01
–2 00
4 (3
6) Ti
m e
se rie
s (P
ro sp
) 1
US H os
p (1
) M
ed ic
al an
d Su
rg ic
al IC
U M
D s
– CD
S N
R M
RS A
In fe
ct io
n ra
te s
P 0.
00 74
% AR
R .0
2 Ra
te of
no so
co m
ia lM
RS A
in fe
ct io
ns de
cr ea
se d
us in
g a
CD S
in te
rv en
tio n
to de
cr ea
se flo
ur oq
ui no
lo ne
us e.
Ra te
of no
so co
m ia
lg ra
m -n
eg at
iv e
or ga
ni sm
s si
g- ni
fic an
tly in
cr ea
se d
by 22
.6 6%
.R at
e of
tr im
et ho
pr im
-s ul
fa m
et ho
xa zo
le an
d pi
pe r-
ac ill
in -t
az ob
ac ta
m in
cr ea
se d
in us
ag e.
M ay
na rd
et al
. (2
01 0)
.7 3
20 05
–2 00
7 (3
6) O
bs er
v (b
ot h
Re tr
o an
d Pr
os p)
5 US H
os p
(1 )
Al la
du lt
w ar
ds ex
ce pt
ps yc
h an
d ob
/g yn
w ar
ds
M D
s –
CP O
E CD
S 29
24 H
os pi
ta l-
ac qu
ire d
VT Ea
he al
th IT
PP X-
re la
te d
bl ee
di ng
P 39
% RR
R –
Th er
e w
as a
re du
ct io
n of
th e
ra te
of H
A VT
E af
te r
in tr
od uc
tio n
of CP
O E
an d
CD S.
N ei
th er
he al
th IT
no r
pr op
hy la
xi s
re la
te d
bl ee
di ng
w as
in cr
ea se
d by
th is
in te
rv en
tio n.
M cC
ow an
e al
. (2
00 1)
.7 4
(6 )
RC T
(P ro
sp )
4 UK Am
b (1
7) Pr
im ar
y Ca
re M
D s
– CD
S 47
7 Ac
ut e
as th
m a
ex ac
er -
ba tio
ns a
H os
pi ta
liz at
io n
ED vi
si ts
M O
R 0.
43 –
A si
gn ifi
ca nt
ly lo
w er
nu m
be r
of pa
tie nt
s ex
pe rie
nc ed
as th
m a
ex ac
er ba
tio ns
if th
ei r
ph ys
ic ia
ns us
ed CD
S. H
os pi
ta liz
at io
ns an
d ED
vi si
ts ha
d no
ef fe
ct .
(c on
tin ue
d)
R EVIEW
Brenner S.K, et al. J Am Med Inform Assoc 2016;23:1016–1036. doi:10.1093/jamia/ocv138, Review
1022
Ta bl
e 1:
Co nt
in ue
d
Au th
or s
(y ea
r) St
ud y
Pe rio
d (m
on th
s)
St ud
y D
es ig
n (P
ro sp
or Re
tr os
p)
Q ua
lit y
Sc or
e (0
–1 0)
Co un
tr y
St ud
y Si
te (n
)
Se tti
ng Cl
in ic
ia n
Af fe
ct ed
N o.
Cl in
ic ia
ns H
ea lth
IT In
te rv
en tio
n N
o. Pa
tie nt
s Pa
tie nt
O ut
co m
ea O
ut co
m e
Ef fe
ct Si
ze b
P- va
lu e
Ke y
Fi nd
in gs
M cM
ul lin
et al
. (2
00 6)
.7 5
19 99
(3 )
20 01
–2 00
2 (1
2) 20
02 –
20 03
(3 )
Co ho
rt (P
ro sp
) 3
Ca na
da H
os p
(1 )
IC U
M D
s –
CD S
Ph as
e 1:
68 Ph
as e
2: 26
1 Ph
as e
3: 10
1
VT E
ra te
s M
N S
– D
VT an
d pu
lm on
ar y
em bo
lis m
ra te
s w
er e
si m
ila r
fo r
al lp
ha se
s de
sp ite
in cr
ea se
d co
m pl
ia nc
e w
ith pr
op hy
la xi
s gu
id el
in es
in ph
as es
2 an
d 3.
Ph as
e 2
w as
ch ar
ac te
r- iz
ed by
be ha
vi or
al ap
pr oa
ch es
pl us
co m
- pu
te riz
ed al
er ts
,w hi
le Ph
as e
3 co
ns is
te d
of al
er ts
al on
e.
M cM
ul lin
et al
. (1
99 9)
.7 6
19 94
–1 99
7 (4
8) O
bs er
v (R
et ro
) 3
US H os
p (1
) Al
lw ar
ds M
D s
– CD
S 28
6 AD
E M
N S
– Us
in g
m ed
ic at
io n
al er
ts di
d no
ts ig
ni fi-
ca nt
ly re
du ce
th e
ra te
s of
ad ve
rs e
ev en
ts as
re la
te d
to dr
ug -i
nt er
ac tio
ns .R
at es
of da
ng er
ou s
dr ug
co m
bi na
tio ns
an d
le ng
th of
tim e
fo r
w hi
ch th
os e
co m
bi na
tio ns
w er
e ap
pl ie
d w
as re
du ce
d.
M en
ac he
m ie
ta l.
(2 00
7) .7
7 N
R O
bs er
v (R
et ro
) 5
US H os
p (9
8) Al
lw ar
ds N
/A –
N o.
of IT
Ap pl
ic at
io ns
N R
Ei gh
tp at
ie nt
sa fe
ty in
di ca
to rs
(P SI
s) M
� 1.
82 RR
R in
ra te
s of
m or
ta lit
y .0
24 Th
e gr
ea te
r th
e nu
m be
r of
cl in
ic al
IT ap
pl ic
at io
ns ad
op te
d by
a gi
ve n
ho sp
ita l,
th e
lo w
er th
e ad
ve rs
e ev
en tr
at es
in th
re e
of th
e ei
gh tP
SI s:
de at
h in
lo w
-m or
ta lit
y D
RG s,
de cu
bi tu
s ul
ce rs
,a nd
po st
-o pe
ra -
tiv e
se ps
is .T
he ot
he r
fiv e
in di
ca to
rs di
d no
tr ea
ch st
at is
tic al
si gn
ifi ca
nc e.
M ila
ni et
al (2
01 1)
.7 8
20 09
–2 01
0 (2
4) Co
ho rt
(P ro
sp )
5 US H
os p
(1 )
M ed
ic al
w ar
ds M
D s
35 CD
S CP
O E
47 w
rit te
n or
de rs
33 CP
O E
In -h
os pi
ta lb
le ed
in ga
LO S
90 -d
ay m
or ta
lit y
M AR
R 52
% .0
02 CP
O E
w ith
de ci
si on
su pp
or tr
ed uc
ed ho
s- pi
ta lb
le ed
in g
am on
g pa
tie nt
s w
ith CK
D ad
m itt
ed w
ith AC
S. LO
S an
d 90
-d ay
m or
- ta
lit y
w er
e no
ta ffe
ct ed
.
M or
ris s
et al
. (2
01 1)
.7 9
20 05
–2 00
6 (1
0) O
bs er
v (P
ro sp
) 3
US H os
p (1
) N
eo na
ta lI
CU RN
s –
Ba rc
od e
61 8
Pr ev
en ta
bl e
AD E
as so
ci at
ed w
ith op
io id
ad m
in is
tr at
io n
P 0.
48 .0
45 Ba
rc od
e m
ed ic
at io
n ad
m in
is tr
at io
n sy
s- te
m w
as fo
un d
to si
gn ifi
ca nt
ly re
du ce
th e
ris k
of pr
ev en
ta bl
e AD
Es as
so ci
at ed
w ith
op io
id ad
m in
is tr
at io
n.
M or
ris s
et al
. (2
00 9)
.8 0
N R
(1 2.
5) Co
ho rt
(P ro
sp )
9 US H
os p
(1 )
N eo
na ta
lI CU
RN s
an d
RT s
– Ba
rc od
e eM
AR 95
8 Pr
ev en
ta bl
e AD
Es P
AR R
3% .0
4 Th
e ba
rc od
e m
ed ic
at io
n ad
m in
is tr
at io
n sy
st em
w as
fo un
d to
si gn
ifi ca
nt ly
de cr
ea se
th e
ris k
of pr
ev en
ta bl
e AD
Es .
N ov
is et
al .
(2 01
0) .8
1 20
07 –2
00 8
(1 2)
O bs
er v
(P ro
sp )
3 US H
os p
(1 )
Su rg
ic al
w ar
ds M
D s
– CD
S 80
0 Po
st op
D VT
at 30
da ys
a at
60 da
ys an
d at
90 da
ys
M N
S –
Af te
r im
pl em
en ta
tio n
of CD
S fo
r VT
E pr
o- ph
yl ax
is po
st op
er at
iv el
y, th
er e
w as
a tr
en d
to w
ar d
de cr
ea se
d ra
te s
of po
st -
op er
at iv
e VT
Es ,w
hi ch
di d
no tr
ea ch
si g-
ni fic
an ce
,d ue
to no
tb ei
ng po
w er
ed to
do so
.R at
es of
D VT
pr op
hy la
xi s
or de
rin g
in cr
ea se
d.
O liv
en et
al .
(2 00
2) .8
2 (6
) Co
ho rt
(P ro
sp )
6 Is
ra el
H os
p (1
) Tw
o in
te rn
al m
ed ic
in e
w ar
ds
M D
s N
R CP
O E
13 50
PE s
pr ev
en te
d M
AR R
31 %
< .0
01 Al
th ou
gh no
th e
pr im
ar y
ou tc
om e,
it w
as fo
un d
th at
CP O
E si
gn ifi
ca nt
ly re
du ce
d th
e ra
te of
PE s.
(c on
tin ue
d)
R EVIEW
Brenner S.K, et al. J Am Med Inform Assoc 2016;23:1016–1036. doi:10.1093/jamia/ocv138, Review
1023
Ta bl
e 1:
Co nt
in ue
d
Au th
or s
(y ea
r) St
ud y
Pe rio
d (m
on th
s)
St ud
y D
es ig
n (P
ro sp
or Re
tr os
p)
Q ua
lit y
Sc or
e (0
–1 0)
Co un
tr y
St ud
y Si
te (n
)
Se tti
ng Cl
in ic
ia n
Af fe
ct ed
N o.
Cl in
ic ia
ns H
ea lth
IT In
te rv
en tio
n N
o. Pa
tie nt
s Pa
tie nt
O ut
co m
ea O
ut co
m e
Ef fe
ct Si
ze b
P- va
lu e
Ke y
Fi nd
in gs
O ve
rh ag
e et
al .
(2 00
2) .8
3 19
95 –1
99 6
(1 2)
RC T
(P ro
sp )
7 US H
os p
(2 )
ED M
D s
72 H
IE 32
46 8
H os
pi ta
la dm
is si
on a
Re pe
at ED
vi si
ts M
N S
– Th
er e
w as
no di
ffe re
nc e
in ad
m is
si on
ra te
s or
re pe
at ED
vi si
ts fo
r pa
tie nt
s w
ho se
ED ph
ys ic
ia ns
ha d
ac ce
ss to
el ec
- tr
on ic
m ed
ic al
re co
rd s
fr om
an ot
he r
in st
i- tu
tio n.
Fe w
of th
e ph
ys ic
ia ns
ac tu
al ly
m ad
e us
e of
th e
on lin
e sy
st em
to ch
ec k
O SH
re co
rd s.
Pa re
nt e
an d
M cC
ul lo
ug h.
(2 00
9) .8
4
19 99
–2 00
2 (4
8) O
bs er
v (R
et ro
) 5
US H os
p (N
R) Su
rg ic
al w
ar ds
M D
s RN
s –
EH R
CD S
eM AR
N at
io na
ls am
- pl
e M
ed ic
ar e
cl ai
m s
da ta
In fe
ct io
n du
e to
m ed
i- ca
lc ar
ea Po
st -o
p he
m or
rh ag
e Po
st -o
p VT
E
M N
R –
Ra te
s of
ho sp
ita l-
as so
ci at
ed in
fe ct
io ns
si gn
ifi ca
nt ly
de cr
ea se
d w
ith EH
R us
e, bu
t no
ne of
th e
ot he
r ou
tc om
es w
er e
re du
ce d
by ei
th er
EH R
or th
e ot
he r
he al
th IT
to ol
s st
ud ie
d.
Pa ul
et al
. (2
00 6)
.8 5
20 02
–2 00
4 (1
4) RC
T (P
ro sp
) 5
Is ra
el ,
G er
m an
y, an
d Ita
ly H
os p
(3 )
Al lw
ar ds
M D
s 19
9 co
ho rt
, N
R RC
T CD
S 35
0 in
co ho
rt st
ud y/
23 26
in RC
T
LO Sa
30 -d
ay m
or ta
lit y
M N
S –
At tw
o of
th re
e si
te s,
LO S
w as
si gn
ifi -
ca nt
ly re
du ce
d by
us in
g CD
S fo
r an
tib io
tic se
le ct
io n.
O ve
ra ll,
th er
e w
as no
si gn
ifi ca
nt re
du ct
io n.
30 -d
ay m
or ta
lit y
in th
e in
te n-
tio n
to tr
ea ta
na ly
si s
w as
no ts
ig ni
fic an
tly di
ffe re
nt ei
th er
at an
y of
th e
in di
vi du
al si
te s
no r
ov er
al l.
Pe te
rs on
et al
. (2
00 5)
.8 6
20 01
–2 00
2 (6
) Co
ho rt
(P ro
sp )
3 US H
os p
(1 )
M ed
ic in
e an
d IC
U w
ar ds
M D
s –
CD S
CP O
E 37
18 H
os pi
ta lF
al lr
at ea
LO SD
ay s
of AM
S P
0. 00
36 AR
R fa
lls pe
r 10
0 pt
da ys
.0 01
Th er
e w
as a
si gn
ifi ca
nt re
du ct
io n
of pa
tie nt
in -h
os pi
ta lf
al lr
at es
af te
r im
pl e-
m en
ta tio
n of
CD S.
N o
ef fe
ct w
as fo
un d
on ho
sp ita
ll en
gt h
of st
ay or
da ys
of al
te re
d m
en ta
ls ta
tu s.
Pi on
te k
et al
. (2
01 0)
.8 7
20 01
O bs
er v
(R et
ro )
5 US H
os p
(7 )
Al la
du lt
w ar
ds Ph
ar m
ac is
ts –
AE D
N R
23 0
00 0
ad m
is si
on s
Se ve
rit y-
ad ju
st ed
M or
ta lit
y ra
te sa
LO S
Re ad
m it
Ra te
s
P de
cr ea
se (a
m ou
nt N
R) <
.0 01
Th e
pr im
ar y
ou tc
om e
of se
ve rit
y- ad
ju st
ed m
or ta
lit y
ra te
s w
as si
gn ifi
ca nt
ly lo
w er
in ho
sp ita
ls w
ith AE
D as
co m
pa re
d w
ith co
nc ur
re nt
co nt
ro lg
ro up
an d
pr e-
in te
rv en
tio n
gr ou
p. H
ow ev
er ,t
he re
w as
no si
gn ifi
ca nt
di ffe
re nc
e in
LO S
or re
ad -
m is
si on
ra te
s.
Po on
et al
. (2
01 0)
.8 8
20 05
(9 )
O bs
er v
(P ro
sp )
9 US H
os p
(1 )
Al la
du lt
w ar
ds in
cl ud
in g
IC U
RN s
– Ba
rc od
e eM
AR 17
26 Po
te nt
ia lA
D Es
M AR
R 1.
5% <
.0 01
Th e
ra te
of po
te nt
ia la
dv er
se dr
ug ev
en ts
de cr
ea se
d si
gn ifi
ca nt
ly w
ith us
e of
ba r-
co de
an d
eM AR
.T he
ra te
of po
te nt
ia l
ad ve
rs e
ev en
ts as
so ci
at ed
w ith
tim in
g er
ro rs
di d
no tc
ha ng
e si
gn ifi
ca nt
ly .
(c on
tin ue
d)
R EVIEW
Brenner S.K, et al. J Am Med Inform Assoc 2016;23:1016–1036. doi:10.1093/jamia/ocv138, Review
1024
Ta bl
e 1:
Co nt
in ue
d
Au th
or s
(y ea
r) St
ud y
Pe rio
d (m
on th
s)
St ud
y D
es ig
n (P
ro sp
or Re
tr os
p)
Q ua
lit y
Sc or
e (0
–1 0)
Co un
tr y
St ud
y Si
te (n
)
Se tti
ng Cl
in ic
ia n
Af fe
ct ed
N o.
Cl in
ic ia
ns H
ea lth
IT In
te rv
en tio
n N
o. Pa
tie nt
s Pa
tie nt
O ut
co m
ea O
ut co
m e
Ef fe
ct Si
ze b
P- va
lu e
Ke y
Fi nd
in gs
Po tts
et al
. (2
00 4)
.8 9
20 01
(2 )2
00 2
(2 )
O bs
er v
(P ro
sp )
9 US H
os p
(1 )
Pe di
at ric
w ar
ds M
D s
– CP
O E
51 4
Po te
nt ia
lA D
Es P
AR R
0. 9
po te
nt ia
l AD
Es pe
r 10
0 or
de r
< .0
01 Ra
te s
of po
te nt
ia lA
D Es
w er
e si
gn ifi
ca nt
ly re
du ce
d af
te r
CP O
E im
pl em
en ta
tio n.
Ro llm
an et
al .
(2 00
2) .9
0 19
97 –1
99 8
(2 0)
RC T
(P ro
sp )
1 US Am
b (N
R) Pr
im ar
y Ca
re M
D s
17 CD
S EH
R 22
6 Re
m is
si on
of de
pr es
- si
on at
3a an
d 6
m on
th s
M N
S –
Th er
e w
as no
si gn
ifi ca
nt re
du ct
io n
in ra
te s
of re
co ve
ry fr
om de
pr es
si on
w he
n CD
S w
as ad
de d
to an
EH R
to ac
tiv el
y re
m in
d ph
ys ic
ia ns
to m
an ag
e pa
tie nt
s’ M
D D
.
Ro ss
et al
. (2
00 4)
.9 1
20 01
–2 00
2 (1
3) RC
T (P
ro sp
) 2
US Am b
(1 )
Ca rd
ia c
RN s
M D
s –
Pa tie
nt Po
rt al
10 7
M or
ta lit
ya
H os
pi ta
liz at
io ns
ED vi
si ts
M N
S –
In cr
ea se
d m
ed ic
at io
n ad
he re
nc e
ap pr
oa ch
ed bu
td id
no tr
ea ch
si gn
ifi ca
nc e
in th
e in
te rv
en tio
n gr
ou p
us in
g a
pa tie
nt po
rt al
.T he
nu m
be r
of pa
tie nt
s w
ho vi
si te
d th
e ED
w as
no td
iff er
en t,
bu tt
he nu
m be
r of
vi si
ts w
as hi
gh er
in th
e in
te rv
en tio
n gr
ou p.
M or
ta lit
y an
d ho
sp ita
liz at
io ns
w er
e al
so no
ts ig
ni fic
an tly
di ffe
re nt
.
Ri nd
et al
. (1
99 4)
.9 2
19 90
–1 99
1 (1
8) Ti
m e
Se rie
s (P
ro sp
) 2
US H os
p (1
) Al
la du
lt w
ar ds
M D
s –
CD S
56 2
Re na
li m
pa irm
en t
P RR
0. 45
– W
he n
co m
pu te
r- ba
se d
al er
ts w
er e
ac ti-
va te
d du
rin g
th e
in te
rv en
tio n
pe rio
ds th
e ra
te s
of se
rio us
re na
li m
pa irm
en tw
as di
m in
is he
d.
Ro th
sc hi
ld et
al .
(2 00
5) .9
3 20
02 (1
1) Ti
m e
se rie
s (P
ro sp
) 3
US H os
p (1
) Ca
rd ia
c Su
rg ic
al IC
U an
d st
ep -
do w
n un
its
RN s
– Sm
ar tP
um p
73 5
Se rio
us AD
Es a
N on
- in
te rc
ep te
d po
te nt
ia l
AD Es
M N
S –
Th er
e w
as no
m ea
su ra
bl e
im pa
ct of
sm ar
t pu
m ps
on th
e se
rio us
m ed
ic at
io n
er ro
r ra
te an
d no
n- in
te rc
ep te
d po
te nt
ia l
ad ve
rs e
dr ug
ev en
ts .A
ut ho
rs po
st ul
at ed
th at
la ck
of co
m pl
ia nc
e pl
ay ed
a fa
ct or
in th
e la
ck of
ef fe
ct iv
en es
s ob
se rv
ed .
Sc hn
ip pe
r et
al .
(2 00
9) .9
4 20
05 –2
00 6
(1 2)
O bs
er v
(P ro
sp )
6 US H
os p
(1 )
1 M
ed ic
al w
ar d
Pa s
2 CD
S CP
O E
16 9
Pe rc
en ta
ge of
pa tie
nt da
ys w
ith H
yp og
ly ce
m ia
a LO
S
M N
S –
Pa tie
nt da
ys of
hy po
gl yc
em ia
di d
no t
ch an
ge by
ad di
ng CP
O E
to th
e ad
m is
si on
or de
r se
tf or
di ab
et ic
pa tie
nt s.
Ad ju
st ed
LO S
di d
de cr
ea se
by 25
% .
Sh ul
m an
et al
. (2
00 5)
.9 5
20 01
–2 00
2 (1
5) O
bs er
v (P
ro sp
) 9
UK H os
p (1
) IC
U M
D s
– CP
O E
38 7
AD Es
M N
S .5
1 Ra
te of
m aj
or or
m od
er at
e er
ro rs
af fe
ct in
g pa
tie nt
s w
er e
un ch
an ge
d in
th e
CP O
E gr
ou p
vs th
e ha
nd w
rit te
n pr
es cr
ip tio
n gr
ou p.
Th er
e w
as a
re du
ct io
n of
m aj
or /
m od
er at
e pa
tie nt
ou tc
om es
w he
n no
n- in
te rc
ep te
d an
d in
te rc
ep te
d er
ro rs
w er
e co
m bi
ne d
(.0 1)
.
(c on
tin ue
d)
R EVIEW
Brenner S.K, et al. J Am Med Inform Assoc 2016;23:1016–1036. doi:10.1093/jamia/ocv138, Review
1025
Ta bl
e 1:
Co nt
in ue
d
Au th
or s
(y ea
r) St
ud y
Pe rio
d (m
on th
s)
St ud
y D
es ig
n (P
ro sp
or Re
tr os
p)
Q ua
lit y
Sc or
e (0
–1 0)
Co un
tr y
St ud
y Si
te (n
)
Se tti
ng Cl
in ic
ia n
Af fe
ct ed
N o.
Cl in
ic ia
ns H
ea lth
IT In
te rv
en tio
n N
o. Pa
tie nt
s Pa
tie nt
O ut
co m
ea O
ut co
m e
Ef fe
ct Si
ze b
P- va
lu e
Ke y
Fi nd
in gs
Sm al
le ta
l. (2
00 8)
.9 6
20 05
(5 )
Co ho
rt (P
ro sp
) 6
UK H os
p (1
) O
nc ol
og y
w ar
ds M
D s
3 CP
O E
N R
AD Es
P AR
R 8.
6% <
.0 00
1 CP
O E
re du
ce d
ad ve
rs e
dr ug
ev en
ts .E
rr or
ty pe
di st
rib ut
io n
di ffe
re d
si gn
ifi ca
nt ly
am on
g CP
O E
or de
rs vs
m an
ua lo
rd er
s. CP
O E
w as
as so
ci at
ed w
ith fe
w er
do se
ca lc
ul at
io n
er ro
r ra
te s.
CP O
E de
cr ea
se d
th e
ra te
of m
in or
er ro
rs ,w
ith hi
gh er
pr o-
po rt
io n
of si
gn ifi
ca nt
an d
lif e-
th re
at en
in g
er ro
rs re
la tiv
e to
to ta
le rr
or s.
Sm ith
et al
. (2
01 0)
.9 7
20 06
–2 00
9 (3
1) RC
T (P
ro sp
) 2
UK Am b
(2 9)
Pr im
ar y
Ca re
M D
s –
CD S
91 1
Ex ac
er ba
tio ns
a
H os
pi ta
liz at
io ns
M N
S –
Th er
e w
er e
no si
gn ifi
ca nt
di ffe
re nc
es in
ex ac
er ba
tio ns
w ith
us e
of CD
S. D
is ag
gr eg
at io
n of
th e
co m
po si
te ou
tc om
e de
m on
st ra
te d
th at
in te
rv en
tio n
of al
er ts
w as
fo un
d to
re du
ce th
e od
ds of
pa tie
nt s
re qu
iri ng
ho sp
ita liz
at io
n. Ra
te s
of ED
vi s-
its ,a
fte r
ho ur
s co
nt ac
ts w
ith ph
ys ic
ia ns
w as
no ts
ig ni
fic an
tly di
ffe re
nt .R
at es
of pr
ed ni
so lo
ne us
e w
er e
in cr
ea se
d by
in te
rv en
tio n.
Ti er
ne y
et al
. (2
00 5)
.9 8
19 94
–1 99
6 (3
6) RC
T (P
ro sp
) 6
US Am b
(4 )
Pr im
ar y
Ca re
M D
s an
d Ph
ar m
ac is
ts 27
4 M
D s
20 ph
ar m
ac is
ts CD
S 70
6 ED
vi si
ts a
H os
pi ta
liz at
io ns
M N
S –
Th er
e w
as no
si gn
ifi ca
nt di
ffe re
nc e
in ED
vi si
ts or
ho sp
ita liz
at io
ns ra
te s
be tw
ee n
CO PD
pa tie
nt s
of ph
ys ic
ia ns
w ho
ha d
CD S
an d
th os
e w
ho di
d no
t.
Up pe
rm an
et al
. (2
00 5)
.9 9
20 02
(9 )
Co ho
rt (b
ot h
Re tr
o an
d Pr
os p)
4 US H
os p
(1 )
Al lp
ed ia
tr ic
in pa
tie nt
s M
D s
– CP
O E
N R
AD Es
M N
S –
Al th
ou gh
in ag
gr eg
at e,
ra te
s AD
Es w
er e
no ts
ig ni
fic an
tly di
ffe re
nt af
te r
CP O
E im
pl em
en ta
tio n,
ha rm
fu lA
D Es
w er
e si
g- ni
fic an
tly re
du ce
d w
ith CP
O E
us ag
e w
ith a
N N
T of
1 AD
E pe
r 64
pa tie
nt -d
ay s.
Va n
D oo
rm aa
l et
al .(
20 09
).1 0
0 20
05 –2
00 8
(3 7)
Co ho
rt (P
ro sp
) 5
Th e
N et
he rla
nd s
H os
p (2
)
Tw o
ho sp
ita l
w ar
ds pe
r ho
sp ita
l
M D
s N
R CP
O E
CD S
N R
pA D
Es a
AE s
M N
S –
CP O
E in
co m
bi na
tio n
w ith
CD S
w as
no t
st at
is tic
al ly
as so
ci at
ed w
ith a
re du
ct io
n in
pr ev
en ta
bl e
AD Es
.
W al
sh et
al .
(2 00
8) .1
0 1
20 01
–2 00
2 (1
6) Ti
m e
Se rie
s (P
ro sp
) 5
US H os
p (1
) Al
lp ed
ia tr
ic w
ar ds
M D
s –
CP O
E N
R Se
rio us
M ed
ic at
io n
Er ro
rs M
N S
– O
ve ra
ll, ra
te s
of no
n- in
te rc
ep te
d se
rio us
m ed
ic at
io n
er ro
rs w
er e
no ts
ig ni
fic an
tly le
ss .I
n th
e N
IC U
an d
PI CU
,b ut
no ti
n th
e ge
ne ra
lp ed
ia tr
ic w
ar ds
,s er
io us
m ed
ic a-
tio n
er ro
rs w
er e
si gn
ifi ca
nt ly
le ss
af te
r CP
O E
im pl
em en
ta tio
n. Ti
m e
se rie
s an
al y-
si s
de m
on st
ra te
d th
at ra
te s
of m
ed ic
at io
n er
ro rs
va rie
d si
gn ifi
ca nt
ly w
ith tim
e of
ye ar
,i rr
es pe
ct iv
e of
CP O
E im
pl em
en ta
- tio
n, in
w hi
ch m
on th
s ea
rli er
in th
e ac
a- de
m ic
ye ar
yi el
de d
hi gh
er ro
r ra
te s.
(c on
tin ue
d)
R EVIEW
Brenner S.K, et al. J Am Med Inform Assoc 2016;23:1016–1036. doi:10.1093/jamia/ocv138, Review
1026
Ta bl
e 1:
Co nt
in ue
d
Au th
or s
(y ea
r) St
ud y
Pe rio
d (m
on th
s)
St ud
y D
es ig
n (P
ro sp
or Re
tr os
p)
Q ua
lit y
Sc or
e (0
–1 0)
Co un
tr y
St ud
y Si
te (n
)
Se tti
ng Cl
in ic
ia n
Af fe
ct ed
N o.
Cl in
ic ia
ns H
ea lth
IT In
te rv
en tio
n N
o. Pa
tie nt
s Pa
tie nt
O ut
co m
ea O
ut co
m e
Ef fe
ct Si
ze b
P- va
lu e
Ke y
Fi nd
in gs
W ei
ng ar
te ta
l. (2
00 9)
.1 0
2 20
06 (6
) O
bs er
v (P
ro sp
) 4
US Am b
(N R)
Ad ul
tP rim
ar y
Ca re
, Pe
di at
ric ,
Ps yc
hi at
ry ,
an d
ot he
r sp
ec ia
lti es
M D
s an
d ph
ys ic
ia n-
ex te
nd er
s
23 21
CD S
60 35
2 pA
D Es
a di
sa bi
lit yh
o- sp
ita liz
at io
ns ED
vi si
ts of
fic e
vi si
ts
P 33
1 al
er ts
ne ed
ed to
pr e-
ve nt
1 AD
E
– Al
er ts
w er
e re
sp on
si bl
e fo
r pr
ev en
tin g
AD Es
as w
el la
s ho
sp ita
liz at
io ns
,E D
vi s-
its ,a
nd of
fic e
vi si
ts .
Yu et
al .
(2 00
9) .1
0 3
20 05
–2 00
6 (1
2) Ca
se -c
on tr
ol (R
et ro
) 3
US H os
p (1
22 )
Al lp
ed ia
tr ic
w ar
ds N
/A –
CP O
E 1
15 1
93 2
AD Es
P O
R 1.
42 –
H os
pi ta
ls w
ith ou
tC PO
E ha
d 42
pe rc
en t
hi gh
er ra
te s
of AD
Es .
Za ne
tti et
al .
(2 00
3) .1
0 4
20 00
(4 )
RC T
(P ro
sp )
1 US H
os p
(1 )
O R
M D
s –
CD S
44 9
ca se
s Ra
te of
su rg
ic al
si te
in fe
ct io
ns M
AR R
2% .4
Su rg
ic al
si te
-i nf
ec tio
n ra
te s
w er
e re
du ce
d co
m pa
re d
w ith
pr e-
st ud
y pe
rio d,
bu tn
ot si
gn ifi
ca nt
ly re
du ce
d co
m pa
re d
to co
nc ur
- re
nt co
nt ro
ls w
he n
CD S
w as
us ed
to al
er t
ph ys
ic ia
ns to
gi ve
in tr
ao pe
ra tiv
e an
tib io
tic s.
a P rim
ar y
ou tc
om es
no te
d w
ith as
te ris
ks .b
Al le
ffe ct
si ze
s ar
e re
po rt
ed fo
r th
e pr
im ar
y ou
tc om
e lis
te d.
Ab br
ev ia
tio ns
: Pr
os p ¼
pr os
pe ct
iv e;
Re tr
os p
or Re
tr o ¼
re tr
os pe
ct iv
e; H
ea lth
IT ¼
H ea
lth in
fo rm
at io
n te
ch no
lo gy
; Co
nf In
t¼ co
nf id
en ce
in te
rv al
; O
bs er
v ¼
ob se
rv at
io na
l st
ud y;
RC T ¼
ra nd
om iz
ed co
nt ro
l tr
ia l;
H os
p ¼
ho sp
ita l
se tti
ng ;
Am b ¼
am bu
la to
ry ca
re se
tti ng
; N
R ¼
no t
re po
rt ed
; N
S ¼
no t
si gn
ifi ca
nt pr
im ar
y ou
tc om
e; LT
C ¼
lo ng
te rm
ca re
fa ci
lit y;
IC U ¼
in te
ns iv
e ca
re un
it; RE
I¼ re
pr od
uc tiv
e en
do cr
in ol
og y
an d
in fe
rt ili
ty ;
ED ¼
em er
ge nc
y de
pa rt
m en
t; O
R ¼
op er
at in
g ro
om or
od ds
ra tio
; M
D s ¼
ph ys
ic ia
ns ;
RN s ¼
nu rs
es ;
PA ¼
ph ys
ic ia
ns ’
as si
st an
ts ;
N Ps ¼
nu rs
e pr
ac tit
io ne
rs ;
N /A ¼
no t
ap pl
ic ab
le ;
CD S ¼
cl in
ic al
de ci
si on
su pp
or t;
CP O
E ¼
co m
pu te
riz ed
pr ov
id er
or de
r en
tr y;
EH R ¼
el ec
tr on
ic he
al th
re co
rd ;
H IE ¼
H ea
lth in
fo rm
at io
n ex
ch an
ge ;A
ut oD
is p ¼
au to
m at
ed di
sp en
sa tio
n of
m ed
ic at
io n;
eM AR ¼
el ec
tr on
ic m
ed ic
at io
n ad
m in
is tr
at io
n re
co rd
;A ED ¼
au to
m at
ed er
ro r
de te
ct io
n sy
st em
;e M
ed Re
c ¼
el ec
tr on
ic m
ed ic
at io
n re
co nc
ili at
io n;
Al g ¼
el ec
tr on
ic cl
in ic
al pa
th w
ay ;
G N ¼
gr am
ne ga
tiv e;
LO S ¼
le ng
th of
st ay
; C.
di ff ¼
Cl os
tr id
iu m
di ffi
ci le
; M
RS A ¼
M et
hi ci
lli n-
re si
st an
t St
ap hy
lo co
cc us
au re
us ;
AD E ¼
ad ve
rs e
dr ug
ev en
t; AE ¼
ad ve
rs e
ev en
t; pA
D E ¼
pr ev
en ta
bl e
ad ve
rs e
dr ug
ev en
t; AK
I¼ ac
ut e
ki dn
ey in
ju ry
; AM
I or
M I¼
ac ut
e m
yo ca
rd ia
l in
fa rc
tio n;
CH F ¼
co ng
es tiv
e he
ar t
fa ilu
re ;
PN A ¼
pn eu
m on
ia ;
VT E ¼
ve no
us th
ro m
bo em
bo lis
m ;
PP X ¼
pr op
hy la
xi s;
D VT ¼
de ep
ve no
us th
ro m
bo si
s; PE ¼
pu lm
on ar
y em
bo lis
m ;
po st
-o p ¼
po st
-o pe
ra tiv
el y;
AM S ¼
al te
re d
m en
ta ls
ta tu
s; M ¼
no t
si gn
ifi ca
nt or
m ix
ed re
su lts
st ud
y; P ¼
po si
tiv e
st ud
y; N
eg ¼
ne ga
tiv e
st ud
y (h
ea lth
IT w
as fo
un d
to be
ha rm
fu l);
CO PD ¼
ch ro
ni c
ob st
ru ct
io n
pu lm
on ar
y di
se as
e; AR
R ¼
ab so
lu te
ris k
re du
ct io
n; N
N T ¼
nu m
be r
ne ed
ed to
tr ea
t.
R EVIEW
Brenner S.K, et al. J Am Med Inform Assoc 2016;23:1016–1036. doi:10.1093/jamia/ocv138, Review
1027
non-significant studies were those for which the primary outcome had a non-significant result but secondary outcomes had a positive result, or studies in which all outcomes were non-significant.15 Studies for which patient safety outcomes were not the primary outcome studied were included and the non-patient safety outcome endpoints were not analyzed. Consensus was reached during review discussions. A narra- tive synthesis method was used to integrate the findings into descrip- tive summaries. Sub-analyses of positive studies, mixed studies, and randomized controlled trials were conducted.
RESULTS The search strategy identified 6138 articles. After removal of duplicate articles and articles available only in a foreign language, there were
4736 articles that underwent title review. Based on title alone, 817 (17%) were considered not appropriate for the study. Another 344 arti- cles were added based on a review of the references of the systematic reviews found during the title review process. A total of 4263 articles then underwent abstract review, with 135 articles included for full two-person review. Sixty-eight articles met all of the study inclusion criteria (Figure 1).
Reviewer Agreement Of the 135 papers reviewed by 2 reviewers, agreement about eligibility for inclusion in the systematic review was excellent 90.4% (k¼80.9%; 95% CI, 71.0-90.8%). Of the 69 studies included in the fi- nal review, the level of chance-corrected agreement for scientific merit
Figure 1: Study identification and selection.
Table 2: Study Quality Rating Scale
Potential Source of Bias Score
2 1 0
Allocation Bias Randomized Quasi-randomized Concurrent controls
Unit of Allocation Bias Cluster-analysis (i.e.,: practice or ward) Physician-based analysis Patient-level analysis
Baseline Group Characteristics No baseline differences or appropriate statistical adjustments for differences
Baseline differences present with no statistical adjustments
Baseline differences not reported
Objectivity of Outcome Objective outcomes with blinded assessment
Objective outcomes without blinding Subjective outcomes with no blinding and poorly defined
Completeness of Follow-up >90% 80–90% <80% or not described
R EVIEW
Brenner S.K, et al. J Am Med Inform Assoc 2016;23:1016–1036. doi:10.1093/jamia/ocv138, Review
1028
between reviewers was excellent, with a quadratic-weighted k statis- tic of 88.9% (95% CI, 84.6-93.3%).
Descriptive Analysis of All Studies Types of health IT and outcomes studied There was at least one article for every type of health IT pre-identified. More than one health IT was analyzed in 22 studies (31%), and in those cases, all of the health IT tools studied were included in the anal- ysis. The most common health IT interventions were CDS (n¼40) and CPOE (n¼27) (Table 3). Four health IT tools (electronic medication reconciliation, electronic clinical pathways, patient portal, and smart pumps) were included in only one study, and another 2 tools (HIE and automated medication dispensing) were only found in 2 studies each.
The patient safety outcomes studied varied widely. The most com- mon outcomes studied included: ADEs and adverse events (53% of studies), mortality (26%), thrombosis or bleed (14%), LOS (12%), and infection rates (10%). Secondary outcomes were included in the anal- ysis to capture the broadest number of patient outcomes (Table 3).
Study setting and participants Most of the studies (n¼59, 86%) were performed in inpatient set- tings. A multicenter study design was employed in 19 (28%) of the
Table 3: Summary of Findings
Characteristic Total (%) Positive Studies
Non- significant or Mixed Results Studies
Negative Studies
Total (%) 69 (100) 25 (36) 43 (62) 1 (1)
Study Design
Randomized Control Trial 18 (26) 5 (7) 13 (19) –
Cohort 21 (30) 8 (12) 13 (19) –
Observational 22 (31) 10 (14) 12 (17) 1 (1)
Time Series 7 (10) 2 (3) 5 (7) –
Case-Control 1 (1) 1 (1) – –
Setting
Inpatient 59 (86) 25 (36) 33 (48) 1 (1)
Outpatient 10 (14) 1 (1) 9 (13) –
Long-Term Care 1 (1) – 1 (1) –
Multi-Center 19 (28) 4 (6) 15 (22) –
Clinicians Affected
Physicians 55 (80) 19 (28) 36 (52) 1 (1)
Nurses 10 (14) 4 (6) 6 (9) –
Other 5 (7) 2 (3) 3 (4) –
Pharmacists 2 (3) 1 (1) 1 (1) –
Country
United States 51 (75) 19 (28) 31 (46) 1 (1)
Non-United States 19 (28) 7 (10) 12 (17) –
Methodological Quality Assessment Score
0–3 20 (29) 10 (14) 10 (14) –
4–6 34 (49) 11 (16) 22 (32) 1 (1)
7–10 15 (22) 4 (6) 11 (16) –
Type of Health IT Intervention Studied
Clinical decision support (CDS)
40 (58) 15 (22) 25 (36) –
Computerized provider order entry (CPOE)
27 (39) 10 (14) 16 (23) 1 (1)
Automated error detection
4 (6) 2 (3) 2 (3) –
Electronic medication administration record (eMAR)
4 (6) 1 (1) 3 (4) –
Electronic health record (EHR)
4 (6) – 4 (6) –
Med Administration Barcodes
3 (4) 2 (3) 1 (1) –
Health information exchange (HIE)
2 (3) 1 (1) 1 (1) –
Automated dispensing 2 (3) 1 (1) 1 (1) –
Electronic medication reconciliation
1 (1) – 1 (1) –
(continued)
Table 3: Continued
Characteristic Total (%) Positive Studies
Non- significant or Mixed Results Studies
Negative Studies
Electronic Clinical Pathways 1 (1) – 1 (1) –
Patient Portal 1 (1) – 1 (1) –
Smart pumps 1 (1) 1 (1) 1 (1) –
No. of IT Applications 1 (1) – 1 (1) –
Patient Outcomes Studied
Adverse Drug Events or Adverse Events
37 (53) 17 (25) 20 (29) –
Mortality 18 (26) 5 (7) 12 (18) 1 (1)
Readmission, admission, or Emergency dept. visits
16 (24) 4 (6) 12 (18) –
Thrombosis or Bleed 10 (14) 3 (4) 7 (10) –
Length of Stay 8 (12) 4 (6) 4 (6) –
Infection Rates 8 (10) 5 (7) 3 (4) –
Fall Rates 3 (4) 2 (3) 1 (1) –
Hemodynamic Instability or ICU transfer
3 (4) 1 (1) 2 (3) –
Myocardial Infarction or Cardiac Events
3 (4) – 3 (4) –
Chronic Disease Exacerbations
3 (4) 1 (1) 2 (3) –
Altered Mental Status or Stroke incidence
2 (3) – 2 (3) –
Pressure Ulcers 1 (1) – 1 (1)
Note: Studies can be counted in more than one category where appli- cable. Abbreviations: dept.¼department; ICU¼ Intensive care unit. Other category under clinicians refers to either not reported or not ap- plicable study population.
R EVIEW
Brenner S.K, et al. J Am Med Inform Assoc 2016;23:1016–1036. doi:10.1093/jamia/ocv138, Review
1029
studies with the majority (15 of these 19, 79%) of the multicenter trials resulting in non-significant clinical outcomes. Multicenter design was used in 75% (6 of 8) of outpatient studies as compared to only 22% (12 of 55) of inpatient studies. The vast majority (80%) of studies as- sessed physicians, rather than other healthcare practitioners and most studies were conducted in the United States (75%). There were stud- ies for which authors did not specify the clinicians affected by their health IT, nor was it clear that clinicians were a subject under study from the text. In these cases, reviewers classified the clinicians as “NR,” for not reported (Table 1).
Study quality The study designs were roughly evenly distributed between RCTs, co- hort, and observational design studies, with only a few time-series and case-control designed studies. Study quality was approximately evenly distributed across each grouping of ratings (0–3; 4–6; 7–10) (Table 3). Unlike prior studies, we did not find that there was a signifi- cant increase in the quality of studies over time.11,17 In terms of qual- ity assessment, the weakest aspects of study design tended to be with regard to randomization and allocation. Specifically, the majority of studies failed to have randomization or even a concurrent control group as part of the study design, and most allocation was done at the patient, rather than unit level. Eighty-one percent of studies received a 1 for blinding of outcomes (which meant objective outcomes were as- sessed without blinding), and 61% of studies received a 2 for follow up (indicating >90% follow up achieved and reported). Reporting of baseline characteristics was variable and evenly distributed between scores of 0, 1, and 2. Notably, this pattern for quality assessment held true for all studies as well as in sub-analysis of positive vs mixed and negative studies Only 10 (24%) of the non-significant studies enrolled over 1000 patients whereas, 11 (44%) of the studies which found a positive effect of health IT on patient outcomes had enrolled more than 1000 patients (Table 4). Larger studies (n > 1000 patients) were also more likely to be conducted more recently than smaller studies.
Effects of health IT on patient safety outcomes Of the 69 studies, the majority (n¼43, 63%) had either non-signifi- cant findings with respect to patient safety outcomes, or mixed out- comes. Only 25 (36%) studies showed a statistically significant positive effect of health IT on the primary patient safety outcome as- sessed. There was also 1 (1%) study that found that health IT resulted in an increased mortality rate. There was a significant increase in the number of studies published on health IT and patient safety outcomes over time (Figure 2).
Analysis of Positive Studies The 25 studies that found that health IT had a positive effect on the primary patient safety outcome were mostly observational trials (40%) or cohort trials (30%). The majority of the positive studies were single center trials (n¼20), conducted in the United States (n¼19).
The vast majority of studies that found a positive effect of health IT occurred in the inpatient setting (n¼24, 96%). There was only one trial demonstrating a positive effect of health IT on patient safety out- comes in the outpatient setting, and none in the long-term care set- ting. There was no significant difference in the sample sizes or quality score of the positive studies as compared the mixed result or null studies (Table 3).
Positive benefit on patient safety outcomes was demonstrated in studies evaluating CDS, CPOE, HIE, automated error detection, eMAR, medication administration barcodes, automated dispensing, and smart pumps. The health outcomes involved were adverse events (n¼16
Table 4: Analysis of Studies Categorized as Mixed Results Studies
Characteristic Total Mixed Results Studies (%)
Non-significant Studies (%)
Mixed Studies: some positive results, some non-significant results (%)
Total (%) 43 (100) 24 (56) 19 (44)
Study Design
Randomized Control Trial 13 (30) 9 (21) 4 (9)
Cohort 16 (37) 8 (19) 8 (19)
Observational 14 (33) 7 (16) 7 (16)
Setting
Inpatient 23 (53) 16 (37) 17 (40)
Outpatient 9 (21) 7 (16) 2 (5)
Long-Term Care 1 (2) 1 (2) –
Multi-Center 15 (35) 9 (21) 6 (14)
Methodological Quality Assessment Score
0–3 10 (23) 6 (14) 4 (9)
4–6 22 (51) 12 (28) 10 (23)
7–10 11 (26) 6 (14) 5 (12)
Type of Health IT Intervention Studied
Clinical decision support (CDS) 26 (60) 15 (36) 11 (26)
Computerized provider order entry (CPOE)
17 (40) 7 (16) 11 (26)
Automated error detection 2 (5) 2 (5) –
Electronic medication administration record (eMAR)
3 (7) 1 (2) 2 (5)
Electronic health record (EHR) 3 (7) 2 (5) 1 (2)
Medication Administration Barcode
1 (2) – 1 (2)
Health information exchange (HIE) 1 (2) 1 (2) –
Automated dispensing 1 (2) 1 (2) –
Electronic medication reconciliation
1 (2) 1 (2) –
Patient Portal 1 (2) 1 (2) –
Smart pumps 1 (2) 1 (2) –
No. of IT Applications 1 (2) – 1 (2)
Patient Outcomes Studied
Adverse Drug Events or Adverse Events
18 (42) 9 (21) 9 (21)
Readmission, admission, or Emergency dept. visits
12 (28) 9 (21) 3 (5)
Mortality 10 (23) 7 (17) 3 (5)
Thrombosis or Bleed 9 (21) 6 (14) 4 (9)
Length of Stay 4 (9) – 4 (9)
Infection Rates 4 (9) 1 (2) 3 (5)
Myocardial Infarction or Cardiac Events
3 (7) 2 (5) 1 (2)
Hemodynamic Instability or ICU transfer
2 (5) 2 (5) –
Chronic Disease Exacerbations 2 (5) – 2 (5)
Fall Rates 1 (2) 1 (2) –
Pressure Ulcers 1 (2) 1 (2)
Abbreviations: dept.¼department; ICU¼ Intensive care unit.
R EVIEW
Brenner S.K, et al. J Am Med Inform Assoc 2016;23:1016–1036. doi:10.1093/jamia/ocv138, Review
1030
studies), mortality (n¼4), LOS (n¼4), readmission rates or ED visits (n¼2), prevention or reduction of thrombosis or bleeding (n¼2), in- fection rates (n¼2), and rates of pressure ulcers or falls (n¼2), AMS or stroke incidence (n¼1), and hemodynamic instability or ICU admis- sion (n¼1). The patient safety outcomes for which there were more positive studies than mixed studies were LOS, renal impairment, and fall or pressure ulcer rates (Table 3).
In conducting further sub-analysis of the studies characterized as mixed results studies, it was found that more than half of those stud- ies had non-significant findings with respect to all patient safety out- comes. The remaining studies categorized as mixed results had some secondary patient safety outcomes that were positive (Table 4).
In order to determine which types of outcomes were positively af- fected by which types of health IT, the effective combinations of the two were analyzed. Overall, CDS, CPOE, or CPOE combined with CDS accounted for 73% of the interventions that were successful. The only health outcomes for which those health IT interventions did not consti- tute the majority was for infection rates, pressure ulcers, or hemody- namic instability or transfer to the ICU (Table 5).
Subgroup Analysis: RCTs Only Of all the study types, RCTs had the smallest percentage of studies demonstrating positive effect of health IT on safety outcomes (n¼5, 28%), as compared with all other studies (n¼20, 40%). Again, as for the entire group of studies, inpatient studies, physician studies, and US studies were all more common among RCTs (Table 6). There was a much smaller increase in the number of RCT studies published over time, as compared to all studies (Figure 2).
The quality of the RCTs was significantly higher than the non-RCT studies (P < .001). The quality of the RCT studies did not improve over time (mean RTCs before 2003, 6.9 and after 2003, 7.2), unlike previ- ously reported.11,17
Most RCTs studied patient mortality and readmission, admission, and ED visits. For these outcomes, only one study found a benefit of health IT (Table 6).102
DISCUSSION Overall Significance Our finding that most studies had mixed, rather than positive effects on patient safety outcomes, is consistent with almost all prior system- atic reviews conducted on health IT and patient safety outcomes. We also found a paucity of outpatient studies, studies evaluating large numbers of patients, and randomized control trials. Given the national priority placed on adoption and use of health IT, our work highlights the urgent need to better evaluate the use of multiple types of health IT on a variety of patient safety outcomes and in a variety of healthcare settings.
Summary of Findings Demonstrating the benefit of health IT is challenging for several rea- sons. First, adverse patient outcomes that can be expected to be mod- ified by the implementation of health IT are generally rare events, necessitating large study samples.108,109 We only found 21 studies (31%) that had > 1000 patient study subjects. In addition, randomized control trials evaluating health IT are difficult to conduct, limiting the quality of evidence on this topic. Randomization is generally not feasi- ble within an individual unit or practice, and thus has to be conducted
Figure 2: Number of studies published as a function of publication year. Even though the number of studies published on health informa- tion technology (IT) has increased significantly, the number of randomized controlled trials (RTC) published annually has only seen a mod- est increase over time.
R EVIEW
Brenner S.K, et al. J Am Med Inform Assoc 2016;23:1016–1036. doi:10.1093/jamia/ocv138, Review
1031
across settings. In addition, health IT is costly to purchase, resource- intensive to implement and typically purchased for an entire practice or institution. Not surprisingly, we found that the rates of randomized control trials assessing the effects of health IT on patient safety out- comes are increasing more slowly than the rate of research on this topic overall (Figure 2).
Consistent with those constraints, in regards to the types of health IT studied, we found that CDS was the most commonly studied health IT intervention. This is likely due to its inherent nature—it is a soft- ware-based intervention that can be turned on and turned off, making it well suited for randomized control, before-and-after, or time series designs. Furthermore, because it is software-based it can be trialed at multiple institutions at once; as such, 63% of the multicenter trials fo- cused on CDS. In contrast to CDS, however, many of the individual tools studied had only one or two quantitative publications.
We found only 10 studies conducted in the outpatient setting, despite the fact that the majority of care is given in the outpatient setting.4,110
Similarly, we found only 1 study conducted in the long-term care setting. While it has been shown that ambulatory care settings have until recently lagged behind larger institutions in engaging in health IT adoption,111,112
given the importance of primary care to population health and prevention, the ambulatory care setting stands to gain a lot from rigorous study of the use of health IT to improve patient safety outcomes.
Future Directions and Policy Implications Given our findings, this review underscores important future directions for this field of research. First, additional large studies are needed to evaluate the effect of health IT on patient safety outcomes, particularly
in the outpatient and long-term care settings. Second, a more uniform system for characterizing health IT tools will be needed to facilitate comparison between studies of health IT interventions. CDS, the most commonly studied health IT, for example, covers a very broad range of actual interventions. Third, as the field continues to develop, more cross-institutional studies and collaborations will be required in order to capture the impact of the newest of the emerging health IT tools, such as patient portals and HIE systems.
From a public policy perspective, discussions are occurring in both the academic community and among regulatory agencies as to how to best regulate health IT. The Federal, Food, Drug, and Cosmetic Act re- cently declared health IT a medical device under regulatory jurisdiction of the US Food and Drug Administration (FDA).113 To date, the FDA has not yet exerted its regulatory authority over the vast majority of health IT tools. The Office of the National Coordinator for Health Information Technology has also published Safety Assurance Factors for EHR Resilience (SAFER) guides designed to help organizations assess and optimize health IT safety.114 However, given the mixed findings of many research studies on the effects of health IT on patient safety outcomes, and one study demonstrating a hazardous effect, ongoing studies will be critical to ensure patients remain safe and to better determine which types and features of health IT actually improve care for patients.
Limitations This review has several key limitations. The first is a direct correlate of the quantity and scope of the literature. Despite performing a compre- hensive search, only a limited set of articles with quantitative data were
Table 5: Analysis of the Health IT Found to be Effective in Improving Specific Patient Safety Outcomes
Total studies¼ 25þ18¼43
Patient Outcome
Type of Health IT Intervention
ADEs or AEs
Mortality Length of Stay
Thrombosis or Bleed
Infection Rates
Readm, adm, or ED visits
Hemodynamic Instability or ICU transfer
Fall Rates
Chronic Disease Exacerbations
AMS or CVA
Pressure Ulcers
Total
CDS 6 3 3 2 2 3 1 1 21
CDS/CPOE 3 3 3 1 10
CPOE 7 2 1 10
AED 1 1 1 3
No. of IT Applications
1 1 1 3
Medication Administr-ation Barcode/eMAR
2 1 3
Smart Pumps 1 1
EHR 1 1
Barcode 1 1
HIE 1 1
AED/CDS 1 1
CDS/EHR/CPOE 1 1
Total 21 7 7 7 5 4 2 1 1 1 1
Abbreviations: ADE¼adverse drug event; AE¼adverse event; ED¼emergency department; Readm¼ readmission; adm¼hospital admission; ICU¼ intensive care unit; AMS¼altered mental status; CVA¼stroke; CDS¼clinical decision support; CPOE¼computerized provider order entry; EHR¼electronic health record; HIE¼Health information exchange; AutoDisp¼automated dispensation of medication; eMAR¼electronic medica- tion administration record; AED¼automated error detection system.
R EVIEW
Brenner S.K, et al. J Am Med Inform Assoc 2016;23:1016–1036. doi:10.1093/jamia/ocv138, Review
1032
identified. For many important types of health IT, only a few studies re- porting the impact on actual patient outcomes were found, even among technologies that are being promoted by government policy. As with all systematic reviews, this review also faced the limitations imposed by publication bias, for which studies with positive results are more likely to be published than those with non-significant findings. Proportionally, however, we did find more studies with non-significant findings than not, which would suggest that our findings may be conservative in their estimate of the number of studies for which no significant effect of health IT was found. We also confined our search to English language publications, which may have precluded us from finding additional rele- vant studies. Given these limitations, it is possible that certain types of technology were underrepresented in this review, such as emerging technologies (like mobile technologies), patient portals, or HIE.
For this review, we chose to use a quality scale that has been pre- viously used and published in measuring the quality of the study of health IT.9,11,17 While there are other widely utilized scales that might have been chosen, such as the Cochrane rating system, the scale we utilized has additional bias analysis categories not contained in other scales which we felt made it most rigorous for the quality analysis we were employing. Lastly, there is considerable heterogeneity as to what defines certain types of health IT. For example, CDS has become an umbrella term for many different types of decision support that can be implemented in different ways. We relied on authors’ classifications for health IT tools in determining the type of health IT evaluated, rather than addressing this level of variability. This assumption may have led to an overrepresentation of CDS in the literature.
The authors also recognize that the impact of health IT is greatly influenced by technical, organizational, political, and social factors. Controlling for these in the context of a systematic review is extremely difficult given that authors of the original studies are often not able to measure or quantify these factors, and instead rely on well-matched controls to mitigate these effects. The rating system of study quality is the authors’ attempt to guide readers as to which studies may most effectively control for larger, broader factors.
CONCLUSION This review has important implications relevant to multiple stake- holders in healthcare, including providers, consumers, policymakers, and vendors. As the nation invests more heavily in health IT, under- standing the effects on patient safety outcomes is critical. While there are certain health IT tools that are well studied and are demonstrating safety benefits for patients, there are many areas that are vastly understudied. This review underscores the need for additional, high quality, large-scale studies in multiple settings to better understand how health IT is actually impacting patients. Without such research, we will not be able to identify which health IT tools are indeed effective and in what settings we can expect the greatest benefit.
AUTHOR STATEMENTS Funding: This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors. Competing Interests: The authors have no competing interests to declare.
Author Contribution S.B.: Study design, data collection, data analysis and interpretation, writing,
editing and figure creation, and editing. R.K.: Study design, data interpretation, writing, editing, and figure editing. Z.G.: Data collection, data analysis and interpretation, manuscript editing. C.J.: Data collection, manuscript editing. I.K.: Data collection, manuscript editing.
Table 6: RCT Trials
Characteristic Total (%) Positive Studies
Non-significant or Mixed Results Studies
Total (%) 18 (100) 5 (28) 13 (72)
Setting
Inpatient 10 (56) 5 (28) 5 (28)
Outpatient 7 (39) – 7 (39)
Long-Term Care 1 (6) – 1(6)
Multi-Center 8 (44) 1 (6) 7 (39)
Clinicians Affected
Physicians 17 (94) 4 (22) 13 (72)
Nurses 2 (11) 1 (6) 1 (6)
Other 1 (6) – 1 (6)
Pharmacists 1 (6) – 1 (6)
Country
USA 12 (67) 3 (17) 9 (50)
Non-USA 6 (33) 2 (11) 4 (22)
Study Quality
0–3 5 (28) 1 (6) 4 (22)
4–6 9 (50) 3 (17) 6 (33)
7–10 4 (22) 1 (6) 3 (17)
Type of Health IT Intervention
Clinical decision support 14 (78) 4 (22) 10 (56)
Computerized provider order entry
3 (17) 1 (6) 2 (11)
Electronic health record 1 (6) – 1 (6)
Smart Pumps 1 (6) 1 (6) –
eMedical Reconciliation 1 (6) – 1 (6)
Health information exchange (HIE)
1 (6) – 1 (6)
Patient Portal 1 (6) – 1 (6)
Patient Outcomes Studied
Mortality 6 (33) – 6 (33)
Readmission, admission, or Emergency dept. visits
6 (33) 1 (6) 5 (28)
Adverse drug events or adverse events
3 (17) 1 (6) 2 (11)
Chronic Disease Exacerbations
3 (17) 1 (6) 2 (11)
Hemodynamic Instability or intensive care unit transfer
3 (17) 1 (6) 2 (11)
Thrombosis or Bleed 2 (11) 1 (6) 1 (6)
Length of stay 2 (11) 1 (6) 1 (6)
Myocardial infarction or Cardiac Events
2 (11) – 2 (11)
Infection Rates 1 (6) – 1 (6)
Fall rates 1 (6) 1 (6) –
Altered mental status or stroke incidence
1 (6) – 1 (6)
R EVIEW
Brenner S.K, et al. J Am Med Inform Assoc 2016;23:1016–1036. doi:10.1093/jamia/ocv138, Review
1033
R.A.: Study design, literature search, manuscript writing and editing. D.D.: Study design, literature search, manuscript writing and editing. E.A.: Study design, data collection, data analysis and interpretation, writing,
editing, and figure editing.
SUPPLEMENTARY MATERIAL Supplementary material is available online at http://jamia.oxfordjournals.org/.
REFERENCES 1. Kohn L, Corrigan J, Donaldson M. To Err is Human: Building a Safer Health
System. Washington DC: Institute of Medicine, Committee on Quality of Health Care in America 2000.
2. Health Information Technology: Revisions to Initial Set of Standards, Implementation Specifications, and Certification Criteria for Electronic Health Record Technology. Final Rule, 45 CFR Part 170. USA: Department of Health and Human Services, Statute of the United States Government 2010.
3. Jha AK, DesRoches CM, Campbell EG, et al. Use of electronic health re- cords in U.S. hospitals. N Engl J Med. 2009;360(16):1628–1638.
4. Chaudhry B, Wang J, Wu S, et al. Systematic review: impact of health in- formation technology on quality, efficiency, and costs of medical care. Ann Intern Med. 2006;144(10):742–752.
5. Conroy S, Sweis D, Planner C, et al. Interventions to reduce dosing errors in children: A systematic review of the literature. Drug Saf. 2007;30(12):1111–1125.
6. Eslami S, de Keizer NF, Abu-Hanna A. The impact of computerized physi- cian medication order entry in hospitalized patients–a systematic review. Int J Med Inform. 2008;77(6):365–376.
7. Georgiou A, Williamson M, Westbrook JI, Ray S. The impact of computer- ised physician order entry systems on pathology services: A systematic re- view. Int J Med Inform. 2007;76(7):514–529.
8. Hider P. Electronic Prescribing: A Critical Appraisal of the Literature. NZHTA Report. Christchurch, New Zealand Health Technology Assessment; 2002;5(2).
9. Jerant AF, Hill DB. Does the use of electronic medical records improve sur- rogate patient outcomes in outpatient settings? J Fam Pract. 2000;49(4):349–358.
10. Van Rosse F, Maat B, Rademaker CMA, van Vught AJ, Egberts ACG, Bollen CW. The effect of computerized physician order entry on medication pre- scription errors and clinical outcome in pediatric and intensive care: a sys- tematic review. Pediatrics. 2009;123(4):1184–1190.
11. Hunt DL, Haynes RB, Hanna SE, Smith K. Effects of computer-based clini- cal decision support systems on physician performance and patient out- comes: a systematic review. JAMA. 1998;280(15):1339–1346.
12. Mollon B, Chong JJ, Holbrook AM, Sung M, Thabane L, Foster G. Features predicting the success of computerized decision support for prescribing: A systematic review of randomized controlled trials. BMC Med Inform Decis Mak. 2009;9(11):1–9.
13. Montgomery AA, Fahey T. A systematic review of the use of computers in the management of hypertension. J Epidemiol Community Health. 1998;52(8):520–525.
14. Shojania KG, Jennings A, Mayhew A, Ramsay CR, Eccles MP, Grimshaw J. The effects of on-screen, point of care computer reminders on processes and outcomes of care. Cochrane Database Syst Rev. 2009;(CD001096):1–68.
15. Bryan C, Boren SA. The use and effectiveness of electronic clinical decision support tools in the ambulatory/primary care setting: a systematic review of the literature. Inform Prim Care. 2008;16(2):79–91.
16. Chang J, Langberg M, Silksa P, Dietrich A. Improving outcomes through the use of inpatient order sets: A systematic review. J Gen Intern Med. 2010;25:S308–S309.
17. Garg A, Adhikari N, McDonald H, et al. Effects of computerized clinical de- cision support systems on practitioner performance and patient outcomes. JAMA. 2005;293(10):1223–1238.
18. Johnston ME, Langton KB, Haynes RB, Mathieu A. Effects of computer- based clinical decision support systems on clinician performance and pa- tient outcome. A critical appraisal of research. Ann Intern Med. 1994;120(2):135–142.
19. Schedlbauer A, Prasad V, Mulvaney C, et al. What evidence supports the use of computerized alerts and prompts to improve clinicians’ prescribing behavior? J Am Med Inform Assoc.. 2009;16(4):531–538.
20. Chatellier G, Colombet I, Degoulet P. An overview of the effect of com- puter-assisted management of anticoagulant therapy on the quality of anti- coagulation. Int J Med Inform. 1998;49(3):311–320.
21. Durieux P, Trinquart L, Colombet I, et al. Computerized advice on drug dos- age to improve prescribing practice. Cochrane Database Syst Rev. 2008;3(3):1–62.
22. Shebl NA, Franklin BD, Barber N. Clinical decision support systems and an- tibiotic use. Pharm World Sci PWS. 2007;29(4):342–349.
23. Yourman L, Concato J, Agostini JV. Use of computer decision support in- terventions to improve medication prescribing in older adults: a systematic review. Am J Geriatr Pharmacother. 2008;6(2):119–129.
24. Walton R, Dovey S, Harvey E, Freemantle N. Computer support for deter- mining drug dose: systematic review and meta-analysis. BMJ. 1999;338: 984–990.
25. Sintchenko V, Magrabi F, Tipper S. Are we measuring the right end-points? Variables that affect the impact of computerised decision support on pa- tient outcomes: A systematic review. Med Inform Internet Med. 2007;32(3):225–240.
26. Wong K, Yu SKH, Holbrook A. A systematic review of medication safety outcomes related to drug interaction software. Can J Clin Pharmacol. 2010;17(2):e243–e255.
27. Jamal A, McKenzie K, Clark M. The impact of health information technol- ogy on the quality of medical and health care: a systematic review. Heal Inf Manag J. 2009;38(3):26–37.
28. Ludwick DA, Doucette J. Adopting electronic medical records in primary care: Lessons learned from health information systems imple- mentation experience in seven countries. Int J Med Inform. 2009;78(1): 22–31.
29. Kaushal R, Shojania KG, Bates DW. Effects of computerized physician or- der entry and clinical decision support systems on medication safety: a systematic review. Arch Intern Med. 2003;163(12):1409–1416.
30. Van Rosse F, Maat B, Rademaker CMA, van Vught AJ, Egberts ACG, Bollen CW. The effect of computerized physician order entry on medication pre- scription errors and clinical outcome in pediatric and intensive care: a sys- tematic review. Pediatrics. 2009;123(4):1184–1190.
31. Ammenwerth E, Schnell-Inderst P, Machan C, Siebert U. The effect of elec- tronic prescribing on medication errors and adverse drug events: a sys- tematic review. J Am Med Inform Assoc. 2008;15(5):585–600.
32. Oren E, Shaffer ER, Guglielmo BJ. Impact of emerging technologies on medication errors and adverse drug events. Am J Heal Pharm. 2003;60(14):1447–1458.
33. Wolfstadt JI, Gurwitz JH, Field TS, et al. The effect of computerized physician order entry with clinical decision support on the rates of ad- verse drug events: a systematic review. J Gen Intern Med. 2008;23(4):451–458.
34. Jones SS, Rudin RS, Perry T, Shekelle PG. Health information technology: an updated systematic review with a focus on meaningful use. Ann Intern Med. 2014;160:48–54.
35. Delpierre C, Cuzin L, Fillaux J, Alvarez M, Massip P, Lang T. A systematic review of computer-based patient record systems and quality of care: more randomized clinical trials or a broader approach? Int J Qual Heal Care. 2004;16(5):407–416.
36. Alvis JM, Reves J, Govier A V, et al. Computer-assisted continuous infu- sions of fentanyl during cardiac anesthesia: comparison with manual method. Anesthesiology. 1985;63(1):41–49.
37. Barenfanger J, Short MA, Groesch AA. Improved antimicrobial interven- tions have benefits. J Clin Microbiol. 2001;39(8):2823–2828.
38. Buising K, Thursky K, Robertson M, et al. Electronic antibiotic stewardhsip- reduced consumption of broad spectrum antibiotics using a computerized antimicrobial approval system in a hospital system. J Antimicrob Chemother. 2008;62:608–616.
39. Chapuis C, Roustit M, Bal G, et al. Automated drug dispensing system re- duces medication errors in an intensive care setting. Crit Care Med. 2010;38(12):2275–2281.
R EVIEW
Brenner S.K, et al. J Am Med Inform Assoc 2016;23:1016–1036. doi:10.1093/jamia/ocv138, Review
1034
40. Chertow G, Lee J, Burdick E, Seger DL, Song J, Komaroff AL. Guided medi- cation dosing for inpatients with renal insufficiency. Am Heart J. 2001;286(22):2839–2844.
41. Cho A, Lee J, Yoon J, et al. Effect of an electronic alert on risk of contrast- induced acute kidney injury in hospitalized patients undergoing computed tomography. Am J Kidney Dis. 2012;60(1):74–81.
42. Cook P, Rizzo S, Gooch M, Jordan M, Fang X, Hudson S. Sustained reduc- tion in antimicrobial use and decrease in methicillin-resistant Staphylococcus aureus and Clostridium difficile infections following imple- mentation of an electronic medical record at a tertiary-care teaching hospi- tal. J Antimicrob Chemother. 2011;66(1):205–209.
43. Colpaert K, Claus B, Somers A, Vandewoude K, Robays H, Decruyenaere J. Impact of computerized physician order entry on medication prescription errors in the intensive care unit: a controlled cross-sectional trial. Crit Care. 2006;10(1):R21.
44. Del Beccaro MA, Jeffries HE, Eisenberg MA, Harry ED. Computerized provider order entry implementation: no association with increased mortal- ity rates in an intensive care unit. Pediatrics. 2006;118(1):290–295.
45. Dowding D, Turley M, Garrido T. The impact of an electronic health record on nurse sensitive patient outcomes: an interrupted time series analysis. J Am Informatics Assoc. 2012;19(4):615–620.
46. Dykes P, Carroll D, Hurley A, et al. Fall prevention in acute care hospitals: a randomized trial. JAMA. 2010;304(17):1912–1918.
47. Etchells E, Adhikari NKJ, Wu R, et al. Real-time automated paging and de- cision support for critical laboratory abnormalities. BMJ Qual Saf. 2011;20(11):924–930.
48. Evans RS, Pestotnik SL, Classen DC, et al. A computer-assisted manage- ment program for antibiotics and other antiinfective agents. N Engl J Med. 1998;338(4):232–238.
49. Evans RS, Classen DC, Pestotnik SL, Clemmer TP, Weaver LK, Burke JP. A decision support tool for antibiotic therapy. Proc Annu Symp Comput Appl Med Care. 1995:651–655.
50. Evans RS, Pestotnik SL, Classen DC, Bass SB, Burke JP. Prevention of ad- verse drug events through computerized surveillance. Proc Annu Symp Comput Appl Med Care. 1993:437–441.
51. Fitzmaurice D, Hobbs F, Murray E, Bradley C, Holder R. Evaluation of com- puterized decision support for oral anticoagulation management based in primary care. Br J Gen Pr. 1996;46(410):533–535.
52. Frances CD, Alperin P, Adler JS, Grady D. Does a fixed physician reminder system improve the care of patients with coronary artery disease? A randomized controlled trial. West J Med. 2001;175(3): 165–166.
53. Fiumara K, Piovella C, Hurwitz S, et al. Multi-screen electronic alerts to augment venous thromboembolism prophylaxis. Thromb Haemost. 2010;103(2):312–317.
54. Fleming NS, Ogola G, Ballard DJ. Implementing a standardized order set for community-acquired pneumonia: impact on mortality and cost. Jt Comm J Qual Patient Saf. 2009;35(8):414–421.
55. Gandhi TK, Weingart SN, Seger AC, et al. Outpatient prescribing errors and the impact of computerized prescribing. J Gen Intern Med. 2005;20(9):837–841.
56. Glassman PA, Belperio P, Lanto A, et al. The utility of adding retrospective medication profiling to computerized provider order entry in an ambulatory care population. J Am Med Inform Assoc. 2007;(4):424–431.
57. Graumlich JF, Novotny NL, Stephen Nace G, et al. Patient readmissions, emergency visits, and adverse events after software-assisted dis- charge from hospital: cluster randomized trial. J Hosp Med. 2009;4(7):E11–E19.
58. Gurwitz JH, Field TS, Rochon P, et al. Effect of computerized provider order entry with clinical decision support on adverse drug events in the long- term care setting. J Am Geriatr Soc. 2008;56(12):2225–2233.
59. Han YY, Carcillo JA, Venkataraman ST, et al. Unexpected increased mortal- ity after implementation of a commercially sold computerized physician or- der entry system. Pediatrics. 2005;116(6):1506–1512.
60. Holdsworth MT, Fichtl RE, Raisch DW, et al. Impact of computerized pre- scriber order entry on the incidence of adverse drug events in pediatric in- patients. Pediatrics. 2007;120(5):1058–1066.
61. Jani YH, Barber N, Wong ICK. Paediatric dosing errors before and after electronic prescribing. Qual Saf Health Care. 2010;19(4):337–340.
62. Jha AK, Orav EJ, Ridgway AB, Zheng J, Epstein AM. Does the Leapfrog program help identify high-quality hospitals? Jt Comm J Qual Patient Saf. 2008;34(6):318–325.
63. Jimenez Munoz AB, Muino Miguez A, Rodriguez Perez MP, et al. Comparison of medication error rates and clinical effects in three medica- tion prescription-dispensation systems. Int J Health Care Qual Assur. 2011;24(3):238–248.
64. Keene A, Ashton L, Shure D, Napoleone D, Katyal C, Bellin E. Mortality before and after initiation of a computerized physician order entry system in a critically ill pediatric population. Pediatr Crit Care Med. 2007;8(3): 268–271.
65. King WJ, Paice N, Rangrej J, Forestell GJ, Swartz R. The effect of comput- erized physician order entry on medication errors and adverse drug events in pediatric inpatients. Pediatrics. 2003;112(3 Pt 1):506–509.
66. Kucher N, Koo S, Quiroz R, et al. Electronic alerts to prevent venous throm- boembolism among hospitalized patients. N Engl J Med. 2005;352(10):969–977.
67. Kuperman GJ, Teich JM, Tanasijevic MJ, et al. Improving response to criti- cal laboratory results with automation: results of a randomized controlled trial. J Am Med Inform Assoc. 1999;6(6):512–522.
68. Lecumberri R, Marqués M, Dı́az-Navarlaz MT, et al. Maintained effective- ness of an electronic alert system to prevent venous thromboembolism among hospitalized patients. Thromb Haemost. 2008:699–704.
69. Lesourd F, Avril C, Boujennah A, Parinaud J. A computerized decision sup- port system for ovarian stimulation by gonadotropins. Fertil Steril. 2002;77(3):456–460.
70. Linares LA, Thornton DJ, Strymish J, Baker E, Gupta K. Electronic memo- randum decreases unnecessary antimicrobial use for asymptomatic bacte- riuria and culture-negative pyuria. Infect Control Hosp Epidemiol. 2011;32(7):644–648.
71. MacIvor D, Triulzi DJ, Yazer MH. Enhanced detection of blood bank sample collection errors with a centralized patient database. Transfusion. 2009;49(1):40–43.
72. Madaras-Kelly KJ, Remington RE, Lewis PG, Stevens DL. Evaluation of an intervention designed to decrease the rate of nosocomial methicillin-resis- tant Staphylococcus aureus infection by encouraging decreased fluoro- quinolone use. Infect Control Hosp Epidemiol. 2006;27(2):155–169.
73. Maynard GA, Morris TA, Jenkins IH, et al. Optimizing prevention of hospi- tal-acquired venous thromboembolism (VTE): prospective validation of a VTE risk assessment model. J Hosp Med. 2010;5(1):10–18.
74. McCowan C, Neville R, Ricketts I, Warner F, Hoskins G, Thomas G. Lessons from a randomized controlled trial designed to evaluate computer decision support software to improve the management of asthma. Med Informatics Internet Med. 2001;26(26):191–201.
75. McMullin J, Cook D, Griffith L, et al. Minimizing errors of omission: Behavioural rEenforcement of Heparin to Avert Venous Emboli: The BEHAVE Study. Crit Care Med. 2006;34(3):694–699.
76. McMullin ST, Reichley RM, Watson LA, Steib SA, Frisse ME, Bailey TC. Impact of a Web-based clinical information system on cisapride drug inter- actions and patient safety. Arch Intern Med. 1999;159(17):2077–2082.
77. Menachemi N, Saunders C, Chukmaitov A, Matthews MC, Brooks RG. Hospital adoption of information technologies and improved patient safety: a study of 98 hospitals in Florida. J Healthc Manag. 2007;52(6):398–409; discussion 410.
78. Milani R V, Oleck SA, Lavie CJ. Medication errors in patients with severe chronic kidney disease and acute coronary syndrome: the impact of computer- assisted decision support. Mayo Clin Proc. 2011;86(12):1161–1164.
79. Morriss FJ, Abramowitz P, Nelson S, Milavetz G, Michael S, Gordon S. Risk of adverse drug events in neonates treated with opiods and the effect of a bar-coded-assisted medication administration system. Am J Heal Pharm. 2011;68:57–62.
80. Morriss FH, Abramowitz PW, Nelson SP, et al. Effectiveness of a barcode medication administration system in reducing preventable adverse drug events in a neonatal intensive care unit: a prospective cohort study. J Pediatr. 2009;154(3):363–368, 368.e1.
R EVIEW
Brenner S.K, et al. J Am Med Inform Assoc 2016;23:1016–1036. doi:10.1093/jamia/ocv138, Review
1035
81. Novis SJ, Havelka GE, Ostrowski D, et al. Prevention of thromboembolic events in surgical patients through the creation and implementation of a computerized risk assessment program. J Vasc Surg. 2010;51(3):648–654.
82. Oliven A, Zalman D, Shilankov Y, Yeshurun D, Odeh M. Prevention of prescription errors by computerized, on-line, individual patient related surveillance of drug order entry. Stud Heal Technol Informatics. 2002;90: 632–634.
83. Overhage JM, Dexter PR, Perkins SM, et al. A randomized, controlled trial of clinical information shared from another institution. Ann Emerg Med. 2002;39(1):14–23.
84. Parente ST, McCullough JS. Health information technology and patient safety: evidence from panel data. Health Aff. 2009;28(2):357–360.
85. Paul M, Andreassen S, Tacconelli E, et al. Improving empirical antibiotic treatment using TREAT, a computerized decision support system: Cluster randomized trial. J Antimicrob Chemother. 2006;58(6):1238–1245.
86. Peterson JF, Kuperman GJ, Shek C, Patel M, Avorn J, Bates DW. Guided prescription of psychotropic medications for geriatric inpatients. Arch Intern Med. 2005;165(7):802–807.
87. Piontek F, Kohli R, Conlon P, Ellis JJ, Jablonski J, Kini N. Effects of an ad- verse-drug-event alert system on cost and quality outcomes in community hospitals. Am J Health Syst Pharm. 2010;67(8):613–620.
88. Poon EG, Keohane CA, Yoon CS, et al. Effect of bar-code technology on the safety of medication administration. N Engl J Med. 2010;362(18):1698– 1707.
89. Potts AL, Barr FE, Gregory DF, Wright L, Patel NR. Computerized physician order entry and medication errors in a pediatric critical care unit. Pediatrics. 2004;113(1 Pt 1):59–63.
90. Rollman B, Hanusa B, Lowe HJ, Gilbert T, Kapoor WN, Schulberg HC. A ran- domized trial using computerized decision support to improve treatment of major depression in primary care. J Gen Intern Med. 2002;17:493–503.
91. Earnest MA, Ross SE, Wittevrongel L, Moore LA, Lin CT. Use of a patient-accessible electronic medical record in a practice for congestive heart failure: patient and physician experiences. J Am Med Inform Assoc. 2004;11(5):410–417.
92. Rind D, Safran C, Phillips R, et al. Effect of computer-based alerts on the treatment and outcomes of hospitalized patients. Arch Intern Med. 1994;154:1511–1517.
93. Rothschild JM, Keohane CA, Cook EF, et al. A controlled trial of smart infu- sion pumps to improve medication safety in critically ill patients. Crit Care Med. 2005;33(3):533–540.
94. Schnipper JL, Ndumele CD, Liang CL, Pendergrass ML. Effects of a subcu- taneous insulin protocol, clinical education, and computerized order set on the quality of inpatient management of hyperglycemia: results of a clinical trial. J Hosp Med. 2009;4(1):16–27.
95. Shulman R, Singer M, Goldstone J, Bellingan G. Medication errors: a pro- spective cohort study of hand-written and computerised physician order entry in the intensive care unit. Crit Care. 2005;9(5):R516–R521.
96. Small MDC, Barrett A, Price GM. The impact of computerized prescribing on error rate in a department of Oncology/Hematology. J Oncol Pharm Pract. 2008;14(4):181–187.
97. Smith JR, Noble MJ, Musgrave SD, et al. The at-risk registers in severe asthma (ARRISA) study: A cluster-randomised controlled trial in primary care. Thorax. 2010;65:A62.
98. Tierney WM, Overhage JM, Murray MD, et al. Can computer-generated ev- idence-based care suggestions enhance evidence-based management of asthma and chronic obstructive pulmonary disease? A randomized, con- trolled trial. Health Serv Res. 2005;40(2):477–497.
99. Upperman JS, Staley P, Friend K, et al. The impact of hospitalwide com- puterized physician order entry on medical errors in a pediatric hospital. J Pediatr Surg. 2005;40(1):57–59.
100. Van Doormaal JE, van den Bemt PMLA, Mol PGM, et al. Medication errors: the impact of prescribing and transcribing errors on preventable harm in hospitalised patients. Qual Saf Health Care. 2009;18(1):22–27.
101. Walsh KE, Landrigan CP, Adams WG, et al. Effect of computer order entry on prevention of serious medication errors in hospitalized children. Pediatrics. 2008;121(3):e421–e427.
102. Weingart SN, Simchowitz B, Padolsky H, et al. An empirical model to esti- mate the potential impact of medication safety alerts on patient safety, health care utilization, and cost in ambulatory care. Arch Intern Med. 2009;169(16):1465–1473.
103. Yu F, Salas M, Kim Y-II, Menachemi N. The relationship between comput- erized physician order entry and pediatric adverse drug events: a nested matched case-control study. Pharmacoepidemiol Drug Saf. 2009;18(8): 751–755.
104. Zanetti G, Flanagan HL Jr, Cohn LH, Giardina R, Platt R. Improvement of intraoperative antibiotic prophylaxis in prolonged cardiac surgery by auto- mated alerts in the operating room. Infect Control Hosp Epidemiol. 2003;24(1):13–16.
105. Tanon A, Champagne F, Contandriopoulos A, Pomey M, Vadeboncoeur A, Nguyen H. Patient safety and systematic reviews: finding papers indexed in MEDLINE, EMBASE and CINAHL. Qual Saf Health Care. 2010;19(5): 452–461.
106. Liberati A, Altman D, Tetzlaff J, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ. 2009;339:b2700–b2700.
107. Viera AJ, Garrett JM. Understanding interobserver agreement: the kappa statistic. Fam Med. 2005;37(5):360–363.
108. Evans R, Pestotnik S, Classen D, Horn S, Bass S, Burke J. Preventing ad- verse drug events in hospitalized patients. Ann Pharmacother. 1994;28(4): 523–527.
109. Shojania KG, Duncan BW, McDonald KM, Wachter RM, Markowitz AJ. Making health care safer: a critical analysis of patient safety practices. Evid Rep Technol Assess. 2001;(43):i–x, 1-668.
110. Eslami S, Abu-Hanna A, de Keizer NF. Evaluation of outpatient computer- ized physician medication order entry systems: a systematic review. J Am Med Inform Assoc. 2007;14(4):400–406.
111. Classen DC, Bates DW. Finding the meaning in meaningful use. N Engl J Med. 2011;365(9):855–858.
112. Abramson EL, Malhotra S, Fischer K, et al. Transitioning between elec- tronic health records: Effects on ambulatory prescribing safety. J Gen Intern Med. 2011;26(8):868–874.
113. Karsh B-T, Weinger MB, Abbott PA, Wears RL. Health information technol- ogy: fallacies and sober realities. J Am Med Inform Assoc. 2010;17(6): 617–623.
114. Technology TO of the NC for HI. HealthIT.gov SAFER Guides. Available at: http://www.healthit.gov/safer/safer-guides. Accessed October 10, 2014.
AUTHOR AFFILIATIONS .................................................................................................................................................... 1Department of Healthcare Policy and Research, Weill Cornell Medical College, New York, NY, USA 2Center for Healthcare Informatics and Policy, New York, NY, USA 3Department of Medicine, Stanford School of Medicine, Palo Alto, CA, USA 4Department of Medicine, Weill Cornell Medical College, New York, NY, USA 5Department of Pediatrics, Weill Cornell Medical College, New York, NY, USA 6New York-Presbyterian Hospital, New York, NY, USA
7Department of Emergency Medicine, Weill Cornell Medical College, New York, NY, USA 8Samuel J. Wood Library & C.V. Starr Biomedical Information Center, Weill Cornell Medical College, New York, NY, USA 9Uniformed Services University of the Health Sciences, Bethesda, MD, USA
R EVIEW
Brenner S.K, et al. J Am Med Inform Assoc 2016;23:1016–1036. doi:10.1093/jamia/ocv138, Review
1036
Copyright © 2016 American Medical Informatics Association.
