New Technology Interview

R

D h

K a

b

a

A

R

R

2

A

K

M

C

M

N

T

1 d

i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 7 ( 2 0 0 8 ) 291–304

j o u r n a l h o m e p a g e : w w w . i n t l . e l s e v i e r h e a l t h . c o m / j o u r n a l s / i j m i

eview

efinition, structure, content, use and impacts of electronic ealth records: A review of the research literature

ristiina Häyrinen a,∗, Kaija Saranto a, Pirkko Nykänen b

University of Kuopio, Department of Health Policy and Management, Finland University of Tampere, Department of Computer Sciences, Finland

r t i c l e i n f o

rticle history:

eceived 12 April 2006

eceived in revised form

2 June 2007

ccepted 13 September 2007

eywords:

edical records systems

omputerized

edical informatics

ursing informatics

a b s t r a c t

Purpose: This paper reviews the research literature on electronic health record (EHR) systems.

The aim is to find out (1) how electronic health records are defined, (2) how the structure of

these records is described, (3) in what contexts EHRs are used, (4) who has access to EHRs,

(5) which data components of the EHRs are used and studied, (6) what is the purpose of

research in this field, (7) what methods of data collection have been used in the studies

reviewed and (8) what are the results of these studies.

Methods: A systematic review was carried out of the research dealing with the content of

EHRs. A literature search was conducted on four electronic databases: Pubmed/Medline,

Cinalh, Eval and Cochrane.

Results: The concept of EHR comprised a wide range of information systems, from files com-

piled in single departments to longitudinal collections of patient data. Only very few papers

offered descriptions of the structure of EHRs or the terminologies used. EHRs were used

in primary, secondary and tertiary care. Data were recorded in EHRs by different groups of

health care professionals. Secretarial staff also recorded data from dictation or nurses’ or

physicians’ manual notes. Some information was also recorded by patients themselves; this

information is validated by physicians. It is important that the needs and requirements of

different users are taken into account in the future development of information systems.

Several data components were documented in EHRs: daily charting, medication admin-

istration, physical assessment, admission nursing note, nursing care plan, referral, present

complaint (e.g. symptoms), past medical history, life style, physical examination, diagnoses,

tests, procedures, treatment, medication, discharge, history, diaries, problems, findings and

immunization. In the future it will be necessary to incorporate different kinds of stan-

dardized instruments, electronic interviews and nursing documentation systems in EHR

systems.

The aspects of information quality most often explored in the studies reviewed were

the completeness and accuracy of different data components. It has been shown in sev-

eral studies that the use of an information system was conducive to more complete and

∗ Corresponding author at: University of Kuopio, Department of Health Policy and Management, P.O. Box 1627, FIN-70211 Kuopio, Finland. el.: +358 17162604.

E-mail address: [email protected] (K. Häyrinen). 386-5056/$ – see front matter © 2007 Elsevier Ireland Ltd. All rights reserved. oi:10.1016/j.ijmedinf.2007.09.001

292 i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 7 ( 2 0 0 8 ) 291–304

accurate documentation by health care professionals. The quality of information is particu-

larly important in patient care, but EHRs also provide important information for secondary

purposes, such as health policy planning. Studies focusing on the content of EHRs are

needed, especially studies of nursing documentation or patient self-documentation. One

future research area is to compare the documentation of different health care profession-

als with the core information about EHRs which has been determined in national health

projects. The challenge for ongoing national health record projects around the world is to

take into account all the different types of EHRs and the needs and requirements of different

health care professionals and consumers in the development of EHRs. A further challenge

is the use of international terminologies in order to achieve semantic interoperability.

© 2007 Elsevier Ireland Ltd. All rights reserved.

Contents

1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292 2. Materials and methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293 3. Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293

3.1. How is the EHR defined? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293 3.2. How is the structure of EHRs described? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294 3.3. Where is the EHR used? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 296 3.4. Users of the EHR system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 296 3.5. Studied and used components of EHR system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 296 3.6. Purpose, data collection methods and results of these studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297

3.6.1. Impact of EHR on information quality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297 3.6.2. Impact of EHR on other aspects of information system success factors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299

4. Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 300 5. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 301

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 301

1. Introduction

Research and development projects are ongoing in several countries around the world to develop an infrastructure for national health information; examples include Canada [1], Australia [2], England [3], the United States [4] and Finland [5]. These projects share in common a number of elements, including (1) the aim of involving patients in the use of their own health records; (2) the need to define the core infor- mation of these records; (3) the choice and implementation of standards, nomenclatures, codes and vocabularies; (4) the need to develop the necessary data security infrastructure and policies; (5) the aim of producing open, standardized and interoperable EHR systems for data exchange and information management. Besides national projects, the European Union launched the European eHealth Action Plan in 2004. One chal- lenge is to standardize health information systems, which also means standardization of the content and structure of EHRs [6]. In particular, a patient summary has been seen as the most appropriate way to establish eHealth interoper- ability. A patient summary includes patient history, allergies, active problems, test results, and medications. However, fur-

for current research in the field of health informatics [8,9] but the need for research from different approaches has also been noticed [10] The focus of recent studies concerning EHR has been on the possibilities of current technologies and underly- ing architecture (cf. [11–13]) and on exploring the health care registers as a source for evidence-based medicine [14].

According to the literature, the meaning of EHR is unsta- ble. EHR has many functions and includes many kinds of data, and it is obvious that there is a need to determine explicitly what EHR means. Once that has been done, common ways to develop EHRs will be found, along with common view- points on what kind of research focusing on the content of EHR can be done in the future. The aim of this study is to determine what an electronic health record is and how far its content is standardized. An EHR is used primarily for purposes of setting objectives and planning patient care, documenting the delivery of care and assessing the outcomes of care. It includes information regarding patient needs during episodes of care provided by different health care professionals [15,16]. The amount and quality of information available to health care professionals in patient care has an impact both on the outcomes of patient care and the continuity of care. The infor- mation included in EHRs has several different functions in the

ther information can be included, depending on the intended purpose of the summary and the anticipated context of use. Additionally, investigation into the amount of structured data of the patient summary is needed [7]. EHRs are a major focus

decision-making process in patient care, and it also supports decision-making in management and in health policy. EHRs have so far consisted of unstructured, narrative text but also structured coded data. In the future it will be necessary to

a l i n

i t i p h E o n w t q

w a e i ( e i a

2

A d w B G ( R a l s w u r d c w a r t n

l T i c t P

1 t s s i i u c

i n t e r n a t i o n a l j o u r n a l o f m e d i c

mplement more systematic terminologies and codes so that he data contained in these records can be put to better use n clinical research, health care management, health services lanning, and government reporting [8,9,15,16]. Thiru et al. ave reviewed the literature assessing the quality of data in HRs in primary care. They report that the main focus has been n structured data elements, i.e. codes, classifications and omenclatures. Most of the studies included in their review ere descriptive surveys. Thiru et al. also draw attention to

he lack of standardized methods for the assessment of data uality [17].

The present review focuses on research that is concerned ith the structure and content of EHR systems. It aims to

nswer the following questions: (1) how is the EHR defined in arlier research, (2) how is the structure of EHRs described, (3) n which contexts is the EHR used, (4) who has access to EHRs, 5) what data components of the record system are used by nd-users and studied, (6) what is the purpose of these stud- es, (7) what methods of data collection are used in the studies nd (8) what are the results of these studies.

. Materials and methods

n automated literature search was conducted on four atabases with the assistance of a librarian. The databases ere PubMed/Medline (National Library of Medicine, ethesda, MD, USA), Cinalh (Cinahl Information Systems, lendale, CA, USA), Inventory of Evaluation Publications

University for Health Informatics and Technology, Tirol esearch Group Assessment of Health Information Systems) nd Cochrane (The Cochrane Collaboration). On the Cumu-

ative Index of Nursing and Allied Health Literature (Cinahl), the earch was performed using thesaurus terms and free text ords, combining them in an appropriate way. The terms sed were: content analysis, content validity, evaluation esearch, computerized patient record, documentation, vali- ation, utilization, classification, nomenclature, vocabulary, ontrolled and nursing classification. In addition, free text ords were ANDed with the appropriate thesaurus terms nd ORed with other search statements. The search was then estricted to journal articles. As it was expected that much of he research literature within the scope of the review would ot be indexed, no time limits were applied.

On PubMed/Medline, the search was carried out in a simi- ar way by using both the MeSH terms and free text words. he terms used were medical records systems, computer-

zed, content, assess and evaluate, classification, vocabulary, ontrolled, coding and nursing classification. On Cochrane, he search was carried out using the same terms as on ubmed/Medline.

On the Inventory of Health Information Evaluation Studies 982–2002 database (evaldb), the search was based on the cri- eria that are used to classify studies [18]. In this study the earch was performed using two criteria of the database clas- ification: the focus of the evaluation study and the type of

nformation system. The focus of evaluation study criterion s classified further; one criterion is the quality of the doc- mented and processed information, i.e. completeness and orrectness of documentation. The other database criterion

f o r m a t i c s 7 7 ( 2 0 0 8 ) 291–304 293

is the type of information system. Information systems are classified into several types, of which 12 were chosen for the present study: CIS (general or unspecified clinical information or documentation system) OR ANAEST (anaesthesia informa- tion and documentation system) OR CPOE (physician order entry system) OR GP (GP information system) OR LAB (labo- ratory management system) OR NURSE (nursing information and documentation system) OR OP (operation unit planning and management system) OR PACS (picture archiving and communication system) OR PDMS (patient data management system) OR PHARM (pharmacy information system) OR PIS (patient information systems) OR RIS (radiological information system).

The search yielded 299 papers. These papers were reviewed to exclude articles that did not meet the selection criteria: (1) focus on electronic rather than paper-based health record, (2) data content of EHR assessed or analysed, (3) paper written in English, and (4) articles electronically retrievable as full texts or available locally. Following this initial review, 180 papers were retrieved for more detailed evaluation. Forty eight papers were not available electronically or could not be obtained locally. Three studies had been published both in journals and in con- ference proceedings, and the latter were excluded. A total of 37 papers were excluded on the basis of the criteria specified for this review. The final number of papers included in the review was thus 89 (Fig. 1). The review paper by Thiru [17] is included in this review, but it is only considered under the items of time period, publishers and countries of research.

3. Results

The papers included in the present review were published between 1982 and 2004 in 52 different journals; three of them were published in conference proceedings. The top four jour- nals with the largest number of articles were the Journal of the American Medical Informatics Association (n = 11), Methods of Information in Medicine (n = 6), Computers in Nursing (n = 6) and the International Journal of Medical Informatics (n = 4).

Most of the studies had been done in the United States (n = 43). A total of 37 papers were from European countries (United Kingdom, Germany, Sweden, Netherlands, Norway, Hungary, Italy and Finland); the remainder were from Hong Kong (2), Australia (1), Taiwan (1), and Canada (5) (Table 1).

The discussion below deals in order with each of the research questions. The themes in the articles were exam- ined by means of content analysis. Each section begins with a description of the criteria informing the analysis. This is followed by a presentation of the results, which are finally summarized in tables.

3.1. How is the EHR defined?

EHRs were classified on the basis of the International Organi- zation for Standardization (ISO) definition [19]. According to this definition, the EHR means a repository of patient data in

digital form, stored and exchanged securely, and accessible by multiple authorized users. It contains retrospective, concur- rent, and prospective information and its primary purpose is to support continuing, efficient and quality integrated health

294 i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 7 ( 2 0 0 8 ) 291–304

iagr

Fig. 1 – Flow d

care. ISO also gives a number of other terms commonly used to describe different types of EHRs (Table 2).

The different types of EHR introduced in the articles reviewed are shown in Table 2. Electronic patient records were used both in hospitals [59–63] and in general practice [64–71]. Patients could use electronic interviews concerning their medical history [79–84] or enter information concern- ing their diabetes [85,86]. There were also computerized diaries that patients could use to control their medica- tion [87], urinary voiding [88] or food intake [89] and to assess pain intensity [90]. The concept of computerized medical record was used in seven studies [91–97], but its meaning was the same as for computerized patient record. Furthermore, a separate or integrated computer-based nursing information system had been developed to sup-

port nursing documentation [29,53,55,56,64,98–104]. Standard computerized instruments have also been used by health professionals among other things to assess activities of daily living (ADL) [105] or pain [106]. One study provided

Table 1 – The time period covered, publishers and countries of

Time period n = 89 Publisher

1982–1989 7 Various medical and medical informatics jo

1990–1999 34 Computers in Nursing (n = 4); Journal of the Ame Informatics Association (n = 2); Methods of Infor (n = 1); International Journal of Medical Informat medical and medical informatics journals (n

2000–2004 48 Computers in Nursing (n = 2); Journal of the Ame Informatics Association (n = 9); Methods of Infor (n = 5); International Journal of Medical Informat medical, nursing, medical informatics or nu journals (n = 30)

am of review.

no information on the type of information system assessed [107].

3.2. How is the structure of EHRs described?

The structure and content of EHRs has varied over time. Using earlier classifications of the structure of EHRs [108,109], we made a distinction between time-oriented, problem- oriented and source-oriented EHRs. Nowadays EHRs combine all three elements. In the time-oriented electronic medi- cal record, the data are presented in chronological order. In the problem-oriented medical record (POMR), notes are taken for each problem assigned to the patient, and each problem is described according to the subjective informa- tion, objective information, assessments and plan (SOAP).

In the source-oriented record, the content of the record is arranged according to the method by which the information was obtained, e.g. notes of visits, X-ray reports and blood tests. Within each section, the data are reported in chrono-

origin of research papers included in this review

Country of origin

urnals (n = 7) USA (n = 3); Europe (n = 3); others (n = 1)

rican Medical mation in Medicine ics (n = 2); various = 25)

USA (n = 22); Europe (n = 9); others (n = 3)

rican Medical mation in Medicine ics (n = 2); various, e.g. rsing informatics

USA (n = 18); Europe (n = 25); others (n = 5)

i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 7 ( 2 0 0 8 ) 291–304 295

Table 2 – Types of EHR

Type of EHR (ISO) Definition Reference number

Electronic medical record (EMR) Generally focused on medical care Departmental EMR (n = 29) Contains information entered by a single hospital department

Picture archiving and communication system (PACS) [20–22] Anaesthesia records [23–26] Intensive care records [27–30] Ambulatory records [31] Emergency department systems [32–36] Pathology laboratory system [37] Oncology records [38] Cardiology records [39] Operation theatre records [40] Gynaecology records [41] Internal medicine records [42] Pharmacy systems [43,44] Geriatric centre records [45] Diabetes clinic records [46] Radiology reporting system [47,48]

Inter-departmental EMR (n = 2) Contains information from two or more hospital departments Obstetric records for inpatient and outpatient clinics [49] Prescribing system [50]

Hospital EMR (n = 8) Contains all or most of patient’s clinical information from a particular hospital

[51–58]

Inter-hospital EMR Contains patient’s medical information from two or more hospitals

–

Electronic patient record (EPR) (n = 13) Contains all or most of patient’s clinical information from a particular hospital

[59–71]

Computerized patient record (CPR) (n = 13) Contains all or most of patient’s clinical information from a particular hospital

[72–77,91–97]

Electronic health care record (EHCR) (n = 1) Contains all patient health information [78]

Personal health record (n = 8) Controlled by the patient and contains information at least partly entered by the patient

[79–86]

Computerized medical record Created by image scanning of a paper-based health record –

Digital medical record A web-based record maintained by a health care provider –

Clinical data repository An operational data store that holds and manages clinical data collected from health service providers

–

Electronic client record Scope is defined by health care professionals other than physicians, e.g. by physiotherapists or social workers

–

e defi

gated

l h a m d [ c

t p m p i a i a

Virtual EHR No authoritativ

Population health record Contains aggre

ogical order [108]. The American Nurses Association (ANA) as developed a framework for nursing documentation which lso corresponds with the SOAP structure for medical docu- entation. The nursing process had four stages: assessment,

iagnosis, planned or delivered interventions and outcomes 109]. In addition to the structure of narrative text in EHRs, lassifications are needed [108,109].

The structure of the EHR is described in only 15 of he papers reviewed. The SOAP structure appears in five apers [65,66,71,74,95], while computerized nursing docu- entation is structured around the nursing process in nine

apers [29,55,56,64,100–104]. The steps included in the nurs-

ng process varied. Nursing documentation included at least ssessment, the identification of nursing problems and nurs- ng care aims, planning and delivering nursing interventions, nd the evaluation of outcomes [29,56,64,94,100–103]. Further-

nition –

and usually de-identified data –

more in one paper the structure of EHR is episode of care oriented [67].

EHRs include both unstructured free text and coded data. Twenty-eight papers also described the terminologies used in these records, i.e. their classifications, vocabularies, nomen- clatures or codes (Table 3).

Various other national classifications were also used in medical information documentation, including the Operatio- nenschlüssel nach §301 SGB-V (OPS-301) [63,76] coding for procedures, the Swedish coding system [71], the problem list vocabulary [91], the controlled terminology medical entities dictionary [31] for problems, medications and adverse reac-

tions and the drug dictionary for coding medication [50]. Different classifications were also used for purposes of nursing documentation (see Table 3). Outcomes were also described by means of unstructured statements, such as expressions

296 i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 7 ( 2 0 0 8 ) 291–304

Table 3 – The international terminologies used in EHRs

Data component International terminology Reference

Diagnoses International Classification of Diseases (ICD) [27,46,48,49,54,57,59,63,65–67,72,76,78,94] Read codes [68,92,93] International Classification of Primary Care (ICPC) [67]

Procedures Current Procedural Terminology (CPT) [27,48,49]

Medication Anatomical Therapeutic Chemical Classification Index (ATC) [54,78]

Pathological findings Systematized Nomenclature of Medicine (Snomed) [37]

Nursing problems North American Nursing Diagnoses (NANDA) [100,101,103,104] International Classification of Nursing Practice (ICNP) [101]

ion (N

(NOC

Nursing interventions Iowa Nursing Intervention Classificat ICNP

Nursing outcomes Iowa Nursing Outcome Classification

of pain [103]. In Sweden the content of nursing documenta- tion had a common structure based on the key words of the Swedish model for the documentation of nursing care, VIPS. The key concepts for nursing were Well-being, Integrity, Pre- vention and Safety. This use of key words from the VIPS model as headings for both assessment and interventions is one way to standardize documentation [64,101].

Standardized instruments for purposes of structuring patient information include the mini-nutritional assessment (MNA) and a modified version of the Norton scale [101], an assessment instrument about the patients’ medical condi- tion, activities of daily living (ADL), skills, behaviour, nursing care needs and rehabilitation potential, RUG II, assessment of patient functioning category (letter code) and Daily living score [105].

3.3. Where is the EHR used?

Health services are organised in different ways in different countries, but most typically they are divided between pri- mary, secondary and tertiary care. Primary care is health care provided in the community by the staff of a general practice. Secondary care is medical attention provided by a specialist

facility upon referral by a primary care physician, and tertiary care is provided by a team of specialists in a major hospital [110]. The context of the studies is represented in Table 4. A few of the studies were concerned with self-monitoring by

Table 4 – The context of the studies reviewed (n = 89)a

Tertiary care (n = 35) Inpatient [21,23,27,28,30,32,35–37,44,47,50,53, 57,58,60,61,62,72,76,80–82,84,95,98,99, 101,103,104]; outpatient [24,38,42,45,52]

Secondary care (n = 34) Inpatient [20,22,25,26,29,31,33,34,37,40,43,48,49,51, 54,55,56,59,63,74,75,79,91,102,106,107]; outpatient [39,41,46,73,77,78,83,87]

Primary care (n = 14) [64–71,92–94,96,97,100]; home health care [105]

Home care (n = 1) [85,86,88–90]

a In one study the context was both tertiary and secondary care.

IC) [100,103,104] [101]

) [104]

patients in their homes (n = 5). Nine of the studies were con- ducted in more than one organisation, for example in two hospitals in one context (Table 4).

3.4. Users of the EHR system

The EHR is used by different health care professionals and also by administrative staff. Among the various health care professionals who use different components of the EHR are physicians, nurses, radiologists, pharmacists, laboratory tech- nicians and radiographers. Furthermore, EHRs are also used by patients or their parents (Table 5).

3.5. Studied and used components of EHR system

The medical data components recorded in the EHRs are here categorized on the basis of the classifications used in the papers reviewed [32,36,71,74,75,93,95]. The following data components are identified: referral, present complaint (e.g. symptoms), past medical history, life style, physical examination, diagnoses, tests, e.g. laboratory and radiology, procedures, treatment, medication and discharge.

The classification of nursing data components is based on the components of nursing charting areas identified by Marr et al. [53] and on the nursing care plan. The components are medication administration, daily charting, physical assess- ments and admission nursing notes. Daily charting includes patients’ daily functional activities such as vital signs, food, elimination, mobility and patient teaching. Physical assess- ment comprises all kinds of status assessments (e.g. skin status or respiratory status). Admission nursing note contains information on allergies, health behaviour (e.g. physical activ- ity or smoking or sleep patterns), physical assessment (e.g. temperature and neurological status), discharge planning and initial care plan.

According to this review the area of the EHR that is studied most often is medical data (n = 37). Various medical data components have been analysed. Some studies have focused on just one data component such as tests; others

have looked at almost all data components of the EHR. Sev- eral papers (n = 22) said that the documentation systems were used by different health care professionals and that secre- tarial staff typed the dictation of nurses or physicians and

i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 7 ( 2 0 0 8 ) 291–304 297

Table 5 – Users of EHR systems and data components studied

User (number of papers) Component of EHR

Nurse (n = 16) Daily charting [29,53,56,64,98,101]; medication administration [53,98]; physical assessment [53,100,105]; admission nursing note [41,53,101,107]; nursing care plan [29,53,55,56,64,98–104]

Physician (n = 37) Referral [68,69,71,93]; present complaint, e.g. symptoms [30,31,65–67,70,71,73–75,77,91,93]; past medical history [32,36,62,75]; life style [68,75,93]; physical examination [23–27,36,62,68,71,75,93,106]; diagnoses [36,58,63,66,67,75,76,92–94]; tests [21,26,32,36,37,42,48,60,67,75,93]; procedures [58,63,67,76,93]; treatment [27,32,36,61,75,93]; medication [31,68–71,77,93]; discharge [32,36,54,59–61]

Patient (n = 9) History [79,83,84]; diaries [85–90]; test [85]

Parents (n = 3) History [80–82]

Secretarial staff (n = 3) Procedures [40]; problems [96]; diagnoses [96]; findings [96]; immunization [97]

Pharmacists (n = 2) Medication [43,44]

Multiprofessional (n = 22): nurse [28,31,33,34,49–51,68,72,78,95]; physician [20,22,31,33–35,38,45,49,50,52,68,72,78,95]; laboratory staff [28,72]; radiology staff

Referral [46]; present complaint, e.g. symptoms [33,46,72,78,95]; past medical history [33,34,38,46,49,52,72,78,95]; life style [46,97]; physical examination [33,38,46,49,52,95]; diagnoses [31,34,46,51,68]; tests [20,22,28,33,38,39,46,47,52,72,95]; procedures [35,49]; treatment

34,49, inist

y cha

s h o d p

3 t

T f a t t b o t q i r t i t a p c d i b a i s

e

[20,22,47,72]; clerk or administrative staff [22,33,35,38,47,49,51,52]; pharmacy personnel [78]; health care professionals [39,46,57]

[31, adm dail

tored it in the information system. Nursing documentation as been studied in 16 papers, and 12 of these have focused n the documentation of nursing care plans. Patient self- ocumentation has been investigated in only a minority of the apers.

.6. Purpose, data collection methods and results of hese studies

o explore the purpose of the studies reviewed, we used the ramework of DeLone and McLean [111]. van der Meijden has lso used the same classification to study the success fac- ors of information system implementation [112]. According o DeLone and McLean [111] information system success can e considered on six different dimensions, where the output f information systems is measured at the technical, seman- ic and effectiveness level. These dimensions are information uality, system quality, information use, user satisfaction,

ndividual impact and organizational impact. System quality efers to the technical level, information quality to the seman- ic level and information use, user satisfaction, individual mpact and organizational Impact to the effectiveness level. In heir more advanced model [113] DeLone and MacLean added third major dimension, service quality, for e-commerce pur- oses. Service quality refers to the service provided to the ustomer. Furthermore, in the advanced model, the success imension information use has an alternative measure in

ntention to use, and individual and organizational impact has een combined in the single variable of net benefits. DeLone nd McLean have also proposed that these dimensions are

nterrelated, which is why it is important to measure the pos- ible interactions between the different success dimensions.

Each major dimension can be measured by various differ- nt success criteria. System quality assesses the information

57,95]; medication [31,34,43,45,46,50,68,72,95]; discharge [51,52]; ration of medication [78]; admission nursing note [34,38,51,72,95]; rting [28,33,34,46,72]

processing system itself, and its attributes (in the original model, 18) include ease of use, ease of learning or usefulness of system. Information quality measures both the output and input of the information system; attributes (23) here include completeness, accuracy, legibility, reliability and format. Infor- mation use measures end-users’ consumption of the output of an information system, with attributes (12) including amount of use and number of queries. User satisfaction measures the end-users’ response to the use of the output of an informa- tion system, and attributes (8) include overall satisfaction and decision-making satisfaction. Individual impact measures the effect of information on the behaviour of the end-user, and attributes (15) include improved individual productivity and information understanding. Organizational impact measures the effect of information on organizational performance, and its attributes (18) include return on investment and increased work volume [111].

The discussion below presents the results of our content analysis, classifying the purposes and results of the studies according to the original framework of DeLone and McLean. The data collection methods used in these studies were also analysed means by content analysis.

3.6.1. Impact of EHR on information quality All of the studies included in the review analysed one or more of the information quality criteria mentioned above (Tables 6 and 7). In this analysis the most frequently used criteria were completeness and accuracy. The completeness of documentation was addressed in 55 papers. In this analysis completeness serves as a measure of the prevalence of missing

information. Several studies indicated that the use of an infor- mation system was conducive to more complete documen- tation by health care professionals [24,27,29,31–34,36,38,39, 41,42,45–48,56–59,63,67,68,73,74,77,78,93,96,99,100], although

298 i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 7 ( 2 0 0 8 ) 291–304

Table 6 – Research focusing on information quality and data collection methods used

Research focus Quality of documented information: completeness (n = 55) [24,27,29–34,36,38,39,41,42,44–48,51,53,56–59,63,64,66,67,68,71,73,74,77,78,80–86,88,89,91–93,95,96,98–104]; accuracy (n = 29) [20,23,25,26,28,35,37,40,43,45,49–51,54,56,59,66,69,70,72,73,76,77,79,90,93,94,105,107]; legibility (n = 2) [64,99]; comprehensiveness (n = 8) [32,36,46,55,64,75,97,106]; consistency (n = 3) [61,62,87]; reliability (n = 5) [57,65,76,90,95]; relevant (n = 1) [60]; format (n = 3) [66,75,95]; timeliness (n = 2) [29,53]; availability (n = 4) [20,21,22,72]

Data collection method Data review (n = 66) [20,23,25–30,32,34–43,45–49,51,53,55–60,62–66,68,71–75,77,80–84,88–91,93–103,105,106]; 0,54,5 ze (n = p (n =

analyze database (n = 22) [23–26,28,33,5 [53]; scanning documents and categori observation (n = 3) [21,22,44]; focus grou

the completeness of records does vary between different data components [34,38,46,77,93]. Furthermore, the documenta- tion seems to include more detailed data [24,27,41,100,102].

In two studies it has been shown that structured data entry improves data completeness [59,74], and further in three studies that completeness improves with time [33,68,103]. Attention has also been drawn to differences between end-

Table 7 – Research focusing on aspects of information system q methods used

Research focus

System quality (n = 32) Self-reporting [2 observation [21, computer [20,24

Ease of use (record keeping time) [20,21,24,27,29,35,41,42,44,47,56,65,73,79,80, 98–100,105–107]

Ease of learning [44,99] Usability [31,50,65,69] Timesaving [22,52,56,81,98]

Individual impact attributes (n = 4) Observation [98] Changed clinical work patterns [98] Changed documentation habits [56] Decision effectiveness

Speed of clinical decision-making [29] Changed habits [70]

User satisfaction (n = 12) Interviews [74,8 Attitude [74] User satisfaction [35,44] User acceptance [28,47,73,80,84,87,88,90,99]

Information use (n = 6) Analyze databas method [62]; usa

Frequency of use [21,92,104] Retrievability [28,62,74]

Organizational impact (n = 17) Questionnaire [2 audiotaping [75]

Communication and collaboration [27,29,56,99] Impact on patient care

Patient satisfaction [55] Physician–patient interaction [24,42,73,75] Length of patient stay [31] Effects on patient care [21,102] Consumer reactions [41] Advantages of glucose meters [86] Satisfaction with radiology services [20]

Training time [105]

Cost (budget) [20]

9,63,67,76,78,85,86,87,89,90,92,97,104,106,107]; computer clock (n = 1) 1) [61]; interview (n = 5) [69,70,79,80,81]; videotaping (n = 2) [31,93]; 1) [98]; questionnaire (n = 4) [20,52,57,99]; search method (n = 1) [60]

users [66]. Documentation by patients or their parents has also been reported to be good [80–86,88,89]. In one study, a mixed structured or directed text entry seems to be con-

ducive to more in-depth documentation by patients [80]. The completeness of different terminologies varies. Some termi- nologies cover all or almost all necessary terms or statements [30,31,71,91,104].

uality other than information quality and data collection

Data collection method

2,44,47,52,56,79,99,105]; questionnaire [21,44,50,65,69,81,99]; 27,29,35,44,47,73,80,98,100,107]; videotaping [31,42]; automatically by ,41,52,65,106]; focus group [69,98]; interview [65,69,79]; log files [65]

; focus group [98]; data review [56]; audit charts [29]; interview [70]

0,87,90,99]; questionnaire [28,35,44,73,80,84,87,88,99]; observation [47]

e [92]; requiring time [74]; interview [28]; semantic tagging search ge data [104]; observation [21]

1,27,29,31,33,41,55,56,99]; interview [20,73,86]; videotaping [42,75]; ; statistical analysis [20,105]; observation [24]; self-rating [102]

a l i n

a I v o m m s p w w i u c

i [ a t d r a r w v a [ p e i O o y a a i i t P

3 s A M s o s i h t T h t o [ i a u t d

i n t e r n a t i o n a l j o u r n a l o f m e d i c

Data accuracy is analysed in 29 papers. Documentation was ccurate according several studies [26,43,49,50,56,73,105,107]. n two studies, data entries by patients have also proved to be alid [79,90]. Structured data entries improved the accuracy f documentation [35,59]. Analyses of the legality of docu- entation found that the requirements of the law had been et in one study [99] but otherwise in one study there were

hortcomings in this respect [64]. Eight studies focused on com- rehensiveness. For the present purposes comprehensiveness as understood in terms of documentation in accordance ith the regulations and guidelines. In this regard shortcom-

ngs were observed in a number of studies [46,55,64,75,97]. The se of an information system has also proved to provide more omprehensive data [32,36,106].

Consistency has been the focus of interest in three stud- es. These studies have drawn attention to inconsistencies 61,62,87]. Reliability has been explored in five studies. Reli- bility is defined as the extent to which measurements yield he same results on repeated trials. It has been shown that ata from EHRs are reliable [90,95] when compared to manual ecords. One study addressed the issue of relevance [60]. In this nalysis relevance is defined as the ability to retrieve mate- ial that satisfies the user’s needs. The medical documents hich were sent from hospital to general practice were rele-

ant as input to the medical record [60]. The format of EHRs was nalysed in three papers. Records have been SOAP structured 66,75], in one paper the record format POMR of EHRs has been referred by physicians [95]. Timeliness was the focus of inter- st in two papers. No significant differences were observed n timeliness between desktop and hand-held computers [53]. ne paper drew attention to a significant delay in the delivery f medication documentation [29]. Data availability was anal- sed in four studies. Availability means that the data were ctually recorded and accessible to the end-user. Data avail- bility was found to be sufficiently good for the data to be used n decision-making [72], and image availability was improved n systems using PACS [20,21]. Another study showed hat there is no difference between conventional film and ACS [22].

.6.2. Impact of EHR on other aspects of information ystem success factors s was pointed out, the model proposed by DeLone and cLean consists of six dimensions of information system

uccess. The main interest in the studies reviewed was n information quality, but other aspects of information ystem success were also addressed (Table 7). System qual- ty has been analysed in 27 studies. The main concern as been with ease of use, which in this analysis means he amount of time taken up by recording-keeping (n = 21). here was no evidence that an information system can elp to save time [29,35,42,99,100], or that documentations ake more time [41,47,53,56,100,105]. Less time was spent n documentation when information systems were used

20,21,24,27,44,65,73,98,106,107]. Self-administered electronic nterviews by patients take up as much time as conducting

full interview [79]. It has been reported in one study that nstructured text is more time-consuming than using struc- ured questions [80]. The use of an information system for ocumentation takes more time, but on the other hand it was

f o r m a t i c s 7 7 ( 2 0 0 8 ) 291–304 299

also reported to help save time for example in the search for paper documentation [22,99].

Four papers have also explored individual impact attributes such as changed clinical work patterns, changed documenta- tion habits, decision effectiveness or altered policies to allow patients to see their own records. No changes have been observed in clinical work patterns. Bedside documentation was not successful [98], but improved quality of documenta- tion was also reported [56]. The use of an information system had no impact on the speed of decision-making. Surprisingly, information system use gave rise to an increased delay in the delivery of medication [29]. Patients themselves thought they had a very limited role in reading their EHR summaries [70].

User satisfaction was the focus of interest in 12 papers. Physi- cians accept the new structured dictation procedure. In their view structured notes have no direct impact on patient care, but they recognize that they might facilitate research. The advantage of using a computerized system is that it makes it much easier to locate cases according to diagnosis codes instead of having to scan the whole record [74]. Physicians [35] and pharmacists [44] preferred the electronic documenta- tion system over manual systems, but in one paper physicians preferred typewritten notes over a computerized system [73]. There is broad user acceptance of computers [28,47,84,99]. Information system use significantly increased acceptance of computers for documentation purposes based on the nursing process [99]. Computers were also readily accepted by patients [80,87,88,90].

Information use was the focus of interest in six studies. The frequency of use has been studied in three papers. The use of Read Codes to code diabetes varied between different prac- tices from 14% to 98% [92]. A significant increase was reported in the average number of radiology images reviewed by clin- icians [21]. It also shows that information was more easily retrievable from structured notes [74]. Physicians’ ability to recall patient data was better when an information system was used [28], and semantic tagging of information signif- icantly improved information retrieval from narrative notes [62].

Organizational impact attributes was the focus of interest in 16 studies. Attention has been drawn to the effects of information system use on communication and collaboration between different stakeholders. Computerized nursing docu- mentation improved communication between physicians and nurses [99]. Communication between primary and secondary care based on a computer system has been described as be useful, and it has been reported to improve the readability of documentation [27]. Significant better experiences were reported of shift reporting when a computer system was used [29]. The nursing charting system also affects the work of other health care practitioners. Four-fifths of physicians indicated that it was very easy to review patient data on terminals [56].

The use of EHRs and its impacts on patient care was investi- gated in 10 studies. Bedside technology did not seem to affect patient satisfaction with the nurse–patient relationship [55]. The computer system did not affect physician–patient inter-

action [24,42,73]. Some negative effects were also reported [42,73]. Monitoring of diabetes at home has a positive impact on patient care [86]. The level of user IT-literacy was reported to influence physician–patient interaction [75].

i c a l

300 i n t e r n a t i o n a l j o u r n a l o f m e d

The use of an information system had no bearing on patients’ length of stay in hospital [20], and the computer system had no effects on patient care [21,102]. Patients have shown no serious reactions to the adoption of electronic sys- tems, such as objections to the electronic interview [41].

No improvements were identified in the quality of radi- ology reporting service [20]. Training periods were long and more costly than expected [105]. The implementation of PACS has driven up costs, but outside radiology the system had also produced savings [20].

Methods of data collection varied, but most studies used qualitative methods (Tables 6 and 7). System quality was assessed by means of observation and time use by means of self-report, by computer or observation. In many cases the quality of the information documented was studied by means of content analysis against standards or guidelines, or by counting data items included in the documents or by quantita- tive analysis. Information use has been studied among other things by analysing databases. Among the methods of data collection used in studies concerning organizational impact or individual impact are semi-structured, in-depth or open- ended interviews, videotaping and questionnaires.

Comparisons of EHRs with manual paper records were pre- sented in 45 studies [20–23,25,27–29,34–36,38–41,44,46,47,50, 51,53,55,56,58,61,63,67,69,72–75,77,93,95,97–100,102,103,105, 106]. Patient self-documentation was also compared with documentation by health care professionals, or patient documentation was validated by health care professionals [80,83,87–90].

4. Discussion

A number of factors need to be considered in assessing the reliability and validity of this review. First of all, finding the right key words for the database search was extremely dif- ficult, and therefore a librarian was consulted. Secondly, the papers were reviewed by just one researcher. Furthermore, the review was confined to papers that could be accessed locally and to English language papers. The classification of the stud- ies according to their purpose was also extremely difficult, not least because they rarely provided explicit accounts of that purposes and therefore the inference had to be made by the author (KH).

The concept of EHR covers a wide range of different information systems from departmental systems to com- prehensive electronic health care records. Various kinds of departmental EHRs such as intensive care records, emergency department records or ambulatory records have now been in use for a long time, but hospital-wide EHRs, primary care or personal health records are less common. A patient-centred electronic health care record was introduced in only one study, and personal health records in eight studies. Interestingly, the definition of EHR does not include nursing information sys- tems or computerized instruments; however, descriptions of these systems or instruments were provided in the articles.

Few studies offered descriptions of the structure of EHRs, i.e. whether they were based on SOAP or the nursing process, even though studies from the 1980s in which the structure has been described were included in this review. The focus

i n f o r m a t i c s 7 7 ( 2 0 0 8 ) 291–304

of the studies has rather been on the use of different nursing and medical classifications, and international, national and local classifications have been applied. Furthermore, patient information has been structured by using different kinds of standardized instruments. Most EHRs are still primarily based on narrative text. Reuse of the data recorded in EHRs requires the use of different terminologies.

Most of the studies reviewed had been conducted in the context of tertiary or secondary care, which is where the first information systems were introduced. However some work has also been done in the context of home care. Research got under way in the early 1990s, and in the future patients will be even more closely involved in their own care. This means that patients will also be using EHRs both in health care organisa- tions and at home.

EHRs are used by many different health care profession- als, and the needs and requirements of all these professionals must be taken into account in the development of the infor- mation systems. EHR systems in multiprofessional use are precisely the information systems in such departments as intensive care unit or emergency department where the work by nature involves closer teamwork. On the wards, nurses and doctors record patient data in their own separate infor- mation systems, and the use of the other’s documentation is difficult, which might also have an effect on patient care. Almost half of the papers concerned research into medi- cal data components. However, nursing documentation, or documentation by other health care professionals such as physiotherapists, is an important part of the EHR and must not be excluded from medical documentation. Different kinds of standardized instruments are also an integral part of EHRs. Patients can also do parts of the documentation them- selves. Patient self-documentation also reduces the workload of health care professionals, but it is obviously important that self-documented data components are validated by profes- sionals. In a few studies, the documentation was done by secretarial staff according to the physician’s dictation. How- ever, the accuracy of documentation suffered when it was done by another person. It is important that all health care professionals who provide information record it themselves.

According to this review, the dimension of Information Quality in information systems was most typically mea- sured by two criteria: completeness and accuracy. However, other dimensions relevant to the success of information sys- tems were also analysed. The aspect of System Quality most frequently addressed was ease of use. Some studies also looked at the dimensions of user satisfaction, information use, individual and organizational impact. Both qualitative and quantitative methods of data collection were used.

The data included in paper-based patient records has pro- vided the golden standard against which the reliability of EHRs has been assessed. The quality of the information recorded in EHRs is extremely important. The success of EHRs depends on the quality of the information available to health care professionals in making decisions about patient care and in the communication between health care professionals dur-

ing patient care. Good quality of documentation improves the quality of patient care. It is important therefore to assess the quality of information entered in electronic systems by dif- ferent health care professionals. Decision-making tools can

i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n

Summary points

What was already known before this study:

• The EHR has been developed for a long time. • The content of EHR consists of unstructured narrative

text but also structured coded data.

What this study has added to our knowledge:

• An overview of all the varieties of information systems included in EHR.

• An overview of the content of EHR. • The finding that in EHR development work, nursing

information systems and the patient’s role in produc-

b t o r a i p r u p c

5

O o d t t p f i t s c a

r

ing data for EHR have not been taken into account.

e integrated in EHRs if the record is structured and defined erminologies are used; however if the data are inaccurate r incomplete, they will have no worth for decision-making, esearch, statistical or health policy purposes. It is not at ll clear and undisputed that record-keeping saves time, but t must also be taken into account that the use of com- uter systems improves the quality of documentation and educes other tasks. The structured data could also have other ses. If patients could enter data on their own health history, hysicians could use their own time more efficiently and con- entrate more on communication with patients, for example.

. Conclusion

n the basis of this review, it is obvious that studies focusing n the content of EHR are needed, especially studies of nursing ocumentation or patient self-documentation. Comparison of he documentation of different health care professionals with he core information of EHRs as determined in national health rojects is one possible focus of future research. The challenge or ongoing national health record projects around the world s to take into account all the different types of EHRs and he needs and requirements of different health care profes- ionals and consumers in the development of EHRs. A further hallenge is the use of international terminologies in order to chieve semantic interoperability.

e f e r e n c e s

[1] Canada Health Infoway, 2007, available at: http://www.infoway-inforoute.ca/en/home/home.aspx, accessed June 13, 2007.

[2] HealthConnect 2006, available at: http://www.healthconnect.gov.au, accessed June 13, 2007.

[3] Connecting for Health 2007, available at:

http://www.connectingforhealth.nhs.uk/, accessed June 13, 2007.

[4] W.A. Yasnoff, B.L. Humphreys, J.M. Overhage, et al., A consensus action agenda for achieving the national health

f o r m a t i c s 7 7 ( 2 0 0 8 ) 291–304 301

information infrastructure, J. Am. Med. Inf. Assoc. 11 (4) (2004) 332–338.

[5] Ministry of Social Affairs and Health, National Definition and Implementation of the Electronic Patient Record System. Working Group Memorandum of the Ministry of Social affairs and Health, 2002 (in Finnish).

[6] European Commission 2004. e-Health—Making Healthcare Better for European Citizens: An Action Plan for a European e-Health Area, available at: http://ec.europa.eu/ information society/doc/qualif/health/COM 2004 0356 F EN ACTE.pdf, accessed June 13, 2007.

[7] European Commission 2006.Connected Health: Quality and Safety for European Citizens, available at http://ec.europa.eu/information society/activities/health/ docs/policy/interoperability report final092006-cover.pdf, accessed June 13, 2007.

[8] J. Brender, C. Nohr, P. McNair, Research needs and priorities in health informatics, Int. J. Med. Inf. 58–59 (2000) 257–289.

[9] R. Haux, E. Ammenwerth, W. Herzog, P. Knaup, Health care in the information society. A prognosis for the year 2013, Int. J. Med. Inf. 66 (1–3) (2002) 3–21.

[10] R. Haux, Health information systems—past, present, future, Int. J. Med. Inf. 7 (3–4) (2006) 268–281.

[11] B. Blobel, Advanced and secure architectural EHR approaches, Int. J. Med. Inf. 7 (3–4) (2006) 185–190.

[12] P. Knaup, S. Garde, A. Merzweiler, N. Graf, F. Schilling, R. Weber, R. Haux, Towards shared patient records: an architecture for using routine data for nationwide research, Int. J. Med. Inf. 7 (3–4) (2006) 191–200.

[13] J. Špidlen, P. Hanzlı́ček, A. Řı́ha, J. Zvárová, Flexible information storage in MUDRII EHR, Int. J. Med. Inf. 7 (3–4) (2006) 201–208.

[14] C. Dostǎl, K. Pavelka, J. Zvárová, P. Hanzlı́ček, M. Olejárová, Some principles of the development of a clinical database/national register of selected inflammatory rheumatic diseases in the Czech Republic, Int. J. Med. Inf. 7 (3–4) (2006) 216–223.

[15] A.M. van Ginneken, The computerized patient record: balancing effort and benefit, Int. J. Med. Inf. 65 (2) (2002) 97–119.

[16] J. Grimson, Delivering the electronic healthcare record for the 21st century, Int. J. Med. Inf. 64 (2–3) (2001) 111–127.

[17] K. Thiru, A. Hassey, F. Sullivan, Systematic review of scope and quality of electronic patient record data in primary care, BMJ 326 (7398) (2003) 1070–1072.

[18] E. Ammenwerth, N. de Keizer, An inventory of evaluation studies of information technology in health care trends in evaluation research 1982–2002, Meth. Inf. Med. 44 (1) (2005) 44–56.

[19] ISO/DTR 20514, Health Informatics – Electronic Health Record – Definition, Scope, and Context, 2004.

[20] S. Bryan, G. Weatherburn, M. Buxton, J. Watkins, J. Keen, N. Muris, Evaluation of a hospital picture archiving and communication system, J. Health Serv. Res. Policy 4 (4) (1999) 204–209.

[21] B.I. Reiner, E.L. Siegel, F. Hooper, Z. Protopapas, Impact of filmless imaging on the frequency of clinician review of radiology images, J. Digit. Imag. 11 (3 Suppl. 1) (1998) 149–150.

[22] G. Weatherburn, S. Bryan, C. Cousins, A comparison of the time required by radiologists for the preparation of clinico-radiological meetings when film and PACS are used, Eur. Radiol. 10 (6) (2000) 1006–1009.

[23] M. Benson, A. Junger, L. Quinzio, et al., Influence of the

method of data collection on the documentation of blood-pressure readings with an Anesthesia Information Management System (AIMS), Meth. Inf. Med. 40 (3) (2001) 190–195.

i c a l

302 i n t e r n a t i o n a l j o u r n a l o f m e d

[24] D.J. Essin, R. Dishakjian, V.L. deCiutiis, C.D. Essin, S.N. Steen, Development and assessment of a computer-based preanesthetic patient evaluation system for obstetrical anesthesia, J. Clin. Monit. Comput. 14 (2) (1998) 95–100.

[25] J.P. Hollenberg, P.A. Pirraglia, P. Williams-Russo, et al., Computerized data collection in the operating room during coronary artery bypass surgery: a comparison to the hand-written anesthesia record, J. Cardiothorac. Vasc. Anesth. 11 (5) (1997) 545–551.

[26] D.A. Jordan, K.R. McKeown, K.J. Concepcion, S.K. Feiner, V. Hatzivassiloglou, Generation and evaluation of intraoperative inferences for automated health care briefings on patient status after bypass surgery, J. Am. Med. Inf. Assoc. 8 (3) (2001) 267–280.

[27] M. Apkon, P. Singhaviranon, Impact of an electronic information system on physician workflow and data collection in the intensive care unit, Intensive Care Med. 27 (1) (2001) 122–130.

[28] A. Kari, E. Ruokonen, J. Takala, Comparison of acceptance and performance of automated and manual data management systems in intensive care, Int. J. Clin. Monit. Comput. 7 (3) (1990) 157–162.

[29] J.A. Menke, C.W. Broner, D.Y. Campbell, M.Y. McKissick, J.A. Edwards-Beckett, Computerized clinical documentation system in the pediatric intensive care unit, BMC Med. Inf. Decis. Mak. 1 (1) (2001) 3.

[30] J.D. Young, C. Goldfrad, K. Rowan, Development and testing of a hierarchical method to code the reason for admission to intensive care units: the INARC coding method. Intensive care national audit & research centre, Br. J. Anaesth. 8 (4) (2001) 543–548.

[31] J.J. Cimino, V.L. Patel, A.W. Kushniruk, Studying the human–computer–terminology interface, J. Am. Med. Inf. Assoc. 8 (2) (2001) 163–173.

[32] D.L. Schriger, L.J. Baraff, W.H. Rogers, S. Cretin, Implementation of clinical guidelines using a computer charting system. Effect on the initial care of health care workers exposed to body fluids, JAMA 278 (19) (1997) 1585–1590.

[33] S.C. Hu, D.H. Yen, W.F. Kao, The feasibility of full computerization in the ED, Am. J. Emerg. Med. 20 (2) (2002) 118–121.

[34] A. Kabir, R. Hanson, C.M. Mellis, P.P. van Asperen, Is asthma documentation improved by computer-facilitated data entry? J. Qual. Clin. Pract. 18 (3) (1998) 187–193.

[35] K.A. Marill, E.S. Gauharou, B.K. Nelson, M.A. Peterson, R.L. Curtis, M.R. Gonzalez, Prospective, randomized trial of template-assisted versus undirected written recording of physician records in the emergency department, Ann. Emerg. Med. 33 (5) (1999) 500–509.

[36] D.L. Schriger, L.J. Baraff, K. Buller, et al., Implementation of clinical guidelines via a computer charting system: effect on the care of febrile children less than three years of age, J. Am. Med. Inf. Assoc. 7 (2) (2000) 186–195.

[37] A. Hasman, L.M. de Bruijn, J.W. Arends, Evaluation of a method that supports pathology report coding, Meth. Inf. Med. 40 (4) (2001) 293–297.

[38] D.L. Kent, E.H. Shortliffe, R.W. Carlson, M.B. Bischoff, C.D. Jacobs, Improvements in data collection through physician use of a computer-based chemotherapy treatment consultant, J. Clin. Oncol. 3 (10) (1985) 1409–1417.

[39] J.W. Kinn, M.F. O’Toole, S.M. Rowley, J.C. Marek, V.J. Bufalino, A.S. Brown, Effectiveness of the electronic medical record in cholesterol management in patients with coronary artery disease (Virtual Lipid Clinic), Am. J. Cardiol. 88 (2)

(2001) 163–165, A5.

[40] L. Knight, M. Yardley, A. Jones, The dangers resulting from inaccurate computer-based operative records, Br. J. Clin. Pract. 45 (1) (1991) 41–42.

i n f o r m a t i c s 7 7 ( 2 0 0 8 ) 291–304

[41] R.J. Lilford, P. Bingham, G.L. Bourne, T. Chard, Computerized histories facilitate patient care in a termination of pregnancy clinic: the use of a small computer to obtain and reproduce patient information, Br. J. Obstet. Gynaecol. 92 (4) (1985) 333–340.

[42] G. Makoul, R.H. Curry, P.C. Tang, The use of electronic medical records: communication patterns in outpatient encounters, J. Am. Med. Inf. Assoc. 8 (6) (2001) 610–615.

[43] D.L. Milliken, A.A. Gervais, The impact of computerization on medication discrepancies in a centralized unit dose drug distribution system, Hosp. Pharm. 25 (7) (1990) 650–652.

[44] G.T. Schumock, R.A. Hutchinson, B.A. Bilek, Comparison of two systems for documenting pharmacist interventions in patient care, Am. J. Hosp. Pharm. 49 (9) (1992) 2211–2214.

[45] M.M. Wagner, W.R. Hogan, The accuracy of medication data in an outpatient electronic medical record, J. Am. Med. Inf. Assoc. 3 (3) (1996) 234–244.

[46] S.A. Smith, M.E. Murphy, T.R. Huschka, et al., Impact of a diabetes electronic management system on the care of patients seen in a subspecialty diabetes clinic, Diabetes Care 21 (6) (1998) 972–976.

[47] W. Hundt, K. Adelhard, C. Hundt, et al., A computer-based reporting system in radiology of the chest, Eur. Radiol. 8 (6) (1998) 1002–1008.

[48] R.A. Rocha, S.M. Huff, P.J. Haug, D.A. Evans, B.E. Bray, Evaluation of a semantic data model for chest radiology: application of a new methodology, Meth. Inf. Med. 37 (4–5) (1998) 477–490.

[49] P.E. Nielsen, B.A. Thomson, R.B. Jackson, K. Kosman, K.C. Kiley, Standard obstetric record charting system: evaluation of a new electronic medical record, Obstet. Gynecol. 96 (6) (2000) 1003–1008.

[50] P.G. Nightingale, D. Adu, N.T. Richards, M. Peters, Implementation of rules based computerised bedside prescribing and administration: intervention study, BMJ 320 (7237) (2000) 750–753.

[51] L.S. Chan, N. Schonfeld, How much information is lost during processing? A case study of pediatric emergency department records, Comput. Biomed. Res. 26 (6) (1993) 582–591.

[52] G.L. Gibby, W.K. Schwab, Availability of records in an outpatient preanesthetic evaluation clinic, J. Clin. Monit. Comput. 14 (6) (1998) 385–391.

[53] P.B. Marr, E. Duthie, K.S. Glassman, et al., Bedside terminals and quality of nursing documentation, Comput. Nurs. 11 (4) (1993) 176–182.

[54] K.L. Movig, H.G. Leufkens, A.W. Lenderink, A.C. Egberts, Validity of hospital discharge International Classification of Diseases (ICD) codes for identifying patients with hyponatremia, J. Clin. Epidemiol. 56 (6) (2003) 530–535.

[55] R. Nahm, I. Poston, Measurement of the effects of an integrated, point-of-care computer system on quality of nursing documentation and patient satisfaction, Comput. Nurs. 18 (5) (2000) 220–229.

[56] T.A. Pryor, Computerized nurse charting, Int. J. Clin. Monit. Comput. 6 (3) (1989) 173–179.

[57] M.H. Samore, R.S. Evans, A. Lassen, et al., Surveillance of medical device-related hazards and adverse events in hospitalized patients, JAMA 291 (3) (2004) 325–334.

[58] C. van Walraven, S.V. Demers, Coding diagnoses and procedures using a high-quality clinical database instead of a medical record review, J. Eval. Clin. Pract. 7 (3) (2001) 289–297.

[59] N.T. Cheung, V. Fung, Y.Y. Chow, Y. Tung, Structured data entry of clinical information for documentation and data

collection, Medinfo 10 (Pt 1) (2001) 609–613.

[60] J.G. Bellika, E. Bones, G. Hartvigsen, PaSent—the patient’s personal health adviser, J. Telemed. Telecare 8 (Suppl. 2) (2002) 41–43.

a l i n

i n t e r n a t i o n a l j o u r n a l o f m e d i c

[61] G. Mikkelsen, J. Aasly, Concordance of information in parallel electronic and paper based patient records, Int. J. Med. Inf. 63 (3) (2001) 123–131.

[62] G. Mikkelsen, J. Aasly, Manual semantic tagging to improve access to information in narrative electronic medical records, Int. J. Med. Inf. 65 (1) (2002) 17–29.

[63] J. Stausberg, D. Koch, J. Ingenerf, M. Betzler, Comparing paper-based with electronic patient records: lessons learned during a study on diagnosis and procedure codes, J. Am. Med. Inf. Assoc. 10 (5) (2003) 470–477.

[64] A. Ehrenberg, C. Birgersson, Nursing documentation of leg ulcers: adherence to clinical guidelines in a Swedish primary health care district, Scand. J. Caring Sci. 17 (3) (2003) 278–284.

[65] G. Nilsson, H. Petersson, H. Ahlfeldt, L.E. Strender, Evaluation of three Swedish ICD-10 primary care versions: reliability and ease of use in diagnostic coding, Meth. Inf. Med. 39 (4–5) (2000) 325–331.

[66] G. Nilsson, H. Ahlfeldt, L.E. Strender, Textual content, health problems and diagnostic codes in electronic patient records in general practice, Scand. J. Prim. Health Care 21 (1) (2003) 33–36.

[67] I.M. Okkes, A. Groen, S.K. Oskam, H. Lamberts, Advantages of long observation in episode-oriented electronic patient records in family practice, Meth. Inf. Med. 40 (3) (2001) 229–235.

[68] M. Porcheret, R. Hughes, D. Evans, et al., Data quality of general practice electronic health records: the impact of a program of assessments, feedback, and training, J. Am. Med. Inf. Assoc. 11 (1) (2004) 78–86.

[69] C. Pyper, J. Amery, M. Watson, C. Crook, Patients’ experiences when accessing their on-line electronic patient records in primary care, Br. J. Gen. Pract. 54 (498) (2004) 38–43.

[70] L. Ward, M. Innes, Electronic medical summaries in general practice—considering the patient’s contribution, Br. J. Gen. Pract. 53 (489) (2003) 293–297.

[71] H. Petersson, G. Nilsson, L.E. Strender, H. Ahlfeldt, The connection between terms used in medical records and coding system: a study on Swedish primary health care data, Med. Inf. Internet Med. 26 (2) (2001) 87–99.

[72] D. Aronsky, P.J. Haug, Assessing the quality of clinical data in a computer-based record for calculating the pneumonia severity index, J. Am. Med. Inf. Assoc. 7 (1) (2000) 55–65.

[73] J.S. Hammer, J.J. Strain, A. Friedberg, G. Fulop, Operationalizing a bedside pen entry notebook clinical database system in consultation-liaison psychiatry, Gen. Hosp. Psychiatry 17 (3) (1995) 165–172.

[74] L.M. Ho, S.M. McGhee, A.J. Hedley, J.C. Leong, The application of a computerized problem-oriented medical record system and its impact on patient care, Int. J. Med. Inf. 55 (1) (1999) 47–59.

[75] V.L. Patel, A.W. Kushniruk, S. Yang, J.F. Yale, Impact of a computer-based patient record system on data collection, knowledge organization, and reasoning, J. Am. Med. Inf. Assoc. 7 (6) (2000) 569–585.

[76] J. Stausberg, H. Lang, U. Obertacke, F. Rauhut, Classifications in routine use: lessons from ICD-9 and ICPM in surgical practice, J. Am. Med. Inf. Assoc. 8 (1) (2001) 92–100.

[77] P.C. Tang, M.P. LaRosa, S.M. Gorden, Use of computer-based records, completeness of documentation, and appropriateness of documented clinical decisions, J. Am. Med. Inf. Assoc. 6 (3) (1999) 245–251.

[78] A. Naszlady, J. Naszlady, Patient health record on a smart card, Int. J. Med. Inf. 48 (1–3) (1998) 191–194.

[79] J.M. DeLeo, F. Pucino, K.A. Calis, K.W. Crawford, T. Dorworth, J.F. Gallelli, Patient-interactive computer system

f o r m a t i c s 7 7 ( 2 0 0 8 ) 291–304 303

for obtaining medication histories, Am. J. Hosp. Pharm. 50 (11) (1993) 2348–2352.

[80] S.C. Porter, I.S. Kohane, Optimal data entry by patients: effects of interface structure and design, Medinfo 10 (Pt 1) (2001) 141–145.

[81] S.C. Porter, M.T. Silvia, G.R. Fleisher, I.S. Kohane, C.J. Homer, K.D. Mandl, Parents as direct contributors to the medical record: validation of their electronic input, Ann. Emerg. Med. 35 (4) (2000) 346–352.

[82] S.C. Porter, Data acquisition in emergency medicine: electronic communication using free text, Pediatr. Emerg. Care 18 (1) (2002) 15–18.

[83] E.S. Glen, D.R. Small, L.M. Morrison, K. Pollock, Urological history-taking and management recommendations by microcomputer, Br. J. Urol. 63 (2) (1989) 117–121.

[84] M.J. Quaak, R.F. Westerman, J.A. Schouten, A. Hasman, J.H. van Bemmel, Appraisal of computerized medical histories: comparisons between computerized and conventional records, Comput. Biomed. Res. 19 (6) (1986) 551–564.

[85] F. Bischof, E.F. Pfeiffer, Experiences with a computerized diabetes information system, Horm. Metab. Res. Suppl. 26 (1992) 146–148.

[86] C. Meyerhoff, F. Bischof, E.F. Pfeiffer, Long-term experiences with a computerized diabetes management and glucose monitoring system in insulin-dependent diabetic patients, Diabetes Res. Clin. Pract. 24 (1) (1994) 1–7.

[87] D.R. Bangsberg, A. Bronstone, R. Hofmann, A computer-based assessment detects regimen misunderstandings and nonadherence for patients on HIV antiretroviral therapy, AIDS Care 14 (1) (2002) 3–15.

[88] J.M. Rabin, J. McNett, G.H. Badlani, “Compu-Void II”: the computerized voiding diary, J. Med. Syst. 20 (1) (1996) 19–34.

[89] A.A. Rivellese, M.M. Ventura, G. Vespasiani, et al., Evaluation of new computerized method for recording 7-day food intake in IDDM patients, Diabetes Care 14 (7) (1991) 602–604.

[90] R.N. Jamison, S.A. Raymond, J.G. Levine, E.A. Slawsby, S.S. Nedeljkovic, N.P. Katz, Electronic diaries for monitoring chronic pain: 1-year validation study, Pain 91 (3) (2001) 277–285.

[91] S.H. Brown, R.A. Miller, H.N. Camp, D.A. Guise, H.K. Walker, Empirical derivation of an electronic clinically useful problem statement system, Ann. Intern. Med. 131 (2) (1999) 117–126.

[92] J. Gray, D. Orr, A. Majeed, Use of Read codes in diabetes management in a south London primary care group: implications for establishing disease registers, BMJ 326 (7399) (2003) 1130.

[93] M. Pringle, P. Ward, C. Chilvers, Assessment of the completeness and accuracy of computer medical records in four practices committed to recording data on computer, Br. J. Gen. Pract. 45 (399) (1995) 537–541.

[94] S. Ratelle, D. Yokoe, C. Blejan, et al., Predictive value of clinical diagnostic codes for the CDC case definition of pelvic inflammatory disease (PID): implications for surveillance, Sex. Transm. Dis. 30 (11) (2003) 866–870.

[95] W.C. Stratmann, A.S. Goldberg, L.D. Haugh, The utility for audit of manual and computerized problem-oriented medical record systems, Health Serv. Res. 17 (1) (1982) 5–26.

[96] D.R. Studney, A.R. Hakstian, Effect of a computerized ambulatory medical record system on the validity of claims data, Med. Care 21 (4) (1983) 463–467.

[97] R. Wilton, A.J. Pennisi, Evaluating the accuracy of transcribed computer-stored immunization data, Pediatrics

94 (6 Pt 1) (1994) 902–906.

[98] M.K. Pabst, J.C. Scherubel, A.F. Minnick, The impact of computerized documentation on nurses’ use of time, Comput. Nurs. 14 (1) (1996) 25–30.

i c a l

304 i n t e r n a t i o n a l j o u r n a l o f m e d

[99] E. Ammenwerth, R. Eichstadter, R. Haux, U. Pohl, S. Rebel, S. Ziegler, A randomized evaluation of a computer-based nursing documentation system, Meth. Inf. Med. 40 (2) (2001) 61–68.

[100] J.M. Daly, K. Buckwalter, M. Maas, Written and computerized care plans. Organizational processes and effect on patient outcomes, J. Gerontol. Nurs. 28 (9) (2002) 14–23.

[101] M. Ehnfors, J. Florin, A. Ehrenberg, Applicability of the International Classification of Nursing Practice (ICNP) in the areas of nutrition and skin care, Int. J. Nurs. Terminol. Classif. 14 (1) (2003) 5–18.

[102] W.L. Holzemer, S.B. Henry, Computer-supported versus manually-generated nursing care plans: a comparison of patient problems, nursing interventions, and AIDS patient outcomes, Comput. Nurs. 10 (1) (1992) 19–24.

[103] J.H. Larrabee, S. Boldreghini, K. Elder-Sorrells, et al., Evaluation of documentation before and after implementation of a nursing information system in an acute care hospital, Comput. Nurs. 19 (2) (2001) 56–65, quiz 66–8.

[104] C.M. Prophet, The evolution of a clinical database: from

local to standardized clinical languages, Proc. AMIA Symp. (2000) 660–664.

[105] C. Kovner, L. Schuchman, C. Mallard, The application of pen-based computer technology to home health care, Comput. Nurs. 15 (5) (1997) 237–244.

i n f o r m a t i c s 7 7 ( 2 0 0 8 ) 291–304

[106] E.G. VanDenKerkhof, D.H. Goldstein, J. Lane, M.J. Rimmer, J.P. Van Dijk, Using a personal digital assistant enhances gathering of patient data on an acute pain management service: a pilot study, Can. J. Anaesth. 50 (4) (2003) 368–375.

[107] M. Curl, D. Robinson, Hand-held computers in clinical audit: a comparison with established paper and pencil methods, Int. J. Health Care Qual. Assur. 7 (3) (1994) 16–20.

[108] H.J. Tange, A. Hasman, P.F. de Vries Robbe, H.C. Schouten, Medical narratives in electronic medical records, Int. J. Med. Inf. 46 (1) (1997) 7–29.

[109] C.B. Averill, K.D. Marek, R. Zielstorff, J. Kneedler, C. Delaney, D.K. Milholland, ANA standards for nursing data sets in information systems, Comput. Nurs. 16 (3) (1998) 157–161.

[110] MedlinePlus, Medical Dictionary 2005, available at: http://www.nlm.nih.gov/medlineplus/mplusdictionary. html, accessed April 3, 2006.

[111] W.H. Delone, E.R. McLean, Information system success: the quest for the dependent variable, ISR 3 (1) (1992) 60–95.

[112] M.J. Van Der Meijden, H.J. Tange, J. Troost, A. Hasman, Determinants of success of inpatient clinical information

systems: a literature review, J. Am. Med. Inf. Assoc. 10 (3) (2003) 235–243.

[113] W.H. Delone, E.R. McLean, The DeLone and McLean model of information systems success: a ten-year update, JMIS 19 (4) (2003) 9–30.