Interoperability Project Plan

O

H W

a

A

R

R

1

A

K

F

H

E

O

M

U

T

T

E

O

1

H o h t f t m o t f u u

1 d

i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 2 ( 2 0 1 3 ) e1–e13

j o u r n a l h o m e p a g e : w w w . i j m i j o u r n a l . c o m

rganizational framework for health information technology

elga E. Rippen ∗, Eric C. Pan, Cynthia Russell, Colene M. Byrne, Elaine K. Swift estat, 1600 Research Blvd, Rockville, MD 20850, United States

r t i c l e i n f o

rticle history:

eceived 21 June 2011

eceived in revised form

0 October 2011

ccepted 26 January 2012

eywords:

ramework

ealth information technology

valuation

rganizational framework

odel

a b s t r a c t

Purpose: We do not yet know how best to design, implement, and use health information

technology (IT). A comprehensive framework that captures knowledge on the implemen-

tation, use, and optimization of health IT will help guide more effective approaches in the

future.

Methods: The authors conducted a targeted review of existing literature on health IT imple-

mentation and use, including health IT-related theories and models. By crosswalking

elements of current theories and models, the authors identified five major facets of an

organizational framework that provides a structure to organize and capture information on

the implementation and use of health IT.

Results: The authors propose a novel organizational framework for health IT implementation

and use with five major facets: technology, use, environment, outcomes, and temporality.

Each major facet is described in detail along with associated categories and measures.

se

echnology

emporal

nvironment

utcomes

Conclusion: The proposed framework is an essential first step toward ensuring a more con-

sistent and comprehensive understanding of health IT implementation and use and a more

rigorous approach to data collection, measurement development, and theory building.

© 2012 Elsevier Ireland Ltd. All rights reserved.

and to indicate gaps where further knowledge is needed.

. Introduction

ow can we maximize the benefits and minimize the risks f health information technology (IT)? We do not yet know ow best to design, implement, and use health IT. Although

here are stellar applications that are implemented success- ully on all fronts within a given organization [1], it is too often he case that applications are partially implemented, imple-

ented but never used, or implemented with disappointing r even adverse health or business impacts. What are the fac- ors that affect whether or not an application is a success or a

ailure? What measures can we use to assess success and fail- re? How can we apply our understanding of how health IT is sed to mitigate the risk of failure and maximize the benefits

∗ Corresponding author. Tel.: +1 240 453 2622. E-mail address: [email protected] (H.E. Rippen).

386-5056/$ – see front matter © 2012 Elsevier Ireland Ltd. All rights res oi:10.1016/j.ijmedinf.2012.01.012

of success? To answer these questions, we must go beyond a piecemeal approach that captures only discrete aspects of health IT such as the health IT product or the outcome. This requires a comprehensive organizational framework to struc- ture the array of information around the implementation and ongoing use of health IT. This organizational framework can provide the foundation for a more rigorous approach to data collection, measurement development, and theory building.

A framework is an effective way to present a clear, par- simonious, but comprehensive understanding of a complex topic. It provides a road map to organize current knowledge

Because frameworks can effectively highlight key dimen- sions, relationships, and research needs, they are often used to guide data collection, measurement development, and

erved.

i c a l

e2 i n t e r n a t i o n a l j o u r n a l o f m e d

theory building. For example, the Institute of Medicine’s landmark report, Crossing the Quality Chasm [2] presented six aims and ten rules for quality improvement as a framework to guide the redesign of the health care system. The framework was subsequently used and refined to guide data collection, measurement development, and theory building across a range of patient care processes, health care settings, and patient populations [3–5]. Frameworks are ideally suited to elucidate a complex field such as health IT. Although the beginnings of medical informatics can be traced back to at least the 1950s, its development as a formal discipline took place more recently with a focus on information and how to collect, analyze, and disseminate it within the health care delivery process [6]. By 1990, medical informatics was defined as “a rapidly developing scientific field that deals with resources, devices and formalized methods for optimizing the storage, retrieval and management of biomedical information for problem solving and decision making” [7].

The development of medical informatics as a scientific field owes much to attempts to understand the use of IT in non-health business areas and in consumer markets, espe- cially its rapid growth and winning and losing applications and investments that resulted. In the 1980s, major theories and approaches to IT included classic diffusion of innovation theory [8], organization assimilation of innovation analysis [9], socio-technical theory [10], the behavioral intention model [11], socio-cognitive theory [12] and change management [13]. From the 1990s through the 2000s, these models were applied to technology in a health care setting, with the focus on the technology alone. In the mid to late 1990s, it became increasing clear that the success of health IT implementa- tion and use involved more than just technology since health care organizations implementing health IT often encountered high failure rates and other significant challenges. However, few probed other factors [14]. Since then other fields such as change management [13] and usability [15] have contributed to a richer understanding of health IT [16].

2. Theories related to health IT

Publications on health IT implementation are often based on case studies that report before-and-after outcomes assess- ments of health IT as an intervention. Although they can provide rich detail on particular examples, they are often so focused on the specific aspects of the cases at hand that they are difficult to use as building blocks for constructing more generalizable theory. In addition, because of their focus on the process and impact of implementation, they offer limited insight into the underlying factors and conditions that shaped the outcomes [17].

To begin to build a more robust approach to the study of health IT, some researchers are assessing the applicability of major theories and models developed outside of health IT to better predict outcomes, to identify the important factors relating to success, and to determine how to mitigate risk.

Table 1 lists the aims and major components of several of these theories. The last column lists major aspects of health IT addressed in these theories and will be discussed in more detail in the following section.

i n f o r m a t i c s 8 2 ( 2 0 1 3 ) e1–e13

Individually and collectively, these approaches make valu- able contributions by calling attention to the role of a range of key factors influencing the implementation and use of health IT beyond the features of the technology itself. For example, some perspectives such as sociotechnical theory and social-cognitive theory focus on the important impact that individuals can have on health IT mediated through social sys- tems such as incentive and value structures, organizational processes, and organizational cultures. Other perspectives, such as technology diffusion and change management, seek to assess health IT use in a broader context of the relation- ship of individuals, groups, organizational features and other elements to the technology. These perspectives underscore the complex, interactive, and often subtle range of influences that shape health IT use and that must be considered in evaluating its initial use and ultimate outcomes. Still other perspectives, such as PRECEDE/PROCEED and multi-method, underscore temporal dimensions as initial health IT imple- mentation and use over time is affected by change over time in the environment or other factors.

While these theoretically driven approaches are broader and often richer than case studies, they are still highly focused, which allows them to deeply explore the impact of a limited number of factors. However, this prevents them from explain- ing the effects of others. For example, change management theory can be used to address environmental variables criti- cal for successful implementation, but it will neither predict nor explain an implementation that fails because the technol- ogy does not work (e.g., shuts down unexpectedly or does not scale). In addition, many of the measures used to substantiate them have not been validated in the context of health IT as indicated by a paucity of validation studies in the literature.

3. The organizational framework for health IT

An organizational framework for health IT would provide a critical step toward the development of a comprehensive model of implementation by supplying a structure to organize and capture information around its use, the relevant mea- sures and tools, and the relationships between and among different factors. Based upon our understanding of the health IT field, a targeted review of the health IT implementation literature [12–14,17,18,27–39], and the key theory-based com- ponents highlighted in Table 1, we have identified five major facets of an organizational framework. These facets are:

1. Technology—elements relevant to the specific health IT; 2. Use—elements relating to the actual use of the technology; 3. Environment—elements relating to the context influencing

the use of the technology; 4. Outcomes—elements capturing the end results of the tech-

nology in use in that environment; 5. Temporality—time and the developmental trajectory of

other elements such as implementation and clinical dis- ease processes.

The following explains each of the major facets in more detail, along with associated categories and measures.

i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 2 ( 2 0 1 3 ) e1–e13 e3

Table 1 – Health IT related theories, their components and overlap to framework facets.

Theory Exploratory aim Major components Corresponding organizational framework facet

Technology diffusion [8,18,19]

How diffusion of an innovation/technology spreads across a social system, including individuals, groups and organization

Innovation characteristics Compatibility Relative advantage Complexity Cost Communicability Divisibility Profitability Social approval Trialability Observability

Technology Use Environment Outcomes

Change management [13]

Relationship of people and organizational issues to the change process; Four stages around change

Organizations Individuals Groups Management of change process

Use Environment

Precede/proceed [12]

Integrated framework for implementing health IT

Five phases with levels of assessment - Organizational needs and goals - IT specifications and match with goals - Behavior and environmental - Educational and organizational - Points for system use Variables of interest Evaluation phase - Implementation - Process evaluation - Impact evaluation - System evaluation - Outcome evaluation

Use Environment Outcomes Technology Temporality

Sociotechnical theory [12]

How individuals interact with technologies relating to a task

Technical work processes Social systems within organization (users, their practices, their mental constructs and their interactions, management values)

Technology Use Environment

Multi-method [20]

Computerized provider order entry (CPOE) systems

Work and communication pattern Organizational culture Safety and culture

Environment Use Temporality

Social-cognitive theory [11,12,21]

Learning theory and behavioral change

Environment Situation Self-efficacy Outcome-expectation Reciprocal determinism Reinforcement

Environment Use Outcomes

Task-technology fit model [22,23]

Use of technology and how well it fits

Task characteristics Technology characteristics Performance impacts Utilization

Technology Use

Technology acceptance model [24]

Individual intention to use the system

Perceived usefulness Perceived ease of use Behavioral intention to use Actual use of system

Use Environment

Theory of planned behavior

Intention to use and human behavior

Attitudes Perceived behavioral control

s

Use Environment

4

T d r o h

[25,26] Subjective norm Intention Behavior

. Technology

he health IT implementation literature often does not

escribe the technology in a detailed way. However, details elating to a technology can be critical to the success r failure of an implementation. The following examples ighlight the importance of capturing information around

health IT implementation in a more detailed and uniform manner:

• Papers published on Computerized Provider Order Entry

systems invariably display screen shots of what a user sees. However, they often do not depict or otherwise examine the extent of the technology’s functionality and the smooth- ness of its incorporation into workflow. For example, order

i c a l

e4 i n t e r n a t i o n a l j o u r n a l o f m e d

entry without integration into the physician documentation process interrupts a clinician’s workflow.

• Information relating to performance and system downtime such as one to five minute click delays are often not men- tioned in studies, even when they may significantly affect the use of the system.

• Emergency department use of a health information exchange application to look for patient information does not note that for every 1000 patients seen, only 10 patients have some information available. This will influence the use of that application.

• An application to access patient information was built for phone X, but 90% of the physicians have phone Y that does not support the application, affecting the adoption rate.

Table 2 provides examples of categories associated with the technology facet. Ideally, each category, such as functionality, should be composed of subcategories. Currently, there is no standard set of categories and subcategories for technology.

Table 2 – Examples of categories associated with the technolog

Category Characteristics Current state of

Functionality • Describes the functionality and design purpose of the technical application

• Detailed specifica systems are often c but not made avail outside of developm • Can use taxonom concepts

Non-functional Requirements

• Indicates how well the system performs

• Reliability • Availability • Performance • Security • Scheduled downt • Update schedule

Data and interoperability

• Captures the attributes related to the data and their ability to be shared with other systems • Includes concepts of validation and data integrity, quality, currency, semantic interoperability and health IT-related standards

• Wide range of con with varying levels specificity

User-based design

• Includes user interface design but also the workflow that the health IT was designed to support

• Wide range of opt formats

Cost • There are several layers relating to cost: hardware; software; operation and maintenance; implementation costs

• Some costs are m difficult to measure others

Product • Describes the specific technology product (i.e., hardware, software)

• Can be very speci complex and inclu hardware and softw e.g., operating syst coding language, P

i n f o r m a t i c s 8 2 ( 2 0 1 3 ) e1–e13

Ideally, categories should also be associated with measures. For technology, measures range from percentages (e.g., avail- ability) to detailed technical specifications/artifacts. Although measures should have specified formats, currently there is also no standard set of categories or standard data formats for measures.

5. Use

Categories that are tied to the actual use of a technology are captured under the use facet (Table 3). This facet covers not only the “individual” user but also the “group” user discussed in many of the models. Also included are the individual mea- sures relevant to many of the models such as ownership, usability, motivation, workflow, perception of usefulness, ade-

quate training, and comfort with a technology. As with the other facets, many of these components are not adequately assessed in the literature. Many studies focus on one aspect of the facet, for example, an analysis of the usability of a system,

y facet.

measures Examples

tions for reated,

able ent

ies to tag

• Drug-drug alert compares an entered drug in a CPOE application against an XX product interaction list. If a positive match is identified, an alert based on that interaction is triggered and text from the list is presented to the user with an auditory alert such as a beep. The user is required to select one of two choices: (1) “Ignore and continue,” which progresses the ordering process with the information entered; and (2) “Edit,” which brings the user back to the drug entry screen. • Concepts: alerts; CPOE; alert lists; XX product

ime

• Availability is the percent of projected up time over total time that has passed, e.g., 99.99%, including scheduled downtime • Time delay between clicks is 200 ms

cepts of

• RxNorm used for medication • 50% of clinical data is shared with the system • The data is refreshed every 24 h rather than “immediately”

ions and • Developed based on user-centered design principles Detailed workflow mapping to support use in setting with assessment of match

ore than

• Cost for initial license and recurring yearly cost such as operations and maintenance

fic but des

are, em, DA type

• Microsoft 7 operating system; iPad 2; Product Z ambulatory Electronic Health Record (EHR)

i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 2 ( 2 0 1 3 ) e1–e13 e5

Table 3 – Examples of categories associated with the use facet.

Category Characteristics Current state of measures Examples

User attitudes • Cover a wide range of concepts such as user satisfaction, perceived usefulness and usability, and user acceptance

• Some measures are standardized while others are not

• User satisfaction with the drug-drug alert is low (scale)

Usability and Workflow

• Covers usability and actual workflow of the user

• Some usability constructs but significant variations. Workflow should be detailed and consistent to ensure comparison over time

• Physician not using CPOE so the drug-drug alert is seen by the pharmacist; doctor is called to question the prescription several hours later

Ownership/buy- in

• Captures the amount of user involvement and participation in the health IT implementation process

• No standard measures • Physicians were asked to help set the alert threshold for the drug-drug alerts

Knowledge • Includes concepts around adult learning, training, capability

• Training and effectiveness of training are related

ainin al p

ner, s

• Physicians, while being trained to use CPOE, were also trained about

b o

•

•

•

n i u i t s s w

• Tr form trai

ut fail to account for the others. Examples of the importance f these details include:

A nursing documentation system was implemented with a goal of ensuring real time data capture of vitals for physi- cians to access on the computer. However, the nurses’ workflow involved documentation of vitals on a piece of paper in the patient’s room for the physician. The final doc- umentation in the chart took place at the end of the shift. The implementation of the system changed nothing related to the timing of the final documentation, which was now entered into the computer at the end of the shift. This meant that physicians did not have ‘real time’ access to the vitals on the computer.

Physicians were told they had to use the CPOE system and were not involved in the selection of the system or the development of order sets. When the system was imple- mented, many of the physicians did not use the predefined order sets, ordering took a significant time, and resistance dramatically increased when errors were discovered. There was no ownership or sense of responsibility to solve prob- lems that arose, and the CPOE system was subsequently abandoned.

Some of the older physicians were uncomfortable with com- puters. When the hospital moved to EHRs and a paper-free record, many physicians considered leaving. These physi- cians were assigned a physician “buddy” who trained them and answered questions. Discomfort with the technology was decreased as a result of this intervention.

It is clear that the categories relevant to use of the tech- ology are important to understanding the success of an

mplementation. Capturing details around user attitudes, sability and workflow, ownership, and knowledge provides

nsights critical to understanding how best to implement

hese applications. Some categories have validated mea- urement tools, e.g., user satisfaction survey. Others have a tandard approach, e.g., cross-functional flowcharts to assess orkflow that may have variations.

g modalities such as resentation, train the uper-user support

the alerts, the threshold settings, how to address them (e.g., override or edit)

Table 3 provides examples of categories associated with the use facet. As was true for the technology facet, each category, such as user attitudes, should be composed of subcategories. Moreover, there are no standard set of categories and sub- categories for use or related measures and data formats. For use, measures range from surveys of user attitudes to user feedback on training experiences/value. Additionally there are metrics that capture the percent use of a specific functionality within a technology as in the case of physician acknowledg- ment of drug alerts.

6. Environment

The environment facet captures categories that influence the implementation and use of the technology. Some examples that highlight the importance of capturing information related to this facet include:

• The financial incentives being offered by the federal gov- ernment to implement health IT in a meaningful way, combined with the ability of hospitals to provide support to clinicians because of changes to STARK [40], have enabled hospitals to more freely provide EHRs at a time when addi- tional incentives to physicians are being offered.

• The IT infrastructure and the challenges to integrate the hospital EHR with the physician’s outpatient EHR are barri- ers to implementation.

• CEO and physician champions can reinforce the commit- ment of the hospital to implementation of a CPOE system and its importance to patient safety. This can enhance communications around the initiative and help clinicians prepare for the change.

Many of these categories contain numerous elements or subcategories. However, as with other facets, there are no established standard measures used to capture and describe

them. The “measures” range from the presence or absence of an element to detailed description of federal and local policies.

Table 4 provides examples of categories associated with the environment facet. These categories describe variables

e6 i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 2 ( 2 0 1 3 ) e1–e13

Table 4 – Examples of categories associated with the environment facet.

Category Characteristics Current state of measures Examples

Cultural/organizational • Captures teamwork climate, values, culture

• Some measures exist, others are descriptive

• The culture supports innovation and teamwork

Business drivers • This includes governmental policies and regulations that influence the organization and business factors, e.g., competition

• These are often descriptive but can be summarized in a categorical measure such as favorable, neutral or negative on the health IT implementation

• STARK enhanced the ability and desire of hospitals to provide EHRs to physicians

Leadership • Senior leaders and champions fit into this category

• Leadership also is descriptive, but can be captured in a scale to indicate level of support and relevant attributes of leadership

• Physician champion is involved, visible, engaging, and trusted

Setting • Which environment the health IT is being used

• Includes traditional descriptors such as inpatient, outpatient

• 400 bed hospital associated with an academic medical center

Resources • This includes not only the resources available to support the implementation of the health IT, but also the IT infrastructure that can enable it

• Resources cover a broad range from financial and human resources to the state of the IT infrastructure (e.g., broadband)

• Availability of broadband within the hospital; presence of dead zones within the hospital

• Sup man impl

Support • Many aspects of implementation management fit under this category, including training

that influence how health IT is used. For example, environ- mental categories such as leadership can influence whether or not the health IT will be successfully implemented [13]. Environment related measures range from surveys of cli- mate/organizational cultural to asset inventories. As was true for the each facet, each category, such as business drivers, should be composed of subcategories with associated mea- sures.

7. Outcomes

The outcomes facet provides the categories that are the indi- cators for failure and success of the health IT implementation. Three categories are highlighted as examples in Table 5. Com- pared to the other facets, outcomes has the most well-defined measures [36,41]. Despite the maturity of outcome measures, the measures captured tend to be very narrowly defined and at times are not aligned with the other facets. Examples that highlight these challenges are described below:

• A disease management system with functionality to sup- port diabetics is implemented to help patients better maintain their blood sugar by using an online personal health record to monitor and report back to the case man- ager. Most patients did not have access to a computer, and the intervention had no effect.

• Physicians taking care of patients within an HMO were given reminders to provide patients with pneumococcal vaccines. After one year, 98% of the appropriate patients had received the vaccine. Cost savings from avoidance of hospitalizations for pneumococcal vaccine were only captured for the first year. True savings are greater since the benefits last beyond one year.

Table 5 provides examples of categories associated with the outcome facet. Ideally, each category, such as clinical, should be composed of subcategories. Currently, there is no standard

port for training, users, and agement of the ementation project

• Help desk to assist with forgotten passwords

set of categories and subcategories or standardized measures for the outcome facet. For outcome, measures range from clin- ical and business measures that are predefined to compliance rates which vary in calculation. Also captured under the out- comes facet is the methodology of the study. Although some outcome measures have specified formats, currently this is not consistent among all the categories.

8. Temporality

Temporality is a final and critical facet for the organizational framework [8,18,42,43]. The principal temporal category is time. Time as an independent variable allows for linking mea- sures across all facets over prescribed periods. The facet also includes two other temporal categories, the implementation cycle and the outcomes lifecycle, summarized in Table 6.

Table 6 provides examples of categories associated with the temporality facet. Ideally, each category, such as implementa- tion cycle, should be composed of subcategories. Currently, there is no standard set of categories and subcategories for temporality. Ideally, categories should also be associated with measures. For temporality, measures range from the unambiguous measurement of time to the less structured measurement associated with the health IT implementa- tion cycle. Although measures should have specified formats, currently there is also no standard set of categories or stan- dard data formats for measuring planning or implementation cycles.

The temporality facet provides a critical axis for evalu- ation. Over time, the characteristics captured in the other facets will change. Linking these changes to time (and/or other temporal categories) allows for comparisons and understand- ing of relationships between and across facets. Information

captured across all facets at the start of a health IT implementation provides a baseline “snapshot.” Information collected across the facets in the middle of an implementa- tion captures another “snapshot.” At another phase of the

i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 2 ( 2 0 1 3 ) e1–e13 e7

Table 5 – Examples of categories associated with the outcomes facet.

Category Characteristics Current state of measures Examples

Clinical • Covers the clinical outcomes related to the use of the health IT • Quality measures

• There are many clinical outcome measures in the literature, and they need to be mapped to the specific intervention

• Reduction in pneumococcal pneumonia (reminders to immunize)

Business/financial • Cost savings or expenditures are part of the business outcomes

• Business measures include increases in efficiencies and reductions in cost or hospital days

• Hospital stays for senior patients dropped due to reduced iatrogenic pneumococcal pneumonia

Adoption • Includes the number of users and the depth of their use

• Captured as a percentage of users to potential users; level of use of a system can be quantified a variety of ways

• 95% of physicians using CPOE to enter their orders

Methodology • Covers the details associated with their study and analysis

• Study type (e.g., case study, case–control, prospective, retrospective)

uanti ualita rview

• Case–control study of two similar practices, with one implementing an EHR (case) and another

“ l w t “ d f

9

E t

• Q • Q inte

implementation,” another snapshot can be captured. The col- ection of these “snapshot” data sets over time across facets ill enable a more robust evaluation of health IT implemen-

ation and the factors relevant to success or failure. The more snapshots” that are assessed, more fidelity and insight can be eveloped on the dynamics between the different facets and actors.

. Cross-walk of facet with five papers

arlier we evaluated health IT-related models to inform he development of the major facets. To further assess the

Table 6 – Examples of categories associated with the temporali

Category Characteristics Curre

Time • External anchor • Construct for understanding duration • Independent variable

•Unamb

Implementation cycle

•Characterizes a step in a lifecycle of health IT implementation •Not dependent upon time •Supports comparison across different implementations independent of time

•No sing phases •Categor

Outcome lifecycle

•Characterizes the lifecycle of when an intervention can be expected to generate a given outcome • Provides the ability to account for findings when a study has not provided for sufficient time for evaluation

•Interven • Need to effective •Comple delivered •Outcom long tim •Tied to

tative (time, cost) tive (e.g., focus groups, s)

continuing paper-based practice (control)

comprehensiveness of our proposed major facets, we also conducted a detailed crosswalk with five different types of papers summarized in Table 7 and listed below:

1. A comprehensive international literature review of EMR studies Häyrinen et al. [32];

2. A systematic review study of Health IT impact based on

English-language publications by Chaudhry et al. [44];

3. A model of organizational change across several disciplines applied to medical informatics systems by Lorenzi et al. [13];

ty facet.

nt state of measures Examples

iguous •Date (e.g., month/day/year); duration (e.g., days, months, years)

le standard definition of

ical variable

•Planning, implementation, evaluation, and optimization

tion specific account for variance in

ness of intervention x if intervention is

over time es can continue across a

e span other facets

•CPOE drug-allergy alerts: outcome is avoidance of event; outcome is “immediate”: the presence or absence of an avoidable drug-allergy event can be realized without a lag •Reminders for pneumococcal vaccine: outcome is reduced costs for pneumococcal pneumonia; outcome is realized across the lifetime of the patient •Smoking cessation reminders and tools: outcome is occurrence of acute myocardial infarction; outcome would be expected in a time frame of months to years

e8 i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 2 ( 2 0 1 3 ) e1–e13

Table 7 – Crosswalk between the facets of the organizational framework and components identified in five different types of health IT papers.

Source Have Organizational Framework Facets

Technology Use Environment Outcome Temporality

Häyrinen et al. [32]

EHRs classification: International Organization for Standardization (ISO) definition Medical component: Referral, present complaint (e.g. symptoms), past medical, history, life style, physical examination, diagnoses, tests, procedures, treatment, medications, discharge Data elements: Diagnoses codes (ICD, ICPC), procedure (CPT), medications (ATC), pathological findings (SNOMED), nursing problems (NANDA, ICPN) System quality: (e.g., ease of use, ease of learning, usability*) Information use: (e.g., retrievability) Cost

User: Nurse, physicians, patients, pharmacy, others Usability: Timesaving, clinical work patterns, documentation habits, user satisfaction, attitudes, acceptance

Settings: Inpatient tertiary, secondary care, home health, outpatient

Information quality: (e.g., completeness and accuracy) System quality: (e.g., usability*, timesaving) User effects: information use, user satisfaction, individual impact (e.g., clinical work patterns, changed documentation habits, decision effectiveness or altered policies to allow patients to see their own records)**

Organizational impact: (e.g., communication and collaboration, impact on patient care) Patient effects: Patient satisfaction, physician–patient interactions, length of patient stay, effects on patient care, consumer reactions Cost

Chaudhry et al. [44]

Broad range of Health IT: e.g., CPOE, decision support-stand-alone systems, electronic results reporting, electronic prescribing, consumer health informatics/patient, decision support, mobile computing, telemedicine, data exchange networks, knowledge retrieval systems Human factors: (e.g., user-friendliness)

User: Broad range of users Organizational process change: (e.g., workflow redesign)

Settings: Many settings including: inpatient tertiary, secondary care, home health, outpatient, academic, health system (e.g., VA) Project management: (e.g., achieving project milestones)

Effects on Quality: Reduction of medication errors, adherence to protocol-based care, enhanced capacity to perform surveillance and monitoring for disease conditions Effects on Efficiency: Decreased rates of utilization of redundant or inappropriate therapies, provider time savings Effects on Costs: Related to reductions in utilization Effect on time utilization: Secondary preventive care based on time-series studies (e.g., reduced hospitalizations, reduced pressure ulcers)

i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 2 ( 2 0 1 3 ) e1–e13 e9

– Table 7 (Continued)

Source Have Organizational Framework Facets

Technology Use Environment Outcome Temporality

Disease progression: (primary preventive care)

Lorenzi et al. [13] Ease of use of the technology

User acceptance and satisfaction User involvement and participation during implementation

Organizational cultural: Leadership, politics, strategy, psychology, intelligence, decision making Organizational structure: Simple, bureaucracy, adhocracy, networked, federal, divisional

Ash et al. [45] Interoperability: The system’s ability to communicate

Workflow: Impact on added computer activities on work performance

Financial Incentives Safety improvement to work processes: Organizational Culture Driven top down verse provider promoted

Quality of care and outcomes

Wright et al. [46] System: Brigham Integrated Computing System, MEDIECH MAGIC, EpicCare, Physician Order Entry, Allscripts EMR, Siemens INVISION, McKesson Horizon Expert Orders Order sets: General admission orders, surgical/anesthesia orders, critical care admission, clinical pathways, obstetrics, diabetes orders

Setting: Hospital Size (Staffed beds) Discharges Location: City, State Type: Academic medical center, community, health system

Patient Days Discharges Order set utilization: Total order sets, order set usages/yr, frequency of use of order sets

∗ Some overlap between technology and use exists; for example, usability is an attribute of a technology but when assessed within the context of use, e.g., workflow, it is captured under use.

ome.

4

5

r p t t S e a

∗∗ Behavior is captured as part of use; the change, over time is in outc

. An assessment based on medical informatics directors of gaps in EHR adoption and reasons for the gaps and possible solutions by Ash et al. [45]; and

. A comparison of technology use across seven sites by Wright et al. [46].

Specific components described in each paper were catego- ized under one of the facets in our framework. This crosswalk rovided insightful findings. It became immediately clear that here is little consistency in how components are concep-

ualized and in how terminology is used to describe them. ome of the papers used in the crosswalk, along with oth- rs reviewed for our study, describe contextual factors that ffect health IT implementation, but do not use a framework

to organize those factors. Other studies provide organizational frameworks, although each one is different.

An encouraging finding from this crosswalk is that our framework encapsulates all of the important health IT study components identified in the five studies. Further, we found that some facet components such as workflow, organizational change, and usability are commonly discussed in the stud- ies. However, other facet components such as those related to temporality are not often explicitly discussed, although they are inherently important to the health IT implementation.

Chaudhry et al. [44] noted that the heterogeneity in reporting on categories such as functionality (technology facet) made it difficult to assess whether system capabilities were absent or simply not reported.

e10 i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 2 ( 2 0 1 3 ) e1–e13

Table 8 – Illustrative example of two CPOE implementations is related to outcomes over the course of an implementation.

Temporal

Pre-implementation Implementation Post-implementation

Comparison Tool 1 Tool 2 Tool 1 Tool 2 Tool 1 Tool 2

Technology + +++ + +++ ++ +++ Use ++ − ++ + +++ −

++ +

Environment + −− Outcome n/a n/a

10. Conclusion

Given the complex nature of health IT implementation, an organizational framework is an essential first step toward ensuring more consistent and more comprehensive data col- lection. Ensuring that the framework can support the data collection of different models will facilitate the development of more comprehensive models of effective health IT imple- mentation. Technology, use, environment, outcomes, and temporality facets are the core features of this organizational framework. These facets were developed based on our knowl- edge of the field, a limited literature review, and a crosswalk of the data elements tied to current theories/models used in health IT and the literature.

Although this framework is preliminary, the five facets and the category examples may be able to provide a high level checklist of data categories to consider in designing a study. The framework can also be used to begin exploring the interrelationship between the different facets across studies. An illustrative example is highlighted in Table 8 where two CPOE implementations projects, Tool 1 and Tool 2, are fol- lowed across implementation stages (temporality facet). In this example, each facet is reduced to a global measure with a +++ to −−− scale (+ is positive; − is a negative). Tool 1 has relatively positive technology attributes, with challenges around the performance of the system. This is contrasted by Tool 2, which is a “top of the line” tool, with great perfor- mance. On the attributes around the use facet, Tool 1 was favorable; it fits into the clinician’s workflow, they have been involved in its selection and the staff participated in well- designed training. On the other hand, Tool 2 did not support the workflow of the clinical staff, they did not participate in the selection of the application (no ownership) and the training was mediocre. Environmental characteristics for Tool 1 were positive—resources and technical support available and lead- ership was supportive. For Tool 2, insufficient resources were provided for with minimal support. The leadership assumed not much was needed given that it was a top of the line tool. The implementation and post-implementation course for both of the tools were different. For Tool 1 more resources and leadership support was rallied during implementation and tool performance was improved over time. Use continued to increase as positive outcomes were demonstrated. On the other hand, Tool 2 was used by some during the implemen- tation push, with an increased involvement in the leadership

team. However, post-implementation, things reverted to pre- implementation, as there was no perceived value to sue it.

− + −− 0 ++ 0

From Table 8, it is clear that although the technology in Tool 2 was more favorable than Tool 1, Tool 1 implementation was successful and Tool 2 was not. Each facet can play a critical role in the success or failure in the successful design, implemen- tation, and use of health IT. Pushing the use of a technology when it cannot meet the requirements may result in more harm than good. For example, when laboratory results are not updated in real time in an inpatient EHR system, the lag time may result in duplication of orders or worse, delaying a treat- ment that could result in significant adverse consequences for a patient. If the data is not reliable or available, similar negative consequences may result and a lack of confidence in these sys- tems by the users will ensue. However, in the example above, the technology was not the core factor in the success or failure of the use of the health IT.

The technology is irrelevant without the users and the con- text around its use. Even when the technology is still “a work in progress”, users will use the system if the benefits outweigh the costs. Nuances such as ownership, matches in workflow between what the system was designed for versus what is needed for clinical care, attitudes of the users and training are important to success. In the example above, it was the lack of fit and buy-in at the clinical level that resulted in failure. The environmental conditions around implementation can also play a significant part in the success or failure of a technology. Without leadership support, policies, resources, and business drivers the use of a technology cannot be implemented or sus- tained. Often the use and environmental conditions influence each other either positively or negatively. The outcome of the implementation and use of a technology can be effected by the technology, use and environmental. When temporal shifts are not accounted for, findings may not be possible even when the health IT was effective, e.g., reduced healthcare expenditures for complications of diabetics cannot be expected for a num- ber of years (not medically possible). Hence, the temporality facet provides the ability to track changes in the other facets over time and events.

There are limitations in examining and aggregating infor- mation into global scores for each facet. There may be bias in how scores are assessed unless there is some validation and assessment of inter-rater reliability. Moreover, “global” scores may mask more important categorical effects. For this and other reasons, a “scoring” approach at the facet level for pre- dicting outcomes would be inappropriate. However, given our limited understanding of the field, exploration of tools and

approaches to quantify these facets of health IT may yield some valuable insights. Given the complexity of the factors relevant in the success or failure of an implementation and use of a health IT, it is more likely that categories or subcategories

a l i n f o r m a t i c s 8 2 ( 2 0 1 3 ) e1–e13 e11

w T g c I

b o m e a t c a t a

t t t m I f t t b r

1

W f p i

•

•

•

•

•

Summary points What was already known

• Health IT implementation is complex; “wicked”. • Several theories are being explored to better under-

stand health IT implementation. • There is a need to address our limited understanding.

What this study added to our knowledge

• No model provides a comprehensive construct for health IT implementation.

• Five facets of technology, use, environment, temporal, and outcomes provide the constructs for an organiza- tional framework.

• An organizational framework for health IT should organize data, tools and measures in a consistent man- ner for the field of health IT, facilitating data collection, measurement development, and theory building.

i n t e r n a t i o n a l j o u r n a l o f m e d i c

ithin the facets could be found to be predictors of success. he exploration of the facets and their corresponding cate- ories and subcategories will provide insight for researchers to ontinue to build and refine predictive models around health T.

However, in order for evaluations across studies to e meaningful, each of the facets should be composed f categories and subcategories that can be consistently easured—in standard formats—to facilitate cross-study

valuations. The resulting, more detailed framework with ssociated standard measures and formats will prove essen- ial to providing a mechanism to make the organization and ollection of data and observations uniform across studies nd technologies. This is critical if we are to pool information hat can be used to develop and evaluate models and theories round implementation.

Although evaluation of theories that capture the concep- ualization, development and marketing of a new innova- ion/technology [23] have not been included in the scope of his paper, we agree with Chaudhry et al. that the develop-

ent of uniform standards for the reporting of research health T implementation must be a high priority [44]. The findings rom the crosswalk suggest that our framework can serve as he basis for organizing and reporting research on implemen- ation of health IT, and reducing heterogeneity in reporting y ensuring that all important facets and their components elated to health IT are identified using common terminology.

1. Next steps

e have proposed five facets for an organizational framework or health IT. It is the first step in the development of a com- rehensive organizational framework for the field. Next steps

nclude:

Further refinement of the categories and subcategories obtained from the literature and expert panels. The addition of categories and subcategories will allow a more granular understanding of health IT.

Identification of measurement tools and measures associ- ated with the categories or subcategories. Associating tools and measures with categories and subcategories are a crit- ical step in enabling researchers to compare studies, and test models. These measures will likely include text descrip- tions, categories, scales, and numbers. Construction of a health IT taxonomy from a variety of sources to ensure consistency in use of terms. A taxonomy will facilitate a common understanding of terms and the consistent use of categories and measures. Taxonomies are living concepts that change over time and require a main- tenance process and infrastructure.

Development of a database that supports the collection of complex data, which can be expanded as we learn more. A data-mart to adequately support the complex nature of the data and maintain data relationships required by the

organizational framework will enable the collection of the data and its analysis.

The development of a publicly available tool that researchers can access to obtain and/or add tools and

measures and that they can download and/or upload study data. An online data-mart available to researchers in health IT will accelerate our understanding of health IT and lever- age the multitude of studies currently underway. Moreover, this approach will reinforce the use of the measures and the associated tools to capture them.

• The development of tools to explore the data and rela- tionships. Once the data are available, tools that can help explore complex data sets will facilitate the evaluation of models and our understanding of health IT.

Author contributions

All authors contributed toward (1) the conception and design of the study, or acquisition of data, or analysis and interpre- tation of data, (2) drafting of the article or revising it critically for important intellectual content, and (3) the final approval of the version to be submitted.

Conflict of interest

None of the authors have any conflicts of interest that could bias this work.

Role of funding source

Westat employees the authors of this manuscript and Westat was not involved any aspects of the study.

Acknowledgements

We would like to thank Susan Crystal-Mansour and Ejim Mark for help in editing the document.

i c a l

r

e12 i n t e r n a t i o n a l j o u r n a l o f m e d

e f e r e n c e s

[1] D.W. Bates, The road to implementation of the electronic health record, JAMIA 19 (4) (2006) 311–312.

[2] Institute of Medicine (IOM), Crossing the Quality Chasm: A New Health System for the 21st Century, National Academy Press, Washington, DC, 2001.

[3] Institute of Medicine (IOM), Envisioning the National Health Care Quality Report, National Academy Press, Washington, DC, 2001.

[4] Institute of Medicine (IOM), Improving the Quality of Health Care for Mental and Substance-Use Conditions, National Academy Press, Washington, DC, 2006.

[5] U.S. Department of Health and Human Services (HHS), National Strategy for Quality Improvement in Health Care, 2011, available at: http://www.healthcare.gov/law/resources/reports/nationalq ualitystrategy032011.pdf, accessed 10/05/2011.

[6] C.W. Hanson, History of Computing in Medicine, February 6, 2002, available at: http://www.cs.princeton.edu/courses/archive/spr02/cs495/ History%20of%20Computing%20in%20Medicine.pdf, cited 16/6/2011.

[7] R.A. Greenes, E.H. Shortliffe, Medical informatics: an emerging academic discipline and institutional priority, AMA 263 (1990) 1114–1120.

[8] E.M. Rogers, Diffusion of Innovations, 3rd edition, Free Press, New York, 1983.

[9] A.D. Meyer, J.B. Goes, Organizational assimilation of innovations: a multilevel contextual analysis, Academy of Management Journal 31 (1988) 123.

[10] W. Pasmore, Social science transformed: the sociotechnical perspective, Human Relations 48 (1995) 1–21.

[11] A. Banduar, Organisational applications of social cognitive theory, Australian Journal of Management 13 (December) (1988) 275–302.

[12] R. Kukafka, S.B. Johnson, A. Linfane, J.P. Allegrante, Grounding a new information technology framework in behavioral science: a systematic analysis of the literature on IT use, Journal of Biomedical Informatics 36 (2003) 218– 227.

[13] N.M. Lorenzi, R.T. Riley, A.J. Blyth, G. Southon, B.J. Dixon, Antecedents of the people and organizational aspects of medical informatics: review of the literature, AMIA 4 (1997) 79–93.

[14] R.M. Braude, People and organizational issues in health informatics, JAMIA 4 (1997) 150–151.

[15] Koppe lR, D.A. Kreda, Healthcare IT usability and suitability for clinical needs: challenges of design, workflow and contractual relations, Studies in Health Technology and Informatics 157 (2010) 7–14.

[16] F.C.G. Southon, C. Sauer, C.N.G. Dampney, Information technology in complex health services: organizational impediments to successful technology transfer and diffusion, JAMIA 4 (1997) 112–124.

[17] K. Kawamoto, C.A. Houlihan, E.A. Balas, D.F. Lobach, Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success, British Medical Journal 330 (April) (2005) 765.

[18] J. Ash, Organizational factors that influence information technology diffusion in academic health sciences centers, JAMIA 4 (1997) 102–111.

[19] K.J. Klein, R.S.A. Ralls, Organizational dynamics of computerized technology implementation: a review of the empirical literature, Advances in Global High-Technology Management 5 (1995) 31–79.

i n f o r m a t i c s 8 2 ( 2 0 1 3 ) e1–e13

[20] J.I. Westbrook, J. Braithwaite, A.W. Georgiou, A. Ampt, N. Creskwick, E. Coiera, et al., Multimethod evaluation of information and communication technologies in health in the context of wicked problems and sociotechnical theory, JAMIA 14 (2007) 746–755.

[21] M.D. Slater, Integrating application of media effects, persuasion, and behavioral change theories to communication campaigns: a stages-of-change framework, Health Communications 11 (1999) 335–354.

[22] D. Goodhue, Development and measurement validity of a task-technology fit instrument for user evaluation of information systems, Decision Sciences 29 (1998) 105–138.

[23] E. Ammenwerth, C. Iller, C. Mahler, It-adoption and the interaction of task, technology and individuals: a fit framework and a case study, BMC Medical Informatics and Decision Making 6 (2006) 3.

[24] F.D. Davis, Perceived usefulness, perceived ease of use, and user acceptance of information technology, MIS Quarterly 13 (1989) 319–339.

[25] I. Ajzen, The theory of planned behavior, Organizational Behavior and Human Decision Processes 50 (1991) 179–211.

[26] P.M. Archambault, F. Légare, A. Lavoie, M.P. Gagnon, J. Lapointe, S. St-Jacques, J. Poitras, K. Aubin, S. Croteau, M. Pham-Dinh, Healthcare professionals’ intentions to use wiki-based reminders to promote best practices in trauma care: a survey protocol, Implementation Science 5 (2010) 45.

[27] C. Paglian, Design and evaluation in eHealth: challenges and implications for an interdisciplinary field, Journal of Medical Internet Research 9 (2007) e15.

[28] J.G. Anderson, Social, ethical and legal barriers to e-health, International Journal of Medical Informatics 76 (2007) 480–483.

[29] W.J. Winkelman, K.J. Leonard, Overcoming structural constraints to patient utilization of electronic medical records: a critical review and proposal for an evaluation framework, JAMIA 11 (2004) 151–161.

[30] B.E. Dixon, A. Zafar, J.M. Overhage, A framework for evaluating the costs, effort, and value of nationwide health information exchange, JAMIA 17 (2010) 295–301.

[31] T. Greenhalgh, D. Swinglehurst, Studying technology use as social practice: the untapped potential of ethnography, BMC Medicine 9 (2011) 45.

[32] K. Häyrinen, K. Saranto, P. Nykänen, Definition, structure, content, use and impacts of electronic health records: a review of the research literature, International Journal of Medical Informatics 77 (2008) 291–304.

[33] D.S. Bell, S. Cretin, R.S. Marken, A.B. Landman, A conceptual framework for evaluating outpatient electronic prescribing systems based on their functional capabilities, JAMIA 11 (2004) 60–70.

[34] J.I Westbrook, J. Braithwaite, K. Gibson, R. Paoloni, J. Callen, A. Georgiou, et al., Use of information and communication technologies to support effective work practice innovation in the health sector: a multi-site study, BMC Health Services Research 9 (2009) 201.

[35] G. Pare, Enhancing the rigor of qualitative research: applications of a case methodology to build theories of IT implementation, The Qualitative Report December (2002) 7.

[36] C.M. Cusack, C.M. Byrne, J.M. Hook, J. McGowan, E. Poon, A. Zafar, Health Information Technology Evaluation Toolkit, AHRQ AHRQ Publication No. 09-0083EF, Rockville, 2009.

[37] R. Amarasingham, L. Plantinga, M. Diener-West, D.J. Gaskin, N.R. Powe, Clinical information technologies and inpatient outcomes: a multiple hospital study, Archives of Internal

Medicine 169 (2009) 108–114.

[38] K. Kawamoto, C.A. Houlihan, E.A. Balas, D.F. Lobach, Improving clinical practice using clinical decision support

a l i n

[46] A. Wright, D. Sittig, J.D. Carpenter, M.A. Krall, J.E. Pang, B. Middleton, Order sets in computerized physician order entry systems: an analysis of seven sites, in: AMIA 2010

i n t e r n a t i o n a l j o u r n a l o f m e d i c

systems: a systematic review of trials to identify features critical to success, British Medical Journal 330 (7494) (2005) 765.

[39] A.X. Garg, N.K.J. Adhikar, H. McDonald, M.P. Rosas-Arellano, P.J. Devereaux, J. Beyene, J. Sam, R.B. Haynes, Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review, JAMA 293 (10) (2005) 1223–1238.

[40] Centers for Medicare and Medicaid Services, Physician Self-referral Overview, Centers for Medicare and Medicaid Services, February 23, 2011, available at: http://www.cms.gov/PhysicianSelfReferral/, cited 18/6/2011.

[41] L.M. Kern, R. Dhopeshwarkar, Y. Barron, A. Wilcox, H. Pincus, R. Kaushal, Measuring the effects of health information technology on quality of care: a novel set of proposed metrics for electronic quality reporting, The Joint Commission Journal on Quality and Patient Safety 35 (2009)

359–368.

[42] S.P. Robbings, Organizational Behavior: Concepts, Controversies, Applications, 7th edition, Prentice Hall, Englewood Cliffs, 1996, p. 230.

f o r m a t i c s 8 2 ( 2 0 1 3 ) e1–e13 e13

[43] A. Georgiou, G.A. Westbrook, J. Braithewaite, Time matters—a theoretical and empirical examination of the temporal landscape of a hospital pathology service and the impact of e-health, Social Science and Medicine 72 (2011) 1603–1610.

[44] B. Chaudhry, J. Wang, S. Wu, M. Maglione, W. Mojica, E. Roth, S.C. Morton, P.G. Shekelle, Systematic review: impact of health information technology on quality, efficiency, and cost of medical care, Annals of Internal Medicine 144 (10) (2006) 742–752.

[45] J.S. Ash, D.W. Bates, Factors and forces affecting EHR system adoption: report of a 2004 ACMI discussion, Journal of the American Medical Informatics Association 12 (2005) 8–12.

Symposium Proceeding, 2010, pp. 892–896.