For Archmage ONLY

Reasons cited for computerized provider order entry (CPOE) system adoption include medical error reduc- tion,1–3 and provision of just-in-time feedback to physi- cians,4–6 but use of CPOE is not widespread.7 Presumably, implementation lags because CPOE, by reputation, is hard to implement, expensive, and diffi- cult to coax clinicians (and especially physicians) to use. At organizations with successful CPOE implementa- tions, people instrumental in creating the success pos- sess valuable experiential knowledge. As part of a three- year research grant funded by the National Library of Medicine, a two-day consensus conference enticed experts on CPOE implementation to share their expert- ise through discussions and generation of recommenda- tions for CPOE adoption and usage to aid clinicians, vendors, hospital administrators, and information tech-

nology personnel in addressing the challenges that they will face. Computerized provider order entry was defined for the purpose of the conference as a process by which a clinician with order writing authority sits at a computer to directly enter patient care orders.

METHODS

Goals of the Conference

The conference convened to develop a consensus set of recommendations for CPOE implementation. A consen- sus statement is a document, developed and agreed upon by representatives of multiple perspectives, that provides guidance for practice in specific areas.8 An important aspect of consensus statement development is the focus on agreement and collaboration. Guidelines differ from consensus statements in that the former may be more prescriptive and use more elaborate branching and hierarchical structures. The process for developing consensus statements has been described well by Gamroth et al.8

A team of Oregon Health & Science University infor- matics investigators organized and supported the activ- ities of the two-day conference and analyzed the data. To achieve the goal of generating recommendations, such conferences typically follow a commonly used for- mat, with formal talks by experts followed by small group discussions on preselected topics.9,10 However,

229Journal of the American Medical Informatics Association Volume 10 Number 3 May / Jun 2003

JAMIAThe Practice of Informatics

Affiliations of the authors: Oregon Health & Science University, Portland, OR (JSA, PZS); Partners Healthcare System, Chestnut Hill, MA (GJK). This work was supported by grant LM06942-02 from the National Library of Medicine. Special thanks to Paul Tang, MD, and William Bria, MD, for their comments about the Considerations and to Lara Fournier, MS, for assistance with the manuscript. Correspondence and reprints: Joan S. Ash, PhD, Division of Medical Informatics and Outcomes Research, School of Medicine, Oregon Health & Science University, 3181 SW Sam Jackson Park Rd., Portland, OR 97201-3098; e-mail: [email protected]. Received for publication: 8/15/02; accepted for publication: 12/03/02.

Synthesis of Research Paper ■

A Consensus Statement on Considerations for a Successful CPOE Implementation

A b s t r a c t In May of 2001, thirteen experts on computerized provider order entry (CPOE) from around the world gathered at a 2-day conference to develop a consensus statement on successful CPOE implementa- tion. A qualitative research approach was used to generate and validate a list of categories and considerations to guide CPOE implementation.

■ J Am Med Inform Assoc. 2003;10:229–234. DOI 10.1197/jamia.M1204.

JOAN S. ASH, PHD, MLS, P. ZOË STAVRI, PHD, MLS, GILAD J. KUPERMAN, MD, PHD

the research team organizing this conference was expe- rienced in the use of qualitative methods, and did not want to restrict discussion to preconceived topics. The goal was to generate new insights as the experts shared their multiple perspectives on CPOE. Formal presenta- tions were replaced with prior reading, and the large group was encouraged to determine the direction of the discussion. A facilitator managed the large group dis- cussions. The research team designed an agenda that included a thinking task before participant arrival, brainstorming after arrival, narrative generation, priori- tization of issues, and small team assignments.

Preconference Activities

Six months before the conference, potential participants were identified through literature searches, citation analysis, and recommendations from known experts. Categories of attendees included administrators, clini- cians (physicians, nurses, pharmacists), social scientists, information technology implementers, and vendors (see Appendix for a list of participants and their affiliations). Conference organizers sought geographic diversity, rep- resentation of different system types (either commercial or locally developed), and varying implementation site characteristics (university, community, federal). Approximately half of those invited attended. Many attendees represented more than one of the stakeholder groups involved in CPOE implementation. Those who could not attend were asked if they would instead serve as an expert panel to review the final document.

Participants were asked to read a number of papers as background material to establish a shared knowledge base. The goals were to maximize on-site time for the sharing of expertise and experiences, to generate narra- tives, and to develop consensus statements.

Conference Activities

During the two-day conference, participants’ comments were captured on flip charts that were transcribed dur- ing the conference. Large group discussions were tape- recorded and later transcribed. Research team members were assigned to each of the small break-out groups to assist with taping and facilitation.

On the first conference day, the organizers described the purpose of the conference and introduced the experts and the research team members. A trained facilitator (who was neither a stakeholder nor, at that time, a cen- tral member of the research team) led subsequent activ- ities. Participants were asked to share their perceptions of the “one most important thing,” or success factor, that they had discovered during their own CPOE implemen- tations. Next, participants divided into four small teams, preassigned to promote diversity, which began discussion of the success factors outlined in the large group, prioritizing them and adding others with greater levels of detail. The four teams reported back to the large group. To further explore success factors and to share the lessons each expert had learned through expe- rience, a narrative exercise was devised. The value of

sharing lessons learned about CPOE is illustrated by Massaro’s description and analysis of the implementa- tion at the University of Virginia.11,12 Each participant was asked to tell both a success story and a failure story. This was an enjoyable way of generating descriptions of additional success factors within an organizational and temporal context. Tales of stumbles, missteps, or pitfalls rather than outright failures were also allowed.

The experts then continued to work in small groups to develop lists of success factors based on the two exercis- es. For example, one of the failure stories highlighted inadequate levels of readily available support during a pilot implementation; thus, follow-up discussion cen- tered on ideal levels of support. There were several iter- ations of large and small group work sessions. Thorny issues that provoked disagreement during any of these discussions were placed in a “bank.” The bank served as a way to put aside time-consuming and difficult issues and to deal with them later. Bank issues were discussed during an informal evening session and, after agree- ment, some were added to the list of considerations.

During the second day of the conference, the large group reviewed the work of the previous day and dis- cussed and prioritized the main discussion points. Plans were made for disseminating the results and gaining subsequent feedback about them.

Postconference Activities

The data collected (transcriptions of all large and small group interactions) were analyzed with a grounded the- ory approach, using the words of informants to develop patterns and themes.13 Instead of beginning the analysis with a predetermined list of topics, the researchers allow the data, in the form of words in transcripts, to guide code development. The process entails extracting major statements from all data sources and placing them on cards. The primary investigator led a multidiscipli- nary team of five through a 500 card sort exercise,14 iter- atively grouping similar ideas together. The sorting process created a taxonomy of high-level themes. A “themes document” was generated and shared with all participants, who engaged in three months of electronic discussion and changes. Experts who could not attend the conference were shown the resulting document and did not suggest any major content changes. The docu- ment seems to accurately reflect participating experts’ consensus on CPOE implementation.

Results

The Data

• There were approximately 11.5 hours of tape to be transcribed.

• Two hundred and forty-nine single-spaced pages of transcripts were generated.

• Approximately 12 of the 13 participants and seven research team members submitted comments in response to the initial mailing, and each resulted in modifications. The major change suggested by the

ASH ET AL., A Consensus Statement on Considerations for a Successful CPOE Implementation230

comments was that the considerations should be put into a question rather than a statement format. The purpose was to become less prescriptive.

• Two of the nonparticipant experts offered suggestions that were factored into the final list.

The group realized that guidelines or recipes for success would be difficult to create because “CPOE” means dif- ferent things and raises different concerns at different organizations—academic centers are different from community hospitals, and inpatient care is different from outpatient care. Cost reduction as an overarching goal raises different issues from the goal of patient safe- ty. A desire for high levels of decision support raises dif- ferent issues from a desire for a basic system. However, the experts also felt that despite such variations, certain themes were common across all CPOE projects and could be addressed. What the experts asked to have available as they first began thinking about CPOE was a menu of possibilities—a list of issues that they might not otherwise think about. Consequently, a list of “con- siderations” was targeted as the product of the confer- ence rather than a specific guideline or recipe.

The qualitative analysis began with the identification of approximately 500 major statements and ended with ten themes (subsequently condensed) that represent a list of the overarching issues that the experts believe must be considered before CPOE implementation. An overview of the considerations is presented here. Although the complete list included 144 subconsiderations with com- ments and amplification, the following list offers those derived from them by the authors of this paper. Please visit <www.cpoe.org> for the complete report.

The Considerations

Consideration 1: Motivation for Implementation

The motivation for implementing CPOE influences where funding will come from, who will provide politi- cal support, and who will provide clinical leadership. Implementers should consider the following issues:

• Whether local or national authorities may require CPOE at a future date

• If administrators and/or clinicians are pressing for CPOE adoption

• What stated objectives are for CPOE implementation (e.g., “improve efficiency”)

• If outsiders/external conditions are forcing CPOE adoption (e.g., competitors are doing it)

Consideration 2: CPOE Vision, Leadership, and Personnel

Successful CPOE implementations require effective leadership over extended time periods—in different forms and at multiple levels in the organization. Leadership is needed at the executive level to promote a shared vision and provide funding at the clinical level to ensure champions and buy-in and at the project man- agement level to make practical, effective, and useful

decisions. Before embarking on the serious undertaking of CPOE, organizations should determine whether the following conditions are met:

• Top-level leadership commits unwaveringly and vis- ibly to CPOE.

• A shared vision exists in the organization regarding the purpose of CPOE (e.g., to improve patient care) as well as a common understanding of why the current state is suboptimal and change is needed.

• A single, clearly identified CPOE project leader exists with realistic attitudes about what can/cannot be accomplished, knowledge of how to educate admin- istrators, and skills to foster teamwork.

• There are clearly stated anticipated benefits that staff can embrace.

• A realistic overall understanding exists of the efforts required to implement CPOE, coupled with the abili- ty to communicate the vision and articulate tangible objectives to all levels of the organization.

• The institution has identified and enrolled the sup- port of physician leaders and clinical champions, respected by their peers, who can communicate the shared vision.

• The organization has adequate finances, technical infrastructure, project management expertise, and staff readiness for CPOE—coupled with real and vis- ible commitment of the chief executive and financial officers.

• Clinical staff will trust and support the administra- tion through this difficult effort, and, conversely, the leadership will value, have faith in, and depend on the individual clinicians in the organization who will use CPOE.

• An organizational culture exists, or can be created, that values constructive feedback, changes made for quality improvement, and continuous learning—kept in balance by leadership that can tell the difference between clinicians’ requests for “what would be nice” versus “what is essential or critical for success.”

• The compelling enthusiasm and urgency for CPOE can sustain the organization through the hurdles of implementation—i.e., leaders will maintain momen- tum through communication of appropriate urgency.

• The stability and product quality of the vendor are at least good, if not excellent.

• There is deep understanding that CPOE projects are a vendor “marriage,” not a purchase.

Consideration 3: Costs

Financial considerations are of critical importance. Often, costs are underestimated because purchase of the software is only the beginning of financial outlays; other expenditures such as person-hours for training and sup- port are harder to predict. Decision makers need to con- sider the following issues:

• Whether the total cost of ownership has been consid- ered rather than simply the cost of technology

• Whether the organization can afford the attendant temporary productivity losses that accompany CPOE implementation

231Journal of the American Medical Informatics Association Volume 10 Number 3 May / Jun 2003

• Whether funds have been dedicated solely for CPOE, with the ability to commit additional funds quickly for good (unanticipated) cause

Consideration 4: Integration: Workflow, Health Care Processes

The manner in which a CPOE application alters and integrates into existing environments and workflows is critical to its success. Users resent disruption of their patient care activities; thus, implementers must consid- er the following issues:

• Whether the impact of CPOE on the work processes of physicians, nurses, pharmacists, ward clerks, labora- tory personnel, registration personnel, and other hos- pital staff was carefully considered and will be closely followed during implementation and afterward

• Whether an organization-wide change management strategy exists and has been tested under similar stresses previously

• Whether CPOE will be used for all orders or simply some categories of orders

• How users will view orders during construction, after entry, and after completion

• How new, potentially life-saving orders will be com- municated reliably to nurses or others who need to be aware of them

• How CPOE integrates with other hospital applica- tions such as the laboratory system, pharmacy system, ADT/registration system, and other clinical systems via interface engines and/or messaging protocols

• Whether the impact of CPOE on human communica- tion among key employees such as physicians, phar- macists, nurses, and lab technicians has been worked out for both regular use and during CPOE downtime

Consideration 5: Value to Users/Decision Support Systems

Constituencies affected by CPOE implementation (e.g. physicians, nurses, ancillary department personnel) must understand the CPOE implementation “value proposition,” i.e., they must do things differently but there will be some benefit in return. One benefit for cli- nicians is embedded CPOE decision support logic that helps to improve patient care quality and/or to reduce costs. Related issues include the following:

• There must be a plan for the ongoing management of clinical CPOE system content, including decision support.

• Users must participate in development of decision support and other “benefits” that affect them, and be adequately trained in their usage.

• Users must understand where the CPOE system pro- vides “help” and where it may not, and CPOE behav- ior must be consistent (e.g., for drug-drug interaction, drug-allergy interaction, duplicate medications, duplicate labs, expensive tests, suggested drug level monitoring).

• Order sets—groups of orders to manage a disease

state or for a procedure (prebronchoscopy, postbron- choscopy)—must be developed, reviewed, and main- tained for personal and/or departmental usage.

• Surveillance must be in place to determine that bene- fits for clinicians and for patients are both easy to see and to describe.

• Users must be trained before implementation and on an ongoing basis thereafter as CPOE systems evolve.

• Clinical users must be shown that CPOE usage is not clerical work, emphasizing what cannot be done via manual, paper systems.

Consideration 6: Project Management and Staging of Implementation

Project management dictates that implementation be completed in carefully planned stages. Key considera- tions include the following:

• During all stages, “people issues” must have highest priority—keep employees (clinicians) informed, engaged, and content, through planning and commu- nication.

• Early in the project, the scope must be defined, with clear, reasonable, measurable goals.

• Early milestones must be selected to produce “wins” that help maintain momentum toward more difficult long-term objectives.

• Plans should be detailed enough but not overly so • Multiple mechanisms for collecting feedback from

users and staff must be in place, and analyzed in real time for appropriate action.

• The golden rule should be applied by all involved (do unto others as you would have them do unto you), and leaders should work to develop consensus when disagreements arise (keeping in mind that various ways of doing things may all lead to success).

• Use of consultants should be carefully planned with specific objectives before they are employed (if at all).

• A critical mass of users must be ready for the imple- mentation.

• A plan for involving clinicians must be developed, followed, and evolved.

• Metrics for success should be determined beforehand and evaluated over time.

• Accountability for objectives, large and small, must be established and maintained, as each new concern arises.

• During the pre-implementation phase, the organiza- tion should develop a vision, locate funding, identify people who will lead the implementation, solicit involvement from key people, and exhibit strategic and tactical planning skills.

• During the implementation phase, the organization should hire staff, deploy staff where and when most needed, keep up staff morale, and use communica- tion, publicity, and personnel management skills effectively to maintain project momentum.

• After implementation, the organization should estab- lish maintenance routines, create an environment for ongoing system improvement, and provide manage- ment systems for the long term.

ASH ET AL., A Consensus Statement on Considerations for a Successful CPOE Implementation232

Consideration 7: Technology

Technical details to consider as part of a CPOE imple- mentation include strategic considerations, user consid- erations, task completion flexibility, and the quality of the application—from customizability to user friendli- ness. Questions related to the system itself include:

• Whether there is a plan to authorize all users who need access to the system (e.g., attending and house staff physicians, nurses, medical assistants and unit secretaries)

• Whether the system is sufficiently customizable for the organization’s needs, including the ability to pro- vide decision support where needed

• Whether the balance between customization and standardization has been considered

• Whether individual users can customize some things themselves

• If the system can be modified on site • What special considerations have been made for

replacing older systems • Whether assurance of high-level data quality is possi-

ble and has been implemented • Whether the CPOE system can interface with existing

and planned future systems • If a risk analysis of the project has been conducted,

with specific attention given to addressing the risks • If there is a need for remote access • How great a burden system use places on end-users

from the users’ point of view • Whether the response time is good enough for the

users (one expert cited 0.7 seconds as too slow) • If there are escape routes, such as entering free text,

for frustrated users • If details of the user interface (UI) have been scruti-

nized, focusing on aspects of the UI that are most like- ly to give the users difficulty

Consideration 8: Training and Support 24 � 7

One of the constant themes identified by the experts at the retreat was the importance of live help available “at the elbow” at the time of implementation. In addition to the symbolic importance of supporting the users by being present while they are first using the application, intensive support at “go-live” time allows the imple- mentation team to have direct experience with what is and what is not working well. Most successful imple- mentations have had more post-go-live support than pre-go-live training. Most sites have had 24 � 7 support for at least several days post go-live. Considerations include

• Whether there is a training plan for the support staff • If support staff are able to act as translators between

clinicians and information technology staff • Whether provisions have been made for online help

as well as direct assistance by support staff • If users will train and mentor other users (and with

what methods)

Consideration 9: Learning/Evaluation/Improvement

CPOE implementation is an ongoing effort that benefits from continuous improvement. It is important that mechanisms for feedback and modification of the sys- tem be in place. Questions to consider include

• How the organization can learn from its mistakes • Whether there is a process for responding to prob-

lems in a timely manner • How problems will be addressed in a timely manner • How the system will be “test piloted” without put-

ting patients at risk • What the plan is for formal feedback and evaluation • How the system will be improved upon continuously • Whether a plan exists to revisit decisions on a regular

basis

Discussion The group used a qualitative approach to create a con- sensus statement on the specific issues that organiza- tions contemplating a CPOE implementation face. Each consideration should be reviewed by the leadership and implementation team of any organization considering CPOE installation. Some issues will be more easily addressed than others; some will be more relevant to one particular organization than others; and, some are more applicable at different stages in the implementa- tion than others. Some of these questions and issues will have clear and obvious answers, but most will not and will require effort to address. Organizational represen- tatives should focus on the difficult-to-answer questions rather than avoiding them. All of the detailed consider- ations listed are relevant to a successful implementation.

The qualitative approach allowed us to generate differ- ent results than might have been realized with a more prescriptive approach to consensus creation. Of the major categories of considerations, only one grouping was strictly technical. It is possible that the discussion would have centered more on technology if the induc- tive approach had not been taken. The importance of strong executive leadership at the highest levels in the organization in a CPOE initiative should not be under- estimated. Leadership is a thread running through many of the major considerations. Administrative lead- ers, acting on behalf of the organization, must believe viscerally that CPOE is in the best interest of the institu- tion and be able to communicate that feeling throughout the organization.

Clinical leadership must also be committed to CPOE and communicate this commitment to the clinical staff, who will typically be less than excited about the prospect of CPOE (due to natural resistance to change). Clinical staff will have real concerns that the time to complete work will increase. Arguments about improved safety may appear to end-users as vague and intangible. Clinical leaders must work strenuously to communicate (and physically demonstrate) to their

233Journal of the American Medical Informatics Association Volume 10 Number 3 May / Jun 2003

staffs how CPOE provides opportunities for improved quality and efficiency. Administrative and clinical lead- ers must work together to create a strong sense of “common will” to overcome obstacles that will be encountered during a CPOE implementation.

The expert panel has continued its dialogue about CPOE. It gathered informally during the American Medical Informatics Association 2001 and 2002 Annual Symposia to plan further endeavors. Ongoing efforts will develop suggestions to help organizations find answers to the questions listed under each considera- tion. There is agreement that future research needs to be done to develop valid tools to measure readiness for CPOE, the effectiveness of the process during imple- mentation, and outcomes during and after CPOE implementation.

The list of considerations presented in this paper and the full text are available at <www.cpoe.org>. This is meant to serve as a guide for organizations to help them make appropriate decisions regarding CPOE. The group agreed that implementation of CPOE is difficult and it must be approached with awareness of the potential problems. The combined wisdom of those who have already experienced successful implementation efforts, summarized in the list of considerations, can serve as a resource for those contemplating future implementa- tions. Health care institutions are being pressured by the Leapfrog Group3 and others to rapidly adopt CPOE. To increase chances of success, leaders are urged to look carefully at the Considerations before they “leap” into CPOE projects.

References ■

1. Hurtando MP, Swift EK, Corrigan JM (eds). Crossing the Quality Chasm: A New Health System for the 21st Century. Institute of Medicine, Committee on the National Quality Report on Health Care Delivery. Washington, DC: National Academy Press, 2001.

2. Sittig DF, Stead WW. Computer-based physician order entry: The state of the art. J Am Med Inform Assoc. 1994;1:108–23.

3. The Leapfroggroup for Patient Safety. Computer physician order entry (CPOE) factsheet. November 2000. Available at <www.leapfroggroup.org>.

4. Kaushal R, Bates DW. Computerized physician order entry (CPOE) with clinical decision support systems (CDSSs). Chapter 6 in Making Health Care Safer: A Critical Analysis of Patient Safety Practices, Evidence Report/Technology Assessment no. 43, Agency for Health Care Quality and Research, 2001. Available at <www.ahrq.gov/clinic/ptsafety/ chap6.htm>.

5. Shea S, DuMouchel W, Bahamonde L. A meta-analysis of 16 randomized controlled trials to evaluate computer-based clin- ical reminder systems for preventive care in the ambulatory setting. J Am Med Inform Assoc 1996;3:399–409.

6. Davenport TH, Glaser J. Just-in time delivery comes to knowl-

edge management. Harv Bus Rev 2002;80:107–111. 7. Ash JS, Gorman PN, Hersh WR. Physician order entry in U.S.

hospitals. J Am Med Inform Assoc Symposium Suppl 1998 ;235–239.

8. Gamroth L, Semradek J, et al. (eds). Enhancing Autonomy in Long-Term Care. New York, Springer, 1995.

9. Yasnoff WA, Overhage JM, Humphreys BL, LaVenture M. A national agenda for public health informatics: Summarized recommendations from the 2001 AMIA Spring Congress. J Am Med Inform Assoc 2001;8:535–545.

10. Stroup DF, et al. Meta-analysis of observational studies in epi- demiology: A proposal for reporting. JAMA 2000;283:2008– 2012.

11. Massaro TA. Introducing physician order entry at a major aca- demic medical center. I: Impact on organizational culture and behavior. Acad Med 1993;68:20–5.

12. Massaro TA. Introducing physician order entry at a major aca- demic medical center: II. Impact on medical education. Acad Med 1993;68:25–30.

13. Glaser BG, Strauss AL. The Discovery of Grounded Theory: Strategies for Qualitative Research. Chicago, Aldine, 1967.

14. Lincoln YS, Guba EG. Naturalistic Inquiry. Newbury Park, CA, Sage, 1985, pp 346–347.

Appendix Consensus conference participants, their current affilia- tions, and their roles: Jos Aarts (Erasmus University, The Netherlands, social science and international perspec- tive), Marilyn Davis (El Camino Hospital, Mountain View, CA, nurse and implementation specialist at the first POE site), Dick Gibson (Providence Health Systems, Portland, OR, physician and clinician leader), Homer Chin (Kaiser Permanente Northwest, Portland, OR, physician and clinician leader), Paul Nichol (Puget Sound Veterans Administration, Seattle, WA, physician and clinician leader), Marc Overhage (Regenstrief Institute, Indianapolis, IN, physician and clinician leader), Tom Payne (University of Washington, Seattle, WA, physician and clinician leader), Karen Hughart (Vanderbilt University, Nashville, TN, nurse and imple- mentation specialist), Janet Greenman (IDX, Seattle, WA, vendor representative), John Dulcey, MD (Lansdale, PA, physician and clinical systems consult- ant), Gil Kuperman (Partners Healthcare System, Chestnut Hill, MA, director of research and develop- ment), Brian Churchill (Peacehealth, Eugene, OR, nurse and project manager for POE implementation), and Jim Carpenter (Legacy Healthcare, Portland, OR, clinical pharmacist).

Research team members included Joan Ash, P. Zoë Stavri, Paul Gorman, William Hersh, Mary Lavelle, Lara Fournier, and Jason Lyman.

Views expressed by participants are their own and not necessarily those of agencies or organizations with which they are affiliated.

ASH ET AL., A Consensus Statement on Considerations for a Successful CPOE Implementation234