HIM 301 Introduction to Health Informatics WK4-Case Study

Chapter 21 Improving the User Experience for Health Information Technology Products

Nancy Staggers

sability has a strong, often direct relationship with clinical productivity, error rates, user fatigue and user satisfaction—critical factors for EMR adoption.

Healthcare Information and Management Systems Society, 2009

Objectives

At the completion of this chapter the reader will be prepared to:

1.Compare and contrast the terms user experience, human factors, ergonomics, human–computer interaction, and usability

2.Describe the goals of usability and user-centered design

3.Identify the major components to consider in human–computer interaction and usability studies

4.Analyze methods for conducting usability studies and relate them to a specific purpose of a usability study

5.Outline components of four different usability tests related to their position in the systems life cycle

6.Explain the basic steps in conducting a usability test on a healthcare product

Key Terms

Discount usability methods, 340

Ergonomics, 337

Focused ethnographies, 342

Heuristic evaluations, 340

Human factors, 337

Human–computer interaction (HCI), 338

Joint cognitive systems, 339

Sociotechnical system, 335

Task analysis, 341

Think-aloud protocol, 341

Usability, 338

User-centered design, 338

User interface, 336

Abstract

Many health information technology (health IT) products are not designed for their intended users and are inefficient and even dangerous. Federal agencies are responding to this challenge by developing regulations and reports to guide improved usability. Clinicians and informaticists require a suite of skills to understand concepts about the user experience and to conduct usability tests. This chapter provides the knowledge and skills to meet those needs. First, the chapter outlines the need for attention to the user experience and how it might benefit individuals and organizations. Terms are defined, the concepts of usability goals are presented, and user-centered design precepts are discussed. Available human–computer interaction (HCI) frameworks are listed; one framework is explained in detail and traditional usability methods are offered. Then types of usability tests are discussed and linked to the systems life cycle using the HCI framework. Examples of actual usability studies in health settings are provided. After reading this chapter students will be able to design and conduct usability tests to determine the effectiveness and efficiency of and satisfaction with health IT products.

Introduction to Improving the User Experience

While most students may be able to describe their own frustrations with poorly designed products, the systematic study of human factors and usability, plus the wide variety of resources supporting the development of usable systems, is likely new information. This section of the chapter begins by outlining users' current experience with health IT as well as the benefits that improved usability can provide to users and organizations as documented in the literature. This content is followed by the definitions of common terms and an explanation of their interrelationships.

The Current User Experience with Health IT Products

Issues with the usability of health IT products are now a critical national concern, especially because of the vast expansion of funding for and implementation of electronic health records (EHRs) in the U.S. Health IT involves complex interactions among multiple users, products, and environments, all with varying characteristics, commonly called a sociotechnical system. The language for Meaningful Use Stage 2 now includes usability assessments yet suboptimal user experience with current health IT products is well documented. Stead and Lin evaluated premier EHRs in the U.S. in 2009 and found that even the best EHRs had symptoms of poor usability. Physicians had difficulty finding critical information and developing the “big picture” of the patient. These authors claimed that health IT could decrease the quality of healthcare unless steps were taken to improve health IT support for clinicians' work and thought processes.1 Although their study concentrated on physicians, the results can be extrapolated to the entire health team's interactions with health IT.

The professional literature is rich with evidence demonstrating poor and even dangerous user experience with health IT products. For example, unintended consequences occurred because of EHR usability issues.2,3 In 2011 hospital staff transplanted an infected kidney after failing in at least 12 opportunities to discover a donor's positive hepatitis C result buried in the EHR.4 Potential patient safety issues occurred with an Electronic Medication Administration Record (eMAR) because nurses could not view patients' medications easily to determine missed and due medications.5 New errors emerged when computerized provider order entry (CPOE) facilitated 22 types of medication error risks.6 An ambulatory application serving more than 9 million patients failed to support clinicians during patient encounters, did not allow them to obtain situation awareness of the patient (the “big picture” of the patient), promoted workarounds with nonintegrated systems, and greatly increased frustrations because of required structured documentation.7 This application affected healthcare provider productivity and effectiveness, decreasing patient access worldwide.8,9 These issues could be prevented by incorporating known usability principles and processes to improve the user experience with health IT.

Potential Benefits of Improving the User Experience

A white paper from the Healthcare Information Management and Systems Society (HIMSS), co-led by this author, describes methods for incorporating usability in health organizations.10 This publicly available white paper includes a section on the benefits of usability to healthcare that is summarized here. The return on investment (ROI) was addressed largely outside healthcare in earlier publications as that community documented benefits and then moved on to address them.

Usability can add value to organizations across a range of areas. Usability ROI material is available from (1) nonhealthcare projects such as Bias and Mayhew11 and Nielsen,12 (2) the Usability Professionals' Association website,13 and (3) Dey Alexander Consulting.14 To the author's knowledge no research is yet available about ROI or cost savings for usability efforts in healthcare projects. Thus material is cited from nonhealthcare applications. However, findings from nonhealthcare IT projects are very likely to extend to healthcare because of the often dramatic changes that usability can create. Figure 21-1 outlines potential areas of value when the user experience is improved in health organizations.

Increased Individual Effectiveness

Usability can positively affect at least three areas of particular interest in health IT:

1.Increased user productivity and efficiency

2.Decreased user errors and increased safety

3.Improved cognitive support

Increased User Productivity and Efficiency.

One of the most prevalent complaints about health IT in general and EHRs specifically is that the technology impedes users' productivity. For example, outpatient visits were reduced from four to three per hour after an ambulatory EHR was fielded.8 A cognitive work analysis of the same system in a laboratory setting showed a large number of average steps to complete common tasks, a high average execution time, and a large percentage of required mental operators.15

Employing usability processes helps to improve productivity and efficiency. The Nielsen Norman Group estimated that “productivity gains from redesigning an intranet to improve usability are eight times larger than costs for a company with 1,000 employees; 20 times larger for a company with 10,000 employees; and 50 times larger for a company with 100,000 employees.”12(p5) Website redesign statistics for the 42 cases collected by Nielsen Norman yielded an average increase in user productivity of 161%. After testing intranets for low and high usability, these authors projected a savings of 48 hours per employee if intranets were redesigned for high usability. Souza cited usability research showing that two thirds of buyers failed in shopping attempts on well-known sites.16 Thus poor usability on intranets means poor employee productivity.

Decreased User Errors and Increased Safety.

One of the major reasons why health IT is installed is to reduce errors in healthcare.17,18 While clearly some classes of errors such as

FIG 21-1 The value of usability to health organizations.

(From HIMSS Usability Task Force. Promoting Usability in Health Organizations: Initial Steps and Progress toward a Healthcare Usability Maturity Model. Chicago, IL: Healthcare Information and Management Systems Society; 2011.)

medication errors can be reduced with health IT, technology can create unintended consequences and new errors due to usability.3,19,20 For example, Kushniruk and colleagues were able to identify how certain types of usability problems were related to errors as physicians entered prescriptions into handheld devices.21

Nielsen and Levy collected case studies and found a decrease in user error rates in 46 redesign projects measuring user error.22 A study showed a 25% decrease in user errors after screen redesign.23 Users found needed information only 42% of the time on 15 large commercial websites, even when they were directed to the correct home page; 62% gave up looking for desired items on websites.12 Redesigns could prevent errors in these types of interactions; thus incorporating usability can potentially decrease errors in health IT products.

Improved Cognitive Support.

Stead and Lin concluded that the premier systems in the U.S. did not provide the required cognitive support for clinicians (i.e., tools for thinking about and solving health problems).1 Cognitive support may include designs to provide an overview or summary of the patient, information “at a glance,” intuitive designs, and tailored support for clinicians in specific contexts. An example of how usability can provide cognitive support is the work on novel physiologic monitoring designs. Researchers employed user-centered design and usability testing techniques to create novel designs integrating physiologic data in a graphic object.24,25,26 The new design provided integrated, “at a glance” pictorial data to show changes to clinicians. These graphic objects are now being incorporated in vendors' products as an adjunct to numeric data displays.

Increased Organizational Efficiencies

Well-designed user interfaces and systems translate into organizational efficiencies, including the following:

1.Decreased maintenance costs

2.Decreased customer and individual training and support costs

3.Decreased development time and costs

Decreased Maintenance Costs.

Eighty percent of software life cycle costs occur in the maintenance phase and are related to unmet user requirements and similar usability problems.27 Usability experts estimated that by correcting usability problems early in the design phase of a project, two different American airlines projects reduced the cost of those fixes by 60% and 90%.28 At IBM, researchers concluded that it is more economical to consider users' needs early in the design cycle than to solve them later.29

Decreased Customer and Individual Training and Support Costs.

A study by Microsoft showed that time for support calls “dropped dramatically” after a redesign of the print merge function in Word.28 Business analysts found that a well-designed user interface had an internal rate of return of 32%, realized through a 35% reduction in training, a 30% reduction in supervisory time, and improved productivity.30 Logically, a well-designed user interface will require fewer resources to support, less time and effort in training, and decreased time on support calls. Souza also cited a web redesign at lucy.com that resulted in a 20% reduction in support calls.16

Decreased Development Time and Costs.

According to Marcus, the rule of thumb in many usability-aware organizations is that the cost–benefit ratio for usability is $1:$10:$100.31 Once a system is in development, correcting a problem costs 10 times as much as fixing the same problem during design. If the system has been released, it costs 100 times as much relative to fixing the problem in design. This estimate is frequently quoted and while it may be overly optimistic, its main point is clear: It is far more expensive in time, costs, and effort to correct issues later in the development life cycle than to complete an informed design at the beginning of a project.

Best practices in usability engineering could alleviate major reasons for inaccurate cost estimates by managers in these areas: frequent requests for changes by users, overlooked tasks, users' lack of understanding of their own tasks, and insufficient communication and understanding between users and analysts.28 By including usability techniques, two companies reduced time spent on development, one by 40%28 and another by 33% to 50%.32 An ROI analysis by Karat indicated a $10 return on each dollar invested in usability.33 According to Landauer, when usability is factored in at the beginning of a project, efficiency improvements can be greater than 700%.34 On a national level Landauer estimated in 1995 that the inadequate use of usability engineering methods in software development projects cost the U.S. economy about $30 billion per year in lost productivity. Of course, the cost would be even more substantial now.

Incorporating usability into health IT provides significant value to all such projects; therefore it is essential that healthcare team members, informaticists, and IT staff understand and apply usability principles and processes in their work. This chapter provides students with an understanding of the critical nature of usability as well as the knowledge and skills to conduct usability tests.

Definitions of Terms and Their Relationships

Despite discussion in the literature for more than 30 years, precise definitions for user experience terms are a source of debate and overlapping concepts. The relationship of terms is depicted in Figure 21-2. As demonstrated in the figure, user experience is the most inclusive of these terms, with human factors, ergonomics, human–computer interaction (HCI), and usability embedded within it. Usability and ergonomics overlap and intersect human factors and HCI. These terms overlap in conceptual definitions and also because the physical attributes of ergonomics may be combined with software, which is more the purview of usability and HCI.

User Experience

The term user experience encompasses all aspects of users' interactions.35 International Organization for Standardization (ISO) 9241-11 defines the term as “a person's perceptions and responses that result from the use or anticipated use of a product, system or service.”36 Schaffer indicates that user experience is concerned with a range of experiences from walking into a bank to designs that fit into complex ecosystems with many users interacting.37 To achieve a

FIG 21-2 Terms and their relationships.

(Adapted and expanded from Staggers N. Human factors: imperative concepts for information systems in critical care. AACN Clin Issues. 2003;14(3):310-319; quiz 397-318.)

high-quality user experience a seamless merging must occur from the talents of multiple disciplines, such as engineering, graphic and industrial design, interface design, psychology, and domain experts in the discipline at hand.10,38

Human Factors

According to the Human Factors and Ergonomics Society (HFES), human factors is “the scientific discipline concerned with the understanding of interactions among humans and other elements of a system, and the profession that applies theory, principles, data and methods to design in order to optimize human well-being and overall system performance.”39 Simple examples include how to open a door efficiently, how to turn on the lighting for one area of a room from a bank of light switches, and how to safely operate the controls to drive a car. In healthcare, human factors might concern the design of a new operating room to better support teamwork and patient flow. For additional information on this concept, see Donald Norman's classic book about the design of everyday objects.40

Ergonomics

The term ergonomics is used interchangeably with human factors by the HFES in Europe but in the U.S. and other countries its focus is on human performance with physical characteristics of tools, systems, and machines.41 For example, ergonomics issues might address the design of a power drill to fit a human hand or the design of chairs to promote comfort and safety. In healthcare the number, types, and locations of workstations are typically the purview of ergonomics. The design of a surgical instrument to fit the human hand and perform desired functions effectively and efficiently deals with ergonomics.

Human–Computer Interaction

Human–computer interaction (HCI) is the study of how people design, implement, and evaluate interactive computer systems in the context of users' tasks and work.42 As with human factors, HCI draws on the disciplines of psychology and cognitive science, computer science, sociology, and information science and on the discipline of the user at hand. HCI can be addressed throughout the systems life cycle to include the design, development, purchase, implementation, and evaluation of applications. HCI topics can include the following:

•The design and use of devices such as an intravenous pump or a touchpad on a computer

•User satisfaction with CPOE

•Patient usage rates of personal health records

•Users' perceptions of electronic medication records

•The standardization (or not) and meaning of icons on a health web application

•Principles of effective screen design, including mobile application design

•Issues with social media websites, such as the implications of “defriending” a person

•Analysis of the capabilities and limitations of users

Usability

The term usability is often used interchangeably with HCI when the product is a computer but usability also concerns products beyond computers. Usability is also more focused on interactions within a specific context or environment for a specific product. Formally, the ISO defines usability as the extent to which a product can be used by specific users in a specific context to achieve specific goals with effectiveness, efficiency, and satisfaction.36 A product with good usability allows users in a particular context to achieve their goals when interacting with a product.42 Usability is, however, fundamentally concerned with human performance rather than only subjective data. Usability can include the following dimensions:

•Speed and accuracy with a product

•Ease of learning a product and remembering interactions after time has elapsed

•User satisfaction or perceptions about the interactions with a product

•Efficiency and accuracy of interactions

•Designs to promote error-free or error-forgiving products

•Seamless fit of an information system to the tasks and goals of users

The Goals of Usability

The overall goals of usability have been established by the ISO. Figure 21-3 depicts the ISO usability goals.

Effectiveness is the accuracy and completeness with which specified users achieve specified goals in particular environments, including worker and consumer or patient safety (ISO 9126-1).

Efficiency includes the resources expended in relation to the accuracy and completeness of goals achieved.

FIG 21-3 Usability goals.

(From HIMSS Usability Task Force. Promoting Usability in Health Organizations: Initial Steps and Progress toward a Healthcare Usability Maturity Model. Chicago, IL: Healthcare Information and Management Systems Society; 2011.)

Satisfaction is the comfort and acceptability of users and other people to the product or work system and deals with users' perceptions.10 Dimensions of usability correlate to elements depicted in Figure 21-1 to include improvements for individuals or groups of individuals: productivity and efficiency, effectiveness in product use, safety, and cognitive support (an aspect of effectiveness).

User-Centered Design

User experience experts insist on a process of user-centered design that comprises the following three axioms:

•An early and central focus on users in the design and development of products

•Iterative design

•Systematic measures of the interactions between users and products41,42

These principles were derived nearly 30 years ago by Gould and Lewis and are more salient than ever because contemporary environments are filled with an array of complex tools. An early and central focus on users means understanding users in depth (i.e., their characteristics, environment, and tasks).42 Direct contact with users is needed early and often throughout a design or redesign process. Iterative design means having rounds of design and allowing key users to evaluate product prototypes to determine their effectiveness and efficiency in the care process and health decisions. One design is never adequate and typically three rounds are necessary. Once a design is available, even on paper or in PowerPoint, designers or informaticists work with users to determine any issues by having them interact with the initial design. Specific methods to accomplish this are explained in subsequent sections. The important point is that major usability issues are identified and corrected early in the process. Design and evaluation then occur in a cycle until all major usability issues are corrected. This dynamic, iterative process, which includes the three axioms listed above, is known as user-centered design. Importantly, structured and systematic observations, including identified measures, are necessary. Usability goals and axioms apply not only to developing products but also to the selection, purchase, customization, and redesign of products.

This kind of design allows us to integrate health data, information, and knowledge into health IT products. For example, a well-designed eMAR would filter medication routes to match the particular medication, eliminating options not appropriate (e.g., charting an antacid as an intravenous medication). Although this sounds like common sense, current designs in major EHRs often do not accommodate this kind of filtering.

Human–Computer Interaction Frameworks for Health Informatics

Frameworks provide guidance for understanding essential components that improve the user experience. They are helpful in completing user-centered design processes, usability tests, IT adoption evaluations, and usability research. This section of the chapter provides an overview of existing frameworks and describes in detail the Health Human–Computer Interaction Framework.

Human–Computer Interaction Frameworks

Various HCI frameworks and models with different foci are available.43–50

•FITT (Fit between Individuals, Task, and Technology), with the elements connected by interactions and influences43

•UFuRT (User, Function, Representation, and Task analyses). System knowledge is distributed across multiple users who have differences in expertise and cognitive characteristics. It includes a task analysis portion to describe steps in tasks and interactions.50 Recently this framework was renamed TURF.

•A framework for employing usability methods to redesign a fielded system47

•Framework for technology-induced error44

•A combined health IT adoption and HCI model45

•Joint cognitive systems46

The last item bears more discussion. Hollnagel and Woods coined the term cognitive systems engineering, acknowledging that sociotechnical systems, or complex technologies embedded within social systems, are increasingly prevalent yet have frequent system failures.46 The authors devised a cyclic model called contextual control model, or CoCom, with the following elements: event, modifies, constructs, determines, acts, and produces. Users and context are major components of the model. Importantly, joint cognitive systems imply that information is shared or distributed among humans and technology. This framework is useful for examining teamwork in healthcare where team members work together on patient care.

An analysis of the existing frameworks found each helpful but inadequate for health usability studies. Missing elements across frameworks included (1) interactions among disparate users, (2) characteristics and actions of products and users, (3) context, and (4) a developmental timeline. Context is critical in particular because it defines the kinds of users, tasks, and work design.45,48 A developmental time element is also necessary because it accounts for users changing (maturing) in their interactions over time.45,48 Therefore a new framework was created.

The Health Human–Computer Interaction Framework

The Health Human–Computer Interaction (HHCI) Framework is described here. The current framework builds on early work by Staggers and Parks describing nurse–computer interaction.51,52 It was expanded in 2001 to include groups of healthcare providers and interactions with patients.48,53 The framework is adapted further here to acknowledge that IT may be only one example of an available health IT product (e.g., physiologic monitors or intravenous pumps).

The elements of the framework are outlined in Figure 21-4. Information (e.g., patient care, administrative or educational information) is the exchange mechanism. Interactions occur in a system of mutual influences where elements (e.g., individuals, health IT) act and respond based on specific characteristics. Context is paramount, with all interactions embedded within a context. This means that any outcomes of interactions are distinct, as they are defined by a context. The developmental timeline indicates that interactions change over time. Thus the outcomes of interactions are different based on when an interaction occurs in time.

Humans or products can initiate interactions. The information is processed through either the product or the humans according to characteristics. The recipient then reacts to the information; for example, a healthcare provider could read and respond to email from a patient or a product might process interactions after the “enter” key is pressed. Iterative cycles continue as humans behave and products act according to defined characteristics. Goals and planning are implicit within the tasks displayed in the framework.

Essential Components for Improving the User Experience

The important point in this section is that using a framework or model can greatly assist students to think comprehensively about health IT adoption and usability and the conduct of usability studies. Choose a framework to match the need at hand. Usability methods can then be applied as appropriate while ensuring that critical elements are under consideration. This idea is expanded in later sections to illustrate how the framework assists usability testing. In summary, product interactions are a complex part of a sociotechnical system. Critical elements to consider are as follows:

•A user or users and their characteristics

•Interactions

•Tasks (goals of tasks)

•Information

•Products and their characteristics

•Context

•How interactions mature over time (developmental timeline)

Selecting Methods to Improve the User Experience

Techniques to improve the user experience can be informal or formal, simple or complex, and employ a few individuals or a wide range of users. Students or researchers can design small projects or sophisticated studies by combining usability precepts with usability-specific or traditional research designs and methods such as quantitative, qualitative, or mixed methods. The type of usability study is dependent on the purpose of the project; when the assessment is targeted within the systems life cycle; the desired outcome of the project; and available resources, including time, people, and money. However, it uses the elements from the HHCI framework.

Usability projects can be done at any point in the systems life cycle from initial work (to identify usability issues; clarify requirements; and assess initial designs, technical prototypes, or simple computerized applications) to iterative development, product selection, product customization, or evaluation of the impact of a system after installation.54 The important point is that user experience experts recommend usability tests early and often.

FIG 21-4 Health Human–Computer Interaction Framework.

(Copyright Nancy Staggers. Reprinted with permission.)

User experience methods were developed over decades of work and are robust. This section of the chapter concentrates on proven methods to choose and apply. These include discount usability methods and the more traditional usability methods described below.

Discount Usability Methods

Nielsen developed techniques he called discount usability methods to reduce the number of required users in usability projects and to use early design prototypes. Initially meant for user experience experts, these methods have proven useful for others involved in designing projects.55,56 These methods offer economies of time, effort, and cost and can be completed at any point in the systems life cycle. Two common techniques are heuristic evaluation and think-aloud protocol.

Heuristic Evaluation

The definition of a heuristic is a “rule of thumb” or guideline. Heuristic evaluations compare products against accepted usability guidelines to reveal major and minor usability issues. It is a commonly employed technique and students can complete a heuristic evaluation after only a modest amount of training.

A number of usability heuristics are available to evaluate applications:

•Nielsen57

•Zhang et al.'s 14 heuristics58

•Dix et al.'s 10 heuristics41

•Shneiderman's “eight golden rules”59

•HIMSS's nine usability principles60

Zhang et al.'s guidelines have been used more extensively in health applications and devices. These authors combined Nielsen's and Shneiderman's heuristics and applied them to a project evaluating two infusion pumps, finding 192 and 121 heuristic violations, respectively, categorized into 89 and 52 usability problems. Using this technique they concluded that pump 1 might contribute to more medical errors than pump 2.58 Zhang et al.'s 14 adapted heuristics and definitions are outlined in Table 21-1. Once students understand the meaning of each heuristic, they can evaluate a health IT product against the heuristics as in the following example.

An Example of a Heuristic Evaluation Project.

Guo and colleagues used Zhang's heuristics to evaluate a vendor's eMAR installed at a tertiary care center.5 The authors received training on the eMAR, defined typical tasks that nurses complete using the product, and also modified Zhang's heuristics to include concepts about patient safety. The authors independently completed the defined tasks and compared their interactions to the heuristics, synthesized results, and found 233 violations for 60 usability problems. Problems included having to manually update the screen by clicking an “as of” button to ensure that the most current medication orders were being viewed and nurses' difficulty in determining medications given “at a glance.” The results have implications across all three usability goals of effectiveness, efficiency, and satisfaction as well as point to potential patient safety issues.

Think-Aloud Protocol

Think-aloud protocol also involves a small number of users and has them talk aloud while they interact with a product. Users voice what they are trying to do, indicate where interactions are confusing, and provide other thoughts about the product during interactions. This allows a detailed examination of the specified tasks, in particular to uncover major effectiveness issues. This method may be used in the design, redesign, development, or evaluation of applications at any time in the systems life cycle. Think-aloud methods are often used in conjunction with other techniques.

With this technique researchers first determine a specific set of tasks for users to complete, such as tasks to operate an infusion pump. Defining tasks ahead of time provides structure and consistency across participants and guides users through the procedure. Participants are asked to complete the tasks and talk aloud during the session. Methods to capture the session can include observation with notes, audiotaping or videotaping, automatic capture of keystrokes using software such as Morae, and user and evaluator paper-and-pencil diaries or logs.41 The resulting material is then analyzed. Be aware that the analysis portion of this method can be time consuming depending on the complexity of the product, the number of users tested, and the number of tasks. However, the information gained by using this method is robust and helpful.

Traditional Usability Methods

A large suite of methods is available to guide examination of the user experience of health products and processes. The HHCI framework elements will guide assessments (thinking about different user skills levels, different contexts, etc.). Two of these methods, task analysis and focused ethnographies, are presented here.

Task Analysis

Task analysis is a generic term for a set of more than 100 techniques that range from a focus on cognitive tasks and processes (called cognitive task analysis) to observable user interactions with an application (e.g., a systematic mapping of team interactions during a patient code). Task analyses are systematic methods that are used to understand what users are doing or required to do with a product by focusing on tasks and behavioral actions of the users and products. These methods provide a process for learning about and documenting how ordinary users complete actions in a specific context.55,61,62 Task analyses are helpful to identify task completeness, the correct or existing incorrect sequencing of tasks, accuracy of actions, error recovery, and task allocation between humans and products. Task analysis can be used throughout the systems life cycle to determine user requirements for design and redesign or to identify usability issues for complex products. One type of analysis, cognitive task analysis, is particularly useful for understanding users' goals while interacting with products. These methods can be used, for example, to determine who is attending to patients' preventive health alerts in a clinic because alerts are seen by a variety of healthcare providers.

Methods of task analysis include the following:

•Interviews

•Observations

•Shadowing users at their actual work sites

•Observing users doing tasks

•Conducting ethnographic studies or interviews63

A critique of techniques is available for students who want to find the right method for their project.64 References on performing task analyses are available.61,65

Sample output from task analyses is listed in Table 21-2. After observations and interviews students will record user actions (e.g., a flow chart with task descriptions). Students might videotape users as they interact with an EHR, asking users to perform specific tasks and use a think-aloud protocol to uncover tasks (especially cognitive tasks) and requirements.

Example of a Task Analysis.

Staggers and Kobus videotaped nurses as they interacted with an existing eMAR in a military inpatient application.20 They then observed nurses' medication management tasks in the actual setting in a variety of acute care units. The researchers created a task flow diagram of medication tasks (including cognitive tasks) and delineated deficiencies with the current application. Using the researchers' findings, a novel and more effective eMAR was then developed.

TABLE 21-1 Nielsen-Shneiderman Heuristics and a Severity Rating Scheme

HEURISTIC CATEGORY

DEFINITION

Consistency and standards

Consistency across all aspects of the product: methods of navigation, messages and actions, meaning of buttons, terms and icons. Congruence with known screen design principles for color and screen layout. Consistency with ISO usability guidelines.

Visibility of system state

Users understand what the system is doing and what they can do with the product from the system messages, information, and displays.

Match between system and world

The technology matches the way users think and do work, uses appropriate information flow, has typical options that users need, and includes expected actions by the system.

Minimalist

No superfluous information. System and screen design targeted to primary information users' need. Use of progressive disclosure to display details of a category of information only when needed. The exception can be designs for expert users where screen density is preferred.

Minimize memory load

Minimizing the amount of information and tasks users have to memorize to adequately use the technology. Product makes use of sample formats for data input such as a calendar for date format.

Informative feedback

The technology provides prompt and useful feedback about users' interactions and actions (e.g., feedback that orders were placed).

Flexibility and efficiency

The ability to tailor and customize to suit individuals' needs. Includes novice and expert capabilities (e.g., string searches).

Good error messages

Tell users what error occurred and how users can recover from the error. Not abstract or general such as “Forbidden!” Need to be precise and polite and not blame the user.

Prevent errors

Catastrophic errors must be prevented (e.g., pediatric medication order dosing mixed between kilograms and pounds, delivering a radiation dose with the device leaves wide open instead of being tailored to tumor size).

Clear closure

Users should know when a task is completed and all information is accepted. Displays should include progress toward 100% completion versus using a series of bars.

Reversible actions

Whenever possible, actions and interactions should be able to be undone within legal limits in electronic health records. If actions cannot be reversed, there is a consistent procedure for documenting the correction of any misinformation in the system.

Use the users' language

The technology uses language and terms the targeted users can comprehend and expect. Health terms are used appropriately.

Users in control

Users initiate actions versus having the perception that the technology is in control. Avoid surprising actions, ending up in unexpected places, and loud sounds with errors.

Help and documentation

Provide help for users within the context the actions occur (context-sensitive). Embed help functions throughout the application.

SEVERITY SCALE RATING ELEMENT

DEFINITION

0—No usability problem

No need to correct the issue.

1—Cosmetic problem

Correct the issue only if extra time and fiscal resources allow. Lowest priority.

2—Minor problem

Annoying issue with minor impact. Low priority to fix.

3—Major usability problem

Issue with major impact to use or training or both. Important to fix. Considerations are the numbers and kinds of users affected by a persistent problem.

4—Usability catastrophe

Severe issue that must be corrected before product release, especially those related to patient safety.

Data from Staggers N. The impact of screen density on clinical nurses' computer task performance and subjective screen satisfaction. Int J Man Mach Stud. 1993;39:775-792; Staggers N. Improving the usability of health informatics applications. In: Hebda T, Czar P, eds. Handbook of Informatics for Nurses and Health Professionals. Upper Saddle River, NJ: Pearson Education; 2012:170-193; and Zhang J, Johnson TR, Patel VL, Paige DL, Kubose T. Using usability heuristics to evaluate patient safety of medical devices. J Biomed Inform. 2003;36(1-2):23-30.

ISO, International Organization for Standardization.

Focused Ethnographies

Ethnography methods are borrowed from anthropology and sociology where fieldwork and analyses of people in cultural and social settings are completed. Focused ethnographies concentrate on individuals' points of view, their experiences and interactions in social settings, rather than on just the actions of those individuals.66 The researcher is an observer rather than a part of the society. During observations, detailed descriptions are generated with an emphasis on social relationships and their impact on work. Ethnographies have become important in understanding the user experience and to describe the impact of complex products.

TABLE 21-2 Sample Output from Task Analyses

TYPE OF TASK ANALYSIS OUTPUT

DESCRIPTION

Profiles of users

Short narrative, visual descriptions, and/or summaries about the characteristics of users

Workflow diagrams

A flow diagram of tasks or cognitive processes performed by users

Task sequences or hierarchies

Lists of tasks order by sequence or arranged to show interrelationships

Task scenarios

Detailed descriptions of events or incidents, including how users handle situations

Usability issues

A list and classification of usability problems with a product

Affinity diagrams

Bottom-up groupings of facts and issues about users, tasks, and environments to generate design ideas

Video and audiotape highlights

Clips that illustrate particular observations about users and tasks in a context

Adapted from Staggers N. Human–computer interaction. In: Englebardt S, Nelson R, eds. Information Technology in Health Care: An Interdisciplinary Approach. Philadelphia, PA: Harcourt Health Science Company; 2001:321-345.

Example of a Focused Ethnography.

Ash and colleagues used this method to research the impact of CPOE on users in acute care facilities in the United States.3,67,68,69 They completed interviews, focus groups, and observations and found unintended consequences for CPOE: new and more work, workflow issues, unusual system demands, disruptions in routine communications, extreme user emotions, and overdependence on the technology.

Usability Measures and Tests

Usability measurements are to user interface design what physical exams are to patient care. 70

A critical aspect of conducting a usability study is measuring human performance. To assist students, a taxonomy of usability measures is presented in Table 21-3. This table is adapted from Sweeney, Maguire, and Shackel and from Staggers and expanded here.48,53,71 The taxonomy includes measures from three perspectives: users, experts, and organizations. The important points are that usability is measurable and that a suite of measures is available. In addition to the objective measures in Table 21-3, questionnaires are available to measure users' perceptions of or satisfaction with their product interactions.

Usability Questionnaires

At least four questionnaires are available to measure user interaction or interface satisfaction:

•System Usability Scale (SUS)72,73

•Questionnaire for User Interaction Satisfaction (QUIS)74

•Purdue Usability Testing Questionnaire75

•Software Usability Measurement Inventory (SUMI)76

The SUS is considered an industry standard among user experience professionals and has been used widely on a variety of products outside health.72,73 The SUS is a publicly available, 10-item scale developed in 1986 by John Brooke at Digital Equipment Corporation.65

The QUIS has been used in numerous health informatics studies. Typically a computer system or application is assessed. Developed in the late 1990s, QUIS addresses users' overall perceptions of a product, including overall reaction, terminology, screen layout, learning, system capabilities, and other subscales such as multimedia applications. QUIS subscales can be mixed and matched to fit the application at hand. Participants can complete the QUIS in about 5 to 10 minutes. Reliability and validity assessments are available for this tool.

The Purdue Usability Testing Questionnaire has 100 open-ended questions about how features adhere to accepted guidelines. Students would need to be familiar with design guidelines before using this questionnaire. However, reliability and validity assessments of the questionnaire are not reported.

Less information is available about the SUMI, including its assessed reliability and validity. The instrument has three components: an overall assessment, a usability profile, and an item consensus analysis. The usability profile examines areas such as efficiency, helpfulness, control, and learnability. The consensus component addresses adherence to well-known design alternatives such as categorical ordering of data in a simple search task.

Selecting a Type of Usability Test

A key decision before beginning a usability assessment is determining the type of usability study to conduct in a specific case. This section expands on work by Rubin and Chisnell and uses the systems life cycle to organize the types of tests available.42

Determining User Needs and Requirements

At the beginning of the systems life cycle, during initial design or redesign process, informaticists determine user needs and requirements from the following:

•Users' characteristics

•Tasks (including cognitive tasks)

•Work design

•Interactions among workers and tasks and products

TABLE 21-3 Sample Usability Measures

USABILITY FOCUS

USABILITY MEASURES

User behaviors (performance)

Task times (speed, reaction times)

Percentage of tasks completed

Number, kinds of errors

Percentage of tasks completed accurately

Time, frequency spent on any one option

Number of hits and/or amount of time spent on a website

Training time

Eye tracking

Facial expressions

Breadth and depth of application usage in actual settings

Quality of completed tasks (e.g., quality of decisions)

Users' comments (think-aloud) as they interact with technology

System set-up or installation time, complexity of set-up

Model of tasks and user behaviors

Description of problems when interacting with an application

User behaviors (cognitive)

Description of or systems fit with cognitive information processing

Retention of application knowledge over time

Comprehension of system

Fit with workflow

User behaviors (perceptions)

Usability ratings of products

Perceptions about any aspect of technology (speed, effectiveness)

Comments during interviews

Questionnaires and rating responses (workload, satisfaction)

User behaviors (physiologic)

Heart rate

EEG

Galvanic skin response

Brain-evoked potentials

User behaviors (perceptions about physiologic reactions)

Perceptions about anxiety, stress

User behaviors (motivation)

Willingness to use system

Enthusiasm

Expert evaluations (performance)

Model predictions for task performance times, learning, ease of understanding

Observations of users as they use applications in a setting to determine fit with work

Expert evaluations (conformance to guidelines)

Level of adherence to guidelines, design criteria, usability principles (heuristic evaluation)

Expert evaluations (perception)

Ratings of technology, informal or formal comments

Context (organization)

Economic costs (increased FTEs for the help desk for a new application)

Number of support staff, time needed to support product

Number of training staff, time needed to support product

Costs (for support, training, loss of productivity)

Observations about the fit with work design and workflow in departments, organizations, networks of institutions

Combined

Videotaping and audiotaping users as they interact with an application and capturing keystrokes. Can capture any combination of the above.

Adapted from Staggers N. Human–computer interaction. In: Englebardt S, Nelson R, eds. Information Technology in Health Care: An Interdisciplinary Approach. Philadelphia, PA: Harcourt Health Science Company; 2001:321-345; and Staggers N. Improving the usability of health informatics applications. In: Hebda T, Czar P, eds. Handbook of Informatics for Nurses and Health Professionals. Upper Saddle River, NJ: Pearson Education; 2012:170-193.

EEG, Electroencephalography; FTE, full-time equivalent.

•Requirements about the specific environments and particular needs related to the context of interactions

Studies can be conducted with limited resources if the scope of the investigation is focused. As the complexity increases, resource consumption increases concomitantly. Assessments early in the systems life cycle seek to answer the following questions:

•Who are the users and what are their characteristics?

•What are basic activities and tasks in this context?

•How do users cognitively process information?

•What information processing can be supported by products?

•What special considerations should be made for users in this environment?

•What attributes need to be in place for an initial design?

Task analysis and think-aloud protocol, both discussed above, can be used to determine users' needs and requirements and answer the questions listed here.

Example of a Requirements Determination Usability Study

Staggers and colleagues completed two studies focused on nurses' acute care hand-offs or change of shift reports to determine the current state of the activity and to develop requirements to support hand-off tasks.77,78 Hand-offs are highly complex and cognitively intensive periods where nurses going off shift synthesize information about patients and communicate it to nurses coming on shift. Methods that would generate rich details about the process, such as observation, field notes, and interviews, were selected. The HHCI framework guided the thinking about requirements for different aspects of the hand-off process to be studied. For example, nurses (expertise levels, regular versus travel nurses), types of units (critical care, emergency department, medical and surgical), and types of product support in place (EHRs, CPOE, eMAR) were identified using this framework. Hand-off tasks can be completed in a variety of ways, including audiorecorded, face-to-face, and bedside reports. The researchers completed a focused ethnography across available medical and surgical units in different facilities. They observed change of shift reports, audiotaped nurses, photographed nurses' tools, and took field notes about nurses' interactions with the existing EHRs in the facilities. From the findings the researchers were able to derive detailed information about requirements for computerized support for change of shift activities.79

Exploratory Test

An exploratory test is conducted early in the systems life cycle after requirements are determined. These tests are conducted on very basic or preliminary designs or redesigns where few resources have been committed to programming the product. The objective of an exploratory test is to assess the effectiveness of emerging design concepts by asking the following:

•Is the basic functionality of value to users?

•Is basic navigation and information flow intuitive?

•Is fundamental content missing?

•How much computer experience does a user need to use this module?42

Exploratory tests are more informal, with extensive interaction between testers and users. The usability focus is on the goal of effectiveness. Users are asked to perform common tasks with the prototype or step through paper mockups of the application using a think-aloud protocol. Researchers strive to understand why users are behaving as they do with the application rather than how quickly they perform.79 To assess effectiveness the researcher or informaticist is interested in finding cognitive disconnects with basic functions, missing information or steps and assessing how easily users understand the task at hand.

Nielsen recommends having five users talk aloud or “think aloud” as an observer watches them and records any issues. This number of users can detect as much as 60% to 80% of design errors.56 Later research confirms that as few as five to eight users are sufficient for most usability tests.80–81

Example of an Exploratory Test

A public health researcher wanted to develop an application to display reportable patient conditions across jurisdictions. After researching available applications and conducting a requirements determination, she developed a prototype using PowerPoint to meet initial requirements and assess usability for the following sample tasks: find out whether chlamydia is a reportable condition in Utah, Colorado, or Washington; determine the time frame for reporting the condition; and ascertain whether a specimen must be submitted and the location for the submission. She selected key public health, clinical, and laboratory users. These users were asked to think aloud as they completed common tasks. The researcher took notes to record the data related to their responses. Using these data several iterations of the prototype were designed to improve the user experience in completing the tasks.82

Assessment Test

An assessment test is conducted early in or midway through the development of a product application.42 After the organization and general design are determined, this kind of test assesses lower-level operations of the application, stressing the efficiency goals of the product (versus effectiveness) and how well the task is presented to users. Questions during this test might include the following:

•How quickly and accurately can users perform selected tasks?

•Are the terms in the system consistent across modules?

•Are operations displayed in a manner that allows quick detection of critical information?

Users perform common tasks with a product that is partially developed. Usability measures (see Table 21-3) such as performance time and errors are selected. Users can perform tasks silently or researchers can use think-aloud methods to elicit issues. Again, designers use the results to craft a redesigned prototype to correct issues. Iterative development typically takes three rounds of design to eliminate major usability issues.

Example of an Assessment Test

The researcher in the exploratory test example above assessed a later version of the public health application. She asked the same key users to use the same tasks but now participants commented on operations, icons, and the arrangement of the radio buttons.

Validation Test

A validation test is completed later in the systems life cycle using a more mature product. This type of test assesses how this particular product compares to a predetermined standard, benchmark, or performance measure. A second purpose might be to assess how all modules in a technology application work as an integrated whole. A validation test can also be useful in a system selection process to decide how a new vendor supports critical tasks such as medication bar-coding or medication reconciliation. Questions for a validation test might include the following:

•Can 80% of users retrieve the correct complete blood count (CBC) test results within 10 seconds of interacting with the system?

•How many heuristic violations are identified for this product?

•Can users complete admission orders for a trauma patient with no errors?

This kind of test is more structured so it precludes extensive interactions between testers and users. The specific methods are carefully structured similar to the processes used in an experimental study.

Example of a Validation Test

A nurse researcher wanted to ensure that a new mobile device for rural care in Tanzania mirrored the established algorithms on paper. The goal was to assess whether the algorithms used with the mobile device were 100% accurate. She enlisted key users and informaticists to interact with each pathway in the device. Deviations from the established algorithms were documented and corrected.83

Comparison Study

Students can conduct comparison studies at any point in the systems life cycle but they are more commonly done to compare an existing design with a redesign or an early prototype with a more mature product. The major objective of this usability test is to determine which application, design, or product is more effective, efficient, and satisfying.42 The study design can range from an informal side-by-side comparison with structured tasks or a classic experimental study. Results, however, are more dramatic if the designs are substantially different.

Examples of a Comparison Study

The purpose of this study was to determine whether a new user interface for orders management was different than an older interface in terms of performance times, errors, and user satisfaction.84 The tasks and interactions were planned to minimize the amount of time that nurses would be away from patient care. The informaticists used an HCI framework, similar to the HHCI framework, to guide elements in the study. Identical computers were used to test both interfaces. Tasks were “real-world” nursing orders and the same across the two designs. The environment was a computer training room, a quiet room away from patient care units and distractions. The developmental trajectory was considered in this study to ensure that results were not affected by practice time. Therefore 40 tasks for each interface allowed nurses to become practiced at each user interface. The number of tasks was determined in pilot work. Tasks, keystrokes, and errors were captured by the computer. The QUIS was administered after each interface. Each nurse interacted with both interfaces but the order in which they were presented was randomized. The results showed a significant difference on all three variables: performance times, errors, and user satisfaction.

In a second study a nurse researcher wanted to compare the traditional design of physiologic monitors and other products to a new design that integrated data across physiologic parameters, medication management, and communication. His target population was intensive care unit (ICU) nurses. The tasks were designed so that the study could be completed in about 20 to 30 minutes over two sessions. Paper prototypes were used to assess effectiveness, efficiency, and satisfaction before resources were expended to code bidirectional interfaces to the devices. Tasks were defined and nurses interacted with the prototypes. Findings included faster task times, higher detection of potential medication interactions, lower perceived mental workload, and higher satisfaction with the integrated monitoring view.85,86

Identifying Usability Issues with Fielded Products

As organizations begin to understand the importance of the user experience, leaders may be unsure where to begin identifying usability issues in their current environments. The following list includes symptoms of potential strategic usability issues and provides a framework for determining where initial energy and resources could be focused:

•Products or applications requiring long training times

•Support calls categorized by product or application

•Adverse events related to product interactions

•Lists of requested system change requests typically tracked in a database of system change requests across users and products

•User group requests for updates or changes

•Users' descriptions of their most vexing applications and interactions

•Users' identified delays or errors when they interact with complex applications, especially any requiring information synthesis such as eMARs, clinical summaries, and hand-offs

Once a usability problem is suspected, researchers or students begin assessing the issue more systematically using the techniques described above.

Steps for Conducting Usability Tests

At some point in their careers, students will likely be expected to or will want to conduct a usability project. Step-by-step guides are available.42,60 The texts by Beth Crandell,87 Rubin and Chisnell,42 and Tom Tullis and Bill Albert65 are easy to understand. HIMSS60 and the National Institute of Standards and Technology (NIST)88 have also published guides for conducting usability tests on EHRs. The basic steps for conducting usability tests can be summarized as follows:

1.Define a clear purpose. The specific purpose guides testers to determine the type of study, methods, and users required. For example, if the purpose relates to assessment of a redesign of a CPOE module for an intraoperative surgical team, an exploratory test may be indicated.

2.Assess constraints. Testers are always mindful of study constraints: time; resources; availability of the software to be evaluated; and availability of other equipment such as video cameras, testing labs, or users, especially if the users are specialists. These constraints may drive the type of usability test. For example, if the tester's goal is to evaluate an application to support anesthesiologists, these time-constrained physicians may not be willing to spend more than 15 to 20 minutes participating in a usability test. Tasks, methods, and products are defined to work within this constraint.

3.Use an HCI framework to define pertinent components. Use a framework to assess each component against the planned study. Who are the key users? What are typical tasks? What information needs to be exchanged? What product characteristics and which actions are needed? What is the setting or context? Will it be a naturalistic setting to determine exactly how an application will be used or a laboratory setting to control interruptions? What is a representative time in the developmental trajectory? How much practice time needs to be considered, especially if the design is new? Be sure to examine the latter component carefully to ensure a valid comparison between users' interaction with a new product and a current one by including practice time in the study.

4.Match methods to the purpose, constraints, and framework assessment. Methods that produce rich results such as a think-aloud protocol will match a purpose of understanding key user requirements while a more structured method will allow a comparison of new and old designs. Long training and practice times for complex devices may constrain the number of tasks testers can offer. Other basic methodological steps are as follows:

•Select representative end-users

•Select a usability test appropriate to the purpose and point in the systems life cycle

•Define and validate tasks

•Measure key elements and control for others (e.g., measure performance time but control interruptions unless the effect of interruptions is the focus)

•Define the context to be used

•Consider training and practice for new products

•Pilot test methods before running the main study to smooth out procedures and bugs

Once the methods are defined, all of these pieces can be put into action and the study can be conducted.

Conclusion and Future Directions

In the U.S., federal agencies are now engaging in activities to improve the user experience for health IT products (e.g., NIST, the regulations crafted by the Office of the National Coordinator for Health Information Technology). The Food and Drug Administration has required usability testing for the last decade but EHR vendors and other health organizations are only beginning to employ the principles and processes for improving the user experience. The most immediate future direction concerns user-experience education and understanding of action steps to be taken. The current federal requirements will continue to expand. Organizations need to increase their knowledge and skills related to improving the user experience. One way is to use the material outlined by the HIMSS Usability Task Force in 2011 on a Health Usability Maturity Model.10 The material can guide organizations in assessing their current level of user experience, selling usability to the organization, and increasing the user experience to reach a strategic level.

Future usability methods might include automated methods to ensure that designs conform to known standards, especially basic screen designs. Clearly, the future includes a focus on health IT products that support the way users think and do work in health settings.

This chapter described current issues with technology and the potential benefits of improving the user experience in health organizations. After definitions of terms, the axioms of usability were defined: an early and central focus on users in the design and development of systems, iterative design of applications, and systematic usability measures. Across HCI frameworks these major elements exist: users, products, contexts, tasks, information, interactions, and a developmental trajectory. Common usability methods and tests were discussed. Students are now prepared to conduct discount usability tests and four different types of usability tests: exploratory, assessment, validation, and comparison. Usability measures are available and four steps were outlined for planning and conducting usability tests.

References

1.

Stead, W, Lin, H: Computational Technology for Effective Health Care: Immediate Steps and Strategic Directions . 2009, National Academies Press, Washington, DC.

2.

Ash, JS, Berg, M, Coiera, E: Some unintended consequences of information technology in health care: the nature of patient care information system-related errors. J Am Med Inform Assn. 11(2), 2004, 104–112, doi:10.1197/jamia.M1471.

3.

Ash, JS, Sittig, DF, Dykstra, R, Campbell, E, Guappone, K: The unintended consequences of computerized provider order entry: findings from a mixed methods exploration. Int J Med Inform. 78(suppl 1), 2009, S69–S76, doi:10.1016/j.ijmedinf.2008.07.015.

4.

Silver, JD, Hammil, SD: Doctor, nurse disciplined by UPMC in kidney transplant case: failed to detect hepatitis C in kidney donated for transplant. Pittsburg Post Gazette. 2011, Accessed March 1, 2012 http://www.post-gazette.com/pg/11147/1149429-114-0.stm.

5.

Guo, J, Irdbarren, S, Kapsandoy, S, Perri, S, Staggers, N: eMAR user interfaces: a call for ubiquitous usability evaluations and product redesign. Applied Clinical Informatics. 2(2), 2011, 202–224.

6.

Koppel, R, Metlay, JP, Cohen, A, et al.: Role of computerized physician order entry systems in facilitating medication errors. J Amer Med Assoc. 293(10), 2005, 1197–1203, doi:10.1001/jama.293.10.1197.

7.

Staggers, N, Jennings, BM, Lasome, CE: A usability assessment of AHLTA in ambulatory clinics at a military medical center. Mil Med. 175(7), 2010, 518–524.

8.

Philpott, T: Military access drops. Today in the Military http://www.military.com, 2006, Accessed April 30, 2012.

9.

Philpott, T: Doctors see bigger role in electronic health record reform. Stars and Stripes http://www.stripes.com, 2009, Accessed April 30, 2012.

10.

Healthcare Information and Management Systems Society: Promoting Usability in Health Organizations: Initial Steps and Progress toward a Healthcare Usability Maturity Model . 2011, Health Information and Management Systems Society, Chicago, IL.

11.

Bias, RG, Mayhew, DJ: Cost-Justifying Usability: An Update for the Internet Age . 2005, Morgan Kaufman, San Francisco, CA.

12.

Nielsen, J, Gilutz, S: Usability Return on Investment . 2003, Nielsen Norman Group, Fremont, CA.

13.

Usability Professionals’ Association: http://www.upassoc.org/, 2012, Accessed April 4, 2012.

14.

Dey Alexander Consulting: Return on Investment (ROI) discussion articles. 2009, Dey Alexander Consulting, Accessed April 4, 2012 http://www.deyalexander.com.au/resources/uxd/roi.html.

15.

Saitwal, H, Feng, X, Walji, M, Patel, V, Zhang, J: Assessing performance of an electronic health record (EHR) using cognitive task analysis. Int J Med Inform. 79(7), 2010, 501–506, doi:10.1016/j.ijmedinf.2010.04.001.

16.

Souza, R, Sonderegger, P, Roshan, S, Dorsey, M: Get ROI from Design. 2001, Forrester Research, Cambridge, MA.

17.

Institute of Medicine: Crossing the Quality Chasm: A New Health System for the 21st Century . 2001, National Academy of Sciences, Washington, DC.

18.

World Health Organization: 2007, World Health Organization.

19.

Ash, JS, Sittig, DF, Campbell, EM, Guappone, KP, Dykstra, RH: Some unintended consequences of clinical decision support systems. AMIA Annu Symp Proc. 2007, 26–30.

20.

Staggers, N, Kobus, D, Brown, C: Nurses’ evaluations of a novel design for an electronic medication administration record. Comput Inform Nurs. 25(2), 2007, 67–75.

21.

Kushniruk, A, Triola, M, Stein, B, Borycki, E, Kannry, J: The relationship of usability to medical error: an evaluation of errors associated with usability problems in the use of a handheld application for prescribing medications. St Heal T. 107(Pt 2), 2004, 1073–1076, doi:D040004885 [pii].

22.

Nielsen, J, Levy, J: Measuring usability: preference vs performance. Communications of the ACM. 37(4), 1994, 66–75.

23.

Dray, SM: The importance of designing usable systems. Interactions. 2(1), 1995, 17–20.

24.

Drews, FA, Westenskow, DR: The right picture is worth a thousand numbers: data displays in anesthesia. Hum Factors. 48(1), 2006, 59–71.

25.

Syroid, ND, Agutter, J, Drews, FA, et al.: Development and evaluation of a graphical anesthesia drug display. Anesthesiology. 96(3), 2002, 565–575, doi:00000542-200203000-00010 [pii].

26.

Wachter, SB, Agutter, J, Syroid, N, Drews, F, Weinger, MB, Westenskow, D: The employment of an iterative design process to develop a pulmonary graphical display. J Am Med Inform Assn. 10(4), 2003, 363–372, doi:10.1197/jamia.M1207 [pii].

27.

Pressman, RS: Software Engineering: A Practitioner's Approach . 1992, McGraw-Hill, New York, NY.

28.

Bias, RG, Mayhew, DJ: Cost-Justifying Usability . 1994, Morgan Kaufmann Publishers, San Francisco, CA.

29.

IBM: Cost justifying ease of use: complex solutions are problems http://www-01.ibm.com/software/ucd/ucd.html, 2001, Accessed December 11, 2012.

30.

Dray, SM, Karat, C: Human factors cost justification for an internal development project. In Bias, RG, Mayhew, DJ (Eds.): Cost-Justifying Usability. 1994, Morgan Kauffman Publishers, San Francisco, CA, 111–122.

31.

Marcus, A: Return on Investment for Usable User-Interface Design: Examples and Statistics . 2004, Experience Intelligent Design, AM+A White Paper, Berkeley, CA.

32.

Bosert, JL: Quality Function Deployment: A Practitioner's Approach . 1991, ASQC Quality Press, New York, NY.

33.

Karat C. Iterative testing of a security application. Paper presented at: Proceedings of the Human Factors Society; 1989; Denver, CO.

34.

Landauer, TK: The Trouble with Computers: Usefulness, Usability and Productivity . 1995, MIT Press, Cambridge, MA.

35.

Kuniavsky, M: Observing the User Experience: A Practioner's Guide to User Research . 2003, Morgan Kaufmann Publishers, San Francisco, CA.

36.

International Organization for Standardization (ISO): ISO 9241-9211. Usability Net http://www.usabilitynet.org/tools/r_international.htm#9241-11, 1998, Accessed February 7, 2011.

37.

Schaffer, E: Is usability different than the user experience?. 2010, HFI Connect, Accessed February 7, 2011 http://connect.humanfactors.com/profiles/blogs/is-user-experience-different.

38.

Nielsen Norman Group: User experience: our definition. 2007, Nielsen Norman Group, Accessed February 7, 2011 http://www.nngroup.com/about/userexperience.html.

39.

In Definitions of human factors and ergonomics. 2012, Human Factors and Ergonomics Society, Accessed April 4, 2012 http://www.hfes.org/Web/EducationalResources/HFEdefinitionsmain.html#govagencies.

40.

Norman, D: The Psychology of Everyday Things . 1988, Basic Books, New York, NY.

41.

Dix, A, Finlay, JE, Abowd, GD, Beale, R: Human–Computer Interaction. 3rd ed, 2004, Prentice Hall, Essex, England.

42.

Rubin, J, Chisnell, D: Handbook of Usability Testing: How to Plan, Design and Conduct Effective Tests. 2nd ed, 2008, John Wiley & Sons, New York, NY.

43.

Ammenwerth, E, Iller, C, Mahler, C: IT-adoption and the interaction of task, technology and individuals: a FIT framework and a case study. BioMed Central. 6(3), 2006, 1–13.

44.

Borycki, EM, Kushniruk, AW, Bellwood, P, Brender, J: Technology-induced errors: the current use of frameworks and models from the biomedical and life sciences literatures. Methods of Informatics in Medicine. 51(2), 2012, 95–103, doi:10.3414/ME11-02-0009.

45.

Despont-Gros, C, Mueller, H, Lovis, C: Evaluating user interactions with clinical information systems: a model based on human–computer interaction models. J Biomed Inform. 38(3), 2005, 244–255, doi:10.1016/j.jbi.2004.12.004.

46.

Hollnagel, E, Woods, D: Joint Cognitive Systems: Foundations of Cognitive Systems Engineering . 2005, Yaylor & Francis, Boca Raton, FL.

47.

Johnson, CM, Johnson, TR, Zhang, J: A user-centered framework for redesigning health care interfaces. J Biomed Inform. 38(1), 2005, 75–87, doi:10.1016/j.jbi.2004.11.005.

48.

Staggers, N: Human–computer interaction. In Englebardt, S, Nelson, R (Eds.): Information Technology in Health Care: An Interdisciplinary Approach. 2001, Harcourt Health Science Company, Philadelphia, PA, 321–345.

49.

Yusof, MM, Stergioulas, L, Zugic, J: Health information systems adoption: findings from a systematic review. Stud Health Technol Inform. 129(Pt 1), 2007, 262–266.

50.

Zhang J, Butler KA. UFuRT: a work-centered framework and process for design and evaluation of information systems. Paper presented at: Proceedings of HCI International; 2007; Beijing, China.

51.

Staggers, N, Parks, PL: Collaboration between unlikely disciplines in the creation of a conceptual framework for nurse–computer interactions. Proc Annu Symp Comput Appl Med Care. 1992, 661–665.

52.

Staggers, N, Parks, PL: Description and initial applications of the Staggers & Parks Nurse–computer Interaction Framework. Comput Nurs. 11(6), 1993, 282–290.

53.

Staggers, N: Improving the usability of health informatics applications. In Hebda, T, Czar, P (Eds.): Handbook of Informatics for Nurses and Health Professionals. 2013, Pearson Education, Upper Saddle River, NJ, 170–193.

54.

Thompson, CB, Snyder-Halpern, R, Staggers, N: Analysis, processes, and techniques: case study. Comput Nurs. 17(5), 1999, 203–206.

55.

Nielsen, J: Usability Engineering . 1993, AP Professional, Cambridge, MA.

56.

Nielsen, J: Heuristic evaluation. In Nielsen, J, Mack, RL (Eds.): Usability Inspection Methods. 1994, John Wiley & Sons, Inc, New York, NY.

57.

Nielsen, J: Severity ratings for usability problems. Useit.com http://www.useit.com/papers/heuristic/severityrating.html, 1994, Accessed April 30, 2012.

58.

Zhang, J, Johnson, TR, Patel, VL, Paige, DL, Kubose, T: Using usability heuristics to evaluate patient safety of medical devices. J Biomed Inform. 36(1-2), 2003, 23–30.

59.

Shneiderman, B, Plaisant, K: Designing the User Interface: Strategies for Effective Human–Computer Interaction. 4th ed, 2005, Pearson/Addison-Wesley, Boston, MA.

60.

HIMSS EHR Usability Task Force: Defining and testing EMR usability: principles and proposed methods of EMR usability evaluation and rating. 2009, Healthcare Information and Management Systems Society, Accessed April 12, 2012 http://www.himss.org/content/files/HIMSS_DefiningandTestingEMRUsability.pdf.

61.

Hackos, JT, Redish, JC: User and Task Analysis for Interface Design . 1998, John Wiley & Sons, New York, NY.

62.

Mayhew, D: Requirements specifications within the usability engineering lifecycle. In Sears, A, Jacko, J (Eds.): The Human–Computer Interaction Handbook. 2008, Lawrence Erlbaum Association, New York, NY, 917–927.

63.

Courage, C, Redish, J, Wixon, D: Task analysis. In Sears, A, Jacko, J (Eds.): The Human–Computer Interaction Handbook. 2008, Lawrence Erlbaum Associates, New York, NY, 928–937.

64.

Wei, J, Salvendy, G: The cognitive task analysis methods for job and task design: review and reappraisal. Behav Inform Technol. 23(4), 2004, 273–299.

65.

Tullis, T, Albert, B: Measuring the User Experience: Collecting, Analyzing and Presenting Usability Metrics . 2008, Elseiver, New York, NY.

66.

Hammersley, M, Atkinson, P: Ethnography: Principles in Practice. 3rd ed, 2007, Routledge, London, England.

67.

Ash, JS, Sittig, DF, Dykstra, R, Campbell, E, Guappone, K: Exploring the unintended consequences of computerized physician order entry. Stud Health Technol Inform. 129(Pt 1), 2007, 198–202.

68.

Ash, JS, Sittig, DF, Dykstra, RH, Guappone, K, Carpenter, JD, Seshadri, V: Categorizing the unintended sociotechnical consequences of computerized provider order entry. Int J Med Inform. 76(suppl. 1), 2007, S21–S27, doi:10.1016/j.ijmedinf.2006.05.017.

69.

Ash, JS, Sittig, DF, Poon, EG, Guappone, K, Campbell, E, Dykstra, RH: The extent and importance of unintended consequences related to computerized provider order entry. J Am Med Inform Assn. 14(4), 2007, 415–423, doi:10.1197/jamia.M2373.

70.

Staggers, N: Invited testimony about EHR Usability. 2011, Office of the National Coordinator for Health Information Technology, Hearing on EHR Usability, Washington, DC, April 21, 2011.

71.

Sweeney, M, Maguire, M, Shackel, B: Evaluating user–computer interaction: a framework. Int J Man Mach Stud. 38, 1993, 689–711.

72.

Bangor, A, Kortum, P, Miller, JT: An empirical evaluation of the System Usability Scale. Int J Hum-Comput Int. 24(6), 2008.

73.

Sauro, J, Measuring usability with the System Usability Scale (SUS): Measuring Usability http://www.measuringusability.com/sus.php, 2011, Accessed April 30, 2012.

74.

Norman, K, Shneiderman, B, Harper, B, Slaughter, L: Questionnaire for user interaction satisfaction, version 7.0 http://lap.umd.edu/quis/, 1998, Accessed February 16, 2012.

75.

Lin, H, Choong, Y, Salvendy, G: A proposed index of usability: a method for comparing the relative usability of different software systems. Behav Inform Technol. 16(4/5), 1997, 267–278.

76.

Kirakowski, J, Corbett, M: SUMI: the software measurement inventory. Brit J Educ Technol. 24, 1993, 210–212.

77.

Staggers, N, Clark, L, Blaz, JW, Kapsandoy, S: Nurses’ information management and use of electronic tools during acute care handoffs. Western J Nurs Res. 34(2), 2012, 153–173, doi:10.1177/0193945911407089.

78.

Staggers, N, Jennings, BM: The content and context of change of shift report on medical and surgical units. J Nurs Admin. 39(9), 2009, 393–398, doi:10.1097/NNA.0b013e3181b3b63a 00005110-200909000-00009 [pii].

79.

Staggers, N, Clark, L, Blaz, JW, Kapsandoy, S: Why patient summaries in electronic health records do not provide the cognitive support necessary for nurses’ handoffs on medical and surgical units: insights from interviews and observations. Health Informatics J. 17(3), 2011, 209–223, doi:10.1177/1460458211405809.

80.

Shneiderman, B, Plaisant, C, Cohen, M, Jacobs, S: Designing the User Interface: Strategies for Effective Human–Computer Interaction. 5th ed, 2010, Addison-Wesley, Reading, MA.

81.

Dumas, JS, Fox, JE: Usability testing: current practice and future directions. In Sears, A, Jacko, J (Eds.): The Human–Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications. 2nd ed, 2008, Lawrence Erlbaum, New York, NY.

82.

Rajeev, D: Development and Evaluation of New Strategies to Enhance Public Health Reporting. [dissertation]. 2012, Biomedical Informatics, University of Utah, Salt Lake City, UT.

83.

Perri, S: Using Electronic Decision Support to Enhance Provider–Caretaker Communication for Treatment of Children Under Five in Tanzania. [dissertation]. 2012, College of Nursing, University of Utah, Salt Lake City, UT.

84.

Staggers, N, Kobus, D: Comparing response time, errors, and satisfaction between text-based and graphical user interfaces during nursing order tasks. J Am Med Inform Assn. 7(2), 2000, 164–176.

85.

Koch S, Westenskow D, Weir C, et al. ICU nurses’ evaluations of integrated information integration in displays for ICU nurses on user satisfaction and perceived mental workload. Paper presented at: Medical Informatics Europe; 2012; Pisa, Italy.

PAGE 350

86.

Koch SH, Weir C, Westenskow D, et al. Evaluation of the effect of information integration in displays for ICU nurses on situation awareness and task completion time: a prospective randomized controlled study. Int J Med Inform. epublication January 25, 2013.

87.

Crandell, B, Klein, G, Hoffman, R: Working Minds: A Practioner's Guide to Cognitive Task Analysis . 2006, Massachusetts Institute of Technology, Boston, MA.

88.

National Institute of Standards and Technology (NIST): Technical evaluation, testing and validation of electronic health records. 2011, NIST, Accessed April 30, 2012 http://www.nist.gov/itl/iad/ehr-032012.cfm.

Discussion Questions

1. Assess critical user experience issues in your own organization. Describe one example and apply the heuristic evaluation technique to analyze where the major violations exist.

2. Search the Internet for current policies on the user experience in health IT by searching the Office of the National Coordinator for Health Information Technology and the National Institute of Standards and Technology. Analyze how one of these current policies will affect your organization.

3. Outline a usability test for your current eMAR. Include the elements discussed in the chapter in your proposed usability test. Describe why you chose the methods you did.

4. Your organization is planning to purchase new physiologic monitors for the adult ICUs. You are the informatics person assigned to this project. Describe how you would include usability in the purchase process. Outline participants, methods, and tasks to be tested.

Case Study

A tertiary care center in the western U.S. has an installed base of electronic health records supported by Cerner Corporation for inpatient areas and by Epic for outpatient areas. Other technology includes a suite of about 300 different applications supported by the IT department. The current environment, while including robust capabilities such as computerized provider order entry, is “siloed” with information. Healthcare providers complain that they have difficulty obtaining the “big picture” of the patient across systems and they have to remember information located in disparate systems. They are burdened with integrating information themselves. Not only is this time consuming, it is potentially prone to error. Providers have developed numerous workarounds to the different systems in ambulatory and inpatient areas, including “shadow” files for patients they see frequently. Nurses complain that they have to “jump around” the inpatient system to find information they need for activities such as patient hand-offs.

The organization responds by developing a vision for the future that centered on the concept of knowledge management (KM). This concept is defined as the systematic process of identifying, capturing, and transferring information and knowledge that people can use to create, compete (with other organizations), and improve. A crucial aspect of KM is improving the user experience. As the leaders in the organization begin to address KM and improve the user experience, they are employing the same tactics described in this chapter.

Discussion Questions

1. Assume that you are the leader of the KM effort. Where would you start to improve the user experience?

2. Pharmacists supporting the ICUs are asking for your help to improve their situation (their user experience) because they are forced to use nonintegrated systems. What methods would you use to examine this issue?

3. The institution is in the process of purchasing new physiologic monitors for their step-down unit. Describe how usability should be incorporated as part of the purchasing process. Design a brief usability test to support the purchasing process.

Pageburst Integrated Resources

Chapter 21 Improving the User Experience for Health Information Technology Products

Nancy Staggers

sability has a strong, often direct relationship with clinical productivity, error rates, user fatigue and

user satisfaction

—

crit

ical factors for EMR adoption.

Healthcare Information and Management Systems Society, 2009

Objectives

At the completion of this chapter the reader will be prepared to:

1.Compare and contrast the terms user experience, human factors, ergonomics, h

uman

–

computer

interaction, and usability

2.Describe the goals of usability and user

-

centered design

3.Identify the major components to consider in human

–

computer interaction and usability studies

4.Analyze methods for conducting usability studies and relate them to a specific purpose of a usability

study

5.Outline components of four different usability tests related to their position in the systems life cycle

6.Explain the basic steps i

n conducting a usability test on a healthcare product

Key Terms

Discount usability methods, 340

Ergonomics, 337

Focused ethnographies, 342

Heuristic evaluations, 340

Human factors, 337

Human

–

computer interaction (HCI), 338

Joint cognitive systems, 339

Soc

iotechnical system, 335

Task analysis, 341

Think

-

aloud protocol, 341

Chapter 21 Improving the User Experience for Health Information Technology Products

Nancy Staggers

sability has a strong, often direct relationship with clinical productivity, error rates, user fatigue and

user satisfaction—critical factors for EMR adoption.

Healthcare Information and Management Systems Society, 2009

Objectives

At the completion of this chapter the reader will be prepared to:

1.Compare and contrast the terms user experience, human factors, ergonomics, human–computer

interaction, and usability

2.Describe the goals of usability and user-centered design

3.Identify the major components to consider in human–computer interaction and usability studies

4.Analyze methods for conducting usability studies and relate them to a specific purpose of a usability

study

5.Outline components of four different usability tests related to their position in the systems life cycle

6.Explain the basic steps in conducting a usability test on a healthcare product

Key Terms

Discount usability methods, 340

Ergonomics, 337

Focused ethnographies, 342

Heuristic evaluations, 340

Human factors, 337

Human–computer interaction (HCI), 338

Joint cognitive systems, 339

Sociotechnical system, 335

Task analysis, 341

Think-aloud protocol, 341