HIM 301 Introduction to Health Informatics- WK1-D2

Chapter 5 Program Evaluation and Research Techniques

Charlene R. Weir

Evaluation of health information technology (health IT) programs and projects can range from simple user satisfaction for a new menu or full-scale analysis of usage, cost, compliance, patient outcomes, and observation of usage to data about patient's rate of improvement.

Objectives

At the completion of this chapter the reader will be prepared to:

1.Identify the main components of program evaluation

2.Discuss the differences between formative and summative evaluation

3.Apply the three levels of theory relevant to program evaluation

4.Discriminate program evaluation from program planning and research

5.Synthesize the core components of program evaluation with the unique characteristics of informatics interventions

Key Terms

Evaluation, 72

Formative evaluation, 73

Logic model, 79

Program evaluation, 73

Summative evaluation, 73

Abstract

Evaluation is an essential component in the life cycle of all health IT applications and the key to successful translation of these applications into clinical settings. In planning an evaluation the central questions regarding purpose, scope, and focus of the system must be asked. This chapter focuses on the larger principles of program evaluation with the goal of informing health IT evaluations in clinical settings. The reader is expected to gain sufficient background in health IT evaluation to lead or participate in program evaluation for applications or systems.

Formative evaluation and summative evaluation are discussed. Three levels of theory are presented, including scientific theory, implementation models, and program theory (logic models). Specific scientific theories include social cognitive theories, diffusion of innovation, cognitive engineering theories, and information theory. Four implementation models are reviewed: PRECEDE-PROCEED, PARiHS, RE-AIM, and quality improvement. Program theory models are discussed, with an emphasis on logic models.

A review of methods and tools is presented. Relevant research designs are presented for health IT evaluations, including time series, multiple baseline, and regression discontinuity. Methods of data collection specific to health IT evaluations, including ethnographic observation, interviews, and surveys, are then reviewed.

Introduction

The outcome of evaluation is information that is both useful at the program level and generalizable enough to contribute to the building of science. In the applied sciences, such as informatics, evaluation is critical to the growth of both the specialty and the science. In this chapter program evaluation is defined as the “systematic collection of information about the activities, characteristics, and results of programs to make judgments about the program, improve or further develop program effectiveness, inform decisions about future programming, and/or increase understanding.”1 Health IT interventions are nearly always embedded in the larger processes of care delivery and are unique for three reasons. First, stakeholders' knowledge about the capabilities of health IT systems may be limited at the beginning of a project. Second, the health IT product often changes substantially during the implementation process. Third, true implementation often takes 6 months or longer, with users maturing in knowledge and skills and external influences, such as new regulations or organizational initiatives, occurring over that period. Identification of the unique contribution of the health IT application therefore is often difficult and evaluation goals frequently go beyond the health IT component alone. In this chapter the health IT component of evaluation is integrated with overall program evaluation; unique issues are highlighted for evaluating health IT itself. The chapter is organized into three sections: (1) purposes of evaluation, (2) theories and frameworks, and (3) methods, tools, and techniques.

Purposes of Evaluation

The purpose of evaluation determines the methods, approaches, tools, and dissemination practices for the entire project being evaluated. Therefore identifying the purpose is a crucial first step. Mark, Henry, and Julnes have provided four main evaluation purposes, which are listed in Box 5-1.

Usually an evaluation project is not restricted to just one of these purposes. Teasing out which purposes are more important is a process for the evaluator and the involved stakeholders. The following sections represent a series of questions that can clarify the process.

Formative versus Summative Evaluation

Will the results of the evaluation be used to improve the program or to focus on determining whether the goals of the program have been met? This question refers to a common classification of evaluation activities that fall into two types: (1) formative evaluation and (2) summative evaluation. The difference is in how the information is used. The results of the formative evaluation are used as feedback to the program for continuous improvement.2,3 The results of the summative evaluation are used to evaluate the merit of the program. Formative evaluation is a term coined by Scriven in 1967 and expanded on by a number of other authors to mean an assessment of how well the program is being implemented and to describe the experience of participants.4 Topics for formative evaluation include the fidelity of the intervention, the quality of implementation, the characteristics of the organizational context, and the types of personnel. Needs assessments and feasibility analyses are included in this general category. Box 5-2 outlines several questions that fall into the category of formative evaluation for health IT.

Box 5-1 Main Purposes of Program Evaluation

• Program and organizational improvement

• Assessment of merit or worth

• Knowledge development

• Oversight and compliance

Adapted from Mark M, Henry G, Julnes G. Evaluation: An Integrative Framework for Understanding, Guiding and Improving Policies and Programs. San Francisco, CA: Jossey-Bass; 2000.

In contrast, summative evaluation refers to an assessment of the outcomes and impact of the program. Cost-effectiveness and adverse events analyses are included in this category. Some questions that fall into the summative evaluation category are listed in Box 5-3.

Dividing the evaluation process into the formative and summative components is somewhat arbitrary, as they can be and often are conducted concurrently. They do not necessarily differ in terms of methods or even in terms of the content of the information collected. Formative evaluation is especially important for health IT products where the overall goal is improvement. Because health IT products are “disruptive technologies,” they both transform the working environment and are themselves transformed during the process of implementation.5 Many writers in the informatics field have noted the paucity of information on implementation processes in published studies. In a meta-analysis of health IT by researchers at RAND, the authors noted:

In summary, we identified no study or collection of studies, outside of those from a handful of health IT leaders that would allow a reader to make a determination about the generalizable knowledge of the system's reported benefit. This limitation in generalizable knowledge is not simply a matter of study design and internal validity. Even if further randomized, controlled trials are performed, the generalizability of the evidence would remain low unless additional systematic, comprehensive, and relevant descriptions and measurements are made regarding how the technology is utilized, the individuals using it, and the environment it is used in.6(p4)

Box 5-2 Questions to Pose during Formative Evaluation

• What is the nature and scope of the problem that is being addressed by health IT?

• What is the extent and seriousness of the need?

• How well is the technology working and what is the best way to deliver it?

• How are participants (and users) experiencing the program?

• How did the intervention change after implementation?

Box 5-3 Questions to Pose during Summative Evaluation

• To what degree were the outcomes affected by the product?

• What is the cost effectiveness of the product?

• What were the unintended consequences of the product?

Although written in 2006, this statement is still relevant today.

Generalizability and Scope

Will the results of the evaluation be used to inform stakeholders of whether a particular program is “working” and is in compliance with regulations and mandates? This question refers to issues of the generalizability and scope of the project. It is also a question of whether the evaluation is more of a program evaluation or a research study. An evaluation of a locally developed project usually would be considered a program evaluation. In contrast, if the program was designed to test a hypothesis or research question and described, measured, or manipulated variables that could be generalized to a larger population, then the results of the evaluation study are more like research. However, both approaches use systematic tools and methods. For example, if the program to be evaluated is a local implementation of alerts and decision support for providers at the point of care to evaluate skin breakdown, then the stakeholders are the administrators, nurses, and patients who are affected by use of the decision support program. The evaluation questions would address the use of the program, the impact on resources, satisfaction, and perhaps clinical outcomes. The evaluation would likely use a before and after format and a more informal approach to assess whether the decision support worked. However, if the evaluation question is whether or not computerized guidelines affect behavior in general and under what conditions, then the specific stakeholders matter less and the ability to generalize beyond the contextual situation matters more. A more formal, research-based approach is then used. This question is really about what is to be learned. Vygotsky called these two approaches “patterning” versus “puzzling.”7 In the patterning approach to evaluation the comparison is between what went before at the local level, whereas in the puzzling approach the task is to do an in-depth comparison between different options and puzzle through the differences.

Another way to address this issue is to imagine that evaluation activities fall along a continuum from “program evaluation” to “evaluation research.” Program evaluation tends to have a wide scope, using multiple methods with a diverse range of outcomes. Evaluation research tends to be more targeted, using more selected methods and fewer outcomes. On the program evaluation end of the continuum, evaluation can encompass a range of activities including but not limited to program model development, needs assessment, tracking and performance monitoring, and continuous quality improvement. On the research end of the continuum, activities include theory testing, statistical evaluation of models, and hypothesis testing. At both ends of the continuum and in between, however, evaluators can use a variety of research designs, rigorous measurement methods, and statistical analyses.

Program Continuance versus Growth

Will the results of the evaluation be used to make a decision about continuing the program as is or about expanding it to a larger or different setting if it has generalizable knowledge? Answering the question of whether a program will be continued requires a focus on the concerns and goals of stakeholders as well as special attention to the contextual issues of cost, burden, user satisfaction, adoption, and effectiveness. Answering this question also requires an assessment about the manner in which the program was implemented and its feasibility in terms of resources and efforts. Does the program require ongoing and intense training of staff and technicians? Does it require unique hardware requirements that are a one-time or ongoing cost? Does the program have “legs” (i.e., can it exist on its own once implemented)? Are the benefits accrued available immediately or is it a long-term process?

One specific example to determine whether the program should be continued is to assess whether or not it contributes to the institution being a “learning organization.”8,9 This approach focuses on performance improvement and the following four areas of concern (modified for health IT):

1.What are the mental models and implicit theories held by the different stakeholders about the health IT product?

2.How does the health IT product promote mastery or personal control over the work environment?

3.What is the system-level impact of the health IT product? How does the intervention support a system thinking approach?

4.How does the health IT product create a unified vision of the information environment? Addressing this question requires understanding how individuals view the future computerized environment and whether or not they have come to a shared vision.

Theories and Frameworks

The use of theory in evaluation studies is controversial among evaluators as well as in the informatics community. On the one hand, some authors note that evaluation studies are local, limited in scope, and not intended to be generalizable. On the other hand, other authors argue that theory is necessary to frame the issues adequately, promote generalizable knowledge, and clarify measurement. This author would argue that theory should be used for the latter reason. Theoretical perspectives clarify the constructs and methods of measuring constructs as well as bring forward an understanding of mechanisms of action.

For the purposes of this chapter, theoretical perspectives will be divided into three levels of complexity. At the most complex level are the social science, cognitive engineering, and information science theories. These theories are well established, have a strong evidence base, and use validated measures and well-understood mechanisms of action. This chapter discusses some well-known social science theories as well as two health IT–specific adaptations of these theories that have significant validation. At the next level are the program implementation models, which are less complex and basically consist of a conceptual model. The models are often used to describe processes but few studies attempt to validate the models or test models against each other. Finally, at the most basic level are the program theory models, which are program specific and intended to represent the goals and content of a specific project. All evaluations for health IT products should develop a program theory model to guide the evaluation process itself. The descriptions below are brief and are intended to provide an overview of the possibilities at each level.

Social Science Theories

There are myriad theories relevant to both the design of interventions and products and the structure of the evaluation. A more detailed description of such theories as they apply to informatics is found in Chapter 2. A short description is provided here for the purpose of context. These social science theories include social cognitive theories, diffusion of innovation theory, cognitive engineering theories, and information theories.

Social Cognitive Theories

The social cognitive theories include the theory of planned behavior,10 its close relative the theory of reasoned action,11 and Bandura's social cognitive theory.12 These theories predict intentions and behavior as a function of beliefs about the value of an outcome, the likelihood that the outcome will occur given the behavior, and the expectations of others and self-efficacy beliefs about the personal ability to engage in the activity. The empirical validation of these theories is substantial and they have been used to predict intentions and behavior across a wide variety of settings.

Diffusion of Innovations Theory

Another very commonly used model in informatics is diffusion of innovations theory by Rogers.13,14 In this model characteristics of the innovation, the type of communication channels, the duration, and the social system are predictors of the rate of diffusion. The central premise is that diffusion is the process by which an innovation is communicated through certain channels over time among the members of a social system organization. Individuals pass through five stages: knowledge, persuasion, decision, implementation, and confirmation. Social norms, roles, and the type of communication channels all affect the rate of adoption of an innovation. Characteristics of an innovation that affect the rate of adoption include relative advantage as compared to other options; trialability, or the ease with which it can be tested; compatibility with other work areas; complexity of the innovation; and observability, or the ease with which the innovation is visible.

Cognitive Engineering Theories

The cognitive engineering theories have also been widely used in informatics, particularly naturalistic decision making (NDM),15–17 control theory,18 and situation awareness (SA).19 These theories focus more on the interaction between the context and the individual and are more likely to predict decision making, perception, and other cognitive variables. NDM is a broad and inclusive paradigm. SA is narrower and is particularly useful in supporting design.

SA combines the cognitive processes of orientation, attention, categorization or sense-making, and planning into three levels of performance. These activities are thought to be critical to human performance in complex environments. Endsley refers to a three-level system of awareness: (1) perception, (2) comprehension, and (3) projection. She defines shared SA as the group understanding of the situation.19 For example, in one recent study of health IT, higher SA was significantly associated with more integrated displays for intensive care unit (ICU) nursing staff.20,21 Table 5-1 presents the core components of SA and associated definitions.

TABLE 5-1 Levels of Situational Awareness

LEVEL

DESCRIPTION

Perception of the elements in the environment

What is present, active, salient, and important in the environment? Attention will be driven by task needs.

Comprehension of the current situation

Classification of the event is a function of activation of long-term memory. Meaning is driven by the cognitive process of classification and task identification.

Projection of future status

Expectations of outcomes in the future. Driven by implicit theories and knowledge about the causal mechanisms underlying events.

Information Theories

One of the most influential theories is information theory published in 1948 by Claude Shannon.22 Shannon focused on the mathematical aspects of the theory. Weaver, an oncologist, focused on the semantic meaning of the theory.23 Information theory identifies the degree of uncertainty in messages as a function of the capacity of the system to transmit those messages given a certain amount of noise and entropy. The transmission of information is broken down into source, sender, channel, receiver, and destination. Because information theory is essentially a theory of communication, information is defined relative to three levels of analysis:

1.At the technical or statistical level, information is defined as a measure of entropy or uncertainty in the situation. The question at this level is: How accurately are the symbols used in the communication being transmitted?

2.At the semantic level, information is defined as a reduction of uncertainty at the level of human meaning. The question here is: How well do the symbols that were transmitted convey the correct meaning?

3.At the effectiveness level, information is defined as a change in the goal state of the system. The question here is: How well does the perceived meaning effect the desired outcome?24

This simple framework can provide an effective evaluation model for any system that evaluates the flow of information. For example, Weir and McCarthy used information theory to develop implementation indicators for a computerized provider order entry (CPOE) project.25

Information foraging theory is a relatively new theory of information searching that has proven very useful for analyzing web searching.26 Information foraging theory is built on foraging theory, which studies how animals search for food. Pirolli and Card noticed similar patterns in the processes used by animals to search for food and the processes used by humans to search for information on the Internet. The basic assumption is that all information searches are goal directed and constitute a calibration between the energy cost of searching and the estimated value of information retrieved. The four concepts listed in Box 5-4 are important to measure. Empirical work on foraging theory has validated its core concepts.27

Box 5-4 Four Concepts of Foraging Theory

1. Information and its perceived value

2. Information patches or the temporal and spatial location of information clusters

3. Information scent or the presence of cues value and location of information

4. Information diet or the decision to pursue one source over another

Information Technology Theories

Two well-developed information technology theories are specifically used in the IT domain. As is true of many IT theories, they are compiled from several existing theories from the basic sciences to improve their fit in an applied setting.

Information System Success

An IT model that integrates several formal theories is DeLone and McLean's multifactorial model of IT success developed with the goal of improving scientific generalization.28 Their theory was originally developed in 1992 and revised in 2003 based on significant empirical support. The model is based on Shannon and Weaver's communication theory23 and Mason's information “influence” theory.29 DeLone and McLean used Shannon and Weaver's three levels of information: (1) technical (accuracy and efficiency of the communication system), (2) semantic (communicating meaning), and (3) effectiveness (impact on receiver). These three levels correspond to DeLone and McLean's constructs of (1) “systems quality,” (2) “information quality,” and (3) impact (use, user satisfaction, and outcomes). DeLone and McLean revised the model in 2003 to include recent literature and added a fourth level—“service quality”—referring to the degree to which users are supported by IT staff. Figure 5-1 depicts an adaptation of the updated 2003 model that adds user satisfaction, user characteristics, and task effectiveness to the original model.

Unified Theory of Acceptance and Use of Technology (UTAUT)

An adaptation of the social cognitive theories within the field of informatics is the unified theory of acceptance and use of technology (UTAUT).30 UTAUT, depicted in Figure 5-2, explains users' intentions to use an information system as a function of performance expectancy or self-efficacy beliefs, effort expectancies, social influence, and facilitating conditions. Significant moderators of these variables of intentions

FIG 5-1 A model of information system success

(Adapted from Journal of Management Information Systems, vol. 19, no. 4 [Spring 1993]: 9-30. Copyright © 1993 by M.E. Sharpe, Inc. Reprinted with permission. All Rights Reserved. Not for Reproduction.)

are gender, age, and the degree to which usage is mandated. This model integrates social cognitive theory,12 theory of reasoned action,11 and diffusion of innovations theory.13

Venkatesh and Davis conducted a systematic measurement meta-analysis that tested eight major models of adoption to clarify and integrate the adoption literature.31 All of the evaluated models were based to some degree on the social cognitive models described above but adapted to the question of IT adoption and use. Empirical studies showed that UTAUT explained around 70% of the variance in intention to use, significantly greater than any of the initial models alone. Two key findings of this work are important. First, Venkatesh et al. found that the variables associated with initial intentions to use are different than the variables associated with later intentions.30,32,33 Specifically, the perceived work effectiveness constructs (perceived usefulness, extrinsic motivation, job fit, relative advantage, and outcome expectations) were found to be highly predictive of intentions over time. In contrast, variables such as attitudes, perceived behavioral control, ease of use, self-efficacy, and anxiety were predictors only of early intentions to use.

Second, the authors found that the variables predictive of intentions to use are not the same as the variables predictive of usage behavior itself.30 They found that the “effort factor scale” (resources, knowledge, compatible systems, and support) was the only construct other than intention to significantly predict usage behavior. Finally, these authors found that the model differed significantly depending on whether usage was mandated or by choice. In settings where usage was mandated, social norms had a stronger relationship to intentions to use than the other variables.

Program Implementation Models

Program implementation models refer to generalized, large-scale implementation theories that are focused on

FIG 5-2 An example of unified theory of acceptance and use of technology.

(From Venkatesh V, Morris M, Davis G, Davis F. User acceptance of information technology: toward a unified view. MIS Quart. 2003;27[3]:425-478.)

performance improvement and institution-wide change. Four models are reviewed here: PRECEDE-PROCEED, PARiHS, RE-AIM, and quality improvement.

PRECEDE-PROCEED Model

The letters in the PRECEDE-PROCEED model represent the following terms: PRECEDE Predisposing, Reinforcing, and Enabling Constructs in Educational Diagnosis and Evaluation; PROCEED Policy, Regulatory, and Organizational Constructs in Educational and Environmental Development. This model was originally developed to guide the design of system-level educational interventions as well as evaluating program outcomes. It is a model that addresses change at several levels, ranging from the individual to the organization level. According to the model the interaction between the three types of variables produces change: predisposing factors that lay the foundation for success (e.g., electronic health records or strong leadership), reinforcing factors that follow and strengthen behavior (e.g., incentives and feedback), and enabling factors that activate and support the change process (e.g., support, training, computerized reminders, and templates or exciting content). The model has been applied in a variety of settings, ranging from public health interventions, education, and geriatric quality improvement and alerting studies.34 Figure 5-3 illustrates the model applied to a health IT product.

Promoting Action on Research Implementation in Health Services (PARiHS)

The PARiHS framework outlines three general areas associated with implementation (Fig. 5-4):

Inc.)

FIG 5-4 The PARiHS model.

(From Kitson AL, Rycroft-Malone J, Harvey G, McCormack B, Seers K, Titchen A. Evaluating the successful implementation of evidence into practice using the PARiHS framework: theoretical and practical challenges. Implement Sci. 2008;3:1.)

2.Context: Enhancing leadership support and integrating with the culture through focus groups and interviews

3.Facilitation of the implementation process: Skill level and role of facilitator in promoting action as well as frequency of supportive interactions35

These three elements are defined across several subelements, with higher ratings suggestive of more successful implementation. For instance, a high level of evidence may include the presence of randomized controlled trials (i.e., the gold standard in research), high levels of consensus among clinicians, and collaborative relationships between patients and providers. Context is evaluated in terms of readiness for implementation, including consideration of the context's culture, leadership style, and measurement practices. High ratings of context may indicate an environment focused on continuing education, effective teamwork, and consistent evaluation and feedback. Finally, the most successful facilitation is characterized by high levels of respect for the implementation setting, a clearly defined agenda and facilitator role, and supportive flexibility.36 Successful implementation (SI) is thus conceptualized as a function (f) of evidence (E), context (C), and facilitation (F), or SI = f(E, C, F).37 The PARiHS framework suggests a continuous multidirectional approach to evaluation of implementation. Importantly, evaluation is a cyclic and interactive process instead of a linear approach.

Reach Effectiveness Adoption Implementation Maintenance (RE-AIM)

RE-AIM was designed to address the significant barriers associated with implementation of any new intervention and is particularly useful for informatics. Most interventions meet with significant resistance and any useful evaluation should measure the barriers associated with the constructs (Reach, Effectiveness, Adoption, Implementation, and Maintenance). These constructs serve as a good format for evaluation.38,39 First, did the intervention actually reach the intended target population? In other words, how many providers had the opportunity to use the system? Or, how many patients had access to a new website? Second, for effectiveness, did the intervention actually do what it was intended to do? Did the wound care decision support actually work as intended every time? Did it identify the patients it was supposed to identify? Or, did the algorithms miss some key variables in real life? Third, for adoption, what proportion of the targeted staff, settings, or institutions actually used the program? What was the breadth and depth of usage? Did they use it for all relevant patients or only for some? Fourth, for implementation, was the intervention the same across settings and time? With most health IT products there is constant change to the software, the skill level of users, and the settings in which they are used. These should be documented and addressed in the evaluation. Finally, for maintenance, some period of time should be identified a priori to assess maintenance and whether usage continues and by whom and in what form. For health IT interventions it is especially useful to look for unintended consequences as well as workarounds during implementation and maintenance in particular.

Quality Improvement

Evaluation activities using a quality improvement framework are often based on Donabedian's classic structure-process-outcome (SPO) model for assessing healthcare quality.40,41 Donabedian defines structural measures of quality as the professional and organizational resources associated with the provision of care, such as IT staff credentials, CPOE systems, or staffing ratios. Process measures include the tasks and decisions imbedded in care, such as the time to provision of antibiotics or the proportion of patients on deep vein thrombosis (DVT) prevention protocols. Finally, outcomes are defined as the final or semifinal measurable outcomes of care, such as the number of amputations due to diabetes, the number of patients with DVT, or the number of patients with drug-resistant pneumonia. These three categories of variables are thought to be mutually interdependent and reinforcing. A model for patient safety and quality research design (PSQRD) expands on this well-known, original SPO model.42 For more details on this model, see Chapter 20.

Program Theory Models

Program theory models are the most basic and practical of the evaluation models. They are the implicit theories of stakeholders and participants of the proposed program. A detailed program theory identifies variables, the timing of measures and observations, and the key expectations that reflect their understandings. Most importantly, a program theory model serves as a shared vision between the evaluator team and the participants, creating a unified conceptual model that guides all evaluation activities.

Six Steps

Program theory evaluation is recommended practice for all program evaluation and an important approach regardless of whether the program is the implementation of a new documentation system or an institution-wide information system. Invariably, various participants will have different ideas about what the program is and why it works.43 Creation of a program theory model serves to align the various stakeholders into a single view. The Centers for Disease Control and Prevention's six-step program evaluation framework is one of the best examples in current use.44 It recommends the following six steps:

1.Engage stakeholders to ensure that all partners have contributed to the goals of the program and the metrics to measure its success.

2.Describe the program systematically to identify goals, objectives, activities, resources, and context. This description process involves all stakeholders.

3.Focus the evaluation design to assess usefulness, feasibility, ethics, and accuracy.

4.Gather credible evidence by collecting data, conducting interviews, and measuring outcomes using a good research design.

5.Justify conclusions using comparisons against standards, statistical evidence, or expert review.

6.Ensure use and share lessons learned by planning and implementing dissemination activities.

Logic Models

A logic model is a representation of components and mechanisms of the program as noted by the authors of the W.K. Kellogg Foundation's guide to logic models:

Basically, a logic model is a systematic and visual way to present and share your understanding of the relationship among the resources you have to operate your program, the activities you plan, and the changes or results you hope to achieve. The most basic logic model is a picture of how you believe your program will work. It uses words and/or pictures to describe the sequence of activities thought to bring about change and how these activities are linked to the results the program is expected to achieve.45(p1)

The basic structure of a logic model is illustrated in Figure 5-5. The logic model starts with a category of “inputs,” which include staff, resources, prior success, and stakeholders. In health IT the inputs are the programs, software, the IT staff, hardware, networks, and training. “Outputs” include the activities that are going to be conducted, such as training, implementation, software design, and other computer activities. Participation refers to the individual involved. Outcomes are divided into short, medium, and long outcomes or short-term versus long-term outcomes. The goal is to make sure that the components of the program are easy to see and the mechanisms are made explicit.

The specific methods used to apply a model are not prescribed (although creating a logic model is recommended). Use of a wide range of methods and tools is encouraged. Engaging the stakeholders is the first step and that process could involve needs assessment, functionality requirements analyses, cognitive task analyses, contextual inquiry, and ethnographic observation, to name a few approaches. In all cases, the result is the ability to provide a deep description of the program, the expected mechanisms, and the desired outcomes. Once there is agreement on the characteristics of the program at both the superficial level and the deeper structure, designing the evaluation is straightforward. Agreement among stakeholders is needed not only to identify concepts to measure, but also to determine how to meaningfully measure them.

In a health IT product implementation, agreement, especially about goals and metrics, should be made early in the program implementation process. Because health IT products are unique, creating a shared vision and a common understanding of the meaning of the evaluation can be challenging. Using an iterative process for implementation can mitigate this problem where design and implementation go hand in hand and the stakeholder's vision is addressed repeatedly throughout the process.

Methods, Tools, and Techniques

The need for variety in methods is driven by the diversity in population, types of projects, and purposes that are characteristic of research in informatics. Many evaluation projects are classified as either qualitative or quantitative. This division may be somewhat artificial and limited but using the terms qualitative and quantitative to organize methods helps to make them relatively easy to understand. A central thesis of this section is that the choice of method should fit the question and multiple methods are commonly used in evaluations.

Quantitative versus Qualitative Questions

Because evaluation is often a continuous process throughout the life of a project, the systems life cycle is used to organize this discussion. Evaluation activities commonly occur during the planning, implementation, and maintenance stages of a project (see Chapter 2 for more details about these stages and others in the systems life cycle). At each stage both quantitative and qualitative questions might be asked. At the beginning of a project the goal is to identify resources,

FIG 5-5 Example of a logic model structure.

(From University of Wisconsin–Extension, Cooperative Extension, Program Development and Evaluation website. Logic model. http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html. 2010.)

feasibility, values, extent of the problem to be solved, and types of needs. The methods used to answer these questions are essentially local and project specific. During the project, evaluation questions focus on the intensity, quality, and depth of the implementation as well as the evolution of the project team and community. Finally, in the maintenance phase of the project the questions focus on outcomes, cost–benefit value, or overall consequences.

Table 5-2 presents a matrix that outlines the different stages of a project and the types of questions that might be asked during each stage. The questions are illustrations of possible evaluation questions and loosely categorized as either qualitative or quantitative.

Qualitative Methods

Many individuals believe that qualitative methods refer to research procedures that collect subjective human-generated data. However, subjective data can be quantitative, such as the subjective responses to carefully constructed usability questionnaires used as outcome end points. Diagnostic codes are another example of quantitative forms of subjective data. Qualitative methods, rather, refer to procedures and methods that produce narrative or observational descriptive data that are not intended for transformation into numbers. Narrative data refer to information in the form of stories, themes, meanings, and metaphors. Collecting this information requires the use of systematic procedures where the purpose is to understand and explore while minimizing bias. There are several very good guides to conducting qualitative research for health IT evaluation studies.46

TABLE 5-2 Evaluation Research Questions By Stage of Project and Type of Question

TYPES OF QUESTIONS

STAGE OF PROJECT

QUALITATIVE

QUANTITATIVE

Planning

What are the values of the different stakeholders?

What are the expectations and goals of participants?

What is the prevalence of the problem being addressed by the health IT product?

What are the resources available?

What is the relationship between experiencing a factor and having a negative outcome?

What are the intensity and depth of use of health IT tools?

Implementation

How are participants experiencing the change?

How does the health IT product change the way individuals relate to or communicate with each other?

How many individuals are participating?

What changes in performance have been visible? What is the compliance rate?

How many resources are being used during implementation (when and where)?

Maintenance

How has the culture of the situation changed?

What themes underscore the participants' experience?

What metaphors describe the change?

What are the participants' personal stories?

Is there a significant change in outcomes for patients?

Did compliance rates increase (for addressing the initial problem)?

What is the rate of adverse events?

Is there a significant change in efficiency?

Is there a correlation between usage and outcomes?

IT, Information technology.

Structured and Semistructured Interviews

In-person interviews can be some of the best sources of information about an individual's unique perspectives, issues, and values. Interviews vary from a very structured set of questions conducted under controlled conditions to a very informal set of questions asked in an open-ended manner. Typically, evaluators audio-record the interviews and conduct thematic or content coding on the results. For example, user interviews that focus on how health IT affects workflow are especially useful. Some interviews have a specific focus, such as in the critical incident method.47 In this method an individual recalls a critical incident and describes it in detail for the interviewer. In other cases the interview may focus on the individual's personal perceptions and motivations, such as in motivational interviewing.48 Finally, cognitive task analysis (CTA) is a group of specialized interviews and observations where the goal is to deconstruct a task or work situation into component parts and functions.49–51 A CTA usually consists of targeting a task or work process and having the participant walk through or simulate the actions, identifying the goals, strategies, and information needs. These latter methods are useful for user experience studies outlined in Chapter 21.

Observation and Protocol Analysis

An interview may not provide enough information and therefore observing users in action at work is necessary to fully understand the interactions of context, users, and health IT. Observation can take many forms, from using a video camera to a combination of observation and interview where individuals “think aloud” while they work. The think-aloud procedures need to be analyzed both qualitatively for themes and quantitatively for content, timing, and frequency.52

Interviews, CTAs, and observation are essential in almost every health IT evaluation of users. Technology interventions are not uniform and cannot simply be inserted into the workflow in a “plug and play” manner. In addition, the current state of the literature in the field lacks clarity about the mechanisms of action or even delineating the key components of health IT. Thus understanding the user's response to the system is essential. For more information about the user experience, see Chapter 21.

Ethnography and Participant Observation

Ethnography and participant observation are derived from the field of anthropology, where the goal is to understand the larger cultural system through observer immersion. The degree of immersion can vary, as can some of the data collection methods, but the overall strategy includes interacting with all aspects of the context. Usually ethnography requires considerable time, multiple observations, interviews, and living and working in the situation if possible. It also includes reading historical documents, exploring artifacts in current use (e.g., memos, minutes), and generally striving to understand a community. These methods are particularly useful in a clinical setting, where understanding the culture is essential.46,53

Less intensive ethnographic methods are also possible and reasonable for health IT evaluations. Focused ethnography is a method of observing actions in a particular context. For example, nurses were observed during patient care hand-offs and their interactions with electronic health records were recorded. From these observations design implications for hand-off forms were derived.54,55

Quantitative Methods

Research Designs

Quantitative designs range from epidemiologic, descriptive studies to randomized controlled trials. Three study designs presented may be particularly useful for health IT and clinical settings. Each of these designs takes advantage of the conditions that are commonly found in health IT projects, including automatically collected data, the ubiquitous use of pre-post design, and outcome-based targets for interventions.

Time Series Analysis

This design is an extension of the simple pre-post format but requires multiple measures in time prior to and after an intervention such as health IT. Evidence of the impact is found in the differences in mathematical slopes between measures during the pre-test and post-test periods. This design has significantly more validity than a simple pre-post one-time measure design and can be very feasible in clinical settings where data collection is automatic and can be done for long periods with little increase in costs. For example, top-level administrators might institute a decision support computerized program to improve patient care for pain management. A straightforward design is to measure the rate of compliance to pain management recommendations during several periods about 12 months prior and several periods up to 12 months after implementation, controlling for hospital occupancy, patient acuity, and staffing ratios. This design is highly recommended for health IT implementations where data can be captured electronically and reliably over long periods.56,57

Regression Discontinuity Design

A regression discontinuity design is similar to a time series analysis but is a formal statistical analysis of the pattern of change over time for two groups. This design is particularly suited to community engagement interventions, such as for low vaccination rates in children or seat belt reminders. Participants are divided into two nonoverlapping groups based on their prescores on the outcome of interest. The example used above of compliance with pain management guidelines in an inpatient surgical unit may also be applicable. Providers with greater than 50% compliance to pain guidelines are put in one group and those with less than 50% compliance with pain guidelines are in the other group. Those with the lowest compliance receive a decision support intervention such as a computerized alert and a decision support system to assess and treat pain while the rest do not. Post measures are taken some time after the intervention and the difference between the predicted scores of the low compliance group and their actual scores as compared to the nonintervention group are noted. The validity of this design is nearly as high as a randomized controlled trial, but this design is much easier to implement because those who need the intervention

FIG 5-6 Example of a multiple baseline study.

receive it. However, this design requires large numbers, which may not be available except in a system-wide or multisite implementation.58

Multiple Baseline with Single Subject Design

This design adds significant value to the standard pre-post comparison by staggering implementation systematically (e.g., at 3-month intervals) over many settings but the measurement for all settings starts at the same time. In other words, measurement begins at the same time across five clinics but implementation is staggered every 3 months. Figure 5-6 illustrates the pattern of responses that might be observed. The strength of the evidence is high if outcomes improved after implementation in each setting and they followed the same pattern.59

Instruments

Data are commonly gathered through the use of instruments. User satisfaction, social network analyses, and cost-effectiveness tools are discussed briefly below.

User Satisfaction Instruments

User satisfaction is commonly measured as part of health IT evaluations but is a complex concept. On the one hand it is thought to be a proxy for adoption and on the other hand it is used as a proxy for system effectiveness. In the first conception, the constructs of interest would be usability and ease of use and whether others use it (usability and social norms). In the second conception, the constructs of interest would refer to how well the system helped to accomplish task goals (usefulness). One of the most common instruments for evaluating user satisfaction is the UTAUT.31 This well-validated instrument assesses perceived usefulness, social norms and expectations, perceived effort, self-efficacy, ease of use, and intentions to use. Reliability for these six scales ranges from 0.92 to 0.95.

A second measure of user satisfaction focuses on service quality (SERVQUAL) and assesses the degree and quality of IT service. Five scales have been validated: reliability, assurance, tangibles, empathy, and responsiveness. These five scales have been found to have reliability of 0.81 to 0.94.60

A third measure is the system usability scale (SUS), which is widely used outside health IT.61 It is a 10-item questionnaire applicable to any health IT product. Bangor and colleagues endorsed the SUS above other available instruments because it is technology agnostic (applicable to a variety of products) and easy to administer and the resulting score is easily interpreted.62 The authors provide a case study of product iterations and corresponding SUS ratings that demonstrate the sensitivity of the SUS to improvements in usability.

Social Network Analysis

Methods that assess the linkages between people, activities, and locations are likely to be very useful for understanding a community and its structure. Social network analysis (SNA) is a general set of tools that calculates the connections between people based on ratings of similarity, frequency of interaction, or some other metric. The resultant pattern of connection is displayed as a visual network of interacting individuals. Each node is an individual and the lines between nodes reflect the interactions. Although SNA uses numbers to calculate the form of the networked display, it is essentially a qualitative technique because the researcher must interpret the patterns of connections and describe them in narrative form. Conducting an SNA is useful if the goal is to understand how an information system affected communication between individuals. It is also useful to visualize nonpeople connections, such as the relationship between search terms or geographical distances.63 For example, Benham-Hutchins used SNA to examine patient care hand-offs from the emergency department to inpatient areas, finding that each hand-off entailed 11 to 20 healthcare providers.64

Cost-Effectiveness Analysis

Cost-effectiveness analysis (CEA) attempts to quantify the relative costs of two or more options. Simply measuring additional resources, start-up costs, and labor would be a rudimentary cost analysis. A CEA is different than a cost–benefit analysis, which gives specific monetary analysis. A simple CEA shows a ratio of the cost divided by the change in health outcomes or behavior. For example, a CEA might compare the cost of paying a librarian to answer clinicians' questions as compared to installing Infobuttons per the number of known questions. Most CEA program evaluations will assess resource use, training, increased staff hiring, and other cost-related information. This would not necessarily be a full economic analysis that would require a consultation with an economist. The specific resources used could be delineated in the logic model, unless it was part of hypothesis testing in a more formal survey. The reader is directed to a textbook if further information is needed.37

Conclusion and Future Directions

Evaluation of health IT programs and projects can range from simple user satisfaction for a new menu to full-scale analysis of usage, cost, compliance, patient outcomes, observation of usage, and data about patients' rate of improvement. Starting with a general theoretical perspective and distilling it to a specific program model is the first step in evaluation. Once overall goals and general constructs have been identified, then decisions about measurement and design can be made. In this chapter evaluation approaches were framed, focusing on health IT program evaluation to orient the reader to the resources and opportunities in the evaluation domain. Health IT evaluations are typically multidimensional, longitudinal, and complex. Health IT interventions and programs present a unique challenge, as they are rarely independent of other factors. Rather, they are usually embedded in a larger program. The challenge is to integrate the goals of the entire program while clarifying the impact and importance of the health IT component. In the future, health IT evaluations should become more theory driven and the complex nature of evaluations will be acknowledged more readily.

As health IT becomes integrated at all levels of the information context of an institution, evaluation strategies will necessarily broaden in scope. Outcomes will not only include those related to health IT but span the whole process. The result will be richer analyses and a deeper understanding of the mechanisms by which health IT has its impact. The incorporation of theory into evaluation will also result in more generalizable knowledge and the development of health IT evaluation science. Health practitioners and informaticists will be at the heart of these program evaluations due to their central place in healthcare, IT, and informatics departments.

References

1.

Patton, MQ: Utilization-Focused Evaluation. 4th ed, 2008, Sage, London.

2.

Ainsworth, L, Viegut, D: Common Formative Assessments. 2006, Corwin Press, Thousand Oaks, CA.

3.

Fetterman, D: Foundations of Empowerment Evaluation. 2001, Sage Publications, Thousand Oaks, CA.

4.

Scriven, M: The methodology of evaluation. In Stake, R (Ed.): Curriculum Evaluation, American Educational Research Association, Monograph Series on Evaluation, No 1. 1967, Rand McNally, Chicago.

5.

Christensen, C: The Innovator's Dilemma. 2003, HarperBusiness, New York, NY.

6.

Shekelle, P: Costs and Benefits of Health Information Technology. 2006, Southern California Evidence-Based Practice Center, Santa Monica, CA.

7.

Vygotsky, L: Mind and Society. 1978, Harvard University Press, Cambridge, MA.

8.

Preskill, H, Torres, R: Building capacity for organizational learning through evaluative inquiry. Evaluation. 5(1), 1999, 42–60.

9.

Senge, P: The Fifth Discipline: The Art and Practice of the Learning Organization. 1990, Doubleday, New York, NY.

10.

Ajzen, I: The theory of planned behavior. Organ Behav Hum Decis Process. 50, 1991, 179–211.

11.

Fishbein, M, Ajzen, I: Belief, Attitude, Intention, and Behavior: An Introduction to Theory and Research. 1975, Addison-Wesley, Reading, MA.

12.

Bandura, A: Human agency in social cognitive theory. Am Psychol. 44, 1989, 1175–1184.

13.

Rogers, EM: Diffusion of Innovations. 1983, Free Press, New York, NY.

14.

Cooper, RB, Zmud, RW: Information Technology Implementation Research: A Technological Diffusion Approach. Management Science. 36, 1990, 123–139.

15.

Klein, G: An overview of natural decision making applications. In Zsambok, CE, Klein, G (Eds.): Naturalist Decision Making. 1997, Lawrence Erlbaum Associates, Mahwah, NJ.

16.

Kushniruk, AW, Patel, VL: Cognitive and usability engineering methods for the evaluation of clinical information systems. J Biomed Inform. 37(1), 2004, 56–76.

17.

Zsambok, C, Klein, G: Naturalistic Decision Making. 1997, Lawrence Erlbaum, Mahwah, NJ.

18.

Åström, K, Murray, R: Feedback Systems: An Introduction for Scientists and Engineers. 2008, Princeton University Press, Princeton, NJ.

19.

Endsley, M, Garland, D: Situation Awareness Analysis and Measurement. 2000, Lawrence Erlbaum, Mahway, NJ.

20.

Koch, S, Weir, C, Haar, M, et al.: ICU nurses’ information needs and recommendations for integrated displays to improve nurses’ situational awareness. J Am Med Inform Assn. 2012, March 21 [e-pub].

21.

Koch SH, Weir C, Westenskow D, et al. Evaluation of the effect of information integration in displays for ICU nurses on situation awareness and task completion time: a prospective randomized controlled study. J Am Med Inform Assn. In press.

22.

Shannon, C: A mathematical theory of communication. Bell Syst Tech J. 27, 1948, 379–423, 623-656.

23.

Shannon, C, Weaver, W: The Mathematical Theory of Communication. 1949, University of Illinois Press, Urbana, IL.

24.

Krippendorf, K: Information Theory: Structural Models for Qualitative Data. 1986, Sage Publications, Thousand Oaks, CA.

25.

Weir, CR, McCarthy, CA: Using implementation safety indicators for CPOE implementation. Jt Comm J Qual Patient Saf. 35(1), 2009, 21–28.

26.

Pirolli, P, Card, S: Information foraging. Psychol Rev. 106(4), 1999, 643–675.

27.

Pirolli, P: Information Foraging Theory: Adaptive Interaction with Information. 2007, Oxford University Press, Oxford, United Kingdom.

28.

DeLone, W, McLean, E: The DeLone and McLean model of information systems success: a ten-year update. J Man Inf Syst. 19(4), 2003, 9–30.

29.

Mason, R: Measuring information output: a communication systems approach. Information & Management. 1(5), 1978, 219–234.

30.

Venkatesh, V, Morris, M, Davis, G, Davis, F: User acceptance of information technology: toward a unified view. MIS Quart. 27(3), 2003, 425–478.

31.

Venkatesh, V, Davis, F: A theoretical extension of the technology acceptance model: four longitudinal field studies. Manage Sci. 46(2), 2000, 186–204.

32.

Agarwal, R, Prasad, J: A conceptual and operational definition of personal innovativeness in the domain of information technology. Inform Syst Res. 9(2), 1998, 204–215.

33.

Karahanna, E, Straub, D, Chervany, N: Information technology adoption across time: a cross-sectional comparison of pre-adoption and post-adoption beliefs. MIS Quart. 23(2), 1999, 183–213.

34.

Green, L, Kreuter, M: Health Program Planning: An Educational and Ecological Approach. 4th ed, 2005, Mayfield Publishers, Mountain View, CA.

35.

Stetler, CB, Damschroder, LJ, Helfrich, CD, Hagedorn, HJ: A guide for applying a revised version of the PARiHS framework for implementation. Implement Sci. 6, 2011, 99, doi:1748-5908-6-99 [pii].

36.

Kitson, A, Harvey, G, McCormack, B: Enabling the implementation of evidence based practice: a conceptual framework. Qual Health Care. 7(3), 1998, 149–158.

37.

Kitson, AL, Rycroft-Malone, J, Harvey, G, McCormack, B, Seers, K, Titchen, A: Evaluating the successful implementation of evidence into practice using the PARiHS framework: theoretical and practical challenges. Implement Sci. 3, 2008, 1.

38.

Gaglio, B, Glasgow, R: Evaluation approaches for dissemination and implementation research. In Brownson, R, Colditz, G, Proctor, E (Eds.): Dissemination and Implementation Research in Health: Translating Science to Practice. 2012, Oxford University Press, New York, NY, 327–356.

39.

Glasgow, R, Vogt, T, Boles, S: Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. 89(9), 1999, 1922–1927.

40.

Donabedian, A: Explorations in Quality Assessment and Monitoring: The Definition of Quality and Approaches to Its Assessment. Vol. 1. 1980, Health Administration Press, Ann Arbor, MI.

41.

Donabedian, A: The quality of care: how can it be assessed?. JAMA. 260, 1988, 1743–1748.

42.

Brown, C, Hofer, T, Johal, A, et al.: An epistemology of patient safety research: a framework for study design and interpretation. Part 1: conceptualising and developing interventions. Qual Saf Health Care. 17(3), 2008, 158–162.

43.

Friedman, C: Information technology leadership in academic medical centers: a tale of four cultures. Acad Med. 74(7), 1999, 795–799.

44.

Centers for Disease Control and Prevention: framework for program evaluation in public health. MMWR. 48(RR11), 1999, 1–40.

45.

W.K. Kellogg Foundation: Using Logic Models to Bring Together Planning, Evaluation, and Action: Logic Model Development Guide. 2004, W.K. Kellogg Foundation, Battle Creek, MI.

46.

Patton, M: Qualitative Research and Evaluation Methods. 2001, Sage Publications, 3rd ed. Newberry Park, CA.

47.

Flanagan, JC: The critical incident technique. Psychol Bull. 51(4), 1954, 327–358.

48.

Miller, WR, Rollnick, S: Motivational Interviewing: Preparing People to Change. 2nd ed, 2002, Guilford Press, New York, NY.

49.

Crandall, B, Klein, G, Hoffman, R: Working Minds: A Practitioner's Guide to Cognitive Task Analysis. 2006, MIT Press, Cambridge, MA.

50.

Hoffman, RR, Militello, LG: Perspectives on Cognitive Task Analysis. 2009, Psychology Press Taylor and Francis Group, New York, NY.

51.

Schraagen, J, Chipman, S, Shalin, V: Cognitive Task Analysis. 2000, Lawrence Erlbaum Associates, Mahway, NJ.

52.

Ericsson, K, Simon, H: Protocol Analysis: Verbal Reports as Data. Rev. ed, 1993, MIT Press, Cambridge, MA.

53.

Kaplan, B, Maxwell, J: Qualitative research methods for evaluating computer information systems. In Anderson, JG, Aydin, CE, Jay, SJ (Eds.): Evaluating Health Care Information Systems: Approaches and Applications. 1994, Sage, Thousand Oaks, CA, 45–68.

54.

Staggers, N, Clark, L, Blaz, J, Kapsandoy, S: Nurses’ information management and use of electronic tools during acute care handoffs. Western J Nurs Res. 34(2), 2012, 151–171.

55.

Staggers, N, Clark, L, Blaz, J, Kapsandoy, S: Why patient summaries in electronic health records do not provide the cognitive support necessary for nurses’ handoffs on medical and surgical units: insights from interviews and observations. Health Informatics J. 17(3), 2011, 209–223.

56.

Harris, A, McGregor, J, Perencevich, E, et al.: The use and interpretation of quasi-experimental studies in medical informatics. J Am Med Inform Assoc. 13, 2006, 16–23.

57.

Ramsay, C, Matowe, L, Grill, R, Grimshaw, J, Thomas, R: Interrupted time series designs in health technology assessment: lessons from two systematic reviews of behavior change strategies. Int J Technol Assess. 19(4), 2003, 613–623.

58.

Lee, H, Monk, T: Using regression discontinuity design for program evaluation. American Statistical Association–Proceedings of the Survey Research Methods Section . 2008, ASA, Retrieved from http://www.amstat.org/sections/srms/Proceedings/.

59.

Shadish, WR, Cook, TD, Campbell, DT: Experimental and Quasi-Experimental Designs for Generalized Causal Inference. 2002, Houghton Mifflin, Boston, MA.

60.

Pitt, L, Watso, NR, Kavan, C: Service quality: a measure of information systems effectiveness. MIS Quart. 19(2), 1995, 173–188.

61.

Sauro, J: Measuring usability with the system usability scale (SUS). Measuring Usability LLC http://www.measuringusability.com/sus.php, 2012, February 2, 2011. Accessed September 27.

62.

Bangor, A, Kortum, P, Miller, JT: An empirical evaluation of the system usability scale. Int J Hum-Comput Int. 24(6), 2008, 574–594.

63.

In Durland, M, Fredericks, K (Eds.): New Directions in Evaluation: Social Network Analysis. 2005, Jossey-Bass/AEA, Hoboken, NJ.

64.

Benham-Hutchins, MM, Effken, JA: Multi-professional patterns and methods of communication during patient handoffs. Int J Med Inform. 79(4), 2010, 252–267.

Discussion Questions

1. Of the levels of theory discussed in this chapter, what level would be most appropriate for evaluation of electronic health records? Would the level of theory be different if the intervention was for an application targeting a new scheduling system in a clinic? Why?

2. What is the difference between program evaluation and program evaluation research?

3. Assume that you are conducting an evaluation of a new decision support system for preventative alerts. What kind of designs would you use in a program evaluation study?

4. Using the life cycle as a framework, explain when and why you would use a formative or summative evaluation approach.

5. What are the basic differences between a research study and a program evaluation?

6. Review the following article: Harris A, McGregor J, Perencevich E, et al. The use and interpretation of quasi-experimental studies in medical informatics. J Am Med Inform Assoc. 2006;13:16-23. Explain how you might apply these research designs in structuring a program evaluation.

Case Study

A 410-bed hospital has been using a homegrown provider order entry system for 5 years. It has recently decided to put in bar code administration software to scan medications at the time of delivery in order to decrease medical error. The administration is concerned about medication errors, top-level administration is concerned about meeting The Joint Commission accreditation standards, and the IT department is worried that the scanners may not be reliable and may break, increasing their costs. The plan is to have a scanner in each patient's room; nurses will scan the medication when they get to the room and also scan their own badges and the patient's arm band. The application makes it possible to print out a list of the patients with their scan patterns and the nurses sometimes carry this printout because patient's arm bands can be difficult to locate or nurses do not want to disturb patients while they are sleeping. The bar code software was purchased from a vendor and the facility has spent about a year refining it. The IT department is responsible for implementation and has decided that it will implement each of the four inpatient settings one at a time at 6-month intervals.

The hospital administration wants to conduct an evaluation study. You are assigned to be the lead on the evaluation.

Discussion Questions

1. What is the key evaluation question for this project?

2. Who are the stakeholders?

3. What level of theory is most appropriate?

4. What are specific elements to measure by stakeholder group?

Pageburst Integrated Resources

As part of your Pageburst Digital Book, you can access the following Integrated Resources:

Bibliography and Additional Readings

Web Resources

Chapter 5 Program Evaluation and Research Techniques

Charlene R. Weir

Evaluation of health information technology (health IT) programs and projects can range from simple

user satisfaction for a new menu or full

-

scale analysis of usage, cost, compliance, patient outcomes, and

observation of usage to data about patient's r

ate of improvement.

Objectives

At the completion of this chapter the reader will be prepared to:

1.Identify the main components of program evaluation

2.Discuss the differences between formative and summative evaluation

3.Apply the three level

s of theory relevant to program evaluation

4.Discriminate program evaluation from program planning and research

5.Synthesize the core components of program evaluation with the unique characteristics of

informatics interventions

Key Terms

Evaluatio

n, 72

Formative evaluation, 73

Logic model, 79

Program evaluation, 73

Summative evaluation, 73

Abstract

Evaluation is an essential component in the life cycle of all health IT applications and the key to

successful translation of these applications into cl

inical settings. In planning an evaluation the central

questions regarding purpose, scope, and focus of the system must be asked. This chapter focuses on the

larger principles of program evaluation with the goal of informing health IT evaluations in clinic

al

settings. The reader is expected to gain sufficient background in health IT evaluation to lead or

participate in program evaluation for applications or systems.

Formative evaluation and summative evaluation are discussed. Three levels of theory are pres

ented,

including scientific theory, implementation models, and program theory (logic models). Specific

Chapter 5 Program Evaluation and Research Techniques

Charlene R. Weir

Evaluation of health information technology (health IT) programs and projects can range from simple

user satisfaction for a new menu or full-scale analysis of usage, cost, compliance, patient outcomes, and

observation of usage to data about patient's rate of improvement.

Objectives

At the completion of this chapter the reader will be prepared to:

1.Identify the main components of program evaluation

2.Discuss the differences between formative and summative evaluation

3.Apply the three levels of theory relevant to program evaluation

4.Discriminate program evaluation from program planning and research

5.Synthesize the core components of program evaluation with the unique characteristics of

informatics interventions

Key Terms

Evaluation, 72

Formative evaluation, 73

Logic model, 79

Program evaluation, 73

Summative evaluation, 73

Abstract

Evaluation is an essential component in the life cycle of all health IT applications and the key to

successful translation of these applications into clinical settings. In planning an evaluation the central

questions regarding purpose, scope, and focus of the system must be asked. This chapter focuses on the

larger principles of program evaluation with the goal of informing health IT evaluations in clinical

settings. The reader is expected to gain sufficient background in health IT evaluation to lead or

participate in program evaluation for applications or systems.

Formative evaluation and summative evaluation are discussed. Three levels of theory are presented,

including scientific theory, implementation models, and program theory (logic models). Specific