RESEARCH CONCEPTS FOR HIM - Research Topic/Problem/Question
lynda3495
AB120916_Ch06.pptxHealth Informatics Research Methods: Principles and Practice, Second Edition
Chapter 6: Evaluation Methods
© 2017 American Health Information Management Association
© 2017 American Health Information Management Association
1
Learning Objectives
Demonstrate appropriate use of terms related to evaluation methods and theory.
Explain the rationale for conducting an evaluation or evaluation research.
Apply evaluation or evaluation research methods to a scenario.
Outline the steps in an evaluation of health information technology.
© 2017 American Health Information Management Association
2
Introduction: Evaluation and Evaluation Research
Evaluation
A process to determine perceived value by stakeholders
Evaluation research
A specific kind of research that includes social science methods and a theoretic framework or model as part of exploratory or hypothesis-driven research
© 2017 American Health Information Management Association
Evaluation Research Methods
Use a theoretic framework
Exploratory or hypothesis driven
Determine the variables based on theoretic constructs
Develop research procedures to study the theoretic constructs
Collect and analyze the data
© 2017 American Health Information Management Association
Forms of Evaluation and Evaluation Research
Two categories of evaluation
Formative evaluation occurs during the program cycle to gather feedback from stakeholders or end-users to adjust the process or the software so that the desired effects are achieved.
Summative evaluation occurs at or near the end of the project and is undertaken to assess effectiveness of the program.
© 2017 American Health Information Management Association
5
Formative Evaluation
Needs assessment
The process of determining, analyzing, and prioritizing needs, and in turn, identifying and implementing solution strategies to resolve high-priority needs
Implementation evaluation
Monitoring how well the planned events are actually occurring
Whether the implementation is meeting the expected timeframes
Process evaluation
How well the project was carried out
© 2017 American Health Information Management Association
6
Examples of Formative Evaluation
Improvement in delivery methods with regard to technology used
The quality of implementation of a new process or technology
Information about the organizational placement of a given process
How well a planned process aligned with an actual process
© 2017 American Health Information Management Association
7
Summative Evaluation
Two types
Impact evaluation
Assesses the intended or unintended net effects of the program or technology
Outcome evaluation
Determines whether the program or technology has caused demonstrable effects as defined in the project goals
© 2017 American Health Information Management Association
8
Summative Evaluation (cont.)
Two components
Cost-effectiveness analysis
Compares the costs to the outcomes of two or more courses of action
Cost-benefit analysis
Compares the total costs of a program or project with its benefits
© 2017 American Health Information Management Association
9
Complex Adaptive Systems (CAS)
An entity with many diverse and autonomous components or parts (called agents) that are interrelated and interdependent with many interconnections
The agents behave as a unified whole in learning from experience and in adjusting to changes in the environment.
Each agent maintains itself in an environment which it creates through its interactions with other agents.
10
© 2017 American Health Information Management Association
Healthcare Organizations as CAS
Healthcare organizations have
Diverse agents that learn such as
Providers, patients, and other stakeholders
Nonlinear interdependencies
Small changes in the system may or may not result in a change at the system level
Reduce predictability
Self-organization
Patterns emerge based on the interaction of the components
Emergence
Patterns emerge from the system based on the individual agents as well as the specific interactions undertaken within the system
Coevolution
Both the system and the agent evolve
© 2017 American Health Information Management Association
Sociotechnical Model for Health Information Technology
The eight dimensions provide a means to evaluate aspects of health information technology.
Each can be assessed for what is working well and what could be improved.
Evaluation findings can be used to iteratively revise software, processes, or components of the HIT to improve the use of the technology or the technology itself to improve safe use of technology in healthcare.
© 2017 American Health Information Management Association
Design Principles for Health Information Technology
Five rights framework states that optimal design and implementation will be achieved if we provide:
The right information
To the right person
In the right format
Through the right channe.
At the right time in workflow
The five rights are essentially a framework for formative evaluation for HIT.
© 2017 American Health Information Management Association
User Centered Design
The product, service, technology, or process should be designed for the user and their task.
There should be consistency of design elements with other components of the system.
The language and terminology should be simple and use terminology that the user is familiar with.
© 2017 American Health Information Management Association
User Centered Design Model
Source: Creately 2016. Reprinted with permission.
© 2017 American Health Information Management Association
User Centered Design (cont.)
The user interface (UI) should be simple and transparent.
The user should know that their actions are successful and feedback to this effect should be provided.
Information should be provided about how to navigate the system.
Fundamental aspects of user centered design are to functionality aligned with the task.
Design the technology to reduce errors.
© 2017 American Health Information Management Association
The System Usability Scale (SUS)
A frequently-used tool that is simple yet informative
Consists of a 10-item questionnaire originally created by John Brooke in 1986
The SUS can be used to evaluate a wide variety of products and services including:
Hardware
Software
Mobile devices
Websites
Applications.
© 2017 American Health Information Management Association
17
Evaluation
The evaluator develops the theory of change.
The theory of change is a statement that explains how activities will produce a series of results and impact and is used to develop a logic model.
In the logic model the desired effect is identified first and then the inputs, activities, resources, and outputs required to achieve these effects are identified.
© 2017 American Health Information Management Association
Logic Model Example
Source: University of Kansas Workgroup for Community Health and Development 2016. Reprinted with permission.
© 2017 American Health Information Management Association
19
Evaluation Questions
Evaluation questions should be developed to guide the evaluation.
Questions should be identified and answered based on the logic models and the critical activities.
By posing the questions, data will be collected to answer these key questions and important aspects of the project will be assessed.
© 2017 American Health Information Management Association
Evaluation Data Collection and Activities
Once the evaluation questions are determined, procedures will need to be established to collect the data to answer the questions.
Quantitative and qualitative data can be collected through a variety of methods.
The process of data collection should be feasible and the data accessible for collection.
© 2017 American Health Information Management Association
Evaluation Research Examples
Process evaluation
O’Malley and colleagues (2005) examined key areas related to the use of coded data. In this study, the process of inpatient coding was examined via:
A review of the literature
Flow charting the process
Interviews
Discussion with coders and the users of the coded data
© 2017 American Health Information Management Association
22
Evaluation Research Examples (cont.)
O’Malley and colleagues also provide a useful guide to calculating accuracy measures for coding. The most common statistics are:
Sensitivity
Specificity
Positive predictive value
Negative predictive value
The kappa (k) coefficient
© 2017 American Health Information Management Association
23
Evaluation Research Examples (cont.)
Goal-based and impact evaluation
Shah and colleagues (2005) designed a computerized medication alert system to decrease adverse events with medications but the alerts often were overridden.
The system was redesigned to improve clinician acceptance
The first goal-based evaluation reflect frequent dismissal
The system was redesigned with user-centered design principals and re-evaluated for impact
The impact evaluation reflected that high-severity medication alerts were accepted.
24
© 2017 American Health Information Management Association
Other Evaluation Resources
Office of the National Coordinator (ONC) testing process for certification of HIT
Agency for Healthcare Research and Quality (AHRQ) Evaluation Toolkit
© 2017 American Health Information Management Association
