4

The QUERI Roadmap for Implementation and Quality Improvement

RESOURCE GUIDE

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

The QUERI Roadmap

Beating The Odds

Getting Where You Need To Go

A pragmatic guide on how to adopt, adapt, implement, spread and sustain new clinical practices and innovations across diverse real-world settings

www.queri.research.va.gov

Move effective practices from research to routine clinical care settings

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

VOLTAGE DROP Effectiveness of practices

Often drops in real-world settings

WAITING GAME It often takes years before effective

practices are adopted

NAVIGATE THE PROCESS

Guides you through the implementation process phases:

pre-implementation, implementation, sustainment

FIND RESOURCES Links to deep dives

on topics

APPLY IMPLEMENTATION METHODS Practical mile markers to help implementers know when

and how to apply implementation methods

Welcome to the Quality Enhancement Research Initiative (QUERI) Roadmap for Implementation and Quality Improvement! The QUERI Roadmap combines real- world experience with “state-of-the art” implementation and quality improvement science practices to make this knowledge usable for investigators and practitioners alike. This resource is a practical guide to plan and deploy methods and tools to support health system leaders, managers, and frontline clinicians in promoting the use of effective practices in routine health care settings. Moreover, the QUERI Roadmap aims to help frontline clinicians and staff integrate these practices into routine care using implementation strategies and rigorous evaluation of these efforts to inform policy and improve population health.

Health systems in the United States are rapidly changing, and investigators and clinical leaders must meet the needs of consumers and the clinicians who care for them. Since 1998, QUERI has responded to these demands by promoting the use of implementation science to accelerate the adoption of effective practices in routine care settings using a learning health system framework. Learning health systems use health system data to find opportunities to apply effective practices to improve consumer and service outcomes. QUERI has also led the development and application of implementation strategies, which are highly specified, theory-based tools or methods that are designed to help frontline practitioners overcome barriers to using effective practices in routine care settings, or in some cases, de- implement ineffective or low-value practices. Several success stories where QUERI implementation strategies were used to implement effective practices nationally include integrated mental health treatment in primary care, telemedicine for chronic illnesses ranging from PTSD to stroke, and physical activity and caregiver interventions.

We designed the QUERI Roadmap to serve as a practical guide for anyone interested in improving health care. The QUERI Roadmap’s three phases of Pre-implementation, Implementation, and Sustainment involve multidisciplinary stakeholders and rigorous evaluation methods.

• Pre-implementation: Identifies a high-priority need, selects effective practices to address the need, engages stakeholders to build implementation capacity, specifies needed practice adaptations and evaluation goals, and activates leadership support.

• Implementation: Calls for clinical and research leaders to use strategies that support frontline clinician learning and motivate and inspire clinicians to adopt effective practices.

• Sustainment: Includes the analyses that make the business case for sustaining an effective practice and the handoff to local operational leaders to own practice sustainment over time.

We see this QUERI Roadmap as a living document that will ultimately inform and be informed by the implementation and quality improvement initiatives within your health systems and communities. We hope the QUERI Roadmap becomes an essential tool to your work improving care, and we wish you best of luck in your journey! Amy M. Kilbourne, PhD, MPH

Director, Quality Enhancement Research Initiative (QUERI) Health Services Research and Development Office of Research & Development Veterans Health Administration United States Department of Veterans Affairs

Foreword

The QUERI Implementation Roadmap demystifies implemen-

tation science for stakeholders in a learning health system to

ensure that effective practices are more rapidly implemented into

practice to improve overall health

SUGGESTED READING Kilbourne AM, Glasgow RE, Cham- bers DA. What Can Implementation

Science Do for You? Key Success Stories from the Field. JGIM (2020)

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

The VA QUERI Roadmap | PAGE 2

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

Acknowledgements Suggested Citation Goodrich DE, Miake-Lye I, Braganza MZ, Wawrin N, Kilbourne AM. Quality Enhancement Research Initiative. QUERI Roadmap for Imple- mentation and Quality Improvement. United States Department of Veterans Affairs Veterans Health Administration Office of Research and Development Health Services Research and Development, 2020. Avail- able at https://www.queri.research.va.gov/tools/roadmap/

This Department of Veterans Affairs QUERI Implementation and Qual- ity Improvement Roadmap was developed by the Department of Veterans Affairs Quality Enhancement Research Initiative (QUERI) in collaboration with the Department of Veterans Affairs Health Services Research & Development Evidence Synthesis Program to support partnered initiatives to implement quality improvement innovations and effective practices in health care settings. The views expressed in this document do not necessarily reflect those of the Department of Veterans Affairs.

Contributors We would like to thank the many implementation experts, clinicians, and Department of Veterans Affairs operational program partners who contributed content, thoughtful suggestions, and critical feedback as part of the development of this resource.

The following individuals contributed subject matter expertise, tools and resources, case study examples, and/or reviewed the QUERI Road- map: Veronica Williams, MPH, Nicholas W. Bowersox, PhD, (ABBP) VA QUERI Center for Evaluation and Implementation Resources; Catherine Battaglia, PhD, RN; Borsika Rabin PhD, MPH, PharmD; Russ Glasgow, PhD; Marina McCreight, MPH (VA Eastern Colorado Healthcare Sys- tem); Mark S. Bauer, MD (Boston VA Healthcare System); Diane Hanks, MA (VA Center for Information Dissemination and Education Resourc- es); Austin B. Frakt, PhD, Melissa M. Garrido, PhD; Steven D. Pizer, PhD (VA QUERI Partnered Evidence-based Policy Resource Center); Leslie R. Hausmann, PhD, Shari Rogal, MD, MPH (VA Pittsburgh Healthcare System); JoAnn E. Kirchner, MD (Central Arkansas Veterans Healthcare System); Maureen L. Marks, PhD (VA National Center for Organizational Development); Cynthia K. Perry, PhD, FNP-BC, FAHA (School of Nurs- ing, Oregon Health & Sciences University); Byron J. Powell PhD, LCSW (Brown School at University of Washington in St.Louis); Maria Souden, MSI, PhD; Jenifer Stelmack, MSW; Chandrea Culbreath, MPH: Amanda Taylor, PHD: (VA Information Resource Center); Anne Sales, PhD, RN (VA Ann Arbor Healthcare System); Angela So, MPH; Todd Wagner, PhD (VA HSR&D Health Economics Resource Center); Shannon Wiltsey Stir- man, PhD (National Center for PTSD, Palo Alto VA Health Care System); Thomas J. Waltz, PhD (Eastern Michigan University); and Shawna N. Smith, PHD (Department of Health Management and Policy, University of Michigan School of Public Health).

Case Study Contributors: We would especially like to express our gratitude to the following individuals who contributed clinical case studies: Shari Rogal MD, MPH (VA Pittsburgh Healthcare System); Vera Yakovchenko PhD, MPH; Pa- mela Bellino OT (Boston VA Healthcare System); Elizabeth M. Yano PhD, MSPH; Alison Hamilton, PhD, MPH (VA Greater Los Angeles Healthcare System); and Elizabeth Oliva, PhD (Palo Alto VA Health Care System).

Reviewers We want to thank the following individuals for sharing critical feed- back: Borsika Rabin, PhD, MPH, PharmD (UC San Diego School of Medicine); Rinad Beidas, PhD (University of Pennsylvania Perelman School of Medicine and Penn Implementation Science Center, Leonard Davis Institute of Health Economics); Ashley L. Choate, MPH (Durham VA); Deborah M. Delevan, MA (VA Greater Los Angeles Healthcare System); Alex Dopp, PhD; Lisa V. Rubenstein, MD, MSPH, FACP (RAND Corporation); Margaret Farrell, MPH, RD (National Cancer Institute Divi- sion of Cancer Control & Population Sciences); Erin P. Finley, PhD, MPH (South Texas Veterans Health Care System); Kyler M. Godwin, PhD, MPH (National VA Quality Scholars Program); Samantha M. Harden PhD (Vir- ginia Tech); Blake Henderson, BS, Ryan Vega, MD (VA Center for Innova- tion); Lauren Korshak, MA (VA Office of Health Equity); Nicole Floyd, MPH; Kim Peterson, MS (VA HSR&D Evidence Synthesis Program); Karin Nelson, MD, MSHS (VA Puget Sound Healthcare System); Gretchen A. Piatt, MPH, PhD (University of Michigan Department of Learning Health Sciences); and Lucy A. Savitz, PhD, MBA (Kaiser Permamente Northwest Region).

Graphic Design and Editorial Assistance Graphic design support and editorial assistance was provided by: Natalya Wawrin, MSW (VA Center for Clinical Management Research); Sarah Wonders; Amy Miller; Sarah Papke; Shawn Bauguess; and Kyle Morse (Atlas Research/Prometheus Federal Services).

For More Information Center for Evaluation and Implementation Resources Quality Enhancement Research Initiative U.S. Department of Veterans Affairs [email protected]

The VA QUERI Roadmap | PAGE 3

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

Contents Foreword ............................................................................................................................................................................................................... 2

Acknowledgements .....................................................................................................................................................................................................................................................................................................................3

Introduction .......................................................................................................................................................................................................... 5 Introduction ............................................................................................................................................................................................................................................................6

Who Is This Guide For? ................................................................................................................................................................................................................................................................................................................6 How Do I Use this Guide? ............................................................................................................................................................................................................................................................................................................7 Resources ........................................................................................................................................................................................................................................................................................................................................8 Why a Roadmap? .........................................................................................................................................................................................................................................................................................................................8 Applying the Roadmap ............................................................................................................................................................................................................................................................................................................ 10

Pre-Implementation ........................................................................................................................................................................................... 11 Identify a Problem and a Solution ................................................................................................................................................................................................................... 13

Identify a High-Priority Need and Related Goals ............................................................................................................................................................................................................................................................... 13 Agree on Effective Practices and Settings ............................................................................................................................................................................................................................................................................ 14 Clarify Effective Practice Core Elements and Options for Delivery Adaptations ........................................................................................................................................................................................................ 15 Culturally Tailored Adaptations for Consumer Impact .................................................................................................................................................................................................................................................... 18

Engage Stakeholders .......................................................................................................................................................................................................................................... 19 Cultivate Leadership and Stakeholder Support ................................................................................................................................................................................................................................................................. 19 Assess Capacity Including Barriers and Facilitators to Practice Delivery ..................................................................................................................................................................................................................... 22 Package Effective Practices with Delivery Adaptations ................................................................................................................................................................................................................................................... 28 User-Centered Design to Enhance Practice Adaptations ................................................................................................................................................................................................................................................ 29

Develop Measuresand Data .............................................................................................................................................................................................................................. 30 Design Evaluation to Match Goals ........................................................................................................................................................................................................................................................................................ 30 Identify Measures of Success and Data Sources ................................................................................................................................................................................................................................................................ 32 Establish Baseline Performance ............................................................................................................................................................................................................................................................................................. 40

Pre-Implementation Check List ......................................................................................................................................................................... 42 Implementation .................................................................................................................................................................................................. 43

Implement an Intervention ............................................................................................................................................................................................................................... 45 Select Implementation Strategies ......................................................................................................................................................................................................................................................................................... 45 Tailor Strategies to Local Settings .......................................................................................................................................................................................................................................................................................... 46 Disseminate Implementation Plan and Support Tools .................................................................................................................................................................................................................................................... 49 Planning for the Scale and Spread of an Effective Practice in a Health System ......................................................................................................................................................................................................... 51

Activate Implementation Teams ...................................................................................................................................................................................................................... 54 Leadership Uses Top-Down Practice Support to Push Change to Local Sites ............................................................................................................................................................................................................. 54 Empower Bottom-Up Pull to Enhance Stakeholder Buy-In at the Local Level ............................................................................................................................................................................................................ 55 Create Stakeholder Feedback Channels .............................................................................................................................................................................................................................................................................. 55

Monitor Implementation Progress .................................................................................................................................................................................................................. 56 Report Progress to Stakeholders ............................................................................................................................................................................................................................................................................................ 56 Make Data Accessible to Stakeholders ................................................................................................................................................................................................................................................................................. 56 Adjust Plan Based on Feedback ............................................................................................................................................................................................................................................................................................. 57

Implementation Check List ................................................................................................................................................................................ 58 Sustainment ........................................................................................................................................................................................................ 59

Sustain an Intervention ...................................................................................................................................................................................................................................... 61 Planning for Sustainment ....................................................................................................................................................................................................................................................................................................... 62 Develop a Business Plan to Continue Effective Practice as Routine Care .................................................................................................................................................................................................................... 63 Monitor for Change in Effective Practices and Whether a Different Practice is Needed .......................................................................................................................................................................................... 63 Weigh Costs of Maintaining the Effective Practice .......................................................................................................................................................................................................................................................... 65

Transition Ownership to Stakeholders............................................................................................................................................................................................................ 66 Provide Management Support .............................................................................................................................................................................................................................................................................................. 66 Plan and Budget for Resources .............................................................................................................................................................................................................................................................................................. 66 Support Continuous Learning and Innovation ................................................................................................................................................................................................................................................................ 67

Ongoing Evaluation and Reflection ................................................................................................................................................................................................................. 68 Consumer Outcomes ................................................................................................................................................................................................................................................................................................................ 68 Delivery of the Effective Practice ............................................................................................................................................................................................................................................................................................ 68 Clinician and System Costs ..................................................................................................................................................................................................................................................................................................... 68

Sustainment Check List ...................................................................................................................................................................................... 69 Case Studies ........................................................................................................................................................................................................ 70

The National Hepatitis C Innovation Team Collaborative............................................................................................................................................................................ 71 Tailoring Implementation of a Medical Home Model to Meet Women Veterans’ Needs Using Evidence-based Quality Improvement ..................................... 75

Sustainment ............................................................................................................................................................................................................................................................................................................................... 77 The Department of Veterans Affairs Rapid Naloxone Initiative: National Diffusion of a Promising Practice .................................................................................... 78

VA Resources for Implementation & Quality Improvement ............................................................................................................................ 82 General Resources for Implementation and Quality Improvement ............................................................................................................... 96

Conducting Adaptation Interviews ............................................................................................................................................................................................................... 100 Evaluation and Study Designs for Implementation and Quality Improvement .......................................................................................... 114 Glossary of Terms.............................................................................................................................................................................................. 130 References ......................................................................................................................................................................................................... 133

The VA QUERI Roadmap | PAGE 4

Introduction

Introduction

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

The QUERI Roadmap for Implementation and Quality Improvement describes how implementation science and quality improvement concepts, methods, and evaluation strategies can be used to improve health care outcomes and quality over time.

Learning health systems like the Veterans Health Administration strive to continuously innovate and respond to changing system priorities, policies, and treatment practices. Successful innovation to address these demands is driven by multidisciplinary teams of clini- cians, consumers, managers, and investigators working together to develop reliable ways to implement practices that improve consumer outcomes, or in some cases, de-implement the use of low-value practices.

Technology accelerates the creation of medical knowledge and front- line clinicians need a systematic way to plan, adopt, adapt, and spread

new clinical practices across diverse settings in the face of limited resources. This guide draws upon two decades of Department of Vet- erans Affairs QUERI experience addressing these challenges within the largest integrated health system in the United States and advancing the disciplines of implementation science and quality improvement in health care.2

Our leadership commissioned this guide based on a growing demand for a more user-friendly approach to planning and managing the implementation of effective practices, especially for frontline employ- ees making these changes.3

Our goal for the QUERI Roadmap is to bring together best practices, methods, and metrics to create a common approach for multidisci- plinary implementation teams accelerating knowledge into practice.

Who Is This Guide For? We developed the QUERI Roadmap as a pragmatic guide to enable individuals who wish to work within clinical settings and partner with clinical leaders and other stakeholders to support improvement efforts.4

By stakeholders, we mean anyone who has a direct or indirect role in supporting the adoption and use of an effective practice, including senior organizational leaders, managers, clinicians, support staff, and health care consumers.

Implementation scientists study the optimal way to move effective practices into real-world practice and apply their knowledge to help bring a diverse array of innovations to benefit consumers across vari- ous treatment specialties and settings.

In this guide we try to avoid the use of academic or technical words in order to promote communication across disciplines and share different perspectives, failures, and ideas that enable teams to solve complex health care problems.5

This guide offers a comprehensive approach or “roadmap” to prag- matically apply implementation science methods to overcome barriers to integrating effective practices or promising innovations in routine care. All stakeholders in a health system benefit from using rigorous implementation and evaluation methods.

Strong evaluation methods help us understand not only the perfor- mance of an effective practice, but also why and how a change hap- pened during implementation, its cost, and its potential for sustained use. This practice-based knowledge is valuable to organizational leaders, investigators, and other stakeholders at the local and regional levels of an organization who desire the relevant knowledge and skills for optimizing implementation efforts.6

The VA QUERI Roadmap | PAGE 6

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

PRE-IMPLEMENTATION

IMPLEMENTATION

SUSTAINMENT

Monitor implementation progress

How Do I Use this Guide? The QUERI Roadmap serves as a comprehensive and practical approach to guide you through the process of implementing an effective clinical practice or quality improvement program.3

The roadmap is a framework that takes you through the entire implementation process,7 which we divide into three phases: Pre-implementation, Implementation, and Sustainment. Within each phase, there are three levels of core domains that pose questions to you and your team about activities needed to:

Although the roadmap is organized into three separate phases, in real-world practice, the activities and components described in each phase overlap and are not always distinct. In some cases, failures, setbacks, new information, or discoveries may require you to revisit a prior phase and adjust your planned process to achieve the outcomes you want.

Rather than focus on implementation theory or implementation research design—as covered in a related resource by our colleagues at the National Cancer Institute8—the QUERI Roadmap offers practical mile markers to help you understand when and how to apply specific implementation methods to implement a practice.

We include three Case Studies to illustrate real-world applications of the QUERI Roadmap to implement diverse practices to address health care challenges.

Support uptake of the effective practice

Activate and engage stakeholders and delivery capability

Optimize the use of data and measures to assess progress

Identify a problem and solution

Implement an intervention

Sustain an intervention

Activate implementation teams

Transition ownership to stakeholders

Engage stakeholders

Develop measures and data

Ongoing evaluation and reflection

The VA QUERI Roadmap | PAGE 7

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

Resources When you see Resources, click to see more information on a specific topic related to quality improvement, implementation science, or evaluation, including tools, websites, and other resources.

Bold Blue terms are linked to the Glossary at the back of the guide to define key concepts. Similarly, a comprehensive References list enables you to cross-reference the science underlying this guide.

We also provide links to VA Resources that can help support

implementation and quality improvement efforts in the VA.

Most of the VA Resources are available via public-facing websites. Furthermore, VA resource centers highlighted in this guide illustrate best practices for applying evaluation and implementation methods that you might consider replicating in your health care setting. Finally, please note that some consultative services may only be available to VA health professionals.

Why a Roadmap? A brief history of our role in implementation science and practice QUERI was created in 1998 to deploy effective practices to support the VA’s transformation in the 1990’s from a hospital-based system to a large, integrated health system. Initially, our funded centers focused on implementing effective practices for specific health conditions (e.g., congestive heart failure, diabetes, mental disorders, spinal cord injury) that were identified and operationalized via a traditional “pipeline” approach. Figure 19,10 shows our first Implementation Framework pipeline, which drove significant improvements in health outcomes

and service delivery for Veterans. This process led to the formation of a QUERI-wide network of implementation experts and the international Implementation Research Group, which is now organized by our Cen- ter for Evaluation and Implementation Resources.

To better meet the rapidly changing needs of Veterans, clinicians, and VA health system, we update QUERI Strategic Plan goals every 5 years. The 2021-2025 QUERI Strategic Plan incorporates feedback from 150+ key stakeholders based on a national evaluation of overall program impact and incorporates practical implementation approaches to achieve quality improvements.2,11-14

Figure 1. The Foundation: QUERI Implementation Framework 1.0 *

Identify high-risk/high volume diseases or problems

Identify best practices

Define existing practice patterns and outcomes across the VA and current variation from best practices

Document that best practices improved outcomes

Within Step 4, a pipeline of phases enabled refinement and spread of effective and implementable programs:

1. Single site pilot

2. Small scale, multi-site implementation trial

3. Large scale, multi-region implementation trial

4. System-wide rollout Document that outcomes are associated with improved health-related quality of life

STEP 1

STEP 2

STEP 3

STEP 4

STEP 5

* To learn more, see past QUERI journal supplements: Medical Care, 2000; 38(6).; Journal of General Internal Medicine. 2006; vol. 21.; Implementation Science, 2008; vol. 3.

The VA QUERI Roadmap | PAGE 8

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

Why a Roadmap? (Continued) We saw an opportunity to make implementation science more user- friendly and accessible for frontline stakeholders who need to apply this knowledge. Thus, the QUERI Roadmap was commissioned to ex- tend the previous “pipeline”10 framework for implementation activities that reflected a traditional, linear research process, to one that is more agile, iterative, and responsive to operational priorities.

The QUERI Roadmap also extends the traditional research pipeline by including recent advancements in implementation science while drawing on key lessons from our experiences and the experiences of other health system leaders in implementing new practices. For example, we learned that uptake of effective practices is based on the use of implementation strategies that empower and motivate frontline stakeholders to adopt and sustain a new practice.

In this guide, we view implementation as a dynamic process that re- quires active and engaged input from multilevel stakeholders to iden- tify, implement, and sustain effective practices over time. The roadmap emphasizes designing and implementing practices for sustainability starting in Pre-implementation; this means once start-up resources disappear, the practice must be resilient and fully integrated into local clinical processes, workflows, and operating budgets as standard care.

The Learning Health System Knowledge to Action Framework15-17 (Figure 2) informs the roadmap and combines elements of a learn- ing health system with Graham and colleagues’ Knowledge to Action Framework.

A learning health system is broadly defined as any organization that routinely and continuously seeks to generate and learn from data with the purposes of improving individual and consumer health.17,18 This framework operationalizes a continuous quality improvement ap- proach to promote learning among all stakeholders. In this roadmap, we view implementation as an iterative process that generates new questions and knowledge based on rapid and systematic improve- ment cycles.

The Knowledge to Action Framework is a conceptual framework designed to reduce the long-time gap between when knowledge is created (research or practice-based knowledge) and the actual time it is put into use by clinicians, policymakers, consumers, or the public.15 The framework breaks this process into two primary components: 1) knowledge creation; and 2) an action cycle of implementing the knowl- edge into practice.

The Knowledge to Action Framework observes how these compo- nents interact with each other over time, variation in the settings where the knowledge is to be applied (hospitals, clinics), and barri- ers to knowledge use that may occur inside and outside the context of the adopting organization.15,19 Barriers to knowledge use must be

identified and iteratively addressed through implementation strate- gies and ongoing evaluation. Through this interactive problem-solving process to tailor or adapt the knowledge for local contexts, you can develop further knowledge about this process and share it internally while also sharing it within the larger health care community to im- prove clinical practice.

Figure 2. Learning Health System Knowledge to Action Framework with QUERI Roadmap

• Health care system data are used to set local priorities for improvement

• Data on the priority problems and potential contributors helps find evidence-based solutions to improve local health system care (data to knowledge)

• Evidence-based knowledge informs quality improvement and implementation science guided efforts (knowledge to performance improvement)

• Continuous monitoring of local health system helps clinicians iteratively learn and develop practice-based knowledge while revealing new opportunities for improvement (performance to data)

Adapted from3

���������������� ���

�������������������� ����D

at a

to K

no w

le

dg e

�� �� �� �� �� ��

��� ��� ������

������� �

�����������

����������� ��������

������ ���������������

���������������� ����� ���� �������������� ����� ���������������� ����

���� ��������������� ���������������� ��������

����������������� ��������� �������

������������� ������ ��������� ��� ������� ����� ����

������������ �������������

������������� �������� �� ���

The VA QUERI Roadmap | PAGE 9

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

Applying the Roadmap Is My Project Considered Research or Non-research? The Common Rule is the 1981 baseline standard of ethics by which any government-funded research in the United States is held account- able regarding biomedical and behavioral research involving human subjects. Recent changes to the Common Rule—including the broad- ening of research exemption categories and recognition of program evaluation as non-research—enable certain implementation-focused projects to be designated as non-research or “quality improvement.” As a result, more Institutional Review Boards are designating implemen- tation projects and program evaluations as non-research,20 especially projects that compare different implementation strategies or evaluate the impacts of the program or practice on the health system.

In the VA, projects that are designed to inform improvement efforts within the health system can be designated as non-research if the work is not designed to produce information that expands the knowl- edge base of a scientific discipline or other scholarly field.

Examples of activities in a project that would be considered expanding the knowledge base of a scientific discipline include the development or validation of new surveys or the direct testing of theoretical models that inform care outside the health system. In general, implementation projects can be considered non-research if results are used to inform care improvements to the system and overall setting.

Note: Randomization alone does not define implementation projects as research. Implementation and quality improvement initiatives can involve randomization as long as the focus is on program or imple- mentation evaluation and not testing a new treatment or practice for efficacy or effectiveness on patient health.

More importantly, you can publish non-research projects in scien- tific journals, but you must document non-research status for most journals. Consult with the journals and your local health system or academic research office regarding the best approach for document- ing non-research status and whether the protocol would need to be reviewed by an Institutional Review Board.

The following online seminar provides guidance on how to best navigate this issue in the VA:

Jeans CK, Kilbourne AM, Booker F., Braganza M. VA Quality Enhance- ment Research Initiative (QUERI): Everything you need to know about a QUERI non-research (“quality improvement”) protocols. 2019. Available at: https://www.hsrd.research.va.gov/for_research- ers/cyber_seminars/archives/3669-notes.pdf

In many cases, implementation projects can still be considered “re- search” but not “human subjects research.” In these cases, the projects are further exempt from Institutional Review Board review and may qualify for a waiver of subject-informed consent if the project is con- sidered minimal risk.

For more information on the distinction of research vs. non-research (quality improvement), see RESOURCES and consult your local health system or

academic research office for consultation.

The VA QUERI Roadmap | PAGE 10

Pre-Implementation

What is being implemented?

IDENTIFY A PROBLEM AND A SOLUTION

Who and what settings are involved?

ENGAGE STAKEHOLDERS

How is it being measured?

DEVELOP MEASURES AND DATA

The QUERI Roadmap presents Pre-Implementation as a series of collaborative steps taken by operational and investigator partners that:

• Work with frontline stakeholders to identify an effective practice that addresses a priority area of care

• Engage stakeholders at all levels of the organization to actively contribute to planning the implementation of the practice and develop adaptations that improve fit between the practice and the contexts in which it will be delivered

• Diagnose local delivery capabilities and assess barriers to implementation across settings with the goal of developing implementation strategies to overcome these barriers

• Agree upon an evaluation plan and meaningful measures that enable stakeholders to know whether the implementation of the practice is achieving the desired improvements on key performance benchmarks

WHAT IS BEING IMPLEMENTED?

Identify a Problem and Solution

What’s in this section?

Key Concepts

ADAPTATION The degree to which an evidence-based intervention is changed to fit with the needs, priorities, resources

of the setting or target population

FIDELITY The degree to which an effective practice is

implemented as intended by the intervention developers and as prescribed in the original protocol

VA Evidence Synthesis Program

KEY RESOURCES VA Office of

Health Equity

KEY RESOURCES

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

Identify high-priority need and goals

What kinds of issues make good targets? How should you define the problem?

A “shared understanding” of what? With whom?

Agree on effective practices and settings Why use effective practices (EPs)? Where can you find potential EPs?

What makes an EP a good fit?

Clarify EP core elements, adaptation options

(consumer/provider input) What are the “active ingredients” of an EP?

What changes, if any, need to be made for the EP work in a certain setting?

What kinds of adaptations might you consider?

Identify a High-Priority Need and Related Goals “What keeps you awake at night?” Begin the Pre-implementation phase by identifying a health care problem for systematic improvement. Your search can use empirical data (e.g., financial, epidemiological, or health services data) and input from pragmatic data sources, including suggestions from health system leadership, clinicians, consumers, and other stakeholders, to find a clinical care issue that aligns with a local or national clinical priority need,6,21-25 such as:

This quality gap between current practices and available effective clinical practices constitutes the “problem,” which can be clearly defined by:

• Describing the problem or quality gap • Specifying the clinical settings and consumer populations or the

types of clinical stakeholders who are affected by the problem • The desired improvements or benefits in outcomes over current

performance • Boundaries of the implementation effort (e.g., services involved,

resources, timeline, etc.)26,27

Depending on the scope of the effort, hospital or health system lead- ership can form a planning team to oversee and have ownership of the partnered implementation process. A clinical leader typically oversees the implementation effort and addresses operational challenges. An

embedded implementation scientist with relevant knowledge and a history of working with the clinical partner can co-lead the implemen- tation and evaluation activities.

Implementation planners may then work to develop goals that set quantifiable objectives for improvement over recent baseline per- formance. When setting goals, research questions, and/or evaluation design for the project, engage stakeholders across relevant disciplines, roles, and responsibilities to confirm a shared understanding and obtain buy-in.

Improvement goals are informed by internal and external benchmarks of quality as well as input from stakeholders. Performance goals need to be clinically meaningful to frontline stakeholders, also aligning with local and national clinical care priorities and the organization’s mission and values.

Performance gaps in measures of quality or clinical outcomes

Delays in adopting current evidence-based practices

Mandates to implement a new clinical practice guideline or policy

Urgency to reverse or stop an emerging issue

Identify a Problem and a Solution

PRE-IMPLEMENTATION Identify a problem and a solution

The VA QUERI Roadmap | PAGE 13

Agree on Effective Practices and Settings High-impact implementation begins by finding the best available prac- tice to solve the problem. An effective practice is a health-focused in- novation, intervention, program, policy, or technology with consistent, credible evidence supporting its ability to have a meaningful impact on consumer health behaviors or outcomes while minimizing harms.

Using evidence-based practices enhances the effectiveness of the care provide to consumers and their families and saves time and resources by improving the quality, timeliness, or efficiency of current clinical workflows.

Begin the Pre-implementation phase by searching for empirically supported practices that address the problem by reviewing relevant scientific research, high-quality evidence syntheses based on standard systematic review and/or meta-analytic methods, and/or clinical prac- tice recommendations from professional and government entities.

Note if the evidence underlying the practice has limitations. For ex- ample, was the evidence for a practice based on studies with settings, delivery methods, or populations similar to those of your own orga- nization?28 If these factors differ, you may need to pilot the practice to “scale out” to these new contexts or consumer populations prior to full implementation to verify the practice is effective in your setting.29,30

In some cases, evidence for a solution may not yet exist in a system- atic review or as the outcome of a randomized clinical trial. In this situation, consider expanding your search to identify examples of positive outliers within your organization and promising practices exhibited by similar high-performing organizations.21,31-33

Sometimes, the best solution for a par- ticular problem is a promising practice, which we define as a technique, in- tervention, program, or methodology that, through experience and rigorous evaluation, has proven to reliably lead to a desired result.34

The VA has created a formal process called Diffusion of Excellence, which actively identifies promising practices and innovations developed at the local level as well as evidence-based practices that have not been widely adopted.35-38 Each year, practices are submitted to a national competition, where investigators and health system leaders select the top practices with the greatest potential for health system impact. These practices in turn receive support for scale up and spread across VA.

Going forward in this guide, we use the term “effective practice” to include both evidence-based practices and promising practices with potential for high impact on local-, regional-, or national-level quality improvement or implementation efforts.

Consider the following questions when deciding whether to imple- ment a practice:39

• Are there concurrent improvement initiatives addressing the same problem?

• Do frontline stakeholders and managers perceive the evidence as strong enough to reliably yield the desired improvement in outcome?

• What is the feasibility of implementing the practice?

• Does the practice create a larger workload for frontline stakeholders without supplying more delivery capacity (i.e., time, personnel)?

• Is the practice simple and user-friendly in design and application? Does the practice fit in workflows?

• What is the burden on consumers, their families, and/or caregivers?

Finally, there is a growing recognition of the problem of low-value care services that merit “de-implementation.” In many cases, the goal may be to implement a new effective practice while acting to de- implement a low value care practice at the same time. “Low-value” care may be defined as health care services or treatments that provide little or no benefit to consumers, have the potential to cause harm, incur unnecessary costs to consumers, or waste limited resources.40,41 An estimated 25% to 34% of all health care costs in the United States reflect low-value care practices,42,43 and thus, reveal a large area for improvement.

De-implementation calls for a practice to be reduced, replaced, or stopped because it has been found ineffective, harmful, inefficient, or no longer necessary, even in the absence of a superior alternative practice.44,45 While the focus of strategies to de-implement low-value care services may be slightly different than those to implement a practice,44,46,47 the steps in the QUERI Roadmap are still be applicable to planning and executing a de-implementation initiative.

The VA Evidence Synthesis Program offers timely and accurate syntheses of specific health care topics of importance to clinicians, managers, and policy makers as they work to improve the health and health care of Veterans.

Outside VA, look to resources like Cochrane, the U.S. Preventive Services Task Force, and the Agency for Healthcare Research Quality Evidence-based Practice Center Program.

PRE-IMPLEMENTATION Identify a problem and a solution

The VA QUERI Roadmap | PAGE 14

Clarify Effective Practice Core Elements and Options for Delivery Adaptations Upon selecting an effective practice, outline the core elements of the practice and develop a menu of options that gives implementing facilities flexibility to adapt practice delivery. This also optimizes planning for staffing, training, supervision, and program support needs.48

Core and Peripheral Elements Core elements are the critical features of an effective practice that were tested in rigorous clinical trials and linked with improved outcomes.48,49

Core elements of an effective practice need to be delivered with fidelity to ensure the desired outcomes are achieved. Fidelity refers to the degree to which an effective practice is implemented with- out compromising the core components essential for the practice’s effectiveness.50,51

Core elements can include materials, measures, and protocols on how to deliver a practice, as well as the core logic and underlying theory sup- porting the practice as shown in Figure 3.49,52 The nature and scope of these components depend on the complexity of the effective practice.

Peripheral elements by comparison (e.g., “menu options”) refer to the modifiable aspects of an effective practice that do not affect its core components49 but optimize the fit between the practice and contex- tual influences affecting practice delivery, such as consumer, cultural, and setting characteristics.53

A logic model can enhance clarity. A logic model is a visual graphic that shows how your effective practice produces desired outcomes.55-57 Logic models describe the core logic or casual relationships between program core elements, the problem, and key measures of success (See Using a Logic Model as a Planning Tool and Identify Measures of Success and Data Sources).

PERIPHERAL COMPONENTS Adaptable

Implementor Characteristics Priority Population

Practice Delivery Organizational, Community,

or Service Setting

• Socio demographic characteristics • Culture (identity, values, sensitivity) • Health Literacy • Co-morbid health conditions • Access to care • Patient-centeredness

• Knowledge, skills, training • Beliefs or attitudes toward EP • Past experience • Resources and support

• Who can deliver EP • Fit with/Competition with other EPs • Fit with organizational mission • Budget, resource support • Appropriate types of clinicians

CORE COMPONENTS Stable

• Content • Procedures • Materials • Underlying theory/logic • Mechanisms for change • Philosophy of EP • Number of sessions

• Dose, frequency, or session length • Technology versus in person delivery

Figure 3. Examples of Core and Peripheral Components to Effective Practices (Adapted from 49,54)

Balancing Fidelity and Adaptation It is important to balance the tension between the need for practice fidelity with frontline stakeholders’ need to flexibly adapt practice delivery for the diverse contexts in which they are implementing.58,59

Maintaining fidelity involves consistently assessing the degree to which the core elements of the practice are implemented as intended and as prescribed in the original protocol.8,50 Fidelity can be measured by comparing current practice delivery with protocol in terms of:60

1. Adherence to the original protocol 2. Dose or the amount of the practice delivered 3. Quality of practice delivery, and 4. Consumer reaction and acceptance

Fidelity monitoring is essential because poor delivery of a practice re- sults in less desirable outcomes. Moreover, without fidelity monitoring,

it can be difficult to determine what aspect of the practice contributed to reduced outcomes. However, effective practices are not limited by fidelity to a one-size-fits-all approach.61 Practice adaptations are inevitable and necessary.

Adaptation is best defined as a process of the thoughtful and deliber- ate alteration of the design or delivery of a practice with an aim of improving its fit or effectiveness in a given context.62,63 While adapt- ability refers to the characteristics of a practice that lend themselves to modification, the QUERI Roadmap also views adaptation as an implementation strategy to optimize a practice over time—from Pre- implementation through Sustainment.58,64 Consider stakeholder input as you assess whether to adapt and improve an effective practice for specific settings or population characteristics in light of the capacity and available resources.

PRE-IMPLEMENTATION Identify a problem and a solution

The VA QUERI Roadmap | PAGE 15

While any change to a practice is a modification, consider the follow- ing distinctions:53

• Planned/proactive modifications (adaptations) are made deliberately through a planning process prior to delivery to maximize fit and implementation success while minimizing disruption to the practice.

• Unplanned/reactive modifications occur over the course of practice implementation, either in an impromptu manner or in reaction to constraints or challenges, and may or may not need to be aligned with core elements to be effective.

• Fidelity-consistent modifications preserve or align with core elements that are needed for a practice to be effective.

• Fidelity-inconsistent modifications alter the practice in a way that does not preserve its core elements; this likely results in diminished practice effectiveness.

Modifications are ultimately judged on whether they positively or negatively affect outcomes:62

+ Modifications that increases the fit between an effective practice and end users (i.e., consumers, clinicians), which may lead to im- proved practice engagement, acceptability, and clinical outcomes, particularly among minority or underserved populations.

+ Modifications that improve outcomes by aligning the practice with end users’ needs.

- Modifications that remove core elements or do not align with a

context’s needs; these modifications can result in reduced practice effectiveness.

As Figure 4 shows, fidelity and adaptation are complementary processes that can positively or negatively influence implementation success.61

Figure 4. Optimizing Implementation Effectiveness with Fidelity and Adaptive Fit to Context

Low Low

High

High

Practice Fidelity

Poor Implementation

E�ective Implementation

Variable Implementation

E�ectiveness

Variable Implementation

E�ectiveness

Practice Fit with Local Context

Implementation effectiveness is optimized in the green quadrant when practices are delivered with high fidelity and when the imple- mentation strategy positively adapts the practice to fit local contexts. Conversely, poor implementation is likely when there is a lack of fidelity and practice fit with the local delivery context. The yellow quadrants illustrate how implementation effectiveness can be quite variable if either fidelity or adaptivity are suboptimal.

Adapting Your Practice You might be asking yourself, “How do I know whether an adaptation is needed or appropriate?” Use the tools in the roadmap’s RESOURCES section to help evaluate whether or not to make adaptations to an effective practice:

The Cancer Prevention and Control Research Network’s Adapta- tion Planning Tool65

Helps you consider adaptations based on a simple stoplight system to evaluate their potential effects on core elements:

• Green Light modifications are fidelity consistent • Yellow Light modifications should be made cautiously • Red Light modifications may be fidelity inconsistent and should

be avoided

The Iterative Decision-making Tool for Evaluation of Adaptations (IDEA)66

Helps tailor the adaptation process to the situation by evaluating key

decisions, including whether an adaptation is: • Necessary • Fidelity consistent with core elements • Feasible based on implementation timeframe • Able to be piloted before full implementation • Influential on key implementation or clinical outcomes • Valuable from the stakeholders’ perspective

The Iterative Decision-making for Evaluation of Adaptations (IDEA) framework emphasizes that adaptation is not straightforward. Rather, it is a dynamic process that benefits from planned, deliberative deci- sion points to evaluate and potentially iteratively test the benefit of incorporating an adaptation within practical time and resource con- straints. Several evaluation models and frameworks67 can help make planned, systematic, and theory-based adaptations to practices before trying to deliver a practice at scale. We illustrate one basic, five-step process to adapt a practice during any roadmap phase:8

Figure 5. A Systematic Approach to Adapt Your Clinical Intervention or Effective Practice (Adapted from65)

Assess �t and consider adaptation in your

local setting

Assess the acceptability and importance of the adaptation with end-users of the practice

Make �nal decisions about what and how to adapt with

stakeholders/practice experts

Make adaptations Pretest and pilot test during Pre-implementation before

implementing

Assess Fit Assess Acceptability Make Final Decisions Make Adaptations Pretest and Pilot Test

Repeat steps as needed

PRE-IMPLEMENTATION Identify a problem and a solution

The VA QUERI Roadmap | PAGE 16

Monitoring Adaptations for Organizational Learning Monitoring the fidelity of a practice is important because it helps ensure the desired improvement in outcomes. Similarly, monitoring adaptation during implementation efforts is also critical to support organizational learning. Understanding the nature, origin, timing, and impact of modifications on outcomes can help develop practice-based knowledge to optimize practice fit to a specific context.53,54,68-70

To better study adaptations, implementation scientists developed

methods to systematically evaluate and document adaptations.53,68,71 Figure 6 illustrates an abbreviated version of the updated FRAME (Framework for Reporting Adaptations and Modifications-Enhanced), which facilitates the classification of different types of modifications over time, including whether the modifications were planned/un- planned, fidelity consistent, and when they occurred.

Figure 6. Framework for Reporting Adaptations and Modifications-Enhanced

WHAT was modi�ed or adapted?

Was the adaptation PLANNED?

WHO decided to make the adaptation?

What was the nature of CONTEXT

adaptations?

To what extent was the adaptation FIDELITY

consistent?

WHEN in the implementation process

will it occur?

What was the GOAL behind the adaptation?

What are the contextual REASONS for the

adaptation?

For whom or what were adaptations made?

LEVEL OF DELIVERY

Adapted from53 Note: This is an abbreviated version of the FRAME and does not include all elements.

Using Adaptation Tools In light of the increased interest in understanding how adaptations influence the uptake of clinical innovations, Rabin and colleagues70 suggest the following criteria when selecting ideal pragmatic methods to assess adaptation:

• Documents in real time over the lifetime of an implementation/quality improvement project

• Easily replicable

• Nonburdensome to users and beneficiaries

• Low complexity

• Low cost and requires modest resources

• Provides both quantitative and qualitative information on adaptation

• Assesses adaptations from the perspective of multiple stakeholders

• Uses multiple methods to gather data (e.g., interview, survey, observation, documents)

• Tracks adaptations to both the effective practices and implementation strategies used

In RESOURCES, we include helpful pragmatic tools for tracking adap- tations, including a modified version of the FRAME checklist devel- oped by Wiltsey-Stirman and colleagues53,68 to efficiently interview frontline clinicians about modifications and adaptations to assess the effects of planned and unplanned modifications on implementation effectiveness.

More information on tools to track adaptations based on the FRAME (e.g., FRAME Interview Coding Manual, monthly self-reported Adapta- tion Checklists) can be found on the F.A.S.T Lab website (The Fidelity, Adaptation, Sustainability, and Training Lab at the National Depart- ment of Veterans Affairs Center for PTSD and Stanford University).

We also include a brief interview guide developed by the VA’s Triple Aim QUERI Program70 to help assess whether adaptations have been made to a practice over time; this guide and related methods are derived from the FRAME68 and RE-AIM evaluation frameworks.72 This interview guide is part of a multi-method assessment approach to evaluate adaptation over time and is linked to a case study illustrating the application of these methods in four diverse health systems.

PRE-IMPLEMENTATION Identify a problem and a solution

The VA QUERI Roadmap | PAGE 17

Culturally Tailored Adaptations for Consumer Impact

Health disparities are significant differences in health or health care between two or more groups due to social, economic, or environmental disadvantage and can be based on:73,74

Race/ethnicity Religion Military service era

Gender Socioeconomic status Functional limitation

Age Sexual orientation Geography (rural/urban)

Mental health Sex identity Cognition

In many health settings, effective practice implementations are rarely tailored to high-risk or vulnerable consumers with greater health needs who could most benefit from treatment.73,75,76 When there is poor fit between an effective practice and the individuals who use or consume the practice, clinical and quality outcomes are likely to suf- fer.61 Effective practices can and should be tailored to different cultures while still preserving core functions.

Cultural adaptation is the “systematic modification of an effective practice to consider the importance of language, culture, and context in such a way that the practice is compatible with the consumer’s cultural patterns, meanings, and values.”77

Effective practices can reduce health inequities if they are designed to address the barriers that are preventing individuals and communi- ties from living their healthiest life. During the Pre-implementation phase, both senior- and local-level planners should look to local health system and/or public health epidemiologic reports to consider ways to address health disparities in the target population.73

Effective practices and messaging can be targeted to specific at-risk groups or tailored at the consumer level to increase the relevance and impact of the message to consumers. Like a piece of clothing with a good fit, a tailored practice is more likely to be acted on because of the way it has been personalized to an individual’s personal or cultural characteristics or needs.78

In local health service or clinic settings, a diverse group of stakeholders can be engaged to review effective practice materials and recommend ways to customize practices to local and vulnerable popula- tions. This step can ensure the materials are relevant, acceptable, and personal- ized. Stakeholders may include:

• Consumers and their families • Frontline stakeholders working with an at-risk population • Health education committees • Patient advocacy groups • Hospital health communications specialists • Investigators with diverse backgrounds

In large health systems, central clinical program offices or resource centers may be available to consult with and provide you with resources and tools to adapt practices to your consumers and setting. For VA, an exemplary resource is the VA Office for Health Equity (OHE; https:// www.va.gov/healthequity/)

PRE-IMPLEMENTATION Identify a problem and a solution

The VA QUERI Roadmap | PAGE 18

Cultivate Leadership and Stakeholder Support Gaining the support of local leaders and frontline stakeholders to implement a new practice is a key task during the Pre-implementation phase.48,79 Senior implementation leaders are essential to creating a

compelling vision and empowering frontline stakeholders to adapt the practice to fit their local settings and populations.

WHO ARE MY STAKEHOLDERS? Getting the right people involved in an implementation initiative is a critical first step. Stakeholders make up the broad spectrum of potential partners who may have a direct or indirect role in supporting the design, delivery, or receipt of the practice.

Internal stakeholders are those in a health system—most notably, consumers/Veterans and their families or caregivers—though they might also include those who coordinate, fund, or support implementation efforts, including:8

HEALTH SYSTEM LEADERSHIP CARE DELIVERY TECHNICAL AND SUPPORT ROLES

National program leaders General clinicians Research experts

System or facility leaders Specialty clinicians Information technology

Mid-level clinical managers Medical assistants Bioengineering

Service or department chiefs Lab personnel Systems redesign engineers

Frontline supervisors Peer specialists* Technicians

External stakeholders work outside the health system and can offer unique views and experiences. These stakeholders may include:

Non-profit organizations Clinicians/practices from outside

your health system Patient advocates

Insurance companies and other types of payers

Policymakers/local leaders Local, state, or

community organizations

* Also known as community health workers

Successful implementation requires organizations to design for sus- tainment from the start of Pre-implementation phase through the end of the Implementation phase. Designing an implementation process for sustainability is about empowering local stakeholders to be ac- tively involved in all aspects of planning, delivering, measuring, and re- fining a practice so they feel ownership of the practice and understand

the value of the practice. Conversely, top-down policy directives or mandates give frontline staff little ownership in a practice. Implemen- tation efforts that engage frontline stakeholders are more successful in adapting the practice to their context and identifying resources and the means to sustain a practice once initial funding is expended.

PRE-IMPLEMENTATION Engage Stakeholders

Engage Stakeholders

The VA QUERI Roadmap | PAGE 19

“If you want to go quickly, go alone. If you want to

go far, go together.” – African proverb

Empowering the Frontline to Innovate The communication and management strategies that system and local hospital leaders use to mobilize support of frontline stakeholders can greatly enhance or hinder an implementation effort.80,81 At a basic level, leaders must interact with frontline employees and managers to clearly communicate how a new practice supports local priorities and clinical needs and to understand stakeholders’ perspectives.

To go beyond superficial “buy-in” for a new practice, stakeholders need to understand the strength of the evidence for the practice and how it serves as an improvement over current care with respect to quality, outcomes, and/or costs.

Senior leaders can garner sustained stakeholder commitment by making the effort personally and professionally relevant to frontline staff and clinicians; this requires actively involving stakeholders in the implementation process so they have an ownership stake in the outcomes.48,82 Leaders and managers can achieve this by:5,39,79,83-85

1. Conveying a clear and compelling purpose

2. Aligning the value of an effective practice with stakeholders’ interests and values of:

- Improving work flow - Delivering better care to consumers - Addressing important local care needs or “pain points”

3. Seeking frontline stakeholders’ feedback early and often to give them an ownership stake in the implementation process

4. Giving stakeholders control over how the practice is adapted to improve fit with local contexts86,87

Senior leadership support is essential for implementing and sustain- ing practices. Evidence suggests that health care organizations with a

distributed leadership structure build greater capacity for adopting effective practices than relying simply on top-down appoaches.88

Spreading responsibility across levels of the organization—from ser- vice chiefs to middle managers and frontline supervisors—maximizes the complementary skills, viewpoints, and resources needed for the implementation. Distributed leadership reduces organizational frag- mentation between leaders at the “top” of the health system and those employees at the “bottom” who are charged with carry- ing out changes to frontline care.

Leadership support at implementing facilities is essential for translating the aims of an implementation effort into action88 by committing people, protected time, space, and other resources. While senior hospital leaders’ support and attention for an effort is vital,89 hospital leaders often face competing demands for their time and attention.90 For sustained focus on the implementa- tion, these leaders must engage frontline and mid-level leaders to take ownership of local implementation efforts.

Mid-level managers and frontline supervisors are critical for managing operational oversight of implementation efforts and coordinating with frontline stakeholders and staff who actively work to implement the changes. Mid-level managers are the communication bridge between top-level leaders and frontline innovators trying to make a new prac- tice fit in a local setting. Frontline stakeholders and their clinical teams also look to middle managers to set expectations, clarify roles and re- sponsibilities, support conflict resolution, provide coverage strategies to engage in improvement activities, monitor and reflect on perfor- mance outcomes, and enable frontline-team-initiated innovations.91,92

Navigating Concerns About Change New clinical innovations and improvement efforts can be met with skepticism by frontline stakeholders because these changes are often viewed as complex, costly, challenging, and time-consuming. Make it an ongoing priority to address these concerns with staff.

The following table highlights some common challenges facing hard- to-engage sites, including sources of concern held by key stakeholders as identified by a Department of Veterans Affairs Evidence Synthesis Program report:

Table 1. Characteristics of Hard-to-Engage Sites in Implementation Initiatives

Common Challenge Valid Factors Underlying Implementation Challenges

Limited bandwidth or resource availability

• Understaffing of key personnel • Time constraints • Lack of administrative support • Frontline staff burnout

• Departure of team members • Lack of funding • Implementation as an unfunded collateral duty

Local solutions to the same problem

• Presence of a local innovation/solution • Lower priority for adoption of a different practice • Disruption of existing workflows to switch practices

Competing priorities with other Initiatives

• Local priorities may not align with implementation initiative • Higher-priority local initiatives may cause resource shortages • Leadership attention preoccupied with other initiatives

Interdisciplinary coordination • Stakeholders siloed by specialty or geographic distance • Different priorities, values, or culture among clinical teams, services, or facilities within a system

Adapted from93

PRE-IMPLEMENTATION Engage Stakeholders

The VA QUERI Roadmap | PAGE 20

New practices that increase frontline stakeholders’ workloads with- out additional resources (e.g. protected time, personnel) may face resistance because these changes place a greater burden on clinicians without increasing job control or their ability to deliver better care.39 These forms of work stress can decrease the psychological well- being of clinicians and lead to apathy, burnout, disengagement and turnover, and often poor health outcomes including depression and suicide, which undermines the delivery of high-quality care.

Similarly, top-down mandates or policy directives can meet resistance from frontline managers and stakeholders who have no voice in the process and, therefore, have little stake in whether the practice suc- ceeds or fails. Make sure to frame an implementation effort so it aligns with local stakeholder needs, values, and priorities, and stay alert for any indications of stakeholder discomfort or disengagement with the implementation process.

Initial resistance to an implementation initiative can be a helpful form of feedback. Take this feedback as a sign that a site or local stakeholders need to understand and weigh the added value of the

new practice.93 This form of engagement often reflects a pragmatic approach to manage scarce resources prudently before committing to adoption. This cautious approach is different from situations where sites disengage from the implementation conversation, which sug- gests a more urgent need to better understand local site perspectives and concerns.

Among hard-to-engage facilities or services, a site’s delay in adoption can be framed as a strength. This “long-view” strategy can be more effective in some cases, where rushing to adopt a new practice can lead to shallow levels of practice implementation and failure to sustain outcomes over time.

For sites with a late implementation start, there is an advantage of be- ing able to draw upon the support and experience of earlier adopting sites to develop a plan that readies local stakeholders for successful implementation.94,95 While some late adopters may want to hide their local difficulties from their peers, others may see it as an opportunity to address local priorities and need for support.

Stakeholder input can improve implementation by: • Streamlining or tailoring a practice to ensure its relevance and acceptability among consumers and stakeholders • Identifying potential barriers to practice uptake • Recommending strategies to address health disparities • Developing permanent solutions rather than temporary “workarounds” • Identifying meaningful outcome measures

We believe deeper and engaged stakeholder participation occurs when stakeholders have an equal voice and actively collaborate with imple- mentation scientists and operational planners throughout all phases of the roadmap.96,97

Engage these stakeholders early and often through mechanisms such as steering or advisory committees, interviews, focus groups, user test- ing, and observations of work routines. It is particularly important to engage relevant stakeholders at key decision-points.

Building and Maintaining Effective Partnerships Early and regular communication is essential to building productive and resilient long-term partnerships. Open lines of communication build relationships and ensure mutual respect, trust, and credibility from the outset of the partnership.11,98

Investigators can actively seek to understand the priorities of their operations partners and work closely with them to establish aims, questions, evaluation designs, and measures that remain relevant and meaningful over time.99 Operational partners value flexibility, responsiveness, and interim data and products. Investigators work to maintain the rigor and integrity of the scientific process even if it takes more time (e.g., complex evaluation designs, carefully written results for publication).99

Engagement requires managing the balance between methodological rigor and the need for timely solutions to clinical problems.100

Establishing regular, candid, and psychologically safe communica- tions between partners promotes a shared agenda and increases the probability that implementation efforts will be effective.98,101 Strong partnerships rely on setting joint expectations about project timelines, roles, responsibilities, shared resources, deliverables, and preferences for ongoing communication.99

Using a memorandum of understand- ing helps to formalize the partnership and promote shared understanding of key details and goals. Prioritizing face- to-face meetings between partners is essential to jointly review prog- ress, define expectations, and refine aspects of the partnership and explore potential future collaborations.

Investigators play a key role in foster- ing institutional learning by sharing important information from partnered evaluation and implementation efforts with policy leaders, managers, and frontline stakeholders throughout the system.

Contact the VA Center for Evaluation & Implementation Resources for more information on the QUERI Partnered Evaluation Toolkit for building effective partnerships for quality improvement and implementation science.

PRE-IMPLEMENTATION Engage Stakeholders

The VA QUERI Roadmap | PAGE 21

Assess Capacity Including Barriers and Facilitators to Practice Delivery Developing an implementation plan begins with a diagnostic assess- ment9,23,84,102-105 of an organization’s readiness to implement a practice and the potential barriers to practice delivery.

Capacity assessment is an action-oriented process used to understand multilevel contextual factors that may influence implementation efforts. The goal of this step is to design an implementation plan that uses theory-based strategies to change and organizational behaviors to support not only practice adoption and implementation but also enable sustainment.

Organizational readiness refers to an organization’s willingness and ability to adopt and implement an effective practice and is comprised of three components:106,107

• Motivation – Incentives or disincentives that make an effective practice more or less desirable, a priority to adopt by local clinicians, a relative advantage over other practices, and a practice’s perceived complexity, compatibility, trialability, and outcomes that are clearly observable by clinical users.108

• General capacity – Attributes of an organization and its connections with other organizations and/or communities that affect its ability to implement a new practice, such as the organization’s culture, climate, innovativeness, resource use, leadership, staffing capacity, and operation structures.

• Innovation-specific capacity – Innovation-specific human, technical, and fiscal conditions necessary to support the successful implementation of an effective practice, including the knowledge, skills, and abilities needed for practice delivery;

clinical program champions; implementation climate, notably support from managers; and supportive relationships inside and outside the organization.

Traditionally, implementation initiatives have favored a top-down model of implementation that “pushes” a new practice into use with capacity-building strategies that are specific to that single innova- tion.109 However, learning health system leaders prefer implementa- tion strategies that build local capacity for implementation, in which the process for prioritizing areas for improvement and selecting an evidence-based solution is started by—or “pulled” by—frontline stake- holders over time.103,107

Organizational readiness varies considerably within large health sys- tems serving diverse communities and settings. General and practice- specific capabilities are often constrained by the quality of available resources, including technology, evaluation infrastructure, materials, time, and money.110 Implementing new effective practices requires changing existing and organizational behaviors within these con- straints to improve processes.

In the figure below, we present a systematic series of steps that can help guide from problem to solution to implementation during the Pre-implementation phase.111 Note the effective methods that can guide you through similar steps, including implementation/interven- tion mapping25,84,105 and many of the implementation science process frameworks.15,24,48,110,112,113

Figure 7. A Systematic Approach to Designing an Implementation Plan

Assess practice gaps using current knowledge

and current practice

Use current knowledge to assess e�ective practices

to reduce the gap

Assess barriers and/or facilitators related

to implementing the e�ective practice

Use theory to link barriers to evidence-based change techniques

Develop a logic model to align activities to measure

and monitor outcomes Design an

implementation strategy to overcome barriers

Map out and analyze processes, behaviors,

and/or decision points that make up an e�ective practice

Re-evaluate �t and repeat steps

if necessary

STEP 1 STEP 2 STEP 3 STEP 4

STEP 5

STEP 6

STEP 7

Adapted from 102,111,114

PRE-IMPLEMENTATION Engage Stakeholders

The VA QUERI Roadmap | PAGE 22

A Systematic Approach to Designing an Implementation Plan Assess gaps using current knowledge

Pre-implementation begins with operational and research partners using organizational quantitative and qualitative data to identify gaps in quality of care or service delivery and, agreeing on a clinical problem area for improvement.

Use available knowledge to identify an effective practice solution to address the gap

Following agreement on a clinical problem area for improvement, conduct a literature review, or evidence synthesis, to find an effective practice that provides a solution to the clinical area for improvement. The practice should be feasible and provide a substantive benefit to resolve the gap in care. Consider if any existing, low-value practices require de-implementation to make room in workflows and/or avoid conflicts with the new practice.

Map out and analyze behaviors, steps, processes, and/or decision points that make up the effective practice

Engage frontline stakeholders at several representative sites involved in the implementation effort to understand variation in the work flows necessary to deliver the practice with quality using one or more of the following methods:

• Process Mapping – Used to understand how clinical processes contribute to outcomes in a new practice by creating a flow diagram that depicts steps of a clinical process, staff roles, and specifications of inputs and outputs to identify modifiable areas for improvement.115,116

• Practice Mapping – Helps makes a practice visible to clinical stakeholders who can71 map out the series of sequential behavioral steps and decisions in a current practice to identify ways to make clinical processes more efficient, reduce errors, or improve outcomes.117

• Root Cause Analysis – Maps out the causes of service failures and safety problems using a systems analysis approach in which interdisciplinary teams ask three questions: “What happened?” “Why did it happen?” and “How do we prevent it from happening again?”118

Consider how mapping practices, programs, guidelines, behaviors, and decision points benefits implementation efforts. Effective practices con- sist of a bundle of behaviors and decisions. In health care, these practices are organized in different ways:111

• Practices are focused on building smaller blocks of care delivery, such as prescribing a medication, dressing a wound, or assessing a consumer.

• Programs bundle practices together into more complete “packages” and may or may not be fully evidence-based in their bundled form (e.g., care coordination or infection-control bundles).

• Clinical practice guidelines summarize many different recommendations for a specific clinical area and are often not operationalized as practices.

Effective practices typically involve many people with different roles and supporting tools (e.g., specific templates in the electronic medical re- cord). Clinicians and consumers are often faced with decision points. Understanding when, where, and how these behaviors and decisions break down and affect outcomes is crucial to developing solutions that address errors and negative variation.

Assessing all the behaviors that go into an effective practice is challenging and local implementation teams can use strategies, like process map- ping. A visual map allows planners to understand where to focus implementation activities to make the behavior changes necessary to imple- ment a new practice with fidelity while reinforcing replicable standards of practice that sustain the change.

Assess barriers and/or facilitators related to implementing the effective practice

By mapping out the behaviors, decisions, roles, and processes that comprise an effective practice, you can begin to see factors affecting the delivery of a practice. Other ways to identify these contextual influences on the implementation process include:25,84

• Brainstorming with local stakeholders and program planners to generate potential influences based on participants’ clinical experience from similar practices

• Reviewing the scientific literature for barriers and facilitators identified by investigators implementing similar effective practices • Observing the physical settings to identify issues • Interviewing stakeholders using theory-based interview guides • Conducting brief surveys of stakeholders (usually after some interviews)

There are many implementation science frameworks7,119 that help identify barriers and facilitation factors. See RESOURCES for a brief overview. The framework selected may have to do with the type of effective practice or the implementation setting.

For more information on choosing dissemination or implementation science theory and models to guide your planning, visit: Dissemination & Implementation Models in Research & Practice.

STEP 1

STEP 2

STEP 3

STEP 4

PRE-IMPLEMENTATION Engage Stakeholders

The VA QUERI Roadmap | PAGE 23

Use theory to link barriers to evidence-based change techniques

Once you identify relevant barriers (and facilitators), work to organize them according to type and influence using a concept or intervention mapping process or quality improvement tools, like driver diagrams or fishbone diagrams.64,120 Using visual methods, you can map barriers to specific behaviors, decision points, and roles/responsibilities that need to change to adopt the practice and achieve the desired outcomes.

Most clinical behaviors are carried out by individual clinicians and staff. Consequently, many implementation strategies incorporate theory- based techniques that help change what people think and how they decide to carry out clinical care duties.

Think

Decide

Behavior

Act

To enact change, consider the context in which individual clinicians work. For example, the Pre-implementation capacity assessment should identify organizational factors that influence how people think and behave, including social norms, organizational culture, job status (hierarchy), leadership characteristics, and performance plans and incentives.

When supporting improvement in a given context, use local strengths or enabling factors for behavior change. For example, consider the resources available to support delivery (e.g., dedicated time, personnel, past experience, supportive leadership) and the degree to which different disciplines communicate and coordinate their health care delivery across a hospital or system.

Then, link barriers with modifiable factors and theoretical mechanisms of behavior change that can be tailored for intervention.103 Theory-based behavioral change tech- niques help specify the:

• Barriers that are altered by a behavior change • Causal mechanism(s) of change • Level of change (e.g., individual, service, organizational) to enhance

implementation • Technique or strategy and the details of who, when, and what outcomes are

affected by the intervention23,71

Table 2 illustrates one method of developing components of a comprehensive imple- mentation intervention for multilevel barriers. In this case example, clinical leaders at VA needed to develop a plan to ensure naloxone, an effective medication for reversing the effects of opiate overdoses, is widely available to clinicians, consumers, and their caregiv- ers. A systematic approach to distributing naloxone was crucial to address a national epidemic of accidental overdose deaths that affected Veterans at even higher rates than the civilian population.

The Department of Veterans Affairs Opioid Overdose Education and Naloxone Distribution (OEND) initiative was the first national quality improvement effort in the United States to develop a standardized approach for providing education and training on naloxone use for various health systems and to at-risk consumers and their family members.129 Implementation planners rapidly assessed potential barriers to delivering OEND across multiple levels of the health system and identified modifiable mechanisms of change to reduce these barriers.

STEP 5

Often, you may use more than one approach to overcome a barrier, and similarly, more than one theory may guide the identification of an implementation strategy to improve implementation outcomes.105

• Link barriers at the clinician- or clinical- team-level to catalogs of behavior change techniques grounded in psychological or social theories and designed to cause changes in important thoughts and behaviors of individual clinicians, managers, or clinical treatment teams.104,121-125

• Link organizational and higher-level barriers to catalogs of implementation strategies (See ERIC in the General Resources section) to cause broader changes among groups of stakeholders ranging from practice-level to system-level changes, such as policy or financing.103,126-128 Frequently, implementation strategies involve more than one strategy whereby discrete strategies are mapped to the causal mechanism of change.

PRE-IMPLEMENTATION Engage Stakeholders

The VA QUERI Roadmap | PAGE 24

Table 2. A Theory-based Approach to Mapping Barriers to Implementation Strategies: The Department of Veterans Affairs OEND Initiative

Barrier Level Barrier Mechanism of Change25,124 Implementation Strategy/Technique121,126

Veteran

Risk awareness Perceived vulnerability Risk communication, use mass media

Ability to use naloxone in overdose

Caregiver knowledge, self- efficacy, skills

Develop and distribute educational materials, obtain family feedback, activate Veterans and family

Cost Alter office fees, make billing easier (naloxone provided free of charge)

Clinician/clinical team

Ability to identify high- risk Veterans

Knowledge, clinical decision-making

Develop clinical analytics to identify at-risk Veterans

Lack of expertise in naloxone prescribing

Knowledge, skills, goals, self- efficacy, subjective norms

Offer educational trainings, train-the-trainer

Prescribing naloxone Behavioral cueing, environ- ment resources

Change electronic medical record templates

Awareness of progress Feedback processes, subjec- tive norms

Audit and feedback, relay data to clinicians

Hospital/practice

Competing priorities Professional role change, reinforcement

Mandate change, policy directives(s) for all facilities, identify and prepare champions

Implementation variability

Knowledge, subjective norms Values, standardize tools, guidance, resources implementa- tion plans

Cost to facilities Reinforcement Policy change, change cost to hospital (no cost)

Health system

Low availability of naloxone

Environmental context, social roles

Use advisory boards and national workgroups

Unstandardized nalox- one kit

Environment resource Place naloxone kits on national formulary

Lack of best practice Knowledge, skills, decision processes, social learning

Create learning collaborative, centralized technical assistance and facilitation

Coordination across service disciplines

Professional role, norms, motivation

Change availability of services and mix of clinicians offering treatment

Union support Professional role, social influ- ences, norms

Obtain formal commitments

Design an implementation strategy to overcome barriers

Implementation strategies are specific tools or methods used to support the uptake of the effective practice (See Select Implementation Strate- gies for more information).

De-implementation of low value services may be necessary to ensure successful use of the effective practice. Low-value care is defined as health care services or treatments that provide little or no benefit to Veterans or have the potential to cause harm, incur unnecessary costs, or waste limited health care resources.40,41 De-implementation calls for a practice to be reduced, replaced, or stopped because it is ineffective, harmful, inefficient, or no longer necessary ,even in the absence of a superior alternative practice.44,45,47

Develop a logic model to align activities and monitor outcomes

A logic model is a useful way to visually summarize the alignment of theory, behavioral change techniques, processes measures, and outcomes. In the next section of Pre-implementation, “Develop Measures and Data,” we describe how to create a logic model to help stakeholders under- stand key activities and measures.

STEP 6

STEP 7

PRE-IMPLEMENTATION Engage Stakeholders

The VA QUERI Roadmap | PAGE 25

Using a Logic Model as a Planning Tool Logic models are tools that visually display how implementation ac- tivities produce desired outcomes. It illustrates the core logic or casual relationships between core elements, the problem, and key measures of success given certain assumptions about how the practice works within certain contexts.

Logic models answer the following questions:57

• What problem is the effective practice trying to solve and what outcomes represent success?

• What activities and supports are necessary to achieve these outcomes?

• What inputs and resources are needed to deliver these activities and supports?

The resulting model helps clarify the implementation process by sup- plying a roadmap that shows elements and activities that lead to the desired outcomes. This helps communicate elements of the clinical practice and the implementation process to different stakeholders and promotes buy-in. Specifically, a model helps:130

• Refine your practice’s mission, values, goals, and objectives with stakeholders

• Foster agreement regarding the most important desired outcomes

• Identify the most effective path to pursue given organizational and resource constraints (see Core Elements)

• Identify existing or additional (or weak and strong) components of the practice and ways to enhance performance (Menu of Adaptations)

The benefit of building a logic model comes from the process of discussing, analyzing, and justifying key relationships and linkages be- tween activities and outcomes among frontline staff, evaluators, and health system leaders. Greater stakeholder buy-in can foster resource commitments for long-term sustainment and help identify potential barriers so frontline stakeholders can address them during Pre-imple- mentation or early Implementation.

A basic logic model usually has two sides comprised of process and outcome elements. The process side includes the effective practice’s inputs (e.g., resources), activities, and outputs (i.e., direct products). The outcome side shows the intended effects of the practice, which can be short-term, intermediate, and/or long-term. Assumptions and relevant details about contextual factors influencing adoption may also be included in a logic model. Often these are noted in a box below or to the left of the model as show in Figure 8.

Figure 8. Layout of a general logic model

PROCESS OUTCOMES

Inputs Activities Outputs Short

Assumptions/Contextual Factors

Intermediate Long

Source.130

PRE-IMPLEMENTATION Engage Stakeholders

The VA QUERI Roadmap | PAGE 26

This next figure shows a more detailed logic model to help define common model components.130

Figure 9. Basic Logic Model adapted from56,131,132

Purpose or Mission of your Practice

Th e

P ro

b le

m

OUTCOMESINTERVENTION

Inputs Activities Outputs Short-term Intermediate Long-term What you

invest What you do Evidence of

Activities Clinical Practice Changes in: Changes in: Changes in:

• Time • Funding • Partners • Sta� • Equipment • Space • Facilities

• How many • Who you reach • Units of

Services

• Knowledge • Skills • Attitudes • Motivation • Awareness • Practice process • Implementation

outcomes

• Behaviors • Function • Symptoms • Practices • Procedures • Policy

• Health system or condition

• Reduce morbidity or mortality

• Access • Equality • E�ciency • E�ectiveness • Timeliness

• Services • Activities

How you improve practice uptake

Implementation Strategies

• Activities • Methods

In�uence of Contextual Factors and External In�uences in the Environment

• Purpose or mission is the clinical problem or need that motivates an implementation or quality improvement effort.

• Theory/Context describes the climate or context in which a change takes place, including the health condition, needs, and history of a specific problem. You may describe the political, policy, or economic climate for supporting a change, including the justification for using an effective practice to solve a problem. What is the theory behind the elements of the effective practice?

• Inputs include the resources, infrastructure, or supports necessary to implement the effective practice (e.g., time, equipment, materials, money, volunteers, training, and technical assistance)—what is invested. Inputs can also include constraints on the practice implementation.

• Activities are specific steps taken to produce desired outcomes—what you do. You can clarify whether activities occur early versus later and whether activities deliver the practice versus implementation strategies to enable effective practice uptake.

• Outputs are direct, tangible measures or results of activities— what you get. These early outputs often serve as documentation of progress and describe what the effective practice does and who it reaches (e.g., number of people reached and treated or the frequency, type, duration, and intensity of the practice delivered [dose]).

• Outcomes are the desired results of the effective practice— what you hope to achieve. These outcomes are described as short, medium, or long-term depending on the objective, length of the practice, and expectations for the practice.

- Short-term outcomes are the immediate effects of the practice. They may include outcomes that reflect the quality of implementation processes or changes in knowledge, attitudes, or beliefs of the intended audience (e.g., clinicians, administrators, consumers).

- Intermediate- or medium-term outcomes are behavior, normative, or policy changes in a target setting.

- Long-term outcomes are the desired changes in clinical indicators, and/or health outcomes.

- Impacts refer to the ultimate impacts of the practice on system or population outcomes (e.g., mortality, health disparities).

• Assumptions are the beliefs about a practice and the resources required. Assumptions include the way we think the practice will work based on research, best practices, experience, or common sense. The decisions made about implementing a practice are often based on these assumptions. Assumptions may cover funding stability over the course of implementation; training and education results in high-fidelity delivery; clinician motivation to attend trainings; staff hiring or re-allocation; or a policy directive leading to individual clinician behavior change.

• Contextual Factors describe the multilevel influences from both inside an organization and the broader social environment that interact and influence the practice. These factors may influence implementation, participation, and the achievement of outcomes and may or may not be modifiable. Implementation strategies and evaluation methods can help clinicians respond and adjust for these factors.

PRE-IMPLEMENTATION Engage Stakeholders

The VA QUERI Roadmap | PAGE 27

Package Effective Practices with Delivery Adaptations Modifying effective practices makes them more user-friendly and ac- ceptable for real-world use after clinical trial validation.133,134 Packag- ing requires developing, pilot testing, and finalizing a user-friendly practice that enables facilities to efficiently adopt and implement the practice. Furthermore, this package prepares local implementation teams to rapidly scale-up and spread the effective practice across diverse settings with impact and fidelity.

The effort and resources to package an effective practice depend on the complexity of the practice and the extent to which there is an existing manualized set of empirically tested materials to support dis- semination and implementation of the practice.

A user-centered design approach135,136 to package development in- volves garnering input from end-users—notably, frontline clinicians— in the use of the effective practice, often through development, validation, and rapid-cycle testing of the practice’s components in a real-world setting. User-centered design emphasizes simplicity, ease of use, aesthetics, and fit with the problem. The central implementa- tion team drafts the package. End users vet the package and provide feedback in an iterative fashion to ensure the format and content are user-friendly.

In health systems like VA, these practice packages are often disseminat- ed as toolkits and posted on electronic websites for easy access. Tool- kits may include an overview of setup procedures, underlying theories and logic, scripts, consumer materials, standard operating procedures, job descriptions, standardized medical record templates, marketing materials, and instructions for implementation across different settings.

An implementation package typically provides descriptions and pro- cedures for fundamental capacity-building strategies for local frontline stakeholders:

1. Tools to plan, implement, or evaluate the effective practice

2. Training content for educational or skill-building sessions

3. Technical assistance that supplies interactive and individualized support to the specific needs of teams

4. Assessment and feedback that describes the systems of monitoring and offering feedback to managers and frontline stakeholders83

Continuous Engagement and Input from Frontline Stakeholders Effective practice package designers can consult regularly with end- users to review and offer feedback on tools and resources and test the effective practice package, staff training and technical assistance plans, and the organization of the implementation plan. When hold- ing these meetings, stakeholders can offer recommendations on the content to customize the effective practice across settings.

Packaging designers (i.e., investigators and operational staff ) can use virtual sessions to test the feasibility of specific tools or resources by using rapid-cycle testing, comparing a standard versus adapted version of materials (A/B testing), and conducting cognitive end-user

walkthroughs to improve the usability of the package over time.135

Once the package has been iteratively developed, it can be pilot test- ed for usability and functionality at a small number of facilities before implementation. Packaging designers can invite diverse facilities to elicit feedback on their needs and priorities, including from potentially negative end-user feedback.36,52,64,136 Facilities need adequate time to pilot the package and provide information on feasibility, acceptance, appropriateness, usability and any other contextual problems with the overall package, so that it can be refined based on their input.

Creating a Menu of Adaptations Evaluators can work with frontline stakeholders who will deliver the practice58 to develop a menu of adaptations before fully implement- ing. Ideally, these adaptations are identified proactively during Pre- implementation rather than reactively during Implementation.53,66,68

Piloting these adaptations52,134,136 with those who will eventually use or deliver the practice—like clinicians and consumers—can help ensure fidelity to the original core elements and sustainment of the practice over time.54,59,68

Systematical assessment of these adaptations over time helps determine which adaptations support or improve implementation

effectiveness (“positive deviance”) and which adaptations result in reduced outcomes (“negative deviance”).54 Investigators can use this process to better understand how, when, and where these adaptations can be replicated to improve the uptake of similar effective practices.

VA investigators created a simple tracking system to help record these practice adaptations with a focus on: 1) by whom the modifica- tion is made; 2) what is modified; 3) the level of the delivery; 4) the context of the modification; and 5) the nature of the modification.68,70 Two approaches for systematically tracking modifications and adapta- tions are described in the General Resources for Implementation and Quality Improvement section of this guide.

PRE-IMPLEMENTATION Engage Stakeholders

The VA QUERI Roadmap | PAGE 28

User-Centered Design to Enhance Practice Adaptations Many effective practices are based upon complex treatment protocols developed for highly resourced clinical trials that may need adaptation or redesign to simplify their use for real-world conditions and diverse consumers. User-centered design is a participatory approach that aims to ground the characteristics of an effective practice with information obtained from the individuals who use that practice (e.g., consumers, clinicians) with a goal of maximizing “usability in context.”135

Usability is the extent to which an effective practice can be used by stakeholders to achieve specific goals in a particular context.133 In the early design process, implementers seek to understand the needs, desires, preferences, values, experiences and recommendations of people who will use the effective practice.135

This type of iterative pilot testing starts in the Pre-implementation phase and may continue through early implementation. Testing ad- aptations and monitoring fidelity to core elements is crucial to ensure that clinical effectiveness of a practice is maintained. Similarly, keep track of adaptations made to the original practice, measure their effect on usability using a standardized measure, and monitor changes in clinical outcomes associated with significant adaptations.136

In participatory intervention research, collaborating with stakeholders to ensure the user-friendliness, acceptability, and feasibility of a clinical intervention is a fundamental step to preparing a program for imple- mentation. User-centered design offers a rigorous and systematic ap- proach to improve practice delivery and impact. Lyon and Koerner136 observed that there is a need to evaluate the application of these principles and understand how the results improve key outcomes at the levels of:

• Consumer (acceptability, improved functioning) • Clinician (adoption, acceptability) • System (appropriateness, efficiency, program reach/penetration)

Qualities of a well-designed and usable effective practice:136

• Fit – Fits the particular context • Simple – Is straightforward and easy to use • Learnable – Enables rapid understanding on how to use and

apply the practice • Efficient – Minimizes the time, effort, and cost of using the

practice • Memorable – Facilitates remembering and applying key

elements without supports • Error reducing – Prevents or allows recovery from

misapplication of practice • Acceptable/satisfying – Ensures the practice is valuable or

acceptable especially compared to other practices

PRE-IMPLEMENTATION Engage Stakeholders

The VA QUERI Roadmap | PAGE 29

Design Evaluation to Match Goals Program evaluation is the systematic collection of information about the activities, characteristics, and outcomes of effective practices. This allows implementers to make judgements about the practice, improve practice effectiveness, and/or inform decisions about future practice development.139

Evaluation encourages the examination of the operational delivery of an effective practice, including which activities take place, who conducts the activities, and who is reached as a result.

Evaluation also substantiates how closely the effective practice adheres to implementation protocols, and when adaptations were made to a practice that improved or reduced effectiveness. Through program evaluation, you can determine whether the practice’s activi- ties and implementation strategies were implemented as planned. Evaluation also identifies strengths and weaknesses in implementa- tion delivery for improvement.139

Evaluating the implementation of effective practices and quality improvement innovations is complex due to the nature of such work, which involves iterative, adaptive, and context-specific changes.140 The design of an evaluation should reflect this complexity and strive to use clearly defined and prioritized measurement plans to understand the practice’s impact on key outcomes and variation across sites.

Accordingly, leaders are encouraged to adopt best planning prac- tices141,142 early in Pre-implementation to understand their effective practice and develop the most acceptable, feasible, and cost-effective evaluation design in light of limits set by resources, time, and political context.

An example is an evaluability assessment, which is an evaluation strat- egy that is useful with relatively new effective practices that have not been implemented at scale or undergone testing’. Evaluability assess- ments are consistent with the QUERI Roadmap and closely align with recommended planning steps in Pre-implementation:

1. Involve the intended users of the information in planning to obtain commitment to the implementation and define the scope and purpose of initiative

2. Clarify the intended practice by engaging stakeholders as part of the process of defining goals and expectations, describing core elements, and seeking feedback about practice usability, relevant contextual factors, available resources, capacity to make change, and possible strategies

3. Create or revise a logic model to clarify program activities and expectations among stakeholders and pilot the practice to identify data/measurement issues for conducting a full evaluation

4. Agree on necessary changes in activities or goals to improve fit (menu of adaptations)

5. Explore evaluation design options to answer high-priority questions and then select a design that realistically considers resource, time, and political constraints

6. Agree on evaluation priorities (key outcomes) and intended uses of information (e.g., quality improvement, budgeting/ planning, performance measurement) to support system learning

PRE-IMPLEMENTATION Develop Measures and Data

Develop Measures and Data

The VA QUERI Roadmap | PAGE 30

Why Should Evaluation Matter to Frontline Stakeholders? New laws and policies require many public health agencies and private health organizations to use rigorous evaluation methods— preferably randomization—to examine the implementation of new practices or policies.143,144 Clinical services may be asked to tie evidence into budget requests through a learning health system approach that provides evidence for practices and policies.145. This increases the system’s capacity to use evidence, evaluation, and data as tools to improve outcomes.

The aim of rigorous, systematic evaluations of effective practices is to help policymakers decide how to best invest resources to avoid waste or expense on ineffective programs and/or implementation methods.146 Strong program evaluations use mixed methods involving both quantitative and qualitative data collection methods because quantitative results are often insufficient to explain why a practice was implemented well—or not. Qualitative data collected from staff meeting minutes, interviews with stakeholders, and focus groups with consumers help provide data that explains why results occurred.

Ultimately, program evaluation is a tool used to demonstrate account- ability to health system stakeholders, who may include consumers, stakeholders, funding sources, policy makers, and/or members of local communities.

While program evaluation seems to be “pushed” on frontline stake- holders as part of a top-down mandate, the goal is for frontline stakeholders to be “pulled” to in- volvement by an inner source of motivation to know how a clinical practice is performing and what can be done to further improve it.5,139,147

While both push and pull management strategies can motivate stakeholders to conduct quality evaluations of their work, these efforts are more likely to be sustained when frontline clinicians see the results as useful through feedback mechanisms (e.g., electronic dashboards, performance monitoring meetings) that can help them do their jobs while better serving consumers.148

Selecting an Evaluation Design As a general rule, use the most rigorous design to answer the agreed- upon evaluation questions while accounting for real-world constraints. Operational partners are often interested in questions about how to show impact as quickly as possible, so time and evaluation measures that show improvement are important considerations.

In contrast, investigators are often interested in questions that center on which implementation strategies to provide, to whom, and when to achieve successful implementation outcomes.149 Accordingly, implementation scientists prefer study designs that compare the ef- fectiveness of strategies or the ideal provision of different types and/or intensities of implementation strategies, including:

• Hybrid implementation-effectiveness trials (types I to III) • Cluster or group randomized trials • Stepped-wedge cluster randomized design • Factorial designs • Sequential Multiple-Assignment Randomized Trials

(SMARTS)

Many Resources are available to aid you in selecting a specific, randomized evaluation design but the embedded implementation research and evaluation team should have expertise in weighing the relative merits of each design.139,140,146,150-153 Also, consider how the de- sign level of randomization, representative sites, and level of analysis (i.e., network, hospital, clinic) affects whether results are replicable across a wide range of settings when you seek to spread the practice further.146,152

Sometimes, the requirements of favored randomized designs conflict with the priorities of operational partners. This can occur when imple- mentation of an effective practice is delayed or the stakeholders want to spread the implementation strategies more rapidly.20,156

Other times, timing and resources do not allow a randomized evalua- tion due to ethical concerns by operational partners or participating sites. This can occur when effective treatments are delayed based on group assignment or time to treatment. Rigorous program evaluations are still possible using observational, quasi-experimental, and natural experimental designs, such as:149,154,155

• Non-equivalent control group design • Stepped wedge design (without randomization) • Interrupted time series • Regression discontinuity design

Evaluators maintain design rigor by striving for group equivalence at pre-intervention assessment by using matching or matched controls. Similarly, to maintain strong internal validity, designs can use multiple pre- and post- test observations and then repeat interventions.

PEPReC The VA QUERI Partnered Evidence-based Policy Resource Center (PEPReC) is dedicated to helping clinical programs meet high standards by providing timely, rigorous data analyses to support the development of high-priority policy, planning, and management initiatives and quantitative program evaluations.

The VA Center for Evaluation & Implementation Resources provides consultation to VA investigators AND operational partners for planning the designs, evaluation methods, and uses of implementation strategies.

PRE-IMPLEMENTATION Develop Measures and Data

The VA QUERI Roadmap | PAGE 31

Previous implementation studies have successfully enabled random- ization by creating equal opportunity for sites to obtain implementa- tion strategies sooner rather than later, especially when the imple- mentation strategies require start-up resources or additional time to deploy, which precludes sites from supporting the implementation at once. Operations stakeholders may also find value in a comparison

group, especially if they want to know whether the financial commit- ment of supporting the implementation strategies in an ongoing way is worth it. For more information on making the case for randomiza- tion, see the Community Partners in Care study157 and the Oregon Medicaid Experience studies.158

For implementation evaluations, Institutional Review Boards determined that randomization alone does not define research.20 Generally, these types of evaluations are considered non-research as long as they do not state that they are contributing to generalizable knowledge but that the results are used to inform overall programmatic or system improvements (see section above on research versus non-research for more information).

However, projects that involve randomization or other means of systematic allocation require registration in ClinicalTrials.gov or Biomed Central before being published in scientific journals. Registration of these trials is not limited to research protocols, and implementation and program evaluations have been registered in these systems.

Identify Measures of Success and Data Sources There are many different types of outcome measures that can be assessed to determine whether the implementation of an effective practice has been successful.

As Figure 10 shows, implementation evaluations pay attention to three categories of outcomes:159

Consumer outcomes – Measure the outcomes an effective practice was designed to impact, such as symptoms, functioning, satisfaction, and other consumer-reported measures of care160 that can be exam- ined at the consumer or population level of impact.

Implementation outcomes – Measure the effects of deliberate and purposeful actions to implement new treatments, practices, and ser- vices using multilevel implementation strategies.

Service or system outcomes – Long-term indicators of the popula- tion impact of an effective practice that pertain to access (timeliness), efficiency, safety, effectiveness, equity, consumer centeredness, or other aspects of quality, such as overuse, underuse, unintended conse- quences, or misuse of care delivery.161,162

Figure 10. Key Implementation Science Evaluation Outcomes

Evaluation OutcomesInterventionStrategies

E�ective Practices

Multi-level Implementation

Strategies

• Systems Environment • Organizational • Group Learning • Supervision • Individual – clinicians

& consumers

Implementation Outcomes

Service Outcomes*

Patient Outcomes

• Acceptability • Adaptation • Adoption • Appropriateness • E�ectiveness • Feasibility • Fidelity • Implementation cost • Penetration • Sustainability

• E�ectiveness • E�ciency • Equity • Patient Centeredness • Safety • Timeliness

• Symptoms • Function • Satisfaction • Cost (a�ordability) • Self-report

measures

*Institute of Medicine standards of care

Adapted from 8,160

PRE-IMPLEMENTATION Develop Measures and Data

The VA QUERI Roadmap | PAGE 32

As shown in Table 3, implementation outcomes measure the effec- tiveness/success of implementation strategies, proximal measures of the implementation process, and intermediate outcomes that influ- ence service and consumer outcomes.

In other words, if an effective practice or clinical innovation does not lead to improved consumer or service outcomes as expected, frontline stakeholders, managers, and senior health system leaders must under- stand whether this failure was due to:

• Poor design or fit of the effective practice with a priority health care context or consumer populations or

• Incomplete or poor implementation of an otherwise effec- tive practice

Table 3 defines ten implementation outcomes, shows when to assess them during implementation, and provides methods to measure them.

Table 3. Implementation Outcomes

Implementation Outcomes

Definition Implementation

Stage Example Ways to

Measure

Acceptability The extent to which implementation stakeholders perceive a treat- ment, service, practice, or innovation as agreeable, palatable, or satisfactory.

• Early for adoption • Implementation • Sustainment

• Survey • Interviews • Focus groups • Administrative data • Refusals

Adaptation The degree to which an effective practice or implementation strategy changes to suit the needs of a local setting or priority population.

• Pre-implementation • Sustainment

• Checklist • Survey • Interview

Adoption Intention, initial decision, or action by stakeholders to try or employ an effective practice (“uptake”).

• Implementation • Administration data • Observation • Interviews • Survey

Appropriateness Perceived fit, relevance, or compatibility of the effective practice for a given setting, stakeholder, and/or problem.

• Pre-implementation • Survey • Interview • Focus groups

Effectiveness** Effect of the implementation strategy on key outcomes. • Implementation • Sustainment

• Observation • Interviews • Focus groups

Feasibility Extent to which an innovation or effective practice can be successfully used or carried out within a given agency or setting.

• Implementation • Survey • Administrative data

Fidelity** Degree to which an effective practice or implementation strategy is delivered as prescribed in the original protocol or intended by developers; this may include multiple dimensions of content, process, exposure, and dosage.

• Implementation • Sustainment

• Observation • Checklists • Self-report • Administrative data

Implementation Cost

Financial impact of an implementation effort; this may include costs of treatment delivery, cost of the implementation strategy, and cost of using the service setting.

• Implementation • Sustainment

• Administrative data • Checklist

Penetration** Extent to which an innovation or effective practice is integrated within a service setting, organization, or health system (“reach”).

• Implementation • Sustainment

• Checklists • Program audits

Sustainability** Extent to which the effective practice is supported and/or institution- alized within a service’s ongoing, stable operations.

• Sustainment • Interviews • Program audits • Checklists

** Adapted from8,159,163 **In RE-AIM evaluation framework, Reach = penetration; Maintenance = sustainability, Effectiveness is the same; Implementation = Fidelity; EP = effective practice

PRE-IMPLEMENTATION Develop Measures and Data

The VA QUERI Roadmap | PAGE 33

The Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) Framework While there are a number of evaluation frameworks,7 the RE-AIM framework is among the most widely used to support systematic plan- ning, mid-course adjustments, evaluation, and reporting. RE-AIM is helpful for assessing program impact at a population level for clinical outcomes (Effectiveness, Maintenance) and implementation outcomes (Reach, Adoption, Implementation). The framework also pays close

attention to external validity by comparing how generalizable findings are to the consumer population(s) served by a health system and its local hospitals.164 The framework’s creators also advocate for the sys- tematic use of qualitative methods to understand and explain contex- tual factors influencing quantitative results for each dimension.165-167

Table 4. The RE-AIM Program Planning and Evaluation Model

RE-AIM Dimension Pragmatic Questions to Consider Measurement Considerations

Reach Who is (was) intended to benefit and who participates or is exposed to the intervention?

The number, proportion, and representativeness of individuals who are willing to participate or use an effective practice.

Effectiveness What are (were) the most important benefits you are trying to achieve and what is (was) the likelihood of negative outcomes?

The impact of an effective practice on key outcomes among subgroups of a population, including negative (unintended consequences), quality-of-life, and economic outcomes.

Adoption Where is (was) the clinical practice or policy applied and who applied it?

The absolute number, proportion, and representation of set- tings and staff who are willing to start an effective practice and those who are not.

Implementation How consistently is (was) the program or policy de- livered (fidelity), how will it be (was it) adapted, how much will (did) it cost, and why will (did) the results come about?

The intervention staff ’s fidelity to the various components of an effective practice’s protocol (i.e., delivery as intended) is one of several implementation outcomes (see Table 3).

Maintenance When will (was) the initiative become operational; How long will (was) it be sustained (setting level); and How long are the results sustained (individual level)? Measured by longevity of effects (individual level) and program sustainment (setting level).

The extent to which an effective practice or policy becomes institutionalized or part of routine practice and policy.

Adapted from168,169 Terms in parentheses are phrased for post-intervention evaluation. The basic questions are phrased for use in program or policy planning.

Tools for Measuring Implementation Outcomes There are many valid measures and methods for evaluating imple- mentation outcomes—some of which can be assessed by using a combination of qualitative and quantitative (mixed methods) strate- gies to assess multilevel stakeholders’ attitudes, beliefs, intentions,

and behaviors. Some of the measures for examining implementation outcomes can be found through the Society of Implementation Re- search Collaboration; however, membership is required to access most measures.

Other Key Outcomes Balancing Measures assess the potential unintended consequences of implementing a new effective practice, such as consumer harm from over- or under-treatment or the decreases in care in other do- mains.170 For example, tight control of many chronic conditions, such as anxiety, high blood pressure, diabetes, and pain, have resulted in unintended harms to consumers, their families, and the communities where they live. These examples highlight the need for continuous monitoring of quantitative health outcomes and qualitative data from consumers and frontline stakeholders who are well-positioned to detect signs of unintended consequences.

Economic evaluation outcomes include estimates of the financial impact of adopting competing effective practices or whether to use different implementation strategies to achieve individual- and population-level impacts on health outcomes. There is currently very

little information available to health system and local regional/hospital administra- tors about the comparative cost-effectiveness of different implementation strategies. It is currently a high prior- ity within implementation science and health systems to understand the economical logistics of using one clinical intervention (implementation strategy) over another to ad- dress a clinical priority to use resources efficiently.171,172

The VA’s Health Services Research and Development Service Health Economics Resource Center (HERC) is as an embedded national resource center that supports VA economic research173 to increase health care efficiency and value for Veterans and the nation. HERC provides tools and resources to assist researchers and operational leaders in conducting economic analyses for implementation research.

PRE-IMPLEMENTATION Develop Measures and Data

The VA QUERI Roadmap | PAGE 34

Health economists can help program evaluators design, collect, and evaluate economic data to conduct analyses on cost-effectiveness, inform return on investment decisions, and forecast the potential scale-up or de-implementation of an effective practice

There are different types of health economic evaluations, and they dif- fer based on whose perspective the analysis is conducted (i.e., whose cost is included, such as the health care sector, consumer/caregiver, or

societal) and the length of time to evaluate impact. Economic evalua- tions often use a ratio that compares costs in the numerator to changes resulting from the effective practice in the denominator, as shown in Figure 11.

Figure 11. Economic Consequences of Implementing Effective Health Practices

Changes in use of health care resources

Changes in use of non-health care resources

Changes in use of informal caregiver time

Change in use of patient time (for treatment)

Changes in future related and unrelated costs

Changes in Productivity

EFFECTIVE PRACTICE Including Implementation Strategy

Changes in health status

Numerator “Costs”

Denominator “Health E�ects”

Adapted from �gure 8.1, Neumann et al 2016

Historically, the gold standard was a cost-effectiveness analysis in which all costs are included in the numerator (a societal perspec- tive) and quality adjusted life years are the denominator.174,175 Cost- effectiveness analyses have long been preferred over cost-benefit analyses, which requires measuring the denominator in dollars. Cost- benefit analyses are often used in other fields, such as environmental economics, but rarely used in health care because of disagreement among economists regarding how to monetize some types of health outcomes.172

In the past 20 years, many decision makers have struggled with cost-effectiveness analyses because they include costs outside their purview (i.e., costs to society) and the evaluation time frame is much longer than their budget cycles. Consequently, budget impact analy- ses became increasingly popular, and the International Society for Pharmacoeconomics and Outcomes Research promulgated standards for business impact analyses, which has been widely used in drug and device evaluations.176

A budget impact analysis focuses only on the costs (the numerator) of implementing a new practice from a specific payer’s perspective (e.g., health care system) over a shorter timeframe of one to three years. Thus, many of the costs in the numerator, such as caregiver or consumer costs, are excluded. This narrower and more pragmatic approach has appealed to many decision makers who have limited budgets and simply want to know, “Can I afford it?” A budget impact analysis also incorporates the number of people affected by adopting a new practice to calculate a true “unit” cost (cost/consumer) and is useful for estimating the total budget to fund the new practice and its related implementation costs.

Budget impact analysis is sometimes used interchangeably with return on investment, although this can be misleading because a return on investment often includes a “return” that is non-financial, such as consumer outcomes or reduced staff burnout. For evaluators, the added complexity of valuing these non-financial returns for multiple perspectives makes the return on investment less practical than a budget impact analysis.

Finally, while economic evaluations may have been conducted on the effective practice, there is a scarce amount of data available to compare the relative costs of using different implementation strate- gies to support uptake of different types of effective practices across settings and consumer populations. In situations where these costs do not exist, economists rely on micro-costing methods to estimate costs. Micro-costing requires direct measurements of relevant activities and assigns costs to them.

In recent years, frameworks have been developed to track implemen- tation activities for micro-costing, 177,178 which generally focuses on the costs needed to replicate the effective delivery of a practice. Some- times, the implementation and effective practice costs are intertwined, and it may be necessary to measure costs collectively (e.g., treatments adapted for delivery by a special video or e-health delivery mode). These costs must be recorded regularly using costing surveys, stan- dardized templates, or other methods.

Non-VA Resources on economic evaluation can be found under RESOURCES (e.g., U.S. Centers for Disease Control & Prevention Polaris Economic Evaluation website).

PRE-IMPLEMENTATION Develop Measures and Data

The VA QUERI Roadmap | PAGE 35

Table 5. Comparison of Implementation and Effective Practices to Track for Micro-Costing

Implementation Costs Effective Practice Costs

Implementation strategy costs (labor) All costs to deliver the effective practice

Consumer/stakeholder engagement costs Practice administration, consumer recruitment

Program adaptation costs Creating/adapting program materials, supplies

Planning, staff recruitment, training, monitoring Mailings, phone calls, text messages

Printing, adapting materials, supplies, data collection Data entry, performance monitoring

Trainings, meetings, travel

Diagnostic or surveillance lab sample collection and testing

All staffing

Note: Costs may change over time from Pre-implementation to Sustainment

Process Evaluation Data Qualitative evaluation refers to an approach that does not rely on quantitative measurement or statistical analysis and aims to under- stand and explain why clinical interventions may or may not work by seeking multiple perspectives from individuals involved at different levels of the health care system.179 Qualitative methods often include individual and group interviews, participant observations, ethnogra- phy, field notes, and analyses of key documents (e.g., tracking iterative Plan-Do-Study-Act improvement cycle outcomes). Qualitative meth- ods and data are forms of process evaluation. There are many excellent Resources on qualitative research methods.180-183

Process evaluation aims to identify influences on the processes deliv- ering the clinical effective practice and/or implementation strategies. Process evaluation can be collected throughout the implementation process and help determine:

• How well a practice or implementation is working • The extent to which a practice or strategy is implemented as

designed (e.g., fidelity, dose, attendance, dropout) • Whether a program is accessible or acceptable to a priority

population of consumers

Process evaluation usually involves qualitative methods but may involve quantitative methods or a mix of both qualitative and quan- titative (mixed methods) to study influences on the delivery of the effective practice and the strategies used to implement it. Process evaluation is useful for gaining insight into the contextual factors that explain why or how something occurred. Process evaluations can be formative or summative process evaluation methods.184

Formative evaluation is a rigorous assessment process designed to identify and make use of potential and actual influences on the prog- ress, quality, and potential sustainment of implementation processes (i.e., effective practices and/or implementation strategies).185,186 Forma- tive data is important in implementation evaluations because they:

• Help refine, improve, and evolve the implementation process and in some cases, adapt an implementation strategy and/or effective practice itself

• May be conducted before, during, and/or after implementation activities to provide data for immediate use to optimize an implementation effort and interpret findings after formal implementation activities end30,186

Formative methods are often guided by a theory or framework, particularly to organize semi-structured interview guides and plan key points of time to sample key clinical stakeholders during the imple- mentation process.

Summative evaluation methods are used to make a number of sum- mative judgements about the worth of an effective practice and/or implementation strategy relative to the intended purpose. Summative data provides an indication of an effective practice’s success in achiev- ing the desired clinical impacts. This occurs at the end of an implemen- tation evaluation.30,186

Table 6. Contrasting the Perspectives of Formative and Summative Evaluation

Formative Evaluation Summative Evaluation

• How is the practice implemented?

• To what extent are desired changes occurring? For whom?

• Is the practice delivered at capacity?

• Is the practice making an impact?

• Are practice/implementation activities delivered as intended?

• What seemed to work? Not work?

• Are consumers being reached as intended?

• What were unintended outcomes?

• What are consumer reactions?

• Did the implementation activity have the intended effect?

Adapted from55

Several Resources are available to help evaluators use process evalua- tion methods.187,188

PRE-IMPLEMENTATION Develop Measures and Data

The VA QUERI Roadmap | PAGE 36

Sources of Health System Data Health system consumer care and administrative databases are the primary source of data used to monitor outcomes during an imple- mentation or quality improvement initiative. In many instances, these systems provide reliable data collection and reporting tools that track clinical outcomes from the electronic health record, as well as other valuable information related to consumer care. If you have access

to information specialists (e.g., programmers, analysts, statisticians), administrative data can add value to projects evaluating clinical out- comes and process improvement.

Within the Veterans Health Administration, there are various sources of administrative and clinical data including:

Table 7. Select Sources and Types of Department of Veterans Affairs Clinical and Administrative Data Circa 2020

Data By Source

VA Corporate Data Warehouse (CDW): national repository comprising data from several VHA clinical and administrative systems, including the electronic health record (EHR)

VHA User Enrollment Databases: Assistant Deputy Under Secretary for Health (ADUSH) Enrollment Files, Planning Systems Support Group (PSSG) Geocoded Enrollee Files

VA Patient Portal and Survey Data: MyHealtheVet, Bereaved Family Survey, and the Survey of Health Care Experiences of Patients (SHEP)

Tools for EHR review: CPRS, Joint Legacy Viewer (JLV), Compensation and Pension Records Interchange (CAPRI), and Cerner Millennium

Data By Topic

Cost Data: Health Economics Resource Center (HERC) cost data, Managerial Cost Accounting (MCA) National Data Extracts, Centers for Medi- care & Medicaid Services (CMS) claims data

Utilization Data: CDW, HERC Average Cost Dataset, ADUSH Enrollment Files, MCA, Medical SAS (MedSAS) Inpatient and Outpatient files

Death and Vital Status Data: CDW, VHA Vital Status File (VSF), Mortality Data Repository (MDR)

Pharmacy Data: CDW, Pharmacy Benefits Management Database (PBM), Pharmacy National Data Extracts (NDE)

Centers for Medicare & Medicaid Services Data for Veterans: Medicare, Medicaid, Patient Assessment, Healthcare Effectiveness and Infor- mation Set (HEDIS), United States Renal Data System (USRDS)

VA Facility and Employee Data: CDW, Veteran Administration Site Tracking (VAST) Database, Primary Care Management Module (PCMM) Web

Special Populations Data: VA Central Cancer Registry (VACCR), National Center for Homeless Veterans Registry, VA Airborne Hazards and Open Burn Pit Registry

Some data may be made available through clinical dashboards to ap- prise clinicians and policy leaders of the performance of related services.

To understand whether an improvement or implementation initiative is having the intended impact on clinical and quality outcomes, you need a clearly defined data and measurement plan that specifies:

• The source of data • The frequency of data collection • How the data is collected (e.g., self-reported, clinician-

contributed, or system-generated) • Who validates, processes, and analyzes the data before

dissemination • How the data are standardized for comparability across sites and

stakeholders • How the data is synthesized to be functional and usable by

stakeholders at different levels of the organization • How stakeholders and sites are selected to participate • Whether data for comparator sites are obtained and included in

reports • When, where, and how qualitative methods (e.g., interviews,

focus groups, observations) and surveys are conducted • Who performs qualitative and quantitative analyses and what

methods are used to ensure timeliness • Whether some form of local, regional, or centralized technical

assistance is available to help sites overcome data collection and reporting challenges

Developing a data collection and measurement plan early in Pre- implementation ensures proposed data reporting is feasible and can meet the expectations of implementation planners. To effectively use the wide variety of data available within a large health system like the VA, it is important to have a team with data expertise. Skilled programmers and analysts can guide the evaluation of relevant health system data throughout the processes of data extraction, cleaning, analysis, interpretation, and reporting. Many implemen- tation failures are associated with an initiative’s inability to use and apply data, information, and knowledge to influence current practices and motivate change.189

Many implementers underesti- mate the time, complexity, and resources necessary to develop functional data reports and systems, while overestimating the health care system’s ability to provide the desired data. Notably,

The VA Information Resource Center (VIReC) is a key resource for navigating VA’s complex data environment. VIReC meets this challenge by developing and disseminating an array of resources for understanding and working with VA data as well as advocating for data user needs. As an embedded resource center with national scope, VIReC is dedicated to increasing access to data, methods, and knowledge resources, and supporting the use of the EHR for operational and research evaluations.

PRE-IMPLEMENTATION Develop Measures and Data

The VA QUERI Roadmap | PAGE 37

electronic health record systems may have functional limitations that make it difficult to collect and provide certain types of data that are re- quired by an improvement project. Using a logic model helps identify

key data outcomes and ensures the measurement strategies used to collect this data are feasible, cost efficient, and timely.

Using a Logic Model as an Evaluation Tool A logic model is often used to guide evaluation planning. In particular, it can help:57,130

• Determine what to evaluate • Identify appropriate evaluation questions based on your practice • Know what information to collect to answer these questions

(indicator metrics) • Determine when to collect data • Determine data collection sources, methods, and measures

Logic models are also useful to help define the focus on four key evaluation domains:

1. Implementation (process) – Is the practice implemented as planned? Were all the activities carried out as expected? What influences the quality of the activities?

2. Effectiveness (outcomes) – Is the practice achieving its desired short-, medium-, and/or long-term effects/outcomes?

3. Efficiency – How much “product” or benefit is produced for a given level of inputs and resources? (See Economic Evaluation Outcomes)

4. Causal Attribution – Is the progress toward outcomes due to the practice? In health care settings, causal attribution can be more difficult to determine, especially in terms of program impacts. However, implementation science evaluation methods have made it easier to determine the causality between activities and outputs and short- to medium-term outcomes. This can usually be accomplished with minimal effort by using a combination of surveys, interviews, and analyses from clinical care databases to establish causality.

Figure 12. Evaluation Domains Using a Logic Model

Implementation E�ectiveness

Inputs Activities Outputs Short-term E�ects/

Outcomes

Intermediate E�ects/

Outcomes

Long-term E�ects/

Outcomes

E�ciency (link between boxes)

Causal Attribution (progression between boxes)

Graphic source.130

The boxes and arrows in the figure above indicate evaluation points where it is helpful to ask evaluation questions. As the practice pro- gresses through the logic model (i.e., as the implementation effort ma- tures), a new series of evaluation questions can be developed. While

summative outcome evaluation looks back at the entire model, quality process evaluation models identify reasons for less-than-successful practice implementation (i.e., where did the model break down?).

PRE-IMPLEMENTATION Develop Measures and Data

The VA QUERI Roadmap | PAGE 38

Figure 13. Illustrates how a logic model helps formulate evaluation questions and indicator metrics for your practice Process Outcomes

Inputs

EVALUATION QUESTIONS

INDICATOR MEASURES

Activities Outputs Short-term Outcomes

Intermediate Outcomes

Long-term Outcomes

Impacts Are resources adequate to

implement the practice?

Is practice implemented as

planned?

How many, how much was produced?

Change in knowledge, policy,

environment?

Change in system behavior?

Change in health status?

Change in population

health status?

What will be measured?

What will be measured?

What will be measured?

What will be measured?

What will be measured?

What will be measured?

What will be measured?

Graphic source 137

The logic model in Figure 14 shows key parameters of a nationwide, partnered implementation initiative that aimed to reduce health risks in a vulnerable Veteran population. The Reengaging Veterans With Serious Mental Illness in Treatment (SMI Re-Engage) program was a joint effort by mental health leaders and VA investigators to reduce premature mortality in Veterans with serious mental illnesses, such as bipolar or schizo- phrenia spectrum disorders.190,191 Veterans with these psychiatric diagnoses already face greater health risks than those without these diagnoses, but these health risks can be compounded when an individual goes long periods of time without treatment for both mental and physical health needs. The following logic model communicates the essential aspects of this population-level improvement initiative to all that are involved, even as the program has been sustained over time.192

Figure 14. Example of a Logic Model for Implementation of a Public Safety Intervention Re-engage Veterans with serious mental illness in VA services

OUTCOMESINTERVENTION

Inputs Activities Outputs Short-term Intermediate Long-term What you invest

Practice update: VA service use �����������������������

Evidence of Activities:

Clinical Practice Changes in: Changes in: Changes in: • Time • Funding • Partners • Sta� • Health

information technology

• 158 VA medical centers facilities engaged

• 5075 unique patients identi�ed for outreach in year1

• 88 sites received external facilitation beyond basic implementation strategies

• Total facilitation time per site

• Proportion of sites < 80% of updated documentation of Veteran status on outreach list

• General medical and mental health visits

• Inpatient hospitalizations

• Emergency department use

• Housing Assistance

• Psychosocial recovery use

• Veteran all-cause mortality

• physical health morbidity

• Housing status • Employment • Continuity of care

• Identify eligible cohort • Provide outreach list

to clinicians • Conduct outreach to

Reengage Veterans • Expedite return to care • Document outcomes

• Policy mandate • Education with training • Implementation toolkit • Centralized technical

assistance • External Facilitation • Audit with Feedback

Implementation

Contextual Factors Enabling factors: National VA e�ort to reduce homelessness among Veterans (many of which had a serious mental illness (SMI) diagnosis of a schizophrenia or bipolar spectrum disorder) combined with operational Leadership recognition that Veterans with SMI diagnosis are at increased risk for premature mortality if they experience gaps in coordinated physical and mental health services. Barriers: Many local clinicians faced competing demands and lacked dedicated time, information, and support to �nd this population prior to creation of SMI Re-Engage program.

El ev

at ed

m o

rb id

it y

an d

m o

rt al

it y

am o

n g

V et

er an

s w

it h

S M

I w it

h g

ap s

in s

er vi

ce s

en g

ag em

en t

> 1

y ea

r

While these models vary in size and complexity, understanding the implementation of an effective practice from start to finish assists in devel- oping measures of implementation outcomes, clinical outcomes, and longer-term service outcomes (population impact indictors). Over time, logic models will likely be refined and updated as real-world experience clarifies paths toward consistent outcomes (vs. theorized pathways) and measures are revised to better align with implementation activities. Figure 14 shows an example of a logic model in action.

More Resources55-57,130 can be found here for using a logic model for intervention and evaluation planning.

PRE-IMPLEMENTATION Develop Measures and Data

The VA QUERI Roadmap | PAGE 39

Establish Baseline Performance To assess the outcome of implementing an effective practice in a specific setting, you need an estimate of the Pre-implementation level of performance of the targeted measure of interest:193

• What performance data is currently available? Is this information useful to the current practice?

• What other information should be accessed? Is this data readily available?

• Are new data collection, cleaning, analysis, and reporting processes needed? If so, what resources are needed to complete this task?

• How often does data need to be collected and reported?

In most integrated health systems, like the Veterans Health Adminis- tration, operational program offices track performance on key clinical indices of care quality at the national, regional, hospital, clinical, and levels.

Internal and external benchmarks of quality and performance, as well as input from stakeholders, are used to define improvement target goals in the development of a logic model. Performance goals should align with local and national clinical care priorities and the organiza- tion’s mission and values.

Table 8. Sources of Quality Benchmarks for Performance Measurement

Examples of Publicly Available Examples of Department of

Veterans Affairs Specific

• National Quality Forum • National Committee for

Quality Assurance • Hospital Compare (Centers

for Medicare & Medicaid Service/Medicare)

• Medicare Merit-based Payment System Quality Payment Program

• Professional society clinical practice recommendations (e.g., American College of Physicians, ADA, AMA, American Academy of Physicians)

• Joint Commission on Accreditation of Health Care Organizations

• Veterans Health Administration Network Director Performance Plan

• Veterans Health Administration Medical Center Director Performance Plan

• Strategic Analytics for Improvement & Learning (SAIL)

• Department of Veterans Affairs MISSION Act quality standards

• Operational office clinical performance measures and dashboards

• Department of Veterans Affairs clinical practice guideline recommendations

Baseline data also enables the formulation of meaningful

performance goals for improvement by examining recent past per- formance and what is expected in the future based on implementing the effective practice. Baseline evaluations should include multiple aspects related to Pre-implementation performance in the targeted area:

• Clinical outcome (short, intermediate, and long-term) • Appropriateness (overuse, underuse, misuse) • Utilization • Consumer-reported measures (outcomes, experience,

preferences) • Stakeholder-reported experiences • Consumer-interaction • Value (outcome/cost) • Cost • Unintended consequences

A standard aspect of the process of selecting a quality measure should involve conversations with stakeholders (i.e., clinical staff from all levels, customers, and system administrators) to determine if poten- tial measures are considered meaningful and valid, particularly by frontline stakeholders.194 Leaders and frontline stakeholders should agree upon measures that are most important to improve care. Avoid starting with the wrong goal (e.g., beginning with measures of ac- countability stifles stakeholders’ desire to use data).195

Best Practice Unless a benchmark is imposed on your program by outside stake- holders or regulators, take time to collect preliminary results before choosing a goal. Knowing your performance at the start of the project not only provides you with baseline data against which you can measure your progress, but it will guide your decision-making about how high to set your initial goals to improve clinical or quality outcomes.196

When operational partners lack appropriate clinical measures for an implementation project, investigators can assist in developing low- cost, pragmatic evaluation measures that are of clinical importance; evaluate the appropriateness of care; and provide expertise supported by a clinical evidence base, including measure specifications (e.g., nu- merator and denominator, clarity, validity, reliability, risk adjustment) that are feasible to collect and applicable to modifying care.194,197,198 Make sure to specify the underlying source of data for a measure, methods used to calculate the measure, and the time frame for which to report the measure.193 Given the proliferation of performance measures that are inaccurate and invalid, we strongly advise you to consider the American College of Physicians guidance for reviewing whether a performance measure is a valid and clinically meaningful.194

PRE-IMPLEMENTATION Develop Measures and Data

The VA QUERI Roadmap | PAGE 40

For large-scale, multisite initiatives requiring the creation or modification of electronic medical record templates to track clinical metrics and process measures, the VA Engineering Resource Centers can assist in enabling these tracking and reporting system changes. Likewise, at each medical center or health system, local system redesign engineers and clinical applications coordinators can support making changes to the health information system for evaluation and reporting purposes in local settings.

Table 9. Strategies to Develop Meaningful Measures for Frontline Staff

Secure frontline stakeholder buy-in • Provide evidence of effective practice’s effectiveness and benefit to local setting • Ensure measures are perceived as valid, evidence-based, and reliable • Select measures that are relevant to organizational goals and mission

Use measures that matter and are meaningful to frontline stakeholders and managers • Involve frontline stakeholders in the process of selecting and vetting measures • Understand how stakeholders and staff make sense and use data or performance measures • Use measures to inspire and create positive change • Be mindful that some operational measures may not matter to stakeholders • Use measures that reflect direct consumer care

Optimize strategies for data collection and reporting to multilevel stakeholders • Limit number of reported measures • Ensure it is cost effective to collect and report the measure data • Present measures in formats that are accessible, clear, concise, and digestible • Ensure measures do not conflict with each other (compatible) • Develop measures that are comparable across services, sites, regions, and time • Use measures that can be provided in timely manner to change stakeholder behavior • Hold regular accountability meetings to review measures

Avoid planning pitfalls and counterproductive strategies • Overemphasis on measures of accountability or payment implying: 1) frontline stakeholders are motivated by fear or financial

motivations 2) leaders prioritize bottom line over consumer care • Inadequate estimates of time, resources, and analytic support to access, transform, report data • Assume data user literacy • Select measures that create extra work for frontline clinicians to input • Use too many measures that create confusion and disengagement (“metric fatigue”)

Adapted from148,199-201

PRE-IMPLEMENTATION Develop Measures and Data

The VA QUERI Roadmap | PAGE 41

Pre-Implementation Check List

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

Identify a Problem and Solution Identify high-priority need and goals Establish a shared understanding of the problem to be solved by the EP

Agree on effective practices (EP) and settings Use systematic reviews and literature to identify and employ best EPs

Clarify EP core elements, adaptation options (consumer, provider input) Establish fidelity measures, work with end users and consider contextual needs to explore adaptations, pilot and track adaptations

Engage Stakeholders Cultivate leadership/stakeholder support Seek input early and often from leaders, mid-managers, and frontline stakeholders

Assess capacity, including barriers and solutions to EP delivery e.g., Implementation Mapping, process frameworks

Package EP with delivery adaptations Match implementation strategies to barriers, create toolkit with menu of adaptations

Develop Measures and Data Design evaluation to match goals Select an evaluation design that is feasible and answers operational questions

Identify measures of success and data sources Develop data collection plan selecting a limited number of meaningful, pragmatic, and relevant measures, using qualitative methods strategically

Establish baseline performance Establish baseline performance and set realistic target benchmarks

The VA QUERI Implementation Roadmap

Implementation

What is being implemented?

IMPLEMENT AN INTERVENTION

Who and what settings are involved?

ACTIVATE IMPLEMENTATION

TEAMS

How is it being measured?

MONITOR IMPLEMENTATION

PROGRESS

Implementation involves finalizing the selection and deployment of implementation strategies to enhance uptake of an effective practice. Implementation can be enhanced through the strategic use of both transactional and transformational management strategies that help the frontline workforce successfully integrate the evidence-based innovation into routine clinical practice.108

• Implementation requires strategies and an implementation plan adapted/tailored to variations across potential adoption settings and sites to address local barriers and resource constraints.

• Implementation strategies combine technical, adaptive, and relational skill sets to spread and integrate the effective practice for Sustainment and use transformative leadership approaches to empower frontline workers to own their local implementation process.

• Measurement information and reporting systems should provide timely feedback on implementation progress relative to performance benchmarks, engage multilevel stakeholders in iterative problem-solving to adjust implementation plan strategies, and find ways to improve performance and spread the practice to other settings.

WHAT IS BEING IMPLEMENTED?

Implement an Intervention

What’s in this section?

Key Concepts

IMPLEMENTATION STRATEGIES

Action-oriented, theory-based methods that cause behavior change at the clinician, team, service, and

organizational levels to overcome barriers to adopting and sustaining effective practices.

ROGERS’ DIFFUSION OF INNOVATION

Classifies adopters into five categories based on typical characteristics of innovators, early adopters,

early majority, late majority, and late adopters.

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

Select Implementation Strategies

What are implementation strategies? How many do you need? How do you pick them?

Tailor Strategies to Local Settings

How do you use your needs assessment? What kind of adopters are you dealing with?

What should you be keeping track of as you tailor?

Disseminate Implementation Plan and Support Tools

What goes into a dissemination package? Who can help identify assets?

How can you sequence the dissemination?

Select Implementation Strategies Implementation requires you to decide which strategies to use to sup- port practice uptake based on Pre-implementation planning. Imple- mentation strategies are methods or techniques used to enhance the adoption, implementation, scale-up (or spread), and sustainment of a program or practice.71,126,202 They detail how to implement a practice as part of an implementation plan.71

Adopting an innovation is challenging because it requires changing the ways in which people act and think about delivering “routine” care

services. Implementation strategies are action-oriented, theory-based methods that support behavior change at the clinician, team, service, and organizational levels.

Implementation scientists have compiled and defined nearly 80 indi- vidual implementation strategies. The Expert Recommendations for Implementing Change (ERIC) study classified these according to nine categories of similar strategies (Table 10).126-128,203

Table 10. Examples of Implementation Strategies Across Nine Thematic Categories*

Use evaluative and iterative strategies Audit & provide feedback, conduct rapid cycle PDSAs, stage implementation scale-up

Provide interactive assistance Facilitation, local technical assistance, clinical supervision, centralize technical assistance

Adapt and tailor to context Tailor strategies, promote adaptability, use data experts, use data warehousing methods

Develop stakeholder interrelationships Identify and prepare champions, obtain formal commitments, identify early adopters

Train and educate stakeholders Develop educational materials, offer ongoing and dynamic trainings, provide consultation

Support clinicians Clinician reminders, create new clinical teams, revised professional roles, relay clinical data

Engage consumers Involve consumers and family members, use mass media, activate participants

Use financial strategies Alter incentive structures, access new funding, alter consumer fees

Change infrastructure Change record systems, change physical structures/equipment, change credentialing

* Adapted from 126,127

Implementation strategies aim to improve implementation outcomes, such as those detailed by Proctor and colleagues, including acceptabil- ity, adoption, appropriateness, feasibility, implementation cost, penetra- tion, and sustainability. Achieving quality implementation outcomes is related to attaining improved clinical- and service-level outcomes.160

In Resources, we provide a comprehensive list of implementation strategies compiled by the Expert Recommendations for Implement- ing Change (ERIC) study126,127 and others128 that organizes strategies according to functional themes, defines each strategy, and provides information relevant to applying strategies to particular contexts.

Matching Strategies to the Complexity of a Practice Many effective practices are considered “complex” because they con- sist of multiple components and steps as part of routine care. A com- plex practice may require the involvement and coordination of people working across different disciplines and levels of the health system to be successful.204 The interaction of complex practices with diverse set- tings, consumer populations, and delivery staff invites variation in how well a practice is delivered.

To ensure the successful delivery of complex, multi-component

practices, multiple implementation strategies can be selected to enact changes across organizational levels. The Pre-implementation exercise from Table 1 advocates for strategies to target specific barrier(s) identified in your capacity assessment and seek to lessen the influence of these negative contextual factors. When strategies reduce barri- ers, optimize fit with context, and cause changes in practice delivery, efficiency, and consistency, implementation quality should increase, resulting in improved outcomes.

IMPLEMENTATION Implement an Intervention

Implement an Intervention

The VA QUERI Roadmap | PAGE 45

Across implementation strategies, there is significant variation in each strategy’s level of complexity, intensity, and ability to target specific implementation barriers.64,205 Not every strategy is effective or necessary in a setting to achieve an improvement efforts’ desired outcomes.27,103 However, it is helpful to prioritize strategies that are feasible to enact in light of available resources (e.g., time, people) and that have a high probability of making a positive impact on delivery processes.

Avoid targeting barriers that are not feasible to change or have a low impact on outcomes.

This variability across diverse sites and settings underscores the need for an implementation plan that strategically combines technical and adaptive strategies to help local sites efficiently overcome barriers and integrate a practice for a sustained benefit.206 We encourage managers and leaders to broadly think about implementation strategies that fit into one of two broad categories:

Technical strategies include the application of trainings, resources, performance metrics, policies, and incentives to “push” an ef- fective practice into use from a top-down leadership perspective. These strategies are consistent with transactional management practices by promoting accountability to organizational goals and performance standards while supporting frontline stakeholders’ ability to achieve these goals with competence and quality.

Adaptive strategies “pull” the adoption of effective practices at the local level by engaging the intrinsic motivation and expertise of local stakeholders. These strategies are consistent with transformational management practices that seek to align a practice with local stakeholders’ interests, talents, and motivations to want to own and adapt the practice to make it sustainable for long-term use.

Tailor Strategies to Local Settings Select implementation strategies that improve the fit of an effective practice with local settings and consumers.

Implementation has traditionally been a top-down process largely overseen by either organizational leaders or investigators. This ap- proach aims to build delivery system capacity through technical strategies, such as providing frontline workers with tools, trainings, technical assistance and audits with feedback on implementation per- formance.83,110 This approach is well-suited for situations when many of the adopting sites or teams possess sufficient capacity or self-motiva- tion to adopt the effective practice with little support or connection to intrinsic sources of motivation.108

However, it is increasingly clear that basic strategies, such as guide- line dissemination or clinician training, are insufficient alone to improve outcomes among sites facing significant local barriers and resource constraints.207,208 Implementation experts propose several

practical—but theory-guided—ways to help you select strategies that proactively tailor a practice to local contextual needs and capabilities (e.g., practice facilitation, assess and redesign workflows).27,64

Until recently, there has been limited guidance on how to system- atically match barriers and facilitators reported by stakeholders in Pre-implementation with these strategies.64 Evidence suggests that implementers and investigators repeatedly rely on the strategies they are familiar with rather than using individual and organizational behavior change theories to match appropriate strategies to the local barriers and capacity constraints.209-211 We offer a better way to ap- proach this challenge.

Much like developing adaptation options to improve the fit of the effective practice for local end-users, you can tailor implementation strategies to account for local contextual factors as shown in Table 11.

Table 11. Examples of Factors to Tailor Implementation Strategies for Local Context

Factor to Address Through Tailoring Examples

Characteristics of the effective practice212-214 Complexity, effectiveness, cost, cultural appropriateness, compatibility with local workflows

Characteristics of adopting settings27,213 Variation in innovation capacity, resource availability, leadership structures, organizational readiness

Characteristics and preferences of implementing stakeholders213,214

Location (hospital vs. clinic), supporting resources, size of team responsible for outcome, knowledge, discipline, motivation

Level of contextual barriers27,83 Consumer, clinician, clinic/service, hospital, system, or policy/external environment levels

Known health disparities73,76 Race/ethnicity, gender, geography, sexual orientation

Phase deployment or practice integration205 Pre-implementation, Implementation, Sustainment

Target action of strategy103 Dissemination, capacity building, implementation process, integration of practice into local workflows, or scale-up through health system

Stage of effective practice adoption93 Innovator, early adopter, early-majority adopter, late-majority adopter, late adopter

IMPLEMENTATION Implement an Intervention

The VA QUERI Roadmap | PAGE 46

Depending on the complexity and scope of the practices being imple- mented, you should strive to design an implementation blueprint/ plan that uses the most cost-effective mix of strategies to achieve your implementation objectives. This means your plan should have an adequate range of strategies to address the variation in barriers found across adopting sites. This mix of strategies serves as an implementa- tion playbook for local implementation teams to cover essential tasks:

• Planning and preparation strategies • Dissemination strategies to deploy the practice to adopting sites

and contexts • Implementation process strategies • Capacity-building strategies for late adopters or sites with

significant barriers to practice adoption • Scale-up strategies to spread the practice within sites and to

other sites in a system • Strategies to integrate the practice for long-term sustainment

without external support

Finally, as you develop a comprehensive implementation blueprint/ plan, we encourage you to document the following decisions regard- ing the selection and use of implementation strategies.71,103

These evaluation details contribute to a learning health system by advancing institutional knowledge regarding what strategies were selected for deploying a specific effective practice and why. As an ef- fective practice spreads, include these details in the practice packag- ing materials to help clinical end users understand why strategies were selected, what they entail, when they should be employed, and by whom.

Table 12. Evaluator Guidelines for Naming, Defining, and Operationalizing Implementation Strategies

Name it Specify the behavior change technique or implementation strategy

Define it Operationally define the implementation strategy and any discrete components

Specify it

The actors Identify who enacts the strategy

The action Specify the action, steps, behaviors, deci- sions, or processes

Action target Identify strategy targets according to theory (i.e., barriers, facilitators)

Unit of analysis to assess implementa- tion outcomes

Identify the level of the organization or health setting the strategy targets (e.g., indi- vidual, clinic, service, hospital, etc.)

Dose Specify the dose (i.e., how much) of the implementation strategy is delivered

Time/phase Identify when the strategy is used

Implementation outcome

Identify the implementation outcome(s) likely to be affected by each strategy

Justification Determine theoretical or empirical justifica- tion for the implementation strategy

Ongoing process evaluation can help stakeholders understand when these implementation strategies worked, where they were effective (or not), and how much these activities cost to deliver effective outcomes. This information influences future decision-making by creating an evidence base of implementation strategies that would be most effec- tive and yield the greatest return on investment for similar effective practice deployments.71,215

Tailoring Your Implementation Plan for Rates of Practice Adoption For effective practices to impact populations, consider how practices are spread and adopted by clinical sites and consumers.216 The diffu- sion of innovations theory is helpful in understanding the process by which an innovation is communicated over time through members of a social system or network.108 Four key concepts relate to the concept of diffusion and are defined in Table 13.

Table 13. Key Concepts in Diffusion of Innovations Theory

Concept Definition

Innovation An idea, object, or practice that is perceived as new by an individual, organization, or community, and is intended to be spread

Communication channels

The means of transmitting the new idea from one person to another

Social system A group of individuals who adopt the innovation together

Time How long it takes to adopt the innovation

Adapted from8,108,216

Diffusion of an effective practice requires a multilevel change process that usually takes place in diverse settings with different implementa- tion strategies.

• At the individual level, adopting a new clinical practice involves changing the behaviors and thought processes of relevant clinical stakeholders (e.g., attitudes, beliefs, knowledge, motivations).

• At the organization level, it may entail redesigning clinical workflows and processes, changing standard operating procedures, or altering professional roles and responsibilities.

• At the community or systems level, diffusion can include using media, advancing new policies, or building coalitions to support a practice or initiative.

According to E.M. Rogers, the author of the diffusion of innovations theory, several factors determine how quickly and to what extent an innovation is adopted and diffused.108,216 By considering the benefits of an innovation (see 4), it can be introduced effectively to appeal to important clinical opinion leaders.

IMPLEMENTATION Implement an Intervention

The VA QUERI Roadmap | PAGE 47

Table 14. Key Attributes Affecting the Speed and Extent of an Innovation’s Diffusion

Attribute Key Defining Question

Relative advantage Is the innovation better than the innovation it replaces?

Compatibility Does the innovation fit with the intended audience’s needs?

Complexity Is the innovation easy to use?

Trialability Can you test the innovation before making a decision to adopt?

Observability Are the results of the innovation observable and easily measurable?

Adapted from8,108,216

Rogers found that the process of adoption follows a classic “bell curve” distribution of readiness and ability (Figure 15). The diffusion of inno- vations theory specifies five categories of adopters: innovators, early adopters, early-majority adopters, late-majority adopters, and late

adopters.108 When an innovation is introduced, the majority of people are either early- or late-majority adopters with fewer who are early adopters or late adopters. Very few are innovators who are the first to use the innovation.

Figure 15. Scale-up and Spread Process Across the Diffusion of Innovations Curve

Test & Pilot

Test of Spread Strategies

Mass Broadcast

Re-personalize

Innovato rs

Ear ly A

do pt

ers

Ear ly M

ajority Late Majority

Late Adopters

2.5% 13.5% 34% 34% 16%

During scale-up and spread, different types of implementation strate- gies may be more effective in reaching the adopters you are trying to engage. A recent report from the VA Evidence Synthesis Program93 highlights a few ways to reach audiences that are hard to engage in a “re-personalize” phase that emphasizes a tailored approach more often seen in the earliest phases of spread when customizing a clinical practice to a small number of innovator pilot settings

Growing research evidence suggests that early-adopter and early- majority sites are characterized by greater capacity (i.e., buy-in, readiness, or implementation capacity) to adopt and innovate than late-majority sites, which might lack knowledge, motivation, skills, or other resources and supports to readily adopt a practice.108,217 Settings with greater implementation capacity are also likely to self-select and use more advanced implementation strategies, enabling them to achieve better clinical/quality outcomes than sites with less improve- ment capacity.205,218

While early-majority sites may function well with standard implemen- tation strategies, such as training, tools, performance monitoring and

feedback systems, and technical assistance, 219 sites that are not early adopters may require more intensive, multi-component strategies like facilitation220 that employ a variety of discrete strategies over time to help local facilities problem solve implementation barriers. It is impor- tant to explore why these sites are not readily adopting the practice (e.g., due to limited resources or competing demands) in order to work with them to adapt practices and implementation strategies that fit their needs. The VA Evidence Synthesis Program’s Report on Scaling Beyond Early Adopters is a good reference for addressing these types of barriers.93

Roger’s adoption curve can help planners tailor the timing and alloca- tion of resources to build delivery capacity through basic implementa- tion support strategies48,83 for majority adopters and more intensive and expensive implementation strategies for later adopters.221 To plan for Sustainment, it is critical for a practice to be fully implemented prior to the end of initial implementation start-up funding. Thus, selecting implementation strategies to overcome the unique chal- lenges of late adopters helps a health system achieve a greater level of population impact for the long term.

IMPLEMENTATION Implement an Intervention

The VA QUERI Roadmap | PAGE 48

Disseminate Implementation Plan and Support Tools Implementation begins with the dissemination of the effective practice package to participating facilities/clinics and may include training of the effective practice, technical assistance, and monitoring and feedback on key clinical outcomes. Dissemination strategies depend on practical considerations, such as the:

Design of the implementation initiative

Geographic scope

Number of facilities/ networks

Funding to support strategies 

Health system leaders communicate expectations pertaining to the implementation effort, including differentiating resources available to the field, such as:

• Implementation timelines and milestone dates • Data monitoring and reporting systems • Repositories of implementation or effective-practice tools (e.g.,

toolkits) • Centralized delivery support resources, such as technical

assistance or other forms of support personnel (e.g., facilitators, improvement coaches, health information technology programmers)

• Local delivery support obligations (e.g., clinical champion, internal facilitation, obligation of local data programmers to support electronic-medical-record-template troubleshooting)

• Incentive structures, performance measure targets • Implementation and clinical benchmarks relative to an

implementation timeline

Scale-up plans can leverage system-wide assets that span the entire health system, including:38

• Shared information technology servers/networks • Social media and marketing mechanisms (i.e., local and

centralized services/channels) • E-mail listservs • Performance measure dashboards or clinical monitoring/

reporting • Practice-based research networks or quality improvement

collaboratives among clinicians • Existing clinical communities of practice (e.g., clinical program

office calls, social media)

IMPLEMENTATION Implement an Intervention

The VA QUERI Roadmap | PAGE 49

Effective dissemination involves understanding your key end-users, selecting the best communication channels for sharing informa- tion with internal and external stakeholders, and providing the most

appropriate products for each stakeholder group (See Table 15). Effective dissemination increases demand for the effective practice and builds partnerships across levels of the health system.

Table 15. Dissemination Strategies to Increase Practice Uptake Among Stakeholders

Stakeholder Communication Channel Mode Examples

Consumers and caregiv- ers

Interpersonal Word of mouth • Clinician counseling • Consumer peers/influencers

• Town halls at hospitals • Meet with patient advocacy organizations

Mass media

Written

• Brochures, consumer guides, handouts, care summaries

• Posters, flyers in clinics • Mailings

• Newsletters • Emails • Magazines, newspapers

Media • Information screens in clinics • Social media (Facebook, Twitter, YouTube)

• Public websites • TV and radio ads

Clinicians and clinical teams

Interpersonal Word of mouth

• Peers • Clinical champions • Academic detailing • Internal/external facilitators • Cyber seminars

• Conference presentations • Communities of practice • Quality improvement collaboratives • Practice-based research networks

Mass media

Written • Internal or professional newsletters, listservs Implementation toolkits

Media

• Online continuing education trainings • Electronic reminders • Electronic decision aids • Internal websites

• Professional blogs • Clinical practice guidance • Journal articles • Evidence syntheses

Service and facility level workforce

Interpersonal Presentations • In-service • Academic detailing

Mass media

Written • Standard operating procedures • New and ongoing training

• Local and national policies

Media • Centralized e-resources • Podcast • Health system messages

• Newspaper • Radio

System and policy level

Interpersonal Face to face • Policymaker briefs • C-suite briefs

• Patient advocacy

Mass media TV, radio, social media, and internet

• News stories • Editorials

• Commercials

Within the VA health care system, several operational entities support the spread and adoption of new practices through dissemination and diffusion strategies, including through a dedicated VHA Innovation Ecosystem that features devoted personnel, facilities, and program- ming to support the spread and scale of promising practices.35,36,38

The VA’s Center for Information Dissemination and Education Resources (CIDER) is another essential resource center dedicated to

supporting the implementation of effective practices by using vari- ous communication strategies, such as publications, cyberseminars, national meetings and conferences, social media, and both HSR&D and QUERI websites to share information with clinicians, managers, policymakers, investigators, and the general public.

IMPLEMENTATION Implement an Intervention

The VA QUERI Roadmap | PAGE 50

Planning for the Scale and Spread of an Effective Practice in a Health System In large health systems like the Veterans Health Administration, there comes a point when health system leaders must plan for the spread and scale of an effective practice for sustained integration.

The sequence of improvement conceptualized by the Institute for Healthcare Improvement is displayed in Figure 16, which illustrates the lifecycle of an effective clinical practice or quality improvement innovation. Moving from testing to proof of concept under local conditions using either clinical trials or traditional rapid-cycle testing, the goal is to yield a reliable and robust clinical innovation that works well across diverse settings and populations. As the sequence shows, the ultimate goal is to develop a solid implementation plan that not only integrates a practice into routine care processes but enables the process to rapidly diffuse to other settings and locations.

There is not a clear consensus on these popular terms, which we operationally define as the following:222,223

Spread refers to the adoption and replication of an effective practice from an original site of innovation to other facilities in a local hospital system, regional network, or national health system to reach more consumers eligible for the new practice (sometimes called horizontal scale-up).

Scale-up involves deliberate system-level investments in infra- structure and fixed costs to ensure that the effective practice can be delivered to large numbers of consumers with diminishing marginal costs over time for sustainable outcomes. Scale-up often entails strategic planning and use of strategies to influence budgeting, policy and legal regulations, and knowledge sharing across an organization or system.

Figure 16. The Institute for Healthcare Improvement’s Sequence of Improvement*

Plan

D o

S tu d y

Ac t

Plan

D o

S tu d y

Ac t

Plan

D o

Stud y

Ac t

Plan

Do

Study

Ac t

Plan

Do

Study

Ac t

Plan

Do

Study Ac

t

Sustaining improvements and spreading changes to other locations Make part

of routine operations

Implementing a change

Testing a change

Developing a change

Theory and Prediction

Test under a variety of conditions

Dat a ar

e us ed t

hro ugh

out the

seq uen

ce

*Reprinted with permission.

Spread and scale of effective practices is largely an understudied area in implementation science; however, we integrate the key elements necessary to spread an effective practice across units of a hospital and health system:214,223-227

• Emphasis on leadership and stakeholder engagement and alignment throughout the process

• Activation of social networks and communications at all levels of the organization

• Creation of a scalable effective practice intervention package and implementation plan

• Emphasis on an iterative and flexible approach to addressing implementation challenges at the local level while promoting accountability to performance standards

• Use of rigorous evaluation methods to monitor and revise implementation efforts

• Implementation of an effective practice with an emphasis on Sustainment

IMPLEMENTATION Implement an Intervention

The VA QUERI Roadmap | PAGE 51

Igniting an Initiative to Bring a Practice to Scale The Veterans Health Administration has considerable experience as a national health system in terms of developing innovative strategies to scale,36,38,224,228 yet considerable wisdom has been acquired from other sectors on this topic. In a 2015 Institute of Medicine workshop report,229 Joe McCannon, co-founder of the Billions Institute nonprofit for leading large-scale social change efforts, observed that three prerequisite factors are necessary for effectively taking an innovation to scale:

1. A promising practice with promising evidence to be built on through continuous learning and refinement going forward

2. Attention from influential leaders and stakeholders at national and local levels

3. A conducive environment that provides energy and attention for spread and scale fueled by a new policy or legislation, public attention, or an emergent crisis

The challenge is taking an effective local practice and scaling and spreading it more broadly in the face of real barriers. The health care “marketplace” is a crowded arena of competing practices in which any given practice is competing with a huge volume of new informa- tion, ideas, and innovations. There is little evidence that a promising practice will be taken up simply based on its merit or intrinsic value in the face of the following challenges.

Table 16. Barriers Undermining Change and Adoption of New practices

Conflicting values Inertia

Need for business as usual Resignation and apathy

Competition Fear of change

Ten attributes and behaviors distinguish typical scale-up initiatives versus exceptional initiatives as summarized in Table 17.

Table 17. Attributes of Typical Versus Exceptional Spread and Scale Initiatives

TYPICAL EXCEPTIONAL Comprehensive strategy development

Bias toward starting

Emphasis on consensus Invite debate and new ideas

General goals for expansion Explicit, time-bound aims

Design for success Design for success and scale

Broad knowledge of audience Detailed audience segmentation

One stimulant Many stimulants

One teaching/training method Many learning methods

Replication Adaptation

Summative evaluation is the priority

Formative evaluation (daily data) is the priority

Management gives the approval Management removes barriers

Source:229,230

IMPLEMENTATION Implement an Intervention

The VA QUERI Roadmap | PAGE 52

The Institute of Medicine summarized many observations by McCannon for successful scale and spread initiatives:

Get started. The natural tendency in a typical initiative at- tempting to scale a complex health practice is to take con- siderable time to comprehensively plan for all potential contingencies and outcomes that create inertia. The best way to address complexity is to engage in the process and get started.

Avoid pleasing all stakeholders. Typical initiatives place a strong emphasis on attempting to achieve consensus among all stakeholders, which is difficult to achieve in a health system. Focus on obtaining broad consensus on the aim of the initiative but not on the process of smaller steps to enact it.

Understand what full scale looks like. Successful initia- tives have clear, time-bound aims and a concrete vision of what the scale-up initiative needs to accomplish. Define what full scale looks like in terms of a number of sites that will spread the practice.

Design for Success and Scale. Most resource-intensive initiatives struggle to scale because they only design for success and do not address the need to reduce costs and develop economies of scale over time. Consistent with Roger’s diffusion of innovations theory, a scalable practice should be simple, user-friendly, and allow potential adopters the ability to easily test and clearly see the prac- tice’s benefits and value for their setting. Exceptionally designed scale-ups also plan for infrastructure needs as the practice spreads (i.e., people, financings, space, equip- ment and supplies, data collection, technology, logistics and oversite).

Understand the audience. As noted previously by Roger’s diffusion of innovations theory and a recent VA Evidenced Synthesis Report, it is crucial to understand that adoption of a practice occurs over a predictable range of distribution. It helps to segment the audience of stakeholders with the influence to adopt a practice based on different characteristics such as discipline (e.g., administrator, physician, nurse), geography (e.g., state, region), readiness for the practice (e.g., experienced, intermediate, novice), type of facilities (e.g., medical center versus community-based outpatient clinics), and the relevant mix of customers served by a facility (e.g., demographics, military service era).

Positive stimuli. A stimulus is the incentive or driving force for change. A typical health care initiative to spread a practice is dominated by one stimulus—payment. In contrast, successful initiatives use several different stimuli that range from positive to negative to drive change. Effective stimuli provide emotional connection, empower- ment, collaboration, recognition, enjoyment, transpar- ency, sense-making, scientific evidence, and if necessary, regulation or punishment.

Teaching versus learning. Commonly, initiatives are scaled out relying on one teaching method to learn an in- novation centered around a didactic lesson in which prac- tice users are passive learners. Successful scale initiatives rely on multiple methods of learning through hands-on applications like the Plan-Do-Study-Act cycle. Active im- plementation strategies—like practice facilitation—help frontline clinicians actively engage in mastering a practice through interactive problem-solving and reflection.

Replication versus adaptation. Consistent with our prin- ciples, scale and spread cannot rely simply on the static replication of an effective practice, but must have a focus on disciplined adaptation to address changes in consumer populations, policies, and local contexts.

Evaluation. While traditional scale-up initiatives often are heavily funded to determine summative outcomes (i.e., whether a practice works), exceptional initiatives focus on providing timely data (e.g., daily) to make frequent refine- ments to implementation on a regular basis.

Management Approach. In large scale and spread efforts, it is easy for leaders to fall into a trap in which they tightly manage efforts through data-monitoring systems that rank performance and provide approval to high perform- ers and punishment to low performers; this results in a culture of fear and manipulating the system. In compari- son, leaders who are successful at having impact at scale use their time productively to work with local and regional facility leaders to identify specific barriers to practice uptake and use their influence to remove these barriers. This management practice promotes a culture of positive collaboration versus fear and helps frontline stakeholders invest in the outcomes of the scale and spread initiative. Similarly, it is also important to develop shared culture and values among clinical stakeholders to embrace qual- ity improvement and the adoption of new innovations using implementation strategies.

IMPLEMENTATION Implement an Intervention

The VA QUERI Roadmap | PAGE 53

Leadership Uses Top-Down Practice Support to Push Change to Local Sites Implementation includes the use of leadership management styles to help enhance the uptake of new effective practices through a mix of transactional and transformational leadership practices. The behaviors of senior organizational leaders have a significant influence on creat- ing an organizational climate, culture, and environment conducive to innovation, learning, and improvement.

Transactional leadership practices are ideal for mak- ing incremental changes to current organizational poli- cies and procedures by using incentives and disciplinary power to motivate employees to meet performance objectives or targets.231-233 This style is well-suited for pushing out new effective practices and developing the systems to monitor performance.

Transformational leadership practices support collab- orative work through teams to identify a change, create a vision to inspire employees to persevere through the change, and work with employees to achieve a goal for the greater benefit of the organization. This leader- ship style motivates frontline employees and middle managers by engaging and empowering them to take an active role in the implementation process. Trans- formational strategies help improve employee morale and satisfaction by aligning tasks with personal values, encouraging skills development, and inspiring greater ownership of their work.234,235

In supporting the implementation of a new effective practice, transfor- mational leadership is instrumental in bringing stakeholders together to develop an improvement aim that aligns with stakeholders’ needs and demonstrating how it addresses local “pain points”. However, transactional leadership is useful for creating the “push” to get a policy, mandate, or clinical directive disseminated by creating structures that create accountability for local leaders and clinicians who implement the practice. As mentioned above, the use of positive and negative stimuli/incentives is important and generally should fall on an 80/20 ratio of positive- to negative-incentive strategies.

Transactional practices rely on implementation strategies that align rewards for meeting performance benchmarks with respect to goals and targets for adoption, implementation, and change in clinical outcomes. Transactional leadership practices aim to promote ac- countability using measurement-based systems, job descriptions, and clinician/leadership incentive programs aligned with implementation outcomes.

While transactional leadership strategies create structures to support quality implementation of an effective practice, evidence suggests that transformational leadership practices are more consistent with the tenants of a learning health system. In organizations emphasiz- ing these principles, there is greater uptake of practices, improved clinical and implementation outcomes, increased job satisfaction, and decreased employee turnover, burnout, and demoralization among frontline stakeholders.

IMPLEMENTATION Activate Implementation Teams

Activate Implementation Teams

The VA QUERI Roadmap | PAGE 54

Empower Bottom-Up Pull to Enhance Stakeholder Buy-In at the Local Level Implementation strategies should not only aim to build capacity to implement a specific effective practice, but to build broad capacity at the local level to implement future practices and innovations as skills are developed in these methods.

Empowering local frontline clinicians and managers to decide on the strategies to implement the effective practice in their facility fosters ownership in the process, and in turn, motivation and commitment. Clinical champions can garner motivational support by reframing the implementation process as an opportunity for local clinicians to learn

and own the process, rather than using punitive strategies to shame, penalize, or criticize the local team.

For complex innovations, like implementing a new model of care, empowering frontline staff (e.g., medical assistants, support staff ) by changing roles and responsibilities can be challenging,236 but these adaptations offer the benefit of reducing clinician burnout by allowing staff to work at the top of their license. Effective implementation ef- forts require a combination of skill sets to help sites adopt and sustain a practice:

Technical – Content expertise about the effective practice; proficiency in relevant evaluation practices pertaining to fidelity/adapta- tion assessment, performance benchmarking, and process evaluation; and knowledge on how to match capacity-building strategies to meet local site needs

Adaptive – Capability to address variations in local organizational and clinician capacities; readiness to adopt a practice by fostering user-centered design principles; problem-solving solutions to workflow redesign and technology challenges; and identifying and repairing local structural weaknesses to support practice sustainment

Relational – Capability to employ motivational and psychological strategies to encourage organizational change across interdisciplin- ary stakeholders and distributed levels of management and leadership

A learning climate is created when leaders integrate implementation efforts into organizational missions by prioritizing efforts, setting goals, and fostering accountability while empowering frontline stakeholders to make changes that lead to innovation and improvement.5,237 This ap- proach may be a change for leaders accustomed to traditional top-down management practices, but the long-term benefit is the development of resilient implementation capacity (e.g., knowledge, skills, communication channels) among frontline employees to support adoption of other effective practices.

Create Stakeholder Feedback Channels Reinforcing the establishment of formal communication channels al- lows stakeholders to share feedback on the progress of their imple- mentation efforts.238 For rapid learning to occur, stakeholders should feel comfortable not only sharing the details of successful strategies to overcome implementation challenges, but also the failures.

A prerequisite condition for these channels is the establishment of psychological safety,239,240 the perception that stakeholders can take interpersonal risks without suffering punishment.241 In a psychologi- cally safe environment, individuals are willing to contribute ideas, questions, and concerns, which supports learning. In environments where leaders do not foster psychological safety, individuals are less willing to speak up for fear of being perceived as ignorant, incompe- tent, negative, or disruptive.5

Health care environments can foster hierarchical cultures where individuals of lower status self-censor for fear of retaliation or punish- ment.242 Environments like this discourage learning through rapid experimentation, failure, and reflection.243 Effective teamwork is critical in health care settings where clinical teams represent the basic orga- nizational unit; however, a recent review found that a psychologically safe work environment was the most predictive factor of team and

organization success and was the cornerstone of four other predic- tors of performance—clear goals, dependable coworkers, meaningful work, and a belief that work has impact.240

Senior implementation leaders can model safe communication standards to local implementation teams by setting clear ground rules on learning collaborative/community sessions. Implementation strategies, such as external facilitation220 or improvement coaching, also provide an opportunity to coach dysfunctional teams to cultivate positive communication patterns.

Increasing evidence indicates that when implementation strategies help local leaders (e.g., directors, managers, supervisors) adopt posi- tive management behaviors, it translates into psychologically safe and high-functioning teams. This improves organizational outcomes through rapid learning and better clinical, safety, and employee outcomes.5 The ability to foster effective feedback channels could take weeks to years based on the commitment of senior leaders and man- agers to demonstrate consistency in communicating expectations for speaking up about errors, mistakes, or failures by positively affirming rather than punishing the messenger.

IMPLEMENTATION Activate Implementation Teams

The VA QUERI Roadmap | PAGE 55

Report Progress to Stakeholders For health care organizations to learn, innovate, and improve, frontline managers and clinicians need timely feedback about their current level of performance relative to goals and evidence-based bench- marks. Organizational learning that drives improvements at all levels of the health system is not only driven by data but also feedback loops among stakeholders that foster accountability and reflection on what is working or not working in implementing a new practice or innovation.196

In Pre-implementation, leaders and stakeholders develop a data collection and measurement plan that builds a continuous data monitoring and reporting system across all levels of the organiza- tion. Reporting systems must strive to balance performance feedback with engagement strategies to support positive clinician behavior change.200 Implementation progress should therefore include imple- mentation progress at multiple levels, including system, regional network, facility, service, unit/team, and individual clinician (when appropriate).

Performance feedback should be meaningful and relevant to clini- cians’ direct delivery of care, whether it be through a service, team, or individual clinician. Feedback should be presented in easily digestible formats that help clinicians understand what actions to take to modify

their current care behaviors. Moreover, regular performance meetings with senior leaders, managers, and frontline clinicians should support reflection and problem-solving to use feedback data to drive local ef- forts for continuous improvement on key outcomes.199

Audit and feedback are essential implementation strategies used to support thoughtful engagement in performance feedback. Audit refers to the assessment of performance to a specific target or bench- mark. Feedback is the communication of that performance. Frequent- ly, data is provided without a comparison standard to interpret current performance (from within or outside the organization). Evaluators should monitor how frontline clinicians make sense of feedback data and respond to stakeholders’ concerns about the validity, reliability, and accuracy of reported measures.

Program evaluators can also develop communication channels to pe- riodically share the results of formative evaluations with stakeholders across the system through practice-based networks, communities of practice, and virtual learning collaboratives. This process reinforces the participation of stakeholders in formative data collection and can be impactful by rapidly communicating failures and successes across the organization to support improved delivery (e.g., fidelity, adaptations) and/or use of specific implementation strategies.

Make Data Accessible to Stakeholders Operational stakeholders need continuous monitoring of imple- mentation activities and outcomes (performance measures) to know whether they are achieving goals or evidence-based benchmarks. Performance feedback data that is hard to access and share makes it less timely and useful for implementation and quality improvement purposes. However, simply reporting performance data is not enough to ensure high quality, best practices, and effective care.244

Well-designed performance measures help managers and frontline clinicians assess and improve program outcomes and the processes linked to them. In Pre-implementation, stakeholders create a logic model to help stakeholders understand these relationships. In turn, a limited number of measures should have been selected that could be easily obtained at the local level on a regular frequency and shared for iterative improvement efforts.

Data that is accessible and transparent should be available at all levels of the organization, but particularly at the service, unit, and clinician levels. This data must be digestible and relevant to operational data users to make changes in local clinical implementation efforts. Data can be made accessible and usable by posting it on performance dashboards, posting visible summary reports in clinical breakrooms, or sharing and discussing it during recurring meetings between clini- cians, managers, and senior leaders.

Accessible and continuously reported data enables local implementa- tion teams to use data to experiment and refine implementation pro- cesses and then assess the impact of these changes in relatively short periods of time, which is essential for helping an effective practice fit in a local setting.

IMPLEMENTATION Monitor Implementation Progress

Monitor Implementation Progress

The VA QUERI Roadmap | PAGE 56

Adjust Plan Based on Feedback Learning health care systems and high-reliability organizations use data and measurements to drive improvement and innovation. Well-designed performance measures are insufficient to bring about changes without an improvement framework or process to make sense of this feedback.

“Until a new process or practice is completely routine, each use of it is an experiment”

– Organizational Learning Expert, Amy Edmondson5

Senior leaders create the strategic vision for implementing a new practice in a health system by providing the resources and perfor- mance standards to hold local leaders accountable for carrying out

an implementation plan. While implementation strategies provide the overarching strategy for implementing an effective practice and navigating barriers, implementation at the local level is often driven by small, incremental, and cyclically-implemented tests of change.

Effective performance measurement systems encourage organiza- tional learning and improvement by using data to help managers and clinicians assess and improve program outcomes and the processes linked to them. Feedback from performance measures helps lo- cal implementation teams know how they are doing in achieving implementation goals or performance benchmarks. Failure to achieve performance goals encourages the use of improvement methods like Plan-Do-Study-Act cycles.245,246

Figure 17. Plan-Do-Study-Act Cycle for Iterative and Continuous Learning

Study

Ac t Plan

Do

• Ready to Implement?

• What changes need to be made in next cycle?

• Adopt, re�ne, or abandon?

• Compare data to predictions

• Complete data analysis

• Summarize learnings

• Try the plan • Document problems

and unexpected �ndings

• Start Data analysis

• De�ne objective, make predictions

• Create a plan to Do Tests (who, what, when, where)

• Plan for data collection

As Figure 17 shows, a Plan-Do-Study-Act cycle is an iterative problem- solving approach that addresses gaps between current performance and benchmarked goals. Upon identification of a gap in performance, reflect on the possible causes of the failure and develop a solution (plan), test the solution (do), analyze the resulting outcomes (study), and act on whether to refine, stop, or continue the solution.

To improve and adopt new practices, organizations must be will- ing to support systematic experimentation at the local level. While failures are inherent to this process, these are small, deliberate failures that offer feedback to learn and adjust processes and behaviors.

Plan-Do-Study-Act cycles also offer a systematic way to help local improvement teams adjust to changes in clinical resource availability, clinical priorities, or the discovery of unintended consequences caused by a new practice. Finally, Plan-Do-Study-Act cycles offer a micro- approach to selecting, implementing, testing, and adjusting adapta- tions to an evidence-based intervention to assess what helps optimize fit with consumers and clinicians in a local context while maintaining clinical effectiveness.9

IMPLEMENTATION Monitor Implementation Progress

The VA QUERI Roadmap | PAGE 57

Implementation Check List

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

Implement an Intervention Select implementation strategies Deploy implementation strategies to promote uptake of EPs

Tailor strategies to local settings Tailor based on mechanism of behavior change, EP complexity, setting variation, phase of implementation, etc. Consider technical, adaptive, and relational skill sets

Disseminate implementation plan and support tools Create a playbook for frontline providers and interventionists

Activate Implementation Teams Convey top-down practice support “push” to local sites from leadership Transactional-focused strategies promote accountability and technical skill building

Empower bottom-up “pull” to enhance stakeholder buy-in at local level Transformational-focused strategies empower individual stakeholders to develop skills, strengths, and motivation to “own” the change process

Create stakeholder feedback channels Foster psychological safety and learning climates/ cultures

Develop Measures and Data Report progress to stakeholders Provide feedback on goal attainment to multi-level stakeholders

Make data accessible to stakeholders Effective use of data helps foster learning and monitoring of clinical impact

Adjust plan based on feedback Local teams should iteratively refine implementation to optimize local fit and uptake

The VA QUERI Implementation Roadmap

Sustainment

What is being implemented?

SUSTAIN AN INTERVENTION

Who and what settings are involved?

TRANSITION OWNERSHIP TO STAKEHOLDERS

How is it being measured?

ONGOING EVALUATION AND REFLECTION

Sustainment3 is a process that involves:

• Assessing the effective practice’s performance on an ongoing basis

• Adapting the effective practice over time to maintain or improve clinical effectiveness

• Periodically reviewing to decide whether to continue to sustain a practice or de-implement it

• Creating a business plan to support ongoing effective practice delivery

The goal of this phase is to integrate the effective practice into habitual workflows in clinical care settings and employ it at the local health care facility level without special external support, such as grant or start-up funds.

A defining characteristic of a sustainable practice is that it is viewed as an essential part of ongoing routine clinical care and is valued by both providers and by organizational leaders at the local and system levels.

WHAT IS BEING IMPLEMENTED?

Sustain an Intervention

What’s in this section?

Key Concepts

SUSTAINABILITY The extent to which an effective

practice can continue to be delivered, especially if external support or

funding ends outcomes

DE-IMPLEMENTATION A situation when an established effective practice should be reduced, replaced, or stopped because it has been found ineffective, harmful, inefficient, or no longer necessary even in the absence of a

specific superior alternative practice

VA Health Economics Resource Center

KEY RESOURCES

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

Develop business plan to continue EP

What goes into a business plan? What types of analysis might be needed?

What kinds of costs should be considered?

Monitor for changes in EP, whether different EP is needed

Why change your course of action? What are the options for de-implementing?

How can you approach this process?

Weigh costs of maintaining EP

What matters other than EP success?

Sustainability describes the extent to which an effective practice can continue to be delivered, especially if external support or fund- ing ends.8,247 An effective practice only has an impact on its intended consumer population if it is reliably delivered over time.

Sustainment is not a final endpoint for frontline clinicians unless the practice is de-implemented or stopped in favor of a new or more effec- tive practice. The goals for Sustainment are to:

• Integrate the practice into care so it is viewed as a routine practice or habit

• Continue to fully deliver the core components of the effective practice over time

• Take ownership of the effective practice by building ongoing costs and processes to sustain the practice in local operating budgets and decision-making

Sustaining an effective practice involves a combination of disciplined use of data and operational flexibility to handle a rapidly changing health care environment. Local implementation leaders work with frontline stakeholders to develop and test adaptations to support program effectiveness and fit for new and/or changing contexts and population needs.54,59,248 Ideally, implementers strive for continuous improvements over the prior periods’ performance even as returns diminish over time as the improvements become smaller.

Do not assume a practice will continue after start-up funds for an implementation initiative go away. Rather, research suggest that the majority of implementers struggle to sustain the gains achieved during Implementation efforts due to an inability to fully sustain all components of an effective practice.249-251

Reductions in practice outcomes suggest a poor fit between the practice and its context. Following initial implementation, practices that sustain performance gains after adoption (“holding the gains”) represent a neutral to slightly less favorable implementation out- come because outcomes are not continuing to improve. A negative result happens when outcomes relapse over time to levels close to Pre-implementation.252

As Figure 18 shows, multilevel factors can interact to influence an organization’s capacity to sustain a practice. These factors involve

characteristics of the effective practice, the people delivering the inter- vention, implementation processes, factors specific to an organization, and external environmental influences. Many of the specific determi- nants influencing sustainability are cataloged by investigators to help guide planning and decision-making.248,253

Table 18. Multilevel Factors Influencing Sustainment: The Integrated Sustainability Framework

Factors External to Your Health System • Sociopolitical • Funding

Environment • External leadership • Values, needs, and

priorities

Sustainment of Clinical Practice

• Continued program use/ implementation

• Continued health impact or bene�ts

• Capacity building

Factors Internal to Your Health System

• Resources, funding • Leadership support • Sta�ng, turnover • Program champions

Processes • Stakeholder

engagement • Training, supervision • Technical assistance • Program

evaluation/data • Adaptation

Characteristics of Interventionists

• Implementer or provider characteristics

• Knowledge, skills, expertise

Characteristics of the Clinical Practice

• Perceived bene�t/need

• Fit with context and population

• Adaptability, usability

Adapted from248

SUSTAINMENT Sustain an Intervention

Sustain an Intervention

The VA QUERI Roadmap | PAGE 61

Planning for Sustainment The Program Sustainability Assessment Tool offers an efficient way to better understand multilevel factors that can be underdeveloped and negatively affect a practice’s ability to be sustained.

The tool organizes these factors across eight domains, including:

• Funding stability • Partnerships • Communications • Organizational capacity

• Program evaluation • Program adaptation • Strategic planning • Environmental

Strategic planning—the capacity to define the program’s goal, direc- tion, and strategies—is the central domain on which all the other capacities are dependent. Planning for Sustainment can take place during Pre-implementation118,196 by using resources like the Program Sustainability Assessment Tool to assess an organization’s areas of weakness in supporting new clinical practices. Accordingly, you can target specific organizational structures and processes for strengthen- ing during Implementation using strategies suggested by Leeman and colleagues to improve integration of the practice into routine care, build capacity for sustained use, and support the scale-up of the prac- tice to other clinics, services, and facilities in your health system.27,83,103

Table 19. Classifications of Implementation Strategies Used to Build Capacity for Effective Practice Sustainment

Classification Target for strategy action Examples of specific Expert Recommendations for

Implementing Change (ERIC) implementation strategies

Integration strategies

Factors that enable or impede optimal integration of a specific effective practice into a specific setting by helping frontline stakeholders address barriers to implementation among clinicians or inner organiza- tional setting

For a specific effective practice:

• Institute reminder systems • Revise clinician roles and responsibilities • Provide supervision/ feedback • Modify medical record systems • Implement tools for quality monitoring (process, fidelity)

Capacity-building strategies

Motivation and capability to engage in imple- mentation process strategies (in general but not to a specific effective practice) carried about by individuals, teams, and systems that build frontline stakeholders’ general delivery capacity by target- ing individual-level and implementation process determinants

Across multiple settings:

• Training to build general capacity • Technical assistance and facilitation for implementation

processes • Tools to support implementation processes

Scale-up strategies Motivation and capacity to integrate a specific effec- tive practice into a health system using individuals, teams, and systems that build frontline stake- holders’ innovation-specific delivery capacity by targeting individual, inner-setting, and outer-setting barriers or weaknesses

Across multiple settings:

• Training to build capacity specific to effective practice • Technical assistance and facilitation (specific to effective

practice) • Implementation toolkits • Learning/quality improvement collaboratives • Benchmarking • Infrastructure development • Changes to service fee lists or medication formularies

Adapted from103

Some of the general capacity strengthening strategies may be ac- complished by support services funded by centralized operational offices, such as technical assistance, training, or practice facilitation/

improvement coaching. Some innovation-specific capacity-building strategies should be carried out by local implementers to optimize fit of the new practice with the setting.

SUSTAINMENT Sustain an Intervention

The VA QUERI Roadmap | PAGE 62

Develop a Business Plan to Continue Effective Practice as Routine Care Increasingly, local health care organizations are asked to take a more entrepreneurial role in the ownership of effective practices and qual- ity improvement projects after initial implementation resources go away.254,255

During Implementation, local hospital leaders and managers plan for Sustainment in a critical and systematic way. Program evaluation data from Implementation can help develop a customized business plan.

When planning from a business perspective, the focus becomes cen- tered on customer service and creating value in a financially sustain- able way. A business plan requires thinking about:254

• Establishing new sources of funding support • Justifying how a practice fits in the organization’s mission,

structures, and clinical processes • Setting performance benchmarks to determine practice

effectiveness • Spreading costs among and coordinating with partners (other

services, operational offices) • Demonstrating value of the practice • Analyzing efficiency • Determining cost per unit of service • Calculating break-even points

To make a case for sustaining or spreading a practice within a health system, consider presenting the effective practice’s cost analysis, showing the return on investment on the implementation strategies used to promote practice uptake.

Decision makers may also desire a cost-effectiveness analysis, which compares costs to changes in quantitative measures of health-related outcomes or standardized outcomes, such as quality-adjusted/disabil- ity-adjusted life years as in the case of a cost-utility analysis.172,178,256 Consider presenting analyses from the perspectives of multiple stake- holders, such as payers, consumers, and communities.

A business plan can substantiate shared costs by different service departments and systems-level program offices. For example, effec- tive practices designed to treat complex health conditions, such as cancer, heart failure, or serious mental illness, need care coordina- tion across multiple clinicians.

Costs need to be spread across participating services or units. Some fundamental delivery capa- bilities may be best “owned” by a centralized operational office, such as ongoing provision of audit and feedback or technical assistance on complex issues (e.g., billing and service coding decisions). Understanding how these costs are shared locally and with na- tional or regional program offices can help improve local budget decision-making.212

Monitor for Change in Effective Practices and Whether a Different Practice is Needed There are several reasons to reassess the decision to sustain or discon- tinue use of a clinical practice:

• Change in the evidence for the benefits (or harms) of a practice • Change in clinical practice guidelines related to the practice • Release of a new or similar clinical practice with greater benefits

While the health care field puts a strong emphasis on adopting effec- tive practices, less attention is put on the equally important decision to de-implement low-value care practices.

Low-value care can be defined as health care services or treatments that provide little or no benefit to consumers, have the potential to cause harm, incur unnecessary costs to consumers, or waste limited health care resources.40,41

De-implementation is a situation when an established effective practice is reduced, replaced, or stopped because it has been found ineffective, harmful, inefficient, or no longer necessary even in the absence of a specific, superior alternative practice.44,45,47 Table 20 illustrates the continuum or reasons for de-implementing an effective practice.

The VA’s Health Services Research and Development Service Health Economics Resource Center (HERC) is as an embedded national resource center that supports VA economic research173 to increase health care efficiency and value for Veterans and the nation. HERC provides tools and resources to assist researchers and operational leaders in conducting economic analyses for implementation research.

SUSTAINMENT Sustain an Intervention

The VA QUERI Roadmap | PAGE 63

Table 20. The Continuum of De-Implementation of Low-Value Care Practices

Reduction or partial reversal Related

replacement Unrelated replacement

Full reversal or discontinuation

Description Reducing the frequency, breadth, intensity, or scale of an existing practice so it is only provided to a sub-group of consumers who have been proved to receive the most benefit

Replacing an existing practice with a closely related and more effective practice

Replacing an existing practice with a more effective inter- vention unrelated to current care

Universally stopping an inef- fective or contraindicated practice

Examples • Lengthen interval of cervical cancer Pap smear from one to three years

• Initiate breast cancer screening at age 50 vs. age 40 for women of average risk

• De-intensify diabetes medications in older adults

• Replace inhaled corticosteroids with long- acting beta agonists or muscarinic antagonists

• Perform coronary procedures via radial artery rather than femoral artery

• Physical therapy instead of opioids

• Chronic stable angina treated medically instead of with coronary bypass surgery or percutaneous coronary interventions

• Stopping hormone replacement therapy for prevention of chronic conditions in postmenopausal women

• Choosing Wisely recommendation against pre-operative or procedure electrocardiogram screening for asymptomatic consumers with low perioperative risk of death or myocardial infarction

Effects on clinicians

• May reduce clinical efforts by requiring only continued effort on a subset of consumers

• New clinical efforts needed for risk stratification to identify and appropriately treat eligible consumers

• Little impact on clinical efforts and practice patterns over the long term because substituted practice largely fits existing workflows

• Requires engagement and buy-in from at least two different stakeholder groups to implement the unrelated practice and change old clinical behaviors and workflows

• Focus on de- implementation of procedure and no new practices added

Adapted from44,45

In large health care systems like the Veterans Health Administration, de-implementation is highly relevant when addressing the common occurrence of overtreatment and over-testing among older adults. Most clinical practice guidelines continue to focus on how to escalate care intensity rather than on how to de-intensify treatments and tests.

National campaigns—such as Choosing Wisely®257—call for the reduc- tion of inappropriate services or prescriptions; however, most of these recommendations focus on reducing one-time diagnostic procedures or treatment services at the start of a discrete episode of care. There is an absence of guidance for both consumers and clinicians on how to begin to stop or reverse the intensity or frequency of clinical practices that are part of a consumer’s ongoing care for a chronic condition.

Implementation scientists have observed a number of significant barriers to de-implementation among health care clinicians and organizations:258

• Few clinical trials examine de-implementation, leading to less evidence to guide clinicians’ decisions in terms of de- implementing care or stopping a clinical practice

• Clinicians may be suspicious of the degree to which cost savings are behind a de-implementation recommendation

• Clinicians may be concerned about how de-implementing a practice might affect their score on performance measures or their exposure to legal liability

• Lack of time and effective decision-support tools limit discussions related to de-implementing low-value practices in a consumer’s treatment plan

SUSTAINMENT Sustain an Intervention

The VA QUERI Roadmap | PAGE 64

Using the Roadmap to Systematically De-Implement Low-Value Care Practices De-implementation is a new and evolving area of health care. While it is not always clear when and how to de-implement a clinical practice, the roadmap provides a general framework with which to approach this practical problem:

1. Identify and prioritize practices that may be appropriate for de-implementation - Refer to clinical practice guidelines for practices to reduce, replace, or discontinue - Use local outcomes data to identify practices that no longer yield high-value results - Look to health services literature on guidance for de-implementing services in a particular area of clinical care

2. Engage relevant stakeholders in the process of planning and de-implementing - Seek to understand various perspectives on how to most efficiently overcome barriers and change clinician behaviors

3. Assess relevant multilevel factors when de-implementing a practice - Consider how widespread the practice is in use in the system - Anticipate the possible resources needed to de-implement the practice - Consider the impact of de-implementation on clinicians and organization units

4. Use behavioral change theory to identify and select implementation strategies259-261

- Consider training, education, reminder prompts, and audit and feedback strategies as necessary but insufficient to change clinician behavior

- Look to social psychology and behavioral economics for more advanced strategies to change clinician behavior to cease or reduce the low-value practice and adopt new alternative behaviors and decision-making processes262

5. Implement and evaluate the execution of de-implementation strategies - Use performance measures developed to monitor the de-implementation of health care services to overcome clinical inertia and hold

clinicians accountable263,264

Weigh Costs of Maintaining the Effective Practice Use a strong evaluation plan to reassess the continued sustainment of an effective practice. Sustainable clinical practices can demonstrate program value through measurable results and program evaluation.255 Clinical practices are more likely to be maintained when they show— through established quality indicators and performance measures— that the practice benefits key stakeholders.265 Nonetheless, while the strength of practice outcomes is helpful, evidence suggests that some programs continue to persist even when they do not show clinical effectiveness.

Another consideration is the return on investment of scarce resources. Could the resources that are allocated to sustain a specific practice be employed more effectively to generate greater returns on consumer health outcomes and quality indicators? This is both an economic and ethical cost-benefit analysis that should be considered from multiple perspectives, including the perspective of consumers and their fami- lies, the hospital, and the community.

SUSTAINMENT Sustain an Intervention

The VA QUERI Roadmap | PAGE 65

Provide Management Support The goal for the effective practice implementation project is for the effective practice to be supported internally without the need for external support or management. Because of this, for prolonged sus- tainment, leadership and management should recieve guidance and training on how to support the practice, including the identification and use of implementation strategies and other required resources, monitoring of practice use and maintenance over time, and opportu- nities for adaptation where appropriate. 266

An implementation plan (i.e., an implementation playbook) that

describes the implementation processes for the effective practice is essential for operations to ultimately own and sustain the practice over time. As the practice is integrated to become routine, implemen- tation activities should shift from innovation-specific capacity-building strategies to general capacity-building activities, including training (e.g., booster sessions, onboarding new staff ), establishing standard operating procedures, updating tools, and maintaining evaluation and feedback systems.

Plan and Budget for Resources Local leaders must understand how the investment into the imple- mentation impacts their annual operating budget. Commonly, health care leaders want to know the total annual cost of the clinical practice, its components (including implementation costs), return on invest- ment, and budget impact analyses help justify the benefit of com- mitting resources in one practice compared to investments in other clinical solutions.

In an era of rising health care costs, competing demands to imple- ment new practices, and constrained resources, economic evalua- tion measurement practices establish accurate estimates of practice operating costs. Implementation evaluators should work with health system economists and business analysts to adopt evaluation meth- ods to track and develop accurate financial estimates for budgeting purposes.267

Economic evaluation of implementation costs is an evolving area of research. Presently, most estimates of these costs are based on

periodic activity-driven assessments of the effort and resources em- ployed across each phase of the implementation process.268 The Cost of Implementing New Strategies178 is a new, activity-based evalua- tion approach that enables the estimation of both direct and indirect expenses needed to complete an implementation task or activity. The Cost of Implementing New Strategies evaluation demonstrates that resource allocation, particularly of direct costs, varies by stage of implementation. Cost structures can also differ between competing implementation strategies that are delivering the same practice.

The beliefs of middle managers and medical center leadership in terms of program importance are key to garnering support for resources (e.g., space, dedicated full-time equivalent, materials). A manager’s ability or willingness to promote or support budget fund- ing for a practice depends on the alignment of practice activities with hospital priorities and needs.85 Hence, it is essential to work with staff and leadership to understand the value of an effective practice.

SUSTAINMENT Transition Ownership to Stakeholders

Transition Ownership to Stakeholders

The VA QUERI Roadmap | PAGE 66

Support Continuous Learning and Innovation Best practices related to effective practice delivery, evaluation, and conceptualization are continually evolving based on new research findings and practice implementation trials. Similarly, best practices related to program evaluation are under continual refinement to assist staff and leadership in obtaining the most accurate and useful view of their clinical practice performance.

Some complex practices that have been widely implemented in health systems using strategies like communities of practice, virtual learning, or quality improvement collaboratives. These cases represent a cost-effective sustainment strategy for stakeholders to share innova- tions, problem solve emergent challenges, and share new information. Locally, managers and leaders should ensure local training budgets anticipate the need for training new staff, providing booster sessions, and supporting professional development of new skills that support quality improvement for the effective practice.

Finally, new quality metrics are continually being developed and launched to the field by health care leadership with the expectation that sites and operational programs will be aware of such measures and their application. Rollout of new performance measures should plan to provide basic training and education materials to ensure staff and clinicians can accurately use and interpret new metrics. Sites must maintain an awareness of these shifts in knowledge and the ways to apply such shifts to the oversight and delivery of their effective practice.

SUSTAINMENT Transition Ownership to Stakeholders

The VA QUERI Roadmap | PAGE 67

Consumer Outcomes A key aspect of deciding whether to sustain an intervention is the demonstration that an effective practice continues to generate quality outcomes for consumers, particularly at a population level. Health sys- tems should leverage information technology systems to extract ad- ministrative and clinical data to provide low-cost reporting systems to enable system leaders to monitor individual and population consumer outcomes to ensure continued benefits of a practice with respect to access, morbidity, mortality, unnecessary use, accidental harms, and

consumer savings with respect to more efficient care delivery.

Outcomes should include consumer-reported outcomes, such as im- proved quality of life and satisfaction with care. When a practice fails to consistently attain benchmark standards of effectiveness, quality, or consumer satisfaction, system leaders should review alternative options to enhance outcomes, starting a new cycle of de-implementa- tion and Pre-implementation planning.

Delivery of the Effective Practice Consistent with new perspectives on the roles of fidelity and adapta- tion in sustainment, evaluators and clinicians strive to optimize the fit between the practice and the external environment in which it is delivered.59

Ongoing monitoring and assessment of clinical outcomes and contex- tual characteristics are needed to provide feedback regarding practice effectiveness. Using practical measures, such as checklists to assess aspects of fidelity and adaptation, can enable clinicians to be sensitive to changes in the fit between practice and context. Similarly, continu- ously engaging stakeholders over time can increase fit while develop- ing solutions that might cause problems with Sustainment.

Consistent with a learning health system approach, clinicians are encouraged to test effective practice adaptations using pilot trials and rapid-cycle tests to determine how to optimize and improve the practice rather than let it remain static.

As a practice spreads over diverse settings and consumer popula- tions within a system, evaluators should ensure adopting sites fully implement all components of a practice to avoid partial Sustainment. Furthermore, managers and clinicians should establish communica- tion channels to problem solve delivery challenges and share positive adaptations and practices (e.g., communities of practice, virtual quality improvement collaboratives).

Clinician and System Costs The impacts of implementing an effective practice on the delivery sys- tem should be assessed against the impact on clinicians and strategic measures of health system performance.

Commonly, health system leaders are interested in understanding how key clinical practices support whole system measures to assess overall performance of the health care system in achieving strategic goals, such as the Quadruple Aim: improving population health, enhancing the individual consumer experience of care, decreasing per capita cost of care, and improving the work life of health care clinicians, including clinicians and staff.269,270

Evaluators should also consider the costs and benefits of sustaining an effective practice against indicators of workforce resilience. For more intensive clinical practices, it is important to consider the impact

a practice has on clinician performance through key indicators like clinician satisfaction, well-being, and job burden. System administra- tive data and workforce surveys can provide evidence for the benefit or burden associated with the delivery of the effective practice over time compared to clinicians delivering comparable practices. Other indicators might include:

• Job-career satisfaction • Clinician satisfaction • Burnout • Control over work • Clinician turnover • Likelihood of leaving current

practice within X years

• Patient panel overcapacity • Working extended hours • Work-life balance

SUSTAINMENT Ongoing Evaluation and Reflection

Ongoing Evaluation and Ref lection

The VA QUERI Roadmap | PAGE 68

Sustainment Check List

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

Sustain an Intervention Develop business plan to continue EP Consider sources of funding, local fit, value to local organization, etc.

Monitor for changes in EP, whether different EP is needed Keep abreast of needed changes, adaptations, or consider de-implementation/de-intensification

Weigh costs of maintaining EP Analyze the costs, benefits, and return on investment of supporting EP

Transition Ownership to Stakeholders Provide management support Provide implementation playbook to clarify how to support the EP

Plan and budget for resources Plan and budget for local and shared operating resources (budget impact analysis, break-even analysis)

Support continuous learning and innovation in local stakeholder teams Plan for ongoing analysis, sharing between sites, and staff development

Ongoing Evaluation and Reflection Consumer outcomes Ensure EP has a population effect on a priority group(s) of patients

Delivery of EP (fidelity vs. adaptation) Maintain ongoing monitoring of fidelity and testing of adaptations to improve fit

Provider and system costs Local teams should iteratively refine implementation to optimize local fit and uptake

The VA QUERI Implementation Roadmap

Case Studies

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

Pre-Implementation Identify a Problem and Solution The hepatitis C virus is a leading cause of liver cancer and liver failure among United States adults and Veterans. In 2014, new, direct-acting antiviral medications with fewer side effects, shorter courses of treat- ment, and higher cure rates became widely available.

Earlier treatments included injectable interferon and were less than optimal for consumers and clinicians. These treatments were delivered over many months and frequently had adverse side effects, contrain- dications, and low cure rates. As a result, few Veterans had received hepatitis C virus treatment prior to 2014.218

As the new medications became available in 2014, the Veterans Health Administration cared for over 160,000 Veterans who were eligible for

the medications. Recognizing the benefit of these innovative treat- ments on consumer health and quality of life, VA clinical leadership decided to redesign the structure of hepatitis C virus care to make these treatments easily accessible for Veterans.

To support the expansion of treatment, the HIV, Hepatitis, and Related Conditions Program created a national quality improvement col- laborative called the Hepatitis C Innovation Team Collaborative. The collaborative’s goal was to improve the hepatitis C virus system of care. Health system leaders saw this rapid, national effort as an opportunity to study implementation strategies and how they affect outcomes over time. This work informs future similar implementation efforts.218

To support the study, implementation scientists at VA partnered with the collaborative to answer these questions:

1. Which strategies helped make sites successful in providing hepatitis C virus treatment and testing?

2. Was the collaborative an important part of hepatitis C virus treatment success?

Hepatitis C Innovation Team Collaborative Structure National Health System Leadership

HIT Collaborative Leadership Team

Veterans Integrated Service Network (VISN) HCV Innovation Teams (HITs) Clinician Leaders

Hospital Stakeholders

VA O�ce of Specialty Care Services

National Hepatitis C Resources Center (NHCRC) Veterans Engineering Resource Center (VERC) (system redesign and QI expertise)

HIV, Hepatitis, and Related Conditions Programs (HHRC)

Embedded Research

Evaluation Team

VA O�ce of Strategic Integration

The National Hepatitis C Innovation Team Collaborative

The VA QUERI Roadmap | PAGE 71

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

Engage Stakeholders The new antiviral treatments gave clinicians the ability to provide curative hepatitis C virus treatments. Treating the hepatitis C virus had strong political leadership and budgetary support to rapidly make treatments available to qualified Veterans. Given the cost of treatment, between fiscal year 2015 and 2020, Congress appropriated over three billion dollars to VA to purchase hepatitis C virus medication.

The collaborative approached leaders from each of the health system’s 21 regional networks and from Viral Hepatitis Lead Clinicians to gain support. These stakeholders were asked to form a regional team that represented multiple facilities within a network (e.g., medical centers and outpatient clinics). Each team developed and proposed plans to clinical operations offices for review on how they would improve hepatitis C virus care in their network facilities.

Scaling up hepatitis C virus treatment required local care teams to rap- idly change how they treated the virus. Teams also had to expand the reach of treatment to Veterans who were found previously ineligible as well as those previously undiagnosed and connect them with care.

To facilitate implementation, the HIV, Hepatitis, and Related Conditions

Program supported regional teams. The program provided a learning collaborative and training using Lean quality improvement principles to make changes that resulted in clinical improvements and system redesign. This allowed local clinicians more time to select implemen- tation strategies for their facility. It also provided the opportunity for evaluators to compare implementation strategies used by sites with varying implementation-effectiveness outcomes over time.

Originally developed for manufacturing settings, Lean seeks to introduce new ways of operating that empower frontline staff to make continuous improvements to their daily work that focus on improv- ing value by reducing sources of waste and inefficiency causes by complicated daily processes and workarounds. Lean engages frontline employees in an interactive problem-solving to examine areas for improvement, propose solutions, and experiment using rapid PDSA cycles and monitoring of results. To achieve improvements in qual- ity and performance, Lean emphasizes monitoring measures that represent a range of outcomes including clinical outcomes, financial,

consumer and employee aspects of health care delivery.

Develop Measures and Data The HIV, Hepatitis, and Related Conditions Program developed and supported Hepatitis C Innovation Team leadership to deploy national hepatitis C virus measures, including birth cohort testing, treatment, and cure. Team evaluators also developed an annual web-based sur- vey to assess the use of the 73 implementation strategies identified by the Expert Recommendations for Implementing Change (ERIC) Project.126,127 They did so by sampling key facility hepatitis C virus

clinical leads. Survey responses were used to examine associations between implementation activities and hepatitis C virus performance metrics that could be monitored over time. The team did this to compare differences across sites, especially with high versus low-per- forming facilities (i.e., performance based on numbers of new hepatitis C virus treatment starts).

Implementation Implement an Intervention Nationally, Hepatitis C Innovation Team leadership provided a structure for communicating and networking to help overcome bar- riers and spread best practices through the collaborative. Hepatitis

C Innovation Team leadership held annual in-person meetings and weekly calls. They also trained members in Lean and system redesign methods, conducted onsite support, and delivered real-time feedback.

Activate Implementation Teams Regionally, teams received annual funding to support a regional coordinator position. They also received other staff support, clinical pharmacists, supplies, and travel to help implement the intervention.

After building team capacity to conduct hepatitis C virus quality im- provement, teams determined the implementation strategies needed to improve hepatitis C virus care in their local health system.

Monitor Implementation Progress The Hepatitis C Innovation Team Collaborative helped create a nation- al dashboard that acted as a population health management tool. The dashboard helped identify consumers who were eligible for hepatitis C virus testing and treatment. Evaluators and Hepatitis C Innovation Team leaders tracked the progress across hepatitis C virus measures. They focused on ensuring that all Veterans received access to the treat- ment. While only 10% of Veterans with hepatitis C virus were cured by the end of fiscal year 2014, by the end of fiscal year 2016 (Year 2), 43% (84,192 Veterans) were cured.271

The evaluation team then used methods to identify strategies related to higher treatment across implementation years. Strategies related to treatment shifted between Years 1 (Pre-implementation and early Implementation) and 2 (Implementation). Pre-implementation strate- gies in the “training/educating,” “interactive assistance,” and “build- ing stakeholder interrelationships” clusters were more likely to be significantly associated with treatment starts in Year 1. Strategies in the “evaluative and iterative” and “adapting and tailoring” clusters were more likely to be related to treatment in Year 2. Table 21 shows the shift and implementation strategies across nine thematic clusters.

The VA QUERI Roadmap | PAGE 72

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

Table 21. Strategies Related to Hepatitis C Virus Treatment Starts in Years 1 and 2 Mapped onto Strategy Clusters205

Strategy cluster Discrete implementation strategy Year 1

use Year 2

use Both years

Develop stakeholder interrelationships

Inform local opinion leaders of advances X Share knowledge gained from quality improvement efforts with other sites X Identify and prepare clinical champions X Build networks and relationships for information sharing and problem-solving X Partner with a university to share ideas X Visit other sites outside your facility to learn from their experience X Organize support teams of clinicians caring for hepatitis C virus to share lessons learned and support others’ learning

X

Involve executive boards X Build a local team to address challenges X Conduct local consensus discussions X Recruit, designate, and/or train leaders X Use modeling or simulated change X Make efforts to identify early adopters and learn from their experiences X

Train and educate stakeholders

Conduct educational trainings X Have expert in hepatitis C virus care meet with clinicians for educational trainings X Provide ongoing hepatitis C virus training X Vary information delivery methods to cater to different learning styles X Distribute educational materials X Facilitate forming clinician groups and a collaborative learning environment X Use educational institutions to train clinicians X

Engage consumers Promote demand for HCV care among patients through any other means X Engage in efforts to prepare consumers to be active participants in hepatitis C virus care X

Clinician interactive assistance

Offer external technical assistance to support clinics X Use a centralized system to deliver facilitation X Provide clinical supervision X

Change infrastructure

Create or change credentialing and/or licensing standards X Participate in liability reform efforts to make clinicians more willing to support innovation X Change the records system X Change the physical structure and equipment X Change the location of clinical service sites X

Use financial strategies Alter incentive/allowance structures X

Support clinicians

Develop resource sharing agreements X Facilitate the relay of clinical data to clinicians X Create new clinical teams X Revise professional roles X

Adapt and tailor to the context

Use data experts to manage hepatitis C virus data X Use data warehousing techniques X Tailor strategies to deliver HCV care X Promote adaptability X

Use evaluative and iterative strategies

Assess for readiness and identify barriers/facilitators to change X Develop a formal implementation blueprint X Conduct small tests of change, measure outcomes, and then refine these tests X Develop and organize systems to monitor clinical processes or outcomes for purpose of quality assurance/improvement

X

Intentionally examine the efforts to promote hepatitis C virus care X Develop strategies to obtain and use consumer and family feedback X Collect and summarize clinical performance data to feedback to decision makers to iteratively pilot changes before making system-wide changes

X

The VA QUERI Roadmap | PAGE 73

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

Sustainment Sustain an Intervention The Hepatitis C Innovation Team Collaborative structure had great success in treating over 85% of Veterans with the hepatitis C virus. As a result, the team decided to focus on Veterans with advanced liver diseases in 2019. The team changed its name to the “Hepatic Innova- tion Team Collaborative” in recognition of the expanded scope.

To improve the sustainability of the redesign efforts that led to the VA’s success in hepatitis C virus testing and treatment, the collaborative’s leadership team changed the team composition from system redesign experts to liver disease specialists from regional networks across the country.

Transition Ownership to Stakeholders The collaborative provided infrastructure through regional funding, cross-regional networking, technical assistance, and data feedback. Regional coordinators and their teams developed new relationships with stakeholders. They also obtained approval for applying redesign and Lean management to advanced liver disease.

To make the transition successful, the leadership team provided infra- structure support during the transition to the regional networks. This allowed new leaders to receive train-the-trainer support and educa- tion in process improvement and data management. New leaders were also provided coaching and mentorship.

Ongoing Evaluation and Reflection The VA has led the nation in hepatitis C virus elimination efforts. The evaluation of their national efforts led to the creation of a method to measure implementation strategies over time. These strategies link to the stages of Implementation.

The evaluation team continues to review the factors associated with

equal access to these medications. As the collaborative changes their focus to clinical care, the evaluation team continues to learn more about how to help later-adopting sites improve their care of Veterans with liver disease.

Suggested Readings 1. Belperio PS, Backus LI, Ross D, Neuhauser MM, Mole LA. A popula- tion approach to disease management: hepatitis C direct-acting antiviral use in a large health care system. J Manag Care Spec Pharm. 2014;20(6):533-540.

2. Belperio PS, Chartier M, Ross DB, Alaigh P, Shulkin D. Curing Hepatitis C Virus Infection: Best Practices From the U.S. Department of Veterans Affairs. Ann Intern Med. 2017;167(7):499-504.

3. Rogal SS, Yakovchenko V, Waltz TJ, et al. The association between implementation strategy use and the uptake of hepatitis C treatment in a national sample. Implement Sci. 2017;12(1):60.

4. Belperio PS, Chartier M, Gonzalez RI, et al. Hepatitis C Care in the De- partment of Veterans Affairs: Building a Foundation for Success. Infect Dis Clin North Am. 2018;32(2):281-292.

5. Belperio PS, Chartier M, Ross DB. After Curing Hepatitis C Virus Infec- tion. Ann Intern Med. 2018;168(9):682-683.

6. Rogal SS, Yakovchenko V, Waltz TJ, et al. Longitudinal assessment of the association between implementation strategy use and the uptake of hepatitis C treatment: Year 2. Implement Sci. 2019;14(1):36.

7. Yakovchenko V, Miech EJ, Chinman MJ, et al. Strategy Configura- tions Directly Linked to Higher Hepatitis C Virus Treatment Starts: An Applied Use of Configurational Comparative Methods. Med Care. 2020;58(5):e31-e38

The VA QUERI Roadmap | PAGE 74

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

We applied the QUERI Roadmap to our work to modify the VA’s medical home model (i.e., Patient Aligned Care Team) to meet the needs of women Veterans. Women Veterans are an extreme minority in the VA (approximately 8%). However, they have complex physical and mental health needs that must be treated and are the fastest growing segment of new VA users.

Pre-Implementation Identify the Problem and Solution Women Veterans face gender disparities in primary care access, continuity, and coordination. They also face disparities when it comes to preventive screenings that routinely occur in primary care settings (e.g., colorectal cancer screening, depression screening).

The Patient Aligned Care Team was not originally changed for women Veterans’ primary care needs. Therefore, multiple policies guide the implementation of Women’s Health Patient-Aligned Care Team models. These models must be capable of meeting primary care and gender-specific care needs by expert clinicians in gender-sensitive care environments.

In partnership with VA Women’s Health Services, our goal was to test an evidence-based quality improvement approach to tailoring the Patient Aligned Care Team (as the intervention) to meet women Veterans’ care needs. This intervention is team-based and focuses on access to and continuity with identifiable teamlets. These teamlets have assigned Veteran populations for whom they are accountable for preventive care, primary care coordination, and Veteran outcomes.

Engage Stakeholders We engaged stakeholders in the beginning using several strategies. First, we reached out to medical centers that were part of the VA Wom- en’s Health Practice Based Research Network. Each medical center has a trained site lead with knowledge about site characteristics (e.g., care arrangements). Leads helped provide insights on local organizational strengths and challenges. They also helped us identify stakeholders we should further engage.

We then engaged regional network-level leaders to orient them to our plans. We also had to understand their structures for oversight and decision-making and secure their agreement to participate.

Finally, we convened multilevel (i.e., network, medical center, clinic levels) meetings within each of four participating regional networks. In the meetings, we described the problem and evidence-based quality improvement solution. We described the problem in the context of Patient Aligned Care Team core elements and data on local barriers/ gaps (e.g., network-specific gender disparities). Then, we used expert panel methods to agree on a quality improvement roadmap for each regional network adapted to local contexts. Communication strategies for ongoing regional network and medical center engagement were integral to the quality improvement roadmaps.

Develop Measures and Data The evaluation was designed as a cluster-randomized trial with mea- sures of access, clinician/staff gender sensitivity, and gender-specific care delivery. We also measured potential impacts on health care utili- zation when implementing the Patient Aligned Care Team model (e.g., primary care visits, hospitalizations, emergency room visits).

Finally, we determined baseline performance. We measured it using consumer and clinician/staff surveys, key stakeholder and teamlet in- terviews, and administrative data. These data sets were collected after the stakeholder panel meetings.

Tailoring Implementation of a Medical Home Model to Meet Women Veterans’ Needs Using Evidence-based Quality Improvement

The VA QUERI Roadmap | PAGE 75

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

Implementation Implement the Intervention Evidence-based quality improvement is a top-down/bottom-up research-clinical partnership approach to implementation. It includes:

• Leadership buy-in (achieved in pre-implementation) • Quality improvement training for local teams

(e.g., process flow maps) • Formative data feedback • Technical support (e.g., measures development for local Patient

Aligned Care Team quality improvement projects) • External practice facilitation

Local quality improvement teams selected the target related to the Patient Aligned Care Team among many options prioritized in the regional network quality improvement roadmaps. This enabled them

to identify what mattered most locally. The research team helped the local quality improvement teams develop project proposals using a standardized template. The template required:

• Specific problem • A solution • Activities that needed to be accomplished • Other stakeholders who needed to be involved • Measures that needed to be collected • Time/effort needed by local participants to do the project.

The evidence-based quality improvement needed to be adapted to meet local teams’ competing demands (i.e., fewer facilitation calls).

Activate Implementation Teams All teams were activated for an in-person, evidence-based quality improvement training session spanning one-and-a half-days. During the session, they received group training on evidence-based quality improvement principles and heard testimony from previous users. The teams listened to leaders from the VA Central Office endorse the work ahead. Finally, they worked in small groups with investigators to develop and get feedback on their project proposals.

Top-down support from local leaders was not necessary at this stage

because leaders provided support during stakeholder panel meet- ings. Their priorities were reflected in the network-specific quality improvement roadmaps. Feedback channels were also laid out during the pre-implementation stakeholder panel meetings, but specific strategies were added and evolved over the course of implementation. This included a range of briefings to leaders at the medical center and regional network levels that did not participate in the panel meetings.

Monitor Implementation Progress In addition to pre-implementation communication plans, each VA regional network had a different structure or council for reporting progress (e.g., network quality council). One network created a steer- ing committee specific to this implementation effort. We generated formative feedback reports for the local quality improvement teams and provided them with slide decks and templates for their own re- porting at medical center and network levels, giving them both voice and visibility in their respective organizations. Most had never had these reporting opportunities.

We also provided reviews of their own progress reports, leading in a few cases to academic products of their own. Monitoring of imple- mentation was chiefly accomplished through facilitation calls, where we generated brief summaries of progress and action items for the quality improvement and research teams, respectively, and helped teams navigate apparent roadblocks. We also used rapid summaries of key stakeholder interviews at the clinic, medical center, and regional network levels, which were fed back to the teams for applied analysis.

The VA QUERI Roadmap | PAGE 76

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

Sustainment Sustain an Intervention As the VA’s medical home model, Patient Aligned Care Team was already mandated nationally. However, ongoing improvements were underway as part of the Demonstration Laboratory initiative, where evaluation feedback was provided to national VA primary care leaders for their consideration in ongoing adaptations.

To sustain attention to the value of gender-tailoring Patient Aligned Care Team moving forward, we convened another round of stakehold- er panel meetings 24-months post-implementation in each regional network.

These meetings highlighted team accomplishments and ongoing challenges. They also provided the opportunity to decide on sustain- ability plans at the network and medical center levels. The meetings presented opportunities for regional network leaders to identify and select evidence-based improvements that could spread network-wide.

Aspects of both evidence-based quality improvement and Women’s Health Patient Aligned Care Team advances were integrated into the VA Women’s Health Service’s national Women’s Health Patient Aligned Care Team Roadmap. The roadmap was circulated nationally.

Transition Ownership to Stakeholders One of the key beliefs underlying evidence-based quality improve- ment is the partnership model. This model infuses every stage of implementation. It empowered local quality improvement teams to solve problems that got in the way as they worked. We continue to hear about their ongoing improvements and new quality improve- ment projects four years after the study ended.

We focused on transitioning evidence-based quality improvement functions to key leaders. For example, we worked with network-level Women Veteran Program Managers to take on facilitation roles with women’s health leaders under their purview. We also developed and delivered national evidence-based quality improvement training for all women’s health leaders.

Ongoing Evaluation and Reflection There were many impacts of evidence-based quality improvement on Women’s Health Patient Aligned Care Team implementation. Local quality improvement projects achieved better follow-up of abnormal cervical cancer screening, linking Veterans to designated women’s health clinicians. This led to the Women’s Health Services’ adoption of evidence-based quality improvement as an approach to further com- prehend women’s health care delivery. However, this time it occurred in low-performing VA facilities.

We developed technical specifications for contract vendors to use when deploying evidence-based quality improvement. Currently, we are evaluating the fidelity and effectiveness of deployment in these new organizational contexts.

We have also launched an Evidence-Based Quality Improvement Col- laborative in the VA Women’s Health Practice Based Research Network, which tests our ability to spread innovations in increasingly efficient and effective ways.

More work is necessary to determine the time needed between evi- dence-based quality improvement Patient Aligned Care Team changes in this context. Additional research is also needed to determine clini- cian gender sensitivity and burnout, changes in consumer experience (e.g., how much time does it take for women Veterans to see a differ- ence in their care), as well as clinician, staff, and system costs.

Suggested Readings: Those interested in training on evidence-based quality improvement can visit the Evidence-Based Quality Improvement Implementation Strat- egy Learning Network Hub and contact [email protected].

1. Brunner, J., Cain, C.L., Yano, E.M., Hamilton, A.B. (2019).  HYPERLINK “https://www.ncbi.nlm.nih.gov/pubmed/30455089” Local leaders’ perspectives on women Veterans’ health care: what would ideal look like? Women’s Health Issues. 2019;29:64-71.

2. Chuang, E., Brunner, J., Mak, S., Hamilton, A.B., Canelo, I., Darling, J., Rubenstein, L.V. Yano, E.M. Challenges with implementing a patient-centered medical home model for women Veterans. Women’s Health Issues. 2017;27(2):214-220.

3. Hamilton, A.B., Brunner, J., Cain, C., Chuang, E., Luger, T.M., Canelo, I., Rubenstein, L., Yano, E.M. Engaging multilevel stakeholders in an implementation trial of evidence-based quality improvement in VA women’s health primary care. Transl Behav Med. 2017;7(3):478-85.

4. Hamilton, A., Olmos-Ochoa, T.T., Canelo, I., Rose, D., Hoggatt, K.J., Than, C., Yano, E.M. (In press). Dynamic waitlisted design for evaluating a randomized trial of evidence-based quality improvement of comprehensive women’s health care implementation in low performing VA facilities. Implement Sci Commun.

5. Klap, R., Darling, J., Hamilton, A., Rose, D., Dyer, K., Canelo, I., Haskell, S., Yano, E. (2019). Prevalence of stranger harassment of women Veterans at Veterans Affairs Medical Centers and impacts on delayed and missed care. Women’s Health Issues. 2019;29:107-115.

6. Meredith, L. S., Wang, Y., Okunogbe, A., Bergman, A. A., Canelo, I. A., Darling, J. E., & Yano, E. M. (2017). Attitudes, practices, and experiences with implementing a patient-centered medical home for women veterans. Women’s Health Issues. 2017;27(2):221-227.

7. Yano, E.M., Darling, J., Hamilton, A., Canelo, I., Chuang, E., Meredith, L., Rubenstein, L. Cluster randomized trial of a multilevel evidence-based quality improvement approach to tailoring VA Patient Aligned Care Teams to the needs of women veterans. Implement Sci. 2016;11(1):101.

The VA QUERI Roadmap | PAGE 77

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

The Veterans Health Administration has worked quickly to prevent opioid overdose deaths among Veterans at VA facilities. The Rapid Naloxone Initiative is one part of a multi-faceted national strategy to address the opiate epidemic. Notably, this case study illustrates how a promising care innovation developed by a single health care facility can be spread through the system to improve outcomes using the QUERI Roadmap.129

Pre-Implementation Identify a problem and solution The United States is in the midst of an opioid epidemic. In 2017, opioid overdoses took the lives of nearly 47,600 people—an average of 130 fatalities per day. Military Veterans are a vulnerable population and are twice as likely to die from an accidental overdose compared to non-Veterans.

Naloxone is a safe and effective medication that can reverse an opioid overdose when delivered as an intranasal spray or as an intramuscu- lar shot. In 2014, the VA became the first large United States health system to rapidly implement a nationwide Opioid Overdoes Education and Naloxone Distribution (OEND) program.

The goals of Opioid OEND are to educate at-risk Veterans, their friends, their family, and their clinicians to improve access to naloxone and reduce and prevent Veteran deaths. In the context of this urgent national initiative, there is ample opportunity for local innovation and practice-based knowledge sharing across the health system.

Although the administration of intranasal naloxone effectively re- verses opioid overdoses, clinicians and safety experts at the Boston

VA Healthcare System realized that it was often unavailable at the time of an overdose, even when on facility grounds. The accessibility of naloxone is critical because the onset of brain damage can begin after just five minutes without oxygen.

The Boston VA Healthcare System iteratively worked to solve this problem by developing the Opioid Overdose Reversal Program, now named the Rapid Naloxone Initiative. The initiative consists of three core elements:

1. Training at-risk consumers and their families to administer intranasal naloxone

2. Equipping Department of Veterans Affairs Police or other first responders at a facility with intranasal naloxone and training them to use it

3. Stocking automated external defibrillator cabinets with intranasal naloxone to provide quick access

Engage stakeholders The Rapid Naloxone Initiative benefitted from the national Opioid OEND initiative, which started in 2014. The Opioid OEND initiative gained support at every level of the organization. At the Boston VA Healthcare System, clinical champions garnered support for and over- saw OEND implementation with pharmacy staff and clinicians from mental health, pain management, primary care, and substance-use disorder treatment at each local health facility.

A national workgroup of operational program offices also worked collaboratively toward initiative goals. They developed common tools and resources to educate stakeholders and develop implementation plans, policies, and a data collection and evaluation plan.

The Rapid Naloxone Initiative began when a Veteran experienced an opiate overdose at a Boston VA Healthcare System’s facility in 2015.

A local consumer safety manager realized the need to expand OEND programming. She started an improvement project to assess the feasibility of equipping VA Police officers with naloxone. However, pilot testing suggested this strategy alone would be insufficient—they would also need to enable first responders to respond quickly.

In early 2016, the team began placing naloxone in unlocked cabinets containing automated external defibrillators on the walls of VA cafete- rias, gyms, warehouses, clinic waiting rooms, and rehabilitation hous- ing. This allowed trained first responders to rapidly access intranasal naloxone in critical situations. The following implementation barriers were identified during early rapid-cycle testing.

The Department of Veterans Affairs Rapid Naloxone Initiative: National Diffusion of a Promising Practice

The VA QUERI Roadmap | PAGE 78

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

Barriers to Increasing Rapid Access to Naloxone

Gaining support of local unions to support initiative

Making a prescribed medication (albeit safe) readily available on VA campuses

Obtaining permission to develop guidelines for storing naloxone in automated external defibrillator cabinets

Developing standard methods to track and report implementation/ clinical outcomes

After successfully piloting the practice at the Boston VA Healthcare System, the initiative was spread across the regional network of

medical centers, outpatient clinics, and community outpatient clinics located in six New England states (Maine, New Hampshire, Vermont, Massachusetts, Rhode Island and Connecticut).

Later in 2016, the initiative was entered into the Shark Tank competi- tion hosted by the VA Diffusion of Excellence. It competed against hundreds of other promising employee-initiated practices to receive operational funding support from medical center leaders.

The initiative was ultimately selected as a “Gold Status” practice and received funding. The funding allowed the team to implement the practice in the regional network representing VA facilities in Florida and Puerto Rico between December 2016 and May 2017.

Develop measures and data The VA Boston Healthcare System partnered with the National Opioid Overdose Education Distribution Coordinator to obtain baseline data. The partners then developed measures for success that would be

relevant and meaningful once the practice spread, as displayed in Table 22.

Table 22. Measures for Assessing Impact of the Rapid Naloxone Initiative

Outcome Source

Opioid overdose and naloxone education

Number of Veterans prescribed naloxone Standardized medical template in electronic health record

Reported opioid overdose reversals (per year) Self-reported by facility

Department of Veterans Affairs Police naloxone

Number of facilities with Department of Veterans Affairs Police carrying naloxone

Self-reported by facility

Department of Veterans Affairs Police opioid overdose reversals per year

Self-reported by facility

Automated external defi- brillator naloxone

Number of automated external defibrillator cabinets equipped with naloxone per facility

Self-reported by facility

Number of facilities with automated external defibrillator cabinets equipped with naloxone

Self-reported by facility

An automated external defibrillator opioid overdose reversals per year

Self-reported by facility

The VA QUERI Roadmap | PAGE 79

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

Implementation Implement an Intervention In April of 2018, the VA Under Secretary for Health selected the Rapid Naloxone Initiative for national diffusion. In July of 2018, an interdis- ciplinary workgroup with stakeholders from the following programs and agencies met face to face to develop a formal national diffusion strategy:

• Veterans Health Administration Diffusion of Excellence • Pharmacy Benefits Management (including the pharmacy

Academic Detailing Service) • Department of Veterans Affairs:

- Police - National Center for Patient Safety - National Center on Homelessness Among Veterans - Office of Nursing Services - Office of Mental Health and Suicide Prevention

To implement the plan, the Rapid Naloxone Initiative team partnered with the Diffusion of Excellence initiative and the Opioid OEND national coordinator to manage the spread throughout the VA. The team developed two implementation toolkits. One was created for the automated external defibrillator cabinet naloxone program, and another was created for equipping VA Police with intranasal naloxone. The team selected a number of implementation strategies to address barriers identified through pilot and regional spread efforts (Table 23).

Table 23. Implementation Strategies to Facilitate Rapid Naloxone Initiative Uptake

Implementation barrier Implementation strategy

Opioid OEND clinician training and education

Standardized educational resources, academic detailing

Naloxone training for at-risk Veterans, caregivers

Consumer brochures, materials, education delivered by interdisciplinary VA clinicians

Identification of high-risk consum- ers eligible for naloxone

Clinician decision-support tools

Implementation guidance Standardized Opioid OEND implementation plan, clinical guidance, resources via centralized web-based portal, technical assistance by Diffusion of Excellence and Opioid OEND national program staff

Co-pay cost for naloxone Waiver of co-pay fee via policy change to consumers and costs to local facilities

Leadership support National workgroup of operational office, policy mandates, policy changes in formulary

Local union support Memorandum of understanding signed with union groups to allow VA Police to carry naloxone

Automated external defibrillator cabinet regulatory approval

Guidance from the Joint Commission on standards for labeling, securing, monitoring, and maintaining naloxone in automated external defibrillator cabinets

Marketing to Department of Veter- ans Affairs and non-Department of Veterans Affairs stakeholders

Multi-channel marketing campaign using print media, social media (e.g., Facebook, Ted Talks), presenta- tion on national VA clinical and leadership calls, academic detailers, community of practice, centralized websites

Local site barriers External facilitation by Opioid OEND national coordinator and initiative implementation team; sharing problems and successes on a monthly community of practice call

Data collection and outcome monitoring

Creation of standardized clinician templates for electronic medical record, national dashboard on program metrics

Activate Implementation Teams The Rapid Naloxone Initiative was initiated by a top-down memoran- dum sent by VA leadership. The memorandum defined a goal for all facilities to implement the program by December 2018. When this directive was released in September 2018, each VA health system was required to identify a clinical champion and back-up champion for the program.

The role of champions was to coordinate and monitor the implemen- tation of the practice at the local level to improve fit. In addition, each facility was required to submit a registration form, which included the implementation status at their facility and contact information of the champions. Then, a member of the Diffusion of Excellence team was assigned to contact each champion to provide technical assistance and address implementation barriers.

The VA QUERI Roadmap | PAGE 80

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

Monitor Implementation Progress Virtual teleconference calls with regional network stakeholders helped identify common implementation barriers. Tools and resources to overcome barriers were often provided the same day. While barriers related to VA Police naloxone tended to be easily addressed (e.g., gaining police union support), barriers related to automated external defibrillator cabinet naloxone were more difficult. This was because many concerns related to the Joint Commission and department guidance were created specifically for the Rapid Naloxone Initiative.

The national implementation team met with the Joint Commission to address common questions/concerns and provide feedback to the field, including updated guidance that made it easier implement the practice. For example, the Canandaigua VA Medical Center presented their model of automated external defibrillator cabinet naloxone that garnered a best practice designation from the Joint Commission. This model was shared widely with facilities struggling to implement the practice.

Another major barrier to implementation was not having a nationally standardized brief training protocol for staff. In February of 2019, a 10-minute staff training on the VA Rapid Naloxone Initiative was made available through the VA’s online Talent Management System. The system allowed facilities to assign and track staff training. The VA worked with the pharmaceutical company that created the nasal naloxone spray (Nar- can®) to adapt their video for short training.

Working on this initiative identified the need to streamline, synergize, and improve overdose reporting and response. VA Police now systemati- cally document their activities in the Police System/Report-Exec System, Patient Safety Managers report their activities in the Joint Patient Safety Reporting System, and clinical staff report overdoses in the electronic health record system and adverse drug events in the Drug Event Reporting System.

Sustainment Sustain an intervention The initiative continues with ongoing central support from the VA Office of Mental Health and Suicide Prevention. The initiative also benefits from ongoing national and local leadership support from the VA Office of Security and Law Enforcement and National Center for Patient Safety.

At the local level, nasal naloxone is currently free to both facilities and at-risk Veterans, so ongoing cost is not a concern. A directive is also being developed that should help support sustainment.

In May of 2019, the workgroup had another meeting to discuss future plans and the transition of ownership to the Office of Mental Health and Suicide Prevention.

Ongoing evaluation and reflection As of November 2019, 82% of VA medical centers have equipped a total of 2,785 of police officers with naloxone and 56 facilities have deployed naloxone in 693 automated external defibrillator cabinets. As a result, this simple practice has resulted in 126 opioid overdose reversals (120 by VA Police and six by automated external defibrillator cabinet naloxone).

As of December 31, 2019, naloxone has been dispensed to over 217,469 unique Veterans with 32.3% being high-risk consumers with overlap- ping opioid and benzodiazepine prescriptions dispensed naloxone as well as 38.6% of Veterans with ≥50 mg Morphine Equivalent Daily Dose (MEDD) dispensed naloxone. These efforts have resulted in 990 opioid overdose reversals reported since initiation of the program.

Ensuring that VA consumers get the care they need post-overdose is critical and often lifesaving given the high rates of mortality following non- fatal overdoses (e.g., one study found that one in 10 individuals with a non-fatal overdose die within two years).

For additional information on the on-going efforts of the Rapid Naloxone Initiative, please visit VHA Academic Detailing Services’ Opioid Over- dose Education and Naloxone Distribution (OEND) and https://www.ci2i.research.va.gov/CI2IRESEARCH/resources.asp

Suggested Readings 1. Oliva EM, Christopher MLD, Wells D, et al. Opioid overdose education and naloxone distribution: Development of the Veterans Health

Administration’s national program. J Am Pharm Assoc. (2003). 2017;57(2S):S168-S179 e164.

2. Bounthavong M, Harvey MA, Wells DL, et al. Trends in naloxone prescriptions prescribed after implementation of a National Academic Detailing Service in the Veterans Health Administration: A preliminary analysis. J Am Pharm Assoc (2003). 2017;57(2S).

3. Veterans Health Administration Innovation Ecosystem: https://www.va.gov/INNOVATIONECOSYSTEM/views/who-we-are/diffusion-of- excellence.html

The VA QUERI Roadmap | PAGE 81

VA Resources for Implementation & Quality Improvement

VA Resource Centers Fact Sheets:

• Center for Evaluation & Implementation Resources (CEIR)

• Evidence Synthesis Program (ESP)

• Partnered Evidence-based Policy Resource Center (PEPReC)

• HSR&D VA Information Resource Center (VIReC)

• HSR&D Health Economics Resource Center (HERC)

QUERI Implementation Strategy Learning Hubs

VA Quality Scholars

VA RESOURCE CENTERS

www.queri.research.va.gov U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

EVIDENCE SYNTHESIS PROGRAM Identifies best practices & conducts rapid evidence reviews

CENTER FOR EVALUATION & IMPLEMENTATION RESOURCES Time sensitive consultation support to Operational Leaders on evaluation & implementation

PARTNERED EVIDENCE-BASED POLICY RESOURCE CENTER Supports development & evaluation of national policies/program for VA Leaders using rigorous designs

PEPReC

HSR&D VA INFORMATION RESOURCE CENTER Provides guidance on data, information systems & analysis

HSR&D HEALTH ECONOMICS RESOURCE CENTER Estimates return on investment & implementation strategy costs

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

The VA QUERI Roadmap | PAGE 84

CEIR Center for Evaluation and Implementation Resources

VA’s resource center for implementation and improvement science The Center for Evaluation and Implementation Resources (CEIR) is a QUERI resource center launched in 2017 to support the more rapid implementation of effective practices into routine VA clinical practice. CEIR provides time-sensitive consultation and support to VA operational leaders on evaluation and implementation methods to enable scale-up and spread of policies, effective practices, and technologies aligned with key VA priorities.

CEIR Services and Benefits To achieve its mission, CEIR strives to be VA’s centralized resource for implementation science, quality improvement and program evaluation methods that can enable policymakers, managers and researchers to:

CEIR Overview

EVALUATION IMPLEMENTATION DISSEMINATION Use of program evaluation

best practices in everyday VA care settings

Application of implementation science best practices to

accelerate clinical innovation

Dissemination of implementation science and program evaluation information,

tools, and training opportunities

STRATEGIES Select optimal implementation strategies to deploy effective practices across multiple sites and settings

PLANS Develop implementation plans that integrate new practices for sustainability while de-implementing low-value practices

METRICS Choose meaningful metrics to evaluate implementation efforts

BEST PRACTICES Share best practices in evaluation and implementation science

CONSULTATIONS Receive brief consultations to enhance program evaluation and implementation efforts

LINKAGES Create linkages between leaders, frontline clinicians, and VA researchers to address areas of VA care priority

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

Learning Collaboratives CEIR supports the activities of the Implementation Research Group (IRG). The IRG is an international learning collaborative of over 525 VA and non-VA implementation scientists.

The IRG provides a forum for sharing best practices and lessons learned in the field of implementation science. For more details, please email [email protected].

IRG activities include monthly cyber seminars, community of practice calls, and 6 specialty working groups related to implementation approaches and strategies:

Working with CEIR CEIR’s services are particularly relevant during both Pre- Implementation and Implementation phases to develop a rigorous implementation and evaluation plan to support the rapid uptake of effective practices across diverse clinical sites and settings.

The QUERI Roadmap provides a systematic, three-phased approach to help practitioners overcome barriers to implementing effective practice, Pre-Implementation, Implementation and Sustainment. For more details, visit the QUERI Roadmap.

Performance to D ata

Knowledge to Perform an

ce

D at

a to

K no

w le

dg e

�� �� �� �� ��

��� ����� ��������������

�����������

Adaptation, Fidelity, and Tailoring

Qualitative Comparative Analysis

Audit with Feedback

Qualitative Methods and Analysis Group

Applying Implementation Science

(theories and frameworks)

Implementation Facilitation

External Users: www.queri.research.va.gov/ceir Email: [email protected]

VA’s hub for implementation and improvement science CEIR

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

U.S. Department of Veterans A�airs

Veterans Health Administration Health Services Research & Development Service

The VA QUERI Roadmap | PAGE 86

ESP Evidence Synthesis Program

Synthesizing evidence for VA leadership to improve the health and healthcare of Veterans The VA Evidence Synthesis Program (ESP), an HSR&D Resource Center established in 2007, is helping VA fulfill its vision of functioning as a continuously “learning health system.” ESP achieves this by providing timely, targeted, and unbiased syntheses of the medical literature for translation into evidence-based clinical practice, policy, and research.

ESP Overview

RIGOROUS RELEVANT NIMBLE Rigor, transparency, and

minimization of bias underlie all our products

Emphasis on Veteran population ensures our reviews are relevant

to VA decision-makers’ needs

We adapt traditional methods, timelines, and formats to meet

our partners’ specific needs

ESP Services and Benefits The ESP helps health system leaders make the best possible use of current knowledge.

ESP products assist decision-makers in: Identifying appropriate policy or program options

• Characterize benefits and harms

• Identify key elements of complex interventions to facilitate local adaptation

Exploring effective practices for implementation • Characterize effects of broadly implemented complex

interventions

• Identify contributors to a problem to inform intervention development

QUALIFIED ESP Center Directors are VA clinicians and recognized leaders in the field of research synthesis with close ties to the AHRQ Evidence-based Practice Center Program and Cochrane Collaboration.

INNOVATIVE We go beyond “traditional” systematic review methods and integrate qualitative information from providers and patients and quantitative system data.

HIGH IMPACT ESP tackles high-priority issues such as opioid use, suicide prevention, and community care, and provides informed reports to Congress, American College of Physicians guidelines, and VA formularies.

TIMELY Requests are accepted year-round, with more urgent needs prioritized for our rapid products. Our product timeframes range from 1 week to 1 year.

The VA QUERI Roadmap | PAGE 87

ESP Evidence Products We have a broad range of products to address the various evidence needs of our operational partners. Systematic reviews are the gold standard in research, following the most rigorous methods and allowing for the most complete and defensible conclusions. ESP also offers other products that use more streamlined methods to assist with more time-sensitive needs.

Pre-Implementation ESP’s services are particularly relevant during the Pre-implementation phase to identify the best clinical practices to address a need for improvement, i.e., what works and what might work in this setting, but also whether VA’s research has produced results that benefit the health system and whether we can identify research areas where more or less focus could be useful.

Working with ESP Submit a topic nomination form. We will work with you to determine the appropriate research approach to address your questions of interest. If the topic meets our program criteria and is prioritized, then an ESP Center will be assigned to conduct an in-depth review. High- priority, time-sensitive needs will be undertaken at the Coordinating Center as capacity allows.

ESP’s services are particularly relevant during the Pre-implementation phase to identify the best clinical practices to address a need for improvement, i.e., what works and what might work in this setting, but also whether VA’s research has produced results that benefit the health system and whether we can identify research areas where more or less focus could be useful.

The QUERI Roadmap provides a systematic, three-phased approach to help practitioners overcome barriers to implementing effective practice, Pre-Implementation, Implementation and Sustainment. For more details, visit the QUERI Roadmap.

External Users: www.hsrd.research.va.gov/publications/esp/ Internal Users: vaww.hsrd.research.va.gov/publications/esp/reports.cfm Email: [email protected]

Performance to D ata

Knowledge to Perform an

ce

D at

a to

K no

w le

dg e

�� �� �� �� ��

��� ����� ��������������

�����������

Evidence Compendium Brief summary of key features, data abstraction, and bibliography, organized by key features.

Scoping Review Descriptive overview that identi�es gaps and overlap in key concepts and highlights speci�c and/or unique features of interest.

Standard Systematic Review Comprehensive synthesis using the most

methodologically rigorous process. Reviews several broad, overarching key questions.

Rapid Evidence Brief Detailed report that generally follows, but streamlines, accepted systematic review methods and PRISMA reporting guidelines.

Evidence Inventory Bibliography organized by key features (e.g., key question, study design, populations, etc.)

Evidence Map User-friendly visual �gure or graph and interpretive summary of a broad research �eld that provides quick access to questions and answers that previous research has addressed and identi�es gaps that are important for the VHA.

Evidence Assist Consultative memorandum with �exible format.

1- 4

w ks

9- 12

m o

1- 2

m o

4- 12

m o

9-12 mo

2- 4

m o

1- 4

m o

Synthesizing evidence for VA leadership to improve the health and healthcare of VeteransESP

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

The VA QUERI Roadmap | PAGE 88

PEPReC Partnered Evidence-Based Policy Resource Center

Informing VA priorities with rigorous data analysis The Partnered Evidence-Based Policy Resource Center (PEPReC) was established in 2017 and is a QUERI resource center designed to provide timely, rigorous data analysis to support the development of high-priority policy, planning, management initiatives and quantitative program evaluations.

Technical assistance on randomized evaluation design, metrics, and analytic plans

Partnering in learning agenda development, in compliance with the Foundations for Evidence-based Policymaking Act

Facilitation of communication between evaluators and operations partners

Programming support, including programmer to programmer technical assistance during evaluations

PEPReC Overview

ACCESS TO CARE PROGRAM EVALUATION RESEARCH FACILITATION Collaborate with VA operations partners to enhance planning and improve access

to and efficiency/quality of care

Engage with operations partners and investigators to design and implement

randomized program evaluations

Facilitate consortia to expedite operations-relevant evaluation

PEPReC Services and Activities

Availability of resources depends on health system priorities and is greatest for areas of high priority to the Veterans Health Administration. These include issues iden- tified by HSR&D/QUERI with established consortia of investigators — current health system priorities include access to care, MISSION Act, community care, opioid risk mitigation, and suicide risk mitigation.

The VA QUERI Roadmap | PAGE 89

Informing VA priorities with rigorous data analysis

Working with PEPReC PEPReC’s services are particularly relevant during the Pre- implementation and Sustainment phases by helping develop rigorous program evaluation plans to help policy makers with strategic decisions regarding the adoption or continuation of clinical practices, programs, and policies.

The QUERI Roadmap provides a systematic, three-phased approach to help practitioners overcome barriers to implementing effective practice, Pre-Implementation, Implementation and Sustainment. For more details, visit the QUERI Roadmap.

External Users: https://www.peprec.research.va.gov/ Email: [email protected]

Performance to D ata

Knowledge to Perform an

ce

D at

a to

K no

w le

dg e

�� �� �� �� ��

��� ����� ��������������

�����������

PEPReC Process PEPReC’s goal is to be an engine for VA’s learning health system, where research informs practice and practice informs research.

• Designing metrics and incentives, informed by existing research

• Collaborating with operations partners to implement those metrics and incentives

• Evaluating results with randomized, quasi- experimental, or strong observational designs

• Suggesting and making refinements to evaluation and operations plans, reflecting results and operational constraints and objectives

Medical Scribes, Productivity, and Satisfaction – This policy brief describes increases in provider productivity and satisfaction associated with private sector scribe use, offering guidance to VHA policymakers. The MISSION Act mandated a two-year pilot program to study the impact of medical scribe use in VHA emergency departments and specialty care clinics. In partnership with the Office of Veterans Access to Care, PEPReC is evaluating the impact of medical scribes on VHA provider productivity and patient satisfaction during the pilot.

Priority access to health care: Evidence from an exogenous policy shock – PEPReC investigators evaluated the effects of changes in appointment scheduling policies on patients’ access to primary care. Recommendations for optimal allocation of resources include recognizing and explicitly accounting for differences in access between established and new patients.

PEPReC

Practice Inform s Evaluation

PEPReC

Metrics and incentives

Iterative Re�nement

Implementation Evaluation & Monitoring

Outcomes Evaluation

Ev al

ua ti

on In

fo rm

s Pr

ac ti

ce

Ev al

ua ti

on In

fo rm

s Pr

ac ti

ce

Operations

PEPReC

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

The VA QUERI Roadmap | PAGE 90

VIReC VA Information Resource Center

Your guide to navigating the VA data landscape The VA Information Resource Center (VIReC), an HSR&D resource center established in 1998, is a key resource for navigating VA’s complex data environment. VIReC’s mission is to advance VA capacity to use data effectively for research and quality improvement and to foster communication between research data users and the VA healthcare community.

VIReC Overview

DATA KNOWLEDGE & METHODS

DATA RESOURCES

DATA ACCESS & AVAILABILITY

Developing a working knowledge about VA data and information systems

Supporting data users in navigating the VA data landscape

Collaborating within VA to enhance data access and increase available data

VIReC Services and Benefits Understanding which data sources are available to VA researchers, their scope, and their contents can be challenging. VIReC meets this challenge through disseminating a portfolio of products and services. VIReC’s websites offer an array of resources for understanding the VA data landscape, and VIReC HelpDesk and consultations provide individualized responses to questions about VA data and VIReC products and services.

Intranet www.virec.research.va.gov

VHA Data Portal vaww.vhadataportal.med.va.gov

Internet vaww.virec.research.va.gov

SPOTLIGHT Electronic Health Record Modernization (EHRM) is VA’s initiative to migrate its current EHR (VistA/CPRS) to the Cerner Millennium EHR. VIReC leads the Office of EHRM Research Council in advocating for research and quality improvement requirements from the new EHR. The Center also heads up the Office of Research & Development (ORD) Strategic Initiative for Research and Electronic Health Record Synergy (OSIRES) to provide essential communication and strategic planning around the transition.

Visit dvagov.sharepoint.com/sites/VHAPugResearch/RRG/Pages/EHRM-Research.aspx (VA Network Only) for more details, or email [email protected] with questions.

The VA QUERI Roadmap | PAGE 91

Your guide to navigating the VA data landscape

Working with VIReC VIReC products and services can help identify data sources that are relevant during both Pre-Implementation and Implementation phases. Understanding more about available data can help you develop a data and evaluation plan to monitor your efforts.

The QUERI Roadmap provides a systematic, three-phased approach to help practitioners overcome barriers to implementing effective practice, Pre-Implementation, Implementation and Sustainment. For more details, visit the QUERI Roadmap.

External Users: www.virec.research.va.gov Internal Users: vaww.virec.research.va.gov vaww.vhadataportal.med.va.gov Email: [email protected]

Performance to D ata

Knowledge to Perform an

ce

D at

a to

K no

w le

dg e

�� �� �� �� ��

��� ����� ��������������

�����������

VIReC Resources VIReC provides information about the content, structure, utility, and limitations of VA data to support outcomes and effectiveness research, healthcare operations, implementation, and organizational decision making. Visit the VIReC websites to learn more about these products and services.

Data Documentation Producing information and guidance

about VA data and its use

VA REDCap Supporting VA primary data collection and management

Cyberseminars Offering topical sessions on data,

methods, and informatics

VA/CMS Data for Research Providing Medicare, Medicaid, and

USRDS data for VA research methods, and informatics

Data Issues Brief Disseminating news about VA data, information systems, and resources

HSRData Listserv Facilitating community generated

knowledge about VA data

VIReC

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

The VA QUERI Roadmap | PAGE 92

HERC Health Economics Resource Center

Supports pioneering health economics research to improve health care for Veterans and beyond The Health Economics Resource Center (HERC) was established in 1999 and is an HSR&D resource center designed to support VA economic research in order to increase health care efficiency and value for Veterans and the nation. HERC provides tools and resources to assist researchers in conducting economic analyses for implementation research.

HERC estimates the cost of all VA health care encounters, develops guidebooks to VA data, and issues technical reports with analy- ses of economic data. HERC economists have developed extensive expertise through their participation in economic studies that are funded by VA HSR&D, QUERI, the VA Cooperative Studies Program, Agency for Healthcare Research and Quality (AHRQ) and the National Institutes of Health (NIH).

HERC Overview

RESEARCH TRAINING CONSULTING HERC helps VA researchers determine the cost of VA care, assess cost-effectiveness,

and evaluate the efficiency of VA programs and providers.

HERC offers trainings to help VA researchers learn about cost data

and economic evaluation.

HERC offers a consulting service for VA researchers, policy makers

and operational leaders.

HERC Services and Benefits

HOT TOPICS • Choice/ Community

Care Costs

• Risk Adjustment

ANALYTIC METHODS • Budget Impact

Analysis (BIA)

• Cost-Effectiveness Analysis (CEA)

• Implementation costs

• Micro-costing

DATA • Inpatient

• Outpatient

• Pharmacy

• Labor

• Providers

TOOLS & TRAINING • Guidebooks • Technical Reports

• Courses & Seminars

The VA QUERI Roadmap | PAGE 93

Supports pioneering health economics research to improve health care for Veterans and beyond

HERC Economic Analysis Process

Working with HERC HERC’s services are particularly relevant during the Pre- implementation and Sustainment phases to develop an appropriate economic evaluation plan to understand the implementation costs for an effective practice and the business case for spread to other sites for sustained use.

The QUERI Roadmap provides a systematic, three-phased approach to help practitioners overcome barriers to implementing effective practice, Pre-Implementation, Implementation and Sustainment. For more details, visit the QUERI Roadmap.

External Users: www.herc.research.va.gov Email: [email protected]

Performance to D ata

Knowledge to Perform an

ce

D at

a to

K no

w le

dg e

�� �� �� �� ��

��� ����� ��������������

�����������

STUDY DESIGN • HERC developed these

Guidelines for Economic Analysis to consider which form of economic analysis is most appropriate for the implementation study.

• Common forms of economic analysis for implementation studies include:

- Cost-identification; micro-costing is often used when the cost of the intervention is not known.

- Cost-effectiveness analysis (CEA)

- Budget impact analysis (BIA)

DATA COLLECTION • Develop an Economic

Analysis Plan.

• Estimate the costs of implementation, intervention, and consequences following the best practices described in this presentation.

• Create a finder file or crosswalk file to link participants in a study to administrative and utilization files.

• Sometimes the study requires the collection of additional information to estimate the cost of the intervention. In these situations, create a tool used for collecting data for micro-costing.

DATA ANALYSIS • Analyze the data and

estimate costs for patients.

• This Statistical Program provides guidance on how to estimate costs for patients. Users can modify this program to identify the appropriate time interval and cost subtotals.

• For studies that re-organize existing care, the cost of the intervention is included in the administrative data and can be estimated through analysis. In studies where the existing administrative data do not include the cost of the intervention, the cost of the intervention will need to be estimated separately and added to the costs.

DATA REPORTING • Report the cost of

the intervention for the intervention and comparison group (mean and measure of variance).

• Report the cost of the subsequent downstream health care costs for the intervention and comparison group (mean and measure of variance).

• Report the net effect, which is the cost of intervention plus cost of subsequent downstream effects for the intervention and comparison group (mean and measure of variance).

HERC

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

QUERI Implementation Strategy Learning Hubs

Ann Arbor, MI – Virtual Quality Improvement Training for VHA Teams

Bedford, MA – Teamwork Training Hub: Frontline Huddling for Quality Improvement Implementation

Denver, CO – Designing for Dissemination and Implementation Training Hub

Houston, TX – Leading Healthcare Improvement: Leadership Training for Applying Improvement Strategies

Little Rock, AR – Behavioral Health QUERI Implementation Facilitation Training Hub

Los Angeles, CA – Evidence-Based Quality Improvement Learning Network

San Francisco, CA – QUERI Adaptation Training Hub

The goal of the QUERI Implementation Strategy Learning Hubs is to demystify implementation science by offering opportunities for clinicians and researchers to learn how to deploy effective practices using specific implementation strategies (i.e., tools or methods that help existing providers adopt effective practices). QUERI Learning Hubs use evidence-based implementation strategies and are coordinated by the Center for Evaluation & Implementation Resources.

Implementation Facilitation

Designing for Dissemination & Implementation

Learn. Engage. Act. Process.

(LEAP)

Leadership Training

Adaptation Evidence-based

Quality Improvement

Frontline Huddling

The VA QUERI Roadmap | PAGE 94

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

The VA QUERI Roadmap | PAGE 95

VAQS VA Quality Scholars

The VA Quality Scholars program trains leaders and scholars in healthcare improvement for the VA and leads change across the globe. Supported by the United States Department of Veterans Affairs, the VA Quality Scholars program develops leaders and scholars in healthcare improvement. Founded in 1999, VAQS is the premier training program in healthcare improvement and leadership and leads change nationally and internationally.

VAQS Overview The VA Quality Scholars program consists of eleven sites across the United States, as well as an affiliate site in Toronto, Canada. Each site in the US consists of a partnership between a VA hospital and an academic institution. It has an interprofessional emphasis with physicians, doctoral-trained nurses, clinical psychologists and pharmacists as fellows and faculty.

Contact us for more information at [email protected]

VAQS Fellowship Benefits Includes

• Two-year healthcare improvement and leadership curriculum

• Networking with national experts and leaders

• Professional development • Leading QI projects • Individual mentoring • Data analysis training and consultation • Expansive alumni network • Stipend and federal government

benefits

VAQS Faculty Expertise • Quality Improvement • Interprofessional Education • Patient Safety • Implementation Science

• System Redesign • Clinical Informatics • Health Service Research • Methods & Analysis

Resources:

• Implementation versus Improvement Science (Quality Improvement)

• Adaptation Guidance Tool

• The Iterative Decision-making for Evaluation of Adaptations (IDEA) Decision Tree

• Abridged Checklist for Conducting Adaptation Interviews

• Detailed FRAME Adaptations and Modifications Checklist

• What’s a Barrier and Facilitator Framework

• List of Expert Recommendations for Implementing Change (ERIC)

• Evaluation and Study Designs for Implementation and Quality Improvement

• Other Resources

General Resources for Implementation and Quality Improvement

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

Implementation Science Versus Quality Improvement? We often hear that implementation science and quality improvement have distinct origins, but they share common concepts. Both fields share overlapping goals and methods, which can make the distinction between the two confusing at times (see Table 24).

The way to distinguish implementation science from quality improve- ment is to clarify the goal of the project. Many implementation science projects involve the deployment and scale-up of an effective practice or innovation. Quality improvement initiatives involve systematic efforts to improve care processes to reach a clinical priority goal (e.g., decrease infection rates).

Implementation science is the study of methods to promote the adoption and integration of effective practices, interventions, and policies into routine health care and public health settings to improve the impact on population health.8

Implementation science addresses the problem of effective prac- tices that are underutilized or adopted too slowly. Implementation scientists work to find and address the quality gaps from these delays at the clinician-, team-, clinic-, or organizational-level and use scientific theories to develop implementation strategies to solve these care deficits.184

Implementation strategies are methods or techniques that enhance the adoption, implementation, and sustainment of a program or practice.71,126

Quality improvement refers to a range of local facility- to systems- level work to improve the quality, safety, and value of health care ser- vices.272 Many quality improvement strategies are similar to implemen- tation strategies, such as rapid-cycle testing or consultation by experts.

Quality improvement efforts usually begin with a specific problem (i.e., poor performance) at a local hospital that is recognized at the level of the clinician, clinic, or health system. In turn, improvement teams design and conduct rapid-cycle tests of “interventions” to fix the prob- lem. This improvement may then spread to improve outcomes across the health system.

Quality improvement often relies on tacit knowledge (“real-world wis- dom”) of stakeholders and a range of tools and analytical measures to make thoughtful and continual improvements to a problem over time.

Both Implementation strategy studies and quality improvement projects can be considered non-research. See the previous section on determining whether implementation science projects are considered non-research.

Table 24. Concepts, Approaches and Terms Used in Improvement Science and Implementation Science adapted from272

Implementation Science

Shared De�nitions & Scope • System: interrelated structures, people,

processes, and activities that together create health care service delivery

• Context: physical and sociocultural make-up of the local environment

• Focus: systems-level

Common Outcomes • E�ect on people processes and systems • Cost, feasibility, sustainability

De�nition Work to promote the systematic uptake of evidence- based interventions into practice and policy

The Problem Evidence slow to be adopted in clinical practice and uptake may be uneven across settings, with variable quality of care

Principles • Behavior change through focusing on mediating

variables • Generalizable mechanisms of change across

locations

Theory An existing framework or model chosen to guide and implementation process

Common Approaches (“implementation strategies”) Business process re-engineering, experience-based co-design, Lean methodology, Model for Improvement, Six Sigma, Statistical process control, Theory of constraints, Total Quality Management

Shared Principles • Data and

measurement • Understanding

the process

• Involving and engaging sta�

• Involving patients • Systems-level change

Outcomes of Interest • Acceptability • Adoption/uptake

• Appropriateness for setting

Improvement Science De�nition

Systems-level work to improve the quality, safety and value of health care

The Problem Meaningful disruption, failure, inadequacy, distress, confusion, or other dysfunction in the health care system that adversely a�ects patients, sta�, or the system and prevents it from reaching its full potential

Principles • Improving Reliability • Managing demand, capacity, and �ow • Location speci�c

Rationale Informal or formal frameworks, models, concepts, and/or theories used to explain the problem; any reasons or assumptions that were used to develop the interventions; and reasons why the interventions were expected to work

Common Approaches (“interventions”) Business process re-engineering, experience-based co-design, Lean methodology, Model for Improvement, Six Sigma, Statistical process control, Theory of constraints, Total Quality Management

Outcomes of Interest • E�ciency • Safety

• Timeliness • Patient Centeredness

The VA QUERI Roadmap | PAGE 97

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

Adaptation Guidance Tool When choosing an evidence-based intervention, changes may need to be made to increase fit or compatibility with a specific audience and/or setting. Here is general guidance in terms of things that can and cannot be changed from the original intervention.

Adaptation Guidance

Green

Things that CAN be changed: Usually minor and made to increase reach, receptivity, and consumer participation

• Names of health care centers or systems • Pictures of people and places and quotes • Hard-to-read words that affect reading level • Wording to be appropriate to audience

• Cultural indicators based on population (e.g., language, pictures, art, scenarios)

• Ways to reach and recruit the target audience • Incentives for participation • Timeline

Yellow

Things that can be changed with caution: Typically add or modify practice components and contents versus deleting them

• Substituting activities and/or adding new activities • Changing the order of the curriculum or steps (i.e.,

sequence) • Altering the length of program activities • Shifting or expanding the primary audience

• Varying delivery format/process steps • Modifying who delivers the program • Adding activities to address other risk factors or

behaviors

Red

Things that CANNOT be changed: Changes to the core components

• The health model, theory, or core logic • The health topic, behavior • Core components or whole sections of the program

• Reduction of program - Timeline - Dosage (e.g., activities, time/session)

• Adding more strategies that detract from the core components

1. *Tool adapted from: Lesesne, C. A., Lewis, K. M., Moore, C., Fisher, D., Green, D., & Wandersman, A. (2007). Promoting Science-based Approaches to Teen Pregnancy Prevention using Getting To Outcomes: Draft June 2007. Unpublished manual.

2. * Tool adapted from Session 5: Adapting an Evidence-based Intervention to Fit Your Community. Putting Public Health Evidence in Action Training Workshop. Cancer Prevention and Control Research Network: https://cpcrn.org/training. February 2020.

The VA QUERI Roadmap | PAGE 98

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

Iterative Decision-making for Evaluation of Adaptations (IDEA) Decision Tree The Iterative Decision-making for Evaluation of Adaptations (IDEA) decision tree is a tool to guide effective practice adaptation that incorporates important decision points and the dynamic nature of ongoing adaptation. Its use is intended to help implementation scientists, clinicians, and administrators maximize effective practice impact.

Dose stakeholder input, evaluation, published data, or needs assessment

data suggest and adaptation is needed?

Does time-frame allow pilot? Are core elements or core functions of the intervention known?

Are desired outcomes non-inferior or improved over expected/

published outcomes?

Make decision about further adaptation vs. reversion or

de-implementation

Is “voltage drop” acceptable to stakeholders?

Can barrier/concern be addressed while preserving core intervention

elements/function?

No

No

No

Yes

Yes

Proceed but evaluate, identifying opportunities

to re�ne.

No

No No

Small pilot with measurement of key

outcomes.

Yes

Yes Yes Yes

Yes

A

D

E F

B

C

References: 1. Miller CJ, Wiltsey Stirman S, Baumann A. Iterative Decision-making for Evaluation of Adaptations (IDEA): a decision tree for balancing adapta-

tion, fidelity, and intervention impact. J Community Psychol. 2020;In press. doi: 10.1002/jcop.22279.

2. The F.A.S.T. Lab – The Fidelity, Adaptation, Sustainability, and Training Lab at the VA National Center for PTSD and Stanford University: http:// med.stanford.edu/fastlab.html

The VA QUERI Roadmap | PAGE 99

Conducting Adaptation Interviews Abridged Check List

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

For more information on how to use this guide, please refer to: Rabin BA, McCreight M, Battaglia C, et al. Systematic, Multimethod

Assessment of Adaptations Across Four Diverse Health Systems Interventions. Front Public Health. 2018;6:102.

Questions to ask about each adaptation

WHAT component, part of the practice or implementation strategy was changed?

HOW would you describe the type of change involved in this adaptation?

WHO was responsible for first suggesting or initiating this change?

WHEN during the XYZ program was this adaptation first made?

HOW or on what BASIS was this change made?

WHY (1): WHY was this adaptation made?

WHY (2): Was this adaptation a result of EXTERNAL factors or INTERNAL issues

WHAT was the short term IMPACT of this adaptation on WHAT outcomes?

OTHER? Probe any other aspects of adaptations that emerge from Interviewee comments

Who you should interview

• Key staff implementing the clinical practice/program

How to identify key informants

• Identify with help of research staff familiar with local stakeholders

• Include at least 3 staff who have different implementation roles

When to conduct the interview

• A third of the way through the program

• Shortly after conclusion of the program

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

Framework for Reporting Adaptations and Modifications-Enhanced (FRAME) Adaptation Process WHEN did the modification occur? ❏ Pre-implementation/planning/pilot ❏ Implementation ❏ Scale up ❏ Maintenance/sustainment

Were adaptations PLANNED? ❏ Planned/proactive adaptation ❏ Planned/reactive adaptation ❏ Unplanned/reactive modification

WHO participated in the decision to modify? ❏ Ex: Political leaders, funder, administrator,

community members… ❏ Optional: Indicate who made the ulti-

mate decision

WHAT is modified/adapted? ❏ Content

Modifications to content or that impact aspects of treatment delivery

❏ Contextual Modifications to overall way intervention is delivered

❏ Training and Evaluations Modifications to staff training or how the intervention is evaluated

❏ Implementation and scale-up activities Modifications to the strategies used to implement or spread the intervention

Contextual adaptations/modifications are made to which of the following? ❏ Format ❏ Setting ❏ Personnel ❏ Population

At what LEVEL OF DELIVERY (for whom/ what is the modification made?) ❏ Individual ❏ Target intervention group ❏ Cohort ❏ Individual clinician ❏ Clinic/unit level ❏ Organization ❏ Network System/community

What is the nature of the content modification/adaptation? ❏ Tailoring/tweaking/refining ❏ Changes in packaging or materials ❏ Adding/removing/skipping elements ❏ Shortening/condensing/lengthening/

extending (pacing/timing)

❏ Substituting/reordering/spreading of intervention modules or segments

❏ Integrating parts of the intervention into another framework

❏ Integrating another treatment into EBD ❏ Repeating elements or modules ❏ Loosening structure ❏ Departing from the intervention (“drift”)

followed by a return to protocol within the encounter

❏ Drift from protocol without returning

Relationship to fidelity/core elements? ❏ Fidelity consistent/core elements or

functions preserved ❏ Fidelity inconsistent/core elements or

functions changed ❏ Unknown

What was the goal of the adaptation or modification? ❏ Increase reach or engagement ❏ Increase retention ❏ Improve feasibility ❏ Improve fit with recipients ❏ To address cultural factors ❏ Improve effectiveness/outcomes ❏ Reduce cost ❏ Increase satisfaction

Reasons for Adaptations Socio-political ❏ Existing laws ❏ Existing mandates ❏ Existing policies ❏ Existing regulations ❏ Political climate ❏ Funding policies ❏ Historical content ❏ Societal/cultural norms ❏ Funding or resource alloca-

tion/availability

Clinician ❏ Race/Ethnicity ❏ Sexual/gender identity ❏ First/spoken languages ❏ Previous training and skills ❏ Preferences ❏ Clinical judgement ❏ Cultural norms, competency ❏ Perception of intervention

Organization/Setting ❏ Available resources (funds,

staffing, technology, space) ❏ Competing demands or

mandates ❏ Time constraints ❏ Service structure ❏ Location accessibility ❏ Time constraints ❏ Regulatory compliance ❏ Billing constraints ❏ Social context (culture, cli-

mate, leadership support) ❏ Mission ❏ Cultural or religious norms

Recipient/Consumer ❏ Race/ethnicity ❏ Gender identity ❏ Sexual orientation ❏ Access to resources ❏ Cognitive capacity ❏ Physical capacity ❏ Literacy and education level ❏ First/spoken languages ❏ Legal status ❏ Cultural/multimorbidity ❏ Immigration status ❏ Crisis or emergent circum-

stances ❏ Motivation and readiness

References:

1. Stirman SW, Miller CJ, Toder K, Calloway A. Development of a framework and coding system for modifications and adaptations of evidence-based interventions. Implement Sci. 2013;8:65.

2. Wiltsey Stirman S, Baumann AA, Miller CJ. The FRAME: an expanded framework for reporting adaptations and modifications to evidence- based interventions. Implement Sci. 2016;14(1):5

3. The F.A.S.T Lab - The Fidelity, Adaptation, Sustainability, and Training Lab at the VA National Center for PTSD and Stanford University: http:// med.stanford.edu/fastlab.html

The VA QUERI Roadmap | PAGE 101

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

What’s a Barrier and Facilitator Framework? Determinants are contextual factors believed or shown to influ- ence implementation processes or outcomes. They are also referred to as barriers/obstacles or facilitators/enablers to implementation processes.273

The purpose of a determinant framework is to:273

• Diagnose the reasons why a gap in quality exists • Understand or explain what influences an implementation

process • Help support decisions about evaluation/measurement (what to

assess) • Build theory in implementation science

A determinant framework is a catalog or checklist of these factors that specifies types, domains, or classes at multiple levels of the health system.

There are dozens of implementation science determinant frame- works119,273 that describe relationships between some types of deter- minants with the aim of helping to understand and/or explain the influence on implementation outcomes.7 Examples of types or levels of determinants:

• Intervention characteristics • Consumer/caregivers/family • Clinician • Team/unit/service • Organization • System or external environment • Multi-level (time, financing, leadership, social relations/support)

Some types of determinant frameworks focus on summarizing barriers and facilitators to implementing an effective practice for a type of clinical care or consumer population (e.g., psychosocial interventions for mental health/substance use treatment, cancer care) or delivery of a practice using e-health or m-health technologies.

Other frameworks are informed by one or two underlying theories of individual or organizational behavior change. This helps make sense of the multi-level determinants related to implementing a practice to specific consumer populations or settings.

However, the more widely used frameworks bring together determi- nants and theoretical constructs from many theories and blend them

into a single catalog of relevant factors, levels of assessment, and theory for assessing the implementation of most health care effective practices

Three widely used, “blended” determinant frameworks that are helpful in assessing, organizing, and prioritizing the influence of these factors in a local context include:

Theoretical Domains Framework (TDF)274 – An integrative frame- work that synthesizes 33 psychology theories into 14 domains and 84 constructs that can be used to develop theoretical approaches to interventions aimed at individual-level behavior change.

Consolidated Framework for Implementation Research (CFIR)79 – Conceptualizes 39 determinant constructs across five major domains that may affect an intervention’s implementation. They include inter- vention characteristics, inner setting (of organization), outer setting, characteristics of individuals involved in implementation, and the implementation process itself. For more, see Other Implementation Resources.

Tailored Implementation for Chronic Disease (TICD)213 – Uses a checklist with 57 potential individual- and organizational-level deter- minants of clinical practice improvement grouped into seven domains. These domains include guideline factors, individual health profession- al factors, professional interactions, incentives, resources, capacity for organizational change, and social-political-legal factors. The imple- mentation includes additional checklists that guide the development of an implementation strategy to address identified determinants.

• Review epidemiological and health services research studies that report on how these multi-level contextual factors (determinants) affect service delivery in the targeted clinical area to understand their relative influence

• Identify and apply relevant theories on emerging and relevant determinants

• Use formative evaluation methods to interview or survey multi- level stakeholders to understand relevant influences using implementation science theories and determinant frameworks to structure questions and guide analyses

For guidance selecting a dissemination or implementation science theory and models to guide your planning process, visit: Dissemina- tion & Implementation Models in Research & Practice

The VA QUERI Roadmap | PAGE 102

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

Expert Recommendations for Implementing Change (ERIC) This collection of definitions for 73 discrete implementation strategies from the ERIC Study126,127 includes updated definitions and three new strategies from the evaluation128 of the EvidenceNOW Initiative funded by the Agency for Healthcare Research and Quality. Reprinted with permission.

Table 25. Implementation Strategies Organized by Functional Category

Category and Strategy

Definition Ancillary Lessons Material from ERIC Study and from the Agency for Healthcare

Research Quality EvidenceNOW Initiative

Strategies to Develop Stakeholder Relationships

Identify and prepare champions

Identify and prepare individuals who dedi- cate themselves to supporting, marketing, and driving through an implementation, overcoming indifference or resistance that the effective practice may provoke in an organization.

This strategy includes preparing individuals for their role as champions. Champions are primarily internal to the organization. Additional issues raised include the need for guidance regarding:

a) Methods and considerations related to the selection and identification of champions. Social network theory and methods may be useful in this regard.

b) Training and or providing champions support materials.

c) Addressing incentives or disincentives to the champion role.

d) Whether there are needs for champions at different levels of an organization (e.g., clinic, region, national).

Champions are often distinguished from opinion leaders. Opinion leaders may be considered more of an objective third party with relevant expertise

Organize implementation teams and team meetings

Develop and support teams of clinicians, staff, consumers, and other stakeholders who are implementing or may be users of the effective practice. Provide protected time for teams to reflect on the implemen- tation progress, share lessons learned, make refinements to plans, and support one another’s learning.

None.

Recruit, designate, and train for leadership

Recruit, designate, and train leaders for the change effort.

Change efforts require certain types of leaders, and organizations may need to recruit ac- cordingly, rather than assuming that their current personnel can implement the change. Designated change leaders can include an executive sponsor and a day-to-day manager of the effort. Change leaders should consider how to best establish supervisory lines for effective practices that are enacted by clinicians when the change leader does not have similar clinical responsibilities.

Inform local opinion leaders

Inform clinicians identified by colleagues as opinion leaders or “educationally influ- ential” about the effective practice in the hopes that they will influence colleagues to adopt it.

The opinions of individuals who refer people to services, or who initiate the connection to services also function in a key opinion role. Keeping opinion leaders informed from Pre- implementation through maintenance of the effective practice is important. Ensuring that opinion leaders do not serve as implementation obstacles if they are not actively promoting the effective practice is also important.

Build a coalition Recruit and cultivate relationships with partners in the implementation effort.

Partnerships can develop around cost-sharing, shared resources, shared training, and the divi- sion of responsibilities among partners. This work may proceed naturally from local consensus discussions. Coalition members commonly have defined roles in the implementation effort.

Engage community resources

Connect practices and their consumers to community resources outside the practice (e.g., state and county health departments; nonprofit organizations; resources related to addressing the social determinants of health; and organizations focused on self- management techniques and support).

None.

The VA QUERI Roadmap | PAGE 103

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

Category and Strategy

Definition Ancillary Lessons Material from ERIC Study and from the Agency for Healthcare

Research Quality EvidenceNOW Initiative

Obtain formal commitments

Obtain written commitments from key partners that state what they will do to implement the effective practice.

Formal commitments should clarify roles, responsibilities, and communicate tangible and non-tangible benefits (e.g., community partnerships). Ensure that key roles are supported within the organization (e.g., workload release credit for providing and receiving supervision in a new clinical practice). Formal commitments in no way diminish the importance of infor- mal commitments to a change effort.

Identify early adopters

Identify early adopters at the local site to learn from their experiences with the effec- tive practice.

Early adopters are a good pool for identifying implementation champions. Recruit early adopters to attend stakeholder meetings to present their experiences. Investigating the adop- tion chasm between early adopters and the early majority has been found to be useful. Differ- ent engagement techniques for these two groups are typically needed. For further discussion see Moore (2014).

Conduct local consensus discussions

Include local clinicians and other stakeholders in discussions that address whether the chosen problem is important and whether the effective practice to ad- dress it is appropriate.

Identify stakeholders relevant to each project. Further, with each project, there will be a need to identify whether the goal of the consensus discussion is to characterize consensus or build consensus. Utilizing community-based participatory research principles may be relevant to many effective practices. Notably, the chosen problem needs to be a high enough priority compared to other problems that attention and resources will be dedicated to addressing the problem.

Capture and share local knowledge

Capture local knowledge from implemen- tation sites on how implementers and clini- cians made something work in their setting and then share it with other sites.

This strategy is often coordinated with centralized technical assistance and learning collabora- tives. There are multiple techniques for capturing local knowledge, which could be presented in multiple formats. For example, short Tube videos could be created that document testimo- nials from clinicians who have successfully used a given effective practice. Another example would be maintaining a running list of a team's response to specific implementation barriers that could be shared readily through a platform like Google Docs or Microsoft SharePoint. Ad- ditional techniques can be found at www.liberatingstructures.com.

Use advisory boards and workgroups

Create and engage a formal group of multiple kinds of stakeholders to provide input and advice on implementation efforts and to elicit recommendations for improvements.

Consider how group composition (or heterogeneity) impacts stakeholder participation and take active steps to reduce response bias. For example, inclusion of supervisors and super- visees in these groups can be problematic and it may be a desirable strategy to ensure that these situations are avoided due to the power difference in the relationship. Supervisees, for example, may feel pressure to report positively to put a good face on for the supervisors, and supervisors may feel pressure to deny having any problems with implementation to save face. It can be useful to distinguish between internal stakeholders and representatives (in a par- ticipatory approach to maintain buy-in and relevance) versus external experts and advisors. Similarly, depending on the input or oversight need, the workgroup composition may include multiple-level or multidisciplinary stakeholders.

Use an implementation advisor

Seek guidance from experts in implemen- tation, including providing support and training for the implementation work force.

This could include consultation with outside experts such as university-affiliated faculty members or hiring quality improvement experts and/or implementation professionals. Imple- mentation Advisors may provide support and training for facilitators (see Implementation Facilitation) related to broad facilitation skills, how to use and interpret data, or effective use of an electronic health record.

Model and simulate change

Model or simulate the change that will be implemented prior to Implementation.

Computer simulations, walkthrough simulation exercises, or modeling the potential overall impact of stakeholder's behavior change may be used. System dynamics modeling is one example of a specific method that may be used. This approach is often more relevant for complex multi-component effective practices.

Visit other sites Visit sites where a similar implementation effort has been considered successful.

Clarifying the goals of the site visit prior to making the visit is particularly useful. Comparing and contrasting the features of one’s own site with the comparison site in preparation for the visit may better inform the visit objectives. Clarifying goals, in part includes developing a plan for using the information upon returning to your setting. Identify adaptations made in imple- menting the innovation and any perceived impact of the effective practice. It is important to document facilitators and lessons learned. Much can be learned from visiting sites that have a strong track record for successfully implementing a wide variety of other effective practices. Consulting with sites where implementation has stalled or failed can also provide useful information. Sites also benefit from sharing implementation planning and execution notes virtually (i.e., information exchange is not limited to physical visits).

The VA QUERI Roadmap | PAGE 104

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

Category and Strategy

Definition Ancillary Lessons Material from ERIC Study and from the Agency for Healthcare

Research Quality EvidenceNOW Initiative

Involve executive boards

Involve existing governing structures (e.g., boards of directors, medical staff boards of governance) in the implementation effort, including the review of data on implemen- tation processes.

Other types of leadership with ‘top-down’ powers may be involved for settings that do not have a governing board. Examples include administrative leadership, clinical leadership, policy makers, and insurance clinicians or other payment systems.

Develop an implementation glossary

Develop and distribute a list of terms de- scribing the effective practice, implementa- tion, and stakeholders in the organizational change.

When compiling a glossary, reflect as to whether the terms being introduced are essential.

Develop academic partnerships

Partner with a university or academic unit for the purposes of shared training and bringing research skills to an implementa- tion project.

The Health Insurance Portability and Accountability Act (HIPAA), and other legal limitations are common to encounter with academic partnerships. Formal relationships (e.g., contracts, memorandums of understanding) will be required in some instances. Not all academics have a full understanding of practice-level stakeholder needs and this should be considered while developing this partnership. In settings where ‘research’ is not a commonly supported practice, evaluation or developmental evaluation may be more useful ways of characterizing the activity.

Promote network weaving

Identify and build on existing high-quality working relationships and networks within and outside the organization, organization- al units, teams, etc. to promote information sharing, collaborative problem-solving, and a shared vision/goal related to implement- ing the effective practice.

Individuals functioning as network weavers usually have external links outside of the com- munity to bring in information and ideas. An example would be nurses and doctors who staff hospitals and skilled nursing facilities, and the consumers who rotate among these facili- ties. Networks are somewhat more organic than collaboratives and are often enduring and durable. See: http://www.networkweaver.com/

Strategies to Adapt and Tailor to Context

Tailor strategies

Tailor the implementation strategies to address barriers and leverage facilitators that were identified through earlier data collection.

The tailoring process tends to be idiosyncratic and driven by multiple factors. It is important to identify the core components of the effective practice and implementation strategies that are required to maintain fidelity and effectiveness, and to distinguish those components ame- nable to modification/adaptation. Flottorp et al (2013) and Langley et al. (2009) provide guid- ance regarding many of the multiple factors to consider. Wensing et al (2011, 2014) provide an example of a structured approach to tailoring strategies.

Promote adaptability

Identify the ways a clinical innovation can be tailored to meet local needs and clarify which elements of the innovation must be maintained to preserve fidelity.

Preserving fidelity to the effective practice can be an uncertain process if the core elements of the innovation are not empirically defined.

Use data experts

Involve, hire, and/or consult experts to acquire, structure, manage, report, and use data generated by implementation efforts.

It is sometimes necessary to engage data experts early in the implementation planning process. Data Experts may be particularly important to enable effective use of data-driven effective practices and to support other strategies such as: Develop Implementation Tools for Quality Monitoring; Develop and Organize Quality-Monitoring Systems; Audit and Provide Feedback; Change Record Systems; and Use Data Warehousing Techniques.

Use data warehousing techniques

Integrate clinical records across facilities and organizations to facilitate implementa- tion across systems.

A key feature of this strategy is linking local data to data platforms outside the local setting. Often the connection is to a central data repository that provides benchmarking capability (see Audit and Provide Feedback) or other aggregation functions. Records that include vari- ables that can serve as outcome measures are particularly useful. When outcomes of interest are not available, it may be useful to examine proxy measures.

The VA QUERI Roadmap | PAGE 105

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

Category and Strategy

Definition Ancillary Lessons Material from ERIC Study and from the Agency for Healthcare

Research Quality EvidenceNOW Initiative

Strategies to Use Evaluative and Iterative Strategies

Conduct a local needs assessment

Collect and analyze data related to the need for the effective practice.

This assessment could be focused on:

• Outcomes of usual care

• Process of care

• Description of usual care and its distance from the effective practice (e.g., gaps in care)

• Opinions from stakeholders (including consumers) on (a) barriers and facilitators to the desired outcome (e.g., recovery from mental illness); (b) the need for any effective practice (i.e., tension for change); (c) the need for a specific innovation; or (d) the special considerations for delivering the effective practice in the local context.

Common needs assessment methods include surveys, focus groups, key informant interviews, direct observation, and data mining of administrative records utilized to identify priority populations, as well as to identify baseline care process and outcome clinical care data. If the change involves multiple sites or facilities, then it is necessary to examine practice varia- tion across facilities and outline strategies for the needs assessment to support a standard- ized approach across sites. Collecting data from a random sample of stakeholders may be necessary to reduce response bias and decrease chances that the level of need is not over- or under-estimated.

Obtain and use consumers and family feedback

Develop strategies to increase consumer and family feedback on the implementa- tion effort.

This can continue throughout the implementation effort. Strategies could include complaint forms that funnel feedback to change managers or advisory boards. Consider whether anony- mous feedback formats are appropriate.

Assess for readiness and identify barriers and facilitators

Assess various aspects of an organization to determine its degree of readiness to implement an effective practice, identify barriers that may impede implementa- tion, and determine strengths that can be leveraged to facilitate the implementation effort.

Readiness assessments may focus on agency finances, staffing levels, and other material or lo- gistical resources needed, or available, to support the implementation effort. This assessment may also focus on leadership support, the organizational priority for change, and the presence of successful experience with quality improvement techniques and change management. Additional aspects for assessment may include other services provided, as well as community support, stakeholder attitudes, and beliefs and perceptions of evidence for the effective prac- tice. Rationale for current practices, organizational climate and culture, structure, decision- making styles, and the perceived needs of frontline stakeholders to implement the effective practice (consider adaptation needs and limits) are also important aspects to consider in this assessment. Readiness assessments can be used to vet, eliminate, or prioritize implementation sites. More so, the assessment can help inform internal decisions about whether to go ahead with an implementation initiative. Some barriers can be difficult to observe prior to imple- mentation. Specific measures have been created to assess readiness for change, which may be useful. This strategy does not include Assess and Redesign Workflow.

Develop an implementation blueprint

Develop a formal implementation blue- print that includes all goals and strategies. The blueprint should include the following: 1) aim/purpose of the implementation; 2) scope of the change (e.g., what organiza- tional units are affected); 3) timeframe and milestones; and 4) appropriate perfor- mance/progress measures. Use and update this plan to guide the implementation effort over time.

The implementation blueprint or manual may be informed by one or more theories or con- ceptual frameworks and/or data from pre-implementation needs assessments. This blueprint can also provide a useful historical record of the implementation process, as well as provide a mechanism to track changes over time. The implementation blueprint is often useful to ensure that feedback is received from prospective frontline users of the blueprint prior to implementation.

Consider coordinating this strategy with the development of a fidelity monitoring tool.

Issues to consider separately, especially for research purposes:

• Number and type of implementation strategies

• Organizational levels involved—this can vary by type of intervention. It may be possible to do some interventions at the lowest level. Others may require top management.

• Pre-implementation assessments would be separate step

Other examples of how to create an implementation blueprint include the CDC and Preven- tion's Replicating Effective Programs.

The VA QUERI Roadmap | PAGE 106

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

Category and Strategy

Definition Ancillary Lessons Material from ERIC Study and from the Agency for Healthcare

Research Quality EvidenceNOW Initiative

Develop and implement tools for quality monitoring

Develop, test, and introduce into quality- monitoring systems the right input – the appropriate language, protocols, algorithms, standards, and measures (of processes, consumer outcomes, and imple- mentation outcomes) that are often spe- cific to the innovation being implemented.

These tools should be flexible enough to reflect fidelity, even after adaptations to the setting or client. Performance sites can benefit when these tools are available locally, particularly to help clinicians develop a sense of ownership for the change process. Quality-monitoring tools can be coordinated with other strategies to encourage or reward performance that is in align- ment with the effective practice. This strategy may include activities necessary to prepare data needed to support the Develop and Organize Quality-Monitoring Systems and the Audit and Provide Feedback strategies, including data cleaning, validation, and standardization.

Develop and organize quality- monitoring systems

Develop and organize systems and proce- dures that monitor clinical processes and/ or outcomes for the purpose of quality assurance and improvement.

This includes developing systems for monitoring through peer reviews, collecting data from consumers, clinicians, and supervisors, and using administrative and electronic record data. This category of strategies also includes the design of disease-specific clinical registries and dashboards, where clinical information and tools (graphical representations, real-time report cards, comparisons with benchmarks, fidelity monitoring, etc.) are easily available to key stakeholders and users, e.g., care team members. These systems may provide support to the Audit and Provide Feedback strategy. Some intensive fidelity monitoring activities (e.g., psy- chotherapy recordings) are more practical at random, but not infrequent, intervals.

Conduct cyclical small tests of change

Implement changes in a cyclical fashion using small tests of change before taking changes system-wide. Tests of change benefit from systematic measurement, and results of the tests of change are studied for insights on how to do better. This process continues serially over time, and refinement is added with each cycle.

Two common small tests of change cycling strategies are “Plan-Do-Study-Act” from Dem- ing’s quality management work, and Six Sigma’s Define-Measure-Analyze-Improve-Control sequence.

Audit and provided feedback

Develop summaries of clinical performance over a specific time period, often including a comparator, and give it to clinicians and/ or administrators. Summary content (e.g., nature of the data, choice of comparator) and their delivery (e.g., mode, format) are designed to modify specifically targeted behavior(s) or actions of individual clini- cians, teams, or health care organizations.

Brehaut and colleagues (2016) list 15 suggestions for optimal outcomes using this strategy in clinical settings. Feedback may be derived from a variety of sources, including Quality- Monitoring Systems, medical records, computerized databases, observation, or feedback from consumers. Performance evaluations may also be considered as audit and feedback data if the evaluation included specific information on clinical performance. Feedback summaries may include recommendations. Feedback may be displayed publicly, and often involves comparisons to peers or to local, state, national, or international norms or benchmarks. Feed- back may be designed to guide a clinician in improving fidelity. It should also be noted that audit and feedback data can be helpful in promoting the continuation of intended behavior. Performance data may include process variables, outcomes, or fidelity measures. Feedback can include mandatory performance measures, which are related to benchmarks from the literature or normative data within an organization or industry. This strategy may rely on systems described by the Develop and Organize Quality-Monitoring Systems strategy. Audit and Provide Feedback includes pre-defined summaries of data that are specifically designed to trigger change in behavior to improve clinical performance. Feedback is structured and provided at regular pre-defined intervals. These features differentiate this strategy from De- velop and Organize Quality-Monitoring Systems, which is focused on continuous monitoring via, e.g., dashboards or registries.

Purposefully reexamine the implementation

Monitor progress and adjust effective practices and implementation strategies to continuously improve the quality of care.

It is beneficial to use a concrete schedule for monitoring rather than ‘as needed.’ Time-sensitive benchmarks for determining when adjustments are needed have also been found to be useful.

Assess and redesign workflow

Observe and map current work processes and plan for desired work processes, identi- fying changes necessary to accommodate, encourage, or incentivize use of the clinical innovation as designed.

Workflow redesign may comprise consideration of the following:

• Perceived process (what we think is happening);

• Reality process (what the process actually is); and

• Ideal process (what the process could be).

Process mapping is a core technique for system change. Process maps are used to develop plans that may be a part of Develop an Implementation Blueprint.

The VA QUERI Roadmap | PAGE 107

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

Category and Strategy

Definition Ancillary Lessons Material from ERIC Study and from the Agency for Healthcare

Research Quality EvidenceNOW Initiative

Stage implementation scale-up

Phase implementation efforts by starting with small pilots or demonstration projects and gradually move to a system-wide rollout.

This involves an iterative process that often results in adaptations. Strategies for integrating pi- lot feedback into the scale-up or spread process should be established in advanced. Depend- ing on the effective practice, piloting may also involve phasing in elements or components of the practice change. Many effective practices involve more than one service (e.g., inpatient and outpatient; primary care and specialty care), and the scaling-up or spread process may have different needs to address the interactions among services (e.g., needs related to ensure continuity of care while connecting services). For more details see the Institute for Healthcare Improvement’s white paper, which describes a framework for spread.

Strategies to Train and Educate Stakeholders

Conduct ongoing training

Plan for and conduct training in the ef- fective practice in an ongoing way for all individuals involved with implementation and users of the effective practice (e.g., clinicians, implementation staff, practice facilitators).

This can include follow-up training, advanced training, booster training, purposefully spaced training, training to competence, integration of off-the-job and on-the-job training, structured supervision, the introduction of concepts in a specific sequence to ensure mastery, and train- ings based on the level of clinician knowledge. Ongoing training efforts need to reach across shifts and accommodate staff turnover, as well as rotating staff (e.g., residents). Trainings can be in person, on the web, or technology-assisted (e.g., simulation lab training), and may focus on individuals or involve groups. When planning for ongoing training, it is important to describe the training components, including the timing and frequency of trainings. Issues related to the dynamics of training can be found in the strategy, making training dynamic.

Provide ongoing consultation

Provide ongoing consultation with one or more experts in the strategies used to sup- port implementing the effective practice.

Ongoing consultations could include in-person or distance consultation and feedback on taped clinical encounters. Consultations are tailored to the clinician’s actual practice, thus, differentiating a consultation from ongoing trainings. Feedback may be from a consultant external to the organization, which distinguishes consultation from clinical supervision. Some practice changes can involve a recertification process, thus, involving consultation ensures adequate fidelity. Consultation may also be necessary for non-clinical staff such as administra- tors and those responsible for billing, constructing feedback systems, or other staff with duties that impact the implementation process.

Develop educational materials

Develop and format manuals, toolkits, and other supporting materials to make it easier for stakeholders to learn about the effective practice and for clinicians to learn how to deliver it. This can include technology-delivered (e.g., online/smart- phone-based static or dynamic) content and health messaging.

Create eye-catching, easy-to-use educational documents. Distill complex information into easier-to-learn components. Consider teaching skills modularly. Use different forms of media, and target messages for different audiences. Educational materials should reflect principles of adult learning theory. Assessment of current, available technology infrastructure to accommo- date educational media (e.g., firewalls, old hardware, old software) is merited. Consider how the educational materials will be used over time. For example, will the educational materials’ primary use be to train new or rotating staff; to refresh staff knowledge; or to be incorporated into existing supervision, competency, and performance review structures? Educational mate- rials may be refined through the use of formative evaluation feedback.

Relevant suggestions are provided via the Replicating Effective Programs (REP) framework, under its ‘packaging’ domain. Further support related to developing educational materi- als can be found on the Training Within Industry Service website: http://twi-institute.org/ training-within-industry

Make training dynamic

Vary the information delivery methods to cater to different learning styles and work contexts and shape the training to be interactive.

Making training dynamic includes efforts to divide material into small time intervals, the use of small group breakouts, audience response systems, and other measures, such as having learners try new skills between training sessions. Interactive components of training can be very dynamic with participants actively contributing to the training content, engaging in problem-solving, and identifying solutions that can be tested.

Distribute educational materials

Distribute educational materials (including guidelines, manuals and toolkits) in person, by mail, and/or electronically.

None

The VA QUERI Roadmap | PAGE 108

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

Category and Strategy

Definition Ancillary Lessons Material from ERIC Study and from the Agency for Healthcare

Research Quality EvidenceNOW Initiative

Use train- the-trainer strategies

Train designated clinicians or organizations to train others in the effective practice.

Restrictions regarding who can serve as a trainer are idiosyncratic to the change, for example, some effective practices may require that supervisors have specific levels of education, train- ing, or experience, and such restrictions should be explored in the planning phase. Train-the- trainer strategies may also apply to those responsible for administrative procedures, and who are part of implementing the effective practice.

Conduct educational meetings

Hold meetings targeted toward educating multiple stakeholder groups (i.e., clinicians, administrators, other organizational stake- holders, community members, consumers, families) about the effective practice and/ or its implementation.

The content of the education may include information regarding what to expect as imple- mentation moves forward. It may be useful to consider whether meeting attendees are relatively homogeneous or comprised of multiple roles and disciplines so that the education can be most effective in meeting stakeholder needs. For example, some educational meetings may inform the stakeholder group about the effective practice in a way intended to increase demand by including consumers and families, while others may preview technical facets of the effective practice for clinicians and administrators. It is often useful to have recordings or other materials from the educational meetings available to those who cannot attend the meetings and for individuals engaged or hired after the meeting(s).

Conduct educational outreach visits

Have a trained person meet with individu- als or teams in their work settings to edu- cate them about the effective practice with the intent of changing behavior to reliably use the effective practice as designed.

Visits to the site may be in person or virtual. Some initiatives may require regular educational outreach as part of maintaining behavioral changes required to ensure sustained, consistent use of the effective practice. Academic detailing is another commonly used term, although academic detailing typically involves many additional discrete implementation strategies (e.g., conduct ongoing training, modeling, developing and distributing educational materials).

Create a learning collaborative

Facilitate the formation of groups of clini- cians or clinician organizations and foster a collaborative learning environment to improve implementation of the effective practice.

There are several approaches to this in the literature including peer consultation networks, on- line communities of practice, quality circles, and learning collaboratives. Groups may meet in person or interact using a wide variety of media. The inclusion of a quality manager within the collaborative may be useful. Positive deviance approaches use “discovery and action dialogue” among peers to promote collaborative learning.

Resources specific to learning collaboratives include:

• The Agency for Healthcare Research Quality

• The Institute for Healthcare Improvement

Key terms for searching literature specific to collaborative learning include: learning commu- nity, learning network, and community of practice.

Create online learning communities

Create an online portal for clinical staff members to share and access resources, webinars, and frequently asked questions related to the specific effective practice and provide interactive features to encour- age learning across settings and teams (e.g., regular blogs, facilitated discussion boards, access to experts, and networking opportunities).

While shadowing traditionally has involved in-person observation, creative use of technology may provide additional opportunities for individuals to observe and learn from those experi- enced in the innovation.

Shadow other experts

Provide ways for key individuals to directly observe experienced people engage with or use the targeted effective practice.

While shadowing traditionally has involved in-person observation, creative use of technology may provide additional opportunities for individuals to observe and learn from those experi- enced in the effective practice.

Work with educational institutions

Encourage educational institutions to train clinicians in the effective practice.

This strategy fits well with effective practices requiring clinical training and other skills where training expertise is more likely to be housed in educational institutions.

The VA QUERI Roadmap | PAGE 109

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

Category and Strategy

Definition Ancillary Lessons Material from ERIC Study and from the Agency for Healthcare

Research Quality EvidenceNOW Initiative

Strategies to Provide Interactive Assistance

Implementation Facilitation

A multi-faceted process of enabling and supporting individuals, groups, and organizations in their efforts to adopt and incorporate effective practices into routine practices.

Effective practices that are the target of facilitation efforts can include a continuum of innova- tions ranging from evidence-based research evidence to incremental quality improvement efforts. Ideally, facilitation efforts are holistic and help to build capacity and skill within the setting to sustain ongoing effective practices. Facilitation can be provided by individuals who are internal or external to the setting within which Implementation Facilitation occurs. Implementation Facilitation is an interactive support process and draws on a combination of implementation strategies based on the needs of the setting. Implementation Facilitation is sometimes referred to as a “meta” strategy because, as an interactive support process, it often includes a combination of implementation strategies and bundles multiple strategies as needed.

Provide local technical assistance

Develop and use a system to deliver techni- cal assistance within local settings that is focused on implementation issues.

Local technical assistants can be local staff members or affiliated with a broader or central- ized network of technical assistants. The key is providing tailored technical assistance within settings where the effective practice is being implemented. Technical assistance for both the effective practice and the implementation processes may be important. For example, the VA aims to have mental health Evidence-Based Psychotherapy coordinators, Military Sexual Trauma coordinators, and OEF/OIF/OND coordinators in each facility who can provide techni- cal assistance to other local clinicians for relevant initiatives. This strategy may be supported by Centralize Technical Assistance, which may provide necessary infrastructure and support for Provide Local Technical Assistance.

Provide clinical supervision

Provide clinicians with ongoing supervision focusing on the effective practice. Provide training for clinical supervisors who will supervise clinicians who provide the effec- tive practice.

Clearly defining the role of supervision and providing ongoing resources to ensure that it occurs is important. Supervisor training often needs to include specific training in how to supervise the effective practice.

See Nadeem et al. (2013) for a discussion of the distinction between consultation and supervision.

Centralize technical assistance

Develop and use a centralized system to deliver technical assistance focused on implementation issues. This strategy may involve setting up a centralized technical assistance entity that may, in turn, support a network of technical assistants who work directly with local implementation sites. It may provide human-driven ad hoc support as with a helpdesk, from a central- ized location (online or phone) or may be automated e.g., online frequently asked questions or via online community with discussion boards.

This could be the designation of a lead technical assistance organization (could also be responsible for training). The lead technical assistance entity can develop other mechanisms (e.g., call-in lines or websites) in order to share information on how to best implement the ef- fective practice. This centralized function may support Provide Local Technical Assistance and enable local implementation sites to Use Data Experts.

Strategies to Support Clinicians

Facilitate relay of clinical data to clinicians

Provide as close to real-time data as pos- sible about key measures of process/out- comes using integrated modes/channels of communication in a way that promotes use of the targeted effective practice.

For recommendations regarding how to introduce effective practices or change of any kind into existing workflows, please see May (2013).

Remind clinicians

Develop reminder systems designed to help clinicians to recall information and/or prompt them to use the effective practice.

Reminders could be consumer- or encounter-specific, provided verbally, on paper, or elec- tronically. Computer-aided decision support and drug dosages are included in this strategy. Reminders may be delivered at various time points (prior to service, during service, or follow- ing service delivery).

The VA QUERI Roadmap | PAGE 110

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

Category and Strategy

Definition Ancillary Lessons Material from ERIC Study and from the Agency for Healthcare

Research Quality EvidenceNOW Initiative

Develop resource sharing agreements

Develop partnerships with organizations that have resources needed to implement the effective practice.

For example, this could involve data sharing agreements, agreements to share necessary equipment (e.g., telemedicine equipment), or sharing the cost of bringing in experts who provide training and consultation. Resource sharing agreements could involve formal memo- randums of understanding or be much more informal in nature.

Revise professional roles

Shift and revise roles among profession- als who provide care, and redesign job characteristics.

Revising professional roles includes the expansion of roles to cover provision of the effective practice and the elimination of service barriers to care, including personnel policies.

Create new clinical teams

Change who serves on the clinical team, adding different disciplines and differ- ent skills to make it more likely that the effective practice is delivered (or is more successfully delivered).

None

Strategies to Engage Consumers

Involve consumers and family members

Engage or include consumers and families in the implementation effort.

Feedback from stakeholders can be obtained at any stage of the implementation process depending on the needs and goals of project. Involving stakeholders in the pre-implementa- tion phase is advantageous for many effective practices. Training in the practice, and relevant advocacy, may also be included in stakeholder involvement. Informal caregivers such as neighbors, friends, and other key sources of support may also be prudent to include.

Intervene with consumers to enhance uptake and adherence

Develop strategies with consumers to encourage and problem solve around adherence.

This includes consumer reminders, outreach, and financial incentives to attend appointments. Feedback regarding consumers' understanding and use of the treatment is also important to collect.

Prepare consumers to be active participants

Prepare consumers to be active in their care, to ask questions, and specifically to inquire about care guidelines, the evidence behind clinical decisions, or about available effective practices.

Preparing consumers to inquire about specific practices can involve asking questions and educating consumers about the existence of treatments supported by evidence, as well as explicitly inviting them into the process of treatment decision-making.

Increase demand

Attempt to influence the market for the effective practice to increase competition intensity and to increase the maturity of the market.

One way of increasing demand is to educate consumers about the effective practice so that they demand it from their clinicians (e.g., what pharmaceutical companies do).

Use mass media Use media to reach large numbers of people to spread the word about the effective practice.

Mass media may include television, newspapers, magazines, radio, electronic social media, listservs, mass email campaigns, mass mailings, and robocalls as methods for spreading in- formation. Targets of these media campaigns may be clinicians, potential consumers, or their associates. Other commonly used terms include marketing or social marketing.

Strategies to Change Infrastructure

Mandate change

Have leadership declare the priority of the effective practice and their determination to have it implemented.

It is important to ensure that the individuals mandating the change have the power to do so, as implementers often lack such authority. Working with health system leaders to develop buy-in and lobby for a change mandate is often needed. It can also be important to inform other stakeholders (e.g., auditors, groups that review services for billing) about the mandate to ensure they are on the same page.

Change record systems

Change clinical documentation (e.g., elec- tronic medical records) systems to allow better assessment of implementation or clinical outcomes.

This strategy involves changing or upgrading the structure, content, function, or design of record system components. These systems most commonly involve the electronic health or medical records systems. These changes may include modifying the format of progress notes and treatment plans to reflect the effective practice being implemented. This strategy focuses on changes within the clinical setting. Please refer to Use Data Warehousing Techniques for activities that involve links to or integration with outside entities, repositories, or systems.

The VA QUERI Roadmap | PAGE 111

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

Category and Strategy

Definition Ancillary Lessons Material from ERIC Study and from the Agency for Healthcare

Research Quality EvidenceNOW Initiative

Change physical structure and equipment

Evaluate current configurations and adapt, as needed, the physical structure and/or equipment (e.g., changing the layout of a room, adding equipment) to best accom- modate the targeted effective practice.

None

Create or change credentialing and/or licensure standards

Create an organization that certifies clini- cians in the effective practice or encourage an existing organization to do so. Change governmental professional certification or licensure requirements to include deliver- ing the effective practice. Work to alter con- tinuing education requirements to shape professional practice toward the change.

None

Change service sites

Change the location of clinical service sites to increase access.

Changing service sites can include collocating different services to better implement complex effective practices that require multiple disciplines or services, telemedicine, or bringing the services to the client in their home, the community, or other clinically relevant settings, such as busy public spaces for a client with PTSD.

Change accreditation or membership requirements

Strive to alter accreditation standards so that they require or encourage use of the effective practice. Work to alter member- ship organization requirements so that those who want to affiliate with the orga- nization are encouraged or required to use the effective practice.

None

Start a dissemination organization

Identify or start a separate organization that is responsible for disseminating the effective practice. It could be a for-profit or nonprofit organization.

This strategy can address concerns (e.g., conflict of interest) for situations in which it is desir- able to have fidelity monitors that are independent from the care setting. The dissemina- tion organization could be a for-profit or nonprofit organization. The organization could be ‘licensed’ by a university if the effective practice was born within an academic setting. It is important for dissemination organizations to be aware of organizations' approaches to imple- menting other effective practices in order to build upon existing knowledge and practices.

Change liability laws

Participate in liability reform efforts that make clinicians more willing to deliver the effective practice.

Liability reform can also make clinicians less willing to deliver alternatives to the effective practice.

Strategies to Utilize Financial Strategies

Fund and contract (and/ or Negotiate) with Vendors for the clinical innovation

Governments and other payers of services issue requests for proposals to deliver the effective practice, use contracting pro- cesses to motivate clinicians to deliver the effective practice, and develop new fund- ing formulas that make it more likely that clinicians will deliver the effective practice.

None

Access new funding

Access new or existing money to facilitate the implementation.

Accessing new funding sources could involve new uses of existing money, accessing block grants, shifting funding from one program to another, cost-sharing, passing new taxes, raising private funds, or applying for grants. These monies may be used to fund the delivery of an effective practice, or to support other time limited actions needed for initial implementation, such as to purchase material or logistical support, training, and consultations.

Place innovation on fee-for-service lists/formularies

Work to place the effective practice on lists of actions for which clinicians can be reimbursed (e.g., a drug is placed on a for- mulary, a procedure is now reimbursable).

The VA made Naloxone kits available on all formularies free of charge to consumers and facilities as part of Opiate Education and Naloxone Distribution program to prevent opiate overdose mortality.

The VA QUERI Roadmap | PAGE 112

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

Category and Strategy

Definition Ancillary Lessons Material from ERIC Study and from the Agency for Healthcare

Research Quality EvidenceNOW Initiative

Alter incentive/ allowance structures

Work to incentivize the adoption and implementation of the effective practice.

Incentives may be based on the performance of individual clinicians or larger performance units at the organizational level. The incentive could be in the form of an increased rate of pay to cover the incremental costs associated with implementing the effective practice. The incen- tive could be through loan reduction or forgiveness to clinicians to learn the new practice. This category of financial strategies also includes the elimination of any perverse incentives that become a barrier to receiving appropriate care. An incentive suggests the payment is tied to performing a clinical action or improving outcomes. An allowance suggests that the clini- cian or organization is not required to perform the clinical action or meet the performance standard.

Make billing easier

Make it easier to bill for the effective practice.

Making billing easier might involve requiring less documentation, ‘block’ funding for deliver- ing the effective practice, and creating new billing codes. Developing progress note templates to facilitate documentation of the effective practice can also decrease the burden for obtain- ing payment.

Alter consumer fees

Create fee structures where consumers pay less for preferred treatments (the effec- tive practice) and more for less-preferred treatments.

None

Use other payment schemes

Introduce payment approaches (in a catch- all category).

Payment scheme approaches may involve prepayment and prospective payment for service, clinician salaried service, the alignment of payment rates with the attainment of consumer outcomes, and the removal or alteration of billing limits, such as numbers of encounters that are reimbursable. Payment may also be based on measures of treatment fidelity. Payment schemes are implementation strategies to the degree that they free the clinician’s time to provide the effective practice. Others’ strategies motivate clinicians to provide better service.

Develop disincentives

Provide financial disincentives for failure to implement or use the effective practice.

In addition to direct financial disincentives, this strategy could include tying promotion deci- sions to the use of certain practices.

Use capitated payments

Pay clinicians or care systems a set amount per consumer for delivering clinical care.

This is an implementation strategy to the degree that it frees the clinician to provide services that they may have been disincentivized to provide under a fee-for-service structure. This may be helpful to motivate clinicians to use certain effective practices. These changes often come about as part of policy changes that alter fee structures, alter coverage, or add items to reimbursement formularies.

References:

1. Perry CK, Damschroder LJ, Hemler JR, Woodson TT, Ono SS, Cohen DJ. Specifying and comparing implementation strategies across seven large implementation interventions: a practical application of theory. Implement Sci. 2019;14(1):32.

2. Powell BJ, Waltz TJ, Chinman MJ, et al. A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implement Sci. 2015;10:21.

3. Waltz TJ, Powell BJ, Matthieu MM, et al. Use of concept mapping to characterize relationships among implementation strategies and as- sess their feasibility and importance: results from the Expert Recommendations for Implementing Change (ERIC) study. Implement Sci. 2015;10:109.

The VA QUERI Roadmap | PAGE 113

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

Evaluation and Study Designs for Implementation and Quality Improvement

Figure 18. Illustration of a Generic Randomized Controlled Trial Design at the Patient-Level of Analysis

Pre-Intervention Post-Intervention

Impact

E� ec

t

Intervention Control

Intervention

The following is a general overview on the use of common study designs for implementation and quality improvement initiatives. For further information additional details including examples and sug- gested references are also provided.

Experimental and quasi-experimental designs are two broad cat- egories of evaluation designs used to rigorously and systematically assess the implementation of effective practices. They are also used to compare different implementation strategies. Experimental designs randomly assign participants, which could be consumers, clinicians, health care clinics or sites, or other types of organizational health care units (hospitals) to one or more intervention groups or a control group (standard of care) to test the causal impact of these conditions on desired outcomes (Figure 18). Interventions in these examples can include a clinical intervention or effective practice, as well as a quality improvement intervention or implementation strategy (e.g., methods to help clinicians deploy effective practices).

Randomization helps minimize selection bias because it ensures that the groups being compared are similar initially. It can also mitigate the influence of measured and unmeasured variables that might be associated with the outcome of interest (confounding variables**) so that evaluators can feel confident that differences between groups can be attributed to the intervention.

Randomization also ensures participants receive an equal opportunity to be allocated to the intervention versus con- trol group. In contrast, quasi- experimental designs, including observational designs, evaluate interventions without using randomization when randomization is either not feasible or ethical to employ. For a high-level review of the issues in deciding when to randomize or not randomize from the perspective of evaluation and implementation, see:

1. Julnes G, Mark M, Shipman S. Conditions to Consider in the Use of Randomized Experimental Designs in Evaluation. Available at: https://static1.squarespace.com/ static/56534df0e4b0c2babdb6644d/t/5f5a338a67d71a06db27 5fe9/1599746955716/Conditions+to+consider+for+randomized +experimental+designs.pdf. Data Foundation. June 25, 20202. Accessed September 18, 2020

2. Finkelstein A. A strategy for improving U.S. health care delivery - conducting more randomized, controlled trials. N Engl J Med. 2020;382(16):1485-1488.

**Confounding Variables: Systematic differences between intervention and control groups that obscure the effect of an intervention https://www.ncbi.nlm.nih.gov/pmc/ articles/PMC4024462/

The VA QUERI Roadmap | PAGE 114

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

Designs in Implementation Science A fundamental goal of implementation science is to reduce the time it takes for an effective practice to go from testing in a clinical trial to use in real world health care settings. In a randomized-controlled clinical effectiveness study, the focus is on measuring the effectiveness, or benefit, that an intervention/practice provides consumers when used in routine clinical practice settings. The assessment focus is on health outcomes like consumer symptoms, behaviors, function, survival, or satisfaction with care. Thus, consumers are the unit of analysis and randomization in effectiveness trials, not the people or organizations delivering the clinical practice.

Effectiveness trials can range from those with tightly controlled proce- dures under ideal conditions to explain the benefit of an intervention, to highly pragmatic trials that test the benefit of the intervention in the real-world settings where the effective practice is meant to be used. Tools like the Pragmatic-Explanatory Continuum Indicator Summary-2 (PRECIS-2) can help evaluators understand the degree to which key aspects of an intervention protocol are for real world use. Note, pragmatic effectiveness trials are the domain of research studies whereas the QUERI Roadmap focuses on the evaluation of non-re- search implementation and quality improvement activities.

Notably, randomized implementation designs aim to assess the im- pact of different implementation strategies on the uptake of effective practices. The units of analysis and randomization in implementa- tion studies are at the clinician, clinical unit, facility, or system level in which the implementation strategy is designed to change clinical care processes, workflows, behaviors, or decision-making. Implementation outcomes often reflect process or quality measures of practice uptake such as acceptability, adoption, appropriateness, feasibility, fidelity, penetration, and sustainability.

Suggested Readings 3. Coly A, Parry G. Evaluating Complex Health Interventions: A Guide

to Rigorous Research Designs. Washington, D.C.: AcademyHealth; June 2017. Available at: https://www.academyhealth.org/ evaluationguide

4. Landsverk J, Hendricks-Brown C, Smith JD, et al. Design and analysis in dissemination and implementation research. In: Brownson RC, Colditz GA, Proctor EK, eds. Dissemination and Implementation Research in Health. 2nd ed. New York: Oxford University Press; 2018:201-227.

5. Miller CJ, Smith SN, Pugatch M. Experimental and quasi- experimental designs in implementation research. Psychiatry Res. 2020;283:112452.

6. Wholey J, Hatry HP, Newcomer K, eds. Handbook of Practical Program Evaluation. San Francisco: Jossey-Bass, Inc; 2004.

7. Lewis, E., Baumann, A., Gerke, D., Tabak, R., Ramsey, A., Small, S. & Proctor, E. D&I Research Designs. St. Louis, MO: Washington University; 2017 July. Eight toolkits related to Dissemination and Implementation. Available from https://sites.wustl.edu/wudandi

8. Aarons. G. Module 5. Designs for Implementation Studies. National Cancer Institute, open access Training Institute for Dissemination and Implementation Research (TIDIRC) curriculum; July 2019. Available at: https://cancercontrol.cancer.gov/IS/training- education/tidirc/openaccess-module5.html

9. University of Washington Implementation Science Resource Hub. Select Study Design. Available at: https://impsciuw.org/ implementation-science/research/designing-is-research/

Hybrid Effectiveness-Implementation Designs

Types of Hybrids

Clinical E�ectiveness

Research

Hybrid Type 1 Hybrid Type 2 Hybrid Type 3

The Continuum

Implementation Research

Primary Aim: Determine e�ectiveness of a clinical intervention Secondary Aim: Better understand the context for implementation (i.e., potential barriers and facilitating factors)

Primary Aim: Determine e�ectiveness of a clinical intervention Co-Primary Aim: Determine feasibility and/or (potential) impact of an implementation strategy

Primary Aim: Determine impact of an implementation strategy Secondary Aim: Assess clinical outcomes associated with implementation trial

Adapted from: Curran, Bauer, Mittman et al. 2012 Med Care.

To begin understanding the “implementability” of a promis- ing clinical practice earlier in the research pipeline, imple- mentation scientists developed a continuum of “hybrid” effectiveness-implementation designs in which the focus of the research questions changes as the trial becomes more pragmatic. This design continuum combines the best design elements of traditional effectiveness trials with those of implementation research trials to help evaluators to consider factors that affect how an effective practice can be imple- mented at the end of the clinical research pipeline.

For example, in a hybrid 1 design, the primary aim is to evaluate an intervention’s effectiveness with a secondary emphasis on assessing barriers and facilitators to successful practice uptake. The primary focus of a hybrid type 2 study is to assess both the effectiveness of the clinical practice and its implementation strategy. Hybrid type 2 designs are often used to test an implementation strategy in a few sites. In comparison, the primary goal of a hybrid 3 design is to assess the effectiveness of one or more implementation strategies used to get an effective practice into health care settings while also monitoring to ensure that clinical effectiveness of the practice is maintained.

Figure 19. Examples of Evaluation Aims by Hybrid Effectiveness-Implementation Design Type

The VA QUERI Roadmap | PAGE 115

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

Evaluations randomized at the patient level are generally considered effectiveness studies or studies assessing the effectiveness of both the clinical practice and implementation strategy (hybrid type 1 or 2, re- spectively). In contrast, hybrid type 3 designs typically involve multiple sites which are allocated different implementation strategies or imple- mentation strategy versus no implementation strategy to promote the uptake of an effective practice. The primary outcome of a hybrid type 3 design is the uptake of the effective practice at the clinician or site level, and secondary outcomes are consumer-level outcomes.

Suggested Readings 1. Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C.

Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact. Med Care. 2012;50(3):217-226.

2. Loudon K, Treweek S, Sullivan F, Donnan P, Thorpe KE, Zwarenstein M. The PRECIS-2 tool: designing trials that are fit for purpose. BMJ. 2015;350:h2147.

3. Landes SJ, McBain SA, Curran GM. An introduction to effectiveness-implementation hybrid designs. Psychiatry Res. 2019;280:112513.

4. Curran, GM. An Overview of Hybrid Effectiveness-Implementation Designs. VA HS&R&D Cyberseminar. April 2, 2020. Available at: https://www.hsrd.research.va.gov/for_researchers/cyber_ seminars/archives/video_archive.cfm?SessionID=3756

5. Burke RE, Shojania KG. Rigorous evaluations of evolving interventions: can we have our cake and eat it too? BMJ Qual Saf. 2018;27(4):254-257.

Experimental or Randomized Designs As noted previously in the QUERI Roadmap, non-research implementa- tion initiatives and quality improvement projects can still use rigor- ous, randomized designs to enable health care leaders to understand which implementation strategies are more effective to bring a new promising practice into widespread use.

• Cluster randomized trials • Stepped Wedge • Factorial and Sequential, multiple-assignment randomized trials

(SMART)

Cluster or group-randomized trials The two primary types of rigorous experimental designs used in implementation and quality improvement evaluations, are the parallel cluster-randomized and cluster-randomized stepped wedge designs. These trials are referred to as cluster- or group randomized controlled trials in which the unit of randomization is a cluster or group, and outcome measures are obtained from members of the cluster.

These comparative designs are preferable for implementation evalu- ations which aim to compare different implementation strategies designed to cause changes in “clusters” or groups at the social, physi- cal, or environmental level to health care outcomes . These designs randomize centers (hospitals, community sites) or units (clinician groups, departments) rather than individuals, which helps to avoid some of the contamination that might occur when randomization oc- curs within settings at the individual-level.

The goal of randomization in cluster randomized trials is to achieve balance of baseline covariates among the individuals who comprise a cluster or among other cluster characteristics. In addition, evaluators seek to achieve balance in cluster sample size at baseline randomiza- tion. Since both forms of balance play a role in the sample size and power calculations required to design cluster randomized trial designs, we will present three methods that help achieve these aims later in this appendix.

Allocation to an implementation strategy may occur via simple or restricted randomization. Simple randomization is akin to a coin flip

and is more likely to lead to balance in baseline covariates in larger sample sizes. If the number of units being randomized is small, simple randomization can lead to imbalance in unit characteristics by chance. Restricted randomization allocates treatments within constraints and includes strategies such as stratification, matching, minimization, and covariate-constrained randomization.

Stratification includes simple randomization within separate sub- groups of important confounders (e.g., urban vs rural) — this is an especially useful strategy when there are few units in one stratum and evaluators want to ensure that both treatment and control conditions are well-represented in small strata. In matching, one identifies pairs of units with identical values of specific baseline covariates and randomly assigns one member of each pair to the treatment condition.

Matching leads to precise balance in baseline covariates, but if one member of a pair drops out of the study, the other member also must be dropped to retain balance. When there are several confounders that one wishes to balance, minimization and covariate constrained randomization may be considered. For more information, see Sug- gested Readings at the end of this section.

In the AcademyHealth’s Evaluating Complex Health Interventions: A Guide to Rigorous Research Designs, evaluators of randomized evalu- ations are encouraged to report variations in the nature and size of the effects across clusters, not just “average” effects in the study, to enhance learning from variation.

The VA QUERI Roadmap | PAGE 116

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

Some practical considerations for choosing between these designs include:

• Equipoise – the degree to which there is genuine uncertainty among clinicians that there is not one “better” implementation strategy present (for either the control or experimental group) during the design of a cluster randomized controlled trial (RCT).

• Feasibility – practical considerations for conducting design based on the costs or personnel needed to carry out and collect adequate data for a specific design given funding support, stakeholder expectations, timeline to carryout, ethical considerations of withholding a treatment intervention, and other logistical considerations dictated by a setting, priority population of patients, and the nature of the implementation strategies to be tested.

• Statistical power – the ability to detect a meaningful effect of an implementation strategy on an outcome, given the number and size of clusters included in the trial and the outcome of

interest. Recent publications have reported that the relative power of the stepped wedge design and parallel RCT depend on the value of the intracluster correlations (the degree to which individuals within a cluster resemble each other in terms of the outcome of interest). Stepped wedge cluster RCTs are likely to be more efficient than parallel cluster RCTs when intracluster correlation is high.

Suggested Readings 1. Ivers, N.M., Halperin, I.J., Barnsley, J. et al. Allocation

techniques for balance at baseline in cluster randomized trials: a methodological review. Trials 13, 120 (2012). Available at: https://trialsjournal.biomedcentral.com/ articles/10.1186/1745-6215-13-120

2. Turner EL, Li F, Gallis JA, Prague M, Murray DM. Review of Recent Methodological Developments in Group-Randomized Trials: Part 1-Design. Am J Public Health. 2017;107(6):907-915.

Parallel cluster- or group-randomized design A parallel cluster randomized design compares two or more imple- mentation strategy conditions and clusters are randomly assigned to these two strategies, implementation strategies are deployed, and then the results are compared as shown in Figure 20 below. Random- ization is based on assigning implementation treatment conditions.

Figure 20. A Generic Parallel Cluster Design

Time Period

C lu

st er

(c lin

ic s)

1-3

1 2 3 4 5 6 7 8 9 10

4-6

7-9

10-12

Site allocated to Implementation Strategy B

Site not allocated to implementation strategy B (standard implementation as usual)

In some cases, a new implementation strategy may be compared to a standard implementation as usual to bring the same effective prac- tice into use such that some clusters receive an innovative implemen- tation strategy to deploy the effective practice while other clusters employ a standard method of bringing effective practice into use over a similar time period. Process and output measures used as the primary end points are measured for all eligible patients or subjects in both conditions and aggregated to the level of the randomized unit at the cluster level. The goal of this basic design is to understand whether a new implementation strategy results in better or more efficient implementation processes and outcomes compared to the methods

currently employed.

An alternative parallel design is a head-to-head cluster randomized implementation trial that enables the comparison of two distinctly different implementation strategies to determine which is more suc- cessful in implementing an effective practice. In this design, the same effective clinical practice is used for both arms of the trial, and clusters are assigned randomly to one of the two different implementation strategies as shown in Figure 21. Both implementation strategies are manualized and carried out with equivalent attention to fidelity. Addi- tionally, both implementation strategies are compared on the quality, quantity, or speed of implementing the effective practice.

Figure 21. Example of a Generic Head-to Head Parallel Cluster- Randomized Implementation Trial

Health units randomized

Implementation Strategy 1

Implementation Strategy 2

Program Delivery

E�ective Practice

E�ective Practice

Program Delivery

Adapted from Brown CH, Curran G, Palinkas LA, et al. 2017.

Stepped-wedge cluster randomized design A stepped wedge cluster randomized design is a one-directional crossover experimental design in which time is divided into multiple phases enabling clusters to cross over from a routine implementation or control condition to an implementation condition so that all clus- ters eventually receive the same implementation strategy. As shown in Figure 22, the initial phase begins with no clusters receiving the

enhanced implementation strategy and subsequent phases during which one or more clusters are randomized to receive the implemen- tation strategy at regular pre-specified intervals or steps. By the end of the study, all clusters will have randomized to the intervention group. There should be an expectation that the benefits of the intervention exceed the potential harm.

The VA QUERI Roadmap | PAGE 117

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

Figure 22. Stepped-Wedge Cluster Randomized Design Time Period

C lu

st er

(c lin

ic s)

1-3

1 2 3 4 5 6 7 8 9 10

4-6

7-9

10-12

13-15

16-18

19-21

21-23

Site allocated to Implementation Strategy B

Site not allocated to implementation strategy B (standard implementation as usual)

24-26

Adapted from Coly and Parry, 2017.

Evaluators are strongly encouraged to ensure that more than one cluster be assigned to the treatment condition at each time period. Otherwise, it can be difficult to separate out time and site characteris- tics’ impact on the outcome. This is a relevant concern when there is a possibility that treatment effects vary with time.

This design is ideal for situations where operational stakeholders all want the enhanced implementation strategy, it might be perceived as unethical for a cluster to not receive the enhanced implementation

strategy, and all participating health units cannot receive the en- hanced implementation strategy at the same time due to logistical, practical or financial consideration.

A randomized stepped wedge design permits within-cluster effects to be evaluated. Each cluster receives the implementation strategy intervention while also serving as its own control. If there is a signifi- cant cluster effect, this increases the power of the evaluation to detect differences compared to a design where clusters are assigned to only a control or intervention group. Stepped wedge designs also support the iterative evaluation of barriers to the implementation of the inter- vention which can help improve implementation in subsequent steps or phases of the implementation rollout.

A stepped wedge design can have downsides, especially if rapid re- sults are important to an operational partner because this design can take longer than conventional cluster randomized trials, particularly if only one or a few clusters can be allocated to an enhanced implemen- tation strategy at each step. Longer study duration can require more resources and results in higher costs. This design also requires frequent data collection which can be challenging from a measurement per- spective if key outcomes cannot be accessed through administrative databases and primary data collected is needed. Finally, evaluators must control for secular trends in key outcomes and avoid contamina- tion between those receiving the implementation intervention and those waiting to receive the intervention.

Factorial Designs, including Sequential Multiple Assignment Randomized Trials (SMARTs) Given known and well-documented delays in widespread adoption of effective practices, implementation trials that focus on improv- ing uptake or adoption of a particular effective practice commonly forego traditional randomized comparisons between “treatment” and “control” arms. Instead, the focus of these trials is on the comparative effectiveness of different implementation strategies or sets of imple- mentation strategies. Additionally, implementation trials often involve testing interventions or implementation strategies that are multi- component, and questions may revolve around determining which components are most effective.

As such, trial designs that relate to optimization, or determining the

most efficient provision of limited resources for the largest public benefit, are often of interest to implementation scientists. The Multi- phase Optimization Strategy (MOST) borrows from two engineering principles—resource management and continuous optimization—to provide a framework for building and evaluating multicomponent interventions. While there are a number of different experimental methods that can inform optimization within the MOST framework, factorial designs are the most common. Sequential multiple assign- ment randomized trial (SMART) designs are a variant of factorial designs that inform sequential optimization of components.

Factorial designs Factorial designs allow for “screening” of active or important interven- tion components in multi-component interventions. Factorial designs randomize participants (individuals, clinicians, sites) to different intervention components separately to allow for direct evaluation of different intervention components (e.g., each of several implementa- tion strategies bundled together). This includes both the main effects of and interactions between different components. This process allows investigators to “screen” or identify key “active” components (or combinations of components) for optimizing the intervention pack- age prior to evaluating its effectiveness. As multiple components can be screened within a single factorial experiment, a factorial design is more efficient than conducting multiple randomized controls to exam- ine each component individually.

In factorial designs, the levels of two or more intervention compo- nents (or implementation strategies) are “crossed” so that all possible combinations of each component are implemented. Table 26 shows a simple example of a factorial design combining two different imple- mentation strategies—the presence/absence of Strategy A (on rows) and Low vs. High intensity versions of Strategy B (on columns).

Table 26. Illustration of a 2x2 Factorial Design Comparing Two Implementation Strategies

Strategy A Strategy B Condition 1 Not Present Low intensity Condition 2 Not Present High intensity Condition 3 Present Low intensity Condition 4 Present High intensity

The VA QUERI Roadmap | PAGE 118

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

For example, as shown in Figure 23, Strategy A could represent the presence of a virtual learning collaborative for clinicians to share best practices to adopt a new practice whereas Strategy B could represent high vs. low intensity of quality improvement coaching by

experts from a clinical program office. The factorial design thus results in four different experimental conditions to which participants are randomized.

Figure 23. Illustration of a Factorial Design to Compare Implementation Strategies

Quality Improvement Coaching

Virtual Learning

Collaborative

Low Intensity

Yes

No

High Intensity

Virtual Learning Collaborative + Low Intensity QI Coaching

Virtual Learning Collaborative + High Intensity QI Coaching

Low Intensity QI Coaching Only

High Intensity QI Coaching Only

As participants are randomized to the different factors (or strategies) independently, factorial designs allow for evaluation of main effects of each intervention by comparing different experimental conditions. For example:

• Strategy A: the effect of receiving access to the learning collaborative vs. not can be evaluated by comparing Conditions 3+4 with Conditions 1+2.

• Strategy B: the effect of receiving high intensity vs. low intensity expert coaching can be evaluated by comparing Conditions 2+4 with Conditions 1+3.

As both main effects can be evaluated within the context of a single factorial study without needing to increase sample size, factorial designs offer an advantage over other study designs that might test components separately. Interaction effects can also be tested—for example, whether high intensity coaching is more effective when the learning collaborative is also provided—although as with any study, power for detecting interaction effects will be lower than power to detect the factor main effects.

Suggested Readings: 1. Collins LM, Murphy SA, Strecher V. The multiphase optimization

strategy (MOST) and the sequential multiple assignment randomized trial (SMART): new methods for more potent eHealth interventions. Am J Prev Med. 2007;32(5 Suppl):S112-118.

2. Collins LM. Optimization of Behavioral, Biobehavioral, and Biomedical Interventions: The Multiphase Optimization Strategy (MOST). New York, NY: Springer International Publishing; 2018.

3. Gallis JA, Bennett GG, Steinberg DM, Askew S, Turner EL. Randomization procedures for multicomponent behavioral intervention factorial trials in the multiphase optimization strategy framework: challenges and recommendations. Transl Behav Med. 2019;9(6):1047-1056.

4. Spring B, Pfammatter AF, Marchese SH, et al. A Factorial Experiment to Optimize Remotely Delivered Behavioral Treatment for Obesity: Results of the Opt-IN Study. Obesity. 2020;4(5):e16297

5. Quanbeck A, Almirall D, Jacobson N, et al. The Balanced Opioid Initiative: protocol for a clustered, sequential, multiple- assignment randomized trial to construct an adaptive implementation strategy to improve guideline-concordant opioid prescribing in primary care. Implement Sci. 2020;15(1):26.

Sequential Multiple-Assignment Randomized Trials Sequential multiple-assignment randomized trials (SMARTs) are a variant of a factorial design that can be useful for implementation researchers that are interested in optimizing sequences of implemen- tation support that adapt to ongoing to needs and/or implementation success over time. Adaptive interventions, or interventions that adapt the type, intensity and/or duration of treatment to participant needs as they change over time, are often appealing to implementation researchers as there is recognition that implementation strategies are not necessarily “one size fits all” and that some sites (or individuals or clinicians) may need more, less or different implementation support than others. Additionally, adaptive interventions allow for accommoda- tion of heterogeneity in treatment—for example, increasing or chang- ing implementation support for participants that are not responding (or implementing) under an initial form of implementation support.

SMARTs are multistage randomized trials in which some or all partici- pants (who may be sites, clinicians, and/or patients depending on the target of the intervention or implementation strategy) are randomized

more than once, often based on ongoing information (e.g., treatment response, adherence to and/or success under prior implementation support).

For scientists interested in providing better implementation support or testing the comparative effectiveness of different forms of imple- mentation support, SMARTs allow for direct comparisons of different sequences of implementation support to maximize EBP uptake and/ or improvement in downstream clinical outcomes. As such, SMART de- signs are well-suited to answering questions related to what forms of implementation support should be used and in what order to achieve the best results.

A variety of SMART designs are possible, depending on current sci- ence; however, one popular SMART design randomizes all participants (patients, clinicians, sites) upfront to one of two possible first-line treat- ments or strategies. After a certain period of time, those participants that do not show sufficient “response” to the first-line strategy (e.g., by

The VA QUERI Roadmap | PAGE 119

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

not exhibiting a certain amount of adoption, lacking adherence or en- gagement to their first-stage strategy, or exhibiting barriers that their first-stage treatment is not capable of addressing) are re-randomized to, for example, either continue that first-line strategy or augment

their first-line strategy with a different strategy.

One example of this hypothetical design is shown in Figure 24.

Figure 24. Hypothetical SMART Design

Strategy A

Continue

Continue

Add Strategy B

Strategy B

FIRST-STAGE INTERVENTION

INTERMEDIATE OUTCOME

SECOND-STAGE INTERVENTION

EXPERIMENTAL CONDITIONS

R

R

Subgroups

A

B

C

D

E

F

Study Start Month 6 End of study

Response

Non-Response

Continue

Continue

Add Strategy A R

Response

Non-Response

In this design, all participants—let’s imagine they are clinicians--are randomized at study start to receive either Implementation Strategy A or Implementation Strategy B to support implementation of an EBP. After six months, an intermediate evaluation of each clinician’s response to their first-line treatment is conducted, using a pre-defined definition of response vs. non-response (e.g., a responder is a clinician that delivered the EBP to at least 10 patients). Clinicians that were considered responsive to their first-line implementation support are not re-randomized, and instead having their current implementation support continued (Cells A & D). However, clinicians that failed to meet the response threshold are rerandomized to either continue with their current form of implementation support (Cells B & E) or to add the other form of implementation support (Cells C & F).

In addition to their being a large variety of ways to design SMART trials, SMARTs also allow for a number of different possible primary and secondary research questions and related analyses. While a full description of these analyses is outside the scope of this resource, three common primary aims for SMART trials and their basic analytical strategy are:

• Determination of the best first-stage treatment: This question asks which first-stage strategy results in the best end- of-study results, marginalizing over what happens in the second stage, e.g., is it better to start with Strategy A or Strategy B?

- In Figure 24, this would be evaluated by comparing the participants who received Strategy A (experimental conditions A+B+C) with those that received Strategy B (conditions D+E+F). In terms of power, this is equivalent to analyzing a two-arm trial.

• Determination of the best second-stage treatment for non- responders: This question asks, among participants that are deemed non-responsive at month 6 and re-randomized, is it better to continue them on their first treatment or augment with the other strategy?

- In Figure 24, this would be evaluated by comparing end-of- study outcomes for the non-responsive participants who were randomized to continue their first stage treatment (conditions B+E) with the non-responsive participants randomized to add the second strategy (conditions C+F). This is equivalent to analyzing a two-arm trial among participants that were non- responsive after the first stage.

• Comparing two different embedded sequences of treatment or adaptive interventions: Embedded in every SMART are multiple sequences of treatment. When those sequences adapt based on time-varying information (e.g., response status after six months), they are then considered adaptive interventions. Figure 24 contains four different sequences of treatment, two of which (#2 and #4 below) are adaptive: 1. Begin with Strategy A; after six months, continue Strategy A

for all participants (Conditions A+B); 2. Begin with Strategy A; after six months, continue Strategy

A for responders, and augment with Strategy B for non- responders (Conditions A+C);

3. Begin with Strategy B; after six months, continue Strategy B for all participants (Conditions D+E);

4. Begin with Strategy B; after six months, continue Strategy B for responders, and augment with Strategy A for non- responders (Conditions D+F).

The VA QUERI Roadmap | PAGE 120

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

A primary aim that compares different embedded sequences of treatment, thus, might compare sequence #1 with sequence #4 on end-of-study outcomes. This analysis, while similar to a two-arm trial, does require a bit of additional work to account for the fact that some participants contribute to multiple treatment sequences. Details on these analytic methods, as well as associated power calculations, can be found in the references below.

Suggested Readings 1. Almirall D, Compton SN, Gunlicks-Stoessel M, Duan N,

Murphy SA. Designing a pilot sequential multiple assignment randomized trial for developing an adaptive treatment strategy. Stat Med. 2012;31(17):1887-1902.

2. Lei H, Nahum-Shani I, Lynch K, Oslin D, Murphy SA. A “SMART” design for building individualized treatment sequences. Annu Rev Clin Psychol. 2012;8:21-48.

3. Nahum-Shani I, Qian M, Almirall D, et al. Experimental design

and primary data analysis methods for comparing adaptive interventions. Psychol Methods. 2012;17(4):457-477.

4. Collins LM, Nahum-Shani I, Almirall D. Optimization of behavioral dynamic treatment regimens based on the sequential, multiple assignment, randomized trial (SMART). Clin Trials. 2014;11(4):426-434.

5. Kilbourne AM, Almirall D, Eisenberg D, et al. Protocol: Adaptive Implementation of Effective Programs Trial (ADEPT): cluster randomized SMART trial comparing a standard versus enhanced implementation strategy to improve outcomes of a mood disorders program. Implement Sci. 2014;9:132.

6. NeCamp T, Kilbourne A, Almirall D. Comparing cluster-level dynamic treatment regimens using sequential, multiple assignment, randomized trials: Regression estimation and sample size considerations. Stat Methods Med Res. 2017;26(4):1572-1589.

Quasi-experimental Designs Quasi-experimental designs test causal hypotheses like experimental designs but lack random assignment of clusters to intervention arms. Quasi-experimental designs seek to identify a comparison group or time period that is as similar as possible to the enhanced imple- mentation treatment group in terms of baseline (pre-intervention) characteristics.

• Non-equivalent control group design • Stepped wedge design (without randomization) • Interrupted time series • Regression discontinuity design

Pre-post with non-equivalent control group In this type of evaluation as shown in Figure 25, comparisons are made between health care units receiving an implementation strategy condition versus those not receiving the strategy which serve as a con- trol or implementation as usual group. This is a design well-suited for testing a quality improvement innovation or implementation strategy with a smaller number of sites because the design is easier to enact without the cost and complexity of designs requiring more frequent collection of data over time.

Analyses in such designs aim to determine the difference in the

amount of change over time in clinical and implementation outcomes of interest between groups starting with pre-intervention and moving forward to post-intervention time periods. It is important for study and control sites to be comparable, and a number of statistical methods can help control for confounding variables and secular trends that may occur at the health care unit level of analysis as well as among the patients nested within these units. Similarly, pre- and post- implemen- tation intervention periods should be the same across implementation intervention and comparison control sites.

Figure 25. Example of a Pre-Post Design with Non-Equivalent Control Group

Follow-up time >>>Baseline

Cluster site acting as control site

Cluster site receiving intervention

Unexposed individuals

Individuals receiving intervention

Nonequivalent control group

The VA QUERI Roadmap | PAGE 121

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

Stepped Wedge Design As previously introduced above, a stepped-wedge evaluation is a type of rollout design in which sites are systematically assigned to an implementation strategy at specific time points by the evaluation team (Figure 26). Like its randomized cousin, this design allows for all sites to receive the implementation strategy used to deploy an effec- tive practice. Repeated measurement at each phase of assigning sites/ clusters to the implementation strategy condition enables each site to serve as its own control (within site comparisons) while also enabling between site comparisons.

This design can be useful to employ when random assignment of sites/ clusters may not be feasible due to resource constraints or readiness of sites to employ an implementation strategy. As in a randomized stepped wedge design, evaluators are strongly encouraged to ensure that more than one cluster be assigned to the treatment condition at each time period. Otherwise, it can be difficult to separate out time and site characteristics’ impact on the outcome. This is a relevant con- cern when there is a possibility that treatment effects vary with time.

Figure 26. Stepped Wedge Design

Time Period

C lu

st er

(c lin

ic s)

1-3

1 2 3 4 5 6 7 8 9 10

4-6

7-9

10-12

13-15

16-18

19-21

21-23

Site allocated to Implementation Strategy B

Site not allocated to implementation strategy B (standard implementation as usual)

24-26

Interrupted Time Series Design Interrupted time series evaluations are a type of quasi-experimental design that is useful in determining whether a new clinical practice or implementation strategy was effective based on time trend analyses before and after the implementation of a change. This design is often used when it is not possible to randomly assign units such as in the case of implementing a new healthy policy, regulation, or clinical service at a specific point in time.

An interrupted time series design requires data collection at mul- tiple periods of time prior to, during and after the introduction of an implementation initiative which may not be feasible or affordable for some evaluation budgets. When circumstances permit adequate

collection of data over time such as data tracked in electronic patient care databases, analysts can perform relatively simple pre-post test comparisons, and can allow for these comparisons to be adjusted for potential secular trends in the data before and after the introduc- tion of the intervention (Figure 27). When adequate data is available, evaluators can determine how long it takes for an intervention to result in measurable improvements in care as well as the impact on the consumer population targeted for care by the policy or practice. This design can be enhanced by including a comparison group that does not experience the implementation of a change (a comparative or controlled time series analysis ).

Figure 27. Illustration of a Generic Interrupted Time Series Design

Intervention introduced

Change in level

Slope post

Slope pre

Time

O u

tc o

m e

Suggested Readings 1. Coly A, Parry G. Evaluating Complex Health Interventions: A Guide

to Rigorous Research Designs. Washington, D.C.: AcademyHealth; June 2017. Available at: https://www.academyhealth.org/ evaluationguide

2. Cruz M, Bender M, Ombao H. A robust interrupted time series model for analyzing complex health care intervention data. Stat Med. 2017;36(29):4660-76.

3. Cruz M, Gillen DL, Bender M, Ombao H. Assessing health care interventions via an interrupted time series model: Study power and design considerations. Stat Med. 2019;38(10):1734-52.

4. Linden A. A matching framework to improve causal inference in interrupted time-series analysis. J Eval Clin Pract. 2018;24(2):408-15.

The VA QUERI Roadmap | PAGE 122

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

Regression Discontinuity Regression-discontinuity designs are a type of quasi-experimental design used to compare the pre-post impact of implementation interventions. In these evaluations, the implementation strategy is assigned to an intervention or exposed group/unit based on their need for implementation based on a cutoff score for a predetermined assignment variable or qualifying condition. The assignment variable is assessed before the implementation intervention on a continuous measure such as illness severity or patient need for a service. All pa- tients, on scoring below the pre-screening cut-off score, are assigned to one group (e.g., intervention group) and all patients scoring above are assigned to the other condition.

In a hypothetical example by Trochim, the blue Xs to the left of the cutoff show the cases for lower performing facilities on a measure of quality at pre- and post-evaluation. The green circles show the comparison group that is comparatively higher performing at both assessments. The vertical line at the pretest score of 50 indicates the cutoff point with no implementation treatment given. The solid line through the bivariate distribution is the linear regression line. The dis- tribution depicts a strong positive relationship between the pre- and post-assessment scores such that the higher the scores a facility scores at pre-testing, the more likely this facility will score a higher score at post-test assessment at a later time. Similarly, the more site scores low on quality at pre-intervention assessment, the more likely the facility will score lower on the quality indicator at post-intervention.

Figure 28. Example of Pre-Post Distribution with No Treatment Effect

Po st

A ss

es sm

en t

O u

tc o

m e

Pre-Intervention Screening Measure 0 10 50 60 70 80 90 100403020

10

20

30

40

50

60

70

80

Regression line

Cuto�

Evaluators use regression analyses to compare the relationship be- tween the assignment variable and outcome of interest (e.g., health, care utilization) for both groups prior to and after the intervention to infer an intervention effect.

Figure 29 is the same as the prior figure except that all points to the left of the cutoff (i.e., the treatment group) have been raised by 10 points at post intervention assessment. The dashed line in Figure 29 shows what we would expect the treated group’s regression line to look like if the program had no effect as show in Figure 28. An implementation treatment effect is suggested when you see a “jump” or discontinuity in the regression lines at the cutoff point as shown by the orange line in Figure 29.

Figure 29. Regression-Discontinuity Design with Ten-point Treatment Effect

Po st

A ss

es sm

en t

O u

tc o

m e

Pre-Intervention Screening Measure 0 10 50 60 70 80 90 100403020

10

20

30

40

50

60

70

80

Cuto�

Control group regression line

Discontinuity in Regression

Treatment group

regression line

The following example from Trochim illustrates the use of a regression discontinuity design for quality improvement purposes. In this case, a hospital administrator would like to improve the quality of consumer care through the institution of an intensive quality of care training program for staff. Because of financial constraints, the program is too costly to implement for all employees and so instead it will be ad- ministered to the entire staff from specifically targeted units or wards which seem most in need of improving quality of care.

Two general measures of quality of care are available. The first is an aggregate rating of quality of care (QOC) based on observation and rating by an administrative staff member and will be labeled here the QOC rating. The second is the ratio of the number of recorded patient complaints relative to the number of patients in the unit over a set period of time and will be defined here as the Complaint Ratio. In this case example, the administrator could use either the QOC rating or Complaint Ratio as the basis for assigning units to receive the train- ing. Similarly, the effects of the training could be measured on either variable.

The VA QUERI Roadmap | PAGE 123

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

Figure 30. Regression Discontinuity Design for the Purpose of Quality Improvement

Q u

al it

y o

f C ar

e

Quality of Care 0 10 50 60 70 80 90 100403020

10

20

30

40

50

60

70

80 Program

Group

POSITIVE

C o

m p

la in

t R

at e

Quality of Care 0 10 50 60 70 80 90 100403020

10

20

30

40

50

60

70

80 Program

Group

NEGATIVE

In these examples, only the regression lines are shown in the figure and hospital units were assigned to training because they scored below some cutoff score on the QOC rating. The first figure depicts a positive implementation treatment effect because training raised the

program group regression line on the QOC rating over what would have been expected. However, the second figure illustrates a nega- tive effect because the program raised training group scores on the Complaint Ratio indicating increased complaint rates. In either case, a discontinuity in regression lines indicates a program effect in the regression continuity design.

This design is less efficient than many experimental designs and requires more cases to reach statistical power than a randomized trial. Accordingly, evaluators should avoid using this design in situations when assignment is a larger organizational unit (e.g., health system or network).

Suggested Readings 1. Bor J, Moscoe E, Mutevedzi P, Newell ML, Barnighausen

T. Regression discontinuity designs in epidemiology: causal inference without randomized trials. Epidemiology. 2014;25(5):729-37.

2. Coly A, Parry G. Evaluating Complex Health Interventions: A Guide to Rigorous Research Designs. Washington, D.C.: AcademyHealth; June 2017. Available at: https://www.academyhealth.org/ evaluationguide

3. Moscoe E, Bor J, Barnighausen T. Regression discontinuity designs are underutilized in medicine, epidemiology, and public health: a review of current and best practice. J Clin Epidemiol. 2015;68(2):122-33.

4. Trochim, W. M. K. hosted by Conjoint.ly. Research Methods Designs. Regression Discontinuity Design. June 2020. Available at: https://conjointly.com/kb/regression-discontinuity-design/

5. University of Washington Implementation Science Resource Hub. Select Study Design. Quasi-experimental Designs – regression discontinuity. Available at: https://impsciuw.org/ implementation-science/research/designing-is-research/

6. Venkataramani AS, Bor J, Jena AB. Regression discontinuity designs in healthcare research. BMJ. 2016;352:i1216.

7. Walkey AJ, Drainoni ML, Cordella N, Bor J. Advancing Quality Improvement with Regression Discontinuity Designs. Ann Am Thorac Soc. 2018;15(5):523-9.

Observational Designs Observational designs are useful for studying the effectiveness of implementation and quality improvement efforts as they occur naturally over time when it is not feasible for experimental designs to manipulate treatment conditions or to randomly assign groups or sites to different implementation interventions. These study designs are subject to selection bias and confounding variables.

• Cohort Study Design • Cross-sectional design • Case Control Design

Confounding due to selection bias is a concern in observational data due to absence of randomization to treatment – patient characteristics may be associated with both treatment and outcomes. Pre-processing

is a set of strategies that can be used to account for observable selec- tion bias and includes exact matching, coarsened exact matching, propensity scores, and entropy balancing. In all of these strategies, the goal is to make the treatment and comparison groups as similar as possible on observed characteristics, other than the receipt of the treatment. Strategies differ in the extent to which matching is con- ducted on specific variables, moments (e.g., mean, variance) of the dis- tributions of variables, or values of functions of multiple covariates. For more information, see Suggested Readings at the end of this section.

Strategies to balance unobserved and observed characteristics across treatment and comparison groups include regression discontinu- ity (described above), difference-in-differences, and instrumental variables. In difference-in-differences analyses, the treatment effect

The VA QUERI Roadmap | PAGE 124

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

is estimated by observing the difference in changes over time in an outcome over groups that do and do not receive an intervention. In instrumental variable analyses, one identifies a variable (the instru- ment) that is associated with likelihood of receipt of treatment but not the outcome. This method allows for estimation of a treatment effect among individuals whose treatment receipt depends on the value of the instrument.

Suggested Readings 1. Daw, J.R. and Hatfield, L.A. (2018), Matching and Regression to

the Mean in Difference‐in‐Differences Analysis. Health Serv Res, 53: 4138-4156. Available at: https://onlinelibrary.wiley.com/doi/ abs/10.1111/1475-6773.12993

2. Garrido, M. (2018, November 1). Who knew difference- in-differences could be so complicated? The Incidental Economist. https://theincidentaleconomist.com/wordpress/ who-knew-difference-in-differences-could-be-so-complicated/

3. Stuart E. A. (2010). Matching methods for causal inference: A review and a look forward. Stat Sci: a review journal of the Institute of Mathematical Statistics, 25(1), 1–21. Available at: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2943670/

4. Garrido, M. M., Kelley, A. S., Paris, J., Roza, K., Meier, D. E., Morrison, R. S., & Aldridge, M. D. (2014). Methods for constructing and assessing propensity scores. Health Serv Res, 49(5), 1701–1720. Available at: https://www.ncbi.nlm.nih.gov/ pmc/articles/PMC4213057/

5. Zeldow, B., Hatfield, L. (2019). Difference-in-Differences. Available at: https://diff.healthpolicydatascience.org/#introduction

6. Pizer, S.D. An intuitive review of methods for observational studies of comparative effectiveness. Health Serv Outcomes Res Method 9, 54–68 (2009). Available at: https://link.springer.com/ article/10.1007/s10742-009-0045-3

7. Pizer S. D. (2016). Falsification Testing of Instrumental Variables Methods for Comparative Effectiveness Research. Health Serv Res, 51(2), 790–811. Available at: https://www.ncbi.nlm.nih.gov/ pmc/articles/PMC4799892/

The VA QUERI Roadmap | PAGE 125

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

Other Implementation Resources AGENCY FOR HEALTH RESEARCH AND QUALITY: ADVANCES IN PATIENT SAFETY: FROM RESEARCH TO DISSEMINATION – PLANNING TOOL TO GUIDE RESEARCH DISSEMINATION This Dissemination and Implementation toolkit was developed to help investigators evaluate their research and develop effective and appropri- ate dissemination plans. With a focus on assessing “real-world” impact, this tool aids patient-safety investigators to go beyond publishing and presenting and into the workforce. https://www.ahrq.gov/patient-safety/resources/advances/vol4/planning.html

AGENCY FOR HEALTH RESEARCH AND QUALITY’S EVIDENCE-BASED PRACTICE CENTERS In 1997, the Evidence-based Practice Center Program was launched to promote evidence-based practice in everyday care. Evidence-based Prac- tice Centers develop evidence reports and technology assessments on topics relevant to clinical and other health care organization and delivery issues. https://www.ahrq.gov/research/findings/evidence-based-reports/index.html

AGENCY FOR HEALTH RESEARCH AND QUALITY’S PRACTICE FACILITATION HANDBOOK A comprehensive, 21 module handbook designed to support the training of new practice facilitators to acquire relevant knowledge and skills to support meaningful improvement in primary care practices, particularly related to the implementation of the chronic care or consumer-centered medical home models. https://www.ahrq.gov/ncepcr/tools/pf-handbook/index.html

AMERICAN PSYCHOLOGICAL ASSOCIATION DIVISION 12 AND SOCIETY OF CLINICAL PSYCHOLOGY An informational website designed as an integrated resource for clinical psychologists interested in learning more about—and accessing resources related to—dissemination and implementation. This website was developed by the Society of Clinical Psychology Dissemination and Implementation Task Force to promote the involvement of clinical psychology, and Society of Clinical Psychology specifically, in the important and developing field of implementation science. http://www.div12.org/implementation/

CANCER PREVENTION AND CONTROL RESEARCH NETWORK – PUTTING PUBLIC HEALTH EVIDENCE IN ACTION TRAINING An interactive training curriculum to teach community program planners and health educators to use evidence-based approaches, including how to adapt programs. http://cpcrn.org/pub/evidence-in-action/#

CANCER CONTROL P.L.A.N.E.T. (PLAN, LINK, ACT, NETWORK WITH EVIDENCE-BASED TOOLS) A portal that provides access to data and resources that can help planners, program staff, and investigators design, implement, and evaluate evidence-based cancer control programs. https://cancercontrolplanet.cancer.gov/planet

CENTERS FOR DISEASE CONTROL AND PREVENTION INTRODUCTION TO PROGRAM EVALUATION FOR PUBLIC HEALTH PROGRAMS: A SELF-STUDY GUIDE This document is a “how to” guide for planning and implementing evaluation activities. The manual, based on the Centers for Disease Control and Prevention’s Framework for Program Evaluation in Public Health, assists managers and staff of public, private, and community health pro- grams to plan, design, implement, and use comprehensive evaluations in a practical way. https://www.cdc.gov/eval/guide/index.htm

CENTERS FOR DISEASE CONTROL & PREVENTION POLARIS ECONOMIC EVALUATION OVERVIEW This website provides an overview of how economic evaluation can support efforts to identify, measure, value, and compare costs and conse- quences of different health intervention strategies and policies. https://www.cdc.gov/policy/polaris/economics/index.html

COCHRANE Cochrane’s mission is to promote evidence-informed, health decision-making by producing high-quality, relevant, accessible systematic reviews and other synthesized research evidence. Their work is internationally recognized as the benchmark for high-quality information about the ef- fectiveness of health care. https://www.cochranelibrary.com

THE COMMUNITY GUIDE A searchable collection of evidence-based findings of the Community Preventive Services Task Force. It is a resource to help select interventions improve health and prevent disease in a state, community, community organization, business, health care organization, or school. www.thecommunityguide.org

The VA QUERI Roadmap | PAGE 126

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

THE COMMUNITY TOOLBOX A free, online resource supported by the University of Kansas for those working to build healthier communities and bring about social change by offering access to over 300 educational modules and tools on the topics of community assessment, planning, intervention, evaluation, advocacy, and other aspects of community practice. http://ctb.ku.edu/en

CONSOLIDATED FRAMEWORK FOR IMPLEMENTATION RESEARCH – TECHNICAL ASSISTANCE WEBSITE A site created for individuals considering the use of this framework to evaluate an implementation or design an implementation study. http://cfirguide.org

CENTER FOR RESEARCH IN IMPLEMENTATION SCIENCE AND PREVENTION (CRISP) AT UC DENVER: DISSEMINATION AND IMPLEMENTA- TION IN HEALTH TRAINING GUIDE AND WORKBOOK This 2013 Dissemination and Implementation navigation guide, created by the University of Colorado Denver CRISP, seeks to enable and equip investigators to close the gap between research and practice in the health field. It focuses on five main topics: 1) why dissemination and imple- mentation is important; 2) definitions, theories, and concepts; 3) strategies and tools for designing successful dissemination and implementation interventions; 4) recommendations for evaluation design and measurement; and 5) tips for success for investigators and clinicians. http://www.ucdenver.edu/academics/colleges/medicalschool/programs/crisp/training/Documents/Workshops/CRISP%20Dissemination%20 and%20Implementation%20in%20Health%20Workbook.pdf

DISSEMINATION AND IMPLEMENTATION MODELS IN HEALTH RESEARCH AND PRACTICE An interactive database to help investigators and clinicians select, adapt, and integrate the dissemination and implementation model that best fits their research question or practice problem. http://dissemination-implementation.org

QUERI IMPLEMENTATION FACILITATION TRAINING MANUAL VERSION 2 This manual provides information and resources to individuals who want to understand the process of implementation facilitation, a multi- faceted process of enabling and supporting individuals, groups, and organizations in their efforts to adopt and incorporate effective practices into routine practices. The manual seeks to develop the skills needed to help organizations implement effective practices using external and/or internal facilitation. https://www.queri.research.va.gov/tools/implementation.cfm

IMPLEMENTATION SCIENCE AT A GLANCE: A GUIDE FOR CANCER CONTROL PRACTITIONERS An introductory guide to implementation science that provides a succinct overview of the rapidly evolving field of dissemination and imple- mentation science. This 30-page resource is available in three electronic formats. Through summaries of key theories, methods, and models, the guide shows how greater use of implementation science can support the adoption of effective practices. Case studies illustrate how practitioners are successfully applying implementation science in their cancer control programs. https://cancercontrol.cancer.gov/IS/tools/practice.html

INSTITUTE FOR HEALTHCARE IMPROVEMENT An influential, private institute that has been dedicated to redesigning health care into a system without errors, waste, delay, and unsustainable costs for nearly three decades. The Institute for Healthcare Improvement’s website offers a variety of free and fee-based resources, tools, white papers, and links related to improvement and implementation science. The institute has helped develop a number of effective practices in col- laboration with the VA, including rapid-cycle testing (Model for Improvement), break through collaborative series, the Triple Aim, intervention “bundles,” and a framework for spread and scale, to name a few. http://www.ihi.org

KNOWLEDGE TRANSLATION CANADA An online resource that seeks to improve how research results are communicated; to develop a consensus on knowledge translation terminol- ogy and methods for measuring success; to evaluate various knowledge translation approaches; and to find ways to ensure that knowledge translation efforts have a lasting impact across the continuum of care by engaging health professionals, community members, and various health decision-making groups. http://ktcanada.net/

NATIONAL CANCER INSTITUTE - IMPLEMENTATION SCIENCE A website featuring resources, tools, and links related to implementation science training and education, research and practice tools, research funding opportunities, and research initiatives. http://cancercontrol.cancer.gov/is/

The VA QUERI Roadmap | PAGE 127

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

NATIONAL CANCER INSTITUTE, QUALITATIVE RESEARCH IN IMPLEMENTATION SCIENCE (QualRIS GROUP): QUALITATIVE METHODS IN IMPLEMENTATION SCIENCE WHITEPAPER This paper focuses on the multiple ways in which qualitative methods can be effectively used to answer a range of high-priority implementation science questions and describes resources that are available to support the community. https://cancercontrol.cancer.gov/sites/default/files/2020-04/NCI-DCCPS-ImplementationScience-WhitePaper.pdf

NIH EVIDENCE-BASED PRACTICE AND PROGRAMS A collection of several databases and other resources with information on evidence-based disease prevention services, programs, and practices with the potential to impact public health. https://prevention.nih.gov/research-priorities/dissemination-implementation/ evidence-based-practices-programs

NIH OFFICE OF BEHAVIORAL AND SOCIAL SCIENCE RESEARCH (OBSSR) BEST PRACTICES FOR MIXED METHODS RESEARCH A resource developed to provide guidance to implementation researchers on how to rigorously develop and evaluate mixed methods research applications using best practices. https://obssr.od.nih.gov/training/online-training-resources/mixed-methods-research/

PATIENT-REPORTED OUTCOMES MEASUREMENT INFORMATION SYSTEM (PROMIS®) PROMIS® (Patient-Reported Outcomes Measurement Information System) is a set of person-centered measures that evaluates and monitors physical, mental, and social health in adults and children. It can be used with the general population and with individuals living with chronic conditions. http://www.healthmeasures.net/explore-measurement-systems/promis

PROGRAM SUSTAINABILITY ASSESSMENT TOOL (PSAT) A 40-question self-assessment that program staff and stakeholders can take to evaluate the sustainability capacity of a program. Use the results to help with sustainability planning. https://sustaintool.org

REACH, EFFECTIVENESS, ADOPTION, IMPLEMENTATION, AND MAINTENANCE (RE-AIM) FRAMEWORK Resources and tools for those wanting to apply the RE-AIM framework. Includes planning tools, calculation tools, measures, checklists, visual displays, figures, an online RE-AIM module, and more. http://www.re-aim.org

RESEARCH-TESTED INTERVENTION PROGRAMS A searchable database with evidence-based cancer control interventions and programs specifically for program planners and public health practitioners. https://rtips.cancer.gov/rtips/index.do

THE F.A.S.T. LAB – THE FIDELITY, ADAPTATION, SUSTAINABILITY AND TRAINING LAB The overarching goal of the F.A.S.T. Lab is to determine how to facilitate the high-quality delivery of effective psychosocial practices in public sector mental health settings. Areas of emphasis include training and consultation, treatment fidelity and adaptation, and the identification of strategies that promote sustained implementation of effective practices. http://med.stanford.edu/fastlab/research/adaptation.html

THE GRID-ENABLED MEASURES DATABASE Grid-Enabled Measures is a web-based collaborative tool containing behavioral, social science, and other relevant science measures organized by theoretical constructs. Grid-Enabled Measures enable investigators to collaborate with others, encourages the use of common measures, and facilitates the sharing of harmonized data. https://cancercontrol.cancer.gov/brp/research/gem.html

THE UNIVERSITY OF WASHINGTON IMPLEMENTATION SCIENCE RESOURCE HUB The Implementation Science Resource Hub is a resource of University of Washington Department of Global Health’s Implementation Science Program to help further ongoing implementation research and education, providing introduction to the field of implementation science for students and new implementation researchers, and curating selections of supporting resources for further study. https://impsciuw.org/

UNITED STATES PREVENTIVE SERVICES TASK FORCE RECOMMENDATIONS The United States Preventive Services Task Force is an independent, volunteer panel of national experts in disease prevention and evidence- based medicine. The Task Force works to improve the health of all Americans by making evidence-based recommendations about clinical preventive services. https://www.uspreventiveservicestaskforce.org/Page/Name/recommendations

The VA QUERI Roadmap | PAGE 128

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

VA HEALTH SERVICES RESEARCH & DEVELOPMENT AND QUALITY ENHANCEMENT RESEARCH INITIATIVE CYBER SEMINARS A central hub for online presentations on implementation science, evaluation, and health services presentations in health care that occur daily and are hosted by the VA’s Center for Information Dissemination and Education Resources (CIDER). Both VA and non-VA users can subscribe for announcements of upcoming topics; users must register for each seminar. Both the Health Services Research and Development Service and QUERI cyber-seminar pages feature archives of presentations that include slide PDF files and audio files of the recorded presentation that go back to 2016. https://www.hsrd.research.va.gov/cyberseminars/default.cfm

WASHINGTON UNIVERSITY IN ST. LOUIS DISSEMINATION AND IMPLEMENTATION HOME PAGE The one-stop shop for dissemination and implementation activities at Washington University in St. Louis. This is site provides current information about training and consultation in dissemination and implementation at Washington University in St. Louis and links to the many centers and projects engaged in dissemination and implementation research and consultation. https://sites.wustl.edu/wudandi/

BOOKS Albers B, Shlonsky A, Mildon R, eds. Implementation Science 3.0. 1st ed. Charn, Switzerland: Springer International Publishing;2020.

Brownson RC, Colditz GA, Proctor EK, eds. Dissemination and Implementation Research in Health: Translating Science to Practice. 2nd ed. New York, NY: Oxford University Press;2018.

Chambers DA, Vinson CA, Norton WE, eds. Advancing the Science of Implementation across the Cancer Continuum. New York, NY: Oxford University Press;2019.

Grol R, Wensing M, Eccles M, Davis D, eds. Improving Patient Care: The Implementation of Change in Health Care. 2nd ed. Hoboken, NJ: John Wiley & Sons, Ltd.;2013.

Langley GJ, Moen RD, Nolan KM, Nolan TW, Norman CL, Provost LP. The Improvement Guide: A Practical Approach to Enhancing Organizational Performance. 2nd ed. San Francisco, CA: Jossey-Bass;2009.

Nilsen P, Birken SA, eds. Handbook of Implementation Science. Cheltenham, UK. Edward Elgar Publishing Ltd.;2020.

The VA QUERI Roadmap | PAGE 129

Glossary of Terms

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

Glossary of Terms *From the Glossary of Terms from Implementation Science at a Glance: A Guide for Cancer Control Practitioners by the National Cancer Institute, 2019. https://cancercontrol.cancer.gov/sites/default/files/2020-04/NCI- ISaaG-Workbook.pdf

ACTIVE RESISTERS – Health care personnel who vigorously and openly oppose implementing changes in clinical practice and whose resistance typically takes one of two forms: 1) difficulty implementing a change that is incompatible with practices that are engrained as the result of prior clinical training or habitual work flows or 2) resistance from competing authorities on whether to implement the new effec- tive practice.275,276

ADAPTATION – A process of thoughtful and deliberative alteration of the design or delivery of an effective practice with the aim of improv- ing its fit or effectiveness in a given context.62,63

*ADOPTION – A decision to make full use of an effective practice or program as the best course of action available. Also defined as the decision of an organization or community to commit to and initiate an effective practice.108

*COMMUNITY-BASED PARTICIPATORY RESEARCH – A collaborative approach to research that equally involves all partners in the re- search process and recognizes the unique strengths that each brings. Community-based participatory research begins with a research topic of importance to the community and aims to combine knowledge with action to drive social change to improve health outcomes and eliminate health disparities.277

*CONSTRUCTS – Concepts developed or adopted for use in a theory. The key concepts of a given theory are its constructs.278

*DE-IMPLEMENTATION – Reducing or stopping the use of a guide- line, practice, or policy in health care or public health settings.279

DESIGN THINKING – A solution-focused, action-oriented approach to solving problems through the application of user/human-centered design strategies.135

*DISSEMINATION SCIENCE – The study of targeted distribution of information and intervention materials to a specific public health or effective practice audience. The intent is to understand how best to spread and sustain knowledge and the associated evidence-based interventions.280

EFFECTIVE PRACTICE – A health-focused innovation, intervention, program, policy, or technology with evidence from well-conducted scientific research supporting its ability to have a meaningful im- pact on consumer health behaviors or outcomes while minimizing harms.281

*FIDELITY – Degree to which an effective practice is implemented without compromising the core components essential for the prac- tice’s effectiveness.50,51

FORMATIVE EVALUATION – A rigorous and active assessment process designed to identify potential and actual influences on the progress, quality and potential sustainment of implementation effort and then use this data to refine, improve, and evolve the implementation process and, in some cases, adapt an implementation and/or effec- tive practice itself; formative evaluation involves assessment prior to, during, and/or after implementation activities to provide data for im- mediate use to optimize a related implementation effort and for post hoc interpretation of findings.30,186

HEALTH DISPARITY – A particular type of health difference that is closely linked with social or economic disadvantage. Health disparities adversely affect groups of people who have systematically experi- enced greater social and/or economic obstacles to health and/or a clean environment based on: race or ethnicity, gender, age, geograph- ic location, religion, socioeconomic status, sexual orientation, mental health, military service era, cognitive/sensory/physical disability, or other characteristics historically linked to discrimination or exclusion.74

*IMPLEMENTATION OUTCOMES – The effects of deliberate and pur- posive actions to implement new treatments, practices, and services. Implementation outcomes may include acceptability, feasibility, adop- tion, penetration, appropriateness, cost, fidelity, and sustainability.159

*IMPLEMENTATION SCIENCE – The study of methods to promote the adoption and integration of effective practices, interventions, and policies into routine health care and public health settings to improve the impact on population health.8

*IMPLEMENTATION STRATEGIES – Methods or techniques to en- hance the adoption, implementation, and Sustainment of a program or practice.71,126

*KNOWLEDGE SYNTHESIS – A process for obtaining and summariz- ing scientifically derived information, including evidence of effective- ness (risk and protective factors, core components, and key features, etc.).282,283

*KNOWLEDGE TRANSLATION – The process of converting scientific and technically complex research into everyday language and appli- cable actionable concepts in the practice setting.282,283

LOW-VALUE CARE– Health care services or treatments that provide little or no benefit to consumers, have the potential to cause harm, incur unnecessary costs to consumers, or waste limited health care resources.40,41

LEAN (also known as the Toyota Production System [TPS]) – A systematic method for streamlining a process by identifying and elimi- nating unnecessary elements of a process that do not contribute value to the desired outcome or the product being created (also known as waste). The seven types of waste are: motion, defects, waiting, over- processing, overproduction, transport and handling, and inventory.284

The VA QUERI Roadmap | PAGE 131

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

LEARNING HEALTH SYSTEM – Health care systems in which knowl- edge generation processes are embedded in daily practice to produce continuous improvement in quality care outcomes.285,286

LOGIC MODEL – A graphic depiction or roadmap that presents the shared relationships among the resources, activities, outputs, out- comes, and impact for an effective practice; it depicts the relationships among a practice’s activities and its intended effects.287

ORGANIZATIONAL CONSTIPATOR – Individuals characterized as mid- to high-level executive leaders who prevent or delay certain implementation actions without active resistance, thereby acting as insidious barriers to change.275 276

PSYCHOLOGICALLY SAFETY – Individuals’ perceptions related to the degree of interpersonal threat in their work environment288 or the belief that a work setting is safe for interpersonal risk taking and that one will not be punished or humiliated for speaking up with ideas, questions, concerns, or mistakes.239

PROCESS EVALUATION – A rigorous assessment approach designed to identify potential and actual influences on the conduct and quality of a clinical intervention and/or implementation of an effective prac- tice but in which data are not used during the conduct to influence the process; such data can be collected prior to, during, or after the implementation project.30

QUALITY IMPROVEMENT – The systematic and continuous actions by health care professional, consumers and their families, investigators, payers, planners and educators that lead to measurable improvement in health care services and health status of target consumer groups with respect to consumer outcomes (health), better system perfor- mances (care), and professional development of the workforce.289,290

*REACH – The absolute number, proportion, and representativeness of individuals who participate in a given initiative or receive a specific intervention.168,169

*SCALE-UP – Deliberate efforts to increase the spread and use of prac- tices successfully tested in pilot or experimental projects to benefit more people and to foster policy and program development.291

*SERVICE EFFECTIVENESS OUTCOMES – Intervention results ex- amined at the system level, including efficiency, safety, effectiveness, equity, consumer centeredness, and timeliness.160,162

SIX SIGMA – An organization wide error-reduction technique that re- lies on statistical analysis and extensively trained project leaders called black belts and green belts. Six sigma explains exactly how managers can best organize the effort to improve the quality and reduce varia- tion and defects in a process.292

STAKEHOLDERS – Anyone who has a direct or indirect role in sup- porting the adoption and use of an effective practice, including senior organizational leaders, managers, clinicians, support staff, and patients as well as their families (consumers).

SUMMATIVE EVALUATION – A rigorous assessment approach that uses data on the impacts, outputs, products, or outcomes to make judgements about the extent to which the effective practice was implemented as planned and resulted in the desired degree of suc- cess, effectiveness, or goal achievement of an implemented effective practice.30,186

*SUSTAINMENT – The continued use of program components and activities to achieve desirable outcomes.247

USABILITY – The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use.135

USER-CENTERED DESIGN – A design approach that grounds the char- acteristics of an innovation in information about the individuals who use that innovation, with a goal of maximizing “usability in context.”135

USER RESEARCH— Data collection and analyses that are meant to understand the needs, desires, preferences, values, experiences, and recommendations of people who use a particular effective practice.135

The VA QUERI Roadmap | PAGE 132

References

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

References 1. Demakis JG, McQueen L, Kizer KW, Feussner JR. Quality Enhance-

ment Research Initiative (QUERI): A collaboration between re- search and clinical practice. Med Care. 2000;38(6 Suppl 1):I17-25.

2. Atkins D, Kilbourne AM, Shulkin D. Moving From discovery to system-wide change: the role of research in a learning health care system: experience from three decades of health systems research in the Veterans Health Administration. Annu Rev Public Health. 2017;38:467-487.

3. Kilbourne AM, Goodrich DE, Miake-Lye I, Braganza MZ, Bowersox NW. Quality Enhancement Research Initiative Implementation Roadmap: Toward Sustainability of Evidence-based Practices in a Learning Health System. Med Care. 2019;57 Suppl 10 Suppl 3:S286-S293.

4. Bindman AB, Pronovost PJ, Asch DA. Funding Innovation in a Learning Health Care System. JAMA. 2018;319(2):119-120.

5. Edmondson AC. Teaming: How Organizations Learn, Innovate, and Compete In the Knowledge Economy. San Francisco, CA: Jossey- Bass; 2012.

6. Hirschhorn LR, Ramaswamy R, Devnani M, Wandersman A, Simpson LA, Garcia-Elorrio E. Research versus practice in quality improvement? Understanding how we can bridge the gap. Int J Qual Health Care. 2018;30(suppl_1):24-28.

7. Nilsen P. Making sense of implementation theories, models and frameworks. Implement Sci. 2015;10:53.

8. National Cancer Institute. Implementation Science at a Glance: A Guide for Cancer Control Practitioners. Washington, D.C.: U.S. Department of Health & Human Services, National Institutes of Health;2019.

9. Rubenstein LV, Mittman BS, Yano EM, Mulrow CD. From under- standing health care behavior to improving health care: the QUERI framework for quality improvement. Quality Enhancement Research Initiative. Med Care. 2000;38(6 Suppl 1):I129-141.

10. Stetler CB, McQueen L, Demakis J, Mittman BS. An organizational framework and strategic implementation for system-level change to enhance research-based practice: QUERI Series. Implement Sci. 2008;3:30.

11. Kilbourne AM, Atkins D. Partner or perish: VA health services and the emerging bi-directional paradigm. J Gen Intern Med. 2014;29 Suppl 4:817-819.

12. Kilbourne AM, Elwy AR, Sales AE, Atkins D. Accelerating research impact in a learning health care system: VA’s quality enhance- ment research initiative in the choice act era. Med Care. 2017;55 Suppl 7 Suppl 1:S4-S12.

13. Shulkin DJ. Beyond the VA crisis--becoming a high-performance network. N Engl J Med. 2016;374(11):1003-1005.

14. Braganza MZ, Kilbourne AM. The Quality Enhancement Research Initiative (QUERI) Impact Framework: measuring the real world impact of implementation science. J Gen Intern Med. (2020). https://doi.org/10.1007/s11606-020-06143-z

15. Graham ID, Logan J, Harrison MB, et al. Lost in knowledge transla- tion: time for a map? J Contin Educ Health Prof. 2006;26(1):13-24.

16. Greene SM, Reid RJ, Larson EB. Implementing the learn- ing health system: from concept to action. Ann Intern Med. 2012;157(3):207-210.

17. Guise JM, Savitz LA, Friedman CP. Mind the gap: putting evidence into practice in the era of learning health systems. J Gen Intern Med. 2018;33(12):2237-2239.

18. Institute of Medicine. Best Care at Lower Cost: The Path to Con- tinuously Learning Health Care in America. Washington, D.C. The National Academies Press;2012.

19. Field B, Booth A, Ilott I, Gerrish K. Using the Knowledge to Action Framework in practice: a citation analysis and systematic review. Implement Sci. 2014;9:172.

20. Horwitz LI, Kuznetsova M, Jones SA. Creating a learning health system through rapid-cycle, randomized testing. N Engl J Med. 2019;381(12):1175-1179.

21. Langley GJ, Moen RD, Nolan KM, Nolan TW, Norman CL, Provost LP. The Improvement Guide: A Practical Approach to Enhancing Or- ganizational Performance. 2nd ed. San Francisco, CA: Jossey-Bass; 2009.

22. Kraft JM, Mezoff JS, Sogolow ED, Neumann MS, Thomas PA. A technology transfer model for effective HIV/AIDS interventions: science and practice. AIDS Educ Prev. 2000;12(5 Suppl):7-20.

23. Kochevar LK, Yano EM. Understanding health care organization needs and context. Beyond performance gaps. J Gen Intern Med. 2006;21 Suppl 2:S25-29.

24. Neta G, Glasgow RE, Carpenter CR, et al. A framework for enhanc- ing the value of research for dissemination and implementation. Am J Public Health. 2015;105(1):49-57.

25. Fernandez ME, Ruiter RAC, Markham CM, Kok G. Intervention mapping: theory- and evidence-based health promotion pro- gram planning: perspective and examples. Front Public Health. 2019;7:8.

26. Associates in Process Improvement, Institute for Healthcare Improvement (IHI). Science of Improvement: How to Improve. 2017; http://www.ihi.org/resources/Pages/HowtoImprove/Scienceo- fImprovementHowtoImprove.aspx. Accessed February 8, 2019.

27. Leeman J, Calancie L, Kegler MC, et al. Developing theory to guide building practitioners’ capacity to implement evidence- based interventions. Health Educ Behav. 2017;44(1):59-69.

The VA QUERI Roadmap | PAGE 134

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

28. Green LW, Ottoson JM, Garcia C, Hiatt RA. Diffusion theory and knowledge dissemination, utilization, and integration in public health. Annu Rev Public Health. 2009;30:151-174.

29. Aarons GA, Sklar M, Mustanski B, Benbow N, Brown CH. “Scaling- out” evidence-based interventions to new populations or new health care delivery systems. Implement Sci. 2017;12(1):111.

30. Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C. Effectiveness- implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact. Med Care. 2012;50(3):217-226.

31. Bodenheimer TS. Find the “bright spots”. Jt Comm J Qual Patient Saf. 2014;40(4):147.

32. Dilling JA, Swensen SJ, Hoover MR, et al. Accelerating the use of best practices: the Mayo Clinic Model of Diffusion. Jt Comm J Qual Patient Saf. 2013;39(4):167-176.

33. Wilson B, Tseng CL, Soroka O, Pogach LM, Aron DC. Identification of outliers and positive deviants for healthcare improvement: looking for high performers in hypoglycemia safety in patients with diabetes. BMC Health Serv Res. 2017;17(1):738.

34. Washington State Department of Health. Best practices. 2019; https://www.doh.wa.gov/ForPublicHealthandHealthcares/ HealthcareProfessionsandFacilities/BestPractices. Accessed Au- gust 15, 2019.

35. Clancy C. Creating world-class care and service for our nation’s finest: how Veterans Health Administration Diffusion of Excel- lence Initiative is innovating and transforming Veterans Affairs health care. Perm J. 2019;23:18.309

36. Vega R, Jackson GL, Henderson B, et al. Diffusion of Excellence: accelerating the spread of clinical innovation and best practices across the nation’s largest health system. Perm J. 2019;23:18.309.

37. Jackson GL, Cutrona SL, Kilbourne A, White BS, Everett C, Dam- schroder LJ. Implementation science: helping healthcare systems improve. JAAPA. 2020;33(1):51-53.

38. Elnahal SM, Clancy CM, Shulkin DJ. A framework for dissemi- nating clinical best practices in the VA health system. JAMA. 2017;317(3):255-256.

39. Hayes CW, Goldmann D. Highly Adoptable Improvement: A prac- tical model and toolkit to address adoptability and sustainability of quality improvement initiatives. Jt Comm J Qual Patient Saf. 2018;44(3):155-163.

40. Maratt JK, Kerr EA, Klamerus ML, et al. Measures used to assess the impact of interventions to reduce low-value care: a system- atic review. J Gen Intern Med. 2019.

41. de Vries EF, Struijs JN, Heijink R, Hendrikx RJ, Baan CA. Are low- value care measures up to the task? A systematic review of the literature. BMC Health Serv Res. 2016;16(1):405.

42. Shrank WH, Rogstad TL, Parekh N. Waste in the US health care system: estimated costs and potential for savings. JAMA. 2019;322(15):1501-1509

43. Berwick DM, Hackbarth AD. Eliminating waste in US health care. JAMA. 2019;322(15):1501-1509.

44. Wang V, Maciejewski ML, Helfrich CD, Weiner BJ. Working smarter not harder: Coupling implementation to de-implementation. Healthc (Amst). 2018;6(2):104-107.

45. Norton WE, Chambers DA, Kramer BS. Conceptualizing de-imple- mentation in cancer care delivery. J Clin Oncol. 2019;37(2):93-96.

46. Grimshaw JM, Patey AM, Kirkham KR, et al. De-implementing wisely: developing the evidence base to reduce low-value care. BMJ Qual Saf. 2020;29(5):409‐417

47. Norton WE, Chambers DA. Unpacking the complexities of de- implementing inappropriate health interventions. Implement Sci. 2020;15(1):2.

48. Kilbourne AM, Neumann MS, Pincus HA, Bauer MS, Stall R. Imple- menting evidence-based interventions in health care: application of the Replicating Effective Programs framework. Implement Sci. 2007;2:42.

49. Allen JD, Shelton RC, Emmons KM, Linnan LA. Fidelity and its relationship to implementation effectiveness, adaptation, and dissemination. In: Brownson RC, Colditz GA, Proctor EK, eds. Dissemination and Implementation Research in Health: Translating Science to Practice. 2nd ed. New York, NY: Oxford University Press; 2018;267-284.

50. Rabin BA, Brownson RC, Haire-Joshu D, Kreuter MW, Weaver NL. A glossary for dissemination and implementation research in health. J Public Health Manag Pract. 2008;14(2):117-123.

51. Dusenbury L, Brannigan R, Falco M, Hansen WB. A review of research on fidelity of implementation: implications for drug abuse prevention in school settings. Health Educ Res. 2003;18(2):237-256.

52. Kelly JA, Heckman TG, Stevenson LY, et al. Transfer of research- based HIV prevention interventions to community services: fidel- ity and adaptation. AIDS Educ Prev. 2000;12(5 Suppl):87-98.

53. Wiltsey Stirman S, Baumann AA, Miller CJ. The FRAME: an ex- panded framework for reporting adaptations and modifications to evidence-based interventions. Implement Sci. 2019;14(1):58.

54. Chambers DA, Norton WE. The Adaptome: advancing the sci- ence of intervention adaptation. Am J Prev Med. 2016;51(4 Suppl 2):S124-131.

55. Washington State Department of Health. Logic models in public health program management. May 2012.

56. Center for Community Health - University of Kansas. Community tool box chapter 2, section 1. Developing a logic model or theory of change. 2018; https://ctb.ku.edu/en/table-of-contents/over- view/models-for-community-health-and-development/logic- model-development/main. Accessed 2019, August 15.

The VA QUERI Roadmap | PAGE 135

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

57. Petersen D, Taylor PD, Peikes D. Logic models: the foundation to implement, study, and refine patient-centered medical home models. Rockville, MD: Agency for Healthcare Research and Quality;February 2013.

58. Aarons GA, Green AE, Palinkas LA, et al. Dynamic Adaptation Process to implement an evidence-based child maltreatment intervention. Implement Sci. 2012;7:32.

59. Chambers DA, Glasgow RE, Stange KC. The Dynamic Sustain- ability Framework: addressing the paradox of sustainment amid ongoing change. Implement Sci. 2013;8:117.

60. Rohrbach LA, Grana R, Sussman S, Valente TW. Type II translation: transporting prevention interventions from research to real- world settings. Eval Health Prof. 2006;29(3):302-333.

61. Bauman AA, Cabassa LJ, Wiltsey-Stirman S. Adaptation in dissem- ination and implementation science. In: Brownson RC, Colditz GA, Proctor EK, eds. Dissemination and Implementation Research in Health: Translating Science to Practice. 2nd ed. New York: Oxford University Press; 2018:285-300.

62. Stirman SW, Gamarra J, Bartlett B, Calloway A, Gutner C. Empirical examinations of modifications and adaptations to evidence- based psychotherapies: methodologies, impact, and future direc- tions. Clin Psychol. 2017;24(4):396-420.

63. Wiltsey Stirman S, Gutner CA, Crits-Christoph P, Edmunds J, Evans AC, Beidas RS. Relationships between clinician-level attributes and fidelity-consistent and fidelity-inconsistent modifications to an evidence-based psychotherapy. Implement Sci. 2015;10:115.

64. Powell BJ, Beidas RS, Lewis CC, et al. Methods to improve the se- lection and tailoring of implementation strategies. J Behav Health Serv Res. 2017;44(2):177-194.

65. Cancer Prevention and Control Research Network (CPCRN). Ses- sion 5: adapting an evidence-based intervention to fit your com- munity. Putting public health evidence in action training work- shop. November 2017; http://cpcrn.org/pub/evidence-in-action/.

66. Miller CJ, Wiltsey-Stirman S, Baumann AA. Iterative Decision- making for Evaluation of Adaptations (IDEA): A decision tree for balancing adaptation, fidelity, and intervention impact. J Com- munity Psychol. 2020; https://doi.org/10.1002/jcop.222279.

67. Escoffery C, Lebow-Skelley E, Udelson H, et al. A scoping study of frameworks for adapting public health evidence-based interven- tions. Transl Behav Med. 2019;9(1):1-10.

68. Stirman SW, Miller CJ, Toder K, Calloway A. Development of a framework and coding system for modifications and adaptations of evidence-based interventions. Implement Sci. 2013;8:65.

69. Escoffery C, Lebow-Skelley E, Haardoerfer R, et al. A systematic review of adaptations of evidence-based public health interven- tions globally. Implement Sci. 2018;13(1):125.

70. Rabin BA, McCreight M, Battaglia C, et al. Systematic, multi- method assessment of adaptations across four diverse health systems interventions. Front Public Health. 2018;6:102.

71. Proctor EK, Powell BJ, McMillen JC. Implementation strategies: recommendations for specifying and reporting. Implement Sci. 2013;8:139.

72. Glasgow RE, Klesges LM, Dzewaltowski DA, Estabrooks PA, Vogt TM. Evaluating the impact of health promotion programs: using the RE-AIM framework to form summary measures for decision making involving complex issues. Health Educ Res. 2006;21(5):688-694.

73. Chinman M, Woodward EN, Curran GM, Hausmann LRM. Harness- ing implementation science to increase the impact of health equity research. Med Care. 2017;55 Suppl 9 Suppl 2:S16-S23.

74. U.S. Department of Veterans Affairs, Veterans Health Administra- tion, VHA Office of Health Equity. Facts about the Office of Health Equity. 2019; https://www.va.gov/HEALTHEQUITY/Facts_About_ OHE.asp. Accessed July 20, 2019.

75. McNulty M, Smith JD, Villamar J, et al. Implementation research methodologies for achieving scientific equity and health equity. Ethn Dis. 2019;29(Suppl 1):83-92.

76. Woodward EN, Matthieu MM, Uchendu US, Rogal S, Kirchner JE. The health equity implementation framework: proposal and preliminary study of hepatitis C virus treatment. Implement Sci. 2019;14(1):26.

77. Bernal G, Bonilla J, Bellido C. Ecological validity and cultural sen- sitivity for outcome research: issues for the cultural adaptation and development of psychosocial treatments with Hispanics. J Abnorm Child Psychol. 1995;23(1):67-82.

78. Hawkins RP, Kreuter M, Resnicow K, Fishbein M, Dijkstra A. Under- standing tailoring in communicating about health. Health Educ Res. 2008;23(3):454-466.

79. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4:50.

80. Benzer JK, Charns MP, Hamdan S, Afable M. The role of organiza- tional structure in readiness for change: A conceptual integra- tion. Health Serv Manage Res. 2017;30(1):34-46.

81. Edmondson AC. The competitive imperative of learning. Harv Bus Rev. 2008;86(7-8):60-67, 160.

82. Herzer KR, Pronovost PJ. Physician motivation: listening to what pay-for-performance programs and quality improve- ment collaboratives are telling us. Jt Comm J Qual Patient Saf. 2015;41(11):522-528.

83. Leeman J, Calancie L, Hartman MA, et al. What strategies are used to build practitioners’ capacity to implement community-based interventions and are they effective?: a systematic review. Imple- ment Sci. 2015;10:80.

84. Bartholomew-Eldredge LK, Markham CM, Ruiter RAC, Feman- dez ME, Kok g, Parcel GS. Planning Health Promotion Programs: An Intervention Mapping Approach. 4th ed. San Francisco, CA: Jossey-Bass;2016.

The VA QUERI Roadmap | PAGE 136

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

85. Sullivan JL, Adjognon OL, Engle RL, et al. Identifying and over- coming implementation challenges: Experience of 59 nonin- stitutional long-term services and support pilot programs in the Veterans Health Administration. Health Care Manage Rev. 2018;43(3):193-205.

86. Chuang E, Jason K, Morgan JC. Implementing complex innova- tions: factors influencing middle manager support. Health Care Manage Rev. 2011;36(4):369-379.

87. Edmondson AC, Reynolds SS. Building the Future: Big Teaming for Audacious Innovation. Oakland, CA: Berrett-Koehler Publishers, Inc; 2016.

88. Willis CD, Saul J, Bevan H, et al. Sustaining organizational culture change in health systems. J Health Organ Manag. 2016;30(1):2-30.

89. Birken SA, Lee SY, Weiner BJ, Chin MH, Chiu M, Schaefer CT. From strategy to action: how top managers’ support increases middle managers’ commitment to innovation implementation in health care organizations. Health Care Manage Rev. 2015;40(2):159-168.

90. Hagedorn H, Hogan M, Smith JL, et al. Lessons learned about implementing research evidence into clinical practice. Experi- ences from VA QUERI. J Gen Intern Med. 2006;21 Suppl 2:S21-24.

91. Giannitrapani KF, Rodriguez H, Huynh AK, et al. How middle man- agers facilitate interdisciplinary primary care team functioning. Healthc (Amst). 2019;7(2):10-15.

92. Engle RL, Lopez ER, Gormley KE, Chan JA, Charns MP, Lukas CV. What roles do middle managers play in implementation of in- novative practices? Health Care Manage Rev. 2017;42(1):14-27.

93. Miake-Lye I, Mak SS, Lambert-Kerzner AC, et al. Scaling Beyond Early Adopters: A Systematic Review and Key Informant Perspec- tives. Washington (DC): Department of Veterans Affairs (US); 2019.

94. Gardner KL, Dowden M, Togni S, Bailie R. Understanding uptake of continuous quality improvement in Indigenous primary health care: lessons from a multi-site case study of the Audit and Best Practice for Chronic Disease project. Implement Sci. 2010;5:21.

95. Lorig KR, Hurwicz ML, Sobel D, Hobbs M, Ritter PL. A national dissemination of an evidence-based self-management program: a process evaluation study. Patient Educ Couns. 2005;59(1):69-79.

96. Ramanadhan S, Davis MM, Armstrong R, et al. Participatory implementation science to increase the impact of evidence- based cancer prevention and control. Cancer Causes Control. 2018;29(3):363‐369.

97. Goodman MS, Sanders Thompson VL. The science of stakeholder engagement in research: classification, implementation, and evaluation. Transl Behav Med. 2017;7(3):486-491.

98. Midboe AM, Elwy AR, Durfee JM, et al. Building strong research partnerships between public health and researchers: a VA case study. J Gen Intern Med. 2014;29 Suppl 4:831-834.

99. Bergman AA, Delevan DM, Miake-Lye IM, Rubenstein LV, Ganz DA. Partnering to improve care: the case of the Veterans’ Health Administration’s Quality Enhancement Research Initiative. J Health Serv Res Policy. 2017:1355819617693871.

100. Stein BD, Adams AS, Chambers DA. A learning behavioral health care system: opportunities to enhance research. Psychiatr Serv. 2016;67(9):1019-1022.

101. Kirchner JE, Kearney LK, Ritchie MJ, Dollar KM, Swensen AB, Schohn M. Research & services partnerships: lessons learned through a national partnership between clinical leaders and researchers. Psychiatr Serv. 2014;65(5):577-579.

102. Sales A, Smith J, Curran G, Kochevar L. Models, strategies, and tools. Theory in implementing evidence-based findings into health care practice. J Gen Intern Med. 2006;21 Suppl 2:S43-49.

103. Leeman J, Birken SA, Powell BJ, Rohweder C, Shea CM. Beyond “implementation strategies”: classifying the full range of strate- gies used in implementation science and practice. Implement Sci. 2017;12(1):125.

104. Colquhoun H, Leeman J, Michie S, et al. Towards a common terminology: a simplified framework of interventions to promote and integrate evidence into health practices, systems, and poli- cies. Implement Sci. 2014;9:51.

105. Fernandez ME, Ten Hoor GA, van Lieshout S, et al. Implementa- tion mapping: using intervention mapping to develop imple- mentation strategies. Front Public Health. 2019;7:158.

106. Scaccia JP, Cook BS, Lamont A, et al. A practical implementation science heuristic for organizational readiness: your = MC(2). J Community Psychol. 2015;43(4):484-501.

107. Flaspohler P, Duffy J, Wandersman A, Stillman L, Maras MA. Un- packing prevention capacity: an intersection of research-to-prac- tice models and community-centered models. Am J Community Psychol. 2008;41(3-4):182-196.

108. Rogers EM. Diffusion of Innovations. 5th ed. New York: Free Press; 2003.

109. Wilson MG, Lavis JN, Travers R, Rourke SB. Community-based knowledge transfer and exchange: helping community-based organizations link research to action. Implement Sci. 2010;5:33.

110. Wandersman A, Chien VH, Katz J. Toward an evidence-based system for innovation support for implementing innovations with quality: tools, training, technical assistance, and qual- ity assurance/quality improvement. Am J Community Psychol. 2012;50(3-4):445-459.

111. Sales A. Implementation Science: The Basics. Workshop presenta- tion delivered in Kuala Lumpur: University of Malaya; August 2019.

112. Moullin JC, Dickson KS, Stadnick NA, Rabin B, Aarons GA. Sys- tematic review of the Exploration, Preparation, Implementation, Sustainment (EPIS) framework. Implement Sci. 2019;14(1):1.

The VA QUERI Roadmap | PAGE 137

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

113. Stetler CB, Damschroder LJ, Helfrich CD, Hagedorn HJ. A Guide for applying a revised version of the PARIHS framework for imple- mentation. Implement Sci. 2011;6:99.

114. Sales A. Theories and models: pragmatic frameworks and system- atic approaches to intervention design. Presentation delivered at the Training Institute for Dissemination and Implementation in Health, Bethesda, MD. 2015

115. McCreight MS, Gilmartin HM, Leonard CA, et al. Practical use of process mapping to guide implementation of a care coordina- tion program for rural veterans. J Gen Intern Med. 2019;34(Suppl 1):67-74.

116. Antonacci G, Reed JE, Lennox L, Barlow J. The use of process mapping in healthcare quality improvement projects. Health Serv Manage Res. 2018;31(2):74-84.

117. Knox K, Brach C. Module 5. Mapping and Resdesigning Work- flow. In. Practice Facilitation Handbook. Rockville, MD: Agency for Healthcare Research and Quality; May 2018.

118. Hagley G, Mills PD, Watts BV, Wu AW. Review of alternatives to root cause analysis: developing a robust system for incident report analysis. BMJ Open Qual. 2019;8(3):e000646.

119. Tabak RG, Khoong EC, Chambers DA, Brownson RC. Bridging research and practice: models for dissemination and implemen- tation research. Am J Prev Med. 2012;43(3):337-350.

120. Bennett B, Provost L. What’s your theory? Qual Progrss. 2015;48:36.

121. Michie S, Richardson M, Johnston M, et al. The behavior change technique taxonomy (v1) of 93 hierarchically clustered tech- niques: building an international consensus for the reporting of behavior change interventions. Ann Behav Med. 2013;46(1):81-95.

122. Kok G, Gottlieb NH, Peters GJ, et al. A taxonomy of behaviour change methods: an Intervention Mapping approach. Health Psychol Rev. 2016;10(3):297-312.

123. Carey RN, Connell LE, Johnston M, et al. Behavior change techniques and their mechanisms of action: a synthesis of links described in published intervention literature. Ann Behav Med. 2019;53(8):693-707.

124. Connell LE, Carey RN, de Bruin M, et al. Links between behavior change techniques and mechanisms of action: an expert consen- sus study. Ann Behav Med. 2019;53(8):708-720.

125. Michie S, van Stralen MM, West R. The behaviour change wheel: a new method for characterising and designing behaviour change interventions. Implement Sci. 2011;6:42.

126. Powell BJ, Waltz TJ, Chinman MJ, et al. A refined compilation of implementation strategies: results from the Expert Recommen- dations for Implementing Change (ERIC) project. Implement Sci. 2015;10:21.

127. Waltz TJ, Powell BJ, Matthieu MM, et al. Use of concept mapping to characterize relationships among implementation strategies and assess their feasibility and importance: results from the Expert Recommendations for Implementing Change (ERIC) study. Implement Sci. 2015;10:109.

128. Perry CK, Damschroder LJ, Hemler JR, Woodson TT, Ono SS, Cohen DJ. Specifying and comparing implementation strategies across seven large implementation interventions: a practical ap- plication of theory. Implement Sci. 2019;14(1):32.

129. Oliva EM, Christopher MLD, Wells D, et al. Opioid overdose educa- tion and naloxone distribution: Development of the Veterans Health Administration’s national program. J Am Pharm Assoc (2003). 2017;57(2S):S168-S179 e164.

130. U.S. Centers for Disease Control and Prevention Division for Heart Disease and Stroke Prevention, State Heart Disease and Stroke Prevention Program. Evaluation guide: developing and using a logic model. 2020; https://www.cdc.gov/dhdsp/docs/logic_mod- el.pdf. Accessed January 10, 2020.

131. Petersen D, Taylor EF, Peikes D. Logic Models: The Foundation to Implementation, Study, and Refine Patient-Centered Medical Home Models. Rockville, MD: Agency for Healthcare Research and Quality;Feburary 2013.

132. McCawley PF. The logic model for program planning and evalua- tion. 2001. https://www.extension.uidaho.edu/publishing/pdf/ CIS/CIS1097.pdf.

133. Lyon AR, Bruns EJ. User-centered redesign of evidence-based psychosocial interventions to enhance implementation-hospita- ble soil or better seeds? JAMA Psychiatry. 2018.

134. Lyon AR, Munson SA, Renn BN, et al. Use of human-centered de- sign to improve implementation of evidence-based psychothera- pies in low-resource communities: protocol for studies applying a framework to assess usability. JMIR Res Protoc. 2019;8(10):e14990.

135. Dopp AR, Parisi KE, Munson SA, Lyon AR. A glossary of user-cen- tered design strategies for implementation experts. Transl Behav Med. 2019;9(6):1057‐1064.

136. Lyon AR, Koerner K. User-centered design for psychosocial In- tervention development and implementation. Clin Psychol (New York). 2016;23(2):180-200.

137. Hempel S, O’Hanlon C, Lim YW, Danz M, Larkin J, Rubenstein L. Spread tools: a systematic review of components, uptake, and effectiveness of quality improvement toolkits. Implement Sci. 2019;14(1):83.

138. Baumann AA, Morshed AB, Tabak RG, Proctor EK. Toolkits for dis- semination and implementation research: Preliminary develop- ment. J Clin Transl Sci. 2018;2(4):239-244.

The VA QUERI Roadmap | PAGE 138

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

139. U. S. Department of Health and Human Services, Centers for Disease Control and Prevention, Office of the Director, Office of Strategy and Innovation. Introduction to program evaluation for public health programs: a self-study guide. Atlanta, GA: Centers for Disease Control and Prevention; October 2011: https://www.cdc. gov/eval/guide/cover/index.htm.

140. Parry G, Coly A, Goldmann D, et al. Practical recommendations for the evaluation of improvement initiatives. Int J Qual Health Care. 2018;30(suppl_1):29-36.

141. Leviton LC, Khan LK, Rog D, Dawkins N, Cotton D. Evaluability assessment to improve public health policies, programs, and practices. Annu Rev Public Health. 2010;31:213-233.

142. Wholey JS. Assessing the feasibility and likely usefulness of evalu- ation. In: Wholey JS, Hatry HP, Newcomber KE, eds. Handbook of Practical Program Evaluation. 2nd ed. San Francisco: Jossey-Bass; 2004:33-62.

143. U.S. House of Representatives, 113th Congress. H. R. 1287 - Sound Science Act of 2013. 2013; https://www.congress.gov/bill/113th- congress/house-bill/1287. Accessed August 17, 2015.

144. U.S. House of Representatives, 114th Congress. H. R. 185 - Regula- tory Accountability Act of 2015. 2015; https://www.congress.gov/ bill/114th-congress/house-bill/185/text. Accessed August 17, 2015.

145. United States 115th Congress. Public Law No. 115-435: Founda- tions for Evidence-based Policymaking Act of 2018. In. Washington D.C.2019.

146. Kilbourne A, Atkins D. Evidence-based policy making: balancing rigor with real-world health care for veterans and military person- nel. N C Med J. 2015;76(5):339-342.

147. Parker LE, Kirchner JE, Bonner LM, et al. Creating a quality- improvement dialogue: utilizing knowledge from frontline staff, managers, and experts to foster health care quality improve- ment. Qual Health Res. 2009;19(2):229-242.

148. Gray CP, Yakir M, Hung DY. Physician engagement with metrics in lean primary care transformation. Qual Manag Health Care. 2018;27(3):117-122.

149. Miller CJ, Smith SN, Pugatch M. Experimental and quasi-experi- mental designs in implementation research. Psychiatry Res. 2019.

150. Mazzucca S, Tabak RG, Pilar M, et al. Variation in research designs used to test the effectiveness of dissemination and implementa- tion strategies: a review. Front Public Health. 2018;6:32.

151. Brown CH, Curran G, Palinkas LA, et al. An overview of research and evaluation designs for dissemination and implementation. Annu Rev Public Health. 2017;38:1-22.

152. Lewis E, Baumann A, Gerke D, et al. D&I research designs, eight toolkits related to dissemination and implementation research in cancer (DIRC). July 2017; https://sites.wustl.edu/wudandi. Ac- cessed May 1, 2019.

153. Coly A, Parry G. Evaluating Complex Health Interventions: A guide to rigorous research designs. June 2017; https://www.academy- health.org/evaluationguide. Accessed August 2, 2017.

154. Portela MC, Pronovost PJ, Woodcock T, Carter P, Dixon-Woods M. How to study improvement interventions: a brief overview of possible study types. BMJ Qual Saf. 2015;24(5):325-336.

155. Handley MA, Lyles CR, McCulloch C, Cattamanchi A. Selecting and improving quasi-experimental designs in effectiveness and implementation research. Annu Rev Public Health. 2018;39:5-25.

156. Frakt AB, Prentice JC, Pizer SD, et al. Overcoming challenges to evidence-based policy development in a large, integrated deliv- ery system. Health Serv Res. 2018;53(6):4789‐4807

157. Belin TR, Jones A, Tang L, et al. Maintaining internal validity in community partnered participatory research: experience from the community partners in care study. Ethn Dis. 2018;28(Suppl 2):357-364.

158. Allen H, Baicker K, Taubman S, Wright B, Finkelstein A. The Or- egon health insurance experiment: when limited policy resourc- es provide research opportunities. J Health Polit Policy and Law. 2013;38(6):1183-1192.

159. Proctor E, Silmere H, Raghavan R, et al. Outcomes for imple- mentation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. 2011;38(2):65-76.

160. Proctor EK, Landsverk J, Aarons G, Chambers D, Glisson C, Mit- tman B. Implementation research in mental health services: an emerging science with conceptual, methodological, and training challenges. Adm Policy Ment Health. 2009;36(1):24-34.

161. Martin L, Nelson E, Rakover J, Chase A. Whole System Measures 2.0: A Compass for Health System Leaders. Cambridge, MA: Institute for Healthcare Improvement;2016.

162. Institute of Medicine (IOM). Crossing the Quality Chasm: A New Health System for the 21st Century. Washington D.C.: National Academy Press;2001.

163. Gerke D, Lewis E, Prusaczyk B, Baumann A, Proctor E. Implemen- tation outcomes toolkit. Eight toolkits related to dissemination and implementation research in cancer (DIRC). July 2017; https:// sites.wustl.edu/wudandi. Accessed January 17, 2019.

164. Glasgow RE, Vogt TM, Boles SM. Evaluating the public health im- pact of health promotion interventions: the RE-AIM framework. Am J Public Health. 1999;89(9):1322-1327.

165. McCreight MS, Rabin BA, Glasgow RE, et al. Using the Practical, Robust Implementation and Sustainability Model (PRISM) to qualitatively assess multilevel contextual factors to help plan, implement, evaluate, and disseminate health services programs. Transl Behav Med. 2019.

166. Feldstein AC, Glasgow RE. A practical, robust implementation and sustainability model (PRISM) for integrating research findings into practice. Jt Comm J Qual Patient Saf. 2008;34(4):228-243.

The VA QUERI Roadmap | PAGE 139

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

167. Holtrop JS, Rabin BA, Glasgow RE. Qualitative approaches to use of the RE-AIM framework: rationale and methods. BMC Health Serv Res. 2018;18(1):177.

168. Glasgow RE, Estabrooks PE. Pragmatic applications of RE-AIM for health care initiatives in community and clinical settings. Prev Chronic Dis. 2018;15:E02.

169. Glasgow RE, Askew S, Purcell P, et al. Use of RE-AIM to Address Health Inequities: Application in a low-income community health center based weight loss and hypertension self-management program. Transl Behav Med. 2013;3(2):200-210.

170. Aron DC, Tseng CL, Soroka O, Pogach LM. Balancing measures: identifying unintended consequences of diabetes quality perfor- mance measures in patients at high risk for hypoglycemia. Int J Qual Health Care. 2018.

171. Powell BJ, Fernandez ME, Williams NJ, et al. Enhancing the Impact of Implementation Strategies in Healthcare: A Research Agenda. Front Public Health. 2019;7:3.

172. Eisman AB, Kilbourne AM, Dopp AR, Saldana L, Eisenberg D. Eco- nomic evaluation in implementation science: Making the busi- ness case for implementation strategies. Psychiatry Res. 2020;283.

173. Smith MW, Barnett PG. The role of economics in the QUERI pro- gram: QUERI Series. Implement Sci. 2008;3:20.

174. Neumann PJ, Sanders GD, Russell LB, Siegel JE, T.G. G, eds. Cost- effectiveness in health and medicine. 2nd ed. New York: Oxford University Press. 2016.

175. Sanders GD, Neumann PJ, Basu A, et al. Recommendations for conduct, methodological practices, and reporting of cost-effec- tiveness analyses: second panel on cost-effectiveness in health and medicine. JAMA. 2016;316(10):1093-1103.

176. Sullivan SD, Mauskopf JA, Augustovski F, et al. Budget impact analysis-principles of good practice: report of the ISPOR 2012 Budget Impact Analysis Good Practice II Task Force. Value Health. 2014;17(1):5-14.

177. Ritzwoller DP, Sukhanova A, Gaglio B, Glasgow RE. Costing behav- ioral interventions: a practical guide to enhance translation. Ann Behav Med. 2009;37(2):218-227.

178. Saldana L, Chamberlain P, Bradford WD, Campbell M, Landsverk J. The cost of implementing new strategies (COINS): a method for mapping implementation resources using the stages of imple- mentation completion. Child th Serv Rev. 2014;39:177-182.

179. Hamilton AB, Finley EP. Qualitative methods in implementation research: An introduction. Psychiatry Res. 2019;280:112516.

180. Cohen D, Crabtree BF, Damschroder L, et al. Qualitative meth- ods in implementation science. Bethesda, MD: National Cancer Institute, Division of Cancer Control and Population Sciences, U.S. Department of Health and Human Services, National Institutes of Health. 2018.

181. Corbin JM, Strauss AL. Basics of Qualitative Research. 4th ed. Thou- sand Oaks, CA: Sage Publications, Inc. 2015.

182. Palinkas LA, Horwitz SM, Green CA, Wisdom JP, Duan N, Hoag- wood K. Purposeful sampling for qualitative data collection and analysis in mixed method implementation research. Adm Policy Ment Health. 2015;42(5):533‐544.

183. Palinkas LA, Mendon SJ, Hamilton AB. Innovations in mixed methods evaluations. Annu Rev Public Health. 2019;40:423-442.

184. Bauer MS, Damschroder L, Hagedorn H, Smith J, Kilbourne AM. An introduction to implementation science for the non-specialist. BMC Psychol. 2015;3:32.

185. Elwy AR, Wasan AD, Gilman AG, et al. Using formative evaluation methods to improve clinical implementation efforts: Description and an example. Psychiatry Res. 2019:112532.

186. Stetler CB, Legro MW, Wallace CM, et al. The role of formative evaluation in implementation research and the QUERI experi- ence. J Gen Intern Med. 2006;21 Suppl 2:S1-8.

187. Finley EP, Huynh AK, Farmer MM, et al. Periodic reflections: a method of guided discussions for documenting implementation phenomena. BMC Med Res Methodol. 2018;18(1):153.

188. Moore GF, Audrey S, Barker M, et al. Process evaluation of com- plex interventions: Medical Research Council guidance. BMJ. 2015;350:h1258.

189. Dorr DA, Cohen DJ, Adler-Milstein J. Data-driven diffusion of innovations: successes and challenges in 3 large-scale innovative delivery models. Health Aff (Millwood). 2018;37(2):257-265.

190. Kilbourne AM, Abraham KM, Goodrich DE, et al. Cluster random- ized adaptive implementation trial comparing a standard versus enhanced implementation intervention to improve uptake of an effective re-engagement program for patients with serious mental illness. Implement Sci. 2013;8:136.

191. Kilbourne AM, Almirall D, Goodrich DE, et al. Enhancing outreach for persons with serious mental illness: 12-month results from a cluster randomized trial of an adaptive implementation strategy. Implement Sci. 2014;9:163.

192. Abraham KM, Mach J, Visnic S, McCarthy JF. Enhancing treat- ment reengagement for veterans with serious mental illness: evaluating the effectiveness of SMI Re-Engage. Psychiatr Serv. 2018;69(8):887-895.

193. National State Auditors Association. Best practices in perfor- mance measurement Part 1: Developing performance measures. Lexington, KY: National State Auditors Association. 2004.

194. MacLean CH, Kerr EA, Qaseem A. Time out - charting a path for improving performance measurement. N Engl J Med. 2018;378(19):1757-1761.

195. Compton-Phillips A, Mohta NS. Care redesign survey: how data and analytics improve clinical care. NEJM Catalyst. 2019(Febru- ary):18. https://catalyst.nejm.org/data-and-analytics-improve- clinical-care/. Accessed May 2, 2019.

The VA QUERI Roadmap | PAGE 140

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

196. Academy Health, Office of the National Coordinator for Health Information Technology, Health EA. Collecting quality data for performance management: A learning guide In. March 2017: https://www.academyhealth.org/publications/2017-03/collect- ing-quality-data-performance-management-learning-guide.

197. Powell BJ, Stanick CF, Halko HM, et al. Toward criteria for prag- matic measurement in implementation research and practice: a stakeholder-driven approach using concept mapping. Implement Sci. 2017;12(1):118.

198. Glasgow RE. What does it mean to be pragmatic? Pragmatic methods, measures, and models to facilitate research translation. Health Educ Behav. 2013;40(3):257-265.

199. McGinnis JM, Malphrus E, Blumenthal D. Vital Signs: Core Metrics for Health and Health Care Progress. Washington, D.C.: National Academies Press. 2015.

200. McNamara P, De La Mare D, Ivers N. Confidential Physician Feedback Reports: Designing for Optimal Impact on Performance. Rockville, MD: Agency for Healthcare Research and Quality. 2016.

201. Cohen DJ, Dorr DA, Knierim K, et al. Primary care practices’ abili- ties and challenges in using electronic health record data for quality improvement. Health Aff (Millwood). 2018;37(4):635-643.

202. Kirchner JE, Smith JL, Powell BJ, Waltz TJ, Proctor EK. Getting a clinical innovation into practice: an introduction to implementa- tion strategies. Psychiatr Res. 2019.

203. Bunger AC, Powell BJ, Robertson HA, MacDowell H, Birken SA, Shea C. Tracking implementation strategies: a description of a practical approach and early findings. Health Res Policy Syst. 2017;15(1):15.

204. Craig P, Dieppe P, Macintyre S, et al. Developing and evaluat- ing complex interventions: new guidance. London: Medical Research Council;2019; https://mrc.ukri.org/documents/pdf/ developing-and-evaluating-complex-interventions/

205. Rogal SS, Yakovchenko V, Waltz TJ, et al. Longitudinal assess- ment of the association between implementation strategy use and the uptake of hepatitis C treatment: Year 2. Implement Sci. 2019;14(1):36.

206. Waltz TJ, Powell BJ, Fernandez ME, Abadie B, Damschroder LJ. Choosing implementation strategies to address contextual bar- riers: diversity in recommendations and future directions. Imple- ment Sci. 2019;14(1):42.

207. Beidas RS, Kendall PC. Training therapists in evidence-based practice: a critical review of studies from a systems-contextual perspective. Clin Psychol. 2010;17(1):1-30.

208. Grimshaw JM, Thomas RE, MacLennan G, et al. Effectiveness and efficiency of guideline dissemination and implementation strate- gies. Health Technol Assess. 2004;8(6):iii-iv, 1-72.

209. Wensing M, Grol R. Methods to identify implementation prob- lems. In: Grol R, Wensing M, Eccles M, Davis D, eds. Improving Patient Care: The Implementation of Change in Health Care. Chich- ester: John Wiley & Sons, Inc. 2005:165-184.

210. Birken SA, Powell BJ, Shea CM, et al. Criteria for selecting imple- mentation science theories and frameworks: results from an international survey. Implement Sci. 2017;12(1):124.

211. Baker R, Camosso-Stefinovic J, Gillies C, et al. Tailored inter- ventions to address determinants of practice. Cochrane Da- tabase Syst Rev. 2015;(4):CD005470. Published 2015 Apr 29. doi:10.1002/14651858.CD005470.pub3

212. Scheirer MA. Linking sustainability research to intervention types. Am J Public Health. 2013;103(4):e73-80.

213. Flottorp SA, Oxman AD, Krause J, et al. A checklist for identifying determinants of practice: a systematic review and synthesis of frameworks and taxonomies of factors that prevent or enable improvements in healthcare professional practice. Implement Sci. 2013;8:35.

214. Dearing JW, Cox JG. Diffusion of innovations theory, principles, and practice. Health Aff (Millwood). 2018;37(2):183-190.

215. Lewis CC, Klasnja P, Powell BJ, et al. From classification to causal- ity: advancing understanding of mechanisms of change in imple- mentation science. Front Public Health. 2018;6:136.

216. Glanz K, Rimer BK. Theory at a glance: a guide for health promotion practice. In. 2nd ed. Washington, D.C.: National Cancer Institute, National Institutes of Health, U. S. Department of Health and Hu- man Services. 2005.

217. Kilbourne AM, Almirall D, Eisenberg D, et al. Protocol: Adaptive Implementation of Effective Programs Trial (ADEPT): cluster randomized SMART trial comparing a standard versus enhanced implementation strategy to improve outcomes of a mood disor- ders program. Implement Sci. 2014;9:132.

218. Rogal SS, Yakovchenko V, Waltz TJ, et al. The association between implementation strategy use and the uptake of hepatitis C treat- ment in a national sample. Implement Sci. 2017;12(1):60.

219. Stewart RE, Beidas RS, Mandell DS. Stop calling them laggards: strategies for encouraging nonadopters to incorporate evidence- based practices. Psychiatr Serv. 2019;70(10):958-960.

220. Ritchie MJ, Dollar KM, Miller CJ, et al. Implementation Facilitation Training Manual: Using Implementation Facilitation to Improve Care in the Veterans Health Administration (Version 2). 2017; https:// www.queri.research.va.gov/tools/implementation.cfm. Accessed January 1, 2019.

221. Kilbourne AM, Goodrich DE, Lai Z, et al. Reengaging veterans with serious mental illness into care: preliminary results from a national randomized trial. Psychiatr Serv. 2015;66(1):90-93.

222. Institute of Medicine (IOM). Spread, scale, and sustainability in population health: workshop summary. Washington, D.C.: The National Academies Press. 2015.

223. Barker PM, Reid A, Schall MW. A framework for scaling up health interventions: lessons from large-scale improvement initiatives in Africa. Implement Sci. 2016;11:12.

The VA QUERI Roadmap | PAGE 141

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

224. Nolan K, Schall MW, Erb F, Nolan T. Using a framework for spread: The case of patient access in the Veterans Health Administration. Jt Comm J Qual Patient Saf. 2005;31(6):339-347.

225. Aarons GA, Green AE, Trott E, et al. The roles of system and orga- nizational leadership in system-wide evidence-based interven- tion sustainment: a mixed-method study. Adm Policy Ment Health. 2016;43(6):991-1008.

226. Greenhalgh T, Papoutsi C. Spreading and scaling up innovation and improvement. BMJ. 2019;365:l2068.

227. Milat AJ, Bauman A, Redman S. Narrative review of models and success factors for scaling up public health interventions. Imple- ment Sci. 2015;10:113.

228. Kizer KW, Demakis JG, Feussner JR. Reinventing VA health care: systematizing quality improvement and quality innovation. Med Care. 2000;38(6 Suppl 1):I7-16.

229. Institute of Medicine (IOM). Spread, scale, and sustainability in population health: Workshop summary. Washington, D.C.: The National Academies Press. 2015.

230. McCannon J. Accelerating spread and scale in population health. From contemplation to action: keys to getting started and scaling ef- ficiently. The Institute of Medicine Roundtable on Population Health Outcomes: A Workshop on Spread and Scale. Hunter College. New York City, NY. 2014.

231. House RJ. A path-goal theory of leadership effectiveness. Admin Sci Q. 1971;16:321-328.

232. Waldman DA, Bass BM, Yammarino FJ. Adding to contingent- reward behavior: augmenting effect of charismatic leadership. Group and Organizational Studies. 1990;15:381-394.

233. Aarons GA. Transformational and transactional leadership: asso- ciation with attitudes toward evidence-based practice. Psychiatr Serv. 2006;57(8):1162-1169.

234. Bass BM. Good, better, best. Leadership and performance beyond expectations. New York: Free Press. 1985.

235. Aarons GA, Sommerfeld DH. Leadership, innovation climate, and attitudes toward evidence-based practice during a state- wide implementation. J Am Acad Child Adolesc Psychiatry. 2012;51(4):423-431.

236. Solimeo SL, Ono SS, Lampman MA, Paez MB, Stewart GL. The em- powerment paradox as a central challenge to patient centered medical home implementation in the veteran’s health adminis- tration. J Interprof Care. 2015;29(1):26-33.

237. Keroack MA, ngberg BJ, Cerese JL, Krsek C, Prellwitz LW, Trevelyan EW. Organizational factors associated with high performance in quality and safety in academic medical centers. Acad Med. 2007;82(12):1178-1186.

238. Institute for Healthcare Improvement (IHI). The Breakthrough Se- ries: IHI’s Collaborative Model for Achieving Breakthrough Improve- ment. Cambridge, MA: Institute for Healthcare Improvement. 2003.

239. Edmondson AC. Psychological safety and learning behavior in work teams. Adm Sci Q. 1999;44(2):350-383.

240. Duhigg C. What Google learned from its quest to build the perfect team. The New York Times Magazine. February 25, 2016. https://www.nytimes.com/2016/02/28/magazine/what-google- learned-from-its-quest-to-build-the-perfect-team.html. Accessed October 25, 2018.

241. Edmondson AC, Zhike L. Psychological safety: the history, renais- sance, and future of an interpersonal construct. Annu Rev Organ Psychol Organ Behav. 2014;1:23-43.

242. Yanchus NJ, Derickson R, Moore SC, Bologna D, Osatuke K. Communication and psychological safety in veterans health administration work environments. J Health Organ Manag. 2014;28(6):754-776.

243. Derickson R, Fishman J, Osatuke K, Teclaw R, Ramsel D. Psycho- logical safety and error reporting within Veterans Health Admin- istration hospitals. J Patient Saf. 2015;11(1):60-66.

244. Persaud DD. Designing a performance measurement system for accountability, quality improvement, and innovation. Health Care Manag (Frederick). 2019;38(1):82-88.

245. Shewhart WA. Statistical method from the viewpoint of quality control. Lancaster, PA: Lancaster Press. 1939.

246. Deming WE. The New Economics for Industry, Government, Educa- tion. 2nd ed. Cambridge, MA: Massachusetts Institute of Technol- ogy, Center for Advanced Engineering Study. 1993.

247. Scheirer MA, Dearing JW. An agenda for research on the sustainability of public health programs. Am J Public Health. 2011;101(11):2059-2067.

248. Shelton RC, Cooper BR, Stirman SW. The sustainability of evidence-based interventions and practices in public health and health care. Annu Rev Public Health. 2018;39:55-76.

249. Wiltsey Stirman S, Kimberly J, Cook N, Calloway A, Castro F, Charns M. The sustainability of new programs and innovations: a review of the empirical literature and recommendations for future research. Implement Sci. 2012;7:17.

250. Holweg M, Staats BR, Upton DM. Making process improvements stick. 2018:18. https://dx.doi.org/10.2139/ssrn.3240097. Accessed June 1, 2019.

251. Kearsley-Ho EL, Yang HY, Karunananthan S, Laur C, Grimshaw JM, Ivers NM. When do trials of diabetes quality improvement strate- gies lead to sustained change in patient care? BMJ Qual Saf. 2019.

252. Bowman CC, Sobo EJ, Asch SM, Gifford AL, HIV/Hepatitis Qual- ity Enhancement Research Initiative. Measuring persistence of implementation: QUERI Series. Implement Sci. 2008;3:21.

253. Hailemariam M, Bustos T, Montgomery B, Barajas R, Evans LB, Drahota A. Evidence-based intervention sustainability strategies: a systematic review. Implement Sci. 2019;14(1):57.

The VA QUERI Roadmap | PAGE 142

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

254. Orton S, Menkens AJ. Business planning for public health from the North Carolina Institute for Public Health. J Public Health Manag Pract. 2006;12(5):489-492.

255. Justice Research and Statistics Association. Sustaining Evidence- based Practices. Washington, D.C.: Bureau of Justice Assistance, Office of Justice Programs, U.S. Department of Justice;December 2015.

256. Dopp AR, Mundey P, Beasley LO, Silovsky JF, Eisenberg D. Mixed- method approaches to strengthen economic evaluations in implementation research. Implement Sci. 2019;14(1):2.

257. Wolfson D, Santa J, Slass L. Engaging physicians and consum- ers in conversations about treatment overuse and waste: a short history of the Choosing Wisely campaign. Acad Med. 2014;89(7):990-995.

258. Kerr EA, Hofer TP. Deintensification of routine medical services: the next frontier for improving care quality. JAMA Intern Med. 2016;176(7):978-980.

259. Colla CH, Mainor AJ, Hargreaves C, Sequist T, Morden N. Inter- ventions aimed at reducing use of low-value health services: a systematic review. Med Care Res Rev. 2017;74(5):507-550.

260. Whittington MD, Ho PM, Helfrich CD. Recommendations for the use of audit and feedback to ed-implement low-value care. Am J Med Qual. 2019:1062860618824153.

261. McKay VR, Morshed AB, Brownson RC, Proctor EK, Prusaczyk B. Letting go: conceptualizing intervention de-implementation in public health and social service settings. Am J Community Psychol. 2018;62(1-2):189-202.

262. Sacarny A, Barnett ML, Agrawal S. New evidence on stemming low value prescribing. NEJM Catalyst. 2019;April 10. https://cata- lyst.nejm.org/new-evidence-stemming-overprescribing/.

263. Baker DW, Qaseem A, Reynolds PP, Gardner LA, Schneider EC, American College of Physicians Performance Measurement C. Design and use of performance measures to decrease low- value services and achieve cost-conscious care. Ann Intern Med. 2013;158(1):55-59.

264. Kerr EA, Klamerus ML, Markovitz AA, et al. Identifying recommen- dations for stopping or scaling back unnecessary routine services in primary care. JAMA Intern Med. Published online September 14, 2020. doi:10.1001/jamainternmed.2020.4001

265. Johnson K, Hayes C, Center H, Daley C. Building capacity and sus- tainable prevention innovations: a sustainability planning model. Evaluation and Program Planning. 2004;27:135-149.

266. Swensen SJ, Dilling JA, Mc Carty PM, Bolton JW, Harper CM, Jr. The business case for health-care quality improvement. J Patient Saf. 2013;9(1):44-52.

267. Roberts SLE, Healey A, Sevdalis N. Use of health economic evalua- tion in the implementation and improvement science fields-a systematic literature review. Implement Sci. 2019;14(1):72.

268. Raghavan R. The role of economic evaluation in dissemination and implementation research. In: Brownson R, Colditz G, Proctor E, eds. Dissemination and Implementation Research in Health: Translating Science to Practice. New York: Oxford University Press; 2018:89-106.

269. Bodenheimer T, Sinsky C. From triple to quadruple aim: care of the patient requires care of the provider. Ann Fam Med. 2014;12(6):573-576.

270. Stiefel M, Nolan K. A Guide to Measuring the Triple Aim: Population Health, Experience of Care, and Per Capita Cost. IHI Innovation Series white paper. 2012. http://www.ihi.org/resources/Pages/IHIWhite- Papers/AGuidetoMeasuringTripleAim.aspx.

271. Belperio PS, Chartier M, Ross DB, Alaigh P, Shulkin D. Curing hepa- titis C virus infection: best practices from the U.S. Department of Veterans Affairs. Ann Intern Med. 2017;167(7):499-504.

272. Koczwara B, Stover AM, Davies L, et al. Harnessing the synergy between improvement science and implementation science in cancer: a call to action. J Oncol Pract. 2018;14(6):335-340.

273. Nilsen P, Bernhardsson S. Context matters in implementation sci- ence: a scoping review of determinant frameworks that describe contextual determinants for implementation outcomes. BMC Health Serv Res. 2019;19(1):189.

274. Atkins L, Francis J, Islam R, et al. A guide to using the Theoretical Domains Framework of behaviour change to investigate imple- mentation problems. Implement Sci. 2017;12(1):77.

275. Saint S, Kowalski CP, Banaszak-Holl J, Forman J, Damschroder L, Krein SL. How active resisters and organizational constipators affect health care-acquired infection prevention efforts. Jt Comm J Qual Patient Saf. 2009;35(5):239-246.

276. Bearman G, Stevens MP. Pushing beyond resistors and constipa- tors: implementation considerations for infection prevention best practices. Curr Infect Dis Rep. 2014;16(1):388.

277. Community Health Scholars. Program goals and competencies. http://www.kellogghealthscholars.org/about/community/. php#competencies.

278. National Cancer Institute, Division of Cancer Control & Population Sciences. What are constructs? April 28, 2008; https://cancercon- trol.cancer.gov/brp/research/constructs/what-are-constructs. html.

279. Norton WE, Kennedy AE, Chambers DA. Studying de-implemen- tation in health: an analysis of funded research grants. Implement Sci. 2017;12(1):144.

280. National Institutes for Health. Dissemination and implementation research in health. 2017; https://grants.nih.gov/grants/guide/pa- files/PAR-18-007.html.

281. Chambers DA, Vinson CA, Norton WE. Advancing the Science of Implementation Across the Cancer Continuum. New York, NY: Oxford University Press; 2019.

The VA QUERI Roadmap | PAGE 143

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative

282. U.S. Centers for Disease Control and Prevention (CDC). Under- standing the interactive systems framework for dissemination and implementation. Applying science. Advancing practice (ASAP). https://www.cdc.gov/violenceprevention/pdf/ASAP_Issue-a.pdf.

283. Wandersman A, Duffy J, Flaspohler P, et al. Bridging the gap be- tween prevention research and practice: the interactive systems framework for dissemination and implementation. Am J Commu- nity Psychol. 2008;41(3-4):171-181.

284. Agency for Healthcare Research and Quality. Lean. https://heal- thit.ahrq.gov/health-it-tools-and-resources/evaluation-resources/ workflow-assessment-health-it-toolkit/all-workflow-tools/lean. Accessed July 17, 2019.

285. Olsen LA, Aisner D, McGinnis JM, Institute of Medicine (IOM). The Learning Healthcare System: Workshop Summary. Washington, D.C.: The National Academies Press;2007.

286. Institute of Medicine (IOM). Best Care at lower cost: The path to continuously learning health care in America. Washington D.C.: The National Academies Press;2013.

287. U.S. Centers for Disease Control and Prevention (CDC), Program Performance and Evaluation Office. Logic Models. 2020; https:// www.cdc.gov/eval/logicmodels/index.htm. Accessed January 15, 2020.

288. Nembhard IM, Edmondson AC. Psychological safety: a foundation for speaking up, collaboration, and experimentation in organiza- tions. In: Spreitzer GM, Cameron KS, eds. The Oxford Handbook of Positive Organizational Scholarship. New York: Oxford University Press; 2012:490-506.

289. Batalden PB, Davidoff F. What is “quality improvement” and how can it transform healthcare? Qual Saf Health Care. 2007;16(1):2-3.

290. Health Resources and Services Administration (HRSA). Quality Improvement. April 2011. https://www.hrsa.gov/sites/default/ files/quality/toolbox/508pdfs/qualityimprovement.pdf.

291. Simmons R, Fajans P, Ghiron L. Scaling up health service delivery: from pilot interventions to policies and programmes. Geneva, Swit- zerland: World Health Organzation;2007.

292. Agency for Healthcare Research and Quality. Six Sigma. https:// healthit.ahrq.gov/health-it-tools-and-resources/evaluation- resources/workflow-assessment-health-it-toolkit/all-workflow- tools/six-sigma#h=Six%20Sigma. Accessed July 17, 2019.

The VA QUERI Roadmap | PAGE 144

RESOURCE GUIDE

This report was compiled by the VA Quality Enhancement Research Initiative (QUERI) located in

Washington, D.C. funded by the Department of Veterans Affairs, Veterans Health Administration,

Health Services Research and Development. The findings and conclusions in this document are

those of the author(s) who are responsible for its contents; the findings and conclusions do not

necessarily represent the views of the Department of Veterans Affairs or the United States govern-

ment. Therefore, no statement in this article should be construed as an official position of the De-

partment of Veterans Affairs. No investigators have any affiliations or financial involvement (e.g.,

employment, consultancies, honoraria, stock ownership or options, expert testimony, grants

or patents received or pending, or royalties) that conflict with material presented in the report.

U.S. Department of Veterans A�airs

Veterans Health Administration Quality Enhancement Research Initiative