Generating Support for Evidence-Based Practices

Dena87

SubstanceAbuseandMentalHealthServicesAdministration.2012.Aroadmaptoimplementingevidence-basedprograms..pdf

Home >Social Science homework help >Generating Support for Evidence-Based Practices

A Road Map to Implementing Evidence-Based Programs

June 2012

Table of Contents

Course Overview ...................................................................................... 3

About this Course ................................................................................ 3

Intended Audience ............................................................................... 3

What to Expect.................................................................................... 3

Course Topics ..................................................................................... 4

Course Learning Objectives ................................................................... 4

Implementation Language .......................................................................5

The Five Stages of Implementation ..........................................................7

Exploration: Getting Started.....................................................................8

Identifying Community Needs................................................................ 8

Assessing Organizational Capacity.......................................................... 9

Searching Program Registries to Select the Right Program ....................... 11

Understanding Program Fidelity and Adaptation ..................................... 12

Installation: Launching Your Program.................................................... 13

Initial Implementation: Expect the Unexpected ..................................... 15

Full Implementation: The Program is in Place ........................................ 16

Program Sustainability: Maintaining Your Program’s Success................ 18

Appendix A – Resources ......................................................................... 20

Appendix B - References ........................................................................ 24

Appendix C – Supplemental Documents ................................................. 25

Course Overview

About this Course

This course provides guidance to facilitate selection and implementation of one of

the many evidence-based programs related to prevention and treatment that are

publicly available today. You will learn how to (1) select the program that best

matches your organization's needs and (2) carry out the steps necessary to

implement the program you choose.

Intended Audience

Individuals who may benefit from this course include members of an organization

working collaboratively to identify and implement an evidence-based program.

Whether you're looking for a program that addresses bullying, underage drinking,

drug abuse, or treatment of a specific mental health disorder, the information here

can help you in the selection and implementation of a suitable program. Those who

may find the course useful are:

 Administrators, program directors, or clinicians charged with identifying,

selecting, and implementing a program to meet the needs of their target

population, funders, community, etc.

 Individuals interested in learning more about best practices and strategies for successful program selection and implementation

What to Expect

This course provides overall guidance for appropriately selecting and implementing

the program of your choice. The focus is not on individuals choosing a program, but

rather on an organization working collaboratively to select and implement a

program to fit its needs. Experience has shown that organizational commitment,

readiness, and flexibility are all critical to successful selection and implementation

of an evidence-based program or practice.

This course includes several components. A section on terminology will introduce

you to some of the language you may encounter during your program selection and

implementation tasks. A glossary is also available if you need it. Each page is

supplemented with links to other resources that may prove helpful to your learning

experience. The information presented here has been distilled from professional

publications, and full references appear at the end.

A Road Map to Implementing Evidence-Based Programs | 3 http://nrepp.samhsa.gov/AboutLearn.aspx

Course Topics

The following topics are covered in this course:

1. Course Overview

2. Implementation Language

3. Five Stages of Implementation

4. Exploration: Getting Started

5. Installation: Launching Your Program

6. Initial Implementation: Expect the Unexpected

7. Full Implementation: The Program is in Place

8. Program Sustainability: Maintaining Your Program's Success

Course Learning Objectives

This course will help you learn:

 Basic terms related to program selection and implementation

 How to identify your organization's needs

 Where to find available programs

 How to select the best program for your organization's needs, with a focus on

implementation

 The five basic stages of program implementation

A Road Map to Implementing Evidence-Based Programs | 4 http://nrepp.samhsa.gov/AboutLearn.aspx

Implementation Language

This section will introduce you to some of the terms used in the field of

implementation science and research.

To begin, the general term program, as used in this course, refers to an

intervention designed to bring about specific outcomes for specific purposes or

populations. An example would be a program based on scientific principles designed

to prevent drug abuse by children.

Implementation refers to putting the program — and the scientific principles — to

work in a real setting, such as a school or community to bring benefits to a

particular target audience.

Dissemination of programs refers to the targeted distribution of information and

program materials to a specific public health or clinical practice audience. The intent

is to spread knowledge about the programs and encourage their use.

Increasingly, the approach to prevention and treatment includes the use of

evidence-based programs and practices. Evidence-based signifies that the

approach is based in theory and has undergone scientific evaluation. This contrasts

with approaches based on tradition, convention, belief, or anecdotal evidence. The

shift to evidence-based programs seeks to enhance the potential for positive

results. Today, many foundations, government agencies, and state legislatures

encourage or require the use of evidence-based programs in service delivery plans.

Evidence-based programs are designed by program developers such as researchers

at universities, practitioners in the field, and businesses engaged in promoting and

distributing social services programs.

Many evidence-based programs contain a defined set of core components, which

are the essential parts of a program. Some sample core components might be:

 There are five lessons of 30 minutes each that cover five specific themes.

 Sessions are conducted with a group of four to six elementary school

students.

 The intervention is delivered in the home during home visits.  The intervention is delivered in a specific sequence of stages.

In the above example, if you decide to change the length or frequency of the

lessons, or you use the program with a group of 12 middle school students, or you

conduct the program in a classroom instead of at home, or you change the order in

which the core components are introduced to the target population, you have

changed the core components, and you are no longer implementing the program

with fidelity. As a result, you cannot expect the same outcomes the developer

predicted.

A Road Map to Implementing Evidence-Based Programs | 5 http://nrepp.samhsa.gov/AboutLearn.aspx

So, the term fidelity refers to including all the core components of a program

during implementation to help ensure successful outcomes. The term adaptation

refers to the process of changing a program to meet specific needs. If you adapt a

program for any reason, you must maintain the core components to ensure

success. Adaptation will be discussed in more detail later.

Selecting a program for implementation in your setting involves careful planning,

community and organizational involvement, and a comprehensive assessment of

resources. Once this process is completed, you are ready to determine program fit.

Program fit can be described as the (potential) match between your community's

needs, resources, and capacity to implement a program—with the requirements of

the program.

A critical piece of implementing any program or practice is the ability to measure

the effect of the program on the population you are serving. In an outcome

evaluation, it is important to use outcome measures: How is the system

performing? What is the impact or result on what you are trying to change? In a

process evaluation, use process measures: Are the parts/steps in the system

performing as planned?

A Road Map to Implementing Evidence-Based Programs | 6 http://nrepp.samhsa.gov/AboutLearn.aspx

The Five Stages of Implementation

Successfully implementing a program that fits your organization's needs is a

process - not a single event - that occurs in multiple stages of planning, purposeful

action, and evaluating.

It is not enough to simply select a proven evidence-based program and assume

success will automatically follow. Good implementation strategies are essential.

The National Implementation Research Network (NIRN) 1

reviewed more than 2,000

articles on the

implementation of

programs and identified

five main stages of

successful

implementation (Fixsen,

Naoom, Blase, Friedman,

& Wallace, 2005), which

are all interrelated:

1 National Implementation Research Network (NIRN) – http://nirn.fpg.unc.edu

 Exploration

 Installation

 Initial Implementation

 Full Implementation  Program Sustainability

Since the stages are connected, issues addressed (or not addressed) in one stage

can affect another stage. Moreover, changes in your organization or community

may require you to revisit a stage and address activities again to maintain the

program.

A Road Map to Implementing Evidence-Based Programs | 7 http://nrepp.samhsa.gov/AboutLearn.aspx

Exploration: Getting Started

The goal of the Exploration Stage is to select the right evidence-based program.

Your organization will strive to identify the best program fit, which is the match

between needs and resources and the characteristics of the program (this is

discussed in greater detail later). Four main activities are involved in this stage:

 Identify your community's needs to determine the type of program that

will be most appropriate.

 Assess your organizational capacity including financial resources,

organizational commitment, and community buy-in to determine your ability

to implement a program with fidelity.

 Search program registries to select a program that matches your

community needs, your organization's available resources, and available

programs.

 Understand program fidelity and program adaptation.

Focusing on these activities to identify a good program fit is part of ensuring

successful implementation.

Identifying Community Needs

Much has been written about the importance of the community needs assessment.

Identifying your target population and understanding its needs, challenges, and

assets is critical to your success in choosing an appropriate program. Articulating

the outcomes you want to achieve (such as reducing underage alcohol use or

improving parental bonding) will provide the framework for exploring the range of

evidence-based programs and practices and selecting the best fit for your

organization.

See Community Needs Assessment Resources and Tools 2

for more information on

community needs assessments.

2 See Appendix A

A Road Map to Implementing Evidence-Based Programs | 8 http://nrepp.samhsa.gov/AboutLearn.aspx

Assessing Organizational Capacity

Financial and Personnel Costs

Programs cost money. You will likely need program-specific materials to implement

the program (manuals, materials, etc.). However, implementing a new program

may also require additional funds to hire new staff or purchase needed equipment

or space. Contact the developer to discuss the program you are considering. The

developer can clarify basic information such as costs, time needed, what to expect,

etc. You also need to clearly identify your financial and staff and community

resources. Do you have space available? Do you have the funding you need to fully

implement the program? How much will the program itself cost? The program

developer often indicates in the program materials how much it will cost for staff

training, materials, additional equipment, technical assistance support, and all other

costs directly associated with the program. Ask the program developer about these

implementation costs and the cost of service delivery (how a program bills for the

services it provides), if appropriate.

There may also be other less obvious costs to consider related to infrastructure. For

example, if the program indicates the need for staff with specific skills (such as

someone with a master's degree in social work), you will need to consider the skills

of your current staff members and determine if you need to hire someone new or

train a current staff person. If the program requires that all staff members have

access to a computer program or an Internet connection, additional funds may be

required to make such resources available. It is also important to pay attention to

caseload standards as many evidence-based programs require a specific caseload

that may be dramatically different from usual care. The overall size and scope of

the chosen program will influence the potential associated costs of implementing it.

 Questions To Ask: Financial and Personnel Resources3

 See Organizational Capacity Resources and Tools4 for more information on organizational capacity.

3 See Appendix C

4 See Appendix A

A Road Map to Implementing Evidence-Based Programs | 9 http://nrepp.samhsa.gov/AboutLearn.aspx

Commitment and Buy-In

Your organization's staff at all levels will need to be committed to the

implementation process for the long term. It may take one to four years to

implement a program and achieve positive outcomes, and the process will continue

throughout the life of the program. Discussions — and commitments — are extremely important.

It is also important to understand that achieving buy-in is not a one-time event.

Those in positions of leadership, who often make program decisions, and also the

practitioners, support staff, and human resources staff will need to stay committed

to facilitating delivery of the program and eliminating barriers to success.

 Questions To Ask: Commitment and Buy-In5

 See Organizational Capacity Resources and Tools6 for more information on organizational capacity.

5 See Appendix C

6 See Appendix A

A Road Map to Implementing Evidence-Based Programs | 10 http://nrepp.samhsa.gov/AboutLearn.aspx

Searching Program Registries to Select the Right Program

Once you have a clear idea of your program needs and your organizational

resources, you may search registries of programs to select a program that matches

your needs. It is important to read and understand the criteria used to rate or

classify programs as these will differ by source. Registries and other resources may

also be specific to certain topics or service areas such as education, mental health,

juvenile justice, etc.

The Center for Substance Abuse Prevention (CSAP) provides a checklist of

questions (CSAP, 2009) you may ask to see how well your potential selection might

fit your needs:

 Questions To Ask: Center for Substance Abuse Prevention (CSAP)'s Checklist

for Program Fit 7

 See Program Registries Resources and Tools8 for more information on program registries.

7 See Appendix C

8 See Appendix A

A Road Map to Implementing Evidence-Based Programs | 11 http://nrepp.samhsa.gov/AboutLearn.aspx

Understanding Program Fidelity and Adaptation

Two especially important concepts introduced earlier in the section on

implementation language are program fidelity and program adaptation. It is

common for an organization to decide to "adapt" a program at the time of selection

(that is, change it to fit its needs) rather than plan to implement the program with

fidelity (that is, as it was designed by the program developer). Typically,

organizations want to change either (1) the program's content or (2) the mode of

delivery.

You may choose to adapt a program for a variety of reasons, such as

 Differences in the target population: for example, your organization may be

looking for a program suitable for a Latino population and find a good

program that is not specifically targeted to Latinos; you might consider

adjusting the program to serve that community

 Issues with complexity or ease of use: for example, a teacher may want to

deliver a certain classroom-based program, but class time is not long enough

 Potential barriers to implementation such as time, money, resources, or

accessing the target population

 Lack of perceived efficacy, relevance, or acceptance of the program

 Lack of understanding of what makes the program work

Please be cautioned that your adaptations may have implications that will affect the

program's results. Many believe adapting a program is the easier route for

achieving desired outcomes, but the opposite is often true. Adapting an intervention

so that it meets the needs of a certain target population requires a thorough

understanding of the program theory and the components mentioned earlier.

Adapting a component of a program without understanding the underlying rationale

may result in undesired or unintended program outcomes. Once adaptations have

been made (without input from the program developer), you cannot expect to see

the outcomes produced by the original.

Adapting a program may also require additional resources (personnel, time, and

funds) to monitor the adaptation and evaluate the outcomes. Program adaptations

may be necessary, but they will require additional planning and evaluation above

and beyond implementation with fidelity.

See Fidelity and Adaptation Resources and Tools 9

for more information on fidelity

and adaptation.

9 See Appendix A

A Road Map to Implementing Evidence-Based Programs | 12 http://nrepp.samhsa.gov/AboutLearn.aspx

Installation: Launching Your Program

Once a program has been selected and materials purchased, the process of

installing a program begins. Installation refers to making the structural and

instrumental changes necessary to implement the program within an organization.

Establishing an Implementation Team within your organization will be critical at this

stage. The Implementation Team is a core set of individuals charged with

providing guidance through full implementation of the program. This team helps

ensure engagement of the stakeholders, creates readiness for implementation,

ensures fidelity to the program, monitors outcomes, aligns systems, and removes

barriers to implementation. An organization can choose to develop the

Implementation Team during the Exploration Stage; however, the participants may

change as you move into the Installation Stage.

The following are typical tasks to be conducted during this stage:

 Establish space in the organization for the program, including both physical

space and process space.

 Develop easily accessible data systems to collect and measure the effects of

the program and the process.

 Install and debug any needed equipment (e.g., software).

 Interview current staff to ensure a match with needed qualifications.

 Recruit, select, and hire additional staff as needed.

 Train staff to use the program and any new systems or equipment needed to implement the program.

Best Practices

The following are best practices for the Installation Stage:

 Establish an Implementation Team. o A strong understanding of the program to be implemented o Knowledge of how to successfully implement the program with fidelity o An ability to become more effective and efficient over time through

experience

 Identify and engage an individual or group of individuals to "champion" or promote your chosen program.

o Look for people within the organization and the community who are influential, respected, and committed to the program. These people must have the ability to identify problems that arise and to propose

solutions to support and ensure the success of the program. These

individuals should be active participants in the implementation process

and involved with or be members of the Implementation Team.

 Budget for startup costs.

A Road Map to Implementing Evidence-Based Programs | 13 http://nrepp.samhsa.gov/AboutLearn.aspx

o Since installation occurs prior to the provision of the new service, funds will be expended that are not reimbursable or covered through

other funding mechanisms.  Recognize and address issues regarding readiness.

o Individuals do not adapt to change in the same way or along the same time line. Preparing for change and recognizing change occurs in

stages leads to a supportive climate for implementation.

Potential Challenges

The following are potential challenges during the Installation Stage:

 The organization will be spending money before new services are being provided.

 The "status quo" or "business as usual" attitudes and behaviors will be challenged and can cause discomfort and disruption.

 A deeper understanding of the challenges of implementing the new program may surface as practitioners, staff, and leadership individuals are asked to

change behaviors and processes.

The new program may cause disequilibrium in the system. For example, many

agencies and partners will be affected as new services are provided, particularly

with respect to referral procedures.

A Road Map to Implementing Evidence-Based Programs | 14 http://nrepp.samhsa.gov/AboutLearn.aspx

Initial Implementation: Expect the Unexpected

During the initial implementation stage, individuals begin to put into practice all

that has been planned for during exploration and installation. Practitioners and staff

will be changing their behavior, using new skills for the first time, and incorporating

new practices into their everyday routine.

This stage is often awkward because people are now expected to perform new skills

and engage in new processes, which may lead them to perform in an uncoordinated

or hesitant fashion. Practicing and implementing new skills with fidelity will take

time.

Best Practices

The following are best practices for the Initial Implementation Stage:

1. Manage the Change Process. One role for the Implementation Team will

be to guide and manage the change process and help mitigate fear and

resistance. As the team works to remove barriers, it can address issues that

surface and help maintain the momentum of the initiative. The

Implementation Team can help to set realistic goals and expectations

regarding implementation progress, time lines, and the collection and use of

the right data at the right time.

2. Accept Abundant Coaching. Ongoing consistent coaching for line staff and

practitioners is critical to implementation success. A coach can be trained by

the program developer or have extensive experience implementing the

program in a similar organizational environment. A coach can provide

immediate feedback and support to practitioners and staff. Coaching

minimizes resistance, increases positive perceptions of the program and

implementation, and enhances skills development. Coaching also helps

manage expectations and normalize the implementation. Coaches may be

employed by the organization to support staff and assist with monitoring

outcomes with ongoing technical assistance.

Potential Challenges

The following are potential challenges during the Initial Implementation Stage:

 Resistance to change: Putting new and unfamiliar skills and processes to use can be uncomfortable.

 Inertia: Program implementation may get stalled on bumps and snags along the way, causing the initiative to lose momentum.

 Program adaptations: Discomfort and uncertainty can lead to changes to core intervention components.

 Unrealistic expectations: Organizational leaders and funders may have unrealistic expectations regarding implementation progress or consumer

outcomes during this stage.

A Road Map to Implementing Evidence-Based Programs | 15 http://nrepp.samhsa.gov/AboutLearn.aspx

Full Implementation: The Program is in Place

Full implementation occurs when the program is integrated into the service,

organization, and system settings. The processes and procedures to provide the

selected program are now in place. For example, staffing is complete, caseloads are

full, services are being more skillfully provided by staff, and funding streams are in

place. It now becomes important to maintain and improve the program through

excellent monitoring and purposeful improvement to avoid entering program drift

(that is, edging toward a lack of fidelity). Your program or service is ready to be

evaluated, with a focus on assessing program fidelity. Fidelity measures, which can

be provided by the program developer, are commonly used at this point to

determine if the program is being delivered as intended.

After a program has been implemented with fidelity, program modifications may be

considered to help produce better outcomes. For example, if fidelity is high but

outcomes are not as expected, this may require program adaptation, an adjustment

to the program to fit the specific needs of the organization or its recipients.

All adaptations should be carried out carefully and systematically with the help of

the program developer and with attention to both process and outcome

evaluations. Process evaluation determines if the program has been implemented

as intended; for example, if you accomplished what you intended. Outcome

evaluation refers to reviewing the results to determine if the program is working for

the intended population and how this is measured. Outcome evaluation refers to

reviewing the results to determine if the program is working for the intended

population and how this is measured.

Best Practices

The following are best practices for the Full Implementation Stage:

1. Maintain and improve service, linkages, support, and infrastructure.

2. Maintain integrated, fully functioning core program components. If

adaptations must be made, engage the program developer in the adaptation

process.

3. Monitor and evaluate fidelity to ensure the program is being skillfully

delivered, using fidelity measures and indicators developed specifically for

this purpose.

Potential Challenges

The following are potential challenges during the Full Implementation Stage:

 Lack of training or failure to receive the full dosage of training intended

 Lack of required materials

 Use of some but not all of the required lessons and delivery strategies

 Failure to offer the program in the required sequence of stages

A Road Map to Implementing Evidence-Based Programs | 16 http://nrepp.samhsa.gov/AboutLearn.aspx

 Failure to plan for and manage staff turnover

 Lack of adequate referrals

 Lack of time to implement the entire program

 Miscellaneous difficulties or inconveniences, such as technology or equipment

failures

 Day-to-day crises

 Failure to maintain the buy-in and support of staff, administrators, or others

Organizational issues may also contribute to this failure, such as:

 Lack of funding

 Inadequate infrastructure

 Lack of data-driven decisionmaking or autonomous decisionmaking, where

frontline staff are not communicating decisions to leadership or management

 Lack of administrative support

A Road Map to Implementing Evidence-Based Programs | 17 http://nrepp.samhsa.gov/AboutLearn.aspx

Program Sustainability: Maintaining Your Program’s Success

Sustainability is only possible when full implementation has been achieved.

Sustaining change can be difficult. Your program is not frozen in time and must

adapt continually to changes in the community, funding streams, and organizational

priorities. Organizational culture, leadership, and staff need to be nurtured and

maintained. The involvement of high-level administrators in a continuous feedback

loop with the Implementation Team, providers, and recipients is critical. At this

stage, an organization should institutionalize a quality assurance mechanism to

evaluate use of data. This will facilitate assessing the effectiveness and quality of

the program.

Most importantly, sustainability can and should be planned for early in the

implementation process and examined at each stage.

"The bridge from science to service must be built, repaired, maintained, and

improved." - NIRN Brief, January 2009

Best Practices

The following are best practices for the Program Sustainability Stage:

1. Ensure continued funding for the program and the supporting infrastructure

is built into organization's budget.

2. Ensure fidelity to core program components.

3. Develop and implement plans for quality improvement, including regular

review of process and outcome measures and using results to improve the

program.

4. Evaluate data systems that support decisionmaking regarding the

implementation of your program's outcomes.

5. Develop new community partnerships while maintaining existing

relationships.

6. Share positive results with staff, community and others to maintain buy-in

and support.

Celebrate success with program recipients, staff, and community.

Potential Challenges

The following are potential challenges during the Program Sustainability Stage:

 Obstacles and challenges to ongoing success can be many and varied. The

best approach is to continue to monitor results and be alert to changes that

might affect your program.

 Changes that might occur include:

o Funding levels or sources of funding

o Staff turnover

o Organizational priorities o Organizational leadership

A Road Map to Implementing Evidence-Based Programs | 18 http://nrepp.samhsa.gov/AboutLearn.aspx

Thank you for completing our learning module on implementation. We wish you

success in selecting and implementing the right program for your community. If

you have feedback, comments, or further questions, please Contact Us 10

10 Contact Us page of the NREPP website - http://nrepp.samhsa.gov/ContactUs.aspx

A Road Map to Implementing Evidence-Based Programs | 19 http://nrepp.samhsa.gov/AboutLearn.aspx

Appendix A – Resources

Community Needs Assessment

Tools

 CDC Evaluation Working Group: Framework for Program Evaluation http://www.cdc.gov/eval/framework/index.htm

 Child Welfare Information Builder: Evaluation Toolkit and Logic Model Builder http://www.childwelfare.gov/preventing/developing/toolkit/

 Community Toolbox: Implementing Best Process for Community Change and Improvement

http://ctb.ku.edu/en/promisingapproach/

 FRIENDS National Resource Center for Community-Based Child Abuse Prevention: Evaluation Toolkit http://www.friendsnrc.org/evaluation-toolkit

 RE-AIM: Reach, Effectiveness, Adoption, Implementation and Maintenance Framework

http://www.re-aim.org/  W.K. Kellogg Foundation: Evaluation Handbook

http://www.wkkf.org/knowledge-center/resources/2010/W-K-Kellogg-

Foundation-Evaluation-Handbook.aspx

 W.K. Kellogg Foundation: Logic Model Development Guide http://www.wkkf.org/knowledge-center/resources/2006/02/WK-Kellogg-

Foundation-Logic-Model-Development-Guide.aspx