Evidence Based Practice

Toward Creating Synergy Among Policy, Procedures, and Implementation of Evidence-Based Models in Child Welfare Systems: Two Case Examples

Patricia Chamberlain1

Published online: 24 February 2017

� Springer Science+Business Media New York 2017

Abstract Over the past four to five decades, multiple

randomized controlled trials have verified that preventive

interventions targeting key parenting skills can have far-

reaching effects on improving a diverse array of child

outcomes. Further, these studies have shown that parenting

skills can be taught, and they are malleable. Given these

advances, prevention scientists are in a position to make

solid empirically based recommendations to public child

service systems on using parent-mediated interventions to

optimize positive outcomes for the children and families

that they serve. Child welfare systems serve some of this

country’s most vulnerable children and families, yet they

have been slow (compared to juvenile justice and mental

health systems) to adopt empirically based interventions.

This paper describes two child-welfare-initiated, policy-

based case studies that have sought to scale-up research-

based parenting skills into the routine services that case-

workers deliver to the families that they serve. In both case

studies, the child welfare system leaders worked with

evaluators and model developers to tailor policy, admin-

istrative, and fiscal system practices to institutionalize and

sustain evidence-based practices into usual foster care

services. Descriptions of the implementations, intervention

models, and preliminary results are described.

Keywords Scale-up � Foster care � Parenting skills � Implementation

Children and adolescents engaged in the child welfare

system (CWS) are among the most vulnerable in our

society. They have typically experienced trauma, are poor,

and have been exposed to multiple other early adverse

experiences. As noted by Biglan (2016), research in pre-

vention science has shown that it is possible to reduce the

psychological and biological impacts of such toxic stress,

to repair children’s confidence by focusing on their indi-

vidual strengths, and to protect children from opportunities

to participate in risky behavior. Half of boys and girls in

foster care experience mental health problems at a clini-

cally significant level, making them challenging to parent

(NSCAW Research Group 2002). To properly care for and

nurture these youngsters, we need to prepare and support

foster and relative/kinship parents to use parenting skills

that have been proven to be effective. With few exceptions

such as the examples discussed here, our US child welfare

systems have been slow to implement evidence-based

parenting interventions even though these models have

been well established in numerous research studies as

being effective for over a decade. These evidence-based

parenting strategies, although straightforward, are often the

hardest to implement in situations where children need

them most. Foster care is certainly one of those situations.

KEEP (Keeping foster and kin parents supported and

trained) and Parent Management Training Oregon (PMTO)

are well-researched parenting models developed at the

Oregon Social Learning Center as part of a suite of pro-

grams (Dishion et al. 2016) designed to strengthen parent

skills and supports for nurturing children and adolescents

toward optimal development. The goal of these models is

to help parents (foster, relative, biological) learn and

practice daily strategies that optimize the child’s prosocial

development and increase the probability that the home

setting is experienced by the child as a nurturing place

rather than a traumatic one. For example, previous research

shows that providing CWS foster parents with training and

& Patricia Chamberlain [email protected]

1 Oregon Social Learning Center, Eugene, OR, USA

123

Clin Child Fam Psychol Rev (2017) 20:78–86

DOI 10.1007/s10567-017-0226-5

support yields positive results for children and adolescents

and for the CWS (Price et al. 2008). Positive youth out-

comes include lower rates of high-risk behaviors such as

substance use, delinquency, and teenage pregnancy, and

positive outcomes for the CWS include shorter lengths of

stay in foster care and fewer placement disruptions (Hor-

witz et al. 2010). In addition, key mechanisms that drive

outcomes have been identified in several previous studies

(Chamberlain et al. 2008; Price et al. 2008). For example,

high levels of child/adolescent externalizing problems

predict placement disruptions from foster care. Disruptions

from foster home placements and multiple placements

further drive children’s experiences of trauma and increase

mental health problems that are costly to the individual, the

CWS, and society. On the other hand, parental reinforce-

ment, close supervision, the use of non-harsh consistent

limit setting, and school involvement are well-documented

protective factors (Leve et al. 2012).

In this paper we describe two examples where child

welfare system leaders initiated policy-driven reforms in

their foster care services by scaling up two of the above

referenced evidence-based parenting interventions: KEEP

for foster/relative parents and Parenting Through Change

for Reunification (PTC-R) for biological parents (PTC-R

was adapted from the PMTO model for biological parents

with children in foster care; Forgatch and Patterson 2010.).

In addition, we developed and implemented a new case-

work practice model, R 3 , designed to integrate the KEEP

and PTC-R principles into the daily interactions that

casework supervisors have with caseworkers. The imple-

mentation of these models into the existing CWS cultures

involved a phased multi-level effort informed by the EPIS

conceptual framework (Exploration, Preparation, Imple-

mentation, and Sustainability; Aarons et al. (2011)), which

identifies global factors that influence outer and inner

contextual variables affecting implementation of child and

family interventions in child welfare and other public ser-

vice sectors. Examples of outer context factors are the

overall service environment, political climate, and funding;

inner context includes factors such as leadership, agency

characteristics, and worker attitudes. As described below,

our experiences in the implementation in both case

examples affirmed that different contextual variables

played critical roles at different phases of the implemen-

tation process, as is posited by the EPIS model.

The implementation of KEEP, PTC-R, and R 3 into

existing program policies and administrative and fiscal

procedures involved the coordination of multiple levels of

interrelated initiatives designed to work synergistically

with the interventions to optimize their effects. The first

example took place in a large urban child welfare system

that contracts out all foster care casework management to

private agencies. The second is a state-run system with

centralized management where the scale-up began in four

rural regions and is expected to expand statewide. In both

examples the scale-ups were initiated by policy directives

from the CWS agency leadership aimed at improving

outcomes for children and families in foster care. Prior to

the decision to implement the reforms described below, the

CWS leadership teams in both locations had engaged with

Fred Wulczyn and colleagues at the Center for State Child

Welfare Data at Chapin Hall, University of Chicago, to

understand and identify their system needs and to explore

strategic plans to improve their ‘‘business as usual’’ prac-

tices to improve specific outcomes such as decreasing

length of stay in care and rates of placement disruption.

This entailed identifying where, within each system, what

reforms would be launched, and how the selected inter-

ventions designed to accomplish the reforms could be

supported by modifying existing program policies,

administrative rules, and fiscal procedures. The integration

of interventions into the CWS at multiple levels (i.e.,

policy, administrative, and fiscal) has tremendous advan-

tages over implementing interventions as solitary, niche

models in terms of institutionalization of new practices and

the potential for long-term sustainability. First, we provide

a brief overview of the two intervention models that were

selected to accomplish the reforms and the practice model

that was newly developed to strengthen staff implementa-

tion of the principles of the intervention models. Then, we

provide two case examples aimed at illustrating the ways in

which systems with dissimilar cultures and geographies

implemented policy-driven reforms and adapted adminis-

trative procedures and fiscal incentives to work in unison

with key features of the intervention models.

Description of the Interventions

KEEP

KEEP is a training and support intervention developed with

direct input from foster and relative parents that targets the

following outcomes: (a) increasing parenting skills and

confidence, (b) decreasing the number of foster care place-

ment disruptions (lateral moves and step-ups to group care

placements), (c) improving child behavioral and emotional

problems, and (d) increasing the number of positive place-

ment changes (e.g., reunification, adoption). KEEP has been

found to be effective at achieving these outcomes in ran-

domized controlled trials (Chamberlain et al. 2008; Price

et al. 2009), and quasi-experimental designs (Greeno et al.

2015). More specifically, KEEP has been shown to improve

child problem behavior which in turn mediates placement

stability outcomes for children in foster and relative care

(Chamberlain et al. 2008; Price et al. 2008).

Clin Child Fam Psychol Rev (2017) 20:78–86 79

123

A major principle of KEEP is that foster and relative

parents can serve as key agents of change for children. This

is accomplished by strengthening caregivers’ confidence

and skills so they can change their child’s behaviors,

teaching effective parent management strategies, and pro-

viding the caregivers with support. The intervention is

delivered in the context of a parent group, where foster

parents interact with one another guided by two group

leaders. Parents are encouraged to complete a home prac-

tice each week related to the session content. Each session

begins by debriefing the home practice with parents and

tailoring the KEEP strategies to the situation in their home

with their child(ren).

To learn the model, group leaders participate in a 5-day

experiential training that includes information about the

program’s theory and practice in the delivery of group

sessions. During training, each trainee role plays facilitat-

ing several key sessions while other trainees act as foster/

relative parents. KEEP is delivered in 16 weekly group

meetings (90 min each), and includes detailed manuals for

group leaders and for foster/relative parents. Group leaders

tailor the session content based on issues and ideas raised

by the group participants.

Fidelity to the KEEP curriculum is monitored closely

and is measured across three dimensions including: (1)

content, (2) process, and (3) structure, using a standardized

rating protocol (Facilitator Adherence Rating; FAR;

Buchanan et al. 2013). During groups, the facilitators

record each session using a laptop with software that

enables the recording to be uploaded to a HIPAA-com-

pliant, secure Web site designed to allow for direct

observation of group sessions (Fidelity Implementation

Data Observation system; FIDO). KEEP expert consultants

view the recordings, rate them using the FAR, and identify

areas for reinforcement and feedback. The recordings then

are used in weekly consultation meetings (1.5 h each).

Prior to the consultation, group leaders complete a session

review form with questions about what went well and

challenges experienced. They also complete weekly forms

on parent attendance and engagement ratings. Each of

these measures informs the consultation process.

Parenting Through Change for Reunification (PTC-

R)

PTC-R is a parent skill building and support model training

program for biological parents whose children are living in

foster care. PTC-R was adapted from the Parent Manage-

ment Training Oregon model (PMTO; Forgatch and Pat-

terson 2010). PTC-R addresses key clinical issues in a

parent group format. It is designed to increase effective

parenting practices to decrease child behavioral and emo-

tional problems and increase parent and child prosocial

behavior. Parents learn to provide a nurturing, consistent,

and safe family environment with an emphasis on strate-

gies that deal with a wide variety of contextual adversities

and stressors. Parents are engaged in exercises that teach

core intervention components to decrease coercive and

inconsistent parenting and increase effective parenting

(skill encouragement, limit setting, monitoring, positive

involvement, emotional regulation, and problem solving).

Numerous studies have reported improvements in these

parenting practices, which in turn produce positive out-

comes for children, including reduction in behavior prob-

lems, police arrests, rates of out-of-home placement,

delinquency, deviant peer association, and depression and

improved academic performance and social skills (For-

gatch et al. 2009; Patterson et al. 2010; Forgatch et al.

2009; Forgatch and DeGarmo 1999). The program has

been replicated in numerous randomized controlled trials

and quasi-experimental designs by independent research

teams nationally and internationally (e.g., Akin et al. 2014;

Kjøbli et al. 2013; Ogden and Hagen 2008). PTC-R is

delivered in ten group sessions followed by an additional

six sessions near the time of reunification.

Similar to KEEP, PTC-R trainees participate in a 5-day

experiential pre-service training. Once trained, PTC-R

interventionists deliver the program to parent groups in

90-min sessions. Group leaders record each group session

using a laptop with software that enables the recording to

be uploaded to FIDO. The recordings are then viewed by

the developers, coded for fidelity, and used in weekly

consultation meetings (1.5 h each). Prior to the consulta-

tion, facilitators complete a session review form with

questions about what went well and challenges experi-

enced, and complete weekly forms on parent attendance

and engagement ratings.

Because KEEP and PTC-R were shaped from the same

behavioral and social learning frameworks often utilized in

interventions developed at OSLC, the integration of the

two in the project proved to be a natural fit. In the first case

example, CWS leadership determined that all participating

caseworkers and supervisors would be trained to provide

both KEEP (for foster/relative parents) and PTC-R (for

biological/adoptive parents). The reform was designed so

that caseworkers would implement these two interventions

as part of their usual casework routines, rather than the

previous strategy of referring parents out for support and

parenting services.

R 3

To strengthen the reach and impact of KEEP and PTC-R

principles, a casework practice model was developed by

OSLC researchers (Saldana et al. 2016) at the request of

CWS leadership. R 3 stems from the same social learning

80 Clin Child Fam Psychol Rev (2017) 20:78–86

123

principles of reinforcement used in KEEP and PTC-R. The

idea of R 3 was to embed reinforcement strategies into all

interactions that frontline staff have with children and

families in the foster care system To broadly implement R 3 ,

we elected to train casework supervisors to use the R 3

reinforcement principles with their caseworkers in routine

group supervision meetings and, in turn, supervisors

trained the caseworkers to reinforce the efforts, strengths,

and small steps toward achievement of case goals of the

parents on their caseloads (biological, foster, relative) in

the context of their daily interactions.

Case Example 1: New York City

Prior to the foster care reform, under the Bloomberg

administration, the New York City Administration for

Children and Families (ACS) had implemented numerous

evidence-based preventive interventions models in juvenile

justice and in the CWS, including working with the Oregon

Social Learning Center intervention development team.

Capitalizing on this experience, ACS elected to initiate a

foster care system reform in 2012 to achieve the following

four goals:

1. Decrease the rate of foster care placement disruptions

by increasing the supports for and skills training of

foster and relative parents.

2. Decrease the number of children placed per home to a

maximum of 3 (unless children were related; sibling

groups).

3. Decrease the average length of stay in foster care by

developing a parent/caregiver skill building and

engagement model.

4. Decrease re-entry rates into foster care for families

who had been reunified.

To accomplish these goals, ACS elected to focus on

strengthening caseworkers’ skills and supports for inter-

acting effectively with foster/relative parents using KEEP

and with biological parents using PTC-R. Additionally,

ACS requested that a supervisor practice model be devel-

oped to extend the reach of KEEP and PTC-R principles to

interactions between supervisors and caseworkers. KEEP

and PTC-R share roots in social learning theory, which

posits that people modify their behavior by observing

reinforcement received by others, and that all individuals

exist within and respond to their environments in an

adaptive way (behaviors that they are being reinforced for

increase). The R 3 model capitalizes on the idea that the

social context within the system involves relationships

between multiple agents (e.g., leadership, supervisors, and

caseworkers) that ultimately affect outcomes for families.

Supervisors were selected as the target for the R 3 model

because they are positioned central to the organizational

leadership and caseworkers, the on-the-ground workforce

who make daily decisions that affect the outcomes of the

families being served.

Context and Preparation

ACS contracts with over 35 private agencies to do all case

planning and case management services for families

involved in the New York City foster care system. ACS

wanted to select five ‘‘pilot’’ agencies to implement the

reform that varied in size, geography, culture, and level of

past performance on key outcomes. ACS leaders consulted

with Fred Wulczyn and colleagues at Chapin Hall to select

five contracted agencies serving a total of over 2000 chil-

dren per year (ages 2–18 years; approximately 20% of the

foster care population at the time). Chapin Hall researchers

utilized data from their Center for State Child Welfare Data

archives to assist ACS in selecting representative agencies.

Program Policy

A major policy change enacted by ACS under this reform

was to have caseworkers implement the KEEP and PTC-R

interventions themselves rather than to refer parents to

outside sources for these services as had been done pre-

viously. This new ACS policy was motivated by the desire

to provide consistent, high quality, evidence-based par-

enting interventions to all caretakers of foster children

(foster, biological, relative, adoptive). Having caseworkers

directly provide these services had several potential

advantages for families including the possibility of

enhanced family engagement with the CWS, greater

coherence in their individual case plan, and the opportunity

to experience a more supportive relationship with case-

workers. From a system improvement perspective, case-

workers had first-hand knowledge of the parent’s level of

skill, confidence, and commitment, and caseworkers

potentially benefited by improving their own skills after

being trained in well-researched intervention that is shown

to improve parenting skills and decrease child problem

behaviors. We hypothesize (although we have yet to test)

that the R 3 practice model has the potential to decrease

caseworker stress.

Administrative

As described more fully by Wulczyn and Feldman (in

press), the following administrative changes were intro-

duced. Caseworker caseloads were reduced to no more than

12 cases to allow time for the caseworkers to be trained

and to implement the evidence-based models. All agency

caseworkers and supervisors participated in 10 days of in-

Clin Child Fam Psychol Rev (2017) 20:78–86 81

123

person training over approximately 3 months (5 days for

KEEP; 5 days for PTC-R). Staff in each agency was

divided into two cohorts so that while one half of the

caseworkers and supervisors participated in trainings, the

other half could continue to provide all usual services

during the training days. During the experiential training,

each caseworker participated in role plays by facilitating

several key sessions while other caseworkers acted as

participating parents (foster/relative in KEEP; biological in

PTC-R). After caseworkers completed the training, they

conducted 1–3 ‘‘mock’’ groups to practice key content

elements and gain experience with the fidelity monitoring

video uploading system. This system was used to monitor

the delivery of the interventions, to track participation

rates, parent engagement levels, child behavior problems,

parent stress levels, and other activities related to the

implementation. This information was used for assessing

progress and barriers, and for providing feedback and

consultation to group leaders on a weekly basis during

telephone conferences. Once KEEP and PTC-R groups

were up and running, caseworkers and supervisors were

trained in R 3 , a model that aims to influence the ‘‘how’’

(not the ‘‘what’’) of casework practice. In the R 3 model,

supervisors are provided with training and weekly con-

sultation. The supervisory role is critical to organizational

climate because they are centrally positioned in the system

between caseworkers and leadership. The three Rs include

reinforcement of: (1) parental efforts, (2) parental rela-

tionships and roles, and (3) parents’ small steps toward

goal achievement. These three reinforcement targets are

encouraged in interactions between supervisors and their

supervisee-caseworkers and between caseworkers and the

families they serve.

Fiscal

As detailed in Wulczyn and Feldman (in press), the reform

was designed so that the potential cost savings generated

by the outcomes of KEEP and PTC-R would offset the up-

front investments made to facilitate the administrative

changes noted above plus the costs of implementing the

two EBPs and R 3 . The lowered caseloads and implemen-

tation and training costs were considered necessary core

investments in the project that were expected to be offset

by the savings generated from the EBPs and the other

innovations. Therefore, in order to achieve savings suffi-

cient to neutralize costs of the reform, ACS calculated that

a 17% reduction in each of three key project goals

(placement disruptions, length of stay, and re-entry rates)

would need to be attained.

Agencies were reimbursed by ACS for their participa-

tion in the trainings and for providing financial incentives

to parents (foster and biological) for attending weekly

group sessions. The incentives were intended to cover costs

such as transportation and to provide parents with a modest

stipend to acknowledge their commitment to participate;

parents received $25 per session for attendance, and if they

attended 80% of sessions, they received a $100 bonus.

Evaluation

The evaluation was conducted by Chapin Hall researchers

under a separate contract. First, it aimed to examine whe-

ther the KEEP and PTC-R models were implemented as

designed and whether the reforms had the intended impact.

ACS asked Chapin Hall to address the following research

questions in line with the goals of the reform:

• What was the impact on children’s lengths of stay in foster care, both for children who entered placement

after implementation as well as children in care when

the models were implemented?

• What was the impact on the stability of children’s placements?

• What was the likelihood of re-entry into care once permanence was achieved?

Four sources of data are used to evaluate project results.

First, Chapin Hall research staff conducted one-on-one

interviews with system stakeholders over the course of the

first year, including provider agency staff, public agency

staff, and the developers of the clinical models. The

interviews were designed to get an in-depth description of

the implementation experience from various perspectives,

particularly as it related to changes in the process of care,

the quality of care, and the extent to which the necessary

system (administrative) capacity adjustments were made to

support implementation efforts.

Second, an online survey was administered to case-

workers and supervisors from the five agencies. The survey

was completed at the middle and end of the first year of

implementation and included questions about employee job

satisfaction and the extent to which job satisfaction had

changed.

Third, data were used from the KEEP/PTC-R Web-

based fidelity management system that included video

uploads of all KEEP and PTC-R sessions, fidelity ratings,

attendance, parent engagement ratings, and other associ-

ated data. These data were used in conjunction with data

from administrative records, the fourth source of data.

Administrative data included information related to chil-

dren’s placements in out-of-home care such of dates of

entry and exit, type of placement, and placement moves.

Together, these data were used to create an analytic data

file that allowed for the measurement of potential project-

specific effects.

82 Clin Child Fam Psychol Rev (2017) 20:78–86

123

Results on length of stay and placement stability out-

comes are reported in detail by Chamberlain et al. (2016)

and by Wulczyn and Feldman (in press). Analyses on re-

entry rates have not yet been conducted. As this was not a

randomized controlled trial, alternative available compar-

ison groups were used that included: (1) outcomes for

children who had been served by the five agencies prior to

the reform, (2) outcomes for children who were in care at

non-project agencies prior to the time the project was

operational, and (3) children who were admitted to non-

project agencies after the date the project became opera-

tional. To assess the rate of permanency, controlling for

agency and period effects, children exposed to KEEP,

PTC-R, and R 3 were compared with children served by

non-project agencies together with children served by the

agencies outside of the project period. The rate of perma-

nency (days to reunification) for children in the five

intervention agencies was greater than the rate for all other

children in the comparison group. This difference was

statistically significant. Placement stability was assessed

controlling for agency and period effects. An 18% slower

rate of placement movements was observed in the KEEP-

exposed homes; this effect did not reach statistical signif-

icance although the effect showed a trend toward increased

placement stability in intervention agencies.

Sustainment

We implemented a full transfer model of sustainment as

conceptualized by Forgatch et al. (2016). In the full transfer

model, KEEP and PTC-R group leaders who have met

performance and work load criteria are eligible to be

trained as local trainers and consultants. Once trained, they

provide the ongoing services for their agency staff that

model developer team provided during the implementation

phase of the project. These services include viewing video

uploads of group sessions, fidelity ratings, weekly coaching

for group leaders, and training new staff members. The

model developer teams continue to conduct periodic fide-

lity assessments and provide additional training/consulta-

tion as needed. All five agencies in NYC have achieved full

transfer in both KEEP and PTC-R.

Case Example 2: Tennessee

The State of Tennessee was selected by the US Department

of Health and Human Services as one of eight Title IV-E

Waiver states in 2013. Awarded states are allowed greater

flexibility in how they utilize their federal funds to address

their most challenging child welfare problems. Waivers do

not provide new money to the state, but rather allow them

to utilize portions of their dollars in ways not traditionally

covered by federal dollars, such as adopting new practice

strategies in their communities. The Tennessee leadership

elected to implement KEEP and R 3 as part of the In Home

Tennessee model to address their waiver goals which are:

1. To increase permanency for youth,

2. To reduce time in foster care, and

3. To promote positive outcomes for children and fam-

ilies in their homes.

Context and Preparation

Tennessee decided to first focus their efforts for KEEP and

R 3 in four regions (out of 12 total) in East Tennessee that

showed relatively higher placement disruption rates and

children remaining in care for periods above the state

average; KEEP was first implemented in ten counties in the

four regions and then R 3 . A second cohort of an additional

14 counties in those same four regions was then brought on

board with KEEP. The aim is to eventually roll out these

models in all 95 Tennessee counties in the additional eight

regions over the next 3 years. The State IV-E waiver is

time limited, so it is imperative to Tennessee’s CWS

leadership that there is a clear path and strategy to maintain

the investments in workforce training that they are making

under their waiver plan. We collaborated with Tennessee

CWS leaders to structure a full transfer plan whereby a

solid internal infrastructure for training and maintaining

KEEP intervention strategies was devised. In February

2015, we began planning with leaders at Tennessee’s

Department of Children’s Services (DCS) to implement

KEEP with children placed in relative and foster care

homes and to implement R 3 with all supervisors of custo-

dial (foster care) and non-custodial casework staff. We

estimate that once rolled out statewide, 175 regional

administrators, 300 supervisors, and 1200 caseworkers and

over 8000 children will be exposed to KEEP and R 3 .

Program Policy

As in NYC, the Tennessee CWS leadership changed their

service delivery policy such that frontline staff directly

delivered the KEEP intervention to foster parents with the

aim of providing them with consistent evidence-based

parenting skills and to give caseworker and other frontline

staff training and support for enacting those skills. Addi-

tionally, to attempt to extend the reach of KEEP, supervi-

sors are being trained in R 3 . Tennessee leadership elected

to contract with private agencies to implement services to

biological parents; plans are underway at this time to

implement the Nurturing Parent model to support biologi-

cal parents (www.nurturingparenting.com).

KEEP is being implemented with foster and relative

parents of 4–12 year olds, and R 3 is being implemented

Clin Child Fam Psychol Rev (2017) 20:78–86 83

123

with all supervisors of custodial (foster care) and non-

custodial staff. Fidelity monitoring occurs for both models

using a Web-based fidelity monitoring system which has

been continually upgraded to include the ability to generate

reports relevant to CWS leadership, local offices, and

regional administrators. In addition to tracking attendance,

completion rates, session fidelity scores and parent

engagement, we now also track staff participation in con-

sultation and training. These data will be used by the

Chapin Hall research team to evaluate the effects of dosage

on child and system outcomes. As with KEEP, feedback is

provided to supervisors quickly to maximize the potential

that families receive consistent exposure to the model’s

core principles. Additionally, as with KEEP sessions, video

recordings of R 3 supervision sessions are sent to the model

developer for fidelity coding and feedback. This level of

observation-based, rapid feedback, allows for more

immediate reinforcement and modeling of the KEEP and

R 3 strategies during weekly consultation between Oregon

and Tennessee.

Administrative

In Tennessee the state DCS provides all case management

services for foster homes through their regional and local

offices rather than contracting case management services

out to private agencies as was done in New York City

(Case Example 1).

Using lessons learned from the NYC scale-up of KEEP,

we modified the rollout strategy for Tennessee. In NYC, all

agency caseworkers were trained to deliver the interven-

tions whereas in Tennessee only a small group of case-

workers were selected to be KEEP group leaders in each of

the four regions. The advantage of training all workers as

we did in NYC is that all became familiar with the inter-

vention goals and theory and therefore were in a good

position to support foster and relative parents to enact the

KEEP parenting skills. Three disadvantages of intensively

training the total caseworker workforce were: (1) the cost

of training large groups of caseworkers, (2) the time

required to be off-line to receive the training, and (3) the

fact that once their initial cadre of foster/relative parents

completed KEEP, numbers of new parents requiring the

training in subsequent years were smaller thereby needing

the services of fewer KEEP group leaders that we had

trained.

In Tennessee, in order to maintain a culture of case-

worker support for and knowledge of KEEP and to conduct

a more cost-effective rollout, in partnership with DCS

leaders we decided to expose all caseworkers and super-

visors to a 2-day Foundational Training focused on the

theory behind KEEP and an overview of the parenting

skills being taught and reinforced. Like the intensive 5-day

training, this was interactive and included role plays,

exercises, and games. After the Foundational Training, we

collaborated with DCS leaders to select 4–10 staff from

each region to receive the 5-day intensive KEEP training.

Fiscal

As mentioned above, waiver funding allows for the redi-

rection of funding to implement practices to improve child

and family outcomes. Further, the Tennessee DCS fiscal

team elected to reimburse parents for completing KEEP by

increasing their daily foster care reimbursement rate. This

is a creative use of fiscal incentives that appears to be

meaningful to foster parents in that it has facilitated the

successful recruitment of parents to participate in KEEP

groups.

Evaluation

As part of the federal waiver requirements, states are

required to include an evaluation plan to determine if their

goals are impacted. Like in NYC, Tennessee contracted

with Chapin Hall to conduct this independent evaluation.

Administrative data records will be used to determine

length of stay and placement disruption outcomes. Quali-

tative interviews with frontline staff and leadership will

also be conducted. The implementation of KEEP and R 3

are underway. Therefore, the evaluation and results of the

waiver goals are not yet available for this project.

Summary and Conclusions

During the past decade, the reasons for lack of use of

evidence-based interventions in public service systems are

becoming better understood (Saldana et al. 2012) although

there is still much to be learned about how researchers and

policy decision makers can partner to plan for, select,

accomplish, and sustain implementations that result in

better outcomes for children, youth, and families. The EPIS

model provides a conceptual framework that underscores

common factors, barriers, and facilitators that relate to

implementing and sustaining interventions in public ser-

vice systems such as the ones described here. As described

in EPIS, the outer contexts of both of the described system

reforms were influenced by an array of political, social, and

structural factors. For example, each of these CWS systems

was involved in obtaining or implementing system-wide

Title IV-E waivers. In each case the interventions described

here represent only part of those larger waiver efforts; other

waiver activities are simultaneously taking place (such as

enhanced assessments, influences on caseload size, and

other selected services). In both examples, the intention is

84 Clin Child Fam Psychol Rev (2017) 20:78–86

123

to accomplish the implementation of the interventions

during the waiver period and to then continue to sustain the

interventions long-term after the waiver period ends.

In other ways the outer contexts were quite different for

the two case examples. Case Example 1 takes place in a

city-run system in a large urban area where all case man-

agement services are contracted out to private agencies,

and group leaders and supervisors who implement the

interventions are employees of those private agencies. In

Case Example 2 the interventions are implemented by

state-employed caseworkers and supervisors working in a

centralized state-run system that includes both rural and

urban areas. In Case Example 1 the private agencies belong

to a network of private providers which gives them a voice

to negotiate with the funders (i.e., the Administration for

Children’s Services) around specific aspects of the imple-

mentation such as caseload size reductions; a situation not

present in Case Example 2. In terms of the inner contexts,

in Case Example 1, the five participating agencies vary on

organizational characteristics as might be expected.

Examples of some of those variances include: some had

experience implementing EBPs, others did not; some

agencies specialize in serving specific populations (e.g.,

Hispanic families), others are generalists; agency size, the

number of layers of management staff, and where within

those layers the KEEP and PTC-R program champions are

situated vary as well. While there is some organizational

variance in the different DCS regions in Case Example 2,

there is also the desire by this statewide centralized system

to achieve consistency in administrative and fiscal policies

and procedures. How and whether these differences relate

to implementation success and sustainment will be

explored in future work.

Although this work represents a potential contribution

to the literature in that it describes details of how multiple

theory-driven interventions are being implemented in

public child welfare systems, there are also limitations.

Some of these relate to the fact that the interventions are

not implemented in the context of well-controlled research

trials but rather are being conducted in real-world CWSs

within the context of larger waiver-driven reform projects.

Waivers provide the motivation and opportunity for sys-

tems to make innovative reforms but they are also time-

limited. The work described here focuses on implemen-

tations enacted during waiver periods. Although the

intention is to change casework and supervisor practices

in ways consistent with the goals of the interventions, it is

unknown at this time if interventions will be sustained

post-waiver when outer contexts are likely to go through

considerable changes. Second, in each case example, there

are additional waiver activities beyond the interventions

described here that are being conducted in each system. It

is likely to be challenging to determine whether it was the

interventions per se or the interventions plus the addi-

tional waiver activities that drive any positive outcomes

that are observed. Similarly, it is unclear how the indi-

vidual interventions contribute to the overall outcomes

relative to each other within this multi-intervention con-

text. Additionally, it will be difficult to quantify how

apparently key factors such as changes in leadership at the

middle and upper management levels and changes at the

frontline caseworker and supervisor levels will affect

implementation processes and sustainment outcomes.

Partnering with qualitative researchers may improve our

ability to understand the impacts of these

inevitable changes.

The two case examples describe work that was initiated

child welfare system leaders to address the real-world

demands that they and the children and families they serve

face on a daily basis. In these two case examples we have

had the opportunity to help create systems and strategies

for caseworkers and supervisors to provide evidence-based

services directly to families and to integrate those services

into the daily work of their complex systems. This

rewarding work has included high levels of compromise,

coordination, detailed planning, and above all close com-

munication and partnerships among CWS system leaders,

program evaluators, and model developers.

Finally, as noted in Wulczyn and Feldman (in press), in

the field of health care, interventions that are synergistic

with policy, administrative, and fiscal procedures are

becoming more commonplace; (Huang et al. 2009;

Trickett and Beehler 2013), but there remain few exam-

ples of such multi-level interventions in the child welfare

literature. Moreover, child welfare systems are adminis-

tratively and financially interdependent. Single-prong

interventions that address only one level of the service

system while ignoring the interdependencies may ignore

the potential for fully integrating and sustaining evidence-

based interventions in child welfare and other complex

social service systems. The promise of such partnerships is

that the results for children and families will be optimized

through the work of frontline child welfare caseworkers

and supervisors who are supported to deliver research-

based services in synergy with their system’s internal

policies and procedures.

Acknowledgements Support for this work was provided by P50 DA035763 (PIs: Chamberlain and Fisher) and by R01 DA03634 (PI:

Chamberlain) from the National Institute on Drug Abuse, US PHS.

Compliance with ethical standards

Conflict of interest Oregon Social Learning Center owns the intel- lectual property rights on the KEEP model and Dr. Chamberlain

receives a royalties as the model developer.

Clin Child Fam Psychol Rev (2017) 20:78–86 85

123

References

Aarons, G. A., Hurlburt, M., & Horwitz, S. (2011). Advancing a

conceptual model of evidence-based practice implementation in

public service sectors. Administration and Policy in Mental

Health, 38, 4–23. [PMC: 3025110]. Akin, B. A., Mariscal, S. E., Bass, L., McArthur, V. B., Bhattarai, J.,

& Bruns, K. (2014). Implementation of an evidence-based

intervention to reduce long-term foster care: Practitioner

perceptions of key challenges and supports. Children and Youth

Services Review, 46, 285–293.

Biglan, A. (2016). The ultimate goal of prevention and the larger

context for translation. Prevention Science. doi:10.1007/s11121-

016-0635-6.

Buchanan, R., Chamberlain, P., Price, J. M., & Sprengelmeyer, P.

(2013). Examining the equivalence of fidelity over two gener-

ations of KEEP implementation: A preliminary analysis. Chil-

dren and Youth Services Review, 35, 188–193.

Chamberlain, P., Feldman, S. W., Wulczyn, F., Saldana, L., &

Forgatch, M. (2016). Implementation and evaluation of linked

parenting models in a large urban child welfare system. Child

Abuse and Neglect, 53, 27–39.

Chamberlain, P., Price, J., Leve, L. D., Laurent, H., Landsverk, J. A.,

& Reid, J. B. (2008). Prevention of behavior problems for

children in foster care: Outcomes and mediation effects.

Prevention Science, 9, 17–27.

Dishion, T., Forgatch, M., Chamberlain, P., & Pelham, W. E. (2016).

The Oregon model of behavior family therapy: From interven-

tion design to promoting large-scale system change. Behavior

Therapy, 47, 812–837. [PMID: 27993335]. Forgatch, M. S., & DeGarmo, D. S. (1999). Parenting through change:

An effective parenting training program for single mothers.

Journal of Consulting and Clinical Psychology, 67, 711–724.

Forgatch, M. S., & Patterson, G. R. (2010). The Oregon model of

parent management training (PMTO): An intervention for

antisocial behavior in children and adolescents. In J. R. Weisz

& A. E. Kazdin (Eds.), Evidence based psychotherapies for

children and adolescents (2nd ed., pp. 159–178). New York:

Guilford.

Forgatch, M. S., Patterson, G. R., DeGarmo, D. S., & Beldavs, Z. G.

(2009). Testing the Oregon delinquency model with 9-year

follow-up of the Oregon divorce study. Development and

Psychopathology, 21, 637–660.

Forgatch, M. S., Rains, L. A., & Sigmarsdóttir, M. (2016). Early

results from implementing PMTO: Full transfer on a grand scale.

In M. J. Van Ryzin, K. L. Kumpfer, G. M. Fosco, & M.

T. Greenberg (Eds.), Family-based prevention programs for

children and adolescents: Theory, research, and large-scale

dissemination (pp. 113–133). New York: Psychology Press.

Greeno, E. J., Lee, B. R., Uretsky, M. C., Moore, J. E., Barth, R. P., &

Shaw, T. V. (2015). Effects of a foster parent training

intervention on child behavior, caregiver stress, and parenting

style. Journal of Child and Family Studies. doi:10.1007/s10826-

015-0357-6.

Horwitz, S. M., Chamberlain, P., Landsverk, J., & Mullican, C.

(2010). Improving the mental health of children in child welfare

through the implementation of evidence-based parenting inter-

ventions. Administration and Policy in Mental Health and

Mental Health Services Research., 37, 27–39.

Huang, T. T., Drewnosksi, A., Kumanyika, S., & Glass, T. A. (2009).

A systems-oriented multilevel framework for addressing obesity

in the 21st century. Preventing Chronic Disease, 6(3), A82.

Kjøbli, J., Hukkelberg, S., & Ogden, T. (2013). A randomized trial of

group parent training: Reducing child conduct problems in real-

world settings. Behaviour Research and Therapy, 51(3),

113–121.

Leve, L. D., Harold, G. T., Chamberlain, P., Landsverk, J. A., Fisher,

P. A., & Vostanis, P. (2012). Practitioner review: Children in

foster care: Vulnerabilities and evidence-based interventions that

promote resilience processes. Journal of Child Psychology and

Psychiatry, 53, 1197–1211.

NSCAW Research Group. (2002). Methodological lessons from the

national survey of child and adolescent well-being: The first

three years of the USA’s first national probability sample of

children and families investigated for abuse and neglect.

Children and Youth Services Review, 24, 513–543.

Ogden, T., & Hagen, K. A. (2008). Treatment effectiveness of parent

management training in Norway: A randomized controlled trial

of children with conduct problems. Journal of Consulting and

Clinical Psychology, 76(4), 607–621.

Patterson, G. R., Forgatch, M. S., & DeGarmo, D. S. (2010).

Cascading effects following intervention. Development and

Psychopathology, 22(04), 949–970.

Price, J. M., Chamberlain, P., Landsverk, J., & Reid, J. (2009). KEEP

foster parent training intervention: Model description and

effectiveness. Child & Family Social Work, 14, 233–242.

Price, J. M., Chamberlain, P., Landsverk, J., Reid, J., Leve, L. D., &

Laurent, H. (2008). Effects of a foster parent training interven-

tion on placement changes of children in foster care. Child

Maltreatment, 13, 64–75.

Saldana, L., Chamberlain, P., & Chapman, J. E. (2016). A supervisor-

targeted implementation approach to promote system change:

The R 3 model. Administration and Policy in Mental Health and

Mental Health Services Research. doi:10.1007/s10488-016-

0730-9.

Saldana, L., Chamberlain, P., Wang, W., & Brown, H. (2012).

Predicting program start-up using the stages of implementation

measure. Administration and Policy in Mental Health Research,

39, 419–425.

Trickett, E. J., & Beehler, S. (2013). The ecology of multilevel

interventions to reduce social inequalities in health. American

Behavioral Scientist, 57, 1227–1246.

Wulczyn, F., & Feldman, S. (in press). The scale up of linked

multilevel interventions: A case study. In R. Mildon, B. Albers,

& A. Shlonsky (Eds.), The science of implementation. New

York: Springer.

86 Clin Child Fam Psychol Rev (2017) 20:78–86

123

Clinical Child & Family Psychology Review is a copyright of Springer, 2017. All Rights Reserved.