Professor Mitch 530a

O R I G I N A L A R T I C L E

Evidence Use in Mental Health Policy Making for Children in Foster Care

Justeen K. Hyde • Thomas I. Mackie •

Lawrence A. Palinkas • Emily Niemi •

Laurel K. Leslie

Published online: 25 February 2015

� Springer Science+Business Media New York 2015

Abstract Considerable attention is being given to the use

of research evidence to inform public policy making.

Building upon Weiss’s model of research utilization, we

examined the types and uses of evidence that child welfare

administrators used in response to federal policy reforms

requiring psychotropic medications oversight for children

in foster care. Participants relied on a range of ‘‘global’’

and ‘‘local’’ evidence types throughout the policy devel-

opment phase. Global research evidence was used to raise

awareness about problems associated with psychotropic

medication use. Local evidence helped to contextualize

concerns and had problem-solving and political uses. In

most states, policy actions were informed by a combination

of evidence types.

Keywords Foster care � Mental health � Policy � Evidence � Psychotropic medications

Introduction

Over the last decade, there has been growing demand for

the use of research evidence to inform the development of

public policies (Pawson and Tilley 1997; Sanderson 2002;

Marston and Watts 2003; Bowen et al. 2005). Prior studies

examining the use of evidence in policy making has found

that definitions of ‘‘research evidence’’ vary significantly

(Dobrow et al. 2004), with the broadest definition including

findings from qualitative and quantitative studies, admin-

istrative data, testimony, and personal experiences (Nelson

et al. 2009), and the narrowest including information

derived from experimental or quasi-experimental study

designs (Moore 2006). Regardless of definition, the argu-

ment for using research evidence to inform public policies

is that they impact whole populations; decisions about the

content and implementation of policy actions should be

based on the best available research rather than personal

experience, beliefs, or values (Tanenbaum 2005). Those in

favor of evidence-based policy making contend that it may

result in improved outcomes for intended populations

(MacDonald 1998).

In this paper, we present a case study of evidence use in the

development of state-level policies to address a problem with

limited research-based solutions. Specifically, we examined

the types of evidence that state child welfare administrators

acquired, reviewed, and used as they responded to federal

legislation mandating state oversight of psychotropic

medications for children in child welfare custody. Federal

mandates required states to utilize the best available research

to develop system level policies that would impact the

mental health and well-being of all children in state care.

Research is defined here as empirical findings derived from

systematic methods (qualitative, quantitative or mixed) and

analyses (Tseng 2012). Although system-level approaches to

dealing with social problems are believed to be more cost

effective and have greater long-term impact than individual

level approaches, there is often limited research on the

comparative effectiveness of policy actions at this level. One

interest of the current study was the question of how child

J. K. Hyde (&) Institute for Community Health, 163 Gore Street, Cambridge,

MA 02141, USA

e-mail: [email protected]

T. I. Mackie � E. Niemi � L. K. Leslie Tufts University School of Medicine, Boston, MA, USA

L. A. Palinkas

School of Social Work, University of Southern California,

Los Angeles, CA, USA

123

Adm Policy Ment Health (2016) 43:52–66

DOI 10.1007/s10488-015-0633-1

welfare administrators would respond given the field’s his-

torical prioritization of practical experience, professional

judgment, and perceived common sense to set public policies

that affect children and families (Jack et al. 2010).

The Policy Problem

Over the last decade, the U.S. child welfare/child protective

services and partnering youth-serving systems have been

confronted with growing levels of concern regarding psy-

chotropic medication use among children in foster

care (Naylor et al. 2007; Rubin et al. 2012). Pharmacoepi-

demiological studies examining rates of psychotropic

medication use in children have found that rates range from

13 to 30 % among children in community-based foster care

placements (dosReis et al. 2001; Raghavan et al. 2005; Zima

et al. 1999; Zito et al. 2008; Leslie et al. 2011) compared to a

rate of approximately 4 % in the general population with

private insurance (U.S. GAO 2012). Among children in

therapeutic foster care and group homes, the rates of use are

much higher, up to 67 and 77 % respectively (Breland-Noble

et al. 2005; Strayhorn 2006). Recently published research

(dosReis et al. 2005; McMillen et al. 2007; Raghavan et al.

2005; Zito et al. 2008) and government publications (U.S.

GAO 2012, 2011) have called into question the potential

over-reliance on psychotropic medication use, including

polypharmacy (i.e., combinations of psychotropic medica-

tions), to manage these children’s needs.

In response to high levels of psychotropic medication

use among children in foster care, the federal government

passed two laws over the last 6 years to address concerns

about appropriate mental health care for this vulnerable

population. The first law, the Fostering Connections to

Success and Increasing Adoptions Act (denoted now as

P.L. 110-351) of 2008, was a significant yet broad reform

that altered federal policy across five domains, one of

which was improving health and mental health outcomes

for children in child welfare custody. In September 2011,

the Child and Family Service Improvement and Innovation

Act (denoted now as P.L. 112-34) amended the state

planning and oversight requirements of the Fostering

Connections Act to include specific plans for monitoring

psychotropic prescription medication for children in foster

care. In the Informational Memorandum elaborating on the

Act, state child welfare agencies were prompted to develop

policies in five broad areas: (1) evaluation for mental health

needs and development of a treatment plan, (2) informed

consent and shared decision-making for medication use, (3)

child- and population-level monitoring of psychotropic

medication, (4) access to child psychiatric expertise at the

child- and population-levels, and (5) access to up-to-date

information about psychotropic medications (U.S.

Department of Health and Human Services 2012). In de-

veloping a response to the federal mandates to improve

oversight of mental health care generally, and psychotropic

medications specifically, the Administration for Children

and Families (ACF) placed a high value on the use of

research to inform policy responses. Although there was

research evidence to inform certain components of the

policy mandate, such as evidence based assessment and

treatment, little evidence existed regarding innovative, ef-

ficacious systems-level approaches for child welfare or

related state agencies to provide oversight of medication

use, especially for children in foster care (Geen 2009).

Evidence Framework

Although many studies examining the public policy mak-

ing process focus on factors associated with the acquisition

and use of research evidence, there is broad acknowl-

edgement that policy makers have access to and utilize a

broad range of evidence types (Bowen et al. 2009; Con-

tandrioppoulos et al. 2010; Lewin et al. 2009). In this

section, we present a framework for understanding the

different types, applicability, and uses of evidence to in-

form policy actions. (see Fig. 1) At a macro-level, we have

categorized evidence along two axes. Along one axis is the

research continuum, which ranges in the rigor of methods

and analysis used to generate information. Experimental

and quasi-experimental research studies sit on one end of

the continuum and personal experience sits at the other

end. This continuum may be defined by the extent to which

qualitative, quantitative or other types of data are system-

atically and intentionally collected, analyzed and/or used.

Along the second axis of this framework lies the applica-

bility of evidence to social problems. This axis ranges from

global to local relevance. Information produced from peer-

reviewed research studies may have global application to

Fig. 1 Evidence framework

Adm Policy Ment Health (2016) 43:52–66 53

123

policy-relevant issues with an explicit purpose of adding to

the greater knowledge about the existence of a problem,

underlying causes, and/or effective interventions. On the

other end of the second axis is local evidence, or evidence

generated to gain a better understanding of a social prob-

lem at a local level. The type of evidence one may find here

includes personal experience and testimony (e.g., experi-

ences of youth, caregivers, social workers). Other types of

evidence, such as administrative or claims data may fall

somewhere along the middle of this continuum, depending

on the quality of and strategy for obtaining and analyzing

data.

In addition to the types and applicability of evidence,

there is a growing interest in how research is used

throughout the policy making process (Weiss 1977, 1979;

Nutley et al. 2007; Lomas and Brown 2009; Contandri-

oppoulos et al. 2010; Tseng 2012). Weiss’s (1979) models

of research evidence use illustrate different ways in which

evidence can be used in policy arenas. We highlight four of

her seven original models here, as they were most appli-

cable in the current study. One model, which has broad

relevance to our findings, is the Interactive Model of re-

search evidence use. This model describes a process that

policy makers use to help gain an understanding of the

range of evidence and opinions that exist on an issue.

Researchers are one of many sources along with adminis-

trators, practitioners, advocacy groups, clients, and resi-

dents. A second general model is referred to as the

Enlightenment or Conceptual Model (hereafter referred to

as ‘‘Conceptual Model’’), which describes policy makers as

drawing on a general understanding of a body of research

relevant to a particular policy problem rather than deep

knowledge of specific research studies. Policy makers’

general understanding of research can emerge from a range

of sources, from professional journals to the popular media.

A third model portrays policy makers as active seekers of

research for specific purpose. Referred to as the Problem

Solving Model, research evidence is sought to inform

specific policy actions aimed at addressing social problems.

Weiss argues that although this is the most common con-

ception of research use, the conditions required for policy

makers to directly apply research on social problems to

policy development are complex and rarely found in

practice. The final model most relevant to current public

policy making is the Political Model, which highlights the

use of research evidence as a persuasive function, using

evidence to ‘‘neutralize opponents, convince waverers, and

bolster supporters’’ (Weiss 1979: 429). Although Weiss

developed these models with research evidence as a pri-

mary focus, they are relevant across evidence types.

In the remainder of this paper, we utilize the Evidence

Framework to organize our presentation of results regard-

ing how state level child welfare administrators acquired

and used evidence to develop system-level approaches to

oversee psychotropic medications for children in state care.

First we present a typology of evidence that state policy

makers used to inform their policy responses. We next

expand on how these different types of evidence were used

in early phases of policy making. Expanding upon Carol

Weiss’s theoretical framework regarding research evidence

use in public policy making, we highlight the utility of the

framework for the broad range of evidence used by state

policy makers across the policy development phase.

Methods

Semi-structured key informant interviews were conducted

with state policy makers in child welfare agencies in each

of the 50 U.S. states and District of Columbia. In each

state, we targeted the behavioral health or medical ad-

ministrator, or person responsible for behavioral health

care policy development in a state-level child welfare

agency. We selected these state policy makers as key in-

formants because they often occupy mid-level management

positions within child welfare agencies and are likely to

stay in their respective positions despite changes in lead-

ership from political appointments. Not only are they well-

positioned to develop and implement policies, they also

understand the organizational level contexts when consid-

ering the use of research or other types of evidence to

inform policies (Tseng 2012).

Recruitment

An introductory letter was sent to the person in each state

identified as the behavioral health or medical administrator

from national or state websites. If no behavioral health or

medical administrator was identified, we contacted the di-

rector of foster care services. The letter explained the

purpose of the study, the time involved, and the broad

category of questions to be asked. The letter also stated that

the interview could be conducted with a single person or

group of persons working on the development of the state

policy, depending on the lead individual’s preference.

Letters were followed up by a series of emails and phone

calls. The research team continued emailing and calling the

identified individual within each state until a connection

was made and an appointment to conduct an interview was

set up. In a small number of states, we were referred to

another person within the agency or in a sister agency (e.g.,

Medicaid, public mental health) who was better able to talk

about the development of the state’s oversight plan for

psychotropic medication. Recruitment of a representative

from all 50 states and the District of Columbia took ap-

proximately 14 months.

54 Adm Policy Ment Health (2016) 43:52–66

123

Description of Interviews

A semi-structured interview guide was used to facilitate the

collection of qualitative data. Based on findings from a

previous national survey on psychotropic medication

policies conducted in 2009–10 (Mackie et al. 2011), the

research team anticipated that the majority of states would

be early in the process of developing an oversight plan in

response to recent federal mandates. Table 1 provides a

description of the domains discussed during the course of

each interview. Although our initial interest in this study

was on the acquisition and use of research evidence to

inform mental health policies within child welfare agen-

cies, in our initial pilot work we found that participants

drew upon, and were influenced by, a far broader range of

evidence in this policy making context. For this study, we

thus defined evidence broadly as ‘‘information that affects

existing beliefs of decision makers about significant fea-

tures of the problem under study and how it might be

solved or mitigated’’ (Bardach 2000, p. 102). Interviews

lasted approximately 1 h. All interviews were conducted

by two members of the research team. Interviews were

audio-recorded, after obtaining permission from par-

ticipants, and professionally transcribed.

Data Management and Analysis

All transcripts were reviewed for thoroughness upon re-

ceipt. Once approved, they were entered into Dedoose

(V4.5.91), a web-based qualitative data management pro-

gram that is ideally suited for the management and analysis

of structured and unstructured data (SocioCultural Re-

search Consultants 2012). Data were analyzed using a

framework analysis approach (Miles and Huberman 1994;

Richie and Spencer 1994). Similar in many ways to a

grounded theory approach (Glaser and Straus 1967),

framework analysis is particularly well-suited for applied

research that aims to meet specific informational needs and

provide outcomes or recommendations within a short

timescale. The general approach is inductive, but allows for

inclusion of a priori and emergent concepts to be identified

in the data. Framework analysis also consists of five sys-

tematic and visible steps in the analysis process, so that

collaborators can be clear about how the results are ob-

tained from the data.

The first step entailed a process of becoming familiar

with the data by conducting a close reading of each tran-

script. The research team read the first three transcripts

available to develop a broad understanding of the content

as it related to the project’s specific aims. Each team

member identified categories of information in the inter-

views and themes within these categories. The next step

entailed several team meetings focused on identification of

emerging categories. An initial coding framework was

developed based on these discussions. The coding frame-

work included a priori and emerging categories. An ex-

ample of a priori categories included types of evidence,

uses of evidence, stage of policy development, cross-sys-

tem collaborations, and details of child- and population-

level oversight strategies. For each agreed upon category of

information to be coded, the team developed a definition

and inclusion criteria. Once the coding framework was

drafted, two team members individually applied it to a

subsequent sample of 2–3 transcripts to assess general

applicability for the data. The team met again to review the

utility of the framework, make modifications, and finalize

the framework.

Table 1 Domains of semi-structured qualitative interview

Domain Description of measures Question type

Perceived priority level of

psychotropic medication use

Rating from 1 to 10 of how much a concern psychotropic medication use among

children in child welfare custody is in state.

Rating scale

Phase of policy making Perceived rating of where each state is with respect to its oversight plan for

psychotropic medication use.

Rating scale

Evidence use in developing policy

action

Types, sources, and uses of evidence used to help prioritize the problem, mobilize

support for action, and develop policy actions, if applicable.

Open-ended and

close-ended

Evidence use in implementing or

refining policy action

Types, sources, and uses of evidence used to evaluate oversight plans previously put

into place, if applicable.

Open-ended

Current policy components Key components of plans to oversee psychotropic medication use at the child- and

population-levels (i.e., population covered, classes of medications covered, specific

practices for child-level monitoring, specific practices for population-level

monitoring).

Open- and close-

ended

Collaborators Information about who the child welfare agency is collaborating with to develop and

implement oversight plans, and how they are collaborating.

Open- and close-

ended

Informational needs What kinds of evidence child welfare administrators want and need to develop,

implement, and monitor their oversight plans.

Open-ended

Adm Policy Ment Health (2016) 43:52–66 55

123

The third step entailed applying the coding framework

to the remainder of the data. Two members of the team

double coded 20 % of the interviews to ensure consistency

in how codes are applied. They then independently coded

the remaining interviews, with one team member (JH)

checking all coding upon completion. The team met

regularly to review coding questions and identify new in-

formation that could not be appropriately categorized

within the established framework. Questions and new in-

formation were dealt with using a consensus-based deci-

sion-making approach.

Once all of the data were coded, the team began

‘‘charting’’ the data (Step 4). This is a process of reviewing

‘‘reports’’ on each code to gain a deeper understanding of

the information captured. The team began by charting in-

formation coded on the types and uses of evidence in the

development of oversight policies for psychotropic

medications. Qualitative data charts entailed the code and

its definition, a summary of themes emerging from the

coded information, and illustrative quotes. The team then

met to interpret the data (Step 5), which involved review-

ing charts and agreeing to common and unique themes

across states and respondents. We then looked at code co-

occurrences to search for patterns, associations, concepts,

and explanations in the data. The results of this data ana-

lysis process are presented below. This study protocol was

reviewed and approved by the Institutional Review Board

at Tufts Medical Center.

Results

Description of Sample

A total of 72 key informants in mid-level manager posi-

tions were interviewed, with an average of 1.4 key infor-

mants per state (range of 1–5 key informants per state).

Informants represented a range of agencies, including child

welfare agencies (n = 57, 79.2 %), Medicaid and non-

profit affiliates (n = 7, 9.7 %), parent agencies to child

welfare (n = 3, 4.2 %), Departments of Health (n = 2,

2.8 %), contracted consultants (n = 2, 2.8 %), and a

Department of Mental Health (n = 1, 1.4 %). At least one

respondent per state was a child welfare policy maker,

meaning that they were responsible for developing and

often overseeing the implementation of state-level policy

actions for their agency. While all participants had earned a

bachelor’s degree (BA/BS; n = 72, 100 %), additional

academic training included: Master of Social Work (MSW;

n = 22, 30.6 %), Child and Adolescent Psychiatry (MD;

n = 6, 8.3 %), Registered Nurse (RN; n = 6, 8.3 %), Pe-

diatrician (MD; n = 4, 5.6 %), Juris Doctorate (JD; n = 3,

4.2 %), other Medical Doctorate (MD; n = 3, 4.2 %),

other Master’s level degree (MA/MS/MPA; n = 3, 4.2 %),

Doctorate of Pharmacy (PharmD; n = 2, 2.8 %), Doctoral-

level psychologist (PhD; n = 2, 2.8), other Doctorate of

Philosophy (PhD; n = 1, 1.4 %), and Nurse Practitioner

(NP; n = 1, 1.4 %). On average, participants had held their

respective positions for 5.5 years.

Phase of Policy Making

Key informants were asked to assess where their agency

was in the phase of policy development and implementa-

tion based on a four-point implementation scale that

included: ‘‘not started at all,’’ ‘‘development,’’ ‘‘imple-

mentation,’’ and ‘‘evaluation and refinement.’’ All states

had initiated some type of response to federal policy

mandates. The majority of states (67 %) were in the

development phase of policy making. This was a broad

phase that included efforts to: (1) prioritize the scope of the

problem at the national level and within the local context of

their own state, (2) mobilize key stakeholders toward

policy action, and (3) develop policy action(s) (see Fig. 2).

Those in the implementation phase (9.5 %) had developed

plans within the last 1–2 years and were formally or in-

formally assessing the feasibility and effectiveness of their

plans before implementing them across the state. Finally,

about a quarter of states (23.5 %) were described as being

in full implementation of most if not all components of

their plan and were evaluating their current plans and en-

gaging in quality improvement efforts. Participants from

these states reported that they developed oversight policies

in response to class action lawsuits or other legal actions

taken in response to harm associated with psychotropic

medications use among children in child welfare custody,

some as long as two to three decades before the passage of

the federal legislation.

With the majority of states reporting being in the de-

velopment or early implementation phases of a policy re-

sponse (76.5 %), we limit our examination of how research

evidence is used to inform child welfare policy making to

this initial stage. This is a critical stage in policy making as

the types of evidence used shapes policy makers’ under-

standing of the issue, which in turn influences the selection

of policy actions (Lavis et al. 2009). In the absence of

strong evidence that one systemic approach to psychotropic

medication oversight is better than another, the develop-

ment of oversight strategies will be dependent to a large

extent on what policy makers know and understand about

the problem.

As portrayed in Fig. 2, the development phase of policy

making was comprised of three components. The first

component, ‘‘prioritization,’’ included cultivating an un-

derstanding of the global problems or concerns associated

with psychotropic medication use among children in foster

56 Adm Policy Ment Health (2016) 43:52–66

123

care, and then localizing the problem by exploring the

extent to which more global trends were relevant to their

own state foster care population. Once the problems were

localized, policy makers and their collaborators then

needed to mobilize for potential policy actions and make

decisions about which were the most appropriate for their

state. Mobilizing support and resources to develop policy

actions was variable among participants, with some need-

ing to start at the beginning with education and partnership

building, and others taking advantage of partnerships that

were already focused on behavioral health issues. In the

last component of this stage, states developed specific

policy actions to take. Although presented as a linear

process, policy making is often iterative, with deviation

occurring from one context to another. Figure 2 should

thus be interpreted as a heuristic for evidence use across

early phases of policy making.

Types of Evidence Used in the Development Phase

of Policy Making

State-level child welfare administrators involved in mental

health policy making described the need to consider and

negotiate multiple and often competing interests and de-

mands when prioritizing, mobilizing, and developing pol-

icy actions to ensure oversight of psychotropic medications

for children in foster care. This particular policy initiative

involved a number of stakeholders, including but not lim-

ited to the federal government, state government, county

government (if administered at that level), Medicaid and

managed care organizations, mental health providers,

pediatricians, social workers, caregivers, and foster youth.

During the course of this study, these stakeholders also

operated within changing local, state, and federal contexts

impacted by economic austerity measures and significant

reforms in how health and mental health services are pro-

vided under the Affordable Care Act. Given the complexity

of the issue and the multi-disciplinary collaborations re-

quired by federal mandate, it is not surprising that par-

ticipants described relying on a broad range of evidence to

help inform decisions made during the development phase

of policy making. Table 2 provides an overview of the

different types of evidence key informants most commonly

reported acquiring and using.

The study team initially anticipated that key informants

would make clear distinctions between the value of re-

search based evidence and evidence obtained through more

ad hoc or informal ways. This was not found. Rather, as

described below, we found that different types of evidence

were acquired, used, and valued at different stages of the

policy development phase.

Approach to Policy Making

Key informants largely described an Interactive Model of

evidence use as they approached the development of their

state oversight policies. In this model, ‘‘all kinds of people

involved in [the] issue area pool their talents, beliefs, and

understandings in an effort to make sense of a problem… and progressively move closer to potential policy re-

sponses’’ (Weiss 1979: 428). In the vast majority of states,

working groups provided the primary vehicle for bringing

Fig. 2 Heuristic for research evidence use across the development phase of policymaking

Adm Policy Ment Health (2016) 43:52–66 57

123

different stakeholders together. Members of these working

groups comprised a critical social network for child welfare

policy makers through which evidence was shared, debat-

ed, adopted or disregarded. Most working groups were

minimally comprised of mid-level managers and supervi-

sors within child welfare agencies, mental health experts

(either within or external to the agency). All states even-

tually had representatives from state Medicaid offices as

this was a requirement of the Improvement and Innovation

Act (P.L. 112-34). Finally, many also had direct or indirect

input from consumers, including youth in out-of-home

placements, and foster or biological parents. The need for

this diversity in experience and expertise initially felt like

‘‘an arranged marriage’’ in states, but most eventually

found it to be important and necessary.

I think we are at the first trimester of developing our

plans. We have lots of thoughts and ideas and gathering

a lot of information, but specifically what our plan

looks like – we just aren’t there… We have our con- nection with medical services where we can gather

information on all of the kids that are in custody on

psychotropic medications. We have the opportunity to

work with the division director for Community Be-

havioral Health to really think about how we involve

our community mental health centers… We have to bring our group and residential people to the table. And

then there are the psychiatrists, other docs, and psy-

chiatric nurses at the Human Services Center…The clinical director at the Human Services Center sits on

our DSS Management Team along with other division

directors. We need all these people at the table because

we can’t do it ourselves. We’re not experts. Our field is

social work. We are not doctors.

The notion that child welfare policy makers cannot

act in isolation to develop policies to oversee behavioral

health care generally, and psychotropic medication use

Table 2 Types and sources of evidence used to inform mental health policymaking in Child Welfare Agencies

Category

of

evidence

Types of

evidence

Definition Example

Research

evidence

Administrative

data

Data collected primarily for administrative purposes, such

as registration, transactions, and record keeping. May be

used to analyze conditions or trends found within or

across agencies.

Medicaid claims data, child welfare data

(SACWIS database), or other agency data used

in epidemiological studies

Mental health

research or

evaluation

studies

Evidence regarding effective treatment practices and how

they work based on systematic study.

Published studies examining mental health

conditions, trends in psychotropic medication,

effective treatment modalities, qualitative

inquiries focusing on mental health care

experience

Government

investigative

reports

Research studies commissioned by the federal or state

governments to understand the scope and underlying

issues associated with identified social or health

problems. Studies typically undergo internal review

before implementation and dissemination.

Non-partisan technical reports or white papers that

are issued from state or federal General

Accounting Offices (GAO), Centers for Disease

Control (CDC), Environmental Protection

Agency (EPA)

Professional

guidelines

Consensus-based guidelines developed to support practice

and/or policy decisions. Guidelines may be a

combination of local and research evidence used by a

group of professionals to inform guideline content.

Professional organizations (e.g., ACAAP, ACAP),

or ad hoc work groups assembled by

governmental or intermediary organizations

Expert opinion Information obtained through consultation or input

elicited from a professional within the state who was

knowledgeable about one or more aspects of an issue.

Practicing health and mental health professionals,

pharmacists, researchers in state or local

agencies or academic institutions

Local

evidence

Testimony Experience shared with or observations made by key

stakeholders, including consumers, service providers,

administrators, and policy makers. May be provided in

person, writing, or through media channels.

Input from consumer advisory boards, standing

and ad hoc stakeholder meetings, local or state

media producers. Also includes information

shared by child welfare administrators in other

states with experience developing and/or

implementing components of a policy.

Personal

experience

Experiences or observations made by key informant that

inform his/her understanding of an issue.

Stories or anecdotes provided by key informants to

explain his/her knowledge of the problem, how

to mobilize support, or types of policy actions

needed

58 Adm Policy Ment Health (2016) 43:52–66

123

specifically, was heard repeatedly during the course of the

study. Some states had a long history of collaborative work

and involvement of multiple stakeholders across multiple

service sectors to identify and address problems within

their child welfare system or across child serving systems

more generally. Most, however, reported that cross-agency

and multi-stakeholder collaborations were sparked by

federal policy initiatives, especially the Improvement and

Innovations Act (P.L. 112-34). Regardless of whether or

not states had formed these collaborations before the fed-

eral policies were established or after, the collaborative

approach opened up the door to many different types of

evidence being acquired, shared, and debated among child

welfare policy makers and other key stakeholders in each

state.

Understanding the Problem

Global Knowledge

Research evidence was by and large the most commonly

identified type of evidence that key informants identified

when explaining their global understanding of the problem

of psychotropic medication use among children in foster

care. Two types of research evidence were most commonly

identified: (1) pharmacoepidemiological research in peer-

reviewed journals and government-funded investigative

reports; and (2) clinical mental health research and

evaluation studies on the benefits and side effects of psy-

chotropic medications. Participants’ descriptions of the use

of research evidence at this stage were aligned with

Weiss’s description of a Conceptual Model of evidence use

(1979). Few participants were able to talk about the

specific research studies they reviewed to inform the

problem of psychotropic medication use in their state.

Rather, they had a general sense of the published trends

towards higher rates of use and polypharmacy for children

in foster care.

I don’t know that we formally sat down and said

‘‘let’s pull a lot of research and use it to make deci-

sions…’’ We have talked to specialists at our uni- versity medical centers. They have provided us

information during consultation. I have personally

been to conferences where I’ve heard people speak on

findings. Just this past year I heard your university

representative come and speak at the policy to prac-

tice forum for child welfare in DC. So we’ve had

information from research based studies but we

haven’t necessarily sat down and said we’re going to

pull together a bunch of research in the field.

Similar to research studies published in peer review jour-

nals or presented at professional forums, key informants de-

scribed a few government-funded investigative reports that

were important to their understanding of the scope of the

problem. In particular, many referenced a federal report re-

leased by the GAO in December 2011. This report synthesized

clinical research on the effects of psychotropic medications on

children and highlighted psychotropic medication trends

among children in foster care in five states (GAO 2011). For

states interviewed who had been involved in the GAO’s study,

the findings provided not only global evidence of the problem,

but also local evidence. As will be described later, this local

evidence helped to mobilize support and action for some

working groups. For others, the report not only helped to raise

awareness about the scope of the problem, but also provided a

benchmark for interpreting their own state’s pharmacoepi-

demiological data (if available).

Many participants provided similar narratives regarding

being exposed to research studies through a variety of dif-

ferent forums, like conferences and webinars. For example,

in early 2012, the ACF supported a series of webinars fo-

cused on the issue. The first one helped frame the problem,

providing pharmacoepidemiological data from research

studies and two states that have implemented oversight

strategies. Others featured practice-based evidence drawn

from state administrative data and testimony from select

state child welfare agencies that had oversight policies in

place. The majority of participants interviewed after the

launch of these webinars reported participating on-line and

learning about both the research- and practice-based evi-

dence related to psychotropic medication use and oversight.

About a quarter of participants also discussed how re-

ports highlighting increasing or disproportionately high

levels of psychotropic medication use among children in

foster care in combination with growing clinical evidence

on the side effects of psychotropic medications helped raise

the priority level in their state.

Interviewer

(INT):

And what kind of research in particular

have you been using?

Respondent

(R):

I think the bulk of it is just talking

primarily about the impact of psychotropic

med’s based on youth and what we don’t

know about. That’s the large amount that I

have been looking at, because that is one

of my major concerns is that we don’t

know what the side effects are and the long

term residual effects on children will be

from all of this medication. And the fact

that a lot of the medication that has been

prescribed has not been checked for youth

Adm Policy Ment Health (2016) 43:52–66 59

123

Most child welfare policy makers acquired research

evidence on psychotropic medication use among children

in foster care through intermediaries, such as workgroup

members from other state agencies, national non-profit

organizations, foundations focused on children (e.g., Casey

Family foundations), and ACF. However, a few key in-

formants, such as the individual quoted above, described

themselves as being active seekers of research evidence.

These individuals were often doctors serving as medical or

mental health directors within a child welfare agency or

closely aligned with the agency. These intermediaries were

critical in providing access to specific types of research

evidence that were disseminated to the state policy makers,

such as peer reviewed articles or technical reports arising

out of federal investigation.

With the exception of the few key informants who

considered themselves up-to-date on the latest clinical re-

search around psychotropic medications, most participants

were not able to recall specific studies that influenced their

global understanding and concern around psychotropic

medication use among children in foster care. However,

they described a conceptual understanding of the problem,

which was enough to drive an interest in how the global

problem looked within their own states.

Local Knowledge

Regardless of whether key informants described their states

as being early in the development of their respective

oversight plans or into full implementation, nearly all de-

scribed a process of ‘‘localizing the problem.’’ By this we

mean understanding and defining the problem of psy-

chotropic medication use and oversight within their own

states. In this shift from gaining a global to local under-

standing, policy makers described a change in how evi-

dence was used, from Conceptual to Problem-solving and

Political models.

Participants reported acquiring and drawing primarily

on administrative data and testimony to define the breadth

and depth of concerns around psychotropic medication use

among children in their own child welfare system. Re-

flecting Weiss’ conceptualization of the Problem-solving

Model, these state administrators sought to answer the

following questions: (1) How many young people in state

custody are prescribed psychotropic medications? (2) What

types and combinations are being prescribed and how does

this vary by age, placement type, provider type, or geog-

raphy? and (3) What impact does this have on the children

in state custody and the providers who work with them?

State administrative data were employed to address the

first two questions. The two most common types of ad-

ministrative databases reported were the Statewide

Automated Child Welfare Information Systems (SACWIS)

and Medicaid claims data. Both data sets had challenges.

SACWIS data were not usually considered to be the most

reliable, as mental health information is often captured in

narrative fields and the consistency of data entry for mental

health information is varied. Medicaid data were developed

primarily for billing purposes and not for research or

clinical care. There were considerable differences across

states with respect to access to and use of administrative

databases. States in the early phases of developing their

oversight plans most often reported ‘‘working towards the

goal’’ of accessing systematically collected or population-

level data available in state administrative databases. These

states often relied on ad hoc data pulls from their own child

welfare or Medicaid databases. Participants from two states

reported creating separate databases to specifically track

mental health assessment and/or treatment information,

including psychotropic medication use.

…we are gathering the numbers of our children that are in foster care placements that have been pre-

scribed and are taking psychotropic medication. So

that is a number that is reported to us by providers. Is

it research based? No. It is a self-report from our

providers as to whether or not children in custody and

in placement with those providers are taking psy-

chotropic medications. So we are looking at trends

and raw data, but not any research based data.

Administrative data based on Medicaid claims infor-

mation were often considered to be an important and reli-

able source of evidence about psychotropic medication use

among children in foster care. However, accessing this data

required relationships with Medicaid agency representa-

tives who understood the need for the data. Cross-agency

relationships were still forming for some states and were

considered to be necessary in order to access and use this

type of local evidence. Many key informants reported they

had little in-house capacity to pull and analyze these data

on their own, so acquiring and using Medicaid claims data

required analytic support from Medicaid. Contrastingly,

participants from states with strong cross-agency col-

laborations between child welfare and Medicaid had a good

sense of the breadth of the problems associated with psy-

chotropic medication use among children in their state. As

one child welfare policy maker with access to real time

Medicaid data explained:

We’ve looked at the number of kids that are on

medications broken down by age group and we’ve

looked at the number of medications the individual

kids are on to address both polypharmacy and the

specific types of medication they are on, like atypical

60 Adm Policy Ment Health (2016) 43:52–66

123

medications. This heightened our concern. A lot of

kids - even young kids - are on multiple medications

in significant doses. Some on antipsychotics without

good indication. So we’ve used that kind of data to

quantify and understand the problem from a clinical

standpoint.

Several key informants also highlighted the ways in

which global research evidence was used to inform the

types of local evidence that they might examine in their

own state. For example, some key informants described

using research studies based on Medicaid claims data to

determine the kinds of data that they should pull from their

own administrative databases. Published research using

claims data provided some guidance on the types of data

that were possible to examine and that, if pulled, would

permit comparisons to data reported in other states.

Our Medicaid folks have taken a look at a study that

was published in Pediatrics in November 2011. And

what they did is they took our data, and they com-

pared our data with what was published for the

population from the Mid Atlantic states in that Pe-

diatrics article. And we had to have some clarifica-

tions about the data used to make that comparison.

Then they analyzed what they found and we dis-

cussed the implications. I don’t know that we rou-

tinely look at research based articles or other

information in journals, but certainly at that time we

did utilize that.

In addition to the rates of psychotropic medication use,

characteristics (e.g., age) of the children using medications,

and the types, doses, and combinations of psychotropic

medications prescribed to children, administrative data

were also used to explore variation across the state. For

example, participants with access to administrative data

and analytic capacity reported interest in examining whe-

ther there were differences in psychotropic medication use

by demographic variables, placement type (e.g., foster

family vs. congregate care), geographic area (urban vs.

rural), or provider type (psychiatrist vs. pediatrician).

Do we need to do a targeted approach toward certain

counties? Do we need to do training for certain

physicians? I think [administrative data] will help

inform what type of intervention we need to do as

well as any kind of policy changes. I mean I would

think that there would be some targeted training that

would be helpful, but we need to understand exactly

what the training needs are.

Systematic use of administrative data helped policy

makers understand the scope of the problem in their state

and potential places in need of intervention. Testimonial

evidence was also used to understand the scope of problem

and to mobilize or persuade action, reflecting both Weiss’s

Problem-Solving and Political Models, and addressing

questions about the impact of medication use on children in

foster care. The majority of states had consumer advisory

boards comprised of some combination of youth, biological

caregivers, foster parents, and occasionally residential

treatment or group home providers. Sometimes the con-

sumer boards met regularly; other times they were brought

together on an ad hoc basis to focus on a particular issue.

Key informants reported that youth involved in these

consumer boards often described not being involved in

personal clinical decisions about appropriate treatments,

and psychotropic medications being an easy treatment

strategy but not always the right one. For example, one

participant described what their state learned from youth:

The young people were able to give us good insight

into - - so the use and misuse of medication is one

issue that was talked about, particularly among youth

in residential or congregate care. They perceived

medications as being a disciplinary tool; if a child

didn’t take their medication they were put on a lower

level and not given certain rights or if staff consid-

ered them out of control, they were medicated rather

than provided treatment options.

Personal experiences of key informants were also used

to understand and localize the scope of the problem around

psychotropic medication use. Many key informants had

been working within the child welfare system for years.

When asked about the evidence they drew upon to pri-

oritize the issue of psychotropic medication use in their

state, about half of participants provided one of three

narratives grounded in personal experience. The first of

these three narratives was to use personal experience to

highlight the scope of the problem. For example, when one

participant was asked how she knew about increasing rates

of use in her state, she explained:

I review a lot of child welfare cases, interview peo-

ple, and also look at the actual records. I think that

there has been an increasing awareness that many

children are receiving medications while in our cus-

tody. Recently I participated in what are called per-

manency round tables that are sponsored by Casey

Family Programs. I reviewed ten cases. Nine of those

cases, mostly youth over age ten, were receiving

psychotropic medications.

The second of these narratives was to understand some

of the circumstances making it difficult for front line social

workers to assure that children in their care receive ap-

propriate mental health treatment. In their current positions

as mid-level managers within their child welfare agency,

Adm Policy Ment Health (2016) 43:52–66 61

123

participants reported hearing countless stories from front

line staff about the challenges faced in assessing and

making sure that a medical provider’s recommendation for

psychotropic medications was safe and appropriate. Power

dynamics between medical providers and social workers as

well as lack of knowledge and training on psychotropic

medications were commonly reported. Some participants

also noted having these same experiences when they were

front line workers.

Finally, some drew on personal experience to highlight

more systemic problems regarding mental health care

generally, and psychotropic medications specifically. Per-

sonal experiences were recounted not only to understand

the root of the problem, but also to prompt thinking about

what needed to be done to improve behavioral healthcare

for children in child welfare custody. As one participant

explained:

I came up through a masters program, with an in-

ternship - and what happens when you do internships

in the MSW program?… Interns get put in commu- nity mental health or the county mental health

agencies and then who do they get assigned? Foster

kids. And, what are they going to treat them for?

Reactive attachment disorder. Okay, so, this kid

doesn’t attach so why did you give him a student to

attach to who’s going to be gone in six months?

…and I understand, they’re trying to do all of this work and they use those interns to come in and help

out with staffing issues, but the capacity and expertise

that these foster kids need is something very different

in my mind. So, part of it is through my experience

saying there’s got to be other ways of doing this.

Although not usually persuasive by itself, testimonial

evidence helped to humanize the data obtained from re-

search and administrative databases. The broad represen-

tation of people connected to foster youth on working

groups, from youth themselves to providers that prescribe

their medications, helped to ground the administrative data

that was reviewed and deepen participants’ sense of how

and why the global and local data trends were important.

Thus, these data served a critical political function.

Mobilizing Support for Action

There was considerable variation in the types of evidence

that participants used to mobilize stakeholder groups

within a state to take part in the development of policy

actions. For some, testimonial evidence from stakeholders,

youth, caregivers, and social workers provided an emo-

tional and often personal connection to the issue. It pro-

vided fuel for work groups to continue working together to

make conditions for children in foster care better (i.e.,

Weiss’ Political Model) and highlighted particular priority

areas to address (i.e. Weiss’ Problem-solving Model).

Some states, however, indicated that testimonial evidence

was not enough to secure the support and resources needed

to develop statewide policies. For a few participants,

government investigative reports, such as the 2011 GAO-

funded report on psychotropic medication trends among

children in foster care had the necessary political weight

and reach needed to spark action. As one child welfare

administrator noted,

I mean a lot of people read that report so I think it just

gives a broad message to a lot of other folks who

have not been immersed in it and in some ways that

helps because it gives us more support and probably

more willingness to collaborate and share some re-

sources… We really have to collaborate more closely to be able to look at the issue, to be able to share data.

So I think that’s really given us a lot of impetus to

move things forward.

Participants also indicated that stakeholders in the policy

development process were not persuaded by the same types

of evidence. Recognizing the need for input and support

from mental health, child welfare, Medicaid, and con-

sumers, some participants reported being strategic in their

use of evidence with these different stakeholders, paral-

leling Weiss’ Political Model. Medical and mental health

providers were often identified as being the most chal-

lenging but critical to engage as the policies being devel-

oped would likely affect their practices. Understanding the

type of evidence that would move these stakeholders was

important.

…the primary literature that I looked at was also very important, partly in getting medical partners on

board. I was able to point the pediatricians that we

work with to the article in Pediatrics, the recent ar-

ticle in Pediatrics, to get them more on board with the

issue, things like that…when I can point them to something that’s been published in a peer-reviewed

article – that helps them more than if I give them the

Government Accountability Office report.

As the quote above suggests, ensuring that stakeholders

in the policy development process have the right kind of

evidence to persuade their involvement in the development

of or support for policy actions is a political move, de-

signed to persuade and mobilize action.

Developing Policy Actions

The mandate to oversee psychotropic medication at the

child and population levels required determining what

practices should be in place to ensure appropriate mental

62 Adm Policy Ment Health (2016) 43:52–66

123

health treatment and the indicators that can be monitored to

assess compliance with policies. While states worked to

develop strategies to ensure that youth in foster care re-

ceive safe and appropriate psychopharmacological treat-

ment for youth, the question of how to monitor and assure

compliance with these strategies was critical. Most key

informants reported utilizing prescribing guidelines pub-

lished by professional organizations such as the American

Academy of Child and Adolescent Psychiatrists (AACAP)

(Brown 2007; AACAP 2009) or American Pediatric As-

sociation (APA). The guidelines provided recommenda-

tions on when mental health assessments should be

completed for children entering foster care, types of

treatments appropriate for certain mental health conditions,

and outlier practices or ‘‘red flags’’ suggesting potentially

unsafe use of psychotropic medications. Although not

usually used alone, these guidelines were often considered

to be among the most useful sources of evidence as they

provided concrete guidance around certain practices.

Our Department just required that services comply

with the AACAP’s best practice parameters so they

have two or three best practice parameters on

medication usage. Also we have a practice guideline

for children zero through five in terms of psy-

chotropic medications, what can be used, what can’t

be used, what is the check off list, who can approve

it, what oversight is done on those. So we have these

very specific guidelines for the zero through five

populations.

While evidence regarding efficacious treatment and

monitoring approaches for children with mental health

problems exist, many participants questioned how this

evidence could be adopted into system-level policies in

their own state. For example, key informants in states with

large rural areas expressed concerns with equity of care and

the limited mental health expertise that is often available to

deliver evidence based treatments. Mandating the use of

evidence based treatments for children in foster care may

require more resources, such as funding for additional

mental health expertise and supervision, in order to be

responsibly implemented. However, local knowledge of the

context within each state (e.g., recent budget cuts, political

culture) played a role in decision-making. This example

highlights the complexity of factors that influence the

adoption of research evidence in public policies. Par-

ticipants did not describe a ‘‘knowledge deficit’’ to be fil-

led, but a process of utilizing local evidence to determine

what was feasible and necessary to include in policy.

One final source of evidence that key informants re-

ported to be valuable was the practice-based experience of

state child welfare administrators in other states who have

developed strategies to oversee psychotropic medications

for children in state care. One national study was con-

ducted shortly after the Fostering Connections Act was

passed in 2009 to understand what strategies state child

welfare agencies had to oversee psychotropic medications

(Mackie et al. 2011). About one-third of participants re-

porting using the technical report distributed to states from

that study to gain a sense of how other states were ap-

proaching different components of an oversight plan

(Leslie et al. 2010). Key informants also described a

number of less formal ways that they were able to learn

from other states’ experiences, such participating in we-

binars, conferences, and other cross state networks that

allow for peer-to-peer sharing.

What I am finding most helpful now is finding out

what other states are doing. And what they are

looking at… Most of it can be options that are available out there and listening to how some of them

have been successful, the pros the cons.

Those states that were formally asked to share their over-

sight strategies by intermediaries, such as the ACF or the

Casey Family Programs, typically had practice-based evi-

dence regarding their approach, including resources needed

for implementation, the strengths and challenges of their

oversight models, and data on changes in psychotropic

medication use. These states were classified as being in the

last phase of policy making—evaluation and policy

refinement.

By and large, participants described a Problem-Solving

Model of evidence use in the policy development process

as they toggled back and forth between research evidence

and the group’s local knowledge of their state’s child

welfare and mental health systems to determine what pol-

icy actions were reasonable and appropriate to recommend.

When asked about types of data most useful in the policy

development process, one participant noted:

R1: I guess personally, I would say that whatever we

read and discuss is very interesting but I think when

we are talking with our sister agencies, particularly

the Department of Mental Health and amongst the

social workers and our long term disciplinary teams

it is most useful. Probably our personal information

is what we use the most

R2: But I also think that it links back to the Child

Welfare League of America or the American

Academy of Child and Adolescent Psychiatry

because although it is personal, we still link it back

to see what’s in the literature

Local knowledge about the mental health resources and

supports within the state served as an important sieve when

Adm Policy Ment Health (2016) 43:52–66 63

123

examining oversight models found in other states or

guidelines developed by pediatric and psychiatric profes-

sional associations or working groups. Participants de-

scribed discussing these types of evidence within their

groups and making assessments regarding which were

feasible given available resources, geography, and organi-

zational structure.

Discussion

In the conclusion to her 1979 article titled, The Many

Meanings of Research Utilization, Weiss argues that re-

searchers need to change the questions asked of policy

makers, from how to improve use of research in public

policy making to how to improve the contributions that

research can make to ‘‘the wisdom of social policy’’ (431).

Findings from this study contribute to this reframed ques-

tion. In particular, our study highlights the importance of

understanding that research evidence alone is not enough to

inform the development of complex, system-level policies.

In this case example, child welfare administrators relied on

both research and local evidence to develop policies aimed

to improve oversight of psychotropic medications for

children in foster care. Far from being driven solely by

political interests or personal values, most participants

described their state process as being thoughtful, proactive,

and well-rounded in terms of the types of evidence they

were able to draw on to inform their policy making pro-

cesses. By expanding inquiry beyond research evidence,

we were able to examine the contributions of different

types of evidence across the phases of early policy making.

From this vantage point we gained a better understanding

of what evidence was useful to policy makers as well as

how and why it was useful.

Advocates of evidence-based policy making argue that

policy decisions that affect whole populations should be

based on the best available evidence rather than politics,

personal beliefs, or social values. Although well-inten-

tioned, our findings suggest that the best available evidence

is not enough in and of itself to understand the problem,

mobilize for action, and develop effective and feasible

social policies; knowledge of political contexts, values,

beliefs, and resources also are essential ingredients in the

development of public policies.

In addition to finding different types of evidence used to

inform early phases of policy making, we also found that the

same evidence was used in different ways. For example,

evidence from administrative data regarding trends in psy-

chotropic medication was valuable in helping to localize

policy makers understanding of the scope of the problem

(i.e., Weiss’ Problem-solving Model). When compared with

research-based evidence from pharmacoepidemiological

studies, or testimony from youth, administrative data

sometimes reflected Weiss’ Political Model as well. With

respect to the overarching approach to policy making, we

found Weiss’ Interactive Model of evidence use to be useful

and participants described how a variety of different types

and sources of evidence were needed to inform early phases

of policy making. In theoretical models of evidence use in

policy making, rarely is it acknowledged that certain types of

evidence may have multiple uses. Expanding our under-

standing of evidence use in public policy making contexts

will require moving beyond isolated typologies to show the

dynamic and transactional ways in which global and local

evidence is used.

Our findings on the use of both research and local evi-

dence also contribute to recent work highlighting the in-

creasingly blurred line between research producer and

consumer (Davies and Nutley 2008). Policy makers and

practitioners may not only receive and influence re-

searcher-produced evidence (as emphasized in dominant

paradigms), but may actually produce evidence them-

selves. In this study, policy makers gathered a range of

local evidence, including testimonial evidence from key

stakeholders and analyses of Medicaid claims or other

administrative data sets, and synthesized best practices

when the information was available for local work groups.

In addition, some child welfare policy makers also con-

tracted or partnered with researchers to co-produce re-

search evidence on the scope of the problem (e.g., Zito

et al. 2008). Consistent with the call to expand the notion of

evidence-informed policies, research on contemporary

public policy making also needs to reconsider the tradi-

tional lines that are drawn between research producers and

users of evidence.

Public policy making and the use of local and research

evidence is not, in practice, a linear process. Although we

found that evidence acquired in one step (e.g., problem

definition) did inform subsequent steps (e.g., policy ac-

tions), participants also described an interactive process at

each policy phase whereby a range of different types of

evidence were reviewed in isolation and together to shape

understandings and actions. Related questions for future

study include the ways within which certain types of evi-

dence are privileged or adapted to accommodate particular

moments in the policy development process, and factors

associated with the user, the organization, and socio-po-

litical contexts.

Study Limitations

This study had several limitations that are important to

consider. First, the data collected are based on the per-

spectives and experiences of a single individual (in many

64 Adm Policy Ment Health (2016) 43:52–66

123

states) or a small group of individuals (in a few states).

Although we selected individuals based on their central

role in developing mental health policy for the state’s child

welfare agency, these individuals were not the only ones

involved in the process in their state and we know little

about the perspectives of others who contributed to this

process. These individuals may have differences in opinion

about the types of evidence that have been useful in the

policy development process. Future work we are conduct-

ing involves case studies exploring the perspectives of

multiple stakeholders in a state.

A second related limitation to the methodological ap-

proach is associated with the reliance on a small number of

representatives to communicate about a complex policy

change in a short amount of time. Interviews generally

lasted about an hour. Although this was adequate time to

gain a high level overview of each state’s policy devel-

opment process, we do not have direct observational data

to either confirm the reports of participants or gain insights

into the deliberative exchanges that likely occur between

different social actors when evidence is introduced into a

policy discussion.

A third limitation stems from the dynamic policy envi-

ronment in which the study was conducted. Shortly after

the study launched, the ACF issued another policy reform

(PL 112-34, Improvement and Innovations Act). As part of

this policy reform effort, ACF played an important role in

disseminating research- and practice-based evidence re-

garding the scope of the problem around psychotropic

medications and examples of oversight strategies that other

states have developed to help ensure appropriateness of

mental health treatment. The study team was able to ex-

amine in real time how the dissemination of this type of

evidence was acquired and used at the state level. The

challenge of this dynamic process, however, was that we

interviewed a small number of states before these dis-

semination strategies unfolded. The national context was

not the same throughout the study, which influenced the

types and uses of evidence that participants reported. At the

same time, the emergence of this policy window also fa-

cilitated examination of evidence use at a time in which

this issue was relevant.

Conclusions

Examination of the types of evidence used in complex,

dynamic social systems, such as the one in which child

welfare operates, holds important implications for policy

makers and funders. The health and well-being of indi-

viduals in the United States are often not the sole respon-

sibility of one institution. This is particularly true for

vulnerable populations like children in foster care whose

needs may be met by a number of social institutions.

Although complex, multi-agency collaborations to create

public policies designed to improve health and mental

health outcomes make sense, recognition that each agency

has its own priorities and operates within specific so-

ciopolitical and economic contexts is important (Davies

HTO 2000). It is critical to recognize the myriad types of

evidence that each agency generates, acquires, and pri-

oritizes when making policy decisions. Recognition of

these multiple types of evidence, especially locally gener-

ated evidence, aligns strongly with the increased focus of

translational sciences and the need for study across the

transitional stages, including the study of basic biomedical

sciences (the ‘bench’), clinical trials and health services

research (the ‘bed’), population-level policies (the ‘curb’)

and the multidirectional influence of results from any of

these stages to others (Szilagyi 2009). Greater recognition

of the available evidence will help build additional

knowledge around the dynamic use of evidence from the

‘bench’ to the ‘curb’.

References

American Academy of Child and Adolescent Psychiatry. (2009).

Practice parameter on the use of psychotropic medication in

children and adolescents. Journal of American Academy of Child

and Adolescent Psychiatry, 48(9), 961–973.

Bardach, E. (2000). A practical guide for policy analysis: The

eightfold path to more effective problem solving. Chatham House

Publishers, Seven Bridges Press.

Bowen, S., & Zwi, A. B. (2005). Pathways to ‘‘Evidence-Informed’’

policy and practice: A framework for action. PLoS Med, 2(7),

e166. doi:10.1371/journal.pmed.0020166.

Bowen, S., Erickson, T., Martens, P., & Crockett, S. (2009). More

than ‘‘using research’’: The real challenges in promoting

evidence-informed decision-making. Health Policy, 4(3),

87–102.

Breland-Noble, A. M., Farmer, E. M. Z., Dubs, M., Potter, E., &

Burns, B. J. (2005). Mental health and other service use by youth

in therapeutic foster care and group homes. Journal of Child and

Family Studies, 14(2), 167–180.

Brown, R. T., & DuPaul, G. J. A. D. (2007). Practice parameter on the

use of psychotropic medication in children and adoloscents.

Journal of American Academy of Child and Adolescent Psy-

chiatry, 48(9), 961–973.

Contandrioppoulos, D., Lemire, M., Denis, J., & Tremblay, E. (2010).

Knowledge exchange processes in Organizations and Policy

Arenas: A narrative systematic review of the literature. Milbank

Quarterly, 88(4), 444–483.

Davies, H. T. O., & Smith, P. C. (2000). What Works? Evidence-

Based Policy and Practice in Public Services. Bristol: Policy

Press.

Davies, H., & Nutley, S. (2008). Learning more about how research-

based knowledge gets used: Guidance in the development of new

empirical research. New York: Grant Foundation.

Dobrow, M. J., Goel, V., & Upshur, R. E. G. (2004). Evidence-based

health policy: Context and utilization. Social Science and

Medicine, 58, 207–217.

Adm Policy Ment Health (2016) 43:52–66 65

123

dosReis, S., Zito, J. M., Safer, D. J., Gardner, J. F., Puccia, K. B., &

Owens, P. L. (2005). Multiple psychotropic medication use for

youths: A two-state comparison. Journal of Child and Adoles-

cent Psychopharmacology, 15, 68–77.

dosReis, S., Zito, J. M., et al. (2001). Mental health services for

youths in foster care and disabled youths. American Journal of

Public Health, 91(7), 1094–1099.

Fostering Connections to Success and Increasing Adoptions Act of

2008, Pub L. No. 110-351, 42 U.S.C. § 1305 (2008).

Geen, R. (2009). The Fostering Connections to Success and

Increasing Adoptions Act: Implementation issues and a look

ahead at additional child welfare reform. Child Trends, pp. 1–17.

Glaser, B., & Strauss, A. (1967). The Discovery of Grounded Theory.

London: Weidenfield & Nicolson.

Jack, S., Dobbins, M., Tonmyr, L., Dudding, P., Brooks, S., &

Kennedy, B. (2010). Research evidence utilization in policy

development by child welfare administrators. Child Welfare,

89(4), 83–100.

Lavis, J. N., Wilson, M. G., Oxman, A. D., Lewin, S., Fretheim, A.

(2009). SUPPORT tools for evidence-informed health policy-

making (STP)4: Using research evidence to clarify a problem.

Health Research Policy and Systems, 7(suppl 1): S4. http://www.

health-policy-systems.com/content/7/S1/S4.

Leslie, L., Mackie, T. I., Dawson, E. H., Bellonci, C., Schoonover, D.,

Rodday, A. R., et al. (2010). Multi-state study on psychotropic

medication oversight in foster care. Technical report. Boston,

MA: Tufts University Clinical and Translational Sciences

Institute.

Leslie, L., Raghavan, R., Hurley, M., Zhang, J., Landsverk, J., &

Aarons, G. (2011). Investigating geographic variation in use of

psychotropic medications among youth in child welfare. Child

Abuse and Neglect, 35(5), 333–342.

Lewin, S., Oxman, A. D., Lavis, J. N., Fretheim, A., Marti, S. G., &

Munabi-Babigumira, S. (2009). SUPPORT tools for evidence-

informed policymaking in health II: Finding and using evidence

about local conditions. Health Research Policy and Systems,

7(Suppl 1), S11.

Lomas, J., & Brown, A. D. (2009). Research and advice giving: A

functional view of evidence-informed policy advice in a Canadian

Ministry of Health. Milbank Quarterly, 87(4), 903–926.

MacDonald, G. (1998). Promoting evidence-based practice in child

protection. Clinical Child Psychology & Psychiatry, 3(1), 71–85.

Mackie, T., Hyde, J., Rodday, A. M., Dawson, E., Lakshmikanthan,

R., Bellonci, C., et al. (2011). Psychotropic medication oversight

for youth in foster care: A national perspective on state child

welfare policy and practice guidelines. Children and Youth

Services Review, 33, 2213–2220.

Marston, G., & Watts, R. (2003). Tampering with evidence: a critical

appraisal of evidence-based policy-making. Australian Review of

Public Affairs, 3(3), 143–163.

McMillen, J. C., Fedoravicius, N., Rowe, J., Zima, B. T., & Ware, N.

(2007). A crisis of credibility: professionals’ concerns about the

psychiatric care provided to clients of the child welfare system.

Administration & Policy in Mental Health and Mental Health

Services Research, 34, 203–212.

Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis;

an expanded sourcebook. London: Sage.

Moore, R. A. (2006). Evidence-based policy-making. Clinical

Chemistry and Laboratory Medicine, 44(6), 678–682.

Naylor, M., Davidson, C., et al. (2007). Psychotropic medication

management for youth in state care: Consent, oversight, and

policy considerations. Child Welfare, 86(5), 175.

Nelson, S. R., Leffler, J. C., & Hansen, B. A. (2009). Toward a

research agenda for understanding and improving the use of

research evidence. Portland, OR: Northwest Regional

Educational Laboratory. Retrievable at http://www.nwrel.org/

researchuse/report.pdf.

Nutley, S., Walter, I., & Davies, H. T. O. (2007). Using Evidence:

How Research Can Inform Public Services. Bristol: The Policy

Press.

Palinkas, L. A., Aarons, G. A., Chorpita, B. F., Hoagwood, K.,

Landsverk, J., & Weisz, J. R. (2009). Cultural exchange and the

implementation of evidence-based practices. Research on Social

Work Practice, 19(5), 602–612.

Pawson, R., & Tilley, N. (1997). Realist Evaluation. London: Sage

Publications.

Raghavan, R., Zima, B. T., Andersen, R. M., Leibowitz, A. A.,

Schuster, M. A., & Landsverk, J. (2005). Psychotropic medica-

tion use in a national probability sample of children in the child

welfare system. Journal of Child and Adolescent Psychophar-

macology, 15, 97–106.

Richie, J., & Spencer, L. (1994). Qualitative data analysis for applied

policy research. In Bryman & Burgess (Eds.), Analysing

Qualitative Data (pp. 173–194). London: Routledge.

Rubin, D., Matone, M., Huang, Y. S., dosReis, S., Feudtner, C., &

Localio, R. (2012). Interstate variation in trends of psychotropic

medication use among Medicaid-enrolled children in foster care.

Children and Youth Services Review, 34(8), 1492–1499.

Sanderson, I. (2002). Evaluation, policy learning, and evidence-based

policy making. Public Administration, 80(1), 1–22.

SocioCultural Research Consultants, LLC. (2012). Dedoose, Version

4.5.91. Web application for managing, analyzing, and presenting

qualitative and mixed method data. Los Angeles, CA \https:// www.dedoose.com/App/?Version=4.5.91[.

Strayhorn, C. K. (2006). Texas health care claims study-special

report on foster children. Austin, TX: Office of the Comptroller,

Texas Comptroller of Public Accounts.

Szilagyi, P. (2009). Translational research and pediatrics. Academic

Pediatrics, 9(2), 71–80.

Tanenbaum, S. J. (2005). Evidence-based practice as mental health

policy: Three controversies and a caveat. Health Affairs, 24(1),

163–173.

Tseng, V. (2012). The uses of research in policy and practice. Social

Policy Report, 26(2), 3–16.

United States Government Accounting Office. (2011). Foster Chil-

dren: HHS Guidance Could Help States Improve Oversight of

Psychotropic Prescriptions. GAO 12-201.. Washington, D.C.

Retrieved December 1, 2012 from http://www.gao.gov/assets/

590/586906.pdf.

United States Government Accounting Office. (2012). Children’s

Mental Health: Concerns Remain about Appropriate Services

for Children in Medicaid and Foster Care. GAO-13-15.

Washington, D.C. Retrieved April 12, 2013 from http://www.

gao.gov/assets/660/650716.pdf.

U.S. Department of Health and Human Services, Administration on

Children, Youth, and Families. (2012) Oversight of Psychotropic

Medication for Children in Foster Care; Title IV-B Health Care

Oversight and Coordination Plan. Log No: ACYF-CB-IM-12-

03. Issuance Date: 04-11-2012.

Weiss, C. H. (1977). Research for policy’s sake: The enlightenment

function of social research. Policy Analysis, 3, 531–545.

Weiss, C. H. (1979). The many meanings of research utilization.

Public Administration Review, 39(5), 426–431.

Zima, B., Bussing, R., et al. (1999). Psychotropic medication

treatment patterns among school-aged children in foster care.

Journal of Child and Adolescent Psychopharmacology, 9(3),

135–147.

Zito, J., Safer, D., Sai, D., Gardner, J., Thomas, D., Coombes, P., et al.

(2008). Psychotropic medication patterns among youth in foster

care. Pediatrics, 121(1), e157–e163.

66 Adm Policy Ment Health (2016) 43:52–66

123

Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.