Barriers to the APN roles in healthcare change 6 DQ 1

Children and Youth Services Review 39 (2014) 160–168

Contents lists available at ScienceDirect

Children and Youth Services Review

journal homepage: www.elsevier.com/locate/childyouth

Interagency Collaborative Team model for capacity building to scale-up evidence-based practice

Michael Hurlburt a,b,⁎, Gregory A. Aarons b,c, Danielle Fettes b,c, Cathleen Willging d, Lara Gunderson d, Mark J. Chaffin e

a School of Social Work, University of Southern California, Los Angeles, CA, United States b Child and Adolescent Services Research Center, United States c Department of Psychiatry, University of California, San Diego, La Jolla, CA,United States d Pacific Institute for Research and Evaluation, Albuquerque, NM, United States e Department of Pediatrics, University of Oklahoma Health Sciences Center, Oklahoma City, OK, United States

⁎ Corresponding author at: School of Social Work, Un University Park Campus, Los Angeles, CA 90089, United Sta fax: +1 858 675 0857.

0190-7409/$ – see front matter © 2013 Elsevier Ltd. All ri http://dx.doi.org/10.1016/j.childyouth.2013.10.005

a b s t r a c t

a r t i c l e i n f o

Available online 10 October 2013

Keywords: Implementation Sustainment Teams Process model Evidence-based practice

Background: System-wide scale up of evidence-based practice (EBP) is a complex process. Yet, few strategic approaches exist to support EBP implementation and sustainment across a service system. Building on the Explo- ration, Preparation, Implementation, and Sustainment (EPIS) implementation framework, we developed and are testing the Interagency Collaborative Team (ICT) process model to implement an evidence-based child neglect intervention (i.e., SafeCare®) within a large children's service system. The ICT model emphasizes the role of local agency collaborations in creating structural supports for successful implementation. Methods: We describe the ICT model and present preliminary qualitative results from the use of the implemen-

tation model in one large scale EBP implementation. Qualitative interviews were conducted to assess challenges in building system, organization, and home visitor collaboration and capacity to implement the EBP. Data collec- tion and analysis centered on EBP implementation issues, as well as the experiences of home visitors under the ICT model. Results: Six notable issues relating to implementation process emerged from participant interviews, including: (a) initial commitment and collaboration among stakeholders, (b) leadership, (c) communication, (d) practice fit with local context, (e) ongoing negotiation and problem solving, and (f) early successes. These issues highlight strengths and areas for development in the ICT model. Conclusions: Use of the ICT model led to sustained and widespread use of SafeCare in one large county. Although some aspects of the implementation model may benefit from enhancement, qualitative findings suggest that the ICT process generates strong structural supports for implementation and creates conditions in which tensions between EBP structure and local contextual variations can be resolved in ways that support the expansion and maintenance of an EBP while preserving potential for public health benefit.

© 2013 Elsevier Ltd. All rights reserved.

1. Introduction

Introduction of evidence-based practices (EBPs) can lead to substan- tial public health benefits. However, the implementation process can shape whether intended outcomes are actually achieved (Aarons & Palinkas, 2007; Allen, Brownson, Duggan, Stamatakis, & Erwin, 2012; Crea, Crampton, Abramson-Madden, & Usher, 2008; Fixsen, Naoon, Blase, Friedman, & Wallace, 2005; Greenhalgh, Robert, Macfarlane, Bate, & Kyriakidou, 2004; Palinkas & Aarons, 2009). Well-established practice models, implemented poorly or not sustained, will fail to achieve intended goals despite research evidence supporting their

iversity of Southern California, tes. Tel.: +1 858 675 0167x253;

ghts reserved.

clinical effectiveness (Backer, 2000; Bond, Drake, McHugo, Rapp, & Whitley, 2009). Thus, an effective implementation approach is often as important as the practice to be utilized.

Several conceptual models describe factors that can influence imple- mentation effectiveness. Some models emphasize structural features hypothesized to be core components of effective implementation (Aarons, Hurlburt, & Horwitz, 2011; Damschroder et al., 2009; Feldstein & Glasgow, 2008; Greenhalgh et al., 2004; Mendel, Meredith, Schoenbaum, Sherbourne, & Wells, 2008). Other models emphasize implementation processes, outlining key steps (and their timing) hy- pothesized to contribute to successful implementation of service inno- vations (Glisson & Schoenwald, 2005; Sosna & Marsenich, 2006; Stetler, McQueen, Demakis, & Mittman, 2008). Structural and process implementation models are often conceptually aligned. For example, both types of models address the central importance of issues such as

161M. Hurlburt et al. / Children and Youth Services Review 39 (2014) 160–168

strong and effective leadership to support change initiatives, establish- ing a strong fit between change efforts and organizational and service system culture and values, creating methods for ensuring quality pro- gram delivery (i.e., fidelity), and clarifying/addressing financial supports for a change initiative.

This paper describes the Interagency Collaborative Team (ICT) im- plementation process model. The ICT model provides an approach to support successful roll-out of human service innovations in large geo- graphic areas, particularly change efforts involving EBPs. It is directly relevant to improving outcomes of service enhancements in child and family service systems. The ICT model is designed to enable organiza- tions to work together in ways that generate the structural and process supports associated with successful implementation and sustainment of innovations. We discuss some core areas of difference and similarity be- tween the ICT model and other implementation strategies, connecting core features to one structural implementation framework, the Explora- tion, Preparation, Implementation, and Sustainment (EPIS) framework (Aarons et al., 2011). Qualitative data from the scale-up of an EBP in one large county illustrate areas of strength and some limitations in the ICT model and provide perspective on other process models of EBP implementation.

1.1. Interagency Collaborative Team (ICT) model in the EPIS Framework

Like a number of implementation frameworks, the EPIS framework summarizes variables that can positively or negatively affect the imple- mentation of an evidence-based practice. The EPIS framework is unusual in identifying key variables thought to particularly affect implementa- tion efforts during each of four major implementation stages in public sector child welfare and mental health settings. For example, some key variables identified as influencing the preparation and early implemen- tation stages of a quality improvement effort include strength of the leadership supporting change (Aarons, 2006; Edmondson, 2004; Klein, Conn, & Sorra, 2001), the degree of fit of an innovation with the service system context (Klein & Sorra, 1996), clarity of financial support for proposed changes (Aarons, Wells, Zagursky, Fettes, & Palinkas, 2009; Frambach & Schillewaert, 2002), level of involvement of practice devel- opers in the implementation process (Aarons et al., 2011), and the pres- ence of cross-organizational knowledge of and commitment to the new practice (Glisson & Schoenwald, 2005; Sosna & Marsenich, 2006).

The ICT implementation process model outlines steps designed to lead directly to the kinds of key implementation supports described in the EPIS framework. The model takes its name from the fact that it emphasizes the key role of collaboration among stakeholders and staff members at the system level, from multiple partnering organizations, and of developing or utilizing a local “seed” team to embody and sup- port promotion and maintenance of expertise and ongoing fidelity in the practice to be implemented. Inter-agency collaboration and willing- ness to share expertise is central to multiple steps in the implementa- tion process and across organizational levels. Conceptually, the ICT model has much in common with other implementation process models (Chamberlain, Price, Reid, & Landsverk, 2008; Glisson & Schoenwald, 2005; Sosna & Marsenich, 2006), which describe logically ordered sets of activities designed to create a context in which EBP implementation occurs effectively and intended public health benefits are realized.

1.1.1. ICT processes and action steps Fig. 1 provides a graphical representation of key implementation

processes included in the ICT model, with the stages of the EPIS frame- work listed temporally down the left side of the figure. In the ICT model, a process is considered to be a goal-driven domain of focus that extends over a period of time within the longer implementation effort. For exam- ple, the initial EBP education and stakeholder development and align- ment processes involve an initial phase of identifying community- based stakeholders with interests in a particular practice change effort, and discussions and education efforts designed to lead to joint selection

of and commitment to a common practice change initiative. The practice fit assessment process involves a careful analysis by key stakeholders at system and organizational levels of EBPs under consideration to identify aspects of practices that fit with existing policies, contracting, and ser- vice routines and those where modifications might be required. Brief de- scriptions of each ICT process are provided at the bottom of Fig. 1.

Specific ICT model action steps are listed in Table 1 that animate the processes shown in Fig. 1. Their contributions to each implementation process are noted in the figure. For example, the Initial EBP Education process occurs as part of ICT action steps A (convening of stakeholders) and B (soliciting expertise). Education about the EBP becomes an in- tense process focus that occurs in the context of meetings among inter- ested stakeholders, supplemented by expertise about the EBP solicited from appropriate sources. Sources may be multiple, including EBP developers, other users of the EBP, researchers having familiarity with the practice, and/or materials available from sources such as journals or intermediary organizations that summarize information about EBPs. Structural supports designed to arise from the ICT processes are represented as planks beneath the model processes that generate them. We represent the ICT model in this manner because it is best con- ceptualized as a series of major actions that address core implementa- tion processes. Specific action steps animate these processes and give rise to or strengthen key structural supports viewed as creating an envi- ronment that can sustain an innovative practice as it is scaled up.

1.1.2. ICT initial steps: exploration/adoption decision The ICT model initially revolves around a service system and multi-

agency commitment to invest in the long-term viability of an EBP- centered quality improvement initiative, with an ultimate goal to im- prove selected client level outcomes. Partnering agencies may include a range of stakeholder organizations, but particularly involve funding, administration, and service delivery organizations from the outset. Dur- ing an initial exploration phase, stakeholders convene and meet to dis- cuss need for a practice change effort that involves investment by multiple individuals and organizations. Although no specified leader is required to initiate such meetings, it is expected that one local or region- al organization will often take responsibility for convening and leading such discussions. For example, a health and human service administra- tion may convene discussions around maltreatment prevention, reduc- tion in delinquency, or some other practice change effort. Within an ICT model-guided implementation, convening of stakeholders should in- clude efforts to identify those stakeholders with substantial interests in the identified substantive area (e.g., child neglect).

A second important step in the process of exploring a possible prac- tice change involves concentrated efforts to obtain wide-ranging factual information about the costs, benefits, and tradeoffs associated with spe- cific practice changes. Outside expertise is identified and sought to help answer questions and reduce uncertainty about the change effort under discussion. The joint process of participating in education about possible practice change efforts and discussing the advantages and disadvan- tages of various options is aimed at developing a shared commitment and direction among stakeholders at an inter-agency level to a jointly supported EBP implementation.

1.1.3. Interagency seed team development: preparation and implementation Once a specific EBP is selected as the focal point for a broad practice

change effort, stakeholders in the ICT process initiate implementation of the EBP by creating a formative interagency collaborative “seed” team (or ICST), which may consist of employees from several different local organizations that form a core unit of expertise in the selected service model. A seed team intentionally involves multiple organizations in the maintenance of innovation expertise to build broader investment in, commitment to, and communication about an innovation among invested stakeholders and subsequently trained practitioners. The seed team becomes a repository of local expertise for an EBP. It is designed to serve as the ongoing support structure for continued EBP training,

Fig. 1. Implementation processes emphasized by the ICT, including structural supports hypothesized to emerge from following ICT model steps listed in Table 1.

162 M. Hurlburt et al. / Children and Youth Services Review 39 (2014) 160–168

coaching, and roll-out across a geographic area and as a facilitator of minor practice adjustments that help to fit a practice to diverse regional contexts (Aarons et al., 2012). Members of the seed team maintain a central liaison role between the EBP developer and other actors at multiple levels within the service system so that issues, decisions, and adaptations can be negotiated during initial experimentation with implementation.

1.1.4. Seed team: ongoing responsibilities Following inception of the seed team and initial EBP training, certifi-

cation, and service delivery, the team is then responsible for training and supporting additional teams of individuals that can then implement the selected EBP as it is scaled up across a service system. The seed team assumes responsibility for ongoing training of new teams of

practitioners that may consist of employees from several local non- profit organizations, hereafter referred to as Interagency Collaborative Teams (ICTs). These ICTs form for the express purpose of learning and mastering delivery of the EBP to be implemented, under the guidance of the original seed team. Although members include staff from multiple organizations, they meet together with a seed team coach during a su- pervision and knowledge transfer phase. ICTs trained by the seed team are responsible for the primary delivery of the EBP. This imple- mentation structure, with regional teams having interagency composi- tion, results in a network of local providers that allows for high inter- agency communication, and information and possible workload sharing.

The seed team maintains relationships with ICTs following EBP training. In order to maintain and continually enhance quality delivery

Table 1 Steps involved in the Interagency Collaborative Team (ICT) implementation process model.

ICT steps

A. Identify and convene stakeholders with likely interests in a shared quality improvement initiative (may be iterative)

B. Solicit relevant expertise required to address questions about selected quality improvement directions and EBP alternatives

C. Develop commitment and direction among stakeholders to a jointly supported, EBP-centered change effort

D. Create an interagency seed team to: 1. Learn the EBP 2. Conduct initial local delivery of the EBP 3. Train new local EBP practitioners 4. Serve in a liaison role with external EBP developers/trainers 5. Monitor and provide feedback about quality of EBP delivery 6. Communicate and support a commitment to quality EBP delivery 7. Communicate with stakeholders about implementation progress

E. Form additional interagency training teams that: 1. Deliver the EBP 2. Relay feedback about implementation to the seed team 3. Share information with one another about implementation progress

F. Plan a phased reduction in EBP developer involvement

163M. Hurlburt et al. / Children and Youth Services Review 39 (2014) 160–168

of the EBP, newly trained providers continue to receive constructive support and feedback in the form of supervision and coaching from seed team members for a defined period of time, which may vary by practice or situation. In the case of SafeCare, ongoing fidelity monitoring and coaching are integral components of the EBP. Organizing supervi- sion and coaching through the seed team has many potential benefits in the short and longer term. This structure is designed to provide a clear source of leadership and information to newly trained providers. Ideally having the seed team serve as the source of ongoing training and coaching facilitates a gradual reduction of EBP developer involve- ment whereby the local service system and its contracted agencies be- come the repository of expertise in the practice being adopted. Based on its initial key role within a multi-organizational implementation ef- fort, the seed team continues to serve as a locus of information about needs for adaptations to make a practice work within a particular local context. Such adaptations may involve changes to aspects of the inter- vention itself or to the structure of the service setting in which the EBP is delivered (e.g., Finno-Velasquez, Fettes, Aarons, & Hurlburt, under review). Cross-organizational membership on the seed team con- tributes to ensuring a continuing locus of expertise available to all orga- nizations within the ICT partnership, reducing the kinds of expertise loss that regularly occur within individual organizations and agencies due to staff turnover and organizational changes.

As noted above, the ICT model seeks to foster implementation sup- ports in the areas of practice fit, leadership, communication, expertise distribution, EBP quality (fidelity) management, effective developer in- volvement, and program adaptation. A large California county used the ICT model to implement an evidence-based neglect prevention program throughout the county. Qualitative inquiry into this effort helps to illus- trate several key elements of the implementation approach.

1.2. The present study

Implementation of the SafeCare neglect prevention model occurred in one large California county. With approximately 3 million total resi- dents, the county population is similar in size or larger than that of many smaller U.S. states. The county encompasses urban, semi-urban, and rural areas that are home to a diverse cultural mix of residents, in- cluding significant Mexican-American and Native American popula- tions. Planning for many aspects of public human services in the county is organized into local planning regions, each with some of its own local history, demographic and cultural characteristics, climate, and topography. Implementation of any new practice at a county level

represents a large-scale system and organizational change effort that occurs across the planning regions.

In 2007, the Department of Health and Human Services (DHHS) agency and the local chapter of a national foundation embarked upon an effort to transition one category of county maltreatment prevention services toward an EBP. County DHHS leaders (responsible for child wel- fare services), members of the local branch of a national foundation, and research partners convened to consider three different child focused EBPs to improve outcomes for children and families involved with the child welfare system. After consideration of research evidence, program- matic fit, and financial resources required, SafeCare®, an evidence-based child neglect prevention program utilizing home visiting (Chaffin, Hecht, Bard, Silovsky, & Beasley, 2012; Lutzker, Bigelow, Doctor, Gershater, & Greene, 1998), was selected for implementation.

Qualitative interviews and focus groups allowed us to document the roll-out of SafeCare, provided insight into how the ICT model generated key structural supports for implementation, and helped to identify process issues worthy of more careful consideration. The following sec- tion summarizes at a general level what we learned from that qualita- tive work about themes related to implementation process, including: (a) initial commitment and collaboration among stakeholders, (b) lead- ership, (c) communication, (d) practice fit with local context, (e) ongo- ing negotiation and problem solving, and (f) early successes.

2. Methods

2.1. Overview

From August, 2008 to January, 2009 we undertook in-depth qualita- tive interviews with key stakeholders involved in the early stages of system-wide implementation of SafeCare that followed the ICT imple- mentation model. Data collection and informed consent procedures were approved by the appropriate Institutional Review Boards.

2.2. Participants

Participants in this study included 27 stakeholders involved in vari- ous facets of the early implementation process. Participants were re- cruited through an initial telephone call or email describing the study purpose and participation. One of the authors either made the initial contact or was available to answer questions about participation. Our purposive sample consisted of all individuals who took part in initial EBP planning meetings, including representatives from the county (n = 3) a foundation supporting part of the implementation effort (n=9), and the executive directors of the community-based organiza- tions (n=3) that were eventually contracted to deliver SafeCare. Next, we interviewed key individuals involved in supporting delivery of the EBP, including SafeCare supervisors, trainers, and coaches (n = 6) and front line providers (n= 6).

2.3. Semi-structured interviews

Interview guides consisted of open-ended questions that were tai- lored to each stakeholder group. The interviews with representatives of the county, the foundation, and community-based organizations fo- cused on the initial planning process, their roles and responsibilities and interactions with one another, and perceptions overall of SafeCare implementation. These interviews also sought to capture data on organizational- and system-level factors affecting implementation. The interviews with supervisors, trainers, coaches, and seed team members centered on each person's involvement in the ICT approach, knowledge of and experiences with SafeCare, and the “fit” of the intervention with local populations and service delivery contexts. All participants agreed to recorded interviews, which lasted approximately 60 min, and were professionally transcribed. Transcriptions were reviewed for accuracy

164 M. Hurlburt et al. / Children and Youth Services Review 39 (2014) 160–168

by a research assistant. In addition, ethnographer interview notes were typed and uploaded to an electronic database.

2.4. Data analysis

We employed an iterative process to review the textual data from in- terviews and utilized NVivo 9 (2009) qualitative data analysis software to facilitate this work. Data analysis proceeded first by engaging in an open coding approach to locate the themes and issues that emerged from the interview transcripts. Focused coding was then used to deter- mine which of these themes emerged frequently and which represent- ed unusual or particular concerns (Emerson, Fretz, & Shaw, 1995). In this staged approach to analysis, our research team coded sets of tran- scripts, created detailed memos linking codes to each theme and issue, and then passed their work to other team members for review. Discrep- ancies in coding and analysis were identified and resolved through con- sensus during team meetings (Sandelowski & Barroso, 2003). Themes emerging from analyses were cross-walked with objectives of the ICT model to identify areas in which language used by participants might differ from that used in the model but have similar meaning. Principal themes from the interviews are presented, supplemented by relevant comments from participants. In some cases, quotations are edited slight- ly to smooth readability.

3. Results

Results confirmed the significance of a number of the processes and structural supports outlined in the ICT model, including initial commit- ment and collaboration, cross-level leadership, and practice fit to the local context. Other key themes also arose, such as the importance of early successes, and of negotiating roles and responsibilities among stakeholder organizations. Interconnections among these themes are highlighted below.

3.1. Commitment and collaboration

The first step in implementing the ICT model involves identifying and then bringing together key stakeholders invested in a change ef- fort in order to discuss shared interests in undertaking a particular initiative. In this case, interest from stakeholders in the child welfare system and the foundation chapter in supporting a system-wide im- provement in the area of child maltreatment led to the creation of a “Council” focused on a possible quality/capacity enhancement effort that eventually centered on SafeCare. The Council included represen- tatives not simply from county child welfare services and the local foundation, but from community-based non-profit organizations, advocacy organizations, a children's hospital, EBP developers, and researchers. Initial conversations about directions for the intervention were critical in setting the foundation for the long-term collaboration among stakeholders that would be required to implement and sustain SafeCare.

A potentially underappreciated aspect of building initial commit- ment is the fact that stakeholders do not necessarily share the same organizational culture or values when they begin to collaborate. In the case of SafeCare, some stakeholders worked in large government organizations, others in small non-profit organizations. Consequent- ly, stakeholders often had very different ideas about how to pursue change and their respective roles in this process, which at times led to tension and conflict that added complexity to the implementation, some examples of which are described in a later section. Nonethe- less, the stakeholders pushed forward, buoyed by the belief that the specific EBP they had agreed to implement would improve child welfare services and reduce neglect. County officials, for exam- ple, were enthusiastic about ensuring that services provided in the child welfare system were evidence-based. Their confidence in mov- ing forward with SafeCare was bolstered after a presentation and

discussion period with a research team knowledgeable about the EBP, its underlying evidence base, and the benefits of the intervention.

The initial commitment process involved frank discussion and eval- uation of whether interests were broadly shared among stakeholders amidst differences in organizational directions, cultures, and values. Although this process of group reflection may not have been sufficient to guarantee the overall success of the implementation effort, it did ap- pear to have been a central component in laying an appropriate founda- tion for positive outcomes. This process put the diverse interests of the stakeholders on the table, fostered commitment to a common direction, and engendered a sense of top-level leadership support for the change effort that was ultimately reinforced by additional leadership tiers.

3.2. Leadership

Once an intervention is selected, the ICT model calls for develop- ment of a “seed team,” an initial cadre of service providers responsible for acquiring expertise in the service model, for transmitting this knowledge to other teams of individuals involved in day-to-day service provision, and for providing ongoing fidelity assessment and support. By virtue of their roles as trainers and supervisors of future cohorts of SafeCare-trained home visitors, the nine seed team members were placed in a structural position of leadership. As the implementation progressed, three of the original seed team members were selected to assume the roles of trainers/coaches, and one emerged as the team leader and SafeCare supervisor. As noted by several participants, these individuals could be counted on to guide newly trained home visitors in consistent SafeCare practice and thus were paramount to implemen- tation success.

As noted, one SafeCare supervisor became the primary identified team leader and source of support for home visitors. Several home vis- itors commented on the support provided by the SafeCare supervisor in particular. One home visitor stated, “She's very good at answering our questions”, while a second added, “I find her very helpful and she's available if I need her.” Although not expressed in terms of leadership from the home visitor perspective, we interpreted the regular com- ments regarding the support and information provided by the SafeCare supervisor as a reflection of the clinical leadership provided by the seed team, as viewed by home visitors.

Strong leadership was also evident from the directors of each community-based provider organization, the local foundation, and county child welfare services. The fact that the provider organizations collaborated from the outset to respond to the local foundation's Re- quest for Proposals to deliver SafeCare from a multi-agency position, and then facilitated involvement of their staff within a single seed team, reinforced a broad sense of cross-level leadership commitment to SafeCare. During the implementation phase, the local foundation, in partnership with the county, also spearheaded organizational meetings for planning purposes. This higher-level buy-in, commitment, and sup- port communicated a message that this new EBP was not the “flavor of the day” and that there was an expectation for effective implementation and ongoing use of SafeCare.

3.3. Communication

The majority of participants suggested that communication was crucial to successful implementation, but attributed problems en- countered during the roll out of SafeCare to communication chal- lenges. Interviewees reported that the communication structures around implementation were initially insufficient. Some stakeholders were privy to misinformation or to no information regarding issues impacting implementation. In one example, a county staff member in- correctly informed some supervisors and the local foundation that home visitation caseloads pre-SafeCare were half of what they were in reality (typically 20 vs. 10 cases per home visitor). Such unintentional misinformation altered the course of project planning and fueled

165M. Hurlburt et al. / Children and Youth Services Review 39 (2014) 160–168

concerns that implementation of the new EBP would prove to be too ex- pensive in the long run. Problems disseminating information among stakeholders across all levels were also common. Email, in particular, did not function as a dependable mode of communication and informa- tion sharing. There were times when stakeholders felt that only a limit- ed subset of individuals were receiving needed information about program implementation.

Stakeholders also described discussing or even deliberating on SafeCare issues individually outside of group meetings, while alluding to underlying power dynamics that influenced communication. One participant lamented about being left of out of these informal communi- cations about SafeCare: “When I hear [other stakeholders] talking to each other, I can tell that [they] have talked on the side…. It looks like they're able to access the information they need, and [can] find things out.” A second participant reported running into others involved in im- plementation in non-SafeCare milieus and felt freer to share ideas about the intervention and its roll out. They admitted, “We talk about [SafeCare] a bit. [If] I am with the county person, I'm like, ‘Don't ask me [later at a Council meeting], I'll talk to you here. I won't talk to you in the meetings when there are other county people [present]’.” Such comments suggest that perceived power imbalances among stake- holders sometimes interfered with candid dialog.

Structurally, the SafeCare supervisor was expected to serve as a locus of communication and information exchange between home visitors newly trained in SafeCare and the other stakeholder groups (e.g., county child welfare and the local foundation). The SafeCare supervisor com- mented on the challenges she experienced mediating between the home visitors and upper-level leadership: “Right now…we're in the early stages [of implementation] so there's a lot of things that are chang- ing everyday…. [There's a need to] maintain that open communica- tion…. That's really the big part of it, as far as ‘we want you guys to do this’ or ‘we want you guys to do this differently’ or ‘don't use this form’ or ‘use this form.’ I tell the team, ‘There's always going to be some changes and as soon as I know something you guys will know as well. So just bear with me. The first few months are going be like this.’” In some respects it was difficult for the SafeCare supervisor and the home visitors to keep up with frequent changes initiated from above, due to the potential lack of clearly structured communication channels. Apart from the SafeCare supervisor, the home visitors also struggled to some degree with whom else they should be communicat- ing regarding SafeCare implementation. In particular, they were often unsure whether they should turn to or report to their immediate team supervisor, who was not trained in SafeCare, but who managed their workaday lives, or to others involved in the roll-out, such as the re- searchers or trainers.

3.4. Fit with existing practice and fidelity

The ICT model created an active, functional process for addressing the fit of SafeCare to the structure, culture, and local needs of service populations and the organizations delivering those services. This oc- curred at several points. During the initial discussion and commitment phase, possible EBPs were considered with respect to their target audi- ences and outcomes, modes of delivery, and training and resource re- quirements. SafeCare emerged from this phase as a top candidate. It had a jointly held focus of interest to Council members (child neglect), only involved retooling of the curricular component of existing home visitation services rather than a more substantial reformulation of ser- vice models, and had implementation costs viewed as manageable within a large-scale roll out of the practice. This first participatory phase put stakeholders on the path to selecting a model perceived to have good fit with organizational structure, values and needs.

At the practice level, the structure of the seed team created an ideal framework for adapting an EBP as it was progressively implemented across a larger scale service area. A major theme, widely expressed in our interviews, was concern for whether SafeCare could be applied to

all families. The seed team described the service population as having multiple needs, not necessarily consistent with the SafeCare curriculum. Families, for example, struggled with serious problems, ranging from the procurement of basic necessities to shelter, critical issues that fell outside the scope of SafeCare. Commenting on the difficulties of car- rying out SafeCare in such circumstances, one seed team member ex- plained, “[Families] are having a hard time. How can somebody want to learn about safety or health when they don't know what they're going to feed their kids [or] have the money to pay their rent?” This individual admitted to cutting short her visits with families in order “to help them with resources or whatever they're going through.” One solution ad- vanced during the early implementation period was not to begin SafeCare when families were in crisis. In such cases, the seed team member only initiated SafeCare after he or she had the opportunity to help the family address other issues. Over time, the seed team helped to shift home visi- tation practice so that such issues could occur within the context of ongo- ing SafeCare visits. In fact, such issues were fit within the broader problem-solving framework utilized within SafeCare. The seed team played an instrumental role in identifying this issue and facilitating incre- mental solutions that helped tailor SafeCare to the local context.

The seed team was also able to undertake deliberative steps to im- prove the cultural responsiveness of SafeCare to populations in the local region. For example, the seed team actively reviewed and translat- ed SafeCare materials to improve their relevance to Latino, particularly Mexican-American, families. The seed team also discussed and adapted child health focused sessions and materials to the customs of Latino im- migrant families, who sometimes expressed values and preferences for homeopathic remedies not originally considered within the health module of the SafeCare EBP (Finno-Velasquez et al., under review).

The ICT model requires that the seed team become the local cross- agency repository of expertise in an EBP for a group of collaborating or- ganizations. At the outset, this involved the seed team learning and uti- lizing the new practice model until they reached a level of expertise meriting certification by the model developers. While mastering the EBP during early service delivery, the seed team was also immersed in the delivery of the practice in the local context. In their designated roles as future trainers, coaches, and supervisors, the seed team mem- bers were tasked with internalizing the knowledge of and expectations of program developers, but also with a high level of responsiveness to local population needs. As noted above, the pressure to resolve tensions between existing SafeCare structure and knowledge of the local service population put the seed team in the position of developing a locally re- fined expertise that fit the practice to the local area and allowed for a planned decrease in the involvement of the original EBP developer in supporting sustainment of SafeCare. Documentation of the specific ad- aptations made by the seed team is discussed by Finno-Velasquez et al. (under review). Participants viewed the gradual decrease in devel- oper involvement as proceeding effectively, as originally planned.

3.5. Negotiation of rights, roles, responsibilities, and interests

In addition to a need to incorporate further structure in the area of communication patterns, as alluded to by earlier qualitative findings, the ICT model might benefit from further attention to enabling methods for facilitating negotiation of differences among organizations and their members. Although many possible differences among partners may emerge during the implementation of any new change effort, several specific examples from the SafeCre implementation experience illus- trate the kinds of issues that arise and require negotiation during a large-scale collaborative EBP initiative. For example, the appropriate pace for SafeCare implementation and documentable change was one area in which stakeholders held differing expectations and opinions. It arose because the collaborating partners had different needs and expec- tations influencing their participation. The local foundation supporting initial training and development of the seed team had interests in seeing measurable outcomes quickly in order to show progress to donors

166 M. Hurlburt et al. / Children and Youth Services Review 39 (2014) 160–168

supporting the organization, in addition to their core interest in improv- ing outcomes for children and families. Other stakeholders did not share the pressure to accelerate change to the same degree. Discordant expec- tations led to some tensions related to this issue.

Differences also emerged with regard to expected roles and respon- sibilities. For example, one important decision during implementation revolved around the order in which the seed team trained new home visitor teams in the various county regions. As a new partnership in which different parties jointly contributed to the SafeCare implementa- tion, issues around rights and responsibilities needed to be negotiated. County representatives had the authority and perceived responsibility to make decisions about ordering of SafeCare roll-out, since they were entrusted by the public to provide oversight of child welfare services. However, foundation staff also sought to exert authority in this area be- cause the foundation had committed funds to support the seed team. As shared authority was being established, power struggles between parties occasionally ensued. These tensions were recognized by others involved in the implementation. One community-based provider orga- nization executive stated, “It's no one's fault, but I think the foundation and the county still have to have some meetings about whose role is what, and who has decision-making authority on certain things. And I think right now they are still a little messy.”

One further example may also be illustrative. Having significant in- vestment in the success of the SafeCare implementation, especially given that this was the first time the foundation was investing in a single major capital improvement as opposed to multiple smaller local grants, foundation representatives indicated a need and a right to work directly at times with the local community-based organizations contracted by the county to deliver SafeCare services. County representatives, again having direct responsibility for management and oversight of child wel- fare services, also asserted a duty to be involved in conversations and communications around service initiatives under their purview, some- times leading to tensions around roles and responsibilities among collaboration partners. One participant described the resulting tension as a “strange triangulation” between the entities involved. However, “What I think helped move it [the intervention] along was the enthusi- asm of everyone on the Council, [and] the enthusiasm of the County, for transitioning into the SafeCare model.”

3.6. Early successes

The ICT model intentionally involves the staged deployment of an EBP, beginning with the experience of the seed team prior to subse- quent roll out to and support of other practitioners. Participants in qual- itative interviews consistently mentioned that the purposeful transition of SafeCare from an initial seed team did lead to early signs of success that were interpreted as facilitating and supporting efforts to imple- ment and sustain SafeCare. One participant noted that stakeholders across all levels were “…seeing it is working. They are seeing that they're not getting cheated or embarrassed. They are seeing that there is a system in place. They're just calmer. And so their control needs are a little less.” Markers of success appeared to help stakeholders begin to overcome power struggles. Evidence of success took various forms, including positive experiences with initial training and delivery of ser- vices and encouraging reports by the home visitors about how families were responding to the intervention. A county official noted, “The training went well and implementation was successful…. The reception actually at the line level has exceeded my expectations…. Home visitors are comfortable with one, embracing more of a script and, you know, more structure within the visit and that they're receptive to the coaching…” Undertaking implementation in a phased roll out, begin- ning with exploration, appropriate preparation, and planned imple- mentation phases and involving the seed team as a central training and support component, created opportunities for shared successes and further commitment to ongoing problem solving.

4. Discussion

This paper describes the ICT model for EBP implementation, which is designed to facilitate development of many supports hypothesized to be central to successful quality improvement efforts organized around EBP implementation. Qualitative data from implementation of SafeCare in one large geographic area provided the opportunity to reflect on the strengths and limits of the ICT model and to consider it relative to other process models of quality improvement and EBP implementation.

The ICT model departs from traditional service structure and process by distributing local expertise across service teams, and more focally in a seed team, in a way that takes into account challenges faced by real-world public social service systems (Aarons et al., 2011; Aarons, Sommerfeld, Hecht, Silovsky, & Chaffin, 2009). Drawing on computer science and engineering theories of distributed expert systems (Dai, Xie, Poh, & Liu, 2003) and team decision making (Hollenbeck et al., 1995), the ICT model aims to increase effective team functioning through building greater systemic and cross agency trust (Edmondson & Roloff, 2009) and collaboration (Bertram, 2008). This structuring oc- curs at multiple levels, including among administrative and funding stakeholders with interest in a practice change initiative, and at the level of local clinical leadership (i.e., the seed team). The goal of the ICT process is to build interagency relationships at both levels, and be- tween levels, creating the structural supports central to effective adop- tion, implementation, and sustainment of an EBP with positive public health effects. With the seed team playing a substantial role in opera- tional implementation, the ICT model seeks to build structures and pro- cesses that enable the fitting of an EBP to the local context as outside developer involvement is reduced, and potential for EBP sustainment is increased.

Other implementation strategies directly relevant to EBP implemen- tation include such models as the Availability, Responsiveness and Continuity (ARC) and Community Development Team (CDT) models. Each arises out of somewhat different theoretical frameworks than the ICT model. The ARC model emerged from organizational development (Burke, 1993; Nadler & Tushman, 1977; Porras & Robertson, 1992) and interorganizational domain development (Gray, 1990; Trist, 1985) theories. The ARC model focuses strongly on improving organizational culture and climate and organizational processes to support effective care and more effective EBP implementation. ARC relies on an organiza- tional change agent to work with the organization to effect intra- organizational change to improve care. The CDT model was developed based on the experiences of the California Institute for Mental Health (CiMH), a training and technical assistance organization supported by county mental health agencies and child welfare systems. The CDT approach focuses on developing supportive collaborations among stakeholders, often in different counties, that are considering and implementing EBPs. The CDT model, like the ARC, relies heavily on the involvement of an outside consultant to structure communications among stakeholders, set priority topics for discussion, and foster organi- zational problem solving around issues that arise during implementa- tion. The outside consultant brings collected expertise acquired from extensive communications with multiple EBP program developers and local stakeholders to his/her role in supporting EBP implementation ef- forts (Sosna & Marsenich, 2006).

Considerable common ground exists among these models, although there are also some areas of difference, both qualitatively and in overall emphasis. For example, the ICT model devotes considerably less atten- tion to efforts to change intraorganizational culture and climate than the ARC model. However, both have many similarities including pro- cesses specifically targeting initial collaborative work and joint decision making among stakeholders, and the development of mechanisms for monitoring and providing feedback about implementation progress and quality, and for fitting practices to be implemented to the local geo- graphical and cultural context. The ICT and CDT process models also

167M. Hurlburt et al. / Children and Youth Services Review 39 (2014) 160–168

share similarities. Both include processes specifically targeting initial col- laborative work among adopting stakeholders and organizations, using education to reduce uncertainty about EBP adoption during the explora- tion and preparation stages, and developing a core focus on EBP fidelity as part of implementation and sustainment. The models differ, however, in how many of the implementation processes are organized by an exter- nal change agent as opposed to individuals within the service system. Both the ARC and CDT models include a much more extensive role for an external change agent than the ICT model, which proposes that the planned actions and processes illustrated in Fig. 1 will result in the devel- opment of structural supports at the core of effective implementation. The ICT model relies more on establishing a process map for developing local structures to support implementation, and on facilitating desired inter- agency relationships, than on external consultant support.

Given notable common procedural aspects of different implementa- tion process models, the rich qualitative data gathered around imple- mentation of SafeCare in this study informs the ICT model and, to some degree, other implementation process models as well. One key finding from this qualitative study was the interconnectedness of the process components and the implementation supports they facilitated. Initial collaborative efforts among stakeholders, including recognition and discussion of differences, facilitated later problem solving and ne- gotiation around areas of disagreement and potential conflict. Staged roll out of implementation created opportunities for early success, which in turn facilitated communication and problem resolution and helped build “buy-in” and enthusiasm for the EBP. The presence of the seed team solidified perceptions of leadership at multiple levels and di- rectly addressed other key implementation drivers, including sustain- ing a focus on fidelity and fitting of the practice to be implemented to key local contextual variations. Our qualitative work supports argu- ments made by others that multi-component approaches to implemen- tation that address inter- and intra-organizational contextual issues are necessary to create an environment conducive to strong implementa- tion and sustainment (Aarons et al., 2011; Damschroder et al., 2009; Ferlie & Shortell, 2001; Fixsen, Blase, Naoom, & Wallace, 2009; Glisson & Schoenwald, 2005; Grimshaw et al., 2001; Grol & Grimshaw, 1999). Results from interviews clearly connected various aspects of the imple- mentation process to the overall supports for implementation. The in- terviews also are consistent with the idea that core implementation drivers reinforce and support one another (Fixsen et al., 2009).

The ICT model includes notably less involvement of external change agents, such as intermediary or brokering organizations, or outside organizational development consultants than other process models. The foundation and its willingness to support initial imple- mentation phases were important to SafeCare implementation be- cause the foundation served as the initial convening agent and was responsible for providing financial support to initiate implementa- tion of SafeCare. However, the ICT model relies more heavily on orga- nizing a series of actions and local structures designed to create inherent inter-organizational interaction and mutual support around an EBP implementation. Such structures can include service system contracts with community-based organizations that deliver services, or memorandums of agreement that support the ICT structure. The lead- ership at the system level, and at the organizational level is needed to fa- cilitate formal agreements for the ICT structure. In addition, the ICT model presumes that the presence of a local seed team will create a locus of leadership at the practice level that reinforces initial commit- ments from higher-level administrators and accountability to service provision and ongoing intervention fidelity. The model presumes that the role of the seed team, both to learn and to transmit a practice to other colleagues, will require the team to identify and resolve salient ad- aptation issues that arise in the local context. The existence of the seed team, with its coaching and supervision responsibilities, is designed to create a focus on fidelity that is transmitted directly to practitioners working in different organizations delivering the new practice. The seed team is placed in a structural role to carry a voice of leadership

and fidelity focus throughout the course of implementation, rather than having this spearheaded by an external change agent. It is worth noting that the ICT model seeks to create conditions for successful im- plementation of an identified practice change, not to influence how all practice changes occur, although it is presumed that experience with the model with improve local change efforts more broadly.

Initial results suggest that the ICT model steps do lead to many of the intended structural supports or drivers for effective implementation. However, they also reveal that the ICT process may not sufficiently fos- ter some of the supports that receive direct attention in other process models. Our qualitative data suggest that including explicit processes for regular communication in the ICT model is crucial to improve the ability of involved stakeholders to identify and address potential con- flicts in ways that build trust and continued cooperation. Participants' experiences suggested that there are likely to be unexpected areas of conflict and disagreement in any implementation effort and that clear communication and a problem solving orientation will facilitate effec- tive resolution of such issues.

The ICT model does appear to have resulted in a systemic focus on fi- delity in a manner that facilitates appropriate local adaptation. For many years, the potential inflexibility and lack of local cultural relevance of EBPs have been significant limiting concerns (Bernal, 2006; Bernal, Jimenez-Chafey, & Rodriguez, 2009; Bernal & Scharron-del-Rio, 2001; Castro, Barrera, & Martinez, 2004; Kumpfer, Alvarado, Smith, & Bellamy, 2002; Lau, 2006; Matos, Torres, Santiago, Jurado, & Rodriguez, 2006). Ex- periences from participants using the ICT model with SafeCare suggest that such concerns have some validity, but that they are addressable through planned implementation structures and processes. Questions did arise about the fit of SafeCare with local cultural nuances, particular- ly for Latino families. The seed team, with a liaison role between the EBP developer and trained home visitors and families receiving services, did address issues of how to adapt SafeCare to local conditions while re- maining true to core components of the practice. The ICT model envi- sions this as a negotiated process between a locally based team and EBP developers who know that this team will assume responsibility over time for system-wide fidelity maintenance. These kinds of negotia- tions were observed as the seed team recognized the need for SafeCare adaptation for the county and worked with SafeCare developers to make appropriate adaptations (Finno-Velasquez et al., under review).

At a broad level, our qualitative results largely support the role that ICT model processes play in generating structural supports for imple- mentation and sustainment of a system-wide EBP-driven quality im- provement effort. Several areas were identified as likely needing further attention in the implementation model, including a process for detailing appropriate communication patterns early during implemen- tation and a process for creating forums in which differences among participating organizations can be identified, discussed, and resolved.

In the presence of multiple implementation models a natural question that arises concerns the conditions under which a particu- lar implementation model is particularly relevant. The ICT model seems particularly well suited to circumstances in which an EBP is planned for roll-out in a given organization or across a contiguous geographic area in which authority for initiating and supporting the effort falls under the domain of a small number of administrative entities, in this case county child welfare services. As opposed to fo- cusing on collaborative information sharing among organizations implementing an EBP in different locations, or emphasizing the internal organizational cultural and climate of implementing organizations, the ICT model is relevant when local organizations are in a position to work together to develop a shared core infrastructure for implementing and sustaining expertise in a practice that will be scaled up across a broad geo- graphic area. The ICT model is somewhat unique in its emphasis on creat- ing implementation supports through formal and strategic structuring and staging of the implementation process rather than through the exten- sive involvement of an outside organization that organizes and pushes the implementation process forward.

168 M. Hurlburt et al. / Children and Youth Services Review 39 (2014) 160–168

5. Conclusions

The ICT implementation process model developed out of the collab- orative experience of researchers and local agency partners. Use of its implementation processes has led to sustained and widespread use of SafeCare, an evidence-based neglect prevention model, in one large county, and resulted in the phased transitioning of expertise from model developers to the local context. Although some aspects of the im- plementation model may benefit from enhancement, results suggest that the process model generates strong structural supports for imple- mentation and creates conditions in which tensions between EBP struc- ture and local contextual needs can be resolved in ways that support the expansion and maintenance of the EBP while preserving its potential for public health benefit.

Acknowledgments

This study was supported by the National Institute of Mental Health grants 2R01MH072961 and P30MH074678. The authors thank the community-based organizations, case-managers, and supervisors that made this study possible. The authors declare no other conflicts of interest.

References

Aarons, G. A. (2006). Transformational and transactional leadership: Association with at- titudes toward evidence-based practice. Psychiatric Services, 57(8), 1162–1169.

Aarons, G. A., Green, A. E., Palinkas, L. A., Self-Brown, S., Whitaker, D. J., Lutzker, J. R., et al. (2012). Dynamic adaptation process to implement an evidence-based child maltreat- ment intervention. Implementation Science, 7(32), 1–9.

Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2011). Advancing a conceptual model of evidence-based practice implementation in child welfare. Administration and Policy in Mental Health and Mental Health Services Research, 38(1), 4–23.

Aarons, G. A., & Palinkas, L. A. (2007). Implementation of evidence-based practice in child welfare: Service provider perspectives. Administration and Policy in Mental Health and Mental Health Services Research, 34, 411–419.

Aarons, G. A., Sommerfeld, D. H., Hecht, D. B., Silovsky, J. F., & Chaffin, M. J. (2009). The im- pact of evidence-based practice implementation and fidelity monitoring on staff turnover: Evidence for a protective effect. Journal of consulting and clinical psychology, 77(2), 270.

Aarons, G. A., Wells, R. S., Zagursky, K., Fettes, D. L., & Palinkas, L. A. (2009). Implementing evidence-based practice in community mental health agencies: A multiple stakeholder analysis. American Journal of Public Health, 99(11), 2087–2095.

Allen, P., Brownson, R. C., Duggan, K., Stamatakis, K. A., & Erwin, P. C. (2012). The makings of an evidence-based local health department: Identifying administrative and man- agement practices. Frontiers in Public Health Services and Systems Research, 1(2), 2.

Backer, T. E. (2000). The failure of success: Challenges of disseminating effective sub- stance abuse prevention programs. Journal of Community Psychology, 28(3), 363–373.

Bernal, G. (2006). Intervention development and cultural adaptation research with di- verse families. Family Process, 45(2), 143–151.

Bernal, G., Jimenez-Chafey, M. I., & Rodriguez, M. M.D. (2009). Cultural adaptation of treat- ments: A resource for considering culture in evidence-based practice. Professional Psychology-Research and Practice, 40(4), 361–368.

Bernal, & Scharron-del-Rio (2001). Are empirically supported treatments valid for ethnic minorities? Toward an alternative approach for treatment research. Cultural Diversity and Ethnic Minority Psychology, 7, 328–342.

Bertram, R. M. (2008). Establishing a basis for multi-system collaboration: Systemic team development. Journal of Sociology and Social Welfare, 35(4), 9–27.

Bond, G., Drake, R., McHugo, G., Rapp, C., & Whitley, R. (2009). Strategies for improving fidelity in the national evidence-based practices project. Research on Social Work Practice, 19(5), 569.

Burke, M. J. (1993). Computerized psychological testing: Impacts on measuring predictor constructs and future job behavior. In N. Schmitt, & W. C. Borman (Eds.), Personnel selection in organizations (pp. 203–239). : Jossey-Bass.

Castro, F., Barrera, M., & Martinez, C. (2004). The cultural adaptation of prevention inter- ventions: Resolving tensions between fidelity and fit. Prevention Science, 5(1), 41–45.

Chaffin, M., Hecht, D., Bard, D., Silovsky, J. F., & Beasley, W. H. (2012). A statewide trial of the SafeCare home-based services model with parents in Child Protective Services. Pediatrics, 129(3), 509–515.

Chamberlain, P., Price, J., Reid, J., & Landsverk, J. (2008). Cascading implementation of a foster and kinship parent intervention. Child Welfare, 87(5), 24–48.

Crea, T. M., Crampton, D. S., Abramson-Madden, A., & Usher, C. L. (2008). Variability in the implementation of Team Decisionmaking (TDM): Scope and compliance with the fam- ily to family practice model. Children and Youth Services Review, 30(11), 1221–1232.

Dai, Y. S., Xie, M., Poh, K. L., & Liu, G. Q. (2003). A study of service reliability and availability for distributed systems. Reliability Engineering & System Safety, 79(1), 103–112.

Damschroder, L., Aron, D., Keith, R., Kirsh, S., Alexander, J., & Lowery, J. (2009). Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implementation Science, 4, 50.

Edmondson, A.C. (2004). Learning from failure in health care: Frequent opportunities, pervasive barriers. Quality and Safety in Health Care, 13(Suppl. 2), ii3–ii9.

Edmondson, A. C., & Roloff, K. S. (2009). Overcoming barriers to collaboration: Psycholog- ical safety and learning in diverse teams. Team effectiveness in complex organizations: Cross-disciplinary perspectives and approaches, 34.

Emerson, R., Fretz, R., & Shaw, L. (1995). Writing ethnographic fieldnotes. Chicago: University of Chicago Press.

Feldstein, A., & Glasgow, R. (2008). A practical, robust implementation and sustainability model (PRISM). Joint Commission Journal on Quality and Patient Safety, 34(4), 228–243.

Ferlie, E. B., & Shortell, S. M. (2001). Improving the quality of health care in the United Kingdom and the United States: A framework for change. Milbank Quarterly, 79(2), 281–315.

Finno-Velasquez, M. F., Fettes, D., Aarons, G. A., & Hurlburt, M. S. (under review). Cultural adaptation of an evidence-based home visitation program: Latino clients' experiences of service delivery during implementation. (under review).

Fixsen, D., Blase, K., Naoom, S., & Wallace, F. (2009). Core implementation components. Research on Social Work Practice, 19(5), 531.

Fixsen, D. L., Naoon, S. F., Blase, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation research: A synthesis of the literature. Tampa: University of South Florida, Louis de la Parte Florida Mental Health Institute, the National Implementation Research Network.

Frambach, R. T., & Schillewaert, N. (2002). Organizational innovation adoption: A multi-level framework of determinants and opportunities for future research. Journal of Business Research, Special Issue: Marketing Theory in the Next Millennium, 55(2), 163–176.

Glisson, C., & Schoenwald, S. (2005). The ARC organizational and community intervention strategy for implementing evidence-based children's mental health treatments. Mental Health Services Research, 7(4), 243–259.

Gray, B. (1990). Building interorganizational alliances: Planned change in a global envi- ronment. Research in organizational change and development, 4, 101–140.

Greenhalgh, T., Robert, G., Macfarlane, F., Bate, P., & Kyriakidou, O. (2004). Diffusion of in- novations in service organizations: Systematic review and recommendations. Milbank Quarterly, 82(4), 581–629.

Grimshaw, J., Shirran, L., Thomas, R., Mowatt, G., Fraser, C., Bero, L., et al. (2001). Changing provider behavior: An overview of systematic reviews of interventions. Medical Care, 39(8), II2–II45.

Grol, R., & Grimshaw, J. (1999). Evidence-based implementation of evidence-based med- icine. Journal on Quality Improvement, 25, 503–513.

Hollenbeck, J. R., Ilgen, D. R., Sego, D. J., Hedlund, J., Major, D. A., & Phillips, J. (1995). Mutlilevel theory of team decision making: Decision performance in teams incorpo- rating distributed expertise. Journal of Applied Psychology, 80(2), 292.

Klein, K. J., Conn, A.B., & Sorra, J. S. (2001). Implementing computerized technology: An organizational analysis. Journal of Applied Psychology, 86(5), 811–824.

Klein, K. J., & Sorra, J. S. (1996). The challenge of innovation implementation. Academy of Management Review, 21(4), 1055–1080.

Kumpfer, K. L., Alvarado, R., Smith, P., & Bellamy, N. (2002). Cultural sensitivity and adaptation in family-based prevention interventions. Prevention Science, 3(3), 241–246.

Lau, A. S. (2006). Making the case for selective and directed cultural adaptations of evidence-based treatments: Examples from parent training. Clinical Psychology: Science and Practice, 13(4), 295–310.

Lutzker, J. R., Bigelow, K. M., Doctor, R. M., Gershater, R. M., & Greene, B. F. (1998). An ecobehavioral model for the prevention and treatment of child abuse and neglect. In J. Lutzker (Ed.), Handbook of child abuse research and treatment (pp. 239–266). Springer.

Matos, M., Torres, R., Santiago, R., Jurado, M., & Rodriguez, I. (2006). Adaptation of parent– child interaction therapy for Puerto Rican families: A preliminary study. Family Process, 45, 205–222.

Mendel, P., Meredith, L., Schoenbaum, M., Sherbourne, C., & Wells, K. (2008). Interven- tions in organizational and community context: A framework for building evidence on dissemination and implementation in health services research. Administration and Policy in Mental Health and Mental Health Services Research, 35(1–2), 21–37.

Nadler, D. A., & Tushman, M. L. (1977). A diagnostic model for organizational behavior. In R. J. Hackman, E. E. Lawler, L. W. N. Porter, & P S (Eds.), Perspectives on behavior in organizations. New York: McGraw-Hill.

Palinkas, L. A., & Aarons, G. A. (2009). A view from the top: Executive and manage- ment challenges in a statewide implementation of an evidence-based practice to reduce child neglect. International Journal of Child Health and Human Development, 2(1), 47–55.

Porras, J., & Robertson, P. (1992). Organizational development. In M. Dunnette, & L. Hough (Eds.), Handbook of industrial and organizational psychology (pp. 719–822). London: Consulting Psychologists Press.

Sandelowski, M., & Barroso, J. (2003). Classifying the findings in qualitative studies. Qualitative Health Research, 13(7), 905–923.

Sosna, T., & Marsenich, L. (2006). Community development team model: Supporting the model adherent implementation of programs and practices. Sacramento, CA: California Institute of Mental Health.

Stetler, C. B., McQueen, L., Demakis, J., & Mittman, B.S. (2008). An organizational frame- work and strategic implementation for system-level change to enhance research- based practice: QUERI Series. Implementation Science, 3, 30.

Trist, E. (1985). Intervention strategies for interorganizational domains. In R. Tannenbaum, N. Margulies, & F. Massarik (Eds.), Human systems development. San Francisco: Jossey-Bass.