SOCW wk 10 Discussion: Assessing Outcomes

Full Terms & Conditions of access and use can be found at https://www.tandfonline.com/action/journalInformation?journalCode=wasw21

Administration in Social Work

ISSN: 0364-3107 (Print) 1544-4376 (Online) Journal homepage: https://www.tandfonline.com/loi/wasw20

Designing Evaluations in Child Welfare Organizations: An Approach for Administrators

Catherine Lawrence , Jessica Strolin-Goltzman , James Caringi , Nancy Claiborne , Mary McCarthy , Erin Butts & Kelsey O'Connell

To cite this article: Catherine Lawrence , Jessica Strolin-Goltzman , James Caringi , Nancy Claiborne , Mary McCarthy , Erin Butts & Kelsey O'Connell (2013) Designing Evaluations in Child Welfare Organizations: An Approach for Administrators, Administration in Social Work, 37:1, 3-13, DOI: 10.1080/03643107.2011.607887

To link to this article: https://doi.org/10.1080/03643107.2011.607887

Published online: 20 Dec 2012.

Submit your article to this journal

Article views: 1989

View related articles

Citing articles: 1 View citing articles

Administration in Social Work, 37:3–13, 2013 Copyright © Taylor & Francis Group, LLC ISSN: 0364-3107 print/1544-4376 online DOI: 10.1080/03643107.2011.607887

Designing Evaluations in Child Welfare Organizations: An Approach for Administrators

Catherine Lawrence School of Social Welfare, University at Albany, State University of New York,

Albany, New York, USA

Jessica Strolin-Goltzman Department of Social Work, College of Education and Social Services,

University of Vermont, Burlington, Vermont, USA

James Caringi School of Social Work, University of Montana, Missoula, Montana, USA

Nancy Claiborne and Mary MCCarthy School of Social Welfare, University at Albany, State University of New York,

Albany, New York, USA

Erin Butts Institute for Educational Research and Service, University of Montana, Missoula,

Montana, USA

Kelsey O’Connell School of Social Work, University of Montana, Missoula, Montana, USA

Evaluations in child welfare settings present challenges and opportunities for administrators. This paper presents a framework for considering such evaluations and proposes mixed methods to assess participatory interventions for sustainable organizational change. A progression of three studies illustrates this approach and draws on results to demonstrate how evaluation designs impact findings.

Keywords: child welfare, mixed method research, organizational interventions, participatory action research

The child welfare system in the United States serves some of the nation’s most vulnerable chil- dren and families. The organizations and government systems that work to address their needs are increasingly held accountable for the work they do. The Adoption and Safe Families Act of

Correspondence should be addressed to Catherine Lawrence, School of Social Welfare, University at Albany, State University of New York, 135 Western Ave., Albany, NY 12222, USA. E-mail: [email protected]

3

4 LAWRENCE ET AL.

1997, which initiated Child and Family Service Reviews, in addition to program evaluations, case reviews, and other accountability efforts, reflects a genuine desire to serve children and families well. One important area of this accountability is the need for program evaluations that meet the needs of social work managers in child welfare organizations. Ideally, evaluations should be able to 1) promote organizational growth and learning (Cherin & Meezan, 1998) and 2) con- tribute to an evidence base for child welfare practice with the best science available (Samuels, 2010).

To that end, this paper offers an approach to evaluation design that uses mixed methodology to assess participatory interventions for sustainable organizational change. The authors identify why this evaluation approach is suitable for child welfare settings and then go on to describe the three central elements of this approach. While these elements can be applied to human service orga- nizations in many service arenas, this paper uses a progression of three studies in child welfare services to illustrate. This approach can help inform social work managers as they strive to design and conduct feasible evaluations that not only use appropriate scientific methods but also inform improvements in practice and in the delivery of services. In particular, the authors address the use of this approach in rural and American Indian settings.

THE CONTEXT OF CHILD WELFARE

There are three aspects of the child welfare system that are important for social work managers to consider when engaging in evaluations. First, child welfare settings are unique systems that oper- ate within larger contexts. Second, child welfare organizations themselves have a direct influence on workers and practice with families. Third, the goals of evaluation in child welfare often go beyond assessing outcomes for one program and aim to improve child welfare practice. This section describes these unique features and links them to specific evaluation approaches that take these into account and builds on them as strengths. Together, these evaluation approaches make up a part of scaffolding for evaluation in child welfare.

Unique Aspects of Child Welfare Settings

Each state child welfare agency is a distinct organization with its own history and culture, and local district agencies and non-governmental organizations are also unique. Even different units within the same organization are distinct, creating challenges for more controlled evaluation studies.

Further, child welfare organizations operate within our larger social and economic contexts. Changes in internal and external socio-political environments can impact programs, policies, prac- tice, and evaluation. As a public service system, new leaders can be appointed following state or local political changes. Changes in government regulations, funding sources, client demands, and community and stakeholder support have a tremendous impact. Such changes can advance an ini- tiative or distract and immobilize leadership, and create conditions for low morale in the workforce (Golden, 2009).

Identifying and assessing leadership changes and leadership’s subsequent reactions and strategies to environmental changes are therefore important evaluation tasks. Some managers might welcome a research team with a set, manualized intervention. However, agency leaders have many respon- sibilities and are not always able to prioritize research and evaluation efforts. This combination of diversity within and across settings, internal and external pressures, and leadership changes sug- gest that research designs with mixed methodologies can aid in increasing the rigor of evaluation, while also providing in-depth understanding of the relationship between contextual factors and outcomes.

DESIGNING EVALUATIONS IN CHILD WELFARE ORGANIZATIONS 5

The Role of Organizations

Organizations play a central role both in the delivery of child welfare services and the success of evaluation. Understanding this dynamic is important to social work managers in planning for evaluations. For example, there is a clear connection between organizational factors and the frontline workforce (DePanfilis & Zlotnik, 2008; Strolin-Goltzmanet al., 2008). Researchers have identified workforce turnover as a major concern in the field (GAO, 2003; Flower et al., 2005), and there is a growing body of research that identifies the role of organizational factors on worker intention to leave (Drake &Yadama, 1996; Landsman, 2001; Mor Barak, Nissly, & Levin 2001; Smith, 2005; Strolin et al. 2009). Thus, interventions that can address these organizational factors are part of making positive changes in the child welfare system. Such interventions, of course, must include evaluations that assess their impact on the organization and on the workforce.

In addition, researchers have identified several challenges when conducting evaluation in child welfare settings. Worker compliance with evaluation procedures, for example, can vary. The pres- sures of work in human service organizations affect frontline implementation (Lipsky, 1980). Recent research describes the bureaucratic pressures that hinder workers’ ability to utilize best practices in public child welfare (Smith & Donovan, 2003). To manage these demands, workers use discretion in deciding when and how to complete job tasks. Often organizational demands force workers to pri- oritize as a coping mechanism (Brodkin, 1990). For example, workers may prioritize service tasks over evaluation tasks, especially if workers have little participation in or understanding of the eval- uation and rarely see findings influence their practice or work environment. However, if the design and implementation process are participatory, workers may be more invested in the evaluation, can connect findings to positive change, and may prioritize evaluation tasks in their work (Argyris & Schön, 1974, Fabricant & Burghardt, 1992).

The Goal of Evaluation in Child Welfare

A central point worth considering is the goal of evaluation in child welfare. At times, the institution’s goal for evaluation differs from the researchers’ evaluation goal. Researchers may wish for clean hypothesis testing and further understanding of how the results lead to certain outcomes. Child welfare organizations, however, may want to dispatch their evaluation obligations with the least effort possible. Perhaps a worthy goal is to build a body of knowledge that child welfare agencies can apply to their work and to utilize the most appropriate research methods for the task at hand.

An additional goal, however, is for researchers and agencies to become partners in evaluation and, through the evaluation, better understand how the research process can best inform child wel- fare practice and agency change. This kind of partnership in evaluation changes the role of the investigator and changes the assessment of the work and what constitutes success. It also elevates the need for pragmatism in implementing evaluations. In this context, it is helpful to make a dis- tinction between evaluation, research, and implementation study. In projects that are exclusively research studies, a research question drives the design. In implementation study, the context drives the design. In evaluation, it is the program that drives the design.

The opportunity to engage in meaningful systems change through child welfare evaluation is significant. To this end, the following section describes a pragmatic approach to evaluation in child welfare settings that we propose can inform child welfare practice, lead to positive changes that support child welfare goals of safety and permanency, and is perhaps a combination of evaluation science, pragmatic implementation, and the art of building partnerships.

Evaluation Scaffolding for Child Welfare

Evaluation science is not a new field. Program evaluation has been utilized in a variety of organiza- tional settings, from businesses to public services. Rossi et al. (2004) identify five types of program

6 LAWRENCE ET AL.

evaluation: needs assessment, program theory, process analysis, impact analysis, and cost-benefit analysis. However, there has been limited discussion of how to best apply evaluation science in the child welfare field. Although research literature shows the importance of evaluation in foster care and adoption services as well as professional education for the child welfare workforce, evaluation in child welfare has largely focused on staff training (Antle, Barbee, Sullivan, & Christensen, 2009; Collins, Amodeo, & Clay, 2008; Christenson & McMurtry, 2009). Evaluation also has frequently been conducted as an independent evaluation of an individual program rather than as a more rigorous “develop and test” intervention research model (Barth, 2009). Thus a first step in moving the field toward coherent evaluation practices is to identify the pieces necessary to build solid scaffolding for evaluation and intervention research in child welfare.

The challenges and goals discussed above support a participatory evaluation design that obtains buy-in of frontline workers, supervisors, and agency managers. The importance of the organiza- tional and institutional contexts of the child welfare system is a central element in evaluation in child welfare settings. Public and private service agencies are more than the context in which work with families takes place; child welfare organizations have a direct influence on the nature of the work and the workers who manage, supervise, and deliver services (Lipsky, 1980; Fabricant & Burghardt, 1992). Finally mixed methods evaluations have the potential to meet the goals of the researcher as well as the organization with in-depth, generalizable findings that confirm or reject research hypotheses (Plano & Creswell, 2008).

EVALUATION ELEMENTS

The following section describes three elements to include in scaffolding for child welfare evalua- tion: 1) participatory designs, 2) a thorough understanding of organizational context, and 3) mixed methods designs.

Participatory Designs

The participatory nature of evaluation in child welfare settings is a central design element. Participatory action research (PAR) “seeks to understand and improve the world by changing it. At its heart is collective, self-reflective inquiry that researchers and participants undertake, so they can understand and improve upon the practices in which they participate and the situations in which they find themselves” (Baum, MacDougall, & Smith, 2006, p. 854). There are three critical dif- ferences between PAR and conventional research. First, the purpose of PAR is to enable action through a reflexive cycle that includes the collection and analysis of data by participants. Secondly, researchers and community members involved in PAR share power throughout the process from selecting the research topic, collecting data, completing analysis, and deciding on action steps derived from findings. Thirdly, PAR advocates active involvement of study participants. The degree to which this is possible depends on the willingness of participants to be involved in the research (Baum et al. 2006).

When conducting research with child welfare agencies or with vulnerable populations such as minors or American Indian Tribes, sometimes the “develop and test” model will work only if the program was developed in a participatory manner (Hicks, 2009). Further, child welfare evaluation practices should include feedback loops that return the findings back to the field for improvement purposes (Stagner, 2009). Research aimed at solving important problems in child welfare must reflect a basic understanding of the issues. Such understanding can come through research participants’ co-design of research questions and approaches. This also brings a more comprehensive view of the data, as the questions are more likely to explore the antecedents, causes, and correlates of the practices being examined. Finally, such involvement engages key participants

DESIGNING EVALUATIONS IN CHILD WELFARE ORGANIZATIONS 7

in co-designing strategies to develop solutions that are data-driven (Greenwood & Levin, 1998; Kemmis & McTaggart, 2000).

Organizational Contexts

Closely related to participatory design is the need to understand how organizational factors impact interventions and evaluations in child welfare. Because organizational change is fluid, evaluations must assess the change process and factors that impede or promote change. These factors include individual perceptions of the change, what organizational and environmental issues impact the change process, how decisions are made, what organizational data and feedback mechanisms are utilized, and, if workgroups are utilized, how and if team cohesion develops.

Leadership commitment is also crucial to organizational contexts. Heifetz (2006) states that leadership becomes necessary when organizational survival is contingent on facing challenges in a changing world, thus promoting vital changes in organizational priorities, attitudes, and behaviors. Successful leadership in social systems has been defined as having three attributes (Hackman, 2008): clients and stakeholders derive satisfaction and attain expected outcomes from service provision; the organization has capacity to improve performance over time; and workers realize fulfillment and learning from their work. This definition is positioned within the functional approach to leader- ship which, simply stated, ensures necessary functions that foster system viability (Barnard, 1983). Leadership spans the entire functioning of an organization, not just the top tier of agency admin- istration. Administrators, managers, supervisors, and line staff have leadership functions, and they all contribute to critical system functions and accomplishments (For leadership competencies, see Bernotavicz, Brittain, & McDaniel, 2010).

Gaining access to the full span of organizational functioning for evaluation purposes therefore requires support from all levels of leadership. Throughout evaluation, top leadership is required for research legitimacy and access. The characteristics of a leader and his/her interaction within the organization are important factors for effectively initiating and guiding a complex change process. Agency leaders can communicate and reinforce goals, operationalize tasks, and provide support and direction for employees (Vroom & Jago, 2007). Equally important is gaining workforce participa- tion in the change initiative and evaluation. Identifying formal and informal leaders and promoting a participatory process help staff perceive the evaluation as information to identify areas for cele- bration and areas to improve, as opposed to perceptions of judgment, which can impede evaluation engagement.

Mixed Methods

Mixed methods designs can capture the complex organizational phenomena discussed above. Mixed methods designs are common in evaluation, which is a “quintessentially applied field” (Rallis & Rossman, 2003, p. 492). Rallis and Rossman characterize evaluation research designs as reflecting a series of compromises between the needs of stakeholders (including funders), the parameters of research epistemology, and the data and financial resources available for the evaluation (2003).

A particular benefit of organizational evaluations in child welfare is the opportunity to use exist- ing administrative data in combination with other data such as surveys, client and staff interviews, and case studies (Clark-Plano & Creswell, 2008). This allows rich descriptions of findings on the process of change and strengthens all findings through data triangulation, member checking, and searching for discrepant evidence and negative cases (Seidman, 1998).

Qualitative approaches, such as focus groups (with leaders, managers, line workers, and clients) and participant observation (e.g., observing workgroups), provide a fundamental understanding of the nature of the relationship between the change initiative and the change process. Qualitative analysis provides in-depth understanding of the factors operating during the transition to the

8 LAWRENCE ET AL.

desired change. In addition, qualitative approaches can capture decision-making processes from all perspectives of the agency (Patton, 2002).

The inclusion of quantitative approaches, such as pre- and post-surveys, strengthens the program evaluation findings. For example, organizational measures allow researchers to study the role and strength of specific factors on evaluation outcomes. Such quantitative measures allow the agency staff to engage in meaningful continuous quality assessment in identifying areas for practice improve- ment. It also provides pertinent information to important stakeholders for demonstrating agency service outcomes. Finally, when inferential statistics clarify the relationship between measured factors, evaluation findings increase in capacity to generalize to the larger practice community. Mixed method designs are thus a valuable tool for evaluators working with child welfare organizations.

PROGRESSION OF PROJECTS

This section describes a progression of three projects that study worker turnover in child welfare organizations. These three projects were developed consecutively over more than a decade, and each builds on the knowledge and experience of the prior work. The studies highlight one of the three elements of design presented above. The first, a statewide study of worker turnover, presents details of participatory designs. The second, an organizational intervention in public child welfare agencies, highlights the benefits of mixed methods approaches. The third, an organizational inter- vention in nonprofit child welfare organizations, describes the application of organizational research in evaluation. Together, the projects contribute to an evolving understanding of worker turnover, child welfare organizations, and evaluation. In addition, the authors suggest adaptations for urban, rural, and tribal organizations.

A Participatory Study of Turnover

Twelve county commissioners experiencing persistent high turnover in their workforce initiated the first study, a study of high turnover localities that led to a study of comparable low turnover localities. They sought information about factors that would result in retention and requested a survey to help them understand these factors better. A university-based research team had research expertise, and the commissioners had expertise in their organizational systems and thus guided the development of the instrument. Using existing instruments from child welfare turnover studies (Scannapieco & Connell, 1999; Dickinson & Perry, 2002), a work group of commissioners and academics revised the questions and developed the study protocol.

All frontline workers and supervisors were invited to complete the survey at agency-wide meetings. Commissioners and their leadership teams received preliminary findings, which were discussed with members of the research team. Subsequently, final reports were prepared based on a collective understanding of the findings.

The researchers had to manage their concerns about design rigor and let go of questions they believed were relevant to understanding retention. As trust developed, however, commissioners were able to hear the importance of meeting certain methodological criteria to ensure valid findings. The research team emphasized a strong response rate and used interviews with more than 100 staff to triangulate survey data. In particular, the interviews helped agencies better understanding issues like “burnout” and respect. The research team reviewed findings with agency staff who then worked in small groups to confirm or modify interpretations of the findings based on their experiences in the agency. This further increased the validity of the findings.

The commissioners from high-turnover counties then asked whether the predominant factors influencing retention were challenges in all child welfare agencies or whether low turnover child welfare agencies were employing different practices that more positively impacted those factors.

DESIGNING EVALUATIONS IN CHILD WELFARE ORGANIZATIONS 9

With help from the original group of commissioners, we recruited 12 counties with similar demo- graphics but low turnover to participate in the survey. The results supported the conclusion that leaders have a direct impact on the culture, climate, and practices within their organization. The findings were significantly different in areas of administrative and organizational practice, work-life fit, the balance of paperwork and direct client contact, and salary (Author).

The participatory approach used to design and conduct this research influenced the usefulness of the findings for guiding organizational practice changes. The research was designed to answer questions of great import to commissioners struggling with high turnover and they took the findings seriously; several began initiatives to address leadership practices shown to impact retention. As one commissioner stated, “There were two of us sitting in administrative positions in the agency. When 80%+ of the staff identified administrative and organizational practices as the single biggest reason for leaving the job, there was only one place to look for the source of the problem” (Brady, 2008). Several commissioners agreed to participate in the successful submission of a federal grant applica- tion funded by the Children’s Bureau on workforce recruitment and retention. The grant funded a new project to study an organizational intervention designed to support retention, described below.

Using Mixed Methods to Deepen Understanding

The second project combined an organizational intervention with a mixed methods evaluation designed to assess the impact of the intervention on worker intent to leave the agency. The interven- tion itself was participatory. It created agency-based work teams to address organizational factors related to turnover in the agency. These teams, supported by an external facilitator, used data col- lected in the course of the evaluation to identify and select organizational issues the team wanted to address. The team then progressed through a series of steps to implement an organizational initiative.

The initial evaluation design was a multi-site quasi-experimental longitudinal design using a pre-post survey of agency staff. The focus of analysis was changes in actual turnover, intention to leave, burnout, agency commitment, and job satisfaction. As the evaluation progressed, however, it was apparent that it would not capture the process of change within the organization as a result of the intervention. The evaluation design was therefore expanded to include qualitative methods of collecting and analyzing naturally occurring data generated by the agency teams.

In this project, the quantitative and qualitative parts of the evaluation were quite separate. There were two different research questions: one for the quantitative data (What was the impact of the organizational intervention on worker turnover and related factors?) and the other for the qualitative data (What was the nature of the work and process that occurred in the agency work team during the intervention?). The data were from different sources and collected and managed independently. In addition, two distinct teams of researchers did initial analysis. However, following initial analysis the entire evaluation team met to discuss results and a second round of analysis took place. The end results gave a fuller picture of the overall impact of the intervention not only on worker turnover but also on changes the organizations planned and implemented during the intervention.

The expansion of the evaluation design to a mixed method approach deepened the understanding of the change processes that contribute to turnover. For example, quantitative analysis of surveys identified factors associated with intent to leave, but the addition of the qualitative analysis of the work of the agency teams clarified the actual agency changes that occurred. A fuller discussion of these finding are published elsewhere (Strolin-Goltzman et al., 2009).

Expanding the Organizational Research

A third project, currently underway, combines lessons from the first two studies and expands the organizational intervention and research on organizational change into not-for-profit child welfare organizations. The project collaborates with agencies for approximately 15 months to address a

10 LAWRENCE ET AL.

substantial, agency-identified workforce challenge. The same solution-focused, team-based model from the second study is used to develop and implement a solution to the workforce challenge. A facilitator with expertise in organization and management works closely with the team to provide relevant resources.

The study is a pre-post, comparison group design that evaluates if intervention leads to sustain- able system changes that strengthen and support the workforce. It expands the earlier research by adding survey instruments to assess organizational commitment, readiness for change, and culture and climate. The evaluation, building on existing organizational change research, focuses on cli- mate and culture, openness to a change process, i.e., readiness for change (Jones & James, 1979; James, James, & Ashe, 1990; Cooke &Szumal, 1993; Glisson, 2000), employee satisfaction, and job commitment (Spector, 1985; Jaskyte, 2003).

For this project, organizational culture is defined as the behavioral expectations and norms that characterize the way work is done and the way priorities are made within the organization (Cooke & Szumal, 1993; Verbeke et al, 1998, Glisson, 2007). Climate is defined as employees’ shared perceptions of how their work environment impacts them (Jones & James, 1979; Joyce & Slocum, 1984; Glisson, 2007). Workers’ satisfaction and commitment to the job mediates organizational climate and culture, as well as directly impacts and predicts individual motivation and performance, including turnover (Parker et al., 2003). Thus workers’ perceptions of the work environment, work attitudes, and motivation may influence their individual behavior.

The project’s conceptual framework, drawing on the research literature and the work of the ear- lier studies, makes several assumptions. The first is that workers are motivated to the extent that they are satisfied, committed, and involved in their job and organization. Further, their performance is predicted by how these factors interact (mediate) with their perception of the organization’s climate and culture. Thus, measuring climate, culture, and readiness for change in an organization provides insight on the dynamics occurring among employees and between employees and organizational structures (Strolin-Goltzman, Auerbach, McGowan, & McCarthy, 2008). Measures of job satisfac- tion and job commitment give researchers a more complete understanding of the workers’ behaviors.

The advantage of the new surveys used in this project is the multiplicity of functions they pro- vide practitioners and researchers. Traditionally, surveys are used as a pre-post research measure for organizational change and intervention effectiveness. However, they can also isolate specific organi- zational issues and provide data to practitioners for areas needing improvement, thereby advancing the benefits of the intervention for the organization. These surveys also provide detailed information on the organization’s readiness for change. This allows practitioners to assess individual subscales to determine the areas of greatest concern and strategies for building organizational readiness for intervention.

IMPLICATIONS

Several implications for this evaluation approach are addressed below. One specific implication is the use of this approach with special populations such as rural and American Indian communi- ties. Three broader implications are addressed as well: first, that collaboration and partnerships are essential for this type of research to be successful; second, that there needs to be significant time and resources committed to the evaluation process; and, third and perhaps most importantly, that the ultimate goal of child welfare evaluations must be improvement of the system with the aim of better serving children and families.

Adaptations in Rural and American Indian Communities

The evaluation projects in this paper all involved rural and Native American communities, each with unique needs related to research. In these settings, a participatory, mixed methods evaluation

DESIGNING EVALUATIONS IN CHILD WELFARE ORGANIZATIONS 11

approach can offer research methods that are both ethical and effective (Hicks, 2009). Approaches to evaluation with rural and Native American communities deserve further attention than offered here. However, the following considerations, based on the above studies, provide some initial directions for social work managers planning evaluations in such communities.

In rural communities, often small and isolated, researchers must address concerns about confi- dentiality for research participants. Additional concerns include professional boundaries and dual relationships (i.e., a supervisor may be related to a subordinate worker or a CPS worker may inves- tigate a neighbor or relative). These issues can translate into reluctance to participate in evaluation tasks. A participatory design can address these concerns directly with agency leaders and workers by planning strategies to ensure confidentiality and limit dual relationship issues.

Evaluation research in American Indian communities can spark tensions related to diverse traditions, cultures, rural-urban-reservation factors, and the historically negative history Native American communities have with outside researchers. Strategically planned participatory research helps recognize these potential tensions and can honor community values, including trust, flexibility, reciprocity, suspending judgment, inclusion, and bravery, in the engagement process of participatory research. As in rural communities, participatory and mixed methods designs can help ensure that outside researchers are able to learn as much as possible about these differences (Novins et al., 2006). Most importantly, participatory and mixed methods designs can assure that Indian child wel- fare evaluation will result in information that Native American communities can use to improve their organizations and child welfare practice (Israel, Eng, Schulz, & Parker, 2005).

Broader Implications

Collaboration is a form of collective action. It is a developmental process that entails genuine changes in roles, rules, responsibilities and accountabilities, boundaries and jurisdictions, dis- courses, power relations, and both socialization and attribution mechanisms (Claiborne & Lawson, 2005). It results in a collective identity, consensus, and the capacity for collective action. As child welfare agencies have limited time for evaluation and often lack the technical expertise, partner- ships with community and university entities can advance goals for improved service access, quality, efficiency, and effectiveness in order to improve results.

Overall, the approach offered in this paper realistically ties evaluation research to child wel- fare practice. The road map to high-quality child welfare services must include useful evaluation practices.

REFERENCES

Antle, B. F., Barbee, A., Sullivan, J., & Christensen, N. D. (2009). The effects of training reinforcement on training transfer in child welfare. Journal of Child Welfare, 88(3), 5–26.

Argyris, C., & Schön, D. (1974). Theory in practice: Increasing professional effectiveness. San Francisco, CA: Jossey-Bass. Barnard, C. I. (1938). The functions of the executive. Cambridge, MA: Harvard University Press. Barth, R. (2009). What lies in the future for evaluation in the field of child welfare? Presented at the 2009 National Child

Welfare Evaluation Summit, Washington, DC, May 26–28. Baum, F., MacDougall, C., & Smith, D. (2006). Participatory action research. Journal of Epidemiological Community Health,

60(10), 854–857. Bernotavicz, F., Brittain, C., & McDaniel, N. (2010). Leadership competency framework. The National Child Welfare

Workforce Institute. Retrieved from http://ncwwi.org/docs/LeaderCompFrame5-31-2011.pdf Brodkin, E. Z. (1990). Implementation as policy politics. In D. Palumbo & D. J. Calista (Eds.), Implementation and the

policy process: Opening up the black box (pp. 107–118). New York, NY: Greenwood Press. Cherin, D., & Meezan, W. (1998). Evaluation as a means of organizational learning. Administration in Social Work, 22(2),

1–21. Christenson, L. B., & McMurtry, J. (2009). A longitudinal evaluation of the preservice training and retention of kinship and

nonkinship foster/adoptive families one and a half years after training. Journal of Child Welfare, 88(4), 5–22.

12 LAWRENCE ET AL.

Claiborne, N., & Lawson, H. A. (2005). An intervention framework for collaboration: Families in society. Journal of Contemporary Services, 86(1), 93–103.

Collins M. E., Amodeo, M, & Clay, C. (2008). Planning and evaluating child welfare training projects: Working toward a comprehensive conceptual model. Journal of Child Welfare, 87(5), 69–86.

Cooke, R. A., & Szumal, J. L. (1993). Measuring normative beliefs and shared behavioral expectations in organizations: The reliability and validity of the Organizational Culture Inventory. Journal of Psychological Reports, 72(3), 1299–1330.

Clark-Plano, L. P., & Creswell, J. W. (2008). The mixed methods reader. Thousand Oaks, CA: Sage Publications. DePanfilis, D., & Zlotnik, J. (2008). Retention of front-line staff in child welfare: A systematic review of research. Journal

of Children and Youth Services Review, 30(9), 995–1,008. Dickinson, N., & Perry, R. (2002). Factors influencing the retention of specially educated public child welfare workers. In K.

Briar-Lawson & J. Levy-Zlotnick (Eds.), Evaluation research in child welfare: Improving outcomes through university- public agency partnerships (pp. 89–103). Binghamton, NY: Haworth.

Drake, B., & Yadama, G. (1996). A structural equation model of burnout and job exit among child protective service workers. Journal of Social Work Research, 20(3), 179–187.

Fabricant, M., & Burghardt, S. (1992). The welfare state crisis and the transformation of social service work. Armonk, NY: ME Sharpe.

Flower, C., McDonald, J., & Sumski, M. (2005). Review of turnover in Milwaukee county private agency child welfare ongoing case management staff . Milwaukee, WI: Bureau of Milwaukee Child Welfare.

Glisson, C. (2007). Assessing and changing organizational culture and climate for effective services. Journal of Research on Social Work Practice, 17(6), 736–747.

Greenwood, D. J., & Levin, M. (1998). Introduction to action research: Social research for social change. Thousand Oaks, CA: Sage Publications.

Golden, O. (2009). Reforming child welfare. Washington, DC: Urban Institute Press. Hackman, J. R. (2008, June 11). Leadership: Advancing an intellectual discipline. Presented at the 2008 Harvard Business

School Centennial Colloquium, Cambridge, Massachusetts. Heifetz, R. (2006). Anchoring leadership in the work of adaptive progress. In F. Hesselbein & M. Goldsmith (Eds.), The

leader of the future 2: Visions, strategies, and practices for the new era. San Francisco, CA: Jossey-Bass. Hicks, S. (2009). The state of evaluation in child welfare: An Indian country perspective. Presented at the 2009 National

Child Welfare Evaluation Summit, Washington, DC, May 26–28. Israel, B. A., Eng, E., Schulz, A. J ., & Parker, E. A. (Eds.) (2005). Methods in community-based participatory research for

health. San Francisco, CA: Jossey-Bass. James, L. R., James, L. A., & Ashe, D. K. (1990). The meaning of organizations: The role of cognition and values. In

B. Schneider (Ed.), Organizational climate and culture (pp. 40–84). San Francisco, CA: Jossey-Bass. Jaskyte, K. (2003). Assessing changes in employees’ perceptions of leadership behavior, job design, and organizational

arrangements and their job satisfaction and commitment. Journal of Administration in Social Work, 27(4), 25–39. Jones, A. P., & James, L. R. (1979). Psychological climate: Dimensions and relationships of individual and aggregated work

environment perceptions. Journal of Organizational Behavior and Human Performance, 23(2), 201–250. Kemmis, S., & McTaggart, R. (2000). Participatory action research. In N. K. Denzin, Y. S. Lincoln (Eds.) The handbook of

qualitative research. Thousand Oaks, CA: Sage Publications. Landsman, M. (2001). Commitment in public child welfare. Social Service Review, 75, 386–419. Lipsky, M. (1980). Street-level bureaucracy: Dilemmas of the individual in public service. New York, NY: Russell Sage. Mor Barak, M., Nissly, J., & Levin, A. (2001). Antecedents to retention and turnover among child welfare, social work,

and other human service employees: What can we learn from past research? A review and meta-analysis. Social Service Review, 75(4), 625–661.

Novins, D. K., Freeman, B., Thurman, P. J., Iron Cloud-Two Dogs, E., Allen, J. A., LeMaster, P. L., & Deters, P. B. (2006). Principles for participatory research with American Indian and Alaska Native communities: Lessons from the Circles of Care Initiative. [Poster]. 2006 Conference Indigenous Suicide Prevention Research and Programs in Canada and the United States: Setting a collaborative agenda, Albuquerque, New Mexico, February 7–9.

Parker, C. P., Baltes, B. B., Young, S. A., Huff, J. W., Altmann, R. A., Lacost, H. A., & Roberts, (2003). Relationships between psychological climate perceptions and work outcomes: A meta-analytic review. Journal of Organizational Behavior, 24(4), 389–416.

Patton, M. Q. (2002). Qualitative research & evaluation methods (3rd ed.). Thousand Oaks, CA: Sage Publications. Plano Clark, V. L., & Creswell, J. W. (2008). The mixed methods reader. Thousand Oaks, CA: Sage Publications. Rallis, S. F., & Rossman, G. B. (2003). Mixed methods in evaluation contexts: A pragmatic framework. In A. Tashakkori

& C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral sciences. Thousand Oaks, CA: Sage Publications.

Seidman, I. (1998). Interviewing as qualitative research; A guide for researchers in education and the social sciences (2nd

ed.). New York, NY: Teachers Press.

DESIGNING EVALUATIONS IN CHILD WELFARE ORGANIZATIONS 13

Smith, B. D., & Donovan, S. E. F. (2003). Child welfare practice in organizational and institutional context. Social Service Review, 77(4), 541–563.

Smith, B. D. (2005). Job retention in child welfare: Effects of perceived organizational support, supervisor support and intrinsic job value. Children and Youth Services Review, 27(2), 153–169.

Spector, P. E. (1985). Measurement of human service staff satisfaction: Development of the job satisfaction survey. American Journal of Community Psychology, 13, 693–713.

Stagner, M. (2009). Creating and sustaining evaluation partnerships to shape child welfare policy and practice. Presented at the 2009 National Child Welfare Evaluation Summit, Washington, DC, May 26–28.

Strolin-Goltzman, J. S., Auerbach, C., McGowan, B., & McCarthy, M. (2008). The relationship between organizational characteristics and workforce turnover among rural, urban, and suburban public child welfare systems. Administration in Social Work, 32(1), 77–91.

Strolin-Goltzman, J. S., Lawrence, C. K., Auerbach, C., Caringi, J., Clarborne, N., McCarthy, M., & McGowan, B. (2009). Design teams — A promising approach?: The effect of an organizational intervention on improving organizational factors and turnover rates in public child welfare agencies. Child Welfare, 88(5), 149–168.

Strolin-Goltzman, J. S., McCarthy, M., Smith, B., Lawson, H., Bronstein, L., & Caringi, J. (2008). A comparison study of intention to leave among public child welfare systems with high and low turnover rates. Child Welfare, 87(4), 125–143.

U.S. General Accounting Office (2003). Child welfare: HHS could play a greater role in helping child welfare agencies to recruit and retain staff . Washington, DC: US General Accounting Office.

Vroom, V. H., & Jago, A.G. (2007). The role of the situation in leadership. American Psychologist, 62(1), 17–24.