Please read all attachments

439

Administration in Social Work, 33:439–449, 2009 Copyright © Taylor & Francis Group, LLC ISSN: 0364-3107 print/1544-4376 online DOI: 10.1080/03643100903173040

WASW0364-31071544-4376Administration in Social Work, Vol. 33, No. 4, Jul 2009: pp. 0–0Administration in Social Work

Standardizing Practice at a Victim Services Organization: A Case Analysis

Illustrating the Role of Evaluation

Standardizing Practice at a Victim Services OrganizationM. Larsen et al.

MANDI LARSEN Institut für Rechtsmedizin, Universitätsklinikum Hamburg-Eppendorf, Germany

COREY TAX and SHELLY BOTUCK Safe Horizon, New York, New York, USA

This paper provides an example of how an internal evaluation department at a midsize victim services organization led key activities in achieving strategic organizational goals around unifying service delivery and standardizing practice. Using the methods of logic model development and naturalistic observation of services, evaluation staff guided the clarification of program expertise and outcomes, and assessed the necessary resources for standardizing practice.

KEYWORDS program evaluation, standardized practice, victim services, logic models, observation

There is little question that there is a growing demand for program evalua- tion data at nonprofit organizations, stemming from government, founda- tions, and other funding sources that want to know the impact of the programs they are supporting and that require demonstrations of effective- ness (Botcheva, White, & Huffman, 2002; Carman, 2007; Newcomer, Hatry, & Wholey, 2004). This focus on accountability to funders is also an opportunity for organizations to learn what services work best through evidence collec- tion for outcome measurement (Botcheva et al., 2002; Buckmaster, 1999). An organization’s ability to use this evidence and make strategic manage- ment decisions that are evidenced-based or informed is essential in an

Address correspondence to Shelly Botuck, Safe Horizon, 2 Lafayette Street, 3rd Floor, New York, NY 10007, USA. E-mail: sbotuck@safehorizon.org

440 M. Larsen et al.

increasingly competitive environment for funding (Menefee, 1997; Neuman, 2003; Proehl, 2001).

Despite the increased focus on evaluation from funders, limited resources make it difficult for nonprofit organizations to carry out evalua- tion (Hoefer, 2000). A study by Carman (2007) found that very few organi- zations have the discretionary funds necessary to employ internal evaluation staff members. This is in part because demands for evaluation can often seem like detractions from the service provision, especially when funding for services is limited (Kopczynski & Pritchard, 2004; Neuman, 2003). Poorly developed information systems and high staff turnover at many social service organizations also present barriers to implementing evaluation that demonstrates program improvement over time, both in terms of data collec- tion and institutional memory (Kopczynski & Pritchard, 2004). As a result, organizations focus on counting products or services provided through the activities of the organization (e.g., number of counseling sessions, number of trainings conducted) in an attempt to meet funder demands (Carman, 2007). This emphasis on outputs can shift the focus from achieving the mission of the organization to counting services, and may take attention away from case documentation that could be used to monitor practice and assess client outcomes. Despite all of these barriers, evaluation is still the key to understanding the effects of programs and services. Thus, the challenge lays in making evaluation useful to organizations, because without an appreciation of its value and worth, program evaluation will not be efficacious (Chelimsky, 1994).

Using Safe Horizon’s community and criminal justice programs (CCJP) as an example,1 this paper provides a case analysis illustrating the role of evaluation in furthering the implementation of our organization’s strategic plan. It focuses on two key activities, logic modeling and assessing program practice, and highlights the ways that these activities assisted Safe Horizon in standardizing service delivery.

ORGANIZATIONAL CONTEXT

Founded in 1978, Safe Horizon’s mission is to provide support, prevent violence, and promote justice for victims of crime and abuse, their families, and communities. Safe Horizon is New York City’s leading victim assistance organization delivering services to victims of domestic violence, sexual assault, child abuse, stalking, human trafficking, and other crimes through programs in the family and criminal courts, police precincts, child advocacy centers, schools, and other locations. Safe Horizon also operates domestic violence shelters; New York City’s 24-hour domestic violence, rape, and sexual assault hotlines; drop-in centers and emergency shelters for home- less and street-involved youth; case management services; and specialized

Standardizing Practice at a Victim Services Organization 441

mental health programs. Victims’ program involvement may last minutes or years. Safe Horizon’s primary service obligation is to provide victims of crime and abuse with the resources and tools needed to maximize their personal safety and reduce their risk of further harm, whatever the presenting victimization or service setting.

Safe Horizon’s leadership has long recognized the importance of inter- nal research and evaluation. As Whyte (1989) noted, when knowledgeable stakeholders conduct research, they can report on practices without the distortion caused by the presence of an outside observer. However, external funds obtained to answer macro social science and criminal justice ques- tions dictated most of Safe Horizon’s research and evaluation activities. As a result, these activities rarely informed day-to-day direct practice or service delivery. Additionally, it was difficult to agree on measurable outcomes for victims of violence. This was due in part to the context of traditional victim services programs, which are often designed to prevent a negative event from occurring (reabuse), and where the approach often holds that “the survivor is not responsible for preventing, and is indeed often unable to prevent, this negative event from occurring regardless of her actions” (Sullivan & Alexy, 2001, p. 1).

Thus, as a first step in establishing evaluation that would directly inform practice, while acknowledging the challenges of establishing outcomes, program evaluation focused on victim satisfaction surveys. This was helpful in improving practice and began collaboration between evalua- tion and program staff. This also built a foundation for trust and understand- ing that would become important in engaging with programs in thinking about outcomes beyond victim satisfaction.

Over the course of three decades, Safe Horizon grew into a midsize organization with the capacity to serve a wide range of victims in disperse settings, and each program determined its own method for assessing victim needs. As a result, the organization’s service delivery and documentation practices were decentralized and varied. In 2003, this was addressed in the organization’s strategic plan with the goal of unifying service delivery to ensure coordinated and high-quality services.

Standardized practice is the creation of uniformity in the definitions, training, staff role, and procedures for common practices within a discipline or organization, which is “intended to promote the effectiveness of practice, reduce variability in implementing best practice, [and] increase the predict- ability of practice behaviors” (Rosen & Proctor, 2003, p. 1). Using our safety assessment and risk management policy (Safe Horizon, 2007) to standardize practice, Safe Horizon began its first steps towards unifying service delivery and creating a continuum of care across programs. This policy places a “victim’s needs, wishes, resources, and capacities at the center of client work,” and thereby sets a “standard for a dynamic and collaborative process to address the complex challenges that victims of crime or abuse face” (Tax,

442 M. Larsen et al.

Vigeant, & Botuck, 2008, pg. 6). The policy provides a framework that acknowledges change in a victim’s risk over time, while also integrating both the staff’s knowledge and the victim’s perspective into the safety plan- ning process. Its implementation requires attention around standardization because the policy emphasizes “a standard of care that will be upheld across the organization,” while still recognizing that “specific aspects of implementation will depend on the program and the services offered by that program” (Safe Horizon, 2007, p. 2).

To prepare to implement the policy in a way that would unify service delivery across programs, evaluation staff engaged CCJP in a number of key activities, the first of which was the development of logic models. These were intended as blueprints defining the expertise, activities, and goals of each program, clarifying how programs work together, and setting up a framework for monitoring program practice. To identify the necessary resources for implementing standardized practice, evaluation staff assessed program practice to determine to what extent the skills and practices outlined in the policy were already taking place.

LOGIC MODEL DEVELOPMENT

McLaughlin and Jordan (2004) have described logic models as “the basis for a convincing story of the program’s expected performance, telling stake- holders and others the problem the program focuses on and how it is uniquely identified to address it” (p. 8) through a visual representation of a program’s resources, activities, outputs, and a range of outcomes. “A logic model provides a blueprint that delineates all the elements of the program that need to be documented in order to fully understand the program” (Conrad, Randolph, Kirby, & Bebout, 1999, p. 20) and represents how a specific set of resources and activities will bring about intended outcomes. Logic models are useful tools in pinpointing inconsistencies or redundan- cies, as well as determining whether activities are still relevant to program goals. Conrad et al. (1999) also noted the usefulness of logic models for bringing the perspectives of various program stakeholders to consensus, which can serve to establish clear and measurable expectations for a program and a common understanding of staff roles and function across an organization (McLaughlin & Jordan, 2004).

In order to integrate the service delivery model into the organizational culture and everyday decision making, the logic model development process aimed to ensure buy-in at all levels. Evaluation staff (namely, the authors) met with all levels of program management. Prior to these meetings with CCJP leaders and site supervisors, evaluation staff reviewed current funding reports and objectives and investigated reporting and documentation mechanisms in order to gain an initial understanding of

Standardizing Practice at a Victim Services Organization 443

resources and program activities. This served as preparation for building an overall logic model with CCJP leaders to define the vision for this cluster of programs.

Initial meetings with CCJP leaders included discussions about resources (e.g., funding, staff expertise, external partners, documentation systems) and activities, but primarily centered upon the expectations and vision for this cluster of programs. This focus on vision was only possible given the mutual trust and understanding previously built between evaluation and program staff. Due to previous negative experiences in tying program success to the actions of outside systems or the actions of the offender (e.g., receiving an order of protection, successful prosecution of the offender, placement in a domestic violence shelter, desistance of violent behavior), CCJP leaders voiced a general reluctance to define victim outcomes. Given this reluctance and the challenges inherent in establishing and operational- izing outcomes at social service organizations (Neuman, 2003), extra time was devoted to discussing meaningful program outcomes that accurately assess whether the program is having its intended effect. Over the course of these discussions, consensus on appropriate outcomes was achieved through continual grounding in the policy, which was centered upon the organization’s guiding principles and the commitment to “support and promote our client’s self-determination, dignity, and empowerment in a compassionate, non-judgmental environment” (Tax et al., 2008, p. 14). With this grounding, evaluation staff and CCJP leaders developed victim outcomes that were not dependent upon outside actors, but measured pro- gram success through individual victim change. These outcomes, along with quality assurance of standardized practices, have the potential to inform program practice by their measurement.

After developing a draft based on these discussions, evaluation staff led CCJP leaders through the refinement and vetting of the CCJP logic model during a daylong off-site work retreat. In this focused setting, the group walked step-by-step through the logic model, critiquing and offering suggestions for revision. The end result was an overall logic model of CCJP resources and services with victim outcomes that CCJP leaders expected would result from a victim’s involvement with the CCJP cluster.

The next task in creating a blueprint for unified service delivery was the development of a logic model for each of the four main programs in this cluster to clarify each program’s expertise and to define the roles the pro- grams should play in a unified service delivery model. Separate discussions were held with site supervisors from each of the programs (these programs have five sites, one in each borough of New York City), walking through the overall CCJP logic model and breaking down the aspects specific to their program. Document review and discussions with site supervisors revealed that programs performed similar activities, but that these activities were conducted slightly differently in each program. For example, information

444 M. Larsen et al.

provided in a police program focused on police processes, while informa- tion provided in a criminal court program focused on court processes. The expected outcome of information provision (that the victim will be able to strategize and make informed decisions about their situation) remained the same across the CCJP, but the differences in program expertise were clear. Evaluation staff incorporated these commonalities and differences into the initial logic model drafts for each of the programs, totaling four logic models altogether.

Evaluation staff presented these drafts back to CCJP leaders for vetting, walking step-by-step through each program logic model and considering the following questions: Were there gaps in services that needed to be addressed in order to better fit the vision of providing a continuum of care to victims through integrated expert programs? Which services currently offered should change? For example, while all site supervisors indicated performing some type of community outreach, CCJP leaders did not feel that this activity was within the appropriate scope of activities for the court programs. They agreed that community outreach seemed beyond the goals of court programs, whose aim is to serve those already involved in the court systems and who do not receive any funding for outreach activities. CCJP leaders felt that community outreach should be concentrated in the police and community-based programs, which play an important role in ensuring community members are aware of the services offered at Safe Horizon.

Another round of revisions resulted in four individual program logic models that represented the vision for service delivery for the CCJP cluster. To continue fostering a sense of ownership of these logic models, CCJP leadership presented the models to site supervisors. This allowed for the gathering of feedback and also allowed them to be on hand to answer questions about strategic decisions made about standardizing services. By the end of the logic-model development process, clear and measurable expectations for programs were established, as was a common understanding of staff roles and function across the CCJP.

ASSESSING PROGRAM PRACTICE

To assess the extent to which the skills outlined in the new policy were already occurring in day-to-day program practice, observation of service delivery and its documentation across Safe Horizon’s point-of-entry (gateway) programs was conducted over a two-month period. In the absence of documentation that would clearly reflect current practice, prudent use of naturalistic observation—where behavior is observed in its natural environment and is recorded in a manner that is as unobtrusive as possible (Angrosino, 2007)—can provide a representative sample of service delivery.

Standardizing Practice at a Victim Services Organization 445

Based on designated performance indicators of the new risk and safety policy, an observation tool (see Appendix for a detailed explanation) was developed by evaluation staff that would collect information that could: (a) describe current practice, (b) identify differences in practice across programs, (c) examine how practice(s) apply to different types of victim interactions, and (d) inform decision making about future staff training.

Over a three-week period, four observers, which included evaluation staff and interns, were trained by one of the authors to assess client interactions according to a standard and to match to his observations for all sections of the tool. This necessarily included a common understanding and definition of service delivery (e.g., referral, linkage, supportive counseling, crisis interven- tion, etc.), as well as how to be unobtrusive during observations and how to keep appropriate boundaries with victims and staff. A 90% level of inter-rater agreement was established between each of the four observers and the trainer.

All of the observations were scheduled in advance. Every effort was made to ensure that the service delivery was representative of typical sessions and workloads and did not underestimate the frequency or inten- sity of service delivery. Observers refrained from inferring anything about service delivery and gathered information from only directly observed staff comments, actions, or responses to a victim. Observations were always conducted by one observer at a time. Upon arrival at the site, the site super- visor would introduce the observer to the staff and explain what he or she would be doing. To gain the consent of the clients before observing a case management interaction or counseling session, the case manager would introduce the observer to the client and explain the process, emphasizing that the observer was there to observe service provision only.

The data from the observations were entered into an SPSS database, and frequencies were calculated. Twenty program sites were observed for approximately 208 hours, totaling 213 victim interactions (162 telephone and 51 face-to-face interactions).

The analysis of these observations revealed that expected practices were not occurring at the frequency anticipated. Victim safety concerns were documented, and observers noted that the need for assistance was complex and ongoing (Vigeant, Tax, Larsen, & Botuck, 2008). Additionally, even within the same program at different delivery sites, service provision often had wide variability in both practice and documentation. As a result, clients with identical presenting needs might be offered different services depending on which program site they happened to walk into.

DISCUSSION

Evaluation activities, which included the development of logic models and the assessment of current practice, identified gaps between the organization’s

446 M. Larsen et al.

vision for unified service delivery as identified in its strategic goals and current practice. This pointed to additional resource needs that were not anticipated in the original planning. The findings also revealed practice real- ities that included considerable variability in service delivery and documen- tation, lower than expected frequency of specific activities, and complex client need. These findings pointed to a need for changes in existing imple- mentation plans.

The development of logic models served to bring staff with a range of education, experience, and expertise closer to consensus around program practices, services offered, and victim outcomes. Walking through the models with CCJP leaders and site supervisors necessarily focused the dis- cussion around variation in program practice. It was not unusual for there to be a variety of perspectives on program functioning, a lack of shared information across sites, and a variety of documentation systems. This process brought to the forefront the current resources, expertise, and abilities of individual programs. The substantial variation in practice revealed the need for in-depth clarification of roles within each program before implementing standardized practice, a step that had not been adequately accounted for in existing plans.

The observation of services confirmed variations in practice alluded to during logic model discussions. While program observation required a significant amount of time and resources, the time (and cost) provided information that clearly indicated the need for significant ongoing training and support in critical skill areas required to implement the risk manage- ment policy according to the intended standard (e.g., addressing barriers, allowing clients to determine risk factors, building risk management on client’s current protective actions, tailoring plans to fit clients).

Providing this level of support to staff will require increased partner- ship among different departments within the organization and will include: the creation of appropriate training curricula and materials; the develop- ment of staff trainers; the establishment of accountability monitoring mecha- nisms (quality assurance indicators); and the creation of a closer exchange between programs and information technology so that electronic documen- tation systems reflect (or enhance) staff workflow.

Another implication of the variability in practice is in thinking about the role of supervisors. The findings reinforce the importance of providing ongoing training and supports for supervisors so that they are able to effec- tively carry out their roles. Wilkins (2003) has described the deleterious effects on line staff when frontline supervisors are unable to provide guid- ance and assistance in problem solving. This is particularly true in organizations where there are limited resources for staff development and increased needs for services. Moreover, the competence and value of frontline super- visors are vital to achieving the goals of service organizations (Burchard, Gordon, & Pine, 1990; Haas & Robinson, 1998) and in preventing direct

Standardizing Practice at a Victim Services Organization 447

service staff from burnout and vicarious trauma (Baird & Jenkins, 2003; Pearlman & Saakvitne, 1995).

In conclusion, evaluation activities helped determine the resources and first steps in implementing strategic organizational goals around unifying service delivery and implementing standardized practice. Logic model development fostered discussion of program expertise and promoted a shared understanding of standardized practices. These activities, which served to focus service delivery, were successful in furthering the imple- mentation of our organization’s strategic plan. Naturalistic observations served to create a picture of service delivery and helped identify the resources necessary to standardizing practices. Although internal evaluators could present a potential bias, we believe that using internal evaluators familiar with the programs and staff actually facilitated the evaluation activi- ties. The foundation of mutual trust and understanding between evaluation and program are key to overcoming barriers and establishing meaningful outcomes.

NOTE

1. CCJP consists of four distinct programs, each one located in each of the five boroughs of New York City. Each program within CCJP partners with, is regulated by, and may share space with criminal justice and community systems and structures that vary considerably from borough to borough.

REFERENCES

Angrosino, M.V. (2007). Naturalistic observation. Walnut Creek, CA: Left Coast Press.

Baird, S., & Jenkins, S. R. (2003). Vicarious traumatization, secondary traumatic stress and burnout in sexual assault and domestic violence agency staff. Violence and Victims, 18(1), 71–86.

Botcheva, L., White, C. R., & Huffman, L. C. (2002). Learning culture and outcomes measurement practices in community agencies. American Journal of Evaluation, 23(4), 421–434.

Buckmaster, N. (1999). Associations between outcome measurement, accountability, and learning for non-profit organisations. International Journal of Public Sector Management, 12(2), 186–197.

Burchard, S. N., Gordon, L. R., & Pine, J. (1990). Manager competence, program normalization and client satisfaction in group homes. Education and Training in Mental Retardation, 25(3), 277–285.

Carman, J. G. (2007). Evaluation practice among community-based organizations: Research into the reality. American Journal of Evaluation, 28(1), 60–75.

Chelimsky, E. (1994). Making evaluation units effective. In J. Wholey, H. Hatry, & K. Newcomer (Eds.), Handbook of practical program evaluation (pp. 489–509). San Francisco: Jossey-Bass.

448 M. Larsen et al.

Conrad, K. J., Randolph, F. L., Kirby, M. W. J., & Bebout, R. R. (1999). Creating and using logic models: Four perspectives. Alcoholism Treatment Quarterly, 17(1), 17–31.

Haas, P. J., & Robinson, M. G. (1998). The view of nonprofit executives on education nonprofit managers. Nonprofit Management and Leadership, 8(4), 349–362.

Hoefer, R. (2000). Accountability in action? Program evaluation in nonprofit human service agencies. Nonprofit Management and Leadership, 11(2), 167–177.

Kopczynski, M. E., & Pritchard, K. (2004). The use of evaluation by nonprofit organizations. In J. S. Wholey, H. P. Hatry, & K. E. Newcomer (Eds.), Hand- book of practical program evaluation (2nd ed., pp. 649–669). San Francisco: Jossey-Bass.

McLaughlin, J. A., & Jordan, G. B. (2004). Using logic models. In J. S. Wholey, H. P. Hatry, & K. E. Newcomer (Eds.), Handbook of practical program evaluation (2nd ed., pp. 7–32). San Francisco: Jossey-Bass.

Menefee, D. (1997). Strategic administration of nonprofit human service organiza- tions: A model for executive success in turbulent times. Administration in Social Work, 21(2), 1–19.

Neuman, K. M. (2003). Developing a comprehensive outcomes management program: A ten step process. Administration in Social Work, 27(1), 5–23.

Newcomer, K. E., Hatry, H. P., & Wholey, J. S. (2004). Meeting the need for practical evaluation approaches: An introduction. In J. S. Wholey, H. P. Hatry, & K. E. Newcomer (Eds.), Handbook of practical program evaluation (2nd ed., pp. xxxiii–xliv). San Francisco: Jossey-Bass.

Pearlman, L. A., & Saakvitne, K. W. (1995). Treating therapists with vicarious traumatization and secondary traumatic stress disorders. In C. R. Figley (Ed.), Compassion fatigue: Coping with secondary traumatic stress disorder in those who treat the traumatized (pp. 150–177). Bristol, PA: Brunner/Mazel.

Proehl, R. A. (2001). Why is change necessary? Organizational change in human services (pp. 1–10). Thousand Oaks, CA: Sage Publications.

Rosen, A., & Proctor, E. K. (2003). Practice guidelines and the challenge of effective practice. In A. Rosen & E. K. Proctor (Eds.), Developing practice guidelines for social work intervention: Issues, methods, and research agenda (pp. 1–16). New York: Columbia University Press.

Safe Horizon. (2007). Policy on client safety assessment and risk management. New York: Author.

Sullivan, C., & Alexy, C. (2001). Evaluating the outcomes of domestic violence service programs: Some practical considerations and strategies. Retrieved on August 6, 2008 from VAWnet, http://new.vawnet.org/Assoc_Files_VAWnet/ AR_evaldv.pdf.

Tax, C., Vigeant, M., & Botuck, S. (2008). Integrating safety assessment with risk management: A dynamic victim centered approach [White paper]. Retrieved on July 28, 2008 from Safe Horizon http://www.safehorizon.org.

Vigeant, M., Tax, C., Larsen, M., & Botuck, S. (2008, February). Rethinking risk and safety with victims of crime and abuse. Poster session presented at the American Psychological Association’s summit on violence and abuse in relationships: Connecting agendas and forging new directions, Bethesda, MD.

Standardizing Practice at a Victim Services Organization 449

Whyte, W. F. (1989). Advancing scientific knowledge through participatory action research. Sociological Forum, 4(3), 367–385.

Wilkins, E. (2003). Building a committed and effective workforce through strengthen- ing skills of frontline managers. Retrieved on July 31, 2008 from Aon Consulting Worldwide. http://www.ecustomerserviceworld.com

APPENDIX

Observation Tool Assessment Areas

Greeting: Assessed whether staff greeted clients warmly and introduced themselves as employees of Safe Horizon. (1 item)

Identifying Information: Captured what identifying information (e.g., name, date of birth, social security number, address, phone number) was obtained and whether the information was confirmed in accordance with current training. (12 items)

Interaction Measures: Captured specific information about the interac- tion that might impact the service delivery, including the location of the interaction; people present during the interaction; whether the interaction was a scheduled appointment; whether it took place on the phone or was face-to-face; the time the interaction began and ended; and the number of staff-initiated interruptions. Another set of interaction measures included the observer’s assessment as to the level of assistance the observer perceived the client to be seeking, and whether the client was in distress. Notation regarding whether the staff person behaved in a professional manner was also included in this section. (10 items)

Victimization Assessment: Captured whether staff assessed client victimization and, if so, to what extent. This included whether staff assessed for victimization type, whether the client was a primary or secondary victim, information about the offender, recent incident details, duration, scope, system involvement, and prior victimizations. (2 items)

Safety Assessment: Examined whether staff performed a safety assess- ment by capturing initial safety assessment, as well as assessments of current safety concerns, client protective actions, client resources and stressors, and client coping skills. (9 items)

General Skills: Explored whether staff utilized specific skills during client interactions, such as crisis intervention skills, general assessment skills, and engagement skills. (20 items)

Services: Examined which services staff provided to the client, and whether services were provided via exploration, information, referral, advocacy, and/or linkage. (46 items)