Colleague Response

Page 4

Let’s begin by considering three important questions: 1. Is evaluation an important area of social work? 2. Is the evaluator role an important one for social workers? 3. How can evaluations help improve or enhance social work interventions? These questions may be your questions as you begin to read this book. They are questions that many social work students and practitioners have pondered. This book is about evaluation so the responses to the first two questions, in brief, will be no surprise to you. Yes, evaluation is an important area of social work. Further, the evaluator role is an important role for every social worker to prepare to assume. Some social workers will be evaluators of programs, and virtually every social worker will be an evaluator of their own practice. It’s like asking whether social workers need to know whether they are doing a good job, or asking them if they know whether their interventions are effective in helping their clients. The third question, asking how evaluation can help improve social work interventions, is the focus of this text.

The underlying theme driving the book is that evaluation is a vital element of any social work approach and is critical for ensuring that social work actually does work! A reassuring theme is that evaluation is a practice area that BSW and MSW students and practitioners alike can learn. Social workers and students wanting to maximize their impact in their jobs will find that the perspective, knowledge, ethics, and skills of evaluations covered in this book are a central component of practice and ensure that you will have a much greater impact on your clients’ well-being. This book provides the needed preparation for evaluation in both a comprehensive and a readable format. The primary emphasis is on the various kinds of small and mid-range formative evaluations that are often implemented at the local agency level; less emphasis is placed on the large, com-plex national and regional studies that may draw the most coverage under the title evaluation. These smaller formative evaluations are also the critical ones that social work students and graduates either are assigned or should consider taking on in their field placements and employment agencies. Such

Page 5

evaluations often are instrumental in determining whether the programs in which you are working will continue and possibly expand. Example of a Small, Formative Evaluation An agency that provides an anger management program to perpetrators of domestic violence offers a series of ten psychoeducational group sessions to help them manage their anger. The agency also conducts an evaluation of this program that is integral to it. An anger management scale is used to measure changes that occur in the participants’ anger after they have completed all ten sessions of a group program. Throughout the series, the specific items of the anger management scale (e.g., being respectful, having self-control, being self-aware, learning alternatives to violent behavior) identify some of the key discussion topics of the group sessions. In this way, the intervention and its evaluation go hand in hand in helping practitioners and client’s partner to meet the goals of the program.

Evaluation is a multifaceted approach that addresses some of the most vital questions and issues facing programs and practice, including the following: • Who are the clients that need the intervention most? • What kinds of intervention do they need? • How are the interventions to be implemented? • What will the intervention achieve? In other words, evaluations can address important issues at all stages of development of a program or practice area: • Planning, when a vision is created for an intervention and developed into a proposal or plan • Implementation, when the plan is actually carried out • Outcome observation, when the clients’ outcomes or affected behaviors are measured during a post-program period Evaluation is important to consider for both the practice provided by each practitioner and the program itself. At the practice level, an evaluation can investigate inputs such as who will be the client population; the processes the practitioner uses to implement the intervention; and the out-comes or client accomplishments resulting from the practitioner’s interventions. At the program level, an evaluation can investigate the same stages (inputs, processes, and outcomes) of interventions typically provided by several practitioners, for example, the case management services of a men-tal health program.

Page 6

PRACTICE IS EMBEDDED IN A PROGRAM

Programs and individual practice are closely intertwined. Practice is, at times, a microcosm of a program. For example, a small program can consist of a set of similar interventions of a few practitioners (e.g., a family counseling program). In addition, practice is almost always embedded in a pro-gram, and the program is instrumental in informing, shaping, and directing the definition of practice. Practice is not an entity to be developed in a vacuum or at the whim of a practitioner or supervisor, even though this can inadvertently happen in reality. Social work practitioners, once they graduate, usually have a good beginning sense of an overall practice approach to helping clients. However, it is an approach that is still mostly understood in the abstract, based on practice theories being taught in a professional program along with some opportunities for testing the theories in field practicum experiences.

However, when practitioners begin work at an agency, there are numerous new variables that come into play. The client population is one of them. Who are the clients? What kinds of problems and needs do they have? What kinds of interventions do they need the most? In addition, the agency typically has a practice approach (or approaches) that it uses to help clients, and the agency expects its practitioners to know this approach and become competent in using it. The approach may be explicit and detailed or quite vague and mostly abstract. Practitioners new to an agency need to ask how they can fully understand and implement this practice approach, hopefully with assistance from a supervisor. Later on, new practitioners also face growing challenges as they become aware of the need for subtle variations in the application of the agency approach based on the varied and sometimes unique needs of clients. Because this book is about both programs and practice, and their relationship, it offers numerous insights into the vital link between professional practice and its program context. In this light, this book offers several objectives for practitioners to consider:

• Realize that your practice is embedded in a program context

. • Recognize how your program works and informs your practice.

• Understand the bigger picture of the agency, its policies, organizational structures, external environment, and stakeholders, and how they affect your practice.

• Know what is important to evaluate in your practice, taking into account the program context.

• Realize that program evaluations usually have valuable implications for improving your practice.

• Consider using some of the strategies of program evaluations when evaluating your practice.

Page7

COMMON CHARACTERISTICS OF EVALUATIONS

To fully understand the nature of an evaluation, we need to understand its common characteristics. These characteristics include accountability, use of scientific research methods, the logic model as an analytic tool, stakeholders, political processes, an ethical code, and critical thinking. These characteristics continually interact with and influence one another. Being Accountable If there is one overall concept that explains why evaluations are so important, it is accountability. Partially because of past failings of social pro-grams, all governmental and most privately funded agencies are now held accountable for how they use their funds and what they achieve for their clients. Evaluations have become one of the most reliable mechanisms incorporated into program proposals for ensuring such accountability. Agency accountability is now inherent in the jurisdiction of virtually all fund-ing and regulatory agencies, and it has become a key job expectation of agency and program administrators. These funding, regulatory, and administrative roles require accountability to address questions such as the following

Is the intervention focusing on the target population with the greatest need?

• Is the intervention designed to meet the specified needs of the target population?

• Is the intervention being implemented in the way that it was designed and proposed?

• Is the intervention being implemented with high standards?

• How satisfied are clients and their families with the intervention?

• Is the intervention achieving its goals and objectives?

• Is the intervention cost-effective?

Ultimately, it is also important for program sponsors to be accountable to the clients they serve. Because of the power imbalance between an agency and clients, special attention is needed to bring more balance to these two entities in the form of greater power and protection for clients. In addition, agencies need to be accountable to the communities intrinsically connected to clients, such as their family members and the immediate neighbors surrounding residential programs. Accountability to clients and relevant communities often requires the introduction of empowerment strategies, such as client satisfaction surveys and client representation on agency boards. Another strategy is to encourage agencies to involve client groups as participants in program evaluations and to share the results of their evaluations with them. Chapter 2 further elaborates on other empowerment strategies.

Social workers who work in such programs must also be accountable not only to the agency employing them but also to their own professional groups, such as the National Association of Social Workers (NASW), Political Action for Candidate Election (PACE, a political arm of NASW), and their state professional licensing boards. In these instances, accountability refers to abiding by an ethical conduct for social workers, commitments to clients’ dignity and well-being, advocating for social justice, and implementation of sound and evidence-based professional practice.

Using Scientific Research Methods Evaluation activities are scientific and use a wide range of research methodologies (e.g., Dudley, 2011). Scientific research has long-standing values and principles that distinguish it from other types of information gathering. Many of these principles are evident in evaluations, including:

• The search for something that exists rather than something that is desired

• Use of a methodology that minimizes the influence of biases and involves a systematic set of steps or procedures that can be flexibly employed

• Abiding by a special code of ethical conduct that includes a commitment to neutrality in conducting research and a demonstration of concern to protect the people studied

• Assumption of the evaluation having a universal stance, representing the concerns of all society, even though it may focus on a few sub-groups of people or a narrow topic

• Accurate reporting of findings despite whether they are consistent with the researcher’s viewpoint

Although these principles of scientific research are ideals that should be fulfilled to the greatest extent possible in all scientific studies, in reality they are evident in evaluation studies to varying degrees and along a continuum of quality. The more an evaluation rigorously fulfills these ideals, the more confident one can be that it is considered “good” science. The Logic Model as an Analytic Tool The logic model is both an organizing framework and a tool that evaluators use for analyzing a program. The logic model can also be used to analyze a practice approach. The logic model helps highlight how the stages and elements of a program are logically linked. Using this model, it is important to examine the sequence of stages in a program’s development, beginning with identifying the problems of prospective clients, followed by developing and implementing an intervention to address these problems, and

Page 8

culminating in anticipated client outcomes that reflect a resolution of these client problems. These stages can be visually understood in figure 1.1. Figure 1.1 is an important framework that helps begin to organize much of the content of the book and thus will be presented periodically in appropriate chapters to highlight the stages and program elements as they are being covered. As mentioned earlier in the chapter, the logic model comprises three stages: planning, implementing, and observing outcomes. During the planning stage, the problems and needs of the client population are defined and documented and an intervention is conceived and developed to address these problems and needs. During the implementation stage, this intervention is implemented. Finally, during the outcome stage, the out-comes for clients who receive the intervention are observed and measured. The logic model helps the evaluator link the documented problems and needs to the intervention that will address them, and the intervention is, in turn, linked to the client outcomes that are anticipated after the intervention has been implemented. The logic model is elaborated on further in later chapters, particularly chapter 6 and 8.

Example of the Use of the Logic Model in Designing a Program A group of students were asked to design a program that would effectively help clients overcome substance abuse problems. They used the logic model to complete this exercise. They began by identifying some suspected causes of substance abuse, including heredity, peer influence, low self-esteem, social isolation, and inadequate coping skills. Next they decided to design a program to address only the suspected causes that revolved around interpersonal issues, including peer influence, social isolation, low self-esteem, and inadequate coping skills. They decided to offer psychoeducational groups to teach clients the skills needed to manage these and other personal and interpersonal issues. They decided to cover specific topics such as how to find positive peer influences and avoid negative peer influences, how to find and participate in support groups, and some self-esteem-building exercises. They anticipated that once participants had completed the psychoeducational group session

Page 9

they would be able to identify the factors that reduced their self-esteem, to identify specific ways to build more positive self-esteem, and to learn three or four new coping skills. By completion of the program, participants would also have made two or more visits to a support group in the community to help them stop using sub-stances.

Involving Stakeholders and a Political Process

While basic research is often considered apolitical, evaluations involve an overtly political process. Historical events and current political considerations need to be considered when discussing, planning, and implementing an evaluation. Indeed, an evaluation is a special type of research that incorporates political considerations into its execution. An evaluation may have several different stakeholders, and each could have special interests that compete with one another. When talking about an evaluation, political issues almost always come into play, whether explicitly or implicitly. For example, political processes might be involved in any of the following types of questions that agency administrators raise: • How can we help those with the greatest need? • How can an evaluation help our program survive? • How can an evaluation improve the chances of obtaining funding to expand our program? • How can the results of an evaluation be used to enhance our pro-gram’s identity in the larger network of agencies in our field? • How can we report negative findings from an evaluation without jeopardizing our program’s existence?

Example of a Political Consideration

A graduate student conducted an evaluation of staff morale at her field agency. She gave the staff members a questionnaire to fill out, asking them how important they perceived each of several different issues that affected their morale. The issues included salaries, medical benefits, size of caseloads, hours of work, supervision (its quality and frequency), and openness of administration to staff concerns. The findings revealed that their major concerns about morale related to their problems with supervisors and administration. Because the administration of the agency was taken by surprise and unprepared to seriously address these issues, they decided to ignore them and instructed the graduate student to withhold her findings from the staff members.

Page 10

Stakeholders of an evaluation have different interests. Stakeholders often include agency administrators, governmental funding and regulatory agencies, foundations, elected officials, board members, staff members, citizens, clients, advocacy groups, accountants and auditors, and the sur-rounding community. A funding agency, for example, might ask the following questions: • How can I be sure that this program is fulfilling its responsibilities?

• How can I determine whether this program is more or less important than another program that we fund?

• How can I get the most fiscal value out of this program?

• I like this program and its director, but how can I justify funding them when the proposal falls short of what we want?

Political considerations such as these must be taken into account at all stages of evaluating interventions, including planning, implementation, and outcome. An approach to identify and analyze these contextual forces within and outside an agency is elaborated on later in the chapter. In general, this approach can help evaluators consider the political issues and questions that they may need to address or avert in conducting an evaluation before they become serious problems or ethical dilemmas. The approach identifies a wide range of possible constraints that can create conflicts for an evaluation and considers a range of potential resources that can help in conducting the evaluation. The identification of potential constraints and resources before implementation of an evaluation helps address both its feasibility and its ethical standing.

Abiding by an Ethical Code Ethical issues are extremely important to identify when addressing political issues. Actually, the way in which decisions are made or not made should be partially based on an ethical code such as the NASW Code of Ethics (www.naswdc.org) or the ethical principles of the American Evaluation Association (AEA; www.eval.org). As social workers and other human service professionals know, those who participate in research and evaluation are obligated to follow an ethical code. The NASW Code of Ethics is a basic code required of all social workers. It obligates an evaluator to be well informed about ethical issues and well versed in how to implement a variety of measures intended to prevent ethical problems from occurring. The ethical principles of the AEA are designed for professional evaluators specifically conducting evaluation studies. These principles are valuable to consult because they are directed toward issues essential for a professional evaluator to address (the AEA ethical principles are located in appendix A). Ethical problems include such things as physical and psychological harm to research participants, invasion of privacy, and misrepresentation of study findings. Evaluators are obligated to prevent such ethical problems by

Page 11

implementing a variety of ethical safeguards, including an informed consent protocol, confidentiality, and selection of evaluators with appropriate credentials and objectivity. Chapter 3 focuses on an extensive introduction to many of the ethical concerns that are evident in evaluations and how to address these problems. It examines the NASW Code of Ethics and portions of the ethical principles of the AEA, particularly as they pertain to social workers’ ethical obligations related to evaluations.

Thinking Critically Another important characteristic of evaluations is critical thinking. Critical thinkers are natural skeptics about how well an evaluation is conducted, whether it is someone else’s evaluation or one’s own. Gibbs and Gambrill (1996) identify several types of problems that program providers experience when they fail to be critical thinkers:

• Overlooking the people who may need the services of a program the most

• Not understanding the larger social forces that influence the ways clients behave

• Misclassifying or misdiagnosing clients and their problems

• Focusing on irrelevant factors that are not important in helping clients make progress

• Selecting interventions that are weak or inappropriate

• Arranging for interventions to continue either too long or not long enough

• Being overly preoccupied with financial profit and neglecting the impact of such decisions on the clients’ well-being (particularly for-profit agencies)

The Council on Social Work Education (CSWE) views critical thinking as essential to the practice of every social worker. Because of the importance of critical thinking, CSWE mandates that it be one of the basic tenets of the professional foundation of every accredited social work education program at both the BSW and the MSW levels. One of the core program objectives required of the curricula of all programs and to be infused into every course is to “apply critical thinking skills within the context of professional social work.” Distinguishing Evaluations from Social Science Research As we begin to explore evaluations, a basic question emerges. How is an evaluation different from other types of research, particularly social science research conducted in academic settings? A closer look at the above characteristics provides a helpful way of distinguishing them. An evaluation can be distinguished based on its greater emphases on some of the characteristics

Page 12

of evaluations described above. Evaluations, unlike most social science research, give major emphases to four of the above characteristics. First, evaluations are conducted primarily to provide accountability to a funding agency, clients, and the community that an intervention works effectively and efficiently. Second, an evaluation places major emphasis on the logic model while most social science research does not. The logic model helps evaluators focus on the links among client problems, an intervention, and success in helping clients address these problems. Next, effective evaluations seek the involvement of all of the stakeholders while social science research may not. These stakeholders typically include groups with widely varying perspectives such as clients, regulatory agencies, and the agency providing the intervention. Finally, an evaluation is continually engaged in a political process that attempts to bring together these widely different stake-holders so that ideally all of their views are taken into account and all are participating together.

THE AIMS OF EVALUATION

Overall, evaluations of programs and practice are concerned with many purposes, including efficiency, quality, effectiveness, effort, and relevance. Martin and Kettner (1996), among others, stress the importance of efficiency, quality, and effectiveness. Efficiency is concerned with carefully channeling available resources to the target problems. Misdirected and wasted resources are to be avoided or minimized. Efficiency is important because resources for health and human service programs are always likely to be limited or even scarce, and the more efficiently that programs and practices are delivered, the more clients can be helped. Quality refers to services being delivered as intended and done very well or at a high standard. Sometimes this high standard has been referred to as best practices. Quality is obviously important because a program or practice delivered well will have the greatest positive impact on the recipients; the higher the quality, the greater is the likely impact. For example, a social worker who is not adequately prepared to provide intensive counseling to emancipated adolescents about to be released from foster care will fall short of helping them enough to function on their own.

Also, interventions have to be effective or lead to effectiveness. If interventions do not successfully work to bring about the anticipated changes in clients’ lives, the interventions should be reevaluated or the social worker will risk losing funding. All three of these areas—efficiency, quality, and effectiveness—are extremely important to evaluations. One cannot be preferred over the other two, as all three are required to ensure that a program or practice intervention works well. In addition to these three common qualities of programs and practice, two more are also crucial: evidence of effort and relevance to the clients and community. Evidence of effort is important regardless of the achievements

Page 13

of a program. Effort refers to what staff members, volunteers, and adminis-trators put into a program and practice. Effort especially refers to what hap-pens when the interventions are implemented. How much time was spent in the helping process? How many visits or sessions occurred? What hap-pened in these sessions? Most important, was the effort commensurate with the expectations of the funding agency, the program director, and clients? Relevance is another fundamental quality of any intervention. Relevance of the intervention to the clients’ needs and the larger social problems that clients experience is the most important aspect of relevance. A program intervention can be carried out in a high-quality way that uses resources effi-ciently and achieves program goals, but if it is not directly relevant to what the clients need, it is incomplete or misguided. The concept of relevance is related to the seeking of social justice for client populations and determin-ing that the diversity in the client population reflects the larger population that needs the interventions. Both of these efforts, social justice and diver-sity, as well as other important issues of relevance, are covered extensively in this text. Both the NASW Code of Ethics and the AEA Code highlight these issues and the responsibilities of evaluators to take into account the diversity of general and public interests and to be mindful of the rights and needs of all pertinent groups.

DEFINING IMPORTANT TERMS

Several important terms need to be defined before going further. They are relevant to answering numerous basic questions like, what is a program and how is it different from services? What distinguishes programs from the practice of individual workers? What are program evaluations and practice evaluations? How are they similar and different? Finally, what are evidence-based interventions? Let’s take a look at the basic terms: program, program and practice theory, services, professional practice, intervention, program evaluation, practice evaluation, and evidence-based interventions. A program is a subunit of a social agency that provides a set of goods and/or services with common goals. These goods and services are typically provided to a particular population of clients who either voluntarily seek them or are required to receive them. A program typically employs more than one and usually several staff members to provide goods and services.

Chen (1990) expands on this definition by developing the notion of program theory. Program theory is expected to encompass two important sets of documentation. First, it provides a descriptive documentation of the goals, outcomes, and interventions of the program based on the perspectives of various stakeholders. Second, program theory documents the nature of the causal relationship between the program interventions and the desired outcomes for the target group of recipients. It does this by offering research evidence that the proposed program model has been effective in

Page 14

helping, in the past, a group of clients with a particular set of characteristics. Practice theory can easily be described in a similar way. Goals, outcomes, and interventions are documented as well as the causal relationship between a practice intervention and the desired outcomes for a particular client. Services are the activities that programs or one practitioner offer. Services focus mostly on the processes that help clients reach their program goals. They are the means to the ends, not ends in themselves. These helping processes are the major focus of practice courses of professional social work programs and draw from a broad range of practice theories, such as generalist problem solving; locality development; and cognitive behavioral, person-centered, and solution-focused treatments. In addition, in-service training programs periodically provide helpful updates on such knowledge and helping skills.

Example of a Program and Services of a Home Health Agency A home health agency often sponsors one overall program, the goals of which are to help medically indigent clients remain in their own homes independently and prevent placement in a residential program such as an assisted living facility. Such a program offers several services to clients who are homebound, including counseling and referrals, nursing, physical therapy, and occupational therapy. These services, provided by a team of social workers, nurses, physical and occupational therapists, and others, exist to help the pro-gram meet its goals. Home health programs also offer goods, such as medical supplies and incontinence products.

Distinguishing between programs and services is important. For example, if you were to describe a program to someone unfamiliar with what you do, you would likely begin by referring to its goals and what it attempts to accomplish for clients. In contrast, if you begin by describing the services of the program, your explanation may appear incomplete and beg for an explanation of why these services exist or what they intend to accomplish. Note the difference between saying, “Our program is designed to help prevent teenagers from participating in unsafe sex,” and “Our services include individual counseling and psychoeducational groups for teenagers.” Professional practices are the interventions of a human service worker in helping a client system. Professional practice can be offered to one individual, a family, a small group of clients, an organization, an institution, a social policy area, or a larger community. It is important to distinguish between the professional practice of one individual and the services of a pro-gram, which encompass the practices of all staff members within a program.

Page 15

Because this book balances emphasis on programs and the practice of an individual practitioner, the term intervention is often used to refer either to the services and goods of programs or to an individual worker’s practice. As an example, interventions are evident both in the recovery programs of a substance abuse agency and in the clinical practice that one social worker offers to a client’s family. Using working definitions of these key concepts (programs, services, professional practice, and interventions), we can define program evaluations and practice evaluations. A program evaluation is a study of a social program that uses the principles and methods of scientific research. It concerns itself with the practical needs of an organization, not theoretical issues, and it abides by a professional ethical code. The primary purposes of a program evaluation are to provide accountability to its various stakeholders and to improve what the program can accomplish.

A practice evaluation is a study of a professional practitioner’s interventions with a client system, which can be at several different system levels (e.g., individual, group, neighborhood). Like a program evaluation, a practice evaluation uses the principles and methods of scientific research. Unlike a program evaluation, it focuses on only one practitioner’s practice at a time. Its primary purposes are to increase the effectiveness of a practitioner’s interventions and to determine whether the interventions success-fully help clients reach their goals. Although these definitions of program and practice evaluations are meant to be fairly distinct, at times they are blurred. For example, if we decided to evaluate an agency’s group services, would it be a program or practice evaluation? Two issues need to be considered before this question can be answered: (1) does the group service comprise the groups of only one worker or of several different workers? and (2) is the purpose of the evaluation to examine the effectiveness of an individual practitioner’s interventions or the group services of the entire program?

In our example, if only one staff member provided the group service, it could be both a program and a practice evaluation. Therefore, it is important to clarify at the outset what kind of evaluation is intended. For example, is it a practice evaluation intended to evaluate the interventions of individual staff members, or is it a program evaluation concerned with agency-wide issues, such as reaching the goals of the group service generally or deter-mining the use of limited resources? Evidence-based interventions are a central concept of evaluation. We are expected to find out whether an intervention has existing evidence that it works or is effective before we use it. This essentially means that the intervention has been implemented before and found to be effective in helping a particular group(s). The more evidence available that a program has worked, the more confident we can be about its effectiveness in using it again.

Page 16

Evidence can take many forms and can be used with varying degrees of confidence. The best evidence is based on evaluation studies using all of the characteristics of evaluations described earlier in the chapter. Other forms of evidence are also possible to consider, including the wisdom passed on by another professional who has used the intervention, and policies on practice promulgated by respected professional organizations like NASW. However, these sources are weaker than the results of evaluations unless they are ultimately backed up by evaluations. Evidence-based interventions are discussed in much more depth in the next chapter.

What Program Evaluation Is Not Considering what a program evaluation is not can also be helpful in understanding these key concepts. Program evaluations are not evaluations of individual clients; instead, program evaluations typically aim to provide useful information about cohorts of clients. In this case, the emphasis is on the impact of the entire program on the cohort that it serves. Pro-gram evaluations usually have an informed consent protocol that clarifies that the client’s personal identity is not to be revealed in association with any of his or her specific responses. However, a program evaluation occasionally could become an evaluation of individual clients under special circumstances. In this case, the ethical thing to do would be to notify the clients as to why the change is occurring before implementation of the pol-icy change. This preparatory step allows clients the opportunity to withdraw from the study if they wish.

Example of a Change from a Program Evaluation to Evaluation of Individual Clients

The state of North Carolina was successfully sued in a federal district court for failing to provide adequate services and physical safety to a selected group of people with mental illness and mental retardation in state mental hospitals (Dudley & Ahlgrim-Delzell, 2002). For four years, a university sponsor of a longitudinal evaluation of the class members reported only to a state agency on the aggregate data, using a set of statewide indicators of client progress or success. Then the state agency’s needs shifted and it decided to begin requesting information on individual participants in these programs. The original informed consent letter of the academic sponsor of the evaluation specified that data would be released only in aggregate form without identifying individuals. Therefore, a change in informed consent was needed. As a matter of policy, the longitudinal research team provided each class member and his or her guardian or family member with a letter informing of the change in policy. The letter explained the purpose of the study, out-

Page 17

ined the type of data that would be collected, and how the data would be obtained and released to the state on an individual basis. This letter also provided the recipient with the opportunity to with-draw from the study at any time without repercussions. Surprisingly, no one withdrew from the study.

Program evaluations do not evaluate the performance of individual staff members. Although evaluations of staff effort and performance are necessary, it would be confusing and a mistake to mix the purposes of a pro-gram evaluation and an evaluation of individual staff members. Such an initiative would not only confuse the participants in a study but also would likely instill distrust in the evaluation and discourage full cooperation of staff members. It may even encourage staff participants to manipulate their response in their favor or sabotage the study by boycotting it. Program evaluations are not public relations projects. A public relations project could be falsely presented as a program evaluation and used to promote the agency’s programs in annual reports and other material disseminated to the public. In this case, the purpose would be misleading and dishonest, as it gives the impression that a genuine evaluation is going on. Although the results or portions of the results of an evaluation may be use-ful and appropriate to display in public relations materials that promote the agency once an independent evaluation is completed, having a public relations emphasis built into an evaluation could bias the study toward a distorted positive outcome. In this case, some of the dramatic positive findings could end up being used in a report prepared for a funding agency while the neutral or less dramatic findings are disregarded and forgotten.

Example of a Program Evaluation with Potentially Conflicting Purposes

A student was asked to conduct an evaluation for a family agency of two different sets of groups of formerly incarcerated women. In designing the study, the student was told by her super-visor that the agency wanted to use the data in part to determine whether one staff member’s group practice was less effective than the others. After some discussion, the supervisor agreed to remove this concern, a personnel matter, as one of the agency’s study questions, so as not to confuse the study’s intent and to maximize staff cooperation. What Practice Evaluation Is Not What practice evaluation is not is similar to the points made about a pro-gram evaluation. For example, practice evaluations do not evaluate the performance of the individual staff members; they evaluate whether a staff

Page 18

member’s intervention succeeds in helping clients reach their goals. As mentioned previously, evaluations of staff performance are needed, but it would be confusing to mix such evaluations with a practice evaluation. Mixing these two different types of evaluation can easily instill distrust and resistance in staff members. Furthermore, the results of evaluations are likely to be manipulated if success with a client can be translated into a favorable performance review of their work. Practice evaluations are not measures of the effectiveness of a program; they only evaluate individual practice. Although one practitioner in a pro-gram may effectively help his or her clients based on the results of a practice evaluation, it is not logical to conclude that the program is therefore effective. A practice evaluation does not take into account other compo-nents of the program, including the services that other staff members offer. It is also important to note that although it would be unethical to refer to a staff performance evaluation as an evaluation of a program or practice, this can still happen. As mentioned earlier, the politics of evaluations could lead some agencies to considering such things if it could be viewed as ben-eficial to the agency and its programs. Evaluators always need to be ready to confront any ethical dilemmas that can surface, as their obligation is to do the most ethical thing.

UNDERSTANDING THE LARGER CONTEXT OF A PROGRAM

A program does not exist or operate in a vacuum. It is part of a larger dynamic system of many forces and factors, such as a wide range of policies, administrative leadership styles, staff and administrative communication patterns, varied perceptions of clients, and financial issues, all of which must be taken into consideration when conducting a program evaluation. Figure 1.2 provides a sketch of many of these factors in the agency and its environment.

All of these factors and their dynamic interplay can have a major influence on an evaluation conducted by the agency. Social policies have direct influence on evaluations since they give programs and practice approaches meaning and purpose related to the problem to be addressed and the pro-posed solution. Governmental policies (local, state, and federal) are important to consider because they dictate what these problems and solutions are that they will fund. Agency policies also have a direct or indirect influence in a wide range of areas such as financial matters, hiring, personnel, client admissions, and programmatic issues. An agency may have a particular pol-icy, for example, about which client groups to prioritize. Or they may take a strong stand supporting evidence-based interventions or client-centered practice. Or they could have a major commitment to strong fiscal policy and an expectation that benefits be in line with costs. Unfortunately, agency policies that explain what programs are about may not be evident to a new staff member, and may need to be identified for them.

Page 19

FIGURE 1.2 Contextual Factors in Evaluations

AGENCY’S ENVIRONMEN

Leadership style is another factor of influence. One illustration of this dynamic interplay is the leadership style of the administrators. Administrators can assume many styles, including autocrat, collaborator, and delegator. Administrators who are primarily collaborative, for example, are likely to have a different kind of influence on staff members when conducting an evaluation than those of an autocratic administrator. Also, organizational structures and processes, both informal and formal, are important factors with respect to their interplay with decision making (Weissman, Epstein, & Savage, 1983). In this case, while it is usually a good idea to consult everyone in an organization who is interested in and affected by an evaluation, some players will be more important to identify, including the administrators who formally oversee the program in question and those who have informal influence regardless of their formal title. These informal players could be instrumental in supporting or undermining an agency’s for-mal structure. They could be, for example, lower-ranked employees, such as a highly invested secretary or a popular and outspoken staff member. All in all, evaluators can commit a serious and possibly fatal error in an evaluation

Page 20

if they overlook stakeholders who may be potentially central to the success of an evaluation but are excluded from evaluation discussions. Many other contextual factors are also directly relevant to an evaluation. For example, are the agency leaders informed about evaluations, and supporters of evaluations? Or do they comprise novices who may be cautious and reluctant to pursue a study that can be risky? What is the agency’s track record in this regard? Some fairly standard questions of this sort could be asked at the outset: • What kinds of expertise does the agency have for conducting a variety of evaluations? • How cooperative are staff members, both professional and support staff, in taking on additional responsibilities such as filling out questionnaires or searching for client records? • What’s in it for the administration and direct service staff? Are all of the motives openly known and transparent or do some appear to be covert or hidden? • Are there reasons staff members may be suspicious of the motives for an evaluation or reluctant to participate for fear of jeopardizing their jobs?

Several contextual factors could also influence the extent to which the agency will disseminate the findings of an evaluation and implement its recommendations, including whether there are adequate resources, degree of desire to bring about a change in direction, and openness to risk the pro-gram’s future. More attention will be given to these various forces in later chapters within the context of specific topics and the seven steps in con-ducting an evaluation.

SEVEN STEPS IN CONDUCTING AN EVALUATION

A general approach for conducting an evaluation is introduced here and elaborated on throughout the book. The steps of this approach apply to both program and practice evaluations. The approach involves seven gen-eral steps based on a modified version of the steps identified in other evaluation models (e.g., Bamberger, Rugh, & Mabry, 2012; Posavac & Carey, 1997). Since the word steps is also used in other contexts in various parts of the book, whenever steps refers to the seven steps in conducting an evaluation it will be identified as the Seven Evaluation Steps. Seven Evaluation Steps

Step 1: Identify the Problem or Concern to Be Evaluated

Step 2: Identify and Explore Ways to Involve Stakeholders

Step 3: Determine the Purpose of the Evaluation

Page 21

Step 4: Plan the Evaluation

Step 5: Implement the Evaluation

Step 6: Prepare a Written or Oral Report

Step 7: Disseminate the Findings

Step 1: Identify the Problem or Concern to Be Evaluated

During step 1, the evaluator becomes familiar with the problem or concern that an evaluation will examine. Some general questions include the following: What is to be evaluated? Is the program working well or are there some problems that are evident? On the practice level, is my intervention with my client working well? Why or why not? During this step it is also important to begin gathering information about the program or practice context of the problem. It would be helpful to find out more about some of the pertinent components of the program or practice intervention. A description of the intervention should include identifying the client population that is served, the problems and needs that the intervention addresses, and the goals of the intervention. The services and goods that are provided to reach these goals are also important to identify and understand.

Step 2: Identify and Explore Ways to Involve Stakeholders

A concurrent step with the information gathering of step 1 is to identify and explore ways to involve the stakeholders of the program. Stakeholders are the people who are invested in the intervention in some way, such as representatives of the funding and regulatory groups that finance and set standards for the intervention or the administrators and board members of the agency sponsoring the program and those who direct the program. Some stakeholders, especially program directors, are likely to be evaluated on the basis of the intervention’s performance, so they have an obvious stake in what happens. Staff members who deliver the goods and services have an obvious stake in the intervention as well. Their jobs depend on the program’s survival, and their careers, currently and in the future, may also be entwined in the intervention’s outcome. In addition, clients who receive the intervention and their family members have a vital stake in what happens to the inter-vention, as their daily functioning and very survival may depend on how well it performs. Stakeholders are likely to be different for program and practice interventions. Program interventions tend to primarily have macro stakeholders, such as members of a board of directors, community advisory boards, and others in the public sector, while the main stakeholders of practice interventions are often supervisors, practitioners, and client advocates.

Page 22

Step 3: Determine the Purpose of the Evaluation

Once the evaluator is familiar with the program or practice intervention to be evaluated and the problems or concerns, and has developed a relationship with all stakeholders, more information needs to be gathered about who wants the evaluation and why. These discussions can help the evaluator find out how much the stakeholders know about evaluations, their views and past experiences with them, and whether they have a narrow or broad understanding of what an evaluation can be. These discussions should be used to highlight the potential contributions of an evaluation, such as program improvements, the creation of new opportunities to help or understand clients, or assistance to the agency in making an important decision.

Possible Contributions and Resources to Consider

• Expressed concerns are highly relevant to clients’ well-being

• Considerable interest in an evaluation

• Existing policies are supportive

• Financial support

• Awareness of a problem with a program

• Openness to examine a problem further

• Some understanding and respect for evaluation

• Openness to the evaluator as a resource

• Staff support

These discussions should not only uncover any contributions and resources that can support an evaluation, but also any apprehensions or doubts of stakeholders about evaluations generally. For example, could an evaluation be risky for some reason? Too costly? Take too much time? Inter-fere with program or practice operations? Therefore, during this step, it is also important to help the stakeholders identify any real or potential constraints.

Possible Constraints or Resistant Forces

• Limited time

• Costs too high

• Payoff is risky

• Not relevant to existing policies

• Evaluation focus can be too subjective

• Fears it will open up the need to change

• Limits to what can change

• Politics of the system

• Evaluator lacks expertise

Page 223

Problem of access to clients

• No need to justify such an evaluation to funding agency

• Lack of creativity

Step 3 becomes complete when all stakeholders and the evaluator have agreed on the general purpose of an evaluation. If a commonly agreed-on purpose for an evaluation cannot be identified, negotiations would likely discontinue or be delayed until a purpose could be identified. The purpose of a program evaluation could lead to keeping, expanding, or eliminating a program, whereas a practice evaluation may lead to varying an approach with some types of clients.

Step 4: Plan the Evaluation

Once a general purpose for an evaluation is agreed on, a plan for conducting an evaluation follows. As background work, a literature review is often needed to find out more about the problem that is the focus of the evaluation team and to identify reports on research methodologies, pertinent program and practice approaches, and studies on which to base the evaluation. Then several aspects of a research design need to be developed, including a set of study questions and hypotheses to explore or test, a data source (e.g., clients, staff members), a specific data collection method, and a data analysis plan. A plan to protect human participants of the study is also an important task to complete for approval.

All of these aspects of a plan should be discussed and involve the stake-holders so that there is strong support for a specific evaluation plan. When appropriate, the plan should be prepared as a readable written proposal or oral presentation understandable to all stakeholders. With practice evalua-tions, attention needs to be given to engaging the clients in understanding and participating in the evaluation. For example, a goal attainment scale may be used in a practice evaluation to measure the clients’ progress on their outcomes. In this case, the scale should be described to the clients initially, and clients should be encouraged to help define the specific outcome measures that fit their circumstances. Goal attainment scales and the role that clients can play in developing them are described more fully in chapter 9.

Step 5: Implement the Evaluation

The plan that has been developed for an evaluation, often referred to as the evaluation design, is now ready to be implemented. Often its implementation may involve several people in an agency, such as secretaries searching for client case material, staff members who might interview clients

Page 24

individually or in focus groups, and a questionnaire that staff members might need to hand out to clients. In a practice evaluation, staff members are likely to implement one form or another of a single-system design. Along with implementing the data collection effort, collected quantitative data are coded, entered into computer programs, and prepared for analysis. Qualitative data are usually prepared for analysis in narrative form.

Step 6: Prepare a Written or Oral Report

Once the study has been completed, preparation of a report of the results follows. Such reports are designed to address the major questions of stakeholders, usually reflected in the initial purpose worked out in step 3. The report can be oral, written, or both. Report preparation involves several steps, including organizing, analyzing, and interpreting the findings so that they are understandable; developing conclusions and recommendations that are useful and practical; and exploring the use of visual aids, such as tables and graphs, to assist in communication. Reports of program evaluations are usually prepared for different stakeholders (e.g., funding agencies, administrators, boards of directors, community groups) than those of practice evaluations (e.g., supervisors, program coordinators, clients).

Step 7: Disseminate the Findings

The last step of an evaluation is to disseminate the results or findings to stakeholders and others. Unfortunately, this step is often overlooked, or its importance minimized. The results are likely to be disseminated to several different types of stakeholders, some of which are obvious, such as the funding and regulatory agencies and agency administration. Other stakeholders may be easily overlooked but are also important, including former and cur-rent clients and relevant community groups. A report can be communicated in many forms—including oral or written, comprehensive or brief—and in varied formats, such as a technical report, a public meeting, a staff work-shop, a series of discussions, a one-page summary for clients and their families, or other formats.

What’s Next?

The chapters in part II focus on the bigger picture of evaluations, including various evaluation perspectives in chapter 2, professional ethics relevant to evaluations in chapter 3, and some common types of evaluations evident in agencies in chapter 4. These chapters are intended to provide a general background of relevant information for the chapters that follow them. Finally, chapter 5 begins to consider how to conduct an evaluation.