Evaluation model

CHAPTER 5: FORMATIVE AND PROCESS EVALUATION

This chapter begins to introduce a variety of program evaluation approaches. Evaluation methods can be relatively complex and labor intensive or can be uncomplicated and natural (e.g., client satisfaction). The two types of evaluation introduced in this chapter are important but not generally considered all that rigorous because their main focus is not on measurement of client outcomes. Positive client outcomes are the aim of all social service programs but before the outcomes can be studied the program and its clientele have to be well understood so that adjustment or revisions can be made as necessary. Formative and process evaluation efforts shape and refine programs; they do not constitute summative or conclusive evaluation, as we’ll discuss later.

Many different factors can affect the final evaluation design (e.g., the time available to complete the study, the funding, the staff resources, the client population, the seriousness/urgency of the problem, the motivation for the evaluation, its purpose or intended use, the level of expected cooperation from clients, staff, and other stakeholders. Some evaluators have adequate budgets and a relatively well-defined program to assess. Others have “messier” programs and inadequate funding. Where one starts as a first step in designing an evaluation plan might depend on whether it is necessary to begin immediately negotiating for additional release time to conduct the study (perhaps something an internal evaluator might do) or to begin investigating what program documentation exists (perhaps something an external/contract evaluator might do). A good way to start thinking about the evaluation process and its component parts is to construct a logic model.

DEVELOPING A LOGIC MODEL

A logic model is a tool for identifying the processes and components that lead to the proposed program outcomes (Torghele et al., 2007). More precisely, a logic model is a diagram, generally a one-page diagram, that describes how the program should work theoretically to achieve the desired outcomes for participants. That is, it pinpoints the program’s inputs, activities, outputs, and outcomes that capture the “logical flow and linkages that exist” within the program (Savaya & Waysman, 2005, p. 87). See the schematic below:

Program Model → Inputs → Activities → Outputs → Client Outcomes

Inputs are the resources poured into a program. They are the financial, organizational, or human resources invested and include staff, facilities, and budgets. Activities are the actions, tasks, or planned events that constitute the program. It’s what the program staff do (e.g., provide counseling) or create (e.g., workshops). Outputs are the countable products that result from the activities (e.g., the number of educational pamphlets distributed, the number of individual therapy sessions held, the number of clients transported to the day treatment program). Client Outcomes are the positive changes or client accomplishments that result from the program (e.g., improvement in self-esteem, graduation from high school, 6-month or longer abstinence from alcohol and drugs). Many logic models distinguish between short-term, intermediate, and long-term outcomes for clients.

Logic models not only help the evaluator to conceptualize the important program components, they also have these advantages:

· • They assist with undestanding what must happen for outcomes to be achieved.

· • They identify core processes so that re-engineering can occur; they provide a map for programmatic or organizational change.

· • They provide a framework for analyzing alternative strategies to achieve desired client outcomes.

· • They clarify for stakeholders the sequence of events and processes that contribute to program performance.

· • They enable a critical examination of the program and policy logic.

· • They facilitate communication about strategies, activities, and expected outcomes.

· • They provide a focus for achieving the desired end result by linking budget, activities, output, and outcomes. (Millar, Simeone, & Carnevale, 2001)

Logic models, like people, come in a great variety of styles and shapes. Some are arranged vertically, others are horizontal. Sometimes they don’t show three levels of outcome but perhaps only one or two. After all, many social service agencies don’t have the resources to check on clients two to five years after they have terminated services. What’s important with specifying outcomes is to decide upon what impact the program should make. Of course, it helps if these outcomes are easily observed or reported upon (e.g., an unemployed person securing a new job). Occasionally, logic models specify special populations or targets to be served. Logic models may also include programmatic assumptions and even specify activities by the program’s projected stage of development. Logic models can be conceptual as when a program is being planned, or actual as after a program has been operational for a while.

A template that can be used for creating logic models is provided in  Figure 5.1 .  Figure 5.2  contains an example of a vertical logic model from a United Way publication, Measuring Program Outcomes (1996). The Kellogg Foundation is widely recognized as an earlier advocate and source for information about the logic model. Their publication, the W.K. Kellogg Foundation Logic Model Development Guide, is widely cited and can be viewed online at  www.wkkf.org/Pubs/Tools/Evaluation/Pub3669.pdf . The University of Wisconsin Cooperative Extension has a wonderful PowerPoint presentation on logic models at  http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html . Other online resources on logic models can be found at  www.cdc.gov/eval/resources.htm .

FIGURE 5.1: LOGIC MODEL TEMPLATE

Source: Modified from the Kellogg Foundation, 2004.

FIGURE 5.2: EXAMPLE LOGIC MODELS FOR FOUR PROGRAMS OF SOUTHSIDE CHILDREN’S AGENCY

Source: United Way of America (1996). Measuring Program Outcomes: A Practical Approach, p. 45. Used by permission.

Let’s assume that the needs assessment and planning for the new program you wanted to start have been completed. The program has been implemented and has now been in operation about 3 months. If we were to talk to the staff, they probably would acknowledge that there are still some “rough edges” to the program due to its newness. Perhaps a few disgruntled clients have made complaints, and the agency director wants to initiate some sort of program review or evaluation. You are called into the director’s office to design a procedure for obtaining constructive feedback on the program. The agency director is committed to making the program successful and wants a program that the community will be proud of. Because the concern is not whether to continue or discontinue the program, but how to improve the program, what type of evaluation will you recommend to the agency director? How would you go about designing an evaluation that is concerned solely with program improvement?

FORMATIVE EVALUATION

Formative evaluation ought to be your recommendation to the agency director. Formative evaluations are employed to adjust and enhance interventions. They are not used to prove whether a program is worth the funding it receives but serve more to guide and direct programs—particularly new programs. In other words, formative evaluation is used to “form” the program. For this reason, formative evaluations are not as threatening and are often better received by agency staff than other forms of evaluation.

A good analogy for formative evaluation would be an experienced driving instructor sitting beside a beginning driver. If you have taught anyone to drive recently (or can objectively remember your own initial experiences), you may recall the beginning driver’s jerky steering movements and sudden accelerations and decelerations. The driving instructor helps the beginner become a more skillful driver by observing the process of driving and making constructive suggestions. The instructor is more concerned with the process than with any particular destination. Once driving skills have been acquired, it is assumed that the driver will be more likely to reach his or her destination.

The formative evaluator might look at client enrollment and drop-out rates, staffing patterns, management issues, problems and policies in the early months, and the costs associated with a program. This type of evaluation can also be used to determine whether a new or pilot program has been implemented as planned. Formative evaluations reveal “what services were provided, to whom, when, how often, and in what settings” (Moskowitz, 1989). Such evaluations are often considered “internal” agency business. Both strengths and weaknesses of an initial program may be identified.

Formative evaluation does not rely on a specific methodology or set of procedures. Instead, its focus is on acquiring information that would be useful for program improvement—whatever that would be. This information may come from interviewing staff or clients, reviewing agency records and progress notes, or participant observation. One could expect formative evaluators to look for glitches, breakdowns, lengthy delays, and departures from program design. They may find such problems as communication difficulties within the agency, lack of client participation in a program, or a need for additional in-service training for staff.

There is no single recipe for formative evaluation. How you would go about conducting one depends somewhat on the program, your preferences, the agency, and the context of the request for a formative evaluation. Let’s take a hypothetical example and see what options might arise.

Assume the manager of a program for children of substance abusers tells you that they are having some problems getting the program off the ground. The program is called the Strengthening Families Program (SFP), and was implemented approximately 3 months ago. Attendance has been a major problem as well as getting the children to pay attention during the planned activities. In addition, the parents have complained that they did not like the program. The manager believes that a formative evaluation would be a good way to examine these issues and provide some meaningful information for “fine-tuning” the program. The manager asks for your help.

One of the first things you might want to do is to examine the original plan and goals of the program. In this case, the manager followed the steps suggested in an evidence-based practice approach (Melnyk & Fineout-Overholt, 2005) and conducted a search for model parenting and family programs. That search revealed several principles for best practices in family programs for children of substance abusers. These principles meant that any new programs ought to be designed to be: (1) comprehensive, (2) family focused, (3) long term, (4) of sufficient dosage to affect risk or protective factors, (5) tailored to target the population’s needs and cultural traditions, (6) developmentally appropriate, (7) beginning as early in the family life cycle as possible, and (8) delivered by well-trained, effective staff. The manager and staff involved in developing the program concurred with the recommendations and found in the National Institute on Drug Abuse (NIDA) website a resource manual that detailed how to create a drug abuse prevention program for the children of substance abusers. The manual was from a NIDA-funded program previously implemented in over ten different locales across the country. Most importantly, the effectiveness of the program had been established in three different evaluation studies.

Thus, the program itself seems to be well developed and based on sound evidence-based practice standards. Knowing that the goal of formative evaluation is not to provide any sort of “final” or summative evaluation but to organize information needed for program improvement, where would you start a formative evaluation of this program?

CONDUCTING A FORMATIVE EVALUATION

At least three different ways to approach this formative evaluation come to mind. Faced with such a scenario, an evaluator could recommend any one or a combination of approaches.

APPROACH 1: LOCATE MODEL STANDARDS

If standards for similar programs have been developed or proposed by national accrediting or advocacy groups, then the local program could be compared against these standards and any discrepancies identified. This approach is frequently used by governmental units that fund, license, or oversee human services. When there are written standards, they are often put into the form of a checklist, and evaluators can monitor compliance with the standards and identify any areas of deficiency. This approach appears to work best when the expectations are easily defined. For instance, standards for a residential facility might require a window in each bedroom, a fire escape from the second floor, and fire extinguishers every 50 feet. Standards are not helpful when they are vague or difficult to operationalize (as when they state that a program should provide “adequate recreational opportunities”).

One place for the evaluator to begin would be to consider the original program manual/design as the “standard” and to compare it against the content actually being delivered. Are the recommended number and length of sessions to be closely followed? Do the program clients differ from the originally intended population in any way? Are the facilitators trained appropriately? Do the facilitators understand the content of the material they are presenting?

Ideally, standards would exist for every type of human service program that you might need to evaluate. Unfortunately, it is more likely that you will find the situation to be somewhat “hit-or-miss.” Some human service fields have well-developed and substantive standards. Other areas have minimal or no standards. However, you may find that standards developed for one human service program can be utilized for a similar program. The Council on Accreditation (COA), an organization which accredits private and public nonprofit agencies in 38 service areas and 60 different programs, has made their standards available on the website  www.coastandards.org . There is a good chance of finding standards there that might be applicable to any agency or program you need to evaluate. Their website indicates that in 2005 over 1,500 agencies were accredited by COA.

Professional literature is always a source of potentially useful standards or benchmarks. For instance, an article reviewing the literature on inpatient alcoholism treatment may indicate that the average relapse rate in five separate programs across the country was 48 percent during the first 6 months after discharge. If your local inpatient program is experiencing a 78 percent relapse rate within the first 6 months, this may be a strong indication of the need for an in-depth formative evaluation.

However, even if no standards are directly mentioned, a journal article might discuss how one agency dealt with problems such as lack of attendance or client dissatisfaction with certain features of a program. Even marginally relevant articles may contain the name of an agency or an “expert” who could be contacted to provide consultation for your program.

APPROACH 2: GET EXPERT CONSULTATION

Because the staff who work day to day on a program are so close to it, it is not unusual for them to be blind to certain areas where their program could benefit from improvement. (This may be particularly true in those environments where staff are tremendously overworked.) Formative evaluations often bring in experts or outsiders in order to obtain fresh perspectives so that the existing program can be seen in a new or different light.

With this approach you might seek out consultation from a recognized expert or from a similar program with a solid reputation. A person of some authority—the director or program director—from the program could be asked to conduct a site visit of your program. The consultant could review operating policies and procedures; interview residents, staff, and board members; and make suggestions for improvement by making comparisons with his or her own program or some other “model” program. (The standards in this instance may be more informal than formal if they are drawn from the consultant’s experience.) If money to pay the consultant is problematic, a low-cost alternative would be to ask for free consultation from the appropriate state officials who have an interest in the success of your program (e.g., the state department for mental health). It is not unusual for small agencies to have virtually no money for expert consultation. Because the perceived quality of the free consultation may be expected to vary considerably from community to community, some evaluators with no funds for consultation may be interested in still a third approach.

APPROACH 3: FORM AN AD HOC EVALUATION COMMITTEE

An ad hoc committee could be composed of treatment staff, board members, professionals from the community, service consumers, and other concerned stakeholders. The committee might begin by interviewing staff and then move to participating families and children, and then selected professionals outside the agency. Some or all of the committee members could visit similar programs. If this is not economically feasible, the committee could write to other programs asking for copies of their policies and operating procedures. From reviewing these, new policies or procedures may be developed as the evaluation committee devises their own set of standards for the program. The committee may identify a number of areas or discrepancies that, in their opinion, need to be addressed to improve the program.

Members of the ad hoc committee don’t have to be “expert” in the sense of individuals commanding high salaries. Another variation of the ad hoc committee idea would be to interview clients or to create focus groups for feedback. For instance, in a program designed to help pregnant low-income women to stop smoking, both patients and providers reported that the patient education materials (the Guide) looked “too much like work.” It was therefore revised to be more user-friendly, with graphics and inspirational messages, a new cover, and rewritten to a sixth-grade reading level. The 14-minute video was also shortened and a second one was produced so that versions were available with White and Black actors (Crawford, Woodby, Russell, & Windsor, 2005). Formative evaluation can be used to adapt interventions to specific target groups and to identify appropriate strategies for best reaching them ( Box 5.1 ).

BOX 5.1: EXAMPLE FROM THE LITERATURE OF FOCUS GROUPS USED INFORMATIVE EVALUATION

It is no surprise that adolescents experiment with drugs. A question arises, “Can an early intervention be designed for 14–18-year-olds that could be adopted in schools?” Public health nurses in Finland developed a brief instrument designed to classify students’ drug use into one of 4 groups: abstinence/experimental, recurring use, risky use, and hazardous use. With the students’ permission, the instruments were forwarded to the school nurse who reviewed them and then met with students whose scores identified a possible problem. Four focus groups were held with nurses, social workers, doctors, and school counselors who indicated that early intervention was possible. The process worked because a confidential atmosphere allowed trust to develop. The brief instrument helped concretize the assessment, allowing the student to self-evaluate his or her substance use and to reach a consensus about usage with the nurse. The contents of the intervention were made more practical and reformulated as a result of the focus group feedback. A formative evaluation to develop a school health nursing early intervention model for adolescent substance abuse. Public Health Nursing, 24(3), 256–264.

Source: Pirskanen, M., Laukkanen, E., & Pietila, A.M., 2007.

The formative evaluation design, as stated earlier, is employed to shape programs in their early stages. Once the program has been established, it is important to conduct a process evaluation. A process evaluation is defined as “a comprehensive description and analysis of how … programs are conceptualized, planned, implemented, modified, and terminated. Process evaluation attempts to assess the quality and purpose of program activities relative to the desired outcome or results of these programs” (Krisberg, 1980, p. 217). It is to this topic that we will next turn our attention.

PROCESS EVALUATION

Figure 5.3  shows where formative and process evaluation fit into the overall evaluation picture. One major difference between formative evaluation and process evaluation is that while a formative evaluation seeks to influence the initial development of a program, a process evaluation can be conducted anytime during a project—even at its end.

FIGURE 5.3: HOW FORMATIVE AND PROCESS EVALUATION FIT INTO AN OVERALL EVALUATION PLAN

Why would anyone want to conduct a process evaluation at the end of a project? Process evaluations are typically required for research and demonstration projects because sponsors want to know what was learned during the implementation of the project. Such information could be valuable to other communities considering whether to start such a program. In addition, process evaluation can help determine whether the “failure” of a program was due to a poor program or poor intervention, or if it was because implementation of the program was problematic (Harachi et al., 1999; Orwin, 2000).

Specifically, a process evaluation may have the following overarching purposes or goals:

· 1. Program description

· 2. Program monitoring

· 3. Quality assurance

Each of these main purposes may have subgoals or objectives. In addition, a process evaluation may include one or some combination of all three of these goals.

To take an actual example of process evaluation used for program description, in one community an agency developed a proposal to recruit African-American men to serve as mentors for minority teenagers. In the original proposal, the agency specified that 80 mentors would be recruited and trained, and each would be matched with an adolescent. This proved to be a lot tougher than the agency expected. By the project’s end, less than half the desired number of mentors had been obtained.

In this instance, process evaluators looked at the activities used to recruit mentors (e.g., public service announcements, speaking with ministers of black churches)—the evaluator wanted to know what had worked and what had been a waste of time. The evaluation also examined other problems encountered, such as difficulties in recruiting teenagers and their parents, and securing referrals from school counselors and other human service professionals. Process evaluation informs others about what they might expect if they were to launch a similar program.

Another purpose of process evaluation is to assist in explaining why a program did or did not achieve expected outcomes. For example, a large statewide agency wanted to test whether the use of peer volunteers was more effective in reducing subsequent hospitalizations than a new procedure where clients with severe mental illness were assisted in developing a crisis support plan. Three different sites were selected from across the state, and staff received all necessary training. Four years later, at the project’s end, a process evaluation revealed that (1) some staff did not complete crisis plans with their clients—evidently because they were not aware of the necessity to do so; (2) there had been no additional training of staff after the first year; and (3) monitoring of clinical records for clients without crisis support plans also was lacking.

Further, while most clients reported good experiences with their volunteers, the use of volunteers was not a standardized intervention. That is, there was tremendous variation in the volunteers’ activities, responsibilities, and the amount of contact they had with assigned clients. Clients may have had daily, weekly, monthly, or only twice a year contact with a volunteer.

Obviously, even these brief observations would be important to other agencies contemplating similar projects with volunteers and crisis support plans. They surely would want to ensure greater use of crisis support plans by providing ongoing training and closer monitoring of staff completion of these plans. Additionally, to make the interventions of volunteers more uniform, they would need to specify minimum requirements—for example, each volunteer meeting 30 minutes each week, face-to-face with his or her assigned client. And, like the other recommendation, the amount of time volunteers spend with their clients should be carefully recorded and monitored. The process evaluation revealed that documentation of the amount of contact with peer volunteers was practically nonexistent.

PROGRAM DESCRIPTION

One goal of process evaluation is to describe the program. Program descriptions document the operations of a program, which is essential to those who want to replicate or transfer the knowledge/technology from the program. Process evaluations provide the data necessary to judge the intensity and reliability with which services were delivered; they rely heavily on data normally captured by agencies. To provide further detail, examples of information that a process evaluator might want to examine are contained in  Boxes 5.2  and  5.3 . Their contents draw on Scarpitti, Inciardi, and Pottieger (1993) and the Evaluation Guidebook, prepared by the U.S. Department of Health and Human Services, Office of Community Services (1992).  Box 5.2  provides a quick overview of types of data that may be useful in a process evaluation, while  Box 5.3  lists questions that might be asked and links these questions to possible data sources.

BOX 5.2: TYPES OF DATA USEFUL IN PROCESS EVALUATION

· 1. Client sociodemographic characteristics

· 2. Client service usage (type and amount of services received)

· 3. Referral sources (referral and coordinating agency perspectives of program strengths and weaknesses)

· 4. Staff characteristics:

· Professional degrees

Length of experience

Sociodemographics

Staff perceptions of program strengths/weaknesses

· 5. Program activities:

· Special events and meetings

Staff meetings

Training provided

Written program protocols, procedures, and training manuals

Any information to answer the questions: “What happens to clients?” “What is the program?”

Observation of program activities. Is the program being implemented as it is supposed to be?

· 6. Minutes of board, staff, and committee meetings

· 7. Correspondence and internal memos concerning the project

· 8. Client satisfaction data, client reports of program strengths, weaknesses, and barriers

· 9. Financial data; program costs and expenditures

BOX 5.3: QUESTIONS AND DATA SOURCES USEFUL IN PROCESS EVALUATION

General Questions	Examples of Potential Data Collection Sources or Techniques
Why was the program introduced into the community or organization? What need did the program fill? What were the political mechanisms by which the program was initiated and/or maintained?	Literature reviews; local and national legislation review; existing local, state, and national data; legislator interviews; administrative interviews; board member interviews; key community leader interviews
What people and organizations were involved with the implementation? What organizations are involved? What are future plans for involving other organizations? Why? What is being done to ensure the organization will be involved with the program?	Administrative interviews; board member interviews; key community leader interviews; program documentation
What collaborative efforts are utilized with this program? How have the collaboration relationships changed over time?	Administrative interviews; staff interviews; board member interviews; program documentation; interviews with collaborating agencies
What are the norms, assumptions, customs, traditions, and traits of the program? In other words, what is the culture of the program? What principles is the program based on?	Administrative interviews; staff interviews; client interviews; board member interviews; program documentation; interviews with collaborating agencies; focus groups; observations
What is the program? What are the components? What changes were incorporated into the program since inception? Why? What changes are planned for the future? Why?	Staff interviews; client interviews; administrative interviews; program documentation; observations; focus groups
What is the sociodemographic makeup of the program population? Is the program actually serving the population it was intended to serve? If not, why not? What changes are planned to reach the intended target population? How are clients recruited? How satisfied are clients with the program?	Case records; program documentation; client interviews or surveys; staff and administrative interviews
Who are the staff? How effective are the staff members? How are staff trained? How are they evaluated? How satisfied are the staff? How often do staff attend meetings? What are the roles and responsibilities of the staff?	Administrative interviews; staff surveys; program documentation; client interviews
What design changes may be necessary to expand the program or offer it to other sites for replication?	Administrative interviews; board member interviews
What outcome evaluation activities are planned? What would the main outcome variables of interest be?	Administrative interviews; board member interviews; key community leader interviews; collaborating agency interviews
What are the program costs and expenditures? Have there been changes in funding since the inception? What about the future funding plans?	Administrative interviews; board member interviews

As indicated in column 2 of  Box 5.3 , there are numerous ways to collect process evaluation information about a program. These methods include face-to-face and telephone interviews, surveys, key informant interviews, focus groups, organization record analysis, program documentation analysis, observations, and case studies. Each of these data collection techniques is described in other chapters throughout this book.

In addition, specific strategies can be utilized to collect data. Some data collection strategies, such as focus groups and observations, are qualitative data collection methods; and some, like surveys, are typically quantitative data collection techniques. When doing process evaluation, it is probably wise to use some combination of qualitative and quantitative data collection procedures to best capture what is happening with the program. It is also helpful to include representatives from multiple agencies in the interview process to get a picture of the program that is as accurate as possible.

In a process evaluation of a local drug court program, 8 main steps were involved.

Step 1: clarifying with the program administrators what information they were actually interested in. It is critical that all key stakeholders are clear about what information will be produced. If administrators are under the impression they are going find out whether or not the program is successful, process evaluation is not appropriate for providing that information—it is not outcome evaluation.

Step 2: developing a data collection plan to guide the process evaluation. Some of the questions listed in  Box 5.3  guided the instrument development as well as an extensive literature review. The literature review involved a search of published literature about drug court programs as well as unpublished drug court evaluation reports.

Step 3: identifying key program stakeholders to interview. Program administrators are frequently in the best position to discuss what kinds of people come into contact with the program. In this case, judges, clients, staff, police officers, jail personnel, attorneys, and treatment personnel were all important stakeholders; and their perceptions of the program were important to include. Due to funding and time constraints, it was impossible to contact all of the clients. Approximately 20 percent of the clients were randomly selected for interviews.

Step 4: developing process evaluation instruments. Based on the information collected in Step 3, 8 different instruments were developed: (a) administrative interview; (b) judge interview; (c) law enforcement and corrections surveys; (d) treatment provider surveys; (e) prosecutor surveys; (f) defense attorney surveys; (g) client surveys; and (h) staff surveys. Instruments developed combined both quantitative and qualitative questions, and also provided flexibility to probe or write any comments made from quantitative questions.

Step 5: conducting the interviews. The process evaluation included interviews with administrative personnel of the drug court program; each of 5 judges involved in the drug court program; face-to-face interviews with 22 randomly selected active clients. Additionally, surveys were distributed to all drug court staff, community treatment providers, 6 randomly selected defense attorney representatives, 4 prosecuting attorney representatives, 1 representative from the local probation and parole office, 1 representative from the local jail, and 2 local police department representatives. In all, 69 different individuals representing 10 different agency perspectives provided information about the drug court program for the process evaluation.

Step 6: examining program documentation and records.

Step 7: analyzing the information. Responses were reported in specific sections guided by general areas such as those in  Box 5.2 . Specific interviewee wording was used to highlight comments throughout the report. A conclusion and summary section integrated the main findings across all of the respondents in the final report.

Step 8: writing the report. This step included several iterations of the report to include edits and comments by key program administrators. It is important to involve program administrators in the draft stages of the final report. They may have information about the program that was not fully explained initially, and they can correct misinterpretations. These 8 steps can be adapted for the process evaluation of a wide variety of programs.

PROGRAM MONITORING

Program monitoring is the second overarching purpose of process evaluation. Program monitoring can be valuable to the process evaluator who is trying to understand what happened in a program and to whom. The novice evaluator should not develop the opinion that program monitoring is conducted only when a process evaluation has been requested. Ongoing program monitoring is essential to the sound management of all programs.

Like formative evaluation, program monitoring is a basic form of program evaluation. Why is it so elemental? Because a program that is not reaching its intended population is misdirected—perhaps duplicating services to a population already well served. Further, it makes no sense to conduct a more sophisticated evaluation to determine if the intervention worked when it was not applied to the population in need.

Program monitoring does not require elaborate research designs, nor does it usually require an advanced understanding of statistics. Often program monitoring starts with examining a program’s specific goals and objectives and comparing these with the kind of data that most human service agencies routinely collect. These data are “monitored” to ensure that the program is serving those for whom it was designed. It is entirely possible that with the passage of time a program may somehow get diverted and not serve the population originally targeted. (See the discussion on “program drift” in  Chapter 14 .)

Changes occur in practically all programs. Once the initial excitement of starting a new program has worn away, staff and agency resources may be siphoned off or redirected as newer, more urgent problems come along. As the original staff take other jobs, retire, become promoted, or move to other programs, incoming staff may have different notions as to what the program should accomplish or to whom it should be directed. Subtle, almost imperceptible changes in staff, the program philosophy, the composition of the clientele, or the orientation of new employees can result in programs departing significantly from what was first proposed.

It is crucial that conscientious program managers, administrators, and agency boards of directors continuously monitor the progress of programs. Only poor management would benevolently ignore a new program for 10 or 11 months, and then at the end of the funding period attempt to hold the program staff and its manager accountable. To ensure that a program serves the target population and obtains program goals and objectives in the manner expected by the funding source, regular program monitoring is required. Unlike formative evaluations, which tend to be single-episode evaluations, and others that we will discuss later, program monitoring ought to be ongoing. Program monitoring should be thought of as a routine activity where a program director reviews patterns of use data on a regular basis (more often than once a year). Routine monitoring can reveal problems before they become overwhelming and track progress toward meeting the sponsor’s or agency’s expectations.

BECOMING A PROGRAM MONITOR

Human service programs exist to provide either goods or services to clientele. Some programs provide tangible goods: food (soup kitchens), beds (emergency shelter), or clothing. Other programs provide services where the products are more intangible (counseling, mental health education or prevention services, self-esteem groups).

Regardless of whether the client/consumer receives a tangible or intangible product, there is always something that can be counted. For instance, a child protection agency may provide homemaker services to 189 families during the course of a year. This same agency may complete 42 adoptions, approve 64 foster homes, and provide 1,195 hours of individual or group therapy. Each of these program products can be used to provide some measure of accountability. The agency director may be unhappy with the provision of homemaker services to only 189 families because she had hoped that 200 families would receive homemaker services. On the other hand, the director may be pleased (since there had been major staff turnover in the program) that 64 foster homes were approved. (At one point it looked as if only 50 foster homes might be inspected and approved.)

The first step in program monitoring consists of deciding what program products, events, or activities are important enough to count. Not every activity associated with a program is important enough to monitor. For example, we have never seen an annual report that listed the number of times that the stapler was used. It may not be important to count the number of times that calls are placed. However, if you are the manager of a telephone crisis hotline or a telephone information and referral service, it may be important to keep records on the number of telephone contacts categorized by problem (e.g., suicidal ideation, drug use, or unexpected pregnancies).

Just because something can be counted does not mean that it ought to be counted. A telephone counseling hotline service recorded daily (by shift) the number of telephone calls received by problem area. Even though they used 20 or so categories to log the type of call, about 15 percent of the calls fell into the miscellaneous category. After a little investigation it was learned that they were counting incoming phone calls that might be best described as “personal.” A mechanic might call to report that a staff member’s car had been repaired, or a child would call a parent at work on arriving home from school. Although these calls may have been important to the people receiving them, counting such calls in the monthly service report gave the appearance that the telephone hotline was actually a lot busier than it was. Counting these calls would not tell the funding source anything important about whether the program was providing the type of service originally planned. So, while there was accountability, counting for the sake of counting led to some inane results. Program monitoring ought to involve more than tallying the frequency of events.

Program managers should consider what events or activities would give them important information about their programs. For instance, examining the length of time that clients receive service or the amount of time between initial contact and service delivery might be useful. Also, it is not unusual for administrators to be interested in how staff use their time—the proportion of it that is direct service (sometimes known as billable time in mental health centers) and that part that is supportive (“desk work,” traveling, committee meetings).

Box 5.4  provides an example of possible data items that could constitute a monthly report for a drug court program. Managers could use such items to monitor whether the drug court program is “on track” with expectations and benchmarks.

BOX 5.4: DRUG COURT MONTHLY MONITORING REPORT EXAMPLE

· 1. Number of clients currently active:

· 2. Number of new clients accepted into the program:

· 3. Number of participants graduated to next phase:

· a. From phase I to phase II:

· b. From phase II to phase III:

· c. Graduated from phase III:

· 4. Number of court sessions:

· 5. Number of drug screens:

· 6. Number of participants identified as using based on drug screens:

· 7. Number of individual counseling sessions:

· 8. Number of group sessions:

· 9. Number of family support sessions:

· 10. Number of participants referred to outside agencies:

· 11. Number of participants referred to outside agencies for residential services:

· 12. Number of participants employed:

· a. Part time:

· b. Full time:

· c. Disabled:

· d. Homemaker:

· 13. Number of participants in educational pursuit:

· a. High school/GED:

· b. College:

· c. Vocational training/rehabilitation:

· d. Adult education:

· 14. Number of employment/education verifications:

· 15. Number of housing verifications:

· 16. Total amount paid toward court obligations:

· 17. Total number of sanctions:

· 18. Total number of participants rearrested for new charges:

· 19. Total number of terminations:

MISSION STATEMENTS, GOALS, AND OBJECTIVES

In deciding what is important to count or monitor, it is helpful to become familiar with the agency’s mission statement. Mission statements are statements of purpose—they explain what the agency is all about. Mission statements provide a common vision for the organization, a point of reference for all major planning decisions; they answer the question, “Why do we exist?” Mission statements not only provide clarity of purpose to persons within an organization, but also help gain understanding and support from those people outside the organization who are important to its success (Below, Morrisey, & Acomb, 1987). If you are in an agency that does not have a formal mission statement, or if you find it necessary to draft one, start by looking at the agency’s charter, constitution, or bylaws. These documents describe the purpose behind the creation of an agency. Five examples of agency mission statements follow.

· The mission of the Northern County Victims’ Assistance Program is to provide assistance to individuals who have been victims of felony crimes in Northern County. This assistance will be directed at the devastating emotional and psychological consequences that victims of crime and their families experience.

Our mission calls us to live out the interdependent values of love and justice, to lift oppression, and heal brokenness of individuals and families, of groups, and of society itself. [Excerpt from the Mission Statement of a Catholic Social Service Bureau]

The mission of the Western County Mental Health Board is to improve the quality of life in our community by promoting mental health, by preventing and reducing mental and emotional problems, substance abuse problems, and by minimizing their residual effects.

It is the purpose of KET, a unique communications resource linking all Kentuckians by television, to be an institution of learning for children and adults of every age and need, a statewide town hall through which interested citizens can together explore issues of mutual significance, a performance stage for the outstanding talent of Kentucky and the great artists of the world, and a catalyst for uniting the citizens of the Commonwealth in common purpose to solve common problems and to stimulate growth and progress for all.

The mission of the Drug Court program is “to stop the abuse of alcohol and other drugs and related criminal activity” (USDJ, 1997, p. 7).

As can be seen from these examples, mission statements are not going to tell you exactly how the agency will go about its business or when it expects to complete its missions. But, they do inform as to the nature of the organization. One can readily deduce the religious orientation of the agency in the second example. Mission statements are useful in that they communicate the agency’s purpose and they express values, suggesting what is important for the agency to address with its resources. Mission statements are usually stated in somewhat vague terms. They are not specific as to what types of services will be provided or how the client will get those services. How important are mission statements? Sugarman (1988), in listing six major criteria that define a well-managed human service organization, noted that the first characteristic is “a clearly defined mission or purpose, well-understood by its members, and it has goals and plans based thereon” (p. 19).

Occasionally it becomes necessary for an agency to change its mission. Perhaps the best example of this is the March of Dimes. This agency was created because of the problem of polio (an infantile paralysis caused by a virus). With advances in research, vaccines were discovered, and the disease has now been virtually eliminated. The March of Dimes continues to exist, however, but its mission is now to fight birth defects. Whenever there is a change of mission, there must be a corresponding change of program goals and objectives.

Goals follow from mission statements and also tend to be general and global with regard to activities and products. Patton (1982) noted that a goal statement should specify a program direction based on values, ideals, political mandates, and program purpose. Goals are not specific as to when or how something will be accomplished but speak instead to aspirations.

Goals provide the focus, orientation, and direction needed to harness the combined energy and activities of a staff so that chaos and confusion are minimized and clients’ needs are served by the program. Imagine a team of horses hitched to a wagon. Then picture that same wagon with a team of horses attached to each of the four sides. Which wagon is likely to move, and which will go nowhere?

Many people make the mistake of thinking that goals have to be accomplished within a short period of time (perhaps even within one’s lifetime). However, there is no such requirement. Many human service agencies have goals that will likely never be accomplished because they involve continuing needs. How many of the following goals do you feel it will be possible to attain?

· 1. The agency will eliminate all poverty.

· 2. The program will prevent child abuse and neglect.

· 3. The hospital will rehabilitate persons who have problems with alcohol.

· 4. The university will strengthen its commitment to scholarship and academic excellence.

It is perfectly acceptable for an institution to have broad goals that they may never reach. An agency (or a program for that matter) has not failed when a goal is not achieved; the reason is that the goals that human service agencies typically set are not easy to achieve.

Unlike mission statements and goals, objectives are specific and precise. Objectives allow us to measure progress being made toward the achievement of a goal. They declare what will be accomplished by a certain date. Objectives should have a single aim and an end product or result that is easily verifiable. Drucker (1980) notes that program objectives such as “to aid the disadvantaged” or “to provide health care” are sentiments (and vague ones at that) explaining why a program was initiated rather than what it was meant to accomplish. He continues:

To have a chance at performance, a program needs clear targets, the attainment of which can be measured, appraised, or at least judged…. Even “the best medical care for the sick,” the objective of many hospitals in the British National Health Service, is not operational. Rather, it is meaningful to say: “It is our aim to make sure that no patient coming into emergency will go for more than three minutes without being seen by a qualified triage nurse” (p. 231).

Patton (1982) makes the distinction of separating the concept (the goal) from the measurement of it (the objective). This is a useful way to think about the differences between the two. When objectives are properly developed, they leave little doubt about what will be done, the date when its accomplishment can be expected, as well as a clear measure of whether the objective was achieved. To be useful, objectives must specify events or activities that can be independently determined. As an example, see  Box 5.5 . The objectives listed in this example can be used to monitor the program over time or can be used to target areas in need of improvement. As indicated in  Box 5.4 , two types of indicators must be distinguished.

BOX 5.5: EXAMPLE OF A DRUG COURT AGENCY MISSION STATEMENT, GOALS, AND OBJECTIVES

Mission:

The overall mission of Drug Courts is to stop the abuse of alcohol and other drugs and related criminal activity.

Goals:

· 1. Promote client abstinence

· 2. Decrease client recidivism

· 3. Increase community safety

· 4. Increase client life skills

· 5. Increase community awareness about the program as well as about substance abuse

· 6. Expand and maintain resource base

Objective indicators for 1999 compared to 1998

· 1. Promote client abstinence

· a. To increase the number of drug-free babies born

· b. To increase the number of clean urines

· c. To increase the number of meetings attended for each client (AA/NA, treatment groups, education, case specialist meetings)

· 2. Decrease recidivism

· a. To decrease the percentage of clients re-arrested while in the program

· 3. Increase community safety

· a. To lower community drug arrests for FY 1999 compared to FY 1998

· b. To lower community property crime for FY 1999 compared to FY 1998

· 4. Increase life skills

· a. To increase the percentage of clients living in court-approved housing

· b. To increase the percentage of clients in court-approved employment

· c. To increase the percentage of clients obtaining a GED or in college

· d. To increase the percentage of clients gaining or keeping custody of children

· 5. Increase community awareness about the program as well as about substance abuse

· a. To increase the number of staff media contacts

· b. To increase the number of media stories on the program or program clients

· c. To increase program funding

· d. To increase the number of staff requests for speaking engagements

· e. To increase the number of client referrals to the program

· 6. Expand and maintain resource base

· a. To increase the number of agencies for Drug Court client referrals

· b. To increase the number of agencies that will work with the Drug Court program

Source: Logan, T., Williams, K., Leukefeld, C., & Minton, L. (2000). A process evaluation of a drug court: Methodology and results. International Journal of Offender Therapy and Comparative Criminology, 44(3), 369–394. Logan, T., Hoyt, W., McCollister, K., French, M., Leukefeld, C., & Minton, L. (2004). Economic evaluation of drug court: Methodology, results, and policy implications. Evaluation and Program Planning, 27(4), 381–396.

Process indicators are typically data items routinely collected by the agency and used in program monitoring. Outcome indicators are generally collected from participants after program completion while process indicators are generally collected during program participation. Process indicators generally only include information from clients or program participants while outcome indicators include information from control or comparison groups as well as program participants. Finally, outcome indicators answer the question “Does the program work?” while process indicators answer questions like “What happens during the program?” “Is the program implemented according to how it is supposed to run?” “What is the program?” “Is the program meeting its annual goals?” Objectives help to identify what process indicators should be included in the process evaluation.

WRITING PROGRAM OBJECTIVES

To write an objective that provides some measure of accountability (so that it can be determined whether or not the objective was met), think in terms of activities that can be counted or observed. The objective should state what will be accomplished and when it can be expected. A model for writing specific objectives is as follows:

To increase	admissions 10 percent	by June 30, 2010
(verb)	(specific target)	(date)

Some examples of verbs that are useful when writing objectives are:

· To increase, add, develop, expand, enlarge

To decrease, reduce, lessen

To promote, advertise, publicize

To start, create, initiate, begin, establish

However, the choice of the verb may not be as critical as ensuring that the reader can visualize a measurable result. The use of vague terms can make it difficult to determine whether the results were obtained. Avoid language such as that contained in the following program objectives:

· To help clients discover healthier relationships with others.

To help clients develop an appreciation of etiquette.

To help students become better citizens.

To assist clients in getting their lives back together.

To increase the community’s support of ….

To improve clients’ understanding of themselves.

To help families learn about alcohol and alcoholism.

All of these objectives share the same problem—they lack specificity. In other words, it would be difficult to know when these objectives were ever obtained. They do not inform as to how much has to be learned, developed, or understood. (How would we know if clients had improved their understanding of themselves?) Also, they do not provide any indication of dates when these events will be accomplished. And, there is no way of knowing exactly when the objective should be accomplished—the target event that should allow independent verification is too vague or absent.

Sometimes agency directors and program managers, in an effort to make their programs look good, write objectives that will be too easily achieved. Monitoring bodies can contribute to this situation. We once saw an evaluation form that contained these two questions: “Did the project achieve its objectives?” and “How many of the project’s objectives were realized?” Every program manager would like to say that he or she accomplished 100 percent of the program’s objectives. However, if the quality of these objectives is not assessed, program managers may write only objectives that they know they can meet. Setting objectives too low results only in pointless “busywork.” If a program provided 2,200 units of individual counseling one year, then it should be expected to exceed that number in the next year. One exception to this rule might be when the program expects to lose a significant amount of staff, funding, or other resources. Another exception might be if the quality of those sessions were sacrificed. It is also important to balance quality with quantity, as we will see in the next section. Objectives should be set high enough to challenge the staff. They should not be impossible to obtain, but they ought to encourage staff to stretch a bit and perhaps to think a bit “out of the box” about how to meet objectives.

Once program objectives have been developed, monitoring for managerial purposes is possible. When objectives are being developed for new programs, there may be a natural tendency to make conservative estimates of what can be accomplished. Rather than overestimate the number of clients who can be served in a year, program managers may be more likely to underestimate what can be done. These objectives can be tempered by reality if data exist for the start-up phase of other formerly “new” programs. In the absence of such data, educated guesses are appropriate.

However, program monitoring really comes into play when programs have begun to generate service data. In the example from a counseling agency given in  Table 5.1 , it is possible to identify groups that are not getting their “fair share” of the agency’s resources.

TABLE 5.1: CLIENT UTILIZATION DATA, ACCEPT ALL COUNSELING SERVICES, INC.

Variable	2005 (%)	2006 (%)	2007 (%)	2008 (%)	2000 Census (%)
Widowed	3	3.6	2.5	2.3	4.95
Over 60 years old	4	4.5	5.5	5.2	24.75
African-Americans	5	5.5	6.3	6.5	13.25

As can be seen from the table, widowed persons, those over the age of 60, and minorities are not represented in the agency clientele to the extent that would be expected from their proportions in the population. With just this much information, a program manager could develop the following objectives:

· OBJECTIVE 1. To increase the percentage of widowed persons served by the program to 5 percent of the total clientele by December 31, 2009.

OBJECTIVE 2. Through special outreach efforts, to increase the number of older adults served by the program until 15 percent of the program’s clients are 60 or older. This objective to be reached by July 1, 2010.

OBJECTIVE 3. By January 1, 2010 to double the number of African-American clients served by the program in 2008.

With these objectives in place, the program manager and the program’s staff now have a clear set of expectations for their future efforts. Once these objectives are met, new ones can be developed. If they are not met, corrective actions may be needed (providing there were no extenuating factors to explain the nonperformance). The setting of objectives provides a basis against which the program’s accomplishments can be examined.

WHAT SHOULD BE MONITORED?

Program monitoring can be used to check a program’s progress in meeting certain objectives (e.g., increasing the number of minority admissions). In this sense, it is analogous to being told by your physician what kinds of things to monitor to maintain health. For instance, persons with a diagnosis of diabetes must monitor the amount of carbohydrates they consume. Persons with hypertension are told to monitor their salt intake. However, managers need not wait for their programs to become “ill” before employing program monitoring.

Program monitoring can be used most effectively in a diagnostic sense. Managers can use program monitoring to look for “symptoms” that would help them diagnose potential problems. However, the kinds of problems to look for are difficult to state succinctly because of the enormous diversity in human service programs.

Programs range from small volunteer-run ones to programs that employ hundreds of staff. Every program can be expected to have a somewhat unique set of problems. Even similar programs are likely to have different problems. This is due to differences in staff composition, local (and often informal) policies and procedures, relationships with other professionals and agencies within the community, the guidance and leadership of the agency administrator, the amount of financial support, and such factors as the interest and involvement of the board of directors.

Veney and Kaluzny (1984) have summarized the data appropriate for monitoring as inputs, process, and outputs. They describe inputs as consisting of the resources by which the program is carried out. Resources include such categories as project staff, office space, and office equipment and supplies. With regard to inputs, the important thing to monitor is the amount budgeted for a program against what is actually used or allocated. For process, it is important to monitor the activities that were intended to be carried out during program implementation. Outputs are the results of the program—what the program actually produces.

Using this scheme, a manager might check to make sure that a program is not overspending its budget (or that it is getting all that it is entitled to); that planned activities are being conducted on a timely basis; and that regular accounting is being made of the number of service units produced (such as meals provided or other quantifiable products). Thus, a manager would know there were problems if the agency spent 75 percent of its budget by the half-year mark, if scheduled program activities were not being performed, or if the program started off providing 300 service units during the first month but fell off to only 125 service units during the second month.

Monitoring inputs, process, and outputs gives the program manager basic information needed to manage programs. However, this information may not be complete enough to allow a manager to “fine-tune” a program. A program could be meeting expectations in terms of its budget expenditures, its activities, and the number of products that were expected, yet still not be perceived as doing the job it was designed to do.

What other informational items might be helpful to monitor? One would be referral sources. The conscientious manager should monitor where referrals are coming from and in what proportion. It may be perfectly acceptable for a private counseling agency to have 92 percent of its clients self-referred. On the other hand, public agencies may want to see referrals coming from a broad spectrum of the community. The program manager in a public agency who notes that over a 3-month period no referrals have come from the criminal justice system may want to undertake some special efforts to ensure that professionals in that system know whom to contact and how to make a referral. Similarly, the program manager might be concerned if physicians, clergy, or other human service agencies are not referring to the program—or are not referring in the proportion expected.

A more refined level of program monitoring would examine the number of clients who drop out of the program. How many clients complete only one or two sessions? What proportion drop out by the third session? How many clients notify the program staff that they will not be returning? It is also important to know such things as how long it takes for clients to receive service from the time of their initial contact or application. Managers should know how many clients are on their waiting lists so that scheduling and programming can be planned accordingly.

Besides using program monitoring to determine whether the obvious segments of the population are being served (older adults, low-income persons, minorities), program managers can determine whether clients from remote geographical areas and those with special needs (such as persons with mental retardation or physical disabilities) are being served. Additionally, program monitoring can inform as to whether there is an increase of clients with certain types of problems or diagnoses. A significant increase in the number of clients with special problems may necessitate training staff or require some modification in the program. Examining client data by area of residence may indicate the need for a new satellite office.

As management’s use of program information increases, additional items, such as productivity of individual members of the staff, can be added to the items being monitored. As program managers make greater use of program information, often it becomes necessary to develop (or purchase) more sophisticated ways of managing data.

Management information systems depend on documents or files which are increasing electronic files in a computer system. They record transactions such as the specific service a client received, the number of hours of service received or service units provided, the staff member involved, the location, the date, and so on. This information can usually be linked with other data such as the client’s age, gender, street address, and so forth, contained in the initial application or “face sheet.” It could be very useful to see what data items are being kept in the computer files in order to monitor clients associated with particular programs.  Figure 5.4  is a simple illustration of some of the information that may be available to you in terms of daily activities. Note that this information can be used for billing purposes as well as for understanding staff and program productivity.

FIGURE 5.4: PAPER VERSION OF SERVICE DELIVERY DATA ITEMS

Computerized information systems allow for the most sophisticated monitoring of service utilization because of the ease with which the computer can process large quantities of data. Examples of program monitoring questions that a management information system could answer are:

· 1. How many service units, on average, did clients with the disorder receive? (Or, what was the average length of stay for patients with bipolar disorder?)

· 2. Which unit produced the most counseling during the last quarter? (Alternatively, which worker was the least productive last pay period?)

· 3. What percent of the clients were able to pay the full fee?

· 4. Of those clients referred for services last month, how many were referred by the criminal justice system?

· 5. How many cancellations (or no-shows) were there last month? What were the characteristics of those who canceled and gave no notification? (How many were single mothers with small children or unemployed persons with no transportation?)

Outside evaluators need to review any routine monthly, quarterly, or yearly reports that are produced by the agency. These will likely provide a good overview of key variables such as the number of clients who applied for and received service, and so forth.

Another approach is to systematically collect all of the forms used by the agency. Start at the intake department as if you were a client and gather all of the forms that initially must be completed. Then, move on to other departments where clients might be referred to receive service. Finally, ask for copies of forms that are used when clients terminate and cases are closed. All of the forms collectively compose the agency’s management information system (MIS). What you are likely to find is that not every form is entered into the MIS—many forms are filed or kept even though no one accesses the data from them. These forms may yield rich data for analysis or, on the other hand, they may contain repetitious information already accessible from other forms.

It should be noted that program monitoring data, while useful for some management purposes, does not usually inform as to the quality of care provided to the various groups of clients using the agency’s services. Although you may be pleased with a program because poor or minority clients were well represented in the clientele, this does not guarantee that the services they received actually helped them. The program could be serving a large number of persons inadequately or inappropriately. By examining only the characteristics of those clients being served, you still have very little idea about how “good” the program is. When you want to know if the clients are better off as a result of being served by the program, then you need to shift from program monitoring to program evaluation models. However, before considering program outcome evaluation, you might want to examine the quality assurance system.

QUALITY ASSURANCE

Quality assurance is another basic form of evaluation that usually involves determining compliance with some set of standards. The term is often associated with ongoing reviews of medical or clinical care records, although more may be involved than this. Quality assurance aims to identify and correct deficiencies occurring in the process of providing care to consumers of services. A study at the Baylor University Medical Center is a good example of the importance of paying attention to what we provide to our clients. The researchers found a medication error rate of 111.4 per 1,000 orders (Seeley, Nicewander, Page, & Dysert, 2004). Most social workers are employed in programs and organizations where there is much less monitoring than goes on in a large medical center. If these kinds of mistakes can be made in a hospital—and a good one at that—then one can’t help but wonder about the extent of mistakes that are made in social service programs. Are the interventions provided appropriate? Is the treatment plan current and related to the presenting problem? Is the frequency of intervention sufficient? Were the client’s problems correctly assessed? Were referrals made when necessary?

Accreditation standards exist for many, if not most, human services. By way of example, since April 1987, all psychiatric hospitals in the United States must meet the standards developed by the Joint Commission of Accreditation of Health Organizations (JCAHO, 1988, 1990, 1991). Some residential treatment agencies also seek to meet JCAHO’s standards as well as those of other accrediting organizations such as the Council on Accreditation mentioned earlier in this chapter.

Typically, quality assurance standards require agencies to document for all their clients such information as:

· Presenting problem/diagnosis

Treatment plan

Frequency and length of treatment episodes

Service modality/provider

Drug prescriptions

Discharge plans

Staff qualifications

From this information, reviewers can determine if admissions were appropriate, treatment was consistent with generally accepted practice, the least expensive alternative resources were used, and continuity of care and reasonable treatment follow-up were provided. Known in some agencies as “utilization review,” these efforts are increasingly concerned with such issues as length of stay (in treatment) and ensuring that resource usage is fiscally justified.

Unfortunately, many social and human service organizations have considered this “medical model” of quality assurance to be synonymous with program evaluation. All too often, quality assurance has been conceptualized as and limited to checking whether a sampling of reviewed cases was essentially in conformity with accepted standards of care. Such efforts, however, do not indicate the extent to which a program is successful—whether clients improve as a result of intervention—or whether a program is worth funding again next year. Quality assurance efforts, by and large, focus almost exclusively on the process of treatment rather than on treatment outcome.

Because of the confusion that exists, it is necessary to briefly highlight the differences between quality assurance and program outcome evaluation as presented in the remainder of this book. First, quality assurance efforts often stem from legislative mandate. In 1972, amendments to the Social Security Act (PL 92-603) established Professional Standards Review Organizations (PSROs). The intent of this legislation was to establish peer review systems to ensure that federal and state expenditures of Medicare, Medicaid, and the Maternal and Child Health Programs were spent on “medically necessary and high quality care” (Tash & Stahler, 1984). Second, clinicians tend to be involved in quality assurance efforts (peer reviews) and may be one group of many stakeholders contacted in a program evaluation. Third, in quality assurance, recommendations are customarily relayed back to clinicians in order to improve the record-keeping process, whereas evaluation findings may or may not be given as feedback to the clinical staff and are more often used at the administrative level. Fourth, quality assurance often relies on the expert opinion of peer reviewers and consensus that a sample of records met expected standards. Program evaluation methodologies tend to rely much less on peer review and more on quantitative data, research designs, the formal testing of hypotheses, and statistical analysis (Tash, Stahler, & Rappaport, 1982).

Having gone to all this trouble to convince you that quality assurance is not the same as program outcome evaluation, we would not want to leave you with the impression that quality assurance is a waste of time—it is not. It provides a degree of consistency and uniformity by promoting adherence to clinical guidelines. When quality assurance standards are well established and interventions relatively dependable and constant, the program evaluator has an easier time understanding the treatment fidelity and positive effects of an intervention.

Even if quality assurance is not required within an agency, the conscientious manager may want to implement these activities in some form. Coulton (1987) has noted, “A successful organization continually looks for, finds, and solves problems. In this context, quality assurance—with its cycle of monitoring, in-depth problem analysis, and corrective action—serves as a self-correcting function within an organization” (p. 443).

Some human service professionals resent the amount of time it takes to document what they have already done. Especially when caseloads are large, paperwork is an anathema. Workers may feel that time spent on paperwork is time taken away from needy clients. However, viewed from a manager’s perspective, this “paperwork” is needed for a variety of reasons:

· 1. To protect clients from unethical or inappropriate treatment. Consider these “horror stories” of two consulting physicians whose contracts were canceled by different mental health agencies. At one center, the utilization review picked up a pattern of a physician overmedicating clients and using strange combinations of drugs that did not seem to provide any benefit. Staff at the other center found that their consulting psychiatrist was diagnosing an inordinate number of clients as multiple personalities. In a more positive sense, quality assurance activities help to demonstrate that the organization cares about the services provided.

· 2. To protect staff from charges of inappropriate treatment or incompetence. (In this litigious society, the documentation of services rendered is some protection against unfair or untruthful claims.) Quality assurance data also can be used to identify reasons for patient dissatisfaction with services. Satisfied clients always help to improve the marketability of programs and services.

· 3. To recover reimbursements from insurance companies and other third parties. (As noted earlier, quality assurance activities are required by Medicare, Medicaid, and other third-party payers.)

· 4. To better plan for effective and efficient utilization of staff and agency resources. (Having such information as the average length of stay or the numbers of patients with certain diagnoses can help program managers evaluate special and unmet needs as well as better supervise staff whose cases exceed the average.)

Quality assurance efforts in an agency may be elaborate or fairly elementary.  Figure 5.5  provides a short checklist employed by the quality assurance office of a state social services department. Routinely, this office randomly selects cases from each of its district offices, reviews them, and where necessary asks for the records to be amended or corrected. Cases with deficiencies are clearly noted and returned to the caseworker’s supervisor. Supervisors must then see that problems are corrected and report to the quality assurance office.

FIGURE 5.5: CASEWORK EVALUATION FORM

Another important advantage of quality assurance is accountability.

TOTAL QUALITY MANAGEMENT

Green and Attkisson noted as early as 1984 that while program evaluation and quality assurance were distinctly different approaches, they were converging. They observed that quality assurance had embraced the criteria of efficiency or cost effectiveness of services (cost containment in the medical field) and adequacy of services relative to the needs present in the population. At the same time, program evaluators were becoming more comfortable with incorporating features of quality methods into their evaluations.

In the last few years there has been a resurgence of interest in the quality of goods and services available to American consumers. Japanese industries, particularly automobile manufacturers, have popularized and demonstrated enormous success with a concept known as “total quality management.”

Total quality management is based on a series of principles developed by William Deming. One major aim is to reduce variation from every process so that greater consistency of effort is obtained. Quality is defined by the customer, and improvement focuses on what customers want and need. However, total quality management is not a one-shot effort and must come from top management’s commitment to improvement. It often requires a change in thinking that encourages participation in the planning process by all staff members. Deming suggests that employees must be given freedom to dissent and stresses the importance of eliminating all barriers to communication. The organization must create an environment that fosters disclosure without penalty by all members of an organization.

Sometimes known as continuous quality improvement, total quality management emphasizes client satisfaction surveys and uses feedback to make refinements. Although we will talk much more about such methods in a later chapter, it should be pointed out that much of the material we have covered so far (needs assessment, mission statements, goals and objectives, process evaluation, and program monitoring) is consistent with a total quality management orientation.

Martin (1993) has identified 14 different dimensions associated with program quality (see  Box 5.6 ). Although it probably is impossible for a program manager to target all 14 simultaneously for improvement, it does make sense for management to consider how a specific program may fare on selected criteria. For instance, under accessibility: Is the agency open at times convenient for clients who are employed (e.g., evening or Saturday hours)? Is the agency located on a bus line or close to other public transportation? Is it handicapped accessible? Once it is felt that a program is sufficiently accessible, management might then want to target another dimension, such as performance (effectiveness). These dimensions will have to be operationally defined by each program. What is acceptable performance by one might not be as satisfactory for another.

BOX 5.6: ASPECTS OF QUALITY PROGRAMMING

· • Accessibility (few problems are encountered in gaining entrance to the program)

· • Assurance (staff are facilitative)

· • Communication (clientele and potential clientele are kept informed about the program)

· • Competence (staff are skilled and knowledgeable)

· • Conformity (meets generally accepted standards for best practice)

· • Deficiency (not lacking anything needed to make it a quality program)

· • Durability (the impact or change produced by the program lasts)

· • Empathy (therapists and staff are understanding)

· • Humaneness (clients are treated with respect and dignity)

· • Performance (interventions work as intended)

· • Reliability (the interventions are consistent and predictable)

· • Responsiveness (the time from request of assistance to delivery of program is short)

· • Security (there is no danger associated with accessing or receiving the intervention)

· • Tangibles (the physical environment is acceptable—for example, the facility is clean and the furniture is not worn out)

Source: Adapted from Martin (1993).

Most program quality control procedures are not about life or death issues, but if the overall goal or mission of the program is to help people or to contribute to the quality of life for clients, quality assurance procedures can help to ensure that.

It is beyond the scope of this book to outline a quality assurance program to fit your agency. Rather, the purpose of this discussion has been to help you understand how quality assurance can be used for program improvement (e.g., to identify employees who tend to make inappropriate diagnoses or treatment plans, or whose interventions are not consistent with expectations or accepted practice; to identify the need for inservice or continuing education; and to provide other useful data for management decisions).

CHAPTER RECAP

Formative and process evaluation have in common a focus on improving programs. Because of their shared concern, it should be an obligation of every practitioner and every manager to learn more about and support these qualitative forms of evaluation within their agencies or practices. To improve a program’s quality, we cannot focus on only one portion of the process, such as the product at exit. We must examine every aspect of the program—perhaps beginning with goals for the program and the mix of appropriate and inappropriate admissions.

Depending on the age, complexity, and sophistication of the agency whose program you have been asked to evaluate, you may not find mission statements or statements of program goals and objectives. In fact, your first act as an evaluator may be to assist the agency to develop mission statements and program goals. You may find yourself writing goals and objectives simply because that has never been done and no one else has any experience with writing them.

Keep in mind that a program can have more than one goal, and each goal can have multiple objectives. For instance, I once heard of a mental health agency that had purchased a fast-food restaurant. This purchase enabled the agency to employ their clients with chronic mental illness while providing them with necessary training and income to become employable in a competitive job market. The restaurant also brought in needed operational income to the agency. Each of these goals would be evaluated independently.

Patton (1982) has made several astute observations about management information systems. He noted that “if there is nothing you are trying to find out, there is nothing you will find out” (p. 229). He suggested that a management information system is not an “endpoint” but a beginning point for raising issues for additional study. Management information systems only provide data—they do not make decisions. An evaluation does not occur until someone uses data to answer questions that have arisen.

In summary, formative evaluation is used in the beginning stages of a program to help form the program. The specific goals of process evaluation, however, are less clear. They may include one or some combination of all three main goals of process evaluation—describing a program, program monitoring, and quality assurance.

Finally, much of what Deming has taught about how organizations continuously improve can be summarized in four words: Plan, Do, Check, Act. In the PDCA or Deming cycle, Plan means study a program or process by collecting data and deciding what would improve it. In the Do step, the plan is implemented (sometimes on a small scale). In the third phase (Check), staff check the results obtained so that they can make the necessary changes (Act) in the program or process. Whenever we are in a position to provide formative or process evaluation, program monitoring, or quality assurance, we would do well to remember these four simple guides.

QUESTIONS FOR CLASS DISCUSSION

· 1. What is wrong with the following objectives?

· a. To improve statewide planning capacity and capability

· b. To maximize collections from first- and third-party payers

· c. To improve the skills of current staff through appropriate inservice training

· d. To improve staff–patient ratios in state psychiatric hospitals

· e. To participate more actively in economic development activities

· 2. Rewrite the following objectives to improve them.

· a. The Free Clinic will facilitate early initiation of prenatal care by maintaining relations with local physicians and other agencies to facilitate referrals to the clinic.

· b. The Free Clinic will distribute brochures and posters describing the need for early prenatal care and the location of these services.

· c. For high-risk patients, the Free Clinic will perform follow-up counseling as needed.

· d. The chronically mentally ill population will be served by a new “clubhouse” aftercare program to reduce inpatient hospitalizations.

· e. By the end of 8 weeks, all group members will have developed tools to help with panic attacks and flashbacks.

· 3. Discuss how a board of directors would know when a program is in need of a formative evaluation.

· 4. Tell what you know about the various ways in which social and human service agencies in your community conduct quality assurance and program monitoring activities.

· 5. Discuss the extent that social and human service agencies with which your class is familiar utilize computerized management information systems. What are their advantages and disadvantages?

· 6. Briefly describe a local social or human service program to the class. Discuss information that would be useful for program monitoring.

· 7. Refer to  Table 5.1 . What possible explanations could there be for certain populations utilizing services less than might be expected? Could it be argued that some populations have a greater need for services than their proportion in a community’s total population?

· 8. Discuss your experience with quality assurance programs. Viewed from a management perspective, what do you believe to be the benefits of quality assurance?

MINI-PROJECTS: EXPERIENCING EVALUATION FIRSTHAND

· 1. Choose a human service program with which you are familiar and then do the following:

· a. Briefly describe the program.

· b. Write at least one program goal.

· c. Write three specific program objectives.

· 2. Write a mission statement for a fictitious agency of your choosing.

· 3. Imagine that a friend asks you to conduct a formative evaluation of the agency where you now work or intern as a practicum student. What sort of recommendations would you expect? List at least six realistic recommendations that could apply to this agency.

· 4. Outline a strategy you would use to conduct a process evaluation of the same program for which you conducted the formative evaluation in exercise 3, a year after program inception.

· 5. Briefly describe the quality assurance procedures of a social or human service agency with which you are familiar. Draft a short paper outlining how these procedures could be improved.

· 6. Obtain a monthly, quarterly, or yearly report from a social or human services agency. What additional information would be useful if you were a program monitor for that program? What information is missing and should be incorporated in future reports? Draft a set of recommendations based on your reading of the reports.

· 7. Read one of the articles from the References and Resources section and write a short reaction paper.

REFERENCES AND RESOURCES

Below, P. J., Morrisey, G. L., & Acomb, B. L. (1987). The executive guide to strategic planning. San Francisco: Jossey-Bass.

Chelimsky, E. (1985). Program evaluation: Patterns and directions. Washington, DC: American Society for Public Administration.

Commission on Accreditation of Rehabilitation Facilities. (1990). Standards manual for organizations serving people with disabilities. Tucson, AZ: Author.

Coulton, C. J. (1987). Quality assurance. In S. M. Rosen, D. Fanshel, and M. E. Lutz (Eds.), Encyclopedia of social work. Silver Spring, MD: National Association of Social Workers.

Crawford, M. A., Woodby, L. L., Russell, T. V., & Windsor, R. A. (2005). Using formative evaluation to improve a smoking cessation intervention for pregnant women. Health Communications, 17(3), 265–281.

Drucker, P. (1980). The deadly sins in public administration. Public Administration Review, 40(2), 103–106.

Frechtling, J. A. (2007). Logic modeling methods in program evaluation. San Francisco: Jossey-Bass.

Green, R. S., & Attkisson, C. C. (1984). Quality assurance and program evaluation: Similarities and differences. American Behavioral Scientist, 27(5), 552–582.

Harachi, T., Abbott, R., Catalano, R., Haggerty, K., & Fleming, C. (1999). Opening the black box: Using process evaluation measures to assess implementation and theory building. American Journal of Community Psychology, 27(5), 711–731.

Joint Commission on Accreditation of Healthcare Organizations (JCAHO). (1988). The Joint Commission guide to quality assurance. Chicago: Author.

Joint Commission on Accreditation of Healthcare Organizations (JCAHO). (1990). Accreditation manual for hospitals. Chicago: Author.

Joint Commission on Accreditation of Healthcare Organizations (JCAHO). (1991). An introduction to quality improvement in health care: The transition from quality assurance to continuous quality improvement. Chicago: Author.

Krisberg, B. (1980). Utility of process evaluation: Crime and delinquency programs. In M. Klein & K. Teilmann (Eds.), Handbook of criminal justice evaluation (pp. 217–236). Beverly Hills, CA: Sage.

Logan, T., Hoyt, W., McCollister, K., French, M., Leukefeld, C., & Minton, L. (2004). Economic Evaluation of Drug Court: Methodology, Results, and Policy Implications. Evaluation and Program Planning, 27(4), 381–396.

Logan, T., Williams, K., Leukefeld, C., & Minton, L. (2000). A Process Evaluation of a Drug Court: Methodology and Results. International Journal of Offender Therapy and Comparative Criminology, 44(3), 369–394.

Martin, L. (1993). Total quality management in human service organizations. Newbury Park, CA: Sage.

Mawhinney, T. C. (1992). Total quality management and organizational behavior management: An integration for continual improvement. Journal of Applied Behavior Analysis, 25(3), 525–543.

Melnyk, B. M., & Fineout-Overholt, E. (2005). Evidence-based practice in nursing and healthcare: A guide to best practice. Philadelphia, PA: Lippincott, Williams & Wilkins.

Millar, A., Simeone, R. S., & Carnevale, J. T. (2001). Logic models: A systems tool for performance management. Evaluation and Program Planning, 24, 73–81.

Moskowitz, J. M. (1989). Preliminary guidelines for reporting outcome evaluation studies of health promotion and disease prevention programs. Evaluating health prevention programs (New Directions for Program Evaluation, No. 43). San Francisco: Jossey–Bass.

Orwin, R. (2000). Assessing program fidelity in substance abuse health services research. Addiction, 95, Suppl. 3, S309–S327.

Patton, M. Q. (1982). Practical evaluation. Beverly Hills, CA: Sage.

Savaya, R., & Waysman, M. (2005). The logic model: A tool for incorporating theory in development and evaluation of programs. Administration in Social Work, 29(2), 85–103.

Scarpitti, F., Inciardi, J., & Pottieger, A. (1993). Process evaluation techniques for corrections–based drug treatment programs. Journal of Offender Rehabilitation, 19(3/4), 71–79.

Seeley, C. E., Nicewander, D., Page, R., & Dysert, P. A. (2004). A baseline study of medication error rates at Baylor University Medical Center in preparation of a computerized physician order entry system. BUMC Proceedings, 17, 357–361.

Sugarman, B. (1988). The well-managed human service organization: Criteria for a management audit. Administration in Social Work, 12(4), 17–27.

Tash, W. R., & Stahler, G. J. (1984). Current status of quality assurance in mental health. American Behavioral Scientist, 27(5), 608–630.

Tash, W. R., Stahler, G. J., & Rappaport, H. (1982). Evaluating quality assurance programs. In G. J. Stahler & W. R. Tash (Eds.), Innovative approaches to mental health evaluation. New York: Academic Press.

Torghele, K., Buyum, A., Dubruiel, N., Augustine, J., Houlihan, C., Alperin, M., et al. (2007). Logic model use in developing a survey instrument in program evaluation: Emergency preparedness summits for schools of nursing in Georgia. Public Health Nursing, 24(5), 472–479.

United Way of America. (1996). Measuring program outcomes: A practical approach. Alexandria, VA.

U.S. Department of Health and Human Services, Administration for Children and Families, Office of Community Services. Evaluation guidebook: Demonstration partnership program projects. (1992). Washington, DC: U.S. Government Printing Office.

Veney, J. E., & Kaluzny, A. D. (1984). Evaluation and decision making for health services programs. Englewood Cliffs, NJ: Prentice-Hall.