U9A1-28 - Final Paper Submission - final project proposal. "YOU MUST FOLLOW ALL INSTRUCTIONS AS OUTLINED" DO NOT DEIVATE FROM THE INSTRUCTIONS GIVEN...READ EVERYTHING.

6/2/2018 Bookshelf Online: Designing and Managing Programs

https://online.vitalsource.com/#/books/9781483388328/cfi/6/48!/4/2/4/2@0:0 1/21

Chapter 9 Designing Effectiveness-Based Information Systems

Chapter Overview The purpose of this chapter is to explain:

The ways in which computerized data can be used to measure performance The relationship between program evaluation and data collection How a data collection system can be designed to ensure that necessary information will be provided to stakeholders

The following topics are covered in this chapter:

Documentation Practices in Human Services Designing a Data Collection System for a Program

Step 1: Consider the Evaluation Context of Data Collection and Aggregation Step 2: Identify the Programmatic Questions to Be Answered Step 3: Identify Data Elements Step 4: Develop a Strategy for Analysis Step 5: Prepare Format for Monthly Reports

Summary Case Example Review Questions

Documentation Practices in Human Services In Chapter 8 we focused on the data elements that need to be defined within programs in order to establish a framework for measuring client outcomes and program performance. This chapter explains how to organize those elements into a system that will provide data and information on the effectiveness of services both at the individual client level as well as at the program level.

Very few human service agencies and programs in the early 21st century are left untouched by expectations of data collection for the purpose of performance measurement, yet some research suggests that management information is underutilized in human service agencies (Carrilio, 2005; Hatry, 2011).

Because of funding source requirements and because of the advances in information technology, data collection and data entry procedures have changed from handwritten case recording to computerized records that require case-related information to be translated into codes. Business and industry, from the very largest international corporations, have found new uses for data collection and aggregation. They are able to track sales, customer preferences, effectiveness of advertising, and many other business-related techniques. In the same way, human service agencies can track the types of clients receiving services, elements of programs of service, their effectiveness with different types of clients, and many other program components.

Large organizations today that can afford more sophisticated and elaborate systems are utilizing what are referred to as dashboards and scoreboards. Dashboards use a series of metrics to track and analyze organizational activity (e.g., sales, services) while they are in process in the interest of making corrections when the data reveal they are not on target. Dashboards may apply to a whole organization, a specific department or program, or a specific function, and they provide the first glimpse into what is working as expected and what may need adjustments. Figure 9.1 provides an example of how a dashboard might be displayed to track the effectiveness of efforts to increase self-confidence and self-esteem among clients in a domestic violence shelter.

Scorecards tend to focus more on longer range performance in relation to program goals. Key metrics are monitored to determine progress toward achievement of stated goals and objectives. Scorecards tend to make

6/2/2018 Bookshelf Online: Designing and Managing Programs

https://online.vitalsource.com/#/books/9781483388328/cfi/6/48!/4/2/4/2@0:0 2/21

use of dashboard data by incorporating these data into their long-range planning reports. Table 9.1 is an example of how a scorecard might be used to display progress toward achievement of objectives in some of the domestic violence services we have described.

Most organizations that make use of dashboards and scorecards tend to blend them so that one feeds into the other to provide the data and information needed at any given time. A rich array of examples and software for dashboards and scoreboards is available online for interested users. However, a variety of computerized data collection and entry options come with most software packages that allow human service programs to computerize and, in some cases, integrate their client, program, financial, and performance files. Information systems can be built on a basic spreadsheet program such as Microsoft Excel or a database program like Microsoft Access. Some states use templates and data entry screens where data are entered directly into the database. These screens can be constructed in a way that if the wrong type of response is entered, the screen will not accept the data and will not process the transaction until every data field is complete and determined to be accurate.

Figure 9.1 Dashboard Example Using Self-Esteem Scores by Ethnicity Over the First Quarter

Eligibility can be determined by completing a template (a series of data entry screens) on personal computers. When completed, the social worker logs onto the Internet, connects with the state database, and uploads the completed application directly to the department database. The principle is “enter once, use multiple times.”

Regardless of the type of information system utilized, a program manager is confronted with three issues:

1. What questions do I need the system to answer? 2. What data elements must be included in the system to answer the questions? 3. What types of routine reports do I want the system to generate?

6/2/2018 Bookshelf Online: Designing and Managing Programs

https://online.vitalsource.com/#/books/9781483388328/cfi/6/48!/4/2/4/2@0:0 3/21

The role of the program planner is to ensure that these three questions are answered satisfactorily so that the information system will provide the necessary data to manage, monitor, and evaluate the program. The actual work of setting up the information system to address these three needs is best left to IT professionals.

Effectiveness-based program planning requires that data be generated that will support decision making based on what is in the best interests of clients—on information about what services are most effective with what types of clients and what types of problems. The remainder of this chapter focuses on understanding conceptually how the elements of the data collection and reporting system are created, aggregated, and reported. Our assumption is that issues related to technology will be addressed by IT professionals. It should further be noted that what we are proposing here is the development of a data collection system designed to measure performance, to monitor, and to evaluate a program. In dealing with the more complex issues surrounding the development of an information technology application for an entire agency, the reader is referred to the work of Schoech (1999, 2000).

Designing a Data Collection System for a Program A major assumption of effectiveness-based program planning is that each activity is shaped by the completion of previous activities and, in turn, each activity will shape future activities. The process, of course, is not actually linear. Earlier decisions may have to be modified as later activities are addressed. Still, the overall process needs to be guided by commitment to internal consistency between the parts. With these considerations in mind, the design of a data collection system is built on the processes described in the previous chapters, with the logic model providing the basis for ensuring internal integrity. The development of goals and objectives is an articulation of the program hypothesis, simply using different terminology. The identification and definition of inputs, throughputs, outputs, and outcomes is a reformulation of the goals and objectives section, and the development of a data collection system builds on all of these. Table 9.2 illustrates some of these relationships.

Each system component (input, throughput, output, and outcome) has implications for data collection, and we will examine them later in this chapter. But before we deal with data elements, it is important to ensure that thought has been given to the ultimate purposes for which the system is being created. To accomplish this we propose establishing a context in which planners first think through the evaluation function (outcome or performance measurement data needs) of program planning and, second, assess the compatibility of evaluation data with the problem analysis completed at the beginning of the program-planning process.

6/2/2018 Bookshelf Online: Designing and Managing Programs

https://online.vitalsource.com/#/books/9781483388328/cfi/6/48!/4/2/4/2@0:0 4/21

It may seem counterintuitive to “begin at the end,” so to speak, but these steps will help to make the data collection more efficient and will ensure a more useful database once the system is up and running. The approach proposed here is more likely to lead to an understanding of the many ways in which inputs, throughputs, outputs, and outcomes interact with one another to help in solving client and community problems.

The steps involved in designing and implementing such a system are these:

1. Consider the evaluation context of data collection and aggregation. Why is the evaluation being planned? Who will be using the evaluation? What do all potential users expect to learn from it?

2. Identify the programmatic questions to be answered. Does the program hypothesis still appear to be correct and logical? Will the system reveal whether clients are improving? If so, will we be able to explain why? If not, will we be able to explain why not?

3. Identify data elements. What is the absolute minimum we must know about inputs, throughputs, outputs, and outcomes? What do we need to know about client demographics to be able to answer questions about various populations? What metrics will we use to track provision of services, client participation, and results?

4. Develop a strategy for analysis. How can data be displayed so that we can be sure that all users will be able to understand what is happening in the program from their particular perspectives?

5. Prepare format for monthly reports. Who will get monthly reports? What do they need to know?

The details involved in carrying out each of these steps are covered in the following sections.

Step 1: Consider the Evaluation Context of Data Collection and Aggregation

While evaluation design will be addressed in greater detail in Chapters 10 and 11, it is important that the program planner think conceptually about selected evaluation principles before beginning the work of designing the data collection system. When evaluation demands first began to appear in the human services, it was common practice to begin thinking about program evaluation only after services had been provided. This is one of the major reasons early evaluations so often failed to provide the type of information needed for program

6/2/2018 Bookshelf Online: Designing and Managing Programs

https://online.vitalsource.com/#/books/9781483388328/cfi/6/48!/4/2/4/2@0:0 5/21

improvement. We now know that well-planned evaluations begin very early in the program-planning process. We have attempted to reinforce this notion by laying the groundwork from the very first chapter.

As program planners prepare the evaluation strategy, they are attempting to accomplish two purposes: (1) to assess the relative success of programs in meeting their stated objectives and (2) to identify potential areas for program improvement. Measures of program activity and outcome can be compared with the findings from other, similar programs or simply mined for data that appear to indicate potential or actual problems. Here is the central question: What happened as a result of this particular intervention strategy that would not have happened in its absence? The intent of program evaluation is to assist managers in solving problems through the determination of what program strategies work best under what conditions.

There are two categories of evaluation of concern to program planners: formative and summative. These concepts will be addressed briefly here, but will be explained in more detail in later chapters dealing specifically with program evaluation. Formative evaluation is conducted during the actual operation of the program. It attempts to provide information during implementation to help in determining the extent to which the program is being implemented according to the program’s design. Formative evaluation answers questions such as these: Is the program being implemented in accordance with its design? Are the program participants representative of the program’s target population? Are the services provided consistent with those that were promised in the program design? Based on answers to these and other questions, the manager is able to determine whether modifications should be made to program operations even before the program has completed its first year. Formative evaluation focuses on process objectives as discussed in earlier chapters.

Summative evaluation, as the name implies, is carried out at either the end of a program cycle or component (e.g., the budget cycle, the completion of a training program) or at the conclusion of a program. Summative evaluation is designed to provide an assessment of program accomplishments (the relative successes and failures of the program) and focuses on outcome objectives. These two approaches to evaluation are compared briefly in Table 9.3.

Types of Evaluation

Program evaluation may take many forms and may examine a variety of program aspects. The type of evaluation that is undertaken should be based on a realistic assessment of both feasibility and decision makers’ need for information regarding program operations.

Five general types of evaluation can be identified, each capable of providing different kinds of information about program functioning: (1) evaluation of effort, (2) evaluation of cost-efficiency, (3) evaluation of outcome, (4) evaluation of cost-effectiveness, and (5) evaluation of impact (Suchman, 1967). Each category is briefly defined below. This is followed by a more detailed discussion in a later chapter.

Effort evaluation is concerned exclusively with documenting the characteristics of the program participants and the quantity of activity that takes place within a program—how much and what type of service is being provided and to whom. An assessment of effort may be viewed as a reflection of service provision, including both inputs and throughputs. Cost-efficiency evaluation looks at the costs of providing a unit of service. More specifically, it looks at the costs in terms of time, episode, or material units—the costs of the outputs. Outcome evaluation examines the results achieved with clients and seeks to identify the extent to which the program’s outcome objectives have been achieved. Cost-effectiveness evaluation looks at the costs of achieving the results—the cost per successful outcome. Impact evaluation is concerned with the extent to which the community’s needs, as determined in the planning process, have been met by the program.

6/2/2018 Bookshelf Online: Designing and Managing Programs

https://online.vitalsource.com/#/books/9781483388328/cfi/6/48!/4/2/4/2@0:0 6/21

Figure 9.2 illustrates where each of these types of evaluation logically fit within the phases of the logic model.

These five categories, in sum, address all of the information needs a program manager will use to evaluate a program. They can be summarized by the following general questions:

What kinds of clients? (Effort) Experiencing what types of problems? (Effort) Receiving what type and volume of services? (Effort) Get what results? (Outcome and impact) At what costs? (Cost-efficiency and cost-effectiveness)

With these questions as a road map, we are now in a position to build a responsive information system.

Step 2: Identify the Programmatic Questions to Be Answered

An efficient way to generate questions to be answered is to revisit the problem analysis and the working intervention hypothesis. The objective here is to reexamine the reasons for developing the program in the first place and to ensure that it is on track to address the problems it is intended to alleviate or resolve.

For the sake of illustration, we turn once again to the problem of domestic violence (DV). The program hypothesis (made up of the hypothesis of etiology and the working intervention hypothesis) reads as follows:

Because there are women who experience the following:

Low self-esteem and general anxiety disorder Social isolation from friends, family, and community Lack of financial resources to meet their basic needs Lack of basic education and marketable skills to get and hold a steady job with a salary that meets their basic needs

Figure 9.2 The Fit of Five Types of Evaluation With Phases of the Logic Model

the result is women who are vulnerable to becoming victims of domestic violence for the following reasons:

6/2/2018 Bookshelf Online: Designing and Managing Programs

https://online.vitalsource.com/#/books/9781483388328/cfi/6/48!/4/2/4/2@0:0 7/21

A combination of personal, emotional, and psychological problems Isolation from social support systems and networks Dependence on a partner for financial support Severe limitations in their ability to secure and succeed in a steady job with a career path

If the following actions are taken with a population that has been victimized by domestic violence:

provide individual and group counseling to address personal, emotional, and psychological problems; provide case management services to facilitate and support making appropriate community contacts designed to build a sustainable, steady lifestyle following treatment; provide financial planning services designed to manage income and meet basic expenses; provide employment training and placement,

then it is expected that women who participate in the program

will increase their self-esteem and reduce anxiety, will secure housing and child care and will be able to meet other needs necessary to independent living in the community, will be able to manage their finances effectively, and will be employed in a job with a career path and a salary adequate to meet their financial needs.

As we examine this program hypothesis, a number of questions should come to mind.

Input Questions

We have designed a program to deal with a specific problem profile. This particular program is not equipped to deal with certain problems, such as drug addiction, or with a client who is determined to return to her abuser on completion of the program. How can we be sure the participants we recruit will fit the intended profile? Are we collecting data on ethnic and other special populations in this program, and will we be able to answer questions about differences between and among these populations, if any, in efficiency, effectiveness, and quality?

Do members of the at-risk population (victims of domestic violence) reveal the same characteristics as those identified in the literature: namely, low self-esteem, generalized social anxiety, limited independent living skills, limited financial management skills, and limited job skills? If the population recruited to participate in this program does not match this profile, then the relevance of the intervention as designed is questionable. If members of the at-risk population do have some of the same characteristics as those identified in the literature, does it appear that these characteristics have contributed to their becoming victims of domestic violence? These are the feelings and behaviors we are attempting to change or eliminate. We must be certain that if the intervention is successful, it will lead to an independent, violence-free future to test the intervention hypothesis.

Throughput Questions

Are members of the at-risk population participating in the case management, counseling, and training activities on a regular basis as intended? Have ethnic-specific (or other special population) factors been taken into consideration in designing the program and its services?

Output Question

Are program participants completing all the services that are a part of their individual rehabilitation plans (IRPs)?

6/2/2018 Bookshelf Online: Designing and Managing Programs

https://online.vitalsource.com/#/books/9781483388328/cfi/6/48!/4/2/4/2@0:0 8/21

Outcome Questions

Do program participants, after spending time in the planned activities, demonstrate improved self-esteem, lower social anxiety, mastery of independent living skills, mastery of financial management skills, and mastery of job skills? Are results being aggregated by ethnic or other special population groups? These types of questions will prove to be very useful in identifying data elements to be collected as well as in developing a strategy for analysis.

Step 3: Identify Data Elements

Data elements are bits of information about clients, services, and performance that, when aggregated, present a profile of the program in terms of the client population being served, the services being provided, and the results being achieved. Data elements should be organized around program inputs, throughputs, outputs, and outcomes. They can be further organized around the following headings and subheadings. Each of these will be discussed in detail below.

Program Inputs

client demographic and descriptive characteristics client social history data client problem/strength profile material resources facilities equipment staff descriptive characteristics

Program Throughputs

service tasks method of intervention

Program Outputs

intermediate outputs (units of service provided) final outputs (service completions)

Program Outcomes

intermediate client outcomes (change at the point of service completion) final outcomes (change at the designated point of follow-up)

Figure 9.3 illustrates the fit of information system elements to the phases of the logic model.

Figure 9.3 The Fit of Information System Data Elements to Phases of the Logic Model

6/2/2018 Bookshelf Online: Designing and Managing Programs

https://online.vitalsource.com/#/books/9781483388328/cfi/6/48!/4/2/4/2@0:0 9/21

The following sections will lay out, in more detail, what variables should be considered in selecting information system data elements for each of the components of the planning process.

Program Inputs

Client demographic and descriptive characteristics. As discussed in Chapter 8, one of the first data elements to be considered is that of client demographic and descriptive characteristics and social history data. A few of these elements are illustrated in Table 9.4.

The task at this point is not one of developing data collection forms but one of identifying and justifying what data are to be collected. Only when this task is completed can we turn to the task of instrument development.

Considering the questions to be answered in the program to prevent further episodes of domestic violence for clients served, in addition to the above characteristics, we might also want to know something about the domestic partner, housing arrangements, transportation availability, the number of social contacts per week, distance from nuclear family members or other support system, employment status, and other such characteristics. These factors, to a large extent, will help us determine whether we are actually working with the intended population and what factors may play a part in a client’s transition back to the community. For each program, identifying the right demographic variables is a judgment call in the beginning. Adjustments can be made later. It should also be recognized that there are potential risks in identifying too many variables and overloading the system.

Client social history data. Like client demographic data, social history data may help provide clues to factors in clients’ backgrounds that are related to their current situations. It is usually an expectation of caseworkers that they take a social history on their clients at the point of intake so that they can determine patterns of behavior and behavioral changes over time as well as document legal considerations.

To be useful for performance measurement, monitoring, and evaluation purposes, social history data must have a research focus. In other words, there must be a reason, based on the problem analysis, to believe that a particular variable may have some relevance to the program, its clients, and the outcomes achieved.

With the domestic violence program, for example, we know that factors such as low self-esteem and high social anxiety can lead to victimization. We may, therefore, want to explore factors in the client’s history that have diminished her self-confidence. This topic could be explored by asking questions about the number of current friends and associates, about the use of Facebook and other social media to stay in touch with friends, about the

6/2/2018 Bookshelf Online: Designing and Managing Programs

https://online.vitalsource.com/#/books/9781483388328/cfi/6/48!/4/2/4/2@0:0 10/21

nature of relationships, about substance abuse history, and other such factors. Any data to be used for performance measurement, monitoring, or evaluation purposes will have to be quantified and formatted for data entry.

Client problem/strength profile. Clients who come to human service agencies for help typically present a profile that includes both problems and strengths. One may have a steady job but poor financial management skills. Another may have unreliable transportation and a spotty attendance record at work, but good parenting skills. These factors are important to the case management process. In fact, approaches to casework have been built around the concepts of problems and strengths (Reid, 1992; Saleeby, 1992). In the aggregate, these profiles are important to the program planner as well. As discussed in Chapter 7, these become the intermediate outcome objectives the program will attempt to achieve.

In collecting data for performance measurement, it is important to know for later program analysis purposes the number and percentage of clients during the course of a year that are identified as having a particular problem or strength. This information can be drawn from assessment tools completed at the point of intake by the intake and screening person, by the client, by a third party, or by some combination of these. By using a simple scaling device on which the scale ranges from a serious problem at the low end to an area of strength at the high end, a problem and strength profile can be developed for individual clients as well as for all program participants. These assessment tools, either purchased from the author of a standardized scale or developed by staff, can be designed to determine (a) problems or deficits being experienced by clients, (b) level of severity, (c) areas of strength, and (d) degree of strength. A simple 1 to 5 scale like the following is often used:

Many types of measurement scales are available, and most are designed in a way that will yield useful information about deficits and strengths (see, e.g., Fischer & Corcoran, 1994; Hudson, 1982). The following scale might be used with a target group of women who have experienced domestic violence where 1 = severe problem and 5 = major strength:

Some of the questions to be answered by these data might include these:

What problems or deficits are most prevalent? (items most often rated 1 or 2) What strengths are most prevalent? (items most often rated 4 or 5)

6/2/2018 Bookshelf Online: Designing and Managing Programs

https://online.vitalsource.com/#/books/9781483388328/cfi/6/48!/4/2/4/2@0:0 11/21

Do any problems or deficits (e.g., housing, employment) appear to be associated with any demographic or social history factors (e.g., age, education)? Are there any predominant “problem profiles” or problems that tend to cluster together (e.g., income and budgeting with housing and/or transportation)? Are there any common profiles that seem to be associated with abuse? Are there any apparent “strength profiles” that might help in predicting success in overcoming victimization and developing a violence-free lifestyle?

Material resources. When material resources such as cash, food, or clothing are provided for clients, it is important that these items be tracked so that it can be determined what contribution they make to achieving positive outcomes. Some of the items that might be made available to a client on an as-needed basis include the following:

transportation for job interviews or to secure housing, provided by a volunteer articles of clothing for client for job interviews articles of clothing for client’s child or children first and last month’s rent for an apartment

Amount and frequency of receipt of each of these and other such data need to be tracked on a data collection form. Questions to be answered by tracking material resources might include these:

Do resources like clothing and transportation increase the likelihood that clients will have a successful job interview? Does assistance with rent increase stability of housing after clients leave the shelter?

Facilities. In a way, facilities can also be considered resources used for the benefit of clients. Agencies that have more than one branch office may discover that certain locations or settings are more conducive than others to success. Success may be influenced by convenience of location for clients. In the case of residential treatment, it may be valuable to learn whether factors associated with the facility such as rooms, meals, or opportunities for privacy have any influence over whether a client remains for the full duration of treatment. (See Chapter 8 for a related discussion of qualitative dimensions of outputs.) If use of facilities is included in a data collection system, categories might include one or more of the following:

office visit, main office office visit, branch office home visit, client’s home home visit, relative or friend’s home other community facility

Some of the questions to be explored are these:

Is there a relationship between facility location and keeping appointments? Does it appear that certain clients make better progress when there are home visits? Does it appear that there is any relationship between location and client outcomes?

Equipment. The importance of equipment will vary by program. Programs that provide transportation need to have vehicles in good working condition. Some residential treatment centers may make computers available to residents for e-mail and for searching the Internet. Training programs may have the option of using several different software packages, and it is helpful to know which ones achieve the best results with trainees. Providing cell phones to domestic violence victims has made an important contribution to safety and security for some DV programs. In some instances, there may be a correlation between the provision of equipment and social participation. Some of the equipment-related questions to be answered might include these:

Does provision of transportation contribute to keeping appointments? What types of equipment contribute to successful client outcomes?

6/2/2018 Bookshelf Online: Designing and Managing Programs

https://online.vitalsource.com/#/books/9781483388328/cfi/6/48!/4/2/4/2@0:0 12/21

What is the unit cost of equipment currently used by clients? How would costs compare to contracting with another provider for use of this equipment?

Staff descriptive characteristics. Descriptive characteristics about staff are not likely to be entered into an information system for a program, but it may be useful to record them elsewhere (perhaps in a personnel or human resources information system). Typically, in a system designed to record program and service data, there will be some mechanism for connecting clients to workers (e.g., through a worker identification number).

At some point, program evaluators may want to look at successful cases to determine if any particular staff characteristics are correlated with client success. Capturing staff descriptive characteristics will make it possible to identify what factors about staff, if any, make a difference in terms of client outcomes. Keep in mind, we are making a distinction here between criteria for hiring staff (e.g., all case managers will be bilingual) and staff characteristics that may influence outcome (e.g., the level of skill in using a second language by the case manager). It may also be important to record staff demographic information if accrediting organizations need to know about such factors as ethnicity, gender, education, and experience. Some of the staff descriptive characteristics that might be useful include the following:

education license or certification number of years of experience type(s) of professional experience ethnicity gender

Some of the questions to be explored might include these:

How do education and/or experience correlate with client satisfaction and client outcomes? When worker demographics are matched up with client demographics, what are the relationships, if any, to completion of the service plan and to outcomes?

Program Throughputs

Service tasks. The concept of service tasks, introduced in Chapter 8, is used to define the helping process and to break it down into a series of stages or phases. This breakdown is important because, by tying tasks to units of service, the information system can help to identify what tasks were provided in what volume over what period of time. In the absence of service tasks, the service process defaults to just one, simple, undifferentiated task called casework. By defining service tasks, it is possible, for example, to determine how many units of service (either in the aggregate or by a particular worker) are devoted to each task. A program evaluator may learn that the more time spent in intake and screening, the better the eventual outcome of service. Services provided in a program to serve domestic violence victims might include the following:

1. Case management, broken down into the following tasks: intake screening case planning implementation of the case plan completion of the case plan termination follow-up

2. Counseling, broken down into the following tasks: assessment stating the problem analyzing the system setting goals

6/2/2018 Bookshelf Online: Designing and Managing Programs

https://online.vitalsource.com/#/books/9781483388328/cfi/6/48!/4/2/4/2@0:0 13/21

implementing the plan stabilizing the change effort completing the plan termination follow-up

These data elements could help to answer such questions as the following:

How much time is being spent on each service task? Do patterns vary by worker and type of client? What correlations exist between time spent on service tasks and outcomes?

Method of intervention. Defining the method of intervention will help evaluators and program staff learn whether one particular method is more effective than others with certain types of clients. If, for example, a program’s intent is to maintain or improve nutritional levels for seniors, meals can be provided in a congregate setting or can be home delivered. It may be that one method is more effective than the other for certain types of clients. This discovery can lead to adjustments in the method of meal provision prescribed, depending on the likelihood of success, based on findings.

A wealth of information has been developed over the past decade or so about cross-cultural counseling (see, e.g., Gerstein, Heppner, Aegisdottir, Leung, & Norsworthy, 2009; McAuliffe & Associates, 2008; Pedersen, Draguns, Lonner, & Trimble, 2008). Most of these works cover basic issues such as the need for cultural competence and appraisal and assessment across cultural boundaries as well as specific sections on a wide variety of ethnic groups and other special populations. It is at the point of considering the method of intervention that the issue of incorporating what is known about cross-cultural counseling should be explored. It may be that if there are a small number of special groups to be considered, specific concepts and approaches can be factored into the design. At a minimum, however, the methods should be reviewed carefully to ensure that they do not pose barriers to effective counseling with people who may not share important characteristics with counselors.

Methods of counseling might include individual, group, or family, and some may work better than others, depending on the presenting problems. Over the years as research has been conducted on social work practice, certain types of interventions have been found to work better with certain types of clients (see, e.g., Bennett, Stoops, Call, & Flett, 2007; Chen, Jordan, & Thompson, 2006). Tracking method of intervention provides an opportunity to discover whether the method being used currently works best. The program evaluator is interested in any possible relationship between method of intervention and rates of success.

Questions that might be answered by collecting data on method of intervention might include these:

With what methods of intervention are clients most likely to complete the service plan? What methods of intervention lead to the most successful client outcomes with what types of clients? What methods appear to be most effective in cross-cultural counseling?

Program Outputs

Intermediate outputs. Intermediate outputs are defined as units of service and can be measured in any one of three ways: (1) a time unit, (2) an episode or contact unit, or (3) a material unit. Each service included as part of service tasks must define an accompanying unit of service. For counseling, one hour is a commonly used time unit. For information and referral services, one inquiry about service availability might be used as an episode or contact unit. One trip provided to a job interview is an example of a material unit.

The evaluator is interested in the number of units per week/month/year. If a center typically serves a noon meal to 50 seniors per day, that will be defined as 50 units of service per day or 250 units during a normal 5-day week. If a case manager spends one hour per week in a counseling session with each resident in a shelter, the evaluator will be interested in the total number of units (hours) of counseling provided each week/month/year. Units of service provided is a measure of productivity and can be useful in answering such questions as these:

6/2/2018 Bookshelf Online: Designing and Managing Programs

https://online.vitalsource.com/#/books/9781483388328/cfi/6/48!/4/2/4/2@0:0 14/21

What is the mean number and range of units provided by case managers each week? What is the mean number and range of units of each type of service provided to clients each week? Are there correlations between types of units provided and client outcomes?

Final outputs. Final outputs are defined as service completions, and they require that each service provided have a definition of what is meant by “completion.” Usually this means that a client must complete at least a minimum number of sessions. For example, the Safe Haven shelter might define completion of the job-training module as a client’s participation in at least 90% of training sessions, two of which must be the first and last sessions. Detailed attendance records must be kept to make a distinction as to whether the client actually completed the program or whether she should be considered a “dropout” from the program. The following items illustrate the type of data that would be collected to measure service completions for each client and, in the aggregate, for an entire program:

number of training sessions prescribed number of training sessions attended number of counseling sessions prescribed number of counseling sessions attended number of job referrals made number of job interviews completed

This information helps to determine whether clients are actually carrying out the treatment plans prescribed and, if not, to probe for reasons why not. Some of the questions to be answered include these:

What percentage of clients complete the prescribed treatment plan? What factors influence completion and dropout? Are there any significant relationships between completing a plan and positive client outcomes?

Program Outcomes

Intermediate client outcomes. Intermediate client outcomes are defined as those client changes achieved at the point of termination from a program. Client outcomes are calculated by conducting an assessment at the point of intake and producing a score that can be used as a baseline or starting point for measurement purposes. Assessments may be done in terms of numeric counts, standardized measures, level-of-functioning scales or client satisfaction (see Chapter 8).

When standardized measures or level-of-functioning scales are used, intermediate outcomes are calculated by administering the same measurement device at the completion of service as was used at intake (a pre-post assessment). The purpose is to determine whether there has been progress in solving the problems presented at intake, whether they have remained the same, or whether they have gotten worse. Safe Haven case managers and clients would revisit the eight factors assessed at intake and once again rate each on a scale from 1 to 5, with 1 representing a serious problem and 5 representing a strength, assuming that each point on the scale would be operationally defined for each item.

6/2/2018 Bookshelf Online: Designing and Managing Programs

https://online.vitalsource.com/#/books/9781483388328/cfi/6/48!/4/2/4/2@0:0 15/21

These types of measurements emphasize the central issues of effectiveness-based program planning. If all has gone according to plan, the initial assessment will reveal problems and strengths. The treatment plan will focus on dealing with the problems and capitalizing on the strengths. If the treatment plan has been implemented, if the client has completed all the services prescribed, and if the plan has worked, the problem areas should show some improvement. If case managers have collected all needed data during the process, these data can now be aggregated and can begin to inform staff about overall program effectiveness. Questions to be addressed include these:

What types of problems seem most often to be resolved? Which types seem most resistant to change? Are there any patterns in the types of clients who seem to be most successful and least successful with certain problems? How do ethnic and other special populations compare in terms of outcomes achieved?

Final outcomes. Final outcomes are identified and defined by returning to the purpose for which the program was created. The domestic violence program was created and funded at the state level to enable women who have been abused by their partners to become self-sufficient and live violence-free lives, so these are the variables that must be measured. Final outcome measurements are always taken in a follow-up contact with clients, using numerical counts, standardized measures, level-of-functioning scales, or client satisfaction measures.

The period of time that should elapse between the end of the program and the follow-up is a judgment that must be made by those familiar with the program and depends on the results to be achieved. An outcome such as “no further incidents of violence” would probably require follow-up for at least a year, and perhaps longer. If the final outcome required that we measure whether the at-risk population has improved in self-esteem or decreased in social anxiety, a standardized scale would be administered periodically for as long as the client remains in the program, and a follow-up measurement would be taken between 6 months and 1 year after termination if the client is willing to continue to participate. It should be recognized that in some programs, such as long-term care for the elderly, formal, planned termination may never happen. Clients may remain until death. In cases such as these, periodic measurements may be taken after an episode of service (such as a quarterly or annual completion of a treatment plan) to determine whether the effects of treatment are being sustained over time.

Questions to be addressed in analyzing these data would include the following:

What types of clients seem to be having the most success in this program? What are their characteristics? For what types of clients does this program seem to be least effective? What are their characteristics? What services seem to contribute to success? What elements of the program seem to have no impact and should be considered for redesign? What mix of services, given to which types of clients in what volume, has the highest probability of leading to successful outcomes?

6/2/2018 Bookshelf Online: Designing and Managing Programs

https://online.vitalsource.com/#/books/9781483388328/cfi/6/48!/4/2/4/2@0:0 16/21

Step 4: Develop a Strategy for Analysis

The fourth step in designing a data collection system involves examining the data elements identified in Step 3 and determining how variables will be used for performance measurement, monitoring, and program evaluation. As mentioned earlier, some data systems will be structured so that agency-level inputs can be uploaded directly into a larger system to track data needed at the county or state level. Alternatively, funding source requirements may dictate that certain elements be tracked and reported. For example, program planners at the Safe Haven shelter would be expected to collect data as required by the state funding source. In addition, agency-level personnel such as program managers, administrators, board members, and other stakeholders will undoubtedly have a need for information at the agency and/or program level.

When developing a strategy for analysis, it is important to maintain a focus on the program hypothesis as the overall framework so that analysis does not get sidetracked into becoming an exercise in exploring what seem to be interesting issues but are peripheral to the intent of the program. One approach to help keep the data collection system on track is to structure data collection around program objectives, listing each objective and using the following format:

Outcome Objective 1.1: By April 1, 20XX (prior to final outcome), at least 75% of program participants will demonstrate at least a 25% increase in self-esteem scores as measured by the Hudson Self-Esteem Scale. Counselors are responsible for monitoring. Data elements required: client identification number, pretest score, posttest score Data source: counselors’ tracking system on client Data display/analysis: table listing client identification numbers, pretest scores, posttest scores, and percentage difference

This approach helps to keep the focus on objectives and makes clear how data will be used to support the findings in relation to the objective. In addition to the questions generated by the funding source and the program hypothesis, our recurring, compound question will always need to be addressed, as illustrated in Table 9.5.

Each of these subsets of data elements makes up a part of a comprehensive list of all variables that will form the basis for the data collection and performance measurement system. Once all variables to be incorporated into the system have been identified, variables to be aggregated need to be selected. It is important to remember that, while IT professionals are able to develop the necessary computer programs, they will rely on program planners to make clear what data and information they would like to display for performance measurement, monitoring, and evaluation purposes. The aggregation and cross-tabulation of selected data elements begins to reveal relationships between services and client progress that will help answer the questions identified in Step 2 and will shape future program modifications.

Preparing Tables

Tables needed for data analysis are displays of columns and rows that are used as basic documents in compiling information. Where the columns and rows intersect, cells are formed, and the numbers or values calculated (e.g., the number of African American, Asian American, Hispanic, Native American, and White clients for each age grouping) are entered into the cells.

Tables used for data analysis take the data collection and performance measurement system from a simple listing and calculating of discrete elements, like age, education, and income, to the next level of data aggregation. Tables should be designed to answer the questions that were generated in Steps 1, 2, and 3. A spreadsheet program such as Excel may be used to produce tables such as the one illustrated in Table 9.6. For the more advanced systems, many examples and options are available on the Internet using the key words “dashboards” and “scoreboards.”

6/2/2018 Bookshelf Online: Designing and Managing Programs

https://online.vitalsource.com/#/books/9781483388328/cfi/6/48!/4/2/4/2@0:0 17/21

Note: IRP = individual rehabilitation plan; HS = high school.

A table designed to assess program throughputs, outputs, or outcomes might use one or more of the following variables:

Participation

number and percentage of clients who attended at least 80% of the financial management training sessions number and percentage of clients who developed an IRP within 60 days of admission to the program

Knowledge and Skill

number and percentage of clients demonstrating an increase in knowledge and awareness of domestic violence issues number and percentage of clients demonstrating mastery of entry-level skills in the job category for which they were trained

Creating tables for data analysis should be an inclusive process. While program planners may develop a first draft of questions to be asked and tables designed to answer them, review and comment should be solicited from key agency personnel, board members, or others stakeholders. Individuals who are familiar with the problem,

6/2/2018 Bookshelf Online: Designing and Managing Programs

https://online.vitalsource.com/#/books/9781483388328/cfi/6/48!/4/2/4/2@0:0 18/21

the population, and the program from different perspectives (e.g., case managers, supervisors, budget personnel) will invariably identify different sets of questions and create different tables because their interests and need for information differ. Final drafts of tables should be produced in consultation with IT professionals to ensure that they are compatible with the software to be used and will produce the desired reports.

Step 5: Prepare Format for Monthly Reports

The Commonwealth of Australia has made extensive use of performance measurement principles, and based on its experiences to date, its program planners have identified five guiding principles in collecting performance information that may also be useful in tracking program progress:

1. A focus on outcomes—Indicators should reflect whether service objectives have been met. 2. Comprehensiveness—Performance should be assessed against all important objectives. 3. Comparability—Data should be comparable across jurisdictions whenever possible. 4. Progressive data availability—Since progress may vary, data are reported for those jurisdictions that are

prepared to report. 5. Timeliness—Data reported need to be as recent as possible to retain relevance for decision makers.

(Steering Committee for the Review of Government Service Provision, 2006, pp. 1.9–1.10)

The Australian experience has demonstrated that it is necessary to make the system as accurate and as timely as possible while recognizing that there will be imperfections during developmental phases. If all of the important cross-tabulations have been identified in the previous step, the tables produced should be able to provide information for monthly tracking of progress. Data displays can include cross-sectional analysis, time-series comparisons, and comparisons with other data units (Netting, Kettner, & McMurtry, in press). Cross-sectional analysis examines the same population from a number of different perspectives at a particular point in time. Table 9.7 illustrates a cross-sectional analysis.

A time-series comparison illustrates repeated observations over time and displays trends. Time-series comparisons are especially important in monthly reporting in order to track progress toward achievement of outcome and process objectives. Figure 9.4 illustrates a time-series comparison.

Comparison to other data units depicts data from one program side by side with a similar program in another jurisdiction. Some organizations use a technique called benchmarking, which involves a search of comparable programs for best practices. If data and information are available from these agencies, they can be used for comparative purposes. When statewide programs are delivered by a number of agencies, comparisons can be made across agencies to determine how well each agency is performing in relation to established milestones. Table 9.8 illustrates comparison to other data units.

Note: DV = domestic violence.

Figure 9.4 Time–Series Comparison of Completions in Job Training

6/2/2018 Bookshelf Online: Designing and Managing Programs

https://online.vitalsource.com/#/books/9781483388328/cfi/6/48!/4/2/4/2@0:0 19/21

Rapp and Poertner (1992) propose five principles for report design. They suggest that reports (1) provide standards for comparison; (2) be kept simple and not attempt to provide too much information; (3) be amply illustrated with charts and graphs; (4) include labels on graphs and charts in simple, nontechnical language; and (5) present data aggregation at a level that matches the recipient’s place in the organization.

Chapters 10 through 13 provide the tools needed to complete the analysis of the program from a variety of perspectives and present approaches to calculating the cost and value of the intervention.

Summary After all the careful planning and analysis of problem, need, population, community and other factors that go into program planning, the ability to demonstrate effectiveness all comes down to data collection. Program planners should always remember that they can’t analyze data they wish they had collected. Only when there has been a careful thinking through of data needs and development of a framework for answering the important questions about the program is it possible to conclude that stakeholders will know everything they need to know about the program and its ability to bring about the positive changes expected. We have tried to provide an orderly, systematic method for defining data elements to be collected and analyzed in order to avoid a “hit or miss” approach to brainstorming. While it is true that the list of potential data elements is exhaustive, it is important to remember that not every item listed in this chapter is important for every program. As program planners review the input, throughput, output, and outcome elements needed for their particular programs, decisions will have to be made about the minimum needed to evaluate the program and what the agency can afford. These topics are covered in the next four chapters.

Case Example

6/2/2018 Bookshelf Online: Designing and Managing Programs

https://online.vitalsource.com/#/books/9781483388328/cfi/6/48!/4/2/4/2@0:0 20/21

Gateway Behavioral Services is an agency that has provided services to substance abusers over the past 18 years. In its early years its clientele were mostly White, but increasingly there has been recognition that the agency needs to become more effective in serving the African American community. A task force was created consisting of two African American counselors, a male and a female, and a research assistant. The task force proposed the following: (1) create the position of outreach worker to deal with barriers to access; (2) identify current literature to be added to the libraries of all counselors, with the understanding that they would periodically be asked to review articles and present summaries to staff; (3) create assessment and evaluation guidelines designed to explore the client’s world view and racial identity; and (4) develop a model for individualizing intervention strategies based on each individual client’s profile. The agency implemented the program and, after 6 months, prepared to conduct a formative evaluation.

Review Questions 1. What kinds of questions would the staff need to answer for formative evaluation after 6 months of

operation? How might these questions be different from a summative evaluation conducted at the end of the first year?

2. How would you approach evaluation of effectiveness in dealing with barriers to access in the outreach program?

3. In assessing a client’s problem and strength profile, what kinds of values questions and racial identity questions might be explored that would be different from the questions asked of non–African American clients?

4. Assuming that one of the major changes in designing throughputs is that client and counselor will spend a significant amount of time establishing effective communication and exploring client perspectives on his or her current situation, how might you approach defining service tasks in this program?

5. If the ultimate outcome is for clients to become (ideally) free of dependency or (secondarily) free enough of dependency to hold a job and fulfill family responsibilities, what might be some intermediate outcome expectations?

References Bennett, L., Stoops, C., Call, C., & Flett, H. (2007). Program completion and re-arrest in a batterer intervention

system. Research on Social Work Practice, 17, 42–54.

Carrilio, T. (2005). Management information systems: Why are they underutilized in the social services? Administration in Social Work, 29(2), 43–61.

Chen, S., Jordan, C., & Thompson, S. (2006). The effect of cognitive behavioral therapy (CBT) on depression: The role of problem-solving appraisal. Research on Social Work Practice, 16, 500–510.

Fischer, J., & Corcoran, K. (1994). Measures for clinical practice: Vol. 1. Couples, families and children. New York, NY: Free Press.

Gerstein, L., Heppner, P., Aegisdottir, S., Leung, S., & Norsworthy, K. (2009). International handbook of cross- cultural counseling: Cultural assumptions and practices worldwide. Thousand Oaks, CA: Sage.

Hatry, H. (2011). Outcomes measurement in the human services: Lessons learned from public and private sectors. In J. Magnabosco & R. Manderscheid (Eds.), Outcomes measurement in the human services (2nd ed., pp. 17–29). Washington, DC: National Association of Social Workers.

6/2/2018 Bookshelf Online: Designing and Managing Programs

https://online.vitalsource.com/#/books/9781483388328/cfi/6/48!/4/2/4/2@0:0 21/21

Hudson, W. (1982). The clinical measurement package: A field manual. Chicago, IL: Dorsey.

McAuliffe, G., & Associates. (2008). Culturally alert counseling: A comprehensive introduction. Thousand Oaks, CA: Sage.

Netting, F., Kettner, P., & McMurtry, S. (in press). Social work macro practice (6th ed.). Upper Saddle River, NJ: Pearson.

Pedersen, P., Draguns, J., Lonner, W., & Trimble, J. (2008). Counseling across cultures (6th ed.). Thousand Oaks, CA: Sage.

Rapp, C., & Poertner, J. (1992). Social administration: A client-centered approach. New York, NY: Longman.

Reid, W. (1992). Task strategies: An empirical approach to clinical social work. New York, NY: Columbia University Press.

Saleeby, D. (1992). The strengths perspective in social work practice. New York, NY: Longman.

Schoech, D. (1999). Human services technology: Understanding, designing, and implementing computer and Internet applications in the social services. New York, NY: Haworth Press.

Schoech, D. (2000). Managing information for decision making. In R. Patti (Ed.), The handbook of social welfare management (pp. 321–339). Thousand Oaks, CA: Sage.

Steering Committee for the Review of Government Service Provision. (2006). Report on government services 2006. Canberra, Australia: Productivity Commission.

Suchman, E. (1967). Evaluative research. New York, NY: Russell Sage Foundation.

Part IV Calculating the Value and Cost of the Intervention