Final for grant proposal writing and evaluation writing: PLEASE INCLUDE ALL THE DETAILS LISTED IN THE DESCRIPTION AREA

hsm270_week1_reading.pdf

Home >Human Resource Management homework help >Final for grant proposal writing and evaluation writing: PLEASE INCLUDE ALL THE DETAILS LISTED IN THE DESCRIPTION AREA

RATIONALITY AND REALITY OF PROGRAM PLANNING AND EVALUATION

What is a program? What does it intend to achieve? What are the basic properties of planning and evaluation? How are they related? This chapter presents a basic understanding of the char- acteristics of program planning and program evaluation. Planning and evaluation are mostly goal-oriented activities that incorporate many linear-sequential and logical thinking procedures. They are also products of social, economic, and political realities, continuously evolving and moving in a spiral fashion. Successful planning and evaluation demand the arts of balance and inclusiveness of ideology and reality.

Defining Programs

A program is a coordinated change effort that is theory based, goal-oriented, often time limited, target population-specific, and activity driven. It is also an evolving process. A program is the result of some type of planning. Programs are intended to achieve particular goals with their spe- cial intervention approaches and operating procedures.

Mayer (1985) defines a program as “an intervention that constitutes a service or activity that can be provided to facilitate volitional change.” He views planning as “the design of interven- tions” (p. 21). Royse, Thyer, Padgett, and Logan (2001) describe a program as “an organized col- lection of activities designed to reach certain objectives . . . a series of planned actions that are designed to solve some problem” (p. 5). Mika (1996) depicts a program as “a design of activities that, in theory, produce some change or outcome in its participants within a given time frame” (p. 6). Weinbach (1998) portrays a program as a “complex system”; a “self-contained package with its own goals, policies, procedures, and rules, and frequently its own budget” (p. 91).

In human service organizations, programs are often referred as “services.” Each of these programs is supposed to bring the agency closer toward fulfilling its organizational mission. Ideally, all programs supplement and enhance one another. In reality, however, there may be overlapping, as well as competition, among programs for the limited resources available.

Fiscally, there are financially rich programs and there are, more often than not, poorly funded programs. Nevertheless, most of the funding for programs is earmarked for particular purposes and usages. When the agency administrations get too “creative” and start inappro- priately and excessively commingling funding, or “robbing Peter to pay Paul,” the integrity of the program will suffer. Subsequent discontent and disagreements from staff of different pro- grams and their clients are among the critical reasons that can bring down a program.

Similarly, programs are supposed to be time limited. Many human service programs and organizations have outlived their original timeframes, functions, and purposes; however, they continue to exist. A possible situation of goal displacement might have occurred. The survival, continuation, and status quo of a program or organization become the goal of the entity, not meeting the needs of the clients it serves. Clients instead become the means, not the end. The survival goals replace the original service goals.

C H A P T E R

Beyond the general references of programs as coordinated human service interventions at the agency and community levels, there are “social programs” that are the implementations of particular social policies. Social programs are organized interventions that aim to improve a social situation that may be a social problem, a social need, or a condition that requires specific actions. Social programs could also be mandated through formal sanctions such as social poli- cies. These social programs and their sponsoring agencies could be drastically different in size, scope, theoretical and philosophical foundations, practice approaches, mandates, and many other aspects. Social programs are carried out through a variety of specific plans and interven- tions that have particular services/components and activities.

There are no programs or social programs that are “cure all and for all” by addressing every aspect or need of a social condition. A grassroots public education campaign alone certainly can- not cure the disease of AIDS. It could, however, contribute to the better prevention and under- standing of the disease in the target populations, and address specific aspects of needs and issues of this public health concern. Programs therefore should clearly communicate their service intentions by being specific in their scope and associated goals, objectives, activities, and out- comes. It is through program evaluation that the attainment of program goals and the quality of the program are assessed.

A program is inherently a planned process to achieve particular future goals. It is conceived through a careful process of planning and design to bring about the desired change. Good program planning is a necessary, though not sufficient, cause for a successful program. Subsequently, the effectiveness of the program—or the change that programs can actually produce—will not be clearly identified and documented if there is no program evaluation. Planning, implementation, and evaluation are the integral features of programs.

Rubin and Babbie (1997) cited Rossi and Freeman’s (1982) discussion on the purpose of program evaluation. It is “to assess and improve the conceptualization, design, planning, administration, implementation, effectiveness, efficiency, and utility of social interventions and human service programs” (p. 547). Rubin and Babbie further assert that “much of what we call social work research could be called program evaluation research” (p. 545). This is because “program evaluation has more to do with the purposes of research than with specific research methods” (p. 547).

Program Planning and Program Evaluation: Two Sides of a Coin

“A program and evaluation of a program are not separate activities. When planned carefully, these two set of activities can be integrated into an organized set of ongoing operations that pro- mote and support each other” (Mika, 1996, p. 6). Program planning is an organized process through which a set of coordinated activities or interventions is developed to address and facil- itate change in some or all of the identified problems. It is an ongoing process of development with the intention of designing a set of activities that will tackle the identified needs and prob- lems. Program planning is a means for program development. It is a goal-oriented activity based on the assessed needs or problems.

Program planning not only concerns what will happen, it also involves the assessment of what has happened and what really would have happened. It is by design and by its very nature related to needs assessment and program evaluation. Needs or problems assessment is the necessary and fundamental step for any successful program planning. These assessments provide the rationale and direction for the development of program plans. Once the program

2 CHAPTER ONE

interventions are implemented, the performance and the attainment of intended objectives are then evaluated. Assessments represent the front end or the basic appraisals, and evaluations reflect the continuous reviews and judgments.

Kettner, Moroney, and Martin (1999) propose effectiveness-based program planning that “involves taking a program through a series of steps designed to produce a clear understanding of the problem to be addressed, to measure client problem type and severity at entry, to provide a relevant intervention, to measure client problem type and severity at exit, and to examine selected indicators in a follow-up study to determine long-range outcomes” (pp. 6–7). This type of program planning incorporates program evaluation throughout the process and provides con- tinuous assessment of the program performance to gain further improvement.

In practice, funding sources are no longer willing to allocate resources for human services and expect merely a report on whether or not the program and service providers did what they said they would do. Funders want to know what benefits are gained from the services provided. Programs are now required to evaluate their services and show the results (process, outcome, and impact) of the services provided. Consequently, service providers must pay more attention to well developed program plans in order to ensure the highest probability of reaching and pro- ducing the desired results.

United Way of America (1996) discusses the difference between the Traditional Service Program model and the Program Outcome model. The traditional model basically includes inputs, activities, and outputs. “Inputs include resources dedicated to or consumed by the pro- gram” (p. 17). These resources involve a wide range of assets and investments such as money, staff time, facilities, equipment, and supplies. Regulations, mandates, and funding restraints are also part of the inputs. “Activities are what the program does with the inputs to fulfill its mis- sion” (p. 17). They are the service methodologies such as interventions and strategies. “Outputs are the direct products of program activities and usually are measured in terms of volume of work accomplished” (p. 17). In this sense, the outputs are basically process data that testify what and how many activities have been done.

The preferred program outcome model adds an important component of “outcomes” at the end of the traditional program model. The outcome model includes inputs, activities, outputs, and outcomes. “Outcomes are benefits or change for individuals or populations during or after participating in program activities” (United Way of America, 1996, p. 18). Outcomes answer the basic question of “whether, and how much, a program’s participants have changed, how their sta- tus has improved, how they have benefited” (p 18). The focus is beyond what has been done; it is on what changes or benefits have been produced after the participation in program activities. These changes or outcomes “may relate to behavior, skills, knowledge, attitudes, values, condi- tion, status, or other attributes” (p. 18).

Whether it is the traditional or the outcome model, there remains the importance of care- ful and thoughtful assessment, planning, activity design and implementation, and evaluation of program results. Again, there is an interrelated and symbiotic relationship between planning and evaluation.

The Political Economy of Program Planning and Evaluation

Program evaluation supposedly provides useful information for improving the programs and the service delivery systems. In practice, many political and economic variables can make planning and evaluation undesirable and reluctant tasks. Program planning and evaluation are both the

RATIONALITY AND REALITY OF PROGRAM PLANNING AND EVALUATION 3

process and the product of the political and economic environments. Political economy is the interaction and dynamic between politics and economic interests. Netting, Kettner, and McMurtry (1999) state, “political means the process by which the organization obtains power and legitimacy. Economic means the process by which the organization gets resources such as clients, staff, and funding” (p. 219).

Many researchers (e.g., Grinnell, 1997; Rubin & Babbie, 1997) have discussed the various political and economic concerns about program evaluation. First, there are issues with the “political correctness” or “favoritism” of evaluations. Groups with competitive interests want to see the evaluation data work towards their favors, or be presented in ways so that all political bases are covered. They may use their political influence to “guide” the evaluation to present or not to present the “selected” truth.

Although scientific inquiries are supposed to be objective and unbiased, many people per- forming program evaluation tasks receive pressure to make the program look good. Program evaluation activities are often not among the highest priority for administrative and financial support. Consequently, individuals who have the program evaluation assignments may not be the ones who have the competency or commitment to carry out quality research procedures. The original purpose of describing the program as it is, and any improvement that should be made, could easily be replaced. In its place may be the nice and encouraging report that aims to secure current and future funding.

It is common among human service organizations to have in-house evaluators, although certain funding sources may require the use of external evaluators. There are pros and cons for having an internal or external evaluator. Internal evaluators may have better access and familiarity to the organization, staff, and information; they are also part of the organization. Internal politics, personal knowledge, and future plans within the organization may contribute to the less than desirable arrangement to have internal evaluators. External evaluators need time to get to know the organization, the program, and its politics. Meanwhile, staff members can also view them as outsiders that the administrators hire to “spy” on the staff. Whether they are internal or external evaluators, their ability to build trust with the program staff and to maintain independence, objectivity, and ethical and quality practice are among the most important considerations.

Not surprisingly, there are resistances from program staff and clients. Evaluation is per- ceived as an additional, undesirable, and labor-intensive assignment. It takes away the precious time that program staff and clients could use to provide or receive more direct service or activ- ities. Developing buy-in from all levels within the organization is a major issue for any success- ful program evaluation task. The few useful approaches include involvement of concerned parties from the very beginning of the project, and providing useful reports on findings. These involvements have to be real and genuine participation. They include many activities; from con- ceiving the needs and outcomes of the program and its evaluation, to designing evaluation approaches and reporting findings. If staff find they have no time to do evaluation, the agency should be sure that there is built-in or allotted time to complete evaluation tasks.

Staff and clients are frustrated when they provide all the data demanded but never see the reports that come from those data. Even if they do see the reports, often those findings are outdated, not useable, or of no personal significance for them. Certainly, program evaluation reports need to be user friendly and useful to readers. Providing timely reports or feedback of the evaluation findings to the service users or people under study is an important piece of the evaluation puzzle.

4 CHAPTER ONE

The Technology of Program Planning and Program Evaluation

When we talk about technology, we may think it is about computers. Technology in human ser- vices, according to Lewis, Lewis, Packard, and Souflee (2001), is about “the transformation process” (p. 58). It is the transformation of inputs into outputs. Inputs are the resources such as data and information of needs and problems, intervention methods, and staffing. Outputs are the services and goods that are produced as the result of utilization of the inputs. Technology is involved in the service delivery model—the practice method that transforms resources or inputs into services or outputs that serve the clients and the community.

According to Germain (1983), technology is a professional values-guided application of the- ories, knowledge, and skills to practice. Germain further states that “approaches, methods, skills, and techniques are considered to make up technology” (p. 18). To understand technology, one must first distinguish among the often confusing terms: theory, knowledge, approaches, models, methods, skills, and techniques.

Reynolds (1971) notes that theory has been referred to as conceptualization, prescriptions for behaviors, or untested ideas. Yuen (1999b) defines theory as “a set of interrelated proposi- tions or concepts, organized in some systematic manner, that offers an explanation of some phe- nomena” (p. 18).

Yuen (1999a) expands on Atherton and Klemmack’s (1982) listing of four sources of knowl- edge: tradition, experience, common sense, and science. He explains that “tradition is custom and beliefs that have been handed down from generation to generation. It is not necessarily log- ical or rational, but it makes sense to the people who practice it. Experience is the person’s first- hand observation” (p. 106). Sarcastically, everyone expects each other has a common sense and knows what a common sense is. Although in reality it is an odd and somewhat difficult term to define, “common sense is the combination of tradition and experience” (p. 106). Finally, “scien- tific knowledge is developed mainly through logical and rational validations. It is not the source of absolute knowledge, but it provides the objective means of knowing in addition to the other more subjective ways of knowing” (pp. 106–107).

These many ways of knowing lead to the establishment of certain knowledge, which, accord- ing to Germain (1983), forms a “recognizable or recognized perspective entities often called practice model” or approach (p. 31). Method is the specific application of such a model or approach with specific target groups. Germain notes that skill refers to a “particular area of prac- titioner action, such as observation, engagement, data collection, assessment, contracting, set- ting goals and planning, and achieving goals. Technique is used to designate a more specific procedure within such an area of skill” (p. 31). Technology is therefore the combination and application of the theories, knowledge, approaches, methods, skills, and techniques that produce the services.

Program planning could be considered as the selection and decision-making effort that attempts to identify and develop the finest service model. Ideally, this is a model that best uti- lizes the resources and would achieve the most desirable results. Paralleled with the program planning process and the actual implementation of services is the program evaluation process. The program evaluation provides both the formative and summative data that could be used to measure and assess the desirability and quality of the plan and its program.

Many factors contribute to the development of new service programs. Agency staff’s obser- vations and experience of working with the target population certainly generate internal impe- tus for the development of new programs. Similarly, social and political factors including

RATIONALITY AND REALITY OF PROGRAM PLANNING AND EVALUATION 5

legislations and government policies could mandate the development of new services. Needs assessment is an objective way to identify and characterize needs and accordingly develop appropriate services. Once an agency identifies the needs and conceptualizes the program ideas, the work of developing program goals, objectives, and activities begins, in order to define the scope and the results of the program. A set of well defined goals and objectives also provides the base for program monitoring and evaluation. Logic model and goal attainment scale are exam- ples of some of the commonly used program development tools. Chapter 3 discusses the logic model in greater detail.

Program evaluation is the application of different social research designs in assessing the needs and results of programs. Evaluative research utilizes various research designs including exploratory, descriptive, and experimental designs. The exploratory study is designed to gain familiarity and develop hypotheses regarding a particular topic. It is achieved through methods (Atherton & Klemmack, 1982) such as:

1. Literature survey and documentary study 2. Experience study and historical study 3. Study of selected examples, typical cases, and extreme cases 4. Interview with individuals with different viewpoints 5. Review of one’s experience and self-report

Descriptive research provides “description of a phenomenon or the description of the rela- tionships between two or more phenomena” (Atherton & Klemmack, 1982, p. 29). It has a clear statement of problem and is more precise in its data collection, possibly involving sampling. Descriptive research therefore is designed to:

1. Describe some characteristics of the program, the service recipients, and other elements. 2. Describe the use of community resources, facilities, and other resources. 3. Solicit people’s views on an issue. 4. Study the relationship of association among various variables/factors.

Needs assessment, for example, is a commonly used exploratory or descriptive study. Some descriptive research designs involve the use of hypothesis. A hypothesis is an educated guess and a statement that can be tested in an empirical way. It defines the relationship of one or more concepts so that the relationship can be tested. Although both a hypothesis and an assumption are one’s hunch or speculation, they direct very different outcomes. When a person has a hypothesis of a certain situation, he or she will proceed with a scientific inquiry to test the hypothesis that may or may not generate conclusive answers. However, when a person is oper- ating on an assumption, the person already has a preconceived notion or the answer for the iden- tified situation. He or she will proceed as if the answer is true and that testing is not required. The only question is how true his or her assumption is, but not whether it is true.

In many situations, program planners operate on assumptions that are informed by profes- sional literature, relevant data, and people’s experiences. These assumptions will form the ori- entations and the basic service models of the proposed intervention program and become the program philosophy. Program evaluation, meanwhile, provides the needed validation and check- ing of this philosophy and its operations. In order for the program implementation and evalua- tion people to know what they should do to objectively assess program success and failure, specific working program hypotheses are formed and used to guide the implementation and evaluation processes. Atherton and Klemmack (1982) discuss the different hypotheses in descriptive research:

6 CHAPTER ONE

1. X has a particular attribute or characteristic such as “Children who have positive adult role models in their lives tend to have better success in schools.”

2. X happens more often than Y in a given population, such as “Limited employment oppor- tunities that pay a living wage occur more frequently than the lack of vocation training in single parents who are on welfare.”

3. X is substantially associated with Y, such as “The more frequent presence of resilient fac- tors, the less likely the youth will engage in high-risk behaviors.”

Experimental research anticipates causality, in so far as this is possible, as well as association. It involves:

1. Independent variable (X) and dependent variable (Y): change in Y depends on change in X, or change in X will bring about change in Y.

2. Experimental group (the group that receives intervention/administration of the indepen- dent variable) and control group (the group that does not receive intervention).

Atherton and Kelmmack (1982) further discuss the contributory, contingent, and alternative hypotheses in experimental research.

1. Contributory: X is one of the factors that changes the likelihood of Y, such as “Hard work increases the likelihood of getting good grades.”

2. Contingent: Under certain situation and predicament, X may be culpable for the likeli- hood of change in Y, such as “Job placement services will decrease the number of people who are on the welfare roll, when the economy is doing well.”

3. Alternative: Either X or Z alters the likelihood of Y, such as “Changes in either parents’ or their children’s behaviors will lead to fewer child abuse incidents.”

Box 1.1 provides a brief review of various experimental designs.

Summary

Program planning and evaluation should be the integral parts of any programs. This chapter pro- vides the basic understanding of program planning and program evaluation. It emphasizes that planning and evaluation are products of rational thinking as well as political and economical con- siderations. The technology of planning and evaluation explains how it incorporates different types of knowledge and is carried out in a range of basic designs with various working hypothe- ses and assumptions.

References

Atherton, C., & Klemmack, D. (1982). Research methods in social work. Lexington, MA: D.C. Heath.

Germain, C. (1983). Technological advances. In A. Rosenblatt & D. Waldfogel (Eds.), Handbook of clinical social work (pp. 26–57). San Francisco: Jossey-Bass.

Grinnell, R. (1997). Social work research and evaluation: Quantitative and qualitative approaches. Itasca, IL: P.E. Peacock.

Kettner, P., Moroney, R., & Martin, L. (1999). Designing and managing programs: An effec- tiveness-based approach (2nd ed.). Thousand Oaks, CA: Sage.

Lewis, J., Lewis, M., Packard, T., & Souflee, F. (2001). Management of human service pro- grams (3rd ed.). Belmont, CA: Brooks/Cole.

RATIONALITY AND REALITY OF PROGRAM PLANNING AND EVALUATION 7

8 CHAPTER ONE

BOX

1.1 EXAMPLES OF EXPERIMENTAL DESIGNS

X: Administration of the independent variable O: Observation of measurement of the dependent variable R: Random assignment

A. Preexperimental Designs 1. The one-shot case study / cross-sectional case study

X O

2. The one-group pretest-posttest O1 X O2

3. The static group comparison X O (experimental group)

O (control group)

B. True Experimental Designs 1. Cross-sectional survey

R X O

2. The pretest-posttest control-group design R O1 X O2 R O1 O2

3. The posttest-only control-group design R X O R O

4. The Solomon four-group design R O X O R O O R X O R O

C. Quasiexperimental Designs 1. The nonequivalent comparison group

O1 X O2 O1 O2

2. The time-series quasiexperimental design—interrupted time series O1 O2 O3 O4 X O5 O6 O7 O8

3. The multiple time-series design O1 O2 O3 O4 X O5 O6 O7 O8 O1 O2 O3 O4 O5 O6 O7 O8

4. The single-subject design or the single-case study: A (Baseline), B (Intervention). AB ABAB

Mayer, R. (1985). Policy and program planning: A developmental perspective. Englewood Cliffs, NJ: Prentice Hall.

Mika, K. (1996). Program outcome evaluation: A step-by-step handbook. Milwaukee, WI: Families International, Inc.

Netting, E., Kettner, P., & McMurtry, S. (1999). Social work macro practice. (2nd ed.). New York: Longman.

Office for Substance Abuse Prevention. (1992). Three management tools for OSAP Demonstration projects for pregnant and postpartum women and their infants (PPWI): Logic models, GOAMs charts, and evaluation plans. Rockville, MD: Alcohol, Drug Abuse, and Mental Health Administration.

Reynolds, P. (1971). A primer in theory construction. Indianapolis: Bobbs-Merrill. Royse, D., Thyer, B., Padgett, D., & Logan, T. (2001). Program evaluation: An introduction

(3rd ed.). Belmont, CA: Brooks/Cole. Rubin, A., & Babbie, E. (1997). Research methods for social work (3rd ed.). Pacific Grove,

CA: Brooks/Cole. United Way of America (1996). Measuring program outcomes: A practical approach.

Alexandria, VA: Author. Weinbach, R. (1998). The social worker as manager: A practical guide to success. (3rd ed.).

Needham Heights, MA: Allyn & Bacon. Yuen, F. K. O. (1999a). Family health and cultural diversity. In J. T. Pardeck & F. K. O. Yuen

(Eds.), Family health: A holistic approach to social work practice (pp. 101–113). Westport, CT: Auburn House.

Yuen, F. K. O. (1999b). The properties of family health approach. In J. T. Pardeck & F. K. O. Yuen (Eds.), Family health: A holistic approach to social work practice (pp. 17–28). Westport, CT: Auburn House.

RATIONALITY AND REALITY OF PROGRAM PLANNING AND EVALUATION 9