Final for grant proposal writing and evaluation writing: PLEASE INCLUDE ALL THE DETAILS LISTED IN THE DESCRIPTION AREA

hsm270_week7_reading.pdf

Home >Human Resource Management homework help >Final for grant proposal writing and evaluation writing: PLEASE INCLUDE ALL THE DETAILS LISTED IN THE DESCRIPTION AREA

PUTTING IT TOGETHER: MODEL PROGRAM AND EVALUATION PLANS

In the previous chapters, different examples were used to illustrate our discussions on program planning, grant writing, and program evaluation. This chapter draws on the examples of three model plans to: (a) show the incorporation of information that we presented into program and evaluation plans, and (b) highlight the importance and seriousness of ethical considerations in carrying out program evaluation.

These models are intended to serve as examples or sample products of what they could be like. They are not standards or models of perfection for everyone to follow. Instead, they can serve as practical examples and references for readers to use in developing their very own pro- gram and evaluation plan.

Model #1, Project Healthy Families, represents a detailed program plan and evaluation plan for a federally funded demonstration program. It is a substance abuse prevention project tar- geting homeless children and their families. Evaluation tasks are to be carried out by a team of external evaluators.

Model #2, ABC AmeriCorps Project, represents a simple program evaluation plan for a small, local AmeriCorps service project. The project uses AmeriCorps members and volunteers to provide tutoring and other academic support activities to improve the reading abilities of at- risk students. Evaluation tasks are to be carried out by program staff.

Model #3, Ethical Considerations (for Project Hope), shows how ethical considerations in program evaluation are detailed and observed. Ethical practice is an essential component of any evaluation plan. Project Hope is a substance abuse treatment program focused on Asian- American youth. Its confidentiality and ethical concerns represent a relatively detailed write-up that may not be required for smaller evaluation tasks; however, the basic ethical practices and safeguards are still expected to be followed and valued in smaller programs.

Model #1: Project Healthy Families

Program Plan and Description

The Good Place, Inc., proposes a 3-year demonstration project—“Healthy Families”—targeting 800 rural homeless and low-income families and their children and adolescents (ages 0–14) in Springdale. This project will test the efficiency and effectiveness of a set of theoretically based, innovative, and well-coordinated programs that are specific to gender, age, and culture of the target population. These programs aim to promote the current goals set by the Center for Substance Abuse Prevention (CSAP).

The working hypothesis for this project is that “empowering the target population to increase family stability and to promote children and family wellness will decrease the likelihood of involvement in alcohol, tobacco, and other drugs (ATOD) use and other related high-risk behaviors.” In other words, children and parents who have stable living arrangements, positive

100

C H A P T E R

bonding with significant others, congruent self-identities, and the skills to maintain family and individual economic and social well-being are less likely to be involved in ATOD use and other high-risk or violent behaviors. This project targets homeless/low-income families and their chil- dren living at the Good Place’s housing (emergency shelter and transitional apartments) and in the surrounding low-income neighborhoods.

The theoretical base for the proposed project rely mostly on the Family Health Model (Pardeck & Yuen, 1999), which also includes three major perspectives: ecological (Germain & Gitterman, 1987; Meyer, 1983), family systems (Hartman & Laird, 1983; Von Bertalanffy, 1968), and social constructionist (Gelber & Specter, 1987; Gergen & McNamee, 1992; Hoffman, 1990). This combined ecological systems and social constructionist approach views homeless/low-income children’s high-risk behaviors from a holistic and dynamic perspective.

Pardeck and Yuen (1999) define family health as “a state of holistic well-being of the family system. Family health is manifested by the development of, and continuous interaction among, the physical, mental, emotional, social, economic, cultural, and spiritual dimensions of the fam- ily which results in its holistic well-being as well as that of its members” (p. 1). Family health intervention skills include: developmental assessment, resource linkage, supportive interven- tion, supportive intervention, confrontational intervention, empowerment of the client, use of social learning approaches, recognizing the level of prevention and readiness for change, uti- lization of the family unit as change agent, understanding and assessing the construction of a client’s reality, and advocating for the client.

Ecological perspective assesses individuals through the context of one’s interaction with the environment. Homeless and low-income children and families occupy a “negative niche” in today’s society. Some view them as “lazy people” who drain and waste American resources. Meanwhile, their homeless and poverty situations have posed a variety of life stressors that are often traumatic and detrimental. Unfortunately, many of them may not have the required educa- tional and life skills that allow them to cope properly with these stress factors. As a result, many choose alternative coping approaches, which may not be effective or acceptable by society.

Systems theory uses circular causality and views that an individual and her or his environ- ment interact and affect each other mutually. Through input, throughput, output, and feedback, units within the system seek for equilibrium. Often, this feedback system for homeless and low- income children and families has been either absent, negative, or nonfunctional. Both ecologi- cal perspectives and systems theory stress the importance of the reciprocal relationship between individual and family as well as the “wellness” of the systems and their interaction processes.

The social constructionist approach believes that stories, myths, literature, and legends help individuals draw the events of their lives into meaningful wholes that have structure, direction, and purpose. These creations help them to know who they are, how they cope with failure and success, and most importantly, where they are heading—the new reality. Homeless children and parents are likely to become more self-sufficient if they can have pride in themselves and see that there is hope for their future.

Many homeless children and adolescents are living in the stage of anomie. These anomic youngsters are likely to turn to other subcultures to meet their needs for affiliation and may become involved in high-risk behaviors such as substance abuse and gang activities. Individuals who can identify with a stable living arrangement will likely encounter fewer adjustment and personal problems.

The proposed interventions are also based on the risk/protective factors applied to four domains: individual, families, peers, and schools. The proposed strategies are intended to

PUTTING IT TOGETHER: MODEL PROGRAM AND EVALUATION PLANS 101

improve stability and life skills of homeless or low-income children and their families. The pro- posed Healthy Families Project has three main components: (a) Family Together, (b) Health and Case Management, and (c) Community and School Services.

The Family Together component consists of four main sets of interrelated activities.

1. During the regular school year, from around 3:15 to 4:30 P.M., targeted children will par- ticipate in the After-School Learning Center, which is located within the shelter and will provide a variety of tutorial and recreational activities.

2. While these children are attending the center’s activities, their parents (mostly mothers) will attend a separate Parents-to-Parents program, which provides educational and employment readiness activities.

3. Twice a week, after the dinner and individual family hours from 4:30 to 6:00 P.M., children and their families will come together to participate in the Family Night program. This pro- gram is based on the Family Success curriculum for homeless families and aims to strengthen family functioning and resiliency factors.

4. Once a month, special events such as trips to the Six Flags amusement park and local fes- tivals will be organized to provide opportunities for family fun and interaction.

The Health and Case Management component assists targeted families’ abilities to coor- dinate their health/mental health, employment, and social needs. This component includes three main services:

1. Basic health check-up, medical care, and referral services provided on-site at the shelter or through referral to the local St. Francis Hospital.

2. Individual and group counseling and advocacy. 3. Employment and goal-setting assistance.

The Community and School component facilitates comprehensive support from schools and communities to promote healthy families. This component consists of four main activities:

1. Implementation of an Alateen or peer support group for targeted children at Johnson Middle School.

2. Coordination with school officials to develop a mechanism to monitor the academic and overall performance of targeted children from the shelter.

3. Development of a volunteer service group for the project by local college students and youth.

4. Development of an eight-member Health Families Advisory Board. The board consists of parents from the target population, human service providers, a representative of religious groups, a law enforcement officer, a school representative, and community leaders.

Program Evaluation

The objectives of the evaluation plan are: (a) to document and monitor the implementation of the proposed interventions and activities, and (b) to provide data to assess the outcome and overall impact of the proposed project. The results of evaluation will provide information for better understanding of the three key questions for the proposed project:

1. Can a comprehensive and culturally competent project that addresses the multiple needs of participants lead to the reduction of the use of alcohol, tobacco, and other drugs

102 CHAPTER SEVEN

(ATOD) as well as related problems among target homeless/low-income families? If so, what strategies or combinations of strategies produce the best results?

2. What are the relationships between ATOD-related problems and family capacity? 3. Can this proposed model be replicated elsewhere?

These three questions relate directly to Center for Substance Abuse Prevention’s (CSAP) requirements, and to the proposed application’s goals and objectives. They will also serve as principles for process and outcome evaluation.

The evaluation team will meet with the project management staff during the first month of the project to further refine the following evaluation action plan. Project staff will then review this revised action plan, and a youth focus group and a parent focus group will provide com- ments. Their input will help refine the definition of risk and resiliency factors and further deter- mine the focus of evaluation. The revised action plan will be finalized in the third month.

Process Evaluation

Documenting and monitoring the planning, implementation, and interrelationship of the components of the proposed project are the focus of the process evaluation, which aims to: (a) describe program interventions’ development, implementation, and activities, (b) provide quantitative and qualitative data on services delivered and their effectiveness, and (c) document the appropriateness and acceptability of the program within the target community. To achieve these aims, there are three main components of the process evaluation: (a) program planning and development, (b) program interventions, and (c) database management system.

A. Program Planning and Development Planning and development is an ongoing process. The documentation of this process includes compiling information on administrative planning, staff meeting, staff recruitment and training, preparation for program activity, work schedule, organizational support, and community linkage. This information not only documents the progress of the project, but also provides a basis for a more accurate definition of the problems to be addressed and their associated risk and protective factors.

In order to document the process of program development and implementation, the evalu- ation team will collect agendas of project staff meetings, minutes, and other related documents. Evaluation team staff will also participate in selected planning and program activities in order to do observations. Staff and participants will be interviewed, in addition to the biannual question- naire responses, to document satisfaction measures and other concerns. Staff training, work- shops, and conferences will also be documented through records of participation and minutes of discussions. Encounter forms and attendance sheets will document participation and units of service. CSAP Management Information Forms (MIF) require specific information about the target population. The evaluation team will collect relevant data for MIF.

B. Program Interventions The documentation of program intervention should provide both qualitative and quantitative descriptions of the interventions. These documentation procedures include several elements and different data collection approaches. First, descriptions of type of interventions will be recorded along with intervention products such as curriculum. Second, intake and application forms will be used to collect information about project participants. Third, quantity and utilization of project services, as well as the time and locations of these serv- ices, will be documented through encounter forms, activity logs, and attendance sheets. Fourth, both staff and project participants will complete satisfaction survey scales to provide feedback

PUTTING IT TOGETHER: MODEL PROGRAM AND EVALUATION PLANS 103

and assessment regarding the project. Fifth, some ethnographic observations of project activi- ties will be used to collect qualitative information for the project. These process evaluation pro- cedures provide a basis for the understanding of the operation and effectiveness of the project. The following are some of the measures and forms used to document these operations and their effectiveness:

Staff Recruitment, Hiring, and Training Recruitment and hiring procedures, as well as staff training and conferences, will be documented. Feedback from staff through interviews and questionnaires will also be collected.

Planning and Development Activity forms, meeting agendas and minutes, and participatory observation will be used to reflect program development process, problems encountered, and alternatives and solutions to the problems.

Program Usage Attendance records, activity logs, sign-in and sign-out forms, and daily self- reports by staff will be used to document the extent of service used, the achievements and bar- riers, client flow, client penetration, and the characteristics of project participants.

Participant and Staff Satisfaction Project staff and project participants, particularly those in instructional activities, will be asked to provide feedback through questionnaires and interviews. Information collected provides a basis to assess the level of satisfaction toward project activities and organizational and community support.

Target Population The agency’s current General Data Form collects basic demographic infor- mation, including ethnicity, gender, age, family composition, employment and income, and other data. It will be revised to become the intake and application forms for the project. The new forms will include information on risk categories, ATOD histories, and special health con- cerns. This form will be continuously updated and revised upon review of clients’ needs and staff’s recommendation.

Program Outreach All individual and community outreach will be tracked by an outreach encounter form. Content, frequency, and results of these efforts will be documented to assess effectiveness.

Case Management Recording Case management services for participants will be recorded in individual clients’ charts. General social work case management recording requirements will be applied as the standard for charting. These recordings will document the type and progress of interventions, intervention achievements and barriers, frequency and characteristics of referrals, interorganizational linkages, and assessment of clients and services provided.

Linkages with Managed Health Care Project linkages with local managed health care providers at all levels will be documented. The City of Springdale has three major regional health centers and an array of other specialized medical care facilities. It is the mecca of medical service in the region. Project’s linkages with these managed health care systems will be documented from indi- vidual client referrals to institutional agreements. Special attention will be placed on how these linkages will affect the provision of prevention services provided by these organizations.

Project Goals and Objectives Each objective’s and activity’s progress will be collected on a quarterly basis. In addition to the use of forms and self-reports from clients and staff, participa- tory observation by evaluation team staff will also be used for data collection.

104 CHAPTER SEVEN

For Objective 1 (Families Together Component), for its After-School Learning Center activ- ities, process evaluation will document all planning meetings, site development and selection meetings, activities, minutes, and attendance records. For the Parents-to-Parents program, a pre- and posttest will be used to document changes in participants’ attitude and knowledge about parenting and other life skills. Attendance and dropout records will also be used. Family Night and Family Fun activities will be documented by types of activities, attendance, and an observation of interaction among family members. In addition to attendance records and self- reported evaluation from staff and participants, a client satisfaction form will also be used. This survey form will be developed and used by the ninth month of the project.

For Objective 2, Health and Case Management, units of counseling or health screening and medical intervention, as well as referral will be recorded. A case management chart and a med- ical service chart that record intervention process and other related activities will be opened for each client. Types of health and mental health concerns for project participants will be catego- rized for baseline data and trend analysis. The project’s involvement in the development of the local managed care system will be documented through minutes and agendas. Prevention ser- vices developed by local health care providers as the result of interaction with the proposed proj- ect will be tracked and documented.

For Objective 3, School/Community Services, project staff will visit participants’ teachers (with the approval of participants’ parents) to obtain students’ academic records to develop base- line and progress data for comparison. Volunteer group activities will be documented for train- ing model development meetings, training contents and implementation, attendance records, service provided, time used, and volunteer application forms and screening procedures. Teen Support Groups’ activity types and contents, attendance, recruitment and retention records, and the extent of involvement from volunteers will be recorded. Data on the Advisory Board’s mem- bership composition, meeting minutes and agendas, and types of involvement will be collected.

Cost Analysis: The evaluation team will analyze the cost per unit of service and client. Cost analysis is to evaluate the efficiency of the project and the relationship between efforts and effects. Its results will be compared among various project components and to other similar pre- vention projects. Data collected for each component will include: number of clients served, units of service provided, staff time used, personnel and operating cost, and volunteer and other in-kind support.

C. Database Management Information System A well-developed database management information system (MIS) provides the mechanism to store and compile quantifiable data to assess the development and effectiveness of the project. This system involves: (a) the develop- ment of file formats and identification of appropriate software, (b) subsequent data entry, and (c) the generation of reports. With data collection and data analyses, the MIS forms the basis for evaluation.

All clients are required to complete an intake or application form to collect basic demo- graphic and ATOD use-related information. Encounter and attendance forms will collect units and types of services for individual, group, case management, and family activities detailing date of occurrence, numbers served, location, and time. Client chart documentation, including the DSM-IV diagnosis (if applicable), will also serve as process data. Satisfaction scales will provide input and feedback from clients, families, and staff. Observation field notes, pre/post tests results, and other measures will also be stored in the MIS. All project forms and measures will

PUTTING IT TOGETHER: MODEL PROGRAM AND EVALUATION PLANS 105

be updated as needed; baseline and other project data will continue to be collected throughout the project years. Other data, which are available from local private and government organiza- tions, will also be utilized for comparison.

Although the evaluation team has the primary responsibility for the development and main- tenance of the MIS, it is a joint effort between evaluation and project staff to manage this sys- tem. Basic database software, including Microsoft Access and the Statistical Package for Social Science (SPSS), will be used.

Outcome Evaluation

Based on the project’s ecological systems social constructionist theoretical orientations and its focus on the individual, family, peer, and school domains, the outcome evaluation will provide better understanding to the three key evaluation questions: effectiveness of intervention strate- gies, ATOD use and family capacity, and replicability of project.

A sectional time series design with nonrandomized comparison groups, with age and gender cohorts over a period of 3 years, will be used to assess changes. Outcome objectives listed pre- viously, including family functioning and stability, academic readiness, family and school bond- ing, self-image, peer relations and support, physical and mental wellness, and ATOD use and other deviance behaviors, will be assessed. Variables such as frequency and types of interven- tion, duration and nature of participation in various interventions, and individual and family psy- chosocial histories are also assessed for their impact on the outcome of the project.

All project participants will be involved in this evaluation through three evaluation cycles (see Box 7.1). A pretest will be administered to all new participants (age 8 and above) during the first month of their enrollment; the posttest will be administrated 6 months later, or when the clients leave the program if their participation is more than 1 month; and the posttest 2 will be administrated on the twelfth month. A guided interview will be scheduled on the sixth month after the posttest. Minimally, changes over a period of a year will be assessed. Pre- and posttest measures will be implemented in two or more sessions over the period of 2 weeks to avoid wear- down. Project staff and volunteers will assist both children and adults individually to complete forms, pretest, and posttest to address literacy issues and to ensure correct understanding of the questions. For children younger than age 8, observations of their drawings and play will be used to assess changes. Assistance and training from art and play therapists will be used to develop project staff’s capability (see Box 7.1 for a schedule of evaluations).

Due to the transient nature of the target population, it will be difficult to have a stable inter- vention group or a matching group for study. Two approaches will be used to deal with this issue. First, 30 children who participate in project activities will be compared annually to an external comparison group of 30 students. This comparison group will be composed of students in the Johnson school who are not clients of this agency’s services but are comparable in general char- acteristics of children of the intervention group. Variables to be assessed will include academic readiness, peer relations and support, family and school bonding, physical and mental wellness, and ATOD use and other deviance behaviors. Approval for participation will need to be obtained from parents of students of both groups in order to conduct pre- and posttests. Second, intervention group participants who are involved in the project on a regular basis for at least 6 months will be compared to a comparison group consisting of participants who are involved in only a part of the project during a particular time (e.g., summer program) or only involved in regular programs for a period of 1–3 months.

106 CHAPTER SEVEN

For participants in the comparison groups who are 12 years of age and older, outcome assessment will also measure whether there are decreases in the incidence, frequency, and amount of ATOD uses and their related violence. Additionally, participants’ perception of the effects (harm) of ATOD uses and relationships to the total well being of the individuals will be evaluated.

Finally, a random subset of 30 youngsters (15, ages 6–11; 15, ages 12–14), 15 each for the first two years, will be tracked for 24 months to provide information on the long-range effects of the project.

Project participants will be given $5 to $10 or certificates to local food establishments as incentives to participate in each of the pre- and posttests and interviews.

Measures The reliability and validity of applying standard instruments to low-income, tran- sient, or homeless populations have been a concern for this project. Modification of these instru- ments for specific culture/class, age group, literacy level, and language poses the same concerns for validity; it is, however, a necessary step toward more culturally appropriate evaluation. The evaluation team plans to validate and revise all proposed measures during the first 6 months to ensure cultural appropriateness, content validity, and reliability.

Outcome forms being considered include:

● Index of Self-Esteem (Hudson, 1982), a 25-item measure on self-image, self-confidence, and self-satisfaction.

● Symptom Check List (SCL-90-R), a 90-item measure by Derogatis (1992) for psychologi- cal well-being.

● AIDS Risks Reduction (ARR), developed by Springdale County Health Department based on fact sheets provided by the Center for Disease Control and Prevention will be used for HIV and AIDS risk assessment.

● Self-Report Family Inventory (SFI), developed by the local Family Service Council, will be used to measure family competence, style, and relationships.

● Academic Self-Concept from the Effective School Battery (Gottfredson, 1982) will be used to assess attachment to school and education expectation.

● Substance Use Inventory (SUI) is a widely used self-report measure of the patterns of sub- stance use among youth (Skager, Fisher, & Maddahian, 1986).

● High Risk Behaviors Inventory (Jessor & Jessor, 1977) assesses frequency of high-risk behaviors.

● Social/Interpersonal Skills Questionnaire (to be developed by the evaluation team) is a 26-item measure that assesses social/interpersonal skills, shame, face, and community relationship.

PUTTING IT TOGETHER: MODEL PROGRAM AND EVALUATION PLANS 107

TABLE

7.1 SCHEDULE OF EVALUATIONS

Project Month 1st 6th 12th 18th 24th 30th

Cycle 1 Pretest Posttest 1 Posttest 2 Interview Interview Interview Cycle 2 Pretest Posttest 1 Posttest 2 Interview Interview Cycle 3 Pretest Posttest 1 Posttest 2 Interview

● Social Support Questionnaire Short Form (SSQ), developed by the Academic University Social Research Center, will help identify social support patterns.

● Family Stability/Homelessness Questionnaire (to be developed by the evaluation team) measures the length, type, and frequency of homelessness and its various impacts.

● Satisfaction Questionnaire (to be developed by the evaluation team) measures the degree of satisfaction of service and the effectiveness of services.

Implementation schedule, target respondents, time requirements, and purposes of these measures are described in Box 7.2.

Additionally, established assessment tools for programs and curriculum (e.g., the Nurturing Parenting Programs) used in the Healthy Families project will be used during program imple- mentation to measure effectiveness and appropriateness of these programs.

Data Analysis and Reports On a quarterly basis, information including descriptions and data collected will be reviewed and analyzed. Data will be entered into a state-of-the-art desk- top computer and analyzed with updated database and spreadsheet software (Access and SPSS). Descriptive statistics and inferential statistics, as well as analyses of variances, will be utilized to measure program effectiveness and impact. These analyses will determine if there are signifi- cant differences between comparison groups and between pre- and posttests. To assess overall program impact and its influence on prevention programs of other health care providers, meta- analysis and trend analysis will be conducted to evaluate the extent of the impact. During the first year, quarterly reports to the project management will be submitted. Starting in the second year, reports will be prepared on a biannual and annual basis. Biannual and annual reports will discuss problems encountered, options attempted, and solutions presented.

Evaluation Team

The evaluation team will be headed by Dr. Frank Kellen, a professor of social work at the Academic University in Springdale, CA. He received the university’s 2001 Outstanding Instructor Award for his excellent teaching in his graduate-level social research courses. He has many years of experience in working with high-risk populations, and more than 10 years of expe- rience in working with CSAP and its projects. Kellen was a principal investigator for a CSAP high-risk youth project and a Center for Substance Abuse Treatment (CSAT) outpatient treat- ment project. He also helped in the development and implementation of a CSAP violence pre- vention project. Kellen has been involved in various National Institute of Drug Abuse (NIDA) and CSAP research projects. Currently, he is a project evaluator for the Service U.S.A. program. Dr. Kellen will report to Ms. Lovejoy, chief executive officer of the agency.

The evaluation team will be a subcontractor of the project. It will include one part-time research associate who will be either a Ph.D. or doctoral student in human service area, and a part-time research assistant. The evaluation team will have regular biweekly meetings. It will also meet with project staff management once every 2 weeks prior to their meeting with the whole project staff. The evaluation team staff will attend initial project planning and implemen- tation meetings as well as selected program activities for observational measures.

Confidentiality

All information will be kept in the evaluation computer and in a locked file cabinet with access only by evaluation staff and the project director. All evaluation materials will be kept strictly con- fidential as required by the Federal Register, General Provisions Title 42, Chapter 1, Part 2. All

108 CHAPTER SEVEN

forms will utilize numerical codes for identification kept separately. Participation will be strictly voluntary and subject to a signed consent form that will identify all factors related to evaluation including the testing procedures. Consent is required from all parents and youth as well as fam- ily members involved in pre- and posttest assessments. Risks are not anticipated, but we will inform clients about the nature of instruments and the approximate length of time for test com- pletion. Please see the ethical considerations section of this proposal for further discussion of confidentiality and participant protection.

Model #2: Evaluation Plan for ABC Agency AmeriCorps Project

Objectives of the Evaluation Tasks 1. Provide ongoing process data for reporting and improvement. (Are we doing what we set

out to do?)

PUTTING IT TOGETHER: MODEL PROGRAM AND EVALUATION PLANS 109

TABLE

7. 2 IMPLEMENTATION SCHEDULE OF OUTCOME MEASURES

Time of Variable Data Length of Assessment (# of items) Source Admin. Measure

Pre and Post Self-image, Children and 10 min. Index of Self-Esteem satisfaction (25) Youth (C and Y)

ages 8–14

Pre and Post Health status (90) C and Y 8–14 20 min. Symptom Check List and Parents

Pre and Post HIV and AIDS (15) C and Y 8–14 10 min. AIDS Risks Reduction and Parents

Pre and Post Family style C and Y 8–14 15 min. Self-Report Family and competency (34) Inventory

Pre and Post School attachment C and Y 8–14 5 min. Academic Self-Concept and expectation (12)

Pre and Post ATOD use behavioral C and Y 8–14 10 min. Substance Use Inventory patterns (22) and Parents

Pre and Post Frequency of high- C and Y 8–14 5 min. High Risk Behaviors risk behaviors (14) Inventory

Pre and Post, Social skills, shame, C and Y 8–14 10 min. Social/Interpersonal Skills Interview support (26) Parents Questionnaire

Pre and Post, Social support (12) Parents 5 min. Social Support Interview Questionnaire Short Form

Pre, Stability, effects of Parents 15 min. Family Stability/ Interview homelessness (35) Homelessness

Questionnaire

Post, Satisfaction and Clients and 15 min. Satisfaction Questionnaire Interview effectiveness (35) Staff

2. Assess the outcomes/effectiveness of program activities. (How well do we do?) 3. Assess the impacts of the program. (So what?)

Information Management System

The project will develop a data collection system to store and compile quantifiable process and outcome data to evaluate the project. The project will also use specific forms for different objec- tives, a key informant survey, interview guides, a focus group guide, the meeting agenda and minutes, and other written reports to document qualitative and quantitative information.

The project will recruit a volunteer or part-time data processing person to collect, aggregate, and analyze program evaluation data. The project will approach local organizations or universi- ties to identify this assistant person.

Process Evaluation

Program Planning and Development This evaluation process includes compiling informa- tion on administrative planning, activity development meetings, member recruitment and train- ing, activity preparations, schedules, organization support, community linkages, problems encountered, and alternatives and solutions to problems. The project staff and members will col- lect activity logs, agendas, minutes, reports, and other documents as the basis for reports.

Program Interventions 1. Members and Partners Recruitment and Training. The project director will document

the recruitment and training of volunteer members and partner organizations. Feedback from members and partners on orientation and training will be collected through a post- event Training Evaluation Form.

2. Program Usage. The Volunteer Application Form and Volunteer and Service Recipient Sign-In forms will be used to document the extent of service used, achievements made, barriers faced, participant flow and penetration, and the characteristics of project participants.

3. Target Population. The project will collect and analyze target population data, including gender, age, race, and academic performance. Data will be collected through volunteer records, sign-in and sign-out forms, and information provided by partner organizations.

4. Community Linkage. The Community Encounter Form, community meetings, and activity records will track individual and community outreach. These records will include the date, person contacted, organization, content, frequency, and results of these efforts to assess the extent and effectiveness. Other community collaboration efforts will be tracked by meeting minutes, service contracts, memorandum of understanding, and the frequency and type of participation by various community members or agencies.

5. Project Objectives. Each objective and its evaluation strategies are detailed in the attached Objective and Evaluation Plan Form (as seen previously in Box 3.11).

Outcome Evaluation

The project will utilize the evaluation or assessment strategies described previously in the Project Objectives to provide process and outcome measures for each of the program objec- tives within the three main program categories. These categories are: (a) Getting Things Done, (b) Member Development, and (c) Community Strengthening.

110 CHAPTER SEVEN

“Getting Things Done” Category 1. America Reads Component Objective

a. Use the America Reads Participation Form (ARPF) to document the usage of tutoring services by students. These will include the number of hours of service provided, area of study, as well as frequency and comment of participation. If readily available and with proper consents, volunteers and program administration should consider collecting participating students’ scores on a standardized test used by the school at the beginning of the school year/semester and then at the end. Volunteers will also assess students’ per- formance on reading and other skills through observations.

b. Use the Tutorial Goal Information Form (TGIF) to document the extent of achievement of tutorial goals by each student. This information will compliment the narrative com- ments to provide a better assessment of students’ academic performance. Students are encouraged to be involved in identifying items included in the form that will assess them.

2. Volunteer Development Component Objective

a. Use the Positive Self-Development Form (PSDF) to assess the development of volun- teers over the three major areas of development every 4 months. These areas include Knowledge, Skills (Individual, Team/Leadership, Community/Planning), and Personal/Attitude. Volunteers will be involved in the development of the form.

b. Develop a Parent Satisfaction Survey form and a Teacher Satisfaction Survey form to document the level of satisfaction and suggestions for improvement for the project at the end of each academic year. Program directors will use a Training Evaluation Form as a sample to develop these Satisfaction Survey Forms. They should also use the forms (e.g., activity log) to document the extent of participation by parents and teachers.

“Members Development” Category a. Use the Member Development Portfolio Form during initial meeting/orientation, and

again at 6 months and at 12 months to develop baseline data and to assess AmeriCorps members’ developments in each of the areas identified.

b. Use a Leadership Assessment and Civic Involvement Scale at the beginning and the end of the program year to evaluate members’ growth in these two areas.

“Community Strengthening” Category Evaluation data will be collected through the use of the aforementioned forms and other exist- ing forms. Program staff will:

a. Use a Community Encounter Form, Training Evaluation Form, Volunteer Application Form, and other meeting minutes and materials to document the extent, type, frequency, nature, and results of project’s community contacts.

b. Use of focus group meetings and key informant interviews to collect qualitative data.

Impact Evaluation

Based on the project objectives and their process and outcome evaluation results, program staff will identify and assess the areas and the extent of program impacts. This assessment will be further supported by information collected through end-of-the-year focus-group meetings

PUTTING IT TOGETHER: MODEL PROGRAM AND EVALUATION PLANS 111

with specific target populations including community representatives, service recipients, and Agency ABC members. The program will use key informant interviews and a “Top 10 list”- styled approach to promote discussions.

Sampling Plan and Analysis Plan

The evaluation plan for this project may require simple sampling procedures. Sampling, how- ever, can be done on a three-tier level, if the program wishes to draw a sample to evaluate any portion of the program. This will include: (a) all participants for a particular activity, (b) partici- pants who attend activities consistently, and (c) a percentage of the consistent participants who are selected by a random approach.

Basically, the analysis plan the program needs to develop an information management system/database to store evaluation data. The program should update all process data on a reg- ular basis. Single-variable analysis, measurements of central tendency and dispersion (e.g., fre- quency distribution, percentage, mean, mode, median, minimum, maximum, and range) could be used to interpret data collected. Outcome data will be collected through the aforementioned forms. Analysis of changes can be achieved through basic observation and calculation of increase or decrease, single variable analyses, and central tendency measurements and comparisons.

Implementation of Evaluation 1. The program will contact local organizations and universities to recruit a student who will

assist in the data input and analysis. 2. The program should develop a functional database to store and compile quantifiable

process and outcome data to evaluate the project. 3. The program will provide specific orientation sessions to AmeriCorps members on the

purpose and the implementation of evaluation. It will also schedule specific times during each week for members to complete evaluation tasks.

4. Develop the sense of ownership and buy-in by the partner agencies, members, and volunteers.

Model #3: Confidentiality and Ethnical Considerations

New Horizon Association has long-established stringent guidelines regarding confidentiality of all records and charts of clients. Policies and procedures are written, reviewed on an annual basis, and subject to change following current health and mental health licensing procedures of Noble County, the State of Massachusetts, Center for Substance Abuse Prevention, and all fed- eral guidelines.

Target Population 1. Project Hope will target Vietnamese, Cambodian, Filipino, and Amerasian adolescents,

ages 13–21, in South Worthington who have misused and/or abused alcohol, tobacco, and other drugs. Asian Americans account for 7% (7,000) of the county’s population (roughly 100,000), and 60% (4,200) of them live in the city of Worthington. Asian Americans make up 15% of the city’s population (28,000). Vietnamese (2,000) and Filipinos (1,500) are the two largest Asian groups in the city, while Cambodian (500) and Amerasian (200) are the often-neglected groups due to their smaller populations. About 12% of the targeted ethnic

112 CHAPTER SEVEN

groups are adolescents between the ages of 13-21; among them 54% are male and 46% are female. More than half of the target populations are immigrants or refugees. Detailed descriptions of the target population based on socioeconomic status, culture, and other factors are reviewed in the Target Population Section in the proposal.

2. Criteria for inclusion of participants into the substance abuse treatment component of Project Hope include: (a) being Asian-American residents of Noble County age 13–21, (b) having presenting problems of substance abuse, (c) determination of relevance based on the required assessment tools, (d) willingness to commit to entering and participating in the treatment program, (e) willingness to obtain a general medical physical, and (f) demonstration that there is no proclivity to violence. Services at the program’s Wellness Center are open to all clients interested in receiving medical services. Clients with a dual diagnosis of mental illness and substance abuse will be served in conjunction with the agency’s Mental Health Treatment Program.

Applicants may be found ineligible when they are physically/medically unsuitable for the services, are prone to violent behaviors, or have a history of arson.

Participant Recruitment and Selection 1. Criteria for inclusion or exclusion. See the Target Population section in this proposal. 2. Clients for Project Hope will be recruited through various New Horizon Association pro-

grams, local health care providers, government agencies and related programs, education and ethnic community groups, and self-referrals. Project staff will conduct outreach to tar- get ethnic groups to provide community education and recruitment. New Horizon Association’s current clients outreach and recruitment policies will be used and revised for the proposed project.

3. Participation in this project is strictly voluntary. Clients can withdraw from the program at any time. Court-ordered individuals are also free to leave the program. However, the pro- gram may be required to inform proper criminal justice authorities regarding these client’ departures. Individuals who are involuntarily discharged from the program may appeal to the principal investigator or executive director for a review of the decision.

Data Collection ● Data will be collected from participants, their parents, and staff by means of intake

records, drug abuse assessment instruments, written questionnaires, and service logs. ● Detailed description on individuals from whom data will be collected is discussed in this

proposal’s Evaluation section. ● Copies of standard data collection instruments that will be used for this project are provid-

ed in the Appendix.

Privacy and Confidentiality

The current New Horizon Association Personnel Policies, Outpatient Mental Health Program Policies, and Substance Abuse Treatment Program Policies all contain procedures that ensure compliance with local, state, and federal laws and regulations on clients’ privacy and confiden- tiality. These confidentiality policies also regulate access of all charts, files, and information regarding clients. They will be extended to Project Hope.

PUTTING IT TOGETHER: MODEL PROGRAM AND EVALUATION PLANS 113

The policies and all corresponding procedures are in accordance with the Federal Register, General Provisions of Title 42, Chapter 1, Part 2. All clients are required to sign a consent form indicating their willingness to participate in the treatment program prior to their enroll- ment. Clients will also be requested to sign a second consent form to indicate their participa- tion in the collection of evaluative data for the program. However, no clients will be refused for services if they are unwilling to take part in the evaluation. Court ordered clients will be required to sign an additional consent form for the program to inform proper criminal justice authorities on their status.

Participation in the program and its evaluation is voluntary. Clients will be informed of their rights, both verbally and in writing, as well as the voluntary nature of their participation. They will also be informed about their rights to revoke their consent and that no privileges will be denied if they refuse to consent.

The consent forms state who is releasing the information, who will be receiving the infor- mation, and what specific information will be disclosed for what purpose. The consents are time limited, and clients can revoke their consent at any time. Both the client and the program staff need to sign and date the consent forms.

All client charts and records will be kept in separate locked files and are stored in a locked room accessible only to staff with clear sign-out privileges. No charts should be kept on staff desks or shelves, unless being actively used.

Through a coding system, client data will be stored and used for evaluation and to ensure privacy and confidentiality. Identifiers will be stored separately from the data, and only the prin- cipal investigator and the principal evaluator will have access to the identifiers.

Protection from Potential Risks 1. Although potential risks are not anticipated, project staff should adhere to the client confi-

dentiality and protection procedures and policies to ensure the well-being of the clients. 2. Upon hiring, all project staff are required to be trained on confidentiality guidelines and

are provided copies of the procedures and the code of Federal Regulations (42 CFR Part 2). Issues related to client protection and confidentiality will be discussed at least twice a year during staff training.

3. All New Horizon Association staff are trained to provide proper intervention in the event of adverse effects to participants. Additionally, they will be supervised by the agency’s licensed clinical social workers, psychiatrists, and psychologists. These professional staff members are also available to provide additional care if needed.

Consent Procedures 1. Prior to acceptance into the program, clients will be informed, both in English and in their

primary language, about the nature and purpose of their participation in the program and its evaluation component. Procedures that safeguard their privacy and confidentiality will be presented. The voluntary nature of their participation, their rights to withdraw from the program at anytime without prejudice, the potential risks, and the use of data collected through this project will also be discussed. Limitations on confidentiality for adolescents and court-ordered cases will be explained.

2. Informed consent will be obtained from the participants and, for clients under 18, their parents or legal guardians. Participants and their parents/guardians are required to sign a consent form prior to their enrollment into the program and an additional consent form

114 CHAPTER SEVEN

for participating in the evaluation. Consent forms are read to the participants and to the parents/guardians both in English and in their native language to ensure they understand the forms. Copies of the signed consent forms will be given to the participants and their parents/guardians. A copy of the sample blank consent form for this project is provided at the end of the chapter.

3. Separate consent forms will be obtained for the collection of evaluation data, in addition to the consent forms for participation in the services of the Wellness Center and the treat- ment program. Court-ordered individuals will also be asked to sign a separate consent form that allows the program to report their status and progress to the criminal justice personnel. Individuals not consenting to the collection of individually identifiable data for evaluative purposes will be permitted to participate in the program.

Sample Consent Forms The following samples include consent forms for both children and adults for participating in programs and evaluation.

● Sample Program Participation Consent Form for Children and Youth (Box 7.3) ● Sample Evaluation Consent Form for Children and Youth (Box 7.4) ● Sample Program Participation Consent Form for Adults (Box 7.5) ● Sample Evaluation Consent Form for Adults (Box 7.6)

References Derogatis, L. R. (1992). Symptom Checklist-90-Revised. Minneapolis, MN: NCS Pearsons. Gelber, J., & Specter, P. D. (1987). Psychotherapy: Portraits and fiction. Northvale, NJ: Jason

Aronson. Gergen, K. J., & McNamee, S. (1992). Social constructionism in therapeutic process. London:

Sage. Germain, C. B., & Gitterman, A. (1987). Ecological perspective. In A. Minahan, et al. (Eds.),

Encyclopedia of social work (18th ed., pp. 488–499). Silver Spring, MD: National Association of Social Workers.

Gottfredson, D. C. (1982). Handbook for evaluating drug and alcohol prevention programs. College Park, MD: University of Maryland, Institute of Criminal Justice and Criminology.

Hartman, A., & Laird, J. (1983). Family centered social work practice. New York: Free Press. Hoffman, L. (1990). Constructing realities: An art of lenses. Family Process, 29(1), 1–12. Hudson, W. (1982). Index of Self-Esteem (ISE) in The clinical measurement package: A field

manual. Homewood, IL: Dorsey Press. Jessor, R., & Jessor, S. (1977). Problem behavior and psychological development: A longitudi-

nal study of youth. New York: Academic Press. Meyer, C. H. (1983). Clinical social work in the eco-systems perspective. New York: Columbia

University Press. Pardeck, J., & Yuen, F. K. O. (Eds.). (1999). Family health: A holistic approach to social work

practice. Westport, CT: Auburn House. Skager, R., Fisher, D., & Maddahian, E. (1986). A statewide survey of drug and alcohol use

among California students in grade 7, 9, and 11. Sacramento: Office of the Attorney General, Crime Prevention Center.

Von Bertalanffy, L. (1968). General system theory. New York: Braziller.

PUTTING IT TOGETHER: MODEL PROGRAM AND EVALUATION PLANS 115

116 CHAPTER SEVEN

BOX

7.3 SAMPLE PROGRAM PARTICIPATION CONSENT FORM FOR CHILDREN AND YOUTH

Project Hope Children And Youth Program Consent

I, ____________________ (print name), understand that the Project Hope is a demonstration project to improve family functioning and to prevent substance abuse related problems for children and families in City of Worthington. My parent/guardian will need to give permission for me to be in the program. I also understand that my participation is strictly voluntary and I may withdraw from any Project Hope activities at any time. Information that I share during my participation in the project is considered confidential.

I understand the rules and regulations of the project and my responsibilities with regard to the project, and I agree to abide by them. My signature below indicates my complete understanding about the Project Hope and my consent to participate in any project activities.

(Participant’s Signature) (Date) (Staff Name and Signature)

* * * * * * * * * * * *

I, __________________ (print name), parent/guardian of _________________ (print name), read and under- stand this consent form and give my permission for my child to participate in the aforementioned activities.

(Parent/Guardian’s Signature) (Date) (Staff Name and Signature)

PUTTING IT TOGETHER: MODEL PROGRAM AND EVALUATION PLANS 117

BOX

7. 4 SAMPLE EVALUATION CONSENT FORM FOR CHILDREN AND YOUTH

Project Hope Children And Youth Evaluation Consent

As part of a demonstration project, New Horizon Association staff and the evaluation team will evaluate the effectiveness and benefits of the various Project Hope activities. I understand the answers that I give to the evaluators will be totally confidential. My parent/guardian will need to give permission for me to be in the program, but no one from my family and my school will see the information that I give. In addition to my information are to be grouped with others in the final report; I will be given an ID# so that I will not be identified by name in any report. However, project staff and evaluation team reserve the rights to reveal information that they have the legal mandates to report, such as child abuse and immediate danger to self and others. My participation in the project is strictly voluntary. I am free to refuse to participate or with- draw at any time. I will be asked periodically to complete questionnaires or be interviewed during and after my participation in the project.

My signature below indicates my complete understanding about the evaluation component of the Project Hope and my consent to participate in any project evaluation activities.

(Participant’s Signature) (Date) (Evaluator’s Name and Signature)

* * * * * * * * * * * *

(Parent/Guardian’s Signature) (Date) (Staff Name and Signature)

118 CHAPTER SEVEN

BOX

7.5 SAMPLE PROGRAM PARTICIPATION CONSENT FORM FOR ADULTS

Project Hope Program Participation Consent Form for Adults

I, __________________ (print name), understand that the Project Hope is a demonstration project to improve family functioning and to prevent substance abuse related problems for children and families in the City of Worthington. I also understand that my participation is strictly voluntary and I may withdraw from any Project Hope activities at any time. Information that I share during my participation in the project is consid- ered confidential.

(Participant’s Signature) (Date) (Staff Name and Signature)

BOX

7.6 SAMPLE EVALUATION CONSENT FORM FOR ADULTS

Project Hope Evaluation Consent Form for Adult Participants

As part of a demonstration project, The New Horizon Association staff and the evaluation team will evalu- ate the effectiveness and benefits of the various Project Hope activities. I understand the answers that I give to the evaluators will be totally confidential. No one from my family and community will see them. In addition to my information are to be grouped with others in the final report; I will be given an ID# so that I will not be identified by name in any report. However, project staff and evaluation team reserve the rights to reveal information that they have the legal mandates to report, such as child abuse and immediate dan- ger to self and others. My participation in the project is strictly voluntary. I am free to refuse to participate or withdraw at any time. I will be asked periodically to complete questionnaires or be interviewed during and after my participation in the project.

My signature below indicates my complete understanding about the evaluation component of the Project Hope and my consent to participate in any project evaluation activities.

(Participant’s Name and Signature) (Date) (Staff Name and Signature)