Program evaluation

HS 3083 Program Evaluation Project Guidelines -1-

HS 3083

Program Evaluation Project Guidelines

For the HS 3083 final project, students will work in groups of 4-5 to conduct an evaluation of a health education/ promotion activity or program. This is a mock project in, you will not actually implement and evaluate a health promotion program. I want to emphasize that this course project is a reflection of the work accomplished as a group. Failure to contribute a fair share of the workload to the project reflects a lack of professionalism and integrity and could result in removal from the team and/or a loss of points. I expect each student to be a strong and collegial team player – no exceptions.

I suggest you evaluate the intervention you created in HS 3073 Program Planning. Because this is a group project—group members would need to agree on which intervention to use in this mock program evaluation project. As an example, you could develop a simple knowledge test comprised of 10 questions; the easiest step to take is to construct 10 multiple-choice and true-false items. This knowledge test will be used as a pre- and post-test to measure your impact objective. In other words, to measure whether your program impacted participants’ knowledge, you will administer this knowledge test before and after you implement your health promotion program. You may create the participant answers, or data, to complete the project.

This project is designed to resemble a similar project completed in HS3073 Program Planning. Student groups are encouraged to use work from HS3073 as the foundation for this project. Do not plagiarize yourself by re-using the program planning project. You can and should base this evaluation project on the program planning project. Do this by using the same population and/or health issue, the same stakeholders, the same over-arching goals, and the evaluation plan you identified. However, you need to modify and expand those elements based on what you learn in this course so they are new and not simply copied/pasted.

Each project team will complete three draft reports – each composing a portion of the program evaluation process. The final program evaluation report will include all components from the revised draft reports plus additional sections (see guidelines for more detail).

Important Note: This final evaluation report is important for you to include in your respective e-portfolios because it is the primary artifact that demonstrates your key learnings in HS 3083 Program Evaluation in Health Promotion. Therefore, take pride in your work.

Content and Format Specifications

1) Document Format: Submit each paper as a Word document.

2) Typing Requirements: The length of each section of the paper will vary, depending on the required contents. Use 12-point font and double spacing ; set margins at 1 inch on each side (top, bottom, left, and right). Be sure to include page numbers. See the APA Publication Manual (7th ed.) for guidelines regarding tables, figures, graphs, and appendices.

3) Content: The length of each draft report will vary, depending on the contents covered in the report. In general, each section probably will not exceed 4-5 double-spaced pages in length. However, this is merely an estimate. You will revise those draft reports to compose your final project report, which generally does not exceed 25 double-spaced pages (not including title page, table of contents, executive summary, references, or appendix material). Each paper must include the content specified for each of the sections listed in the guidelines. Use headings/sub-headings to delineate content for each section - Do not make me guess what you are addressing within the report; it should be very clear.

4) Writing Mechanics: In order to be an effective program evaluator, you need to be a clear thinker and writer. Therefore, writing mechanics matter. I will deduct points for writing errors, such as misspelled words, sentence fragments, run-on sentences, disorganized thoughts, lack of flow, etc. I recommend that you read Chapter 3 – Writing Clearly and Concisely in the APA Publication Manual (7th ed.)

5) Formatting and References: All in-text citations and the reference list must adhere to APA format – no exceptions. Also, tables, figures, and appendices should be formatted according to APA. See the APA Publication Manual (6th ed.) for details. Be careful not to plagiarize.

6) Cover Page: Each draft report should include a cover page with the following information:

· Draft Report #___

· Student Names

· Course no. and title, semester/year

· Instructor’s name

· Date of submission

Important Note: Use the future tense when writing the evaluation draft reports because you are addressing what you plan to do. Then, when you write the Results section of your final report, you need to use the past tense because you are reporting results of a project that has been completed.

Draft Report #1

Introduction to the Project (1-2 pages)

In this section, you will set the stage for the evaluation project by describing the key health issue that the health education/promotion program or event addresses. For example, if the program being evaluated is offered by the American Heart Association, you should discuss the magnitude of heart-related illness and death. As addressed in HS 3053 and HS 3073, it is helpful to explain the impact of the key health issue in an inverted triangle fashion (nationally, statewide, and locally). It is important that you depict how the key health issue specifically affects your particular target audience.

Program Description and Context of Evaluation (2-3 pages)

In this section, you will write a description of the program you plan to evaluate and a rationale for the evaluation. This discussion should be as brief as possible, but the reader must get a clear understanding of the program you plan to evaluate and why there is a need to evaluate it. In this section, provide a description of the program’s mission statement, goals, and objectives along with a succinct description of the implementation plan.

Note: You already should have this information from your Program Planning Project. However, you will need to condense it and make other necessary revisions for this particular part of the Program Evaluation Project. Pay particular attention to how you write the goal(s) and objectives. Make sure that you write the objectives in clear and measurable terms by using the SMART (or SMART-C) format. For more information about writing clear goals and objectives, see the “Final Project” tab in the main course menu.

Draft Report #2

Evaluation Questions and Program Objectives (~1 page)

Evaluation questions are basically research questions, and they directly connect to the program goals and objectives. To help you determine what you need to evaluate, identify the program goals and objectives. If you do not plan to evaluate the intervention created in HS 3073, you need to identify the goals and objectives for another program you have selected. Once you have identified the program objectives, then you can formulate your evaluation questions.

Examples of program evaluation questions, the corresponding program objectives (process, impact, outcome), and a potential evaluation method are listed below. The project methods will be described in the next section of the paper; however, it is helpful for you to see the evaluation measure matched to the evaluation question and program objective. You can list your objective/question/measure in a similar fashion within your narrative. Some of you may have more than one program goal. However, for the purposes of this project, I recommend that you choose one goal that represents the overall aim of your project. Then, write one process, one impact, and one outcome objective that are linked to that goal. (You already did this in HS 3073; however, many of you probably need to revise your objectives to ensure that they are written in SMART format.) Next, for each objective, write an evaluation question and state an evaluation measure that you can use to assess whether you achieved the particular objective. Group the objectives/questions/measures together in the narrative (see examples below).

Tip: Write the process objective to connect with a sign-in sheet or some other type of tally sheet. Write the impact objective to represent an increase in health knowledge. (See student examples under the Final Project tab).

Examples from a Student Project:

Process Objective: The program committee will deliver 50 program pamphlets to all 41 high schools in Dallas, Texas.

Evaluation Question: Did the number of program participants increase after the pamphlets were delivered to the additional high schools?

Evaluation Measure: To count the number of attendants at the initial meeting and compare it to the count of attendants at the second meeting to see if there was an increase in participation

Impact Objective: After participating in a school wide pregnancy prevention program, 25 students will increase their knowledge of how to prevent teen pregnancy by 80%.

Evaluation Question: Was knowledge regarding teen pregnancy increased after the seminar was conducted?

Evaluation Measure: Conduct a pre-seminar survey and a post-seminar survey to determine if knowledge regarding teen pregnancy was increased after the seminar was conducted.

Outcome Objective: Within the next 10 years, the DISD will achieve a 5% reduction of teen pregnancy.

Evaluation Question: Was there a decrease in teen pregnancy rates for the city of Dallas.

Evaluation Measure: Research teen pregnancy rates by utilizing sources, such as Centers for Disease Control and Prevention (CDC) along with Texas Department of State Health Services.

Draft Report #3

Methodology (2-4 pages)

This section should be a narrative of the evaluation design and process. Include the following information:

a. Evaluation Design: What type of evaluation will you conduct? What type of evaluation design do you plan to use? (Refer to Ch 3 in the text.)

b. Sampling Design and Description: What sampling design will you use? For example, will you use non-probability sampling, such as a convenience sample, snowball, etc.). Or will you use probability sampling (e.g., simple random, cluster, etc.)? In addition, provide a description of your evaluation participants. For the purposes of this project, you need to arrive at a sample of approximately 20-25 people.

c. Instrumentation: Describe the instruments that will be used to collect the evaluation data. You will need instruments to measure one process, one impact, and one outcome objective. For example, to measure your process objective, you can use a sign-in sheet (see examples under the Final Project tab). To measure the impact objective dealing with knowledge improvement, construct a 10-item knowledge test that represents what you want your program participants to learn. (Or, find an existing instrument that is appropriate to use.) Make sure you write the 10 items in objective format (e.g., multiple-choice and true/false items). Then, you also will need some type of instrument to measure your outcome objective. If you plan to use an existing instrument, discuss where you located the instrument and any challenges you encountered in finding an appropriate one. Also, you must cite the original source of the instrument in APA publication style (6th ed.). If you plan to create your own instrument, briefly discuss the instrument development process.

d. Reliability and Validity: Indicate the reliability and validity of the instrument you plan to used if that information is available and/or applicable. For example, if you’re constructing your own knowledge instrument, speak to how that instrument is valid and reliable for measuring pre- and post-intervention knowledge about the particular health topic or issue.

e. Data Collection Methods: Do you plan to use existing (secondary) data, or do you plan to collect new data? How will you collect your data (survey, face-to-face interview, telephone interview, etc.)? When will you distribute the instrument? How many questionnaires/surveys will be distributed and/or completed? For this section, be sure to describe the data collection methods in detail. Also, if you will be using human subjects, you must explain the process you plan to use for informed consent. Include a copy of the consent form in the appendix of the final report.

f. Data Analysis Techniques: Discuss how you plan to analyze your data. In most cases, frequency statistics will be sufficient for your analysis.

Final Program Evaluation Report

For the final report, include all of the revised sections from the draft reports plus the additional sections described below. Before writing the final report, I recommend that you review Ch 11 in your text (Writing a Report) as well as the student example, which is located under the Final Project tab of the course website.

Important Note: Remember to write your final evaluation report in past tense because you have completed the program evaluation.

After revising the three draft reports, add these sections to the final report:

Results (2-4 pages)

Summarize the data collected from your evaluation, but do not discuss the implications. (You will do that later.) The Results section of your report should concentrate on general trends and differences and not on trivial details. This section should be written in narrative form and include tables and figures or graphs. (Data included in a table should not be duplicated in a figure or graph.)

Tables and figures should be self-explanatory, i.e., the reader should be able to understand them without referring to the narrative. However, you must specifically refer to tables and figures in the narrative. For example:

Figure 1 shows that the activity decreased after five minutes.

The activity decreased after five minutes (see Figure 1).

Each table and figure should be numbered separately, contain a descriptive title, and include a legend that identifies symbols, abbreviations, or special methods used. Label all columns and rows in tables and axes in figures. (See the APA Publication Manual [6th ed.] for more information regarding how to format tables and figures.)

Discussion of Findings (1-2 pages)

This section is not a restatement of the results, but rather an interpretation of the results. In this section, include any limitations to your study that may have affected the results (e.g., missing or incomplete data). In addition, discuss whether the program achieved the stated goals and objectives; and provide reasons for the program’s success or failure. You also can report data trends in this section. For example, perhaps there is a finding that is not significant, but shows a trend towards significance that you believe is important to mention. Also, briefly explain how the evaluation findings connect to theory and knowledge in the field of health education/promotion. Phrased another way, discuss the implications for health education/promotion practice or policy.

Conclusions and Recommendations (1-2 pages)

In this section, summarize the program evaluation process, results, and discussion in 2-3 paragraphs. Then, discuss recommendations for improving the health education/promotion program and the evaluation process (for those who may attempt to evaluate this program in the future). Make any other recommendations you deem relevant (e.g., the need for further research in a particular area).

Components of the Final Evaluation Report

Title Page – 1 page

Please include the following information, centered on the page:

· Title of Project

· Student Names

· Course no. and title, semester/year

· Instructor’s name

· Date of submission

Table of Contents – 1 page

List each section title and corresponding page number.

Executive Summary (Abstract) – 1 page

Provide a brief overview of your report. Highlight the “what” and “why” and any major findings that resulted from the evaluation. Also include major conclusions and recommendation. The Executive Summary is often read by someone who does not have time to read the entire evaluation report, such as a representative from an external funding agency or another stakeholder. Level I headings (centered and bold) for your final project:

Introduction to the Project: 1-2 pages

Program Description and Context of Evaluation: 2-4 pages

Evaluation Questions and Objectives: ~1 page

Methodology: 2-4 pages

Results: 2-4 pages

Discussion of Findings: 1-2 pages

Conclusions and Recommendations: 1-2 pages

Reference Page – Typically 1 page (Be sure to start this as a new/separate page.)

Appendices (you may not have all of these for your specific evaluation project) - Page numbers will vary (Each appendix has its own title page; refer to APA manual.)

· Timeline – Provide a Gantt chart of the timeline for your evaluation project.

· Budget – List the costs for conducting your evaluation. If “in kind” donations of materials, photocopying, mailing, etc. were provided, please indicate this in your budget instead of a dollar amount.

· Copy of Instruments

· Consent Form

· Contract (if applicable)

· Any other appendix material that is relevant to your project

1