WHP Porject

Identify everyone in “Your Workplace” who has a stake in knowing the results of your program evaluation:

Goals for Evaluation (Evaluation planning should be closely tied to the development of a program’s goals and objectives).

Two types of goals of greatest interest to stakeholders:

1) Health related goals

2) Financial goals

Evaluation Categories:

Process Evaluation: Focuses on what happens during a program and specific aspects of the intervention that may need attention. Process evaluation can be qualitative (narrative) or quantitative (numbers) questions.

· Qualitative issues: program delivery, program registration, educational content and instructor effectiveness.

· Quantitative issues: financial or attendance numbers, etc.

Impact Evaluation: Assess the overall effectiveness of a program in producing desirable levels of knowledge, attitudes, behaviors, health status and skills among participants. It measures changes in participants’ behavior and attitudes and in organizational culture.

Outcome Evaluation: Quantifies the financial consequences that may occur when an intervention affects absenteeism, productivity, health care utilization etc. Outcome evaluation addresses the question: Did the intervention influence the changes that occurred and, if so, was the cost justifiable?

Measurement Methods for Evaluations

· Process

· Short questionnaires

· Focus groups

· Individual communications / Interviews

· Break-even analysis

· Impact

· Surveys or Questionnaires

· Health-risk appraisals

· Health care claims data reports

· Occupational injury reports

· Absenteeism logs

· Culture audits

· Productivity reports

· Outcome

· Health-claim cost reports

· Monthly expense reports

· Quarterly productivity reports

· Cost-effectiveness analysis

· Benefit-cost analysis

· Break-even analysis

The structure of your evaluation plan is influence greatly by the scope and specificity of your evaluation needs:

Scope: The range of variables and the time frame of your evaluation. (Ex. An evaluation that consists of several outcome variables that are measured over a one-year time frame has a broader scope than an evaluation that has one or two variables spanning six months). The number of variables and the time frame will define the scope of your evaluation.

Specificity: The level or degree of precision that evaluators use in measuring selected variables within an evaluation. (Ex. An outcome variable that is referred to general as absenteeism has virtually no level of specificity, due to the lack of specific type of absenteeism. Lower-back-absenteeism has some level of specificity because it specifies the cause. High levels of specificity are preferred as it gives the evaluators greater opportunity to study the real impact of an intervention.

Evaluation Design

(Non-experimental design will be the only design covered in this module)

Components of this non-experimental evaluation design:

· Experimental group (E) / the group of employees that participate in the program.

· Observation (O) / when a measurement is performed.

· Independent variable (X) / the program intervention designed to produce a positive outcome.

One-group pretest & posttest design

Time Series design

Economic-based Evaluation

Break-Even Analysis (BEA) / Also known as a cost-volume-profit analysis or contribution analysis. A forecasting tool to guide decision-makers in allocating appropriate levels of resources.

Steps in conducting a BEA (text pg. 162)

1) Identify, measure and calculate all costs items

2) Add fixed and variable costs / Total costs = fixed costs + variable costs

3) A) Determine a baseline for predicting future costs (Use representative past data) B) Determine a benefit variable to compare against fixed and variable costs identified in previous step (Should reflect one or more primary goals of the intervention)

Example of a comparison of lower-back injury claims and costs

*Based on initial 20% influence

**Based on past trend of 5% increase per year

Comparison of claims and costs for an issue at ”Your Workplace”

Steps in conducting a BEA (continued)

4) Calculate projected benefits (in dollars) based on any cost reduction

Example of Projected-Generated Savings

Projected-Generated Savings for “Your Workplace”

Evaluation Plan Decision Point Framework

After reviewing this framework for an evaluation plan, fill out the following worksheet for your workplace

Evaluation Plan Decision Point Framework for “Your Workplace”