READ ATTACHMENTS FIRST (before bidding)
appletree123
Process_EvaluationLectureSlides.pptSchool of Management
Program Evaluation
MPA 513
Week 3
School of Management
- Policy in the News
- Review Needs Assessment / Stakeholders
- Process Evaluations
- Exercise:
- Performance Monitoring
- Exercise: City Stat example
- Questions and Conclusions
Class Overview and Objectives
*
School of Management
- In the News
- Public Administration
Evaluation in the News
*
School of Management
- Logic Models
- Stakeholders
Review
*
School of Management
Involving Stakeholders
- Gain broader perspective, avoid blind spots, try to ensure utilization of results
- Key stakeholders:
- Those served or affected by activity
- Those involved in program operations
- Those in a position to make decisions about the activity
- For a manageable process, the list of stakeholders must be narrowed to primary intended users
School of Management
Evaluating Internal Processes
“Now that this is the law of the land, let’s hope we can get our government to carry it out.” John F. Kennedy
School of Management
What is a Process Evaluation?
Process (formative) evaluations are aimed at enhancing your program by understanding it more fully, and whether it is functioning as intended.
Process evaluations study what is being done, and for whom these services are provided
*
Evaluators often distinguish between process/implementation/formative vs. outcome/impact/summative evaluations.
School of Management
Process vs. Outcome Evaluation
Process (Formative) – program managers, front-line staff, program designers, evaluation professionals and other internal and external entities focused on wanting to know why the program (or class of programs) is or is not working and what sort of program adaptations are appropriate.
Outcome (Summative) measures – legislators, accounting entities, interest groups, other levels of government, and other external entities focused on accountability or accreditation.
Evaluators often distinguish between: process or implementation (formative) vs. outcome or impact (summative):
*
School of Management
Illustration of Process Evaluation
Formative
Evaluation
Research
Examines
Inside The
Program
At “The
Process”
(1)Jablonski, J.R. Total Quality Management. Technical Management Consortium Albuquerque, NM.
*
School of Management
Organization Change and Process Evaluation
- Process evaluation supports a program administrator’s desire to correct program deficiencies.
- Problem-solving orientation is different from evaluations that are more outcome-focused.
- Process intervention model provides a framework for planned organizational change.
“We are interested not so much in whether X causes Y as in the question , if Y is not happening, what is wrong with X.” –Sylvia et al, p.70
* Sylvia, Sylvia, and Gunn. 1997. Program Planning and Evaluation for the Public Manager. Waveland Press.
*
School of Management
Activities
Inputs
Outputs
Intermediate Effects/ Outcomes
Short-term Effects/ Outcomes
Long-term Effects/
Outcomes
Context
Assumptions
Stage of Development
Illustration Using the Logic Model
(1)CDC Presentation on Process Evaluation: www.cdc.gov/vaccines/programs/progeval
*
- Lots of this will have been decided when you “staked the claim” but worth considering again.
- How far out the chain is determined by several considerations. These are not a formula that will produce a result, but rather separate conversations which may lead to contradictory conclusions. In that case, then a need to reconcile where to put the resources.
- Key questions include:
- Ripple effect or reasonableness: This one should have been answered in “staking the claim”. The question is how far out the chain of causation is it reasonable to expect the intervention to have an impact. Through the process of refinement in staking the claim, the program should have made decisions about synchronizing the level and mix of activities and the desired “claim.”
- Accountability: This is what the major stakeholders are expecting of you. As mentioned, they might be “wrong” and err in either a too ambitious or not ambitious enough direction. Still, in designing an evaluation, the information of most importance to the stakeholder is a major consideration.
- Measurability: As mentioned, this should never be the reason to scale back an evaluation, except as an admission that NOW we are unable to measure this effect.
School of Management
Activities
Inputs
Outputs
Intermediate Effects/ Outcomes
Short-term Effects/ Outcomes
Long-term Effects/
Outcomes
Context
Assumptions
Stage of Development
What the program and its staff actually do
Results of activities: who, what will change
*
School of Management
Activities
Inputs
Outputs
Intermediate Effects/ Outcomes
Short-term Effects/ Outcomes
Long-term Effects/
Outcomes
Context
Assumptions
Stage of Development
Sphere of control
Sphere of influence
*
School of Management
Activities
Inputs
Outputs
Intermediate Effects/ Outcomes
Short-term Effects/ Outcomes
Long-term Effects/
Outcomes
Context
Assumptions
Stage of Development
Outcome evaluation
Process
evaluation
*
School of Management
Why Study Implementation?
- Implementation is the most over-looked phase of the process approach to program change
- Change is frequently resisted and nearly always disruptive
- Critical components of planned change:
Clear agenda and chain of command;
Proper specificity for assigned responsibilities; and
System for managing and monitoring implementation
*
School of Management
When Can Process Evaluation be Useful?
- When expected outcomes are not observed, process evaluation can suggest reasons.
- When outcomes are positive, process evaluation may help understand which components are especially important.
- When an activity is newly implemented and it is too early to expect changes in outcomes.
- When an outcome evaluation is not feasible due to resource constraints.
School of Management
Look at Coverage and Bias
- How many persons are receiving services?
- What proportion of those who might need the service(s) actually used it/them?
- Has the service, or network of services served the intended clients?
- What were the demographics and clinical characteristics of clients? Are certain groups over/under represented? Is there creaming?
- What proportion of clients completed treatment (or received full services) and what were the characteristics of those who dropped out?
- Are clients aware of the program?
Coverage deals with whether the program is reaching the appropriate population, and answers such evaluation questions as:
*
In general, process evaluations pose questions about coverage and process
School of Management
Common Coverage Problems
- “Self-selection” – bias in coverage due to voluntary over-participation or “creaming”
- Under-coverage due to limited funds, poor public education, poor access, or poor design – e.g., population too broadly defined, such as all poor mothers
- Over-coverage – reaching more than the intended group, e.g., Sesame Street, or special education
*
School of Management
Look at Organizational Function
- Are they receiving the proper amount, type, and quality of services? What are waiting times?
- Is staffing sufficient in numbers and competencies for the function?
- Is the program well-organized? Do staff work well together?
- Are resources, facilities, and funding adequate?
- Is the program in compliance with applicable professional and legal standards?
- Are participants satisfied with services?
- Is there appropriate discharge and follow up?
Process questions deal with whether service delivery and support functions are consistent with program design specs and standards:
*
School of Management
Common Implementation Failures
- “Non-programs”, poor quality, or incomplete intervention, e.g., Title I implementation had no proof of services for targeted audience of poor children
- Wrong or inappropriate intervention, e.g., something that is too sophisticated to go to scale or resisted
- Unstandardized intervention, e.g., Community Action Program and “maximum feasible participation” on the part of the poor
*
School of Management
Coverage vs. Function
- Is the program serving clients it was intended to serve?
- How many direct contact hours does each client receive?
- What is the average age of program participants?
- What percent of clients complete the program?
From Workbook 4, Process Evaluations, WHO
http://www.unodc.org/docs/treatment//process_evaluation.pdf
Test your knowledge. Is the following a coverage question or a function question (1):
*
School of Management
Process Evaluation Methods
- Quantitative
- Qualitative
- Combination
School of Management
Qualitative Methods
- Case studies
- Structured or semi-structured interviews
- Focus groups
- Direct observation
- Reviews of program meeting minutes, progress reports
School of Management
Quantitative Methods
- Surveys
- Information collected from program participants
- Agency records
School of Management
Summary Points
- Program process evaluation tells you if your program operates as intended;
- Service utilization issues typically break down into questions about coverage and organizational function
- Coverage problems can occur due to self-selection, under-coverage, and over-coverage
- Common implementation failures are incomplete interventions, delivery of the wrong intervention, and unstandardized or uncontrolled interventions
*
