Logic Model
Kishon
RoyseCh5-PowerPoint35.pptxChapter 5:
What are Formative and Process Evaluation?
Formative and Process Evaluation
Purpose to shape and refine programs.
Main focus is not measurement of client outcomes.
Does not constitute a summative conclusive evaluation.
Begins with developing a logic model.
How Do I Develop a Logic Model?
Inputs
Financial, organizational, or human resources invested.
Activities
Actions, tasks or planned events that constitute the program.
Outputs
Countable products that result from the activities.
Client
Outcomes
Positive changes or client accomplishments that result from the program.
Advantages of Logic Models
Assist understanding what must happen for outcomes to be achieved.
Identify core processes so re-engineering can occur.
Provide map for programmatic or organizational change.
Provide framework for analyzing alternative strategies.
Clarify events and processes for stakeholders.
Enable critical examination of program and policy logic.
Facilitate communication about strategies, activities, and expected outcomes.
Provide focus for achieving desired end result by linking budget, activities, output, and outcomes.
Logic Model Template
| Resources | Activities | Outputs | Short-term Outcomes | Impact (Long-term) |
| In order to accomplish our set of activities we will need the following: | In order to accomplish our objectives for the program we need to do these activities: | We can monitor our activities by counting or recording these events or products: | We expect our program of activities will lead to these changes: | We expect that the program will eventually lead to these changes: |
https://www.youtube.com/watch?v=XgdZkQbMLBY
Formative Evaluation
Employed to adjust and enhance interventions.
Not used to prove if program worth funding it receives.
Purpose to guide and direct programs (mostly new ones).
Formative evaluations used to “form” programs.
Often better received by staff than other forms of evaluation.
Can be used to determine whether a new or pilot program has fidelity.
Does not rely on specific methodology.
Focus on acquiring useful information.
Methodology depends on nature of program, agency preferences, and context of evaluation request.
Conducting Formative Evaluation
Approach 1: Locate Model Standards
If standards already exist for similar programs, local program can be compared against the standards, with any discrepancies noted.
Frequently used by governmental units.
Council on Accreditation (COA) accredits nonprofit agency and has developed standards on approximately 50 different services:
www.coanet.org/standards/standards-for-public-agencies
Professional literature can be a source for standards or benchmarks.
Conducting Formative Evaluation
Approach 2: Get Expert Consultation
Consult with recognized expert from similar program with solid reputation.
If funds are not available, request a free consultation from appropriate state officials with interest in success of program.
Conducting Formative Evaluation
Approach 3: Form an Ad Hoc Evaluation Committee
Can be composed of treatment staff, board members, professionals from community, service consumers and other stakeholders.
Members do not need to be true “experts”.
Another variation is gathering feedback from several focus groups.
What is Process Evaluation?
Needs Assessment
A need for a program is identified
Formative Evaluation
Initial Program
Information to help “form” and stabilize the program
Process Evaluation
Program description
Program monitoring
Quality assurance
Outcome Evaluation
Does the program work?
Does the program impact the target population’s problems as identified in the needs assessment?
What is Process Evaluation?
Can be conducted at any time during a project.
Purposes/Goals of process evaluation:
Program description.
Program monitoring.
Quality assurance.
Each goal may include sub-goals or objectives and the evaluation may include one or some combination of all three goals.
Another purpose can be to determine why a program achieved or did not achieve its outcomes.
Program Description
Document operations of a program.
Provide necessary data to judge intensity and reliability with which services were delivered.
Rely heavily on data captured by agencies.
Ways to collect information:
Face-to-face and telephone interviews, surveys, key informant interviews, focus groups, organization record analysis, program documentation analysis, observations, and case studies.
Program Monitoring
Understand what happened in a program and to whom.
Ongoing program monitoring is essential to sound management.
Does not require elaborate research design.
Begins with examining specific goals and objectives of program and comparing these with the kind of data that most human service agencies routinely collect.
Data monitored to ensure program is serving those for whom it was designed.
Routine monitoring can reveal problems before they become overwhelming and track progress towards meeting expectations.
Mission Statements, Goals and Objectives
Mission statements explain what agency is about and provide clarity to people within and outside organization.
Goals follow from mission statements and tend to be general and global with regard to activities and products.
Statement of a program’s basic direction.
Derived from a set or values or ideals but not specific as to how or when something will be accomplished.
Provide focus, orientation and direction.
Mission Statements, Goals and Objectives
Objectives are specific and precise.
Allow measurement of progress towards achievement of a goal.
Single aim, with a verifiable outcome or result.
Must specify events or activities that can be independently determined.
Indicators
Process indicators are typically data items routinely collected by an agency and used in program monitoring during program participation.
Generally only include information from clients or program participants.
Answers questions like: “What happens during the program?”
Outcome indicators are generally collected from participants at program completion.
Can include information from control or comparison groups.
Answers question: ‘Does the program work well?”
Writing Program Objectives
Think in terms of activities than can be counted or observed.
State what will be accomplished and when.
Start with a verb: To increase/decrease, add/reduce, enlarge/lessen, promote, create, etc.
Avoid vague terms that lack specificity.
Indicate date when objective will be accomplished.
Balance quality with quantity.
Set objectives high enough to challenge staff but not impossible.
Once developed, monitoring for managerial purposes is possible.
What Should be Monitored?
Program monitoring can check a program’s progress in meeting objectives.
Do not need to wait for problems before monitoring.
Used most effectively in a diagnostic sense.
Data appropriate for monitoring:
Inputs are resources needed by the program and staff.
Process consists of activities intended to be carried out during program implementation.
Outputs are program results.
Quality Assurance
Another basic form of evaluation that evaluates compliance with a set of standards.
Often associated with ongoing review of clinical records.
Identification and correction of deficiencies in the process of providing services.
Accreditation standards exist for most human services.
Quality Assurance
Differs from program evaluation:
Focus is on process of treatment, rather than outcomes.
Often stem from legislative mandate.
Provides degree of consistency by promoting adherence to guidelines and accountability.
Makes it easier for program evaluator to understand treatment fidelity and positive effects of interventions.
Reasons for Documentation “Paperwork”
Protects clients from unethical or inappropriate treatment.
Protects staff from charges of inappropriate treatment or incompetence.
Helps recover reimbursement from insurance companies or other third parties.
Better planning for effective and efficient utilization of staff and agency resources.
Total Quality Management
Based on principles developed by William Deming.
Aim is to reduce variation from processes so greater consistency of effort is obtained.
Quality defined by the customer and improvement focuses on customers wants and needs.
Top management must have commitment to improve.
Encourages participation from all staff members.
Emphasis on client satisfaction surveys and uses feedback to make refinements.
Martin’s Aspects of Quality Programming
Accessibility
Assurance
Competence
Communication
Conformity
Deficiency
Durability
Empathy
Humaneness
Performance
Reliability
Responsiveness
Security
Tangibles
