Creative Exercise
The Five Ps of Evaluation
Those Who Fail to Plan, Plan to Fail
Part 1
Evaluation System
Not every aspect of an organization needs to be evaluated every year
It is useful to determine a system for evaluation
Five areas of evaluation = 5 Ps
Program quality and improvement
Personnel
Places
Policies/administration
Participant outcomes
Since not every aspect of an organization needs to be evaluated every year, it is useful to determine a system for evaluation which can give us better idea of what, when, and how to evaluate.
The five areas of evaluation also called the 5 Ps include program, personnel, places, policies, and participant outcomes.
2
A System for Evaluation
| What What part of our organization will we evaluate? | Why What are we hoping to get out of this evaluation? | When When and how often will this evaluation take place? | How What methods will we use to evaluate this part of our organization? | Who Who will be involved with conducting this evaluation? |
| Program | ||||
| Participants | ||||
| Personnel | ||||
| Place | ||||
| Policy |
So basically the 5 Ps we will be talking about today is covering the WHAT portion of the evaluation system. Because we must decide what part of our organization needs to be evaluated, knowing the 5 Ps is important.
Along with the 5 Ps, why, when, how, and who is also important and we will talk about those as we look into the each category of the five Ps.
3
Practical Considerations …
Time and financial constraints
Evaluations have political overtones
Developing system means you have established appropriate reasons for evaluation
Before moving on to each of the 5 Ps, there is one thing that we must know. That is first acknowledging the practical consideration that we have limited time and financial resource. If we have unlimited amount of time and money, conducting evaluation will be so much easier, however since that is not the case and in most cases it is the opposite where we have so little money and time available for evaluation, we must carefully consider what areas within the organization must be evaluated. A well rounded balance is needed between the investment of time, energy, and money, in order to develop a sustainable evaluation system.
Also, evaluation have political overtones. In determining the worth or value of something, you want to be sure that the criteria and measurement are appropriate in the eyes of those who will use the data. You also must consider the scope of evaluation and what it will cover, the size of the program, dustaion, program input, complexity and span of goals. More about the political implications will be discussed later in this unit.
And Lastly, developing system means that you have established appropriate reasons for evaluation based on whether staff want the information, the funding organization requires it, or you need to make decisions about improving or changing a program.
4
Five Ps: Personnel
In most recreation organizations, the biggest expenditures go to staff salaries and benefits
Yearly personnel evaluations required
Trend is that # of personnel eval. Is increasing
Personnel evaluations are documented and kept as evidence
For promotion, reference, lay off, etc.
So moving on, the first of the five Ps we will talk about is personnel.
In most recreation organizations, the biggest expenditure go to staff salaries and benefits, so personnel is definitely an important portion.
In many cases annual personnel evaluations are required and the trend is that number of personnel evalution is increasing. So, you could now easily find organizations that would conduct a personnel evaluation twice a year.
Personnel evaluations are documented and kept as evidence for promotion, reference, and lay off. Not only the hiring people, but maintaining them at a certain level is part of HR department’s duty. Since those personnel evaluations are used as evidence for promotion, reference, and so on, it is not only an important portion within the organization but also a big motivator for the employees to work hard.
5
Personnel
Often called “performance appraisal/evaluation”
Combination of goals and objectives found in the job descriptions and performance outcomes from actual tasks
Well-written accurate job description is the basis
Personnel evaluation is often called performance review, performance evaluation.
It is very important that personnel evaluation should be conducted based on the standards that were described in their job description. It would not be fair to point out someone during their performance review on the things that they are not required to do. Therefore, well- written and accurate job description is the basis when conducting personnel evalution.
Ideally, personnel evaluation should be the relationship between the given job description and the actual performance. So, those two are most important criteria when conducting a personnel evaluation.
So, when you encounter performance review type of situation later when you get a job, you should always remember that you will and should be evaluated based on the job description of what you were supposed to do.
6
Personnel
Employees at all levels ought to be evaluated
Combination of observation, intuitive judgement, feedback from others (colleagues, participants, supervisors)
Improvement > retainment or punishment
Usually reflects the trilogy of evaluation
When conducing a personnel evaluation, employees at all levels ought to be evaluated
On top of the formal evaluation based on the standards and criteria, observation and intuitive judgement, feedback from others are also important portion of personnel evaluation. Again, that should not be the only form of evaluation, so mix of those informal evaluation and formal evaluation is needed.
The goal and purpose of evaluation should always be to improve instead of retainment or punishment. If that is not the case, it will be very discouraging and demotivating for the employees.
Personnel evaluation follows the trilogy of evaluation, so it will be a combination of criteria, evidence, and judgement.
7
CEO actually conducting a face to face interview
Asking about the position and what the duties are
Instant reward
Goal oriented (fire 3 people)
8
Five Ps: Policies/Administration
Includes analyzing policies, procedures, and administrative practices
Involvement of community & participants
Diverse opinions and information from public
Often called “Needs Assessment”
Big portion of this evaluation is conducting budget analysis
Next of the 5 Ps is the policies which often includes analyzing policies, procedures, and administrative practices.
Evaluating organization’s own policies and administration is not an easy job, and we will not be delving into the details of the administrative aspects of evaluation, but instead, we will talk about two additional aspects that are often included in the policies evaluation. First is called needs assessment. It is basically evaluating what is needed for their organization from community and participant’s perspective. Diverse opinions and information from the public is needed in this case. Because it is important to provide a service or program that participants want, needs assessment is important to satisfy the customers and to improve organization’s reputation.
Another portion of policies and administration evaluation is conducting a budget analysis.
9
Policies/Administration – Budget Analysis
Cost-benefit
Cost of the program to the benefits realized from it
Only $ are easy to quantify
Cost-effectiveness
Ratio of costs to revenue generated
Ideally, cost-effectiveness of 1:1
Performance-based program budget
How money is spent in relation to the outcomes of the program
Breaks down to smaller units
Cost-benefit
Cost of the program to the benefits realized from it
Only $ are easy to quantify
We will look into three different ways to conduct budget analysis.
First is called cost benefit. As it is called, cost benefit is measuring the cost of the program to the benefits realized from it. So it is literally comparing the cost and the benefits earned.
Important thing to know from this budget analysis is that only the monetary benefits are easy to be quantified and therefore measured for the analysis. However, if you think about it, there can be many additional benefits that are not quantifiable in terms of money. For example, from a midnight basketball program we read about from our first creative group exercise, sense of belonging, sense of community, and self-efficacy, or safety of the teens are all important values and benefits of the program, however are hard to be measured in terms of numbers. Therefore, it is important to consider all aspects of benefits when comparing it with the cost.
Second is called cost-effectiveness. Cost-effectiveness analysis is looking into the ratio of costs to revenue. Ideally, cost-effectiveness ratio should be 1:1 meaning that their cost and revenue generated should be same. But that means the organization is not going to make any money, so that is not feasible. To help you guys better understand, example of cost-effectiveness analysis can be used when deciding the league or program fees. For instance, if a little league for baseball costs $20,000 and there is 200 children who signed up for it. Ideally, according to the cost effectiveness analysis, fee for the little league baseball should be $100 per child. That is less likely to happen, but that is one way of calculating and conducting budget analysis.
Last one is called performance based program budget. This budget analysis looks at how money is spent in relation to the outcomes of the program by breaking down into smaller units for detailed information. So instead of just looking into the overall cost and revenue, this analysis looks at each sectors and areas on how much the most was spent and what the outcomes are. So this analysis is better to be used for bigger enterprise or organization because it allows you to look at each units instead of just the entire numbers.
10
Policies/Administration – Economic Impact
Economic impact = amount of revenue activity generated in an area (community) due to particular program/event
ROI is important
(Return on Influence instead of investment)
Lastly, when talking about policies and administration evaluation and especially the budget analysis, it is important to think about economic impact.
Economic impact is the amount of revenue activity generated in an area or community due to particular program or event.
Economic impact is a broad concept that encompasses not only the direct revenues generated from the program, but also all potential benefits that could possibly have an impact on the economy. So going back to the midnight basketball program example, making a safer neighborhood and building an educational friendly environment are all possible factors that can positively have an economic impact to that community. So, that should all be considered.
We know ROI as return of investment and that is an essential guideline in many business models. However, in addition to the traditional ROI, it is also important to understand ROI as Return on influence. Return on influence focuses on the money being spent on the possible economic influence that could have greater value for the community.
11
Something to think about
(Mega)events such as…
Olympics
World Cup
Super Bowl
Marathon
World Tourism Exhibition
What is the economic impact of those events?
Positive & negative?
The Five Ps of Evaluation
Those Who Fail to Plan, Plan to Fail
Part 2
Five Ps: Places (areas & facilities)
Evaluation criteria includes …
How place is being used
Safety (risk management)
Legal mandates
Landscapes and equipment
Routine checks (regular walk through)
Scheduled maintenance
Important to have logs, pictures, or documentations
Out of the 5 Ps, we looked at personnel and policies especially the needs assessment and budget analysis in the part 1 lecture. In part 2, we will continue learning about the 5 Ps. Next P we will learn is place often times an area or facility.
Along with people and policies, place is an important factor to the organization and program which needs ongoing evaluation to maintain its quality.
When evaluating places, evaluation criteria includes how place is being used, safey aspects like risk management, legal madates such as ADA requirements, and landscapes and equipments.
Usually the way of conducting place evaluation is by routine checks and scheduled maintenance. Because outdoor areas and facilities are used by so many people all the time, it is always important to keep a logs whenever conducting an evaluation and also pictures or documentations.
14
Places – Geographical Information System (GIS)
The GIS technique integrates spatial and demographic information
It is being used for numerous applications (urban planning, land use and facility mgmt., environment assessment, wildlife and park, etc.)
One trendy thing that is often used for places evaluation is called Geographical information system (GIS).
It is a technique that integrates spatial and demographic information and it is used in diverse places for different reasons.
15
The picture you see here are all examples of GIS. It is like a Google map but with much more detailed and diverse functions that could address many cool features.
In the same area, using GIS you can see how a water is being used, the ecological status of the water, terrain slopes, hydraulic conductivity, and so many more.
16
Five Ps: Program Quality
Clear purpose and identifiable goals
No program should exist just because …
Evaluation is one of the prime component of a program
Not only at the end, but throughout the program
The next P is the program. Perhaps a program is one of the most often evaluated category amongst the rest. Because no program should exist just because it has been there fore long time, or anything like that. It should have a clear purpose and identifiable goals. Evaluation is one of the prime component of a program that can make the program move forward. Usually we see lots of programs conducting an evaluation at the very end, but that is not always correct. Sometimes depending on the situation, it could be conducted before the program, during the program and of course at the end of the program. We will learn more about them later in this lecture.
17
Program Quality
Individual benefits
Improving health and wellness
Self-esteem, stress, balanced life
Communal benefits
Building stronger families, communities, ethnic and cultural harmony
Loneliness, isolation, crime
Program’s quality can be categorized into four different aspects for evaluation.
First, inidivudal benefits. It is focusing on how program impacts individual’s benefits such as their health, wellbeing, level of self-esteem, stress, balanced life, and so on. A well designed program should definitely benefit each participants in various ways.
Next, is communal benefit. Unlike individual benefit, communal benefit is more about building stronger community, family, ethnic and cultural harmony. Communal benefits include avoiding loneliness and isolation, and decreasing crime rate. Also building a sense of belonging and pride about their community are examples of communal benefits.
18
Program Quality
Economic benefits
Attracting business relocation and expansion
Real estate value, work force, tourism
Environmental benefits
Protecting natural resources and open spaces
Air and water quality, wildlife
Third, economic benefit should also matter. Not always, but depending on the goals and objectives of a program, economic benefits should exist that could attract more business to relocate and expand. If program could increase the value of real estate, work force, and even draw tourists, that would be fantastic. But, like I said, there are many non profit organizations and their programs do not seek for economic benefits.
Lastly, environmental benefits. Although environmental benefit is very important, this tend to be the least benefit that the organization would usually seek for. But now that saving planet earth is more important than ever in our history, environmental benefits have gathered increased attention.
Not all program evaluations should encompass all four of the benefits, but the more it could encompass the better your program would be.
19
Five Ps: Participant Outcome
One of the most common approach
“Participant” consumers, clients, patients, guests, tourists, campers, athletes, etc.
Important to be inclusive!
Last but not least, last P is participant outcome. It is literally evaluating whether participants were satisfied and happy. Depending on the situation, participant can be consumers, clients, patients, guests, tourist, campers, athletes and so on. Since the category of participant can be very broad, it is always important to be inclusive. You would definitely want to consider all aspects about people with disability, or have deep understanding out different culture and respect them in your evaluation. So, among the 5 Ps, two aspects are evaluating people, one was personnel which is evaluating your own staff and participant which is evaluating aspects about your customers.
20
Participant Outcome
Motivation/satisfaction
Changes in attitudes as outcomes
Changes in knowledge as outcomes
Changes in skills and abilities as outcomes
Carry-over into other situations
How individuals interact
Demographic characteristics
Some examples of participant outcomes that we can evaluate are their motivation and satisfaction. Whether they changed in attitudes and gained knowledge, skills, and abilities. Also, whether they were engaged in social interaction or how their cognitive, psychological, and physical abilities have improved. So, anything that could have impacted participants from your program can be evaluated and be used as an evidence to say our program is valuable.
21
Timing of Evaluation
Assessment (beginning)
Formative (during the course of a program/project)
Summative (end of the program/year)
Now that we know what the 5Ps are, let’s talk about different timing of evaluation.
Evaluation can be divided into three types by different timing. Assessment is during the beginning, formative is during, and summative is at the end of a program.
22
Assessments and Planning
Why do we want to conduct an evaluation in the beginning?
To learn and understand the needs and wants
To ensure quality of the program
To know ourselves better
Assessment is doing evaluation in the beginning of a program. Why do we want to conduct an evaluation in the beginning? Well, first we can learn and understand the needs and wants before we start the program, so we could focus our gears on those more. Also to ensure the quality of the program. Conducting an evaluation in the beginning could help us to know about ourselves better, it also helps to understand the program better, and eventually it will help to ensure the overall quality as well.
23
Formative & Summative
Formative – process, efficiency, effectiveness
Make it better before it’s too late
Instant feedback and revisions
Examples
Summative – overall performance objectives, outcomes, or results
Overall impact
Should serve as the starting point of making changes
Examples
Formative evaluation is conducting evaluation during the program. It is measuring the process, efficiency, and effectiveness of a program. You would conduct a formative evaluation to make things better before it’s too late. Also, you can get an instant feedback and revise them right away since your program is still running. An example you could find as a college student is midterm. By taking midterm, I can evaluate your skills and knowledge about the class contents before the end of the semester. Also, as you would do a peer evaluation during the mid point of the semester, that is also an example of a formative evaluation. The reason why I also included a formative peer evaluation was because it’s too late, I wanted to know if there are some people in your group that are not making any contribution at all or anything like that.
Last one is called summative evaluation. It is measuring the overall performance of objectives, goals, outcomes, or results. Usually in summative evaluation you are asked about the overall impact of the program. Although summative evaluation takes place at the end of the program, it should still serve as a starting point to improve things for next year’s program. ICES form you would do for your classes to evaluate the instructor and class is an example of summative evaluation.
Not all program or organization should do all three evaluations because as I said you would always have a limited money and time. So, depending on the given situation, it is evaluator’s role to decide when you will conduct an evaluation or multiple evaluations.
24