Unit5-assignment
bamaboi
unit5_ch12-db8008.pptxDesigning Evaluations in Business
Chapter 12
Doing Research in the Business World
David E. Gray
Los Angeles | London | New Delhi
Singapore | Washington DC
Los Angeles | London | New Delhi
Singapore | Washington DC
Session Objectives
By the end of this session you will be able to:
Describe the purposes of evaluations.
Distinguish between the different schools of evaluation.
Identify suitable data collection sources.
Design valid and reliable evaluation tools.
Produce readable and informative evaluation reports.
Insert your presentation title here
Los Angeles | London | New Delhi
Singapore | Washington DC
The focus of evaluation: Kirkpatrick
Kirkpatrick’s four levels:
Level 1, Reaction: evaluating the reactions of trainees to the programme (usually by the use of a questionnaire).
Level 2, Learning: measuring the knowledge, skills and attitudes that result from the programme and which were specified as training objectives.
Level 3, Behaviour: measuring aspects of improved job performance that are related to the training objectives.
Level, 4 Results: relating the results of the training to organizational objectives and other criteria of effectiveness.
Insert your presentation title here
Los Angeles | London | New Delhi
Singapore | Washington DC
The focus of evaluation: cost–benefit analysis
Insert your presentation title here
Los Angeles | London | New Delhi
Singapore | Washington DC
The focus of evaluation: force field analysis
Insert your presentation title here
Los Angeles | London | New Delhi
Singapore | Washington DC
Types of evaluation (examples)
Insert your presentation title here
Los Angeles | London | New Delhi
Singapore | Washington DC
Schools of evaluation
Adapted from Easterby-Smith, 1994
Insert your presentation title here
Los Angeles | London | New Delhi
Singapore | Washington DC
Schools of evaluation: experimental
| Approach | Potential limitations |
| Seeks to demonstrate that any changes in behaviour or outcomes can be attributed to the (programme) intervention. | Demonstrating that changes are due to the independent variable (the intervention) and not to intervening variables. |
| Focus on quantitative measurement and use of experimental and control groups. | For statistical techniques to work, samples must be sufficiently large. |
| Problem of achieving equivalence between the experimental and control groups. |
Insert your presentation title here
Los Angeles | London | New Delhi
Singapore | Washington DC
Schools of evaluation: illuminative
Seeks views of participants.
Recognises that there are ‘multiple perspectives’.
Develops communal awareness, often using qualitative methods.
(But sponsors will often want measurement of precise outcomes).
Insert your presentation title here
Los Angeles | London | New Delhi
Singapore | Washington DC
Schools of evaluation: interventionist and action research
Close collaboration with programme staff and participants.
Agree on what data are to be collected and the uses of the data.
Seeks to have a direct impact on the programme and participants.
Insert your presentation title here
Los Angeles | London | New Delhi
Singapore | Washington DC
Data collection sources
Informants:
Programme delegates/tutors/facilitators.
Managers and sponsors.
Controls (who do not receive the programme).
Stakeholders (e.g. senior managers).
- Observations (overt/covert: participant/non-participant).
- Records (project plans, emails, content of flipcharts).
Insert your presentation title here
Los Angeles | London | New Delhi
Singapore | Washington DC
Data collection tools
Questionnaires/interview schedules.
Assessment tests.
Repertory grid.
Critical incidents analysis.
Learning logs.
Insert your presentation title here
Los Angeles | London | New Delhi
Singapore | Washington DC
Quality issues: validity High degree of face validity important for programme sponsors. Need to move beyond ‘happiness’ type questions to measure actual impact. Validity improved by covert or longer term observation of programme participants.
Insert your presentation title here
Los Angeles | London | New Delhi
Singapore | Washington DC
Quality issues: reliability
Use evaluations (e.g. observations) by more than one researcher – inter-judge reliability.
Seek multiple perspectives (e.g. sponsors, participants, stakeholders).
Insert your presentation title here
Los Angeles | London | New Delhi
Singapore | Washington DC
Quality issues: objectivity
Outcomes of evaluation have to be compared with something else for them to have any meaning, such as:
The outcomes of similar programmes.
The outcomes of the same programme delivered on a previous occasion.
The outcomes of model programmes in the same field.
The stated goals of the programme.
External standards developed by professional bodies.
Insert your presentation title here
Los Angeles | London | New Delhi
Singapore | Washington DC
Enhancing the impact of evaluation
Involve sponsors in formulating the objectives and the design of the evaluation to gain their interest and commitment.
Get commitment from senior stakeholders in advance of the evaluation that action will be taken on the basis of results.
Consider establishing a steering group to monitor and help the evaluation project and get senior stakeholders onto the group if possible to increase its credibility.
Keep sponsors informed of outputs (particularly unexpected ones) as they occur.
Insert your presentation title here
Los Angeles | London | New Delhi
Singapore | Washington DC
Evaluation involves the systematic collection of data about the characteristics of a programme, product, policy or service.
Evaluation involves different schools or perspectives, ranging from experimental and quasi-experimental, to illuminative perspectives with a focus on processes and the multiple perspectives of participants.
Data collection tools include questionnaires, assessment tests, learning logs and documentation of critical incidents.
The principles of validity, reliability and objectivity apply as much to evaluation as they do to many other aspects of research.
The impact of evaluation is enhanced if stakeholders are kept informed of outcomes as they arise.
Evaluation that fails to take into account ethical issues will often be doomed to failure. Ethical approaches include a focus on the individual needs of people rather than the goals of organizations, on making the purpose of the evaluation transparent to those being evaluated, and encouraging participation in the evaluation process.
Summary
Insert your presentation title here
Los Angeles | London | New Delhi
Singapore | Washington DC
