Discuss the Merits and Demerits of Three Approaches of Evaluation
Name
MSPM 6102 - Practices in Project Management
Walden University
2022
1.0 Definition of project evaluation
Project evaluation is a systematic method for collecting, analyzing, and using information to
answer questions about projects, policies and programs, particularly about their effectiveness and
efficiency. In both the public and private sectors, stakeholders will want to know if the programs
they are funding, implementing, voting for, receiving or objecting to are actually having the
intended effect (and to what cost). This definition focuses on the question of whether the
program, policy or project has, as indicated, the intended effect. However, equally important are
questions such as how the program could be improved, whether the program is worthwhile,
whether there are better alternatives, if there are unintended outcomes, and whether the program
goals are appropriate and useful. Evaluators help to answer these questions, but the best way to
answer the questions is for the evaluation to be a joint project between evaluators and
stakeholders.
2.0 Approaches of evaluation
2.1 Realistic evaluation
Realistic Evaluation (Pawson and Tilley, 2002) is about the meaningful distinctions and
mechanisms which revolve around the curriculum design process. The evaluation process will
involve focus-group activities and other methods (including Soft Systems approaches) of
extracting stakeholder views, theories, distinctions and experiences of curriculum design. As
each iterative stage progresses, the project will seek to test these mechanisms and distinctions,
leading to refinement or rejection. By the end of the project, the intention is that the project will
have identified a number of principle mechanisms between its stakeholders with explanatory and
predictive powers within the broader Higher Education context.
The main strength of the realistic approach is its attempt to link specific contexts to mechanisms
in a way that has perhaps not been considered quite so thoroughly before.This has important
implications for businesses. The ability to extrapolate accurately from one evaluation to decide
matters of security policy on a company-wide basis is both important and costly for businesses.
Sometimes, results from evaluations are used by managers to assess how appropriate a particular
technology is to solve their crime problem. More often, there is no adequate evaluation before
such a decision is made. Yet there is no guarantee that the results of one study will have any
relevance for a different location or context. The commonsense observation that what has an
impact in site A may not necessarily have an impact in site B has, to a large extent, been ignored
by previous research that focuses largely on collecting figures to show whether the measure has
worked at all. The main issue is not so much whether the measure worked but rather how it did
so or, conversely, why it failed to work when logic indicated that it should, or as Pawson and
Tilley (1997) state, "what works, for whom and in what circumstances."
Eventually, of course, the result of conducting evaluations in a realistic manner should be that
the contexts that do not trigger certain mechanisms (and, vice versa, those that do) are identified,
providing a useful base of knowledge for crime prevention practitioners.
Realist evaluation
Proponents of ‘realist evaluation’ (initially called realistic evaluation) are doubtful about the
positivist methodology as a method of finding out which programs do and which do not succeed
(Pawson 2006; Pawson & Tilley 1997). They argue that the classic experimental method of the
natural sciences does conceptual violence to what is normal practice in social policy and social
science research. In particular, in the field of domestic violence it has been argued that practice is
path dependent and thus historically constrained, with the consequence that it is not feasible to
undertake standardized evaluations in a positivist methodological mode.
2.2 (360) Degree Feed Back Evaluation
Merits of degree feed back evaluation
Organization
According to Star, (2003) 360 Feedback, 360-degree feedback has the greatest impact when used
to evaluate and improve the performance of whole organizations. Companies can use data
collected on feedback programs to monitor consistent patterns or areas of weakness for
employees within the organization. According to the site, organizations develop more effective
training programs targeted at collecting common areas of weakness for employees throughout
the organization or in specific departments.
Individual Growth
From an individual perspective, an all-around feedback mechanism helps employees see whether
a consistency exists in the way their performance is viewed by more than just the manager.
Individuals receive specific information that allows them to understand how others perceive
them, notes Star 360 Feedback.
Some employees are bothered when they feel unfairly critiqued by managers who may often
have little direct interaction with them. Hearing feedback directly from colleagues and customers
in a safe, anonymous way may help the employee buy in to the reality of the information.
2.2.1 Demerits of degree feed back evaluation
2.2.1.1 Dishonest Feedback
In his "360 Degree Feedback" article on the site Amazing Results, registered psychologist
Roland Nagel points out that one commonly noted disadvantage of 360 degree feedback is that
employees may not feel comfortable offering honest feedback of supervisors. This same
dilemma could exist in employee-to-employee evaluations.
Without an anonymous system, backlash from supervisors or retaliation from colleagues could
limit workers' willingness to share true feelings in the feedback process.
2.2.1.2 Consistent Interpretation
Another major challenge in the 360 degree feedback process cited by Nagel is the reality that as
each rater sees a different behavior, how do we know the basis upon which the ratings are
observed.
Nagel's point is that different employees interact in different ways with the person being
reviewed. Subordinates observe different behaviors in a supervisor than that supervisor's same-
level colleagues and his managers. Additionally, customers have a different perspective on an
employee's behaviors.
While, you can gain insight on perceptions of the employee from all around, a consistent
interpretation of positive or negative behaviors is challenging.
3.0 Participatory Evaluation
Participatory evaluation can help improve program performance by involving key stakeholders
in evaluation design and decision making, acknowledging and addressing asymmetrical levels of
power and voice among stakeholders, using multiple and varied methods, having an action
component so that evaluation findings are useful to the program’s end users, and explicitly
aiming to build the evaluation capacity of stakeholders (Burke, 1998).
3.1 Merits of participatory evaluation include the following (Patton, 2008):
i. The focus is on participant ownership; the evaluation is oriented to the needs of the
program stakeholders rather than the funding agency.
ii. Participants meet to communicate and negotiate to reach a consensus on evaluation
results, solve problems, and make plans to improve the program.
iii. Input is sought and recognized from all participants.
iv. The emphasis is on identifying lessons learned to help improve program implementation
and determine whether targets were met.
v. The evaluation design is flexible and determined (to the extent possible) during the group
processes.
vi. The evaluation is based on empirical data to determine what happened and why.
Stakeholders may conduct the evaluation with an outside expert serving as a facilitator.
4.0 Empowerment Evaluation
Empowerment evaluation is an approach to help ensure program success by providing
stakeholders with tools and skills to evaluate their program and ensuring that the evaluation is
part of the planning and management of the program (Fetterman, 2008). The major goal of
empowerment evaluation is to transfer evaluation activities from an external evaluator to the
stakeholders. Empowerment evaluation has four steps: taking stock of the program and
determining where it stands, including its strengths and weaknesses; establishing goals for the
future with an explicit emphasis on program improvement; developing strategies to help
participants determine their own strengths that they can use to accomplish program goals and
activities; and helping program participants decide on and gather the evidence needed to
document progress toward achieving their goals (Fetterman, 1994).
5.0 Merits of empowerment evaluation according to Wandersman et al., (2005) include the
following:
i. Values improvement in people, programs, and organizations to help them achieve results.
ii. Community ownership of the design and conduct of the evaluation and implementation of
the findings.
iii. Inclusion of appropriate participants from all levels of the program, funders, and
community.
iv. Democratic participation and clear and open evaluation plans and methods.
v. Commitment to social justice and a fair allocation of resources, opportunities,
obligations, and bargaining power.
vi. Use of community knowledge to understand the local context and to interpret results.
vii. Use of evidence-based strategies with adaptations to the local environment and culture.
viii. Building the capacity of program staff and participants to improve their ability to
conduct their own evaluations.
ix. Organizational learning, ensuring that programs are responsive to changes and
challenges.
x. Accountability to funders’ expectations.
6.0 Demerits of both Participatory and Empowerment Evaluation
The potential demarits of participatory and empowerment evaluation include the possibility that
the evaluation will be viewed as less objective because of stakeholder involvement, difficulties in
addressing highly technical aspects, the need for time and resources when involving an array of
stakeholders, and domination and misuse by some stakeholders to further their own interests.
However, the benefits of fully engaging stakeholders throughout the evaluation outweigh these
concerns (Fetterman et al., 1996).
The potential disadvantages of participatory and empowerment evaluation include the possibility
that the evaluation will be viewed as less objective because of stakeholder involvement,
difficulties in addressing highly technical aspects, the need for time and resources when
involving an array of stakeholders, and domination and misuse by some stakeholders to further
their own interests. However, the benefits of fully engaging stakeholders throughout the
evaluation outweigh these concerns Fetterman et al., (1996).
References
Cummings, OW, Nowakowski, JR, Schwandt, TA, Eichelberger, RT, Kleist, KC, Larson, CL
& Knott, TG 1988, ‘Business perspectives on internal/external evaluation’, in JA McLaughlin,
LJ Weber, RW Covert & RB Ingle (eds), Evaluation Utilization, vol. 39, pp. 59–74.
Czarnecki, MT 1999, Managing by measuring: how to improve your organization’s performance
through effective benchmarking, AMACOM, New York.
Faase, TP & Pujdak, S 1987, Shared understanding of organizational culture’, in J Nowakowski
(ed.), The client perspective on evaluation: New directions for program evaluation, no. 36,
Jossey-Bass, San Francisco. Globerson, A, Globerson, S & Frampton, J 1991, You can’t manage
what you don’t measure: control and evaluation in organizations. Gower Publications, Aldershot.
Goldberg, B & Sifonis, JG 1994, Dynamic planning: the art of managing beyond tomorrow,
Oxford University Press, New York.
Guba, EG & Lincoln, YS 1987, ‘The countenances of fourth-generation evaluation’, in DJ
Palumbo (ed.), The politics of program evaluation, Sage Publications, Newbury Park, California.
Gunn, WJ 1987, ‘Client concerns and strategies in evaluation studies’, in J Nowakowski (ed.),
The client perspective on evaluation: New directions for program evaluation, no. 36, Jossey-
Bass, San Francisco.
Hogben, D 1977, ‘Curriculum evaluation: by whom, for whom?’, paper presented at the Annual
Conference of the American Educational Research Association. Publications, Beverly Hills,
California.
House, ER 1993, Professional evaluation: social impact and political consequences, Sage
Publications, Newbury Park, California.
Kushner, S 2000, Personalizing evaluation, Sage Publications, Thousand Oaks, California.
Love, AJ 1991, Internal evaluation: building organizations from within, Sage Publications,
Newbury Park, California.
Mathison, S 1994, ‘Rethinking the evaluator role: partnerships between organizations and
evaluators’, Evaluation and Program Planning,
vol. 17, no. 3, pp. 299–304.
Meyers, WR 1981, The evaluation enterprise: a realistic appraisal of evaluation careers, methods,
and applications, Jossey-Bass, San Francisco.
Newman, DL & Brown, RD 1996, Applied ethics for program evaluation, Sage Publications, San
Francisco.
Nowakowski, J 1987, ‘Editor’s notes’, in J Nowakowski (ed.), The client perspective on
evaluation: New directions for program evaluation, no. 36, Jossey-Bass, San Francisco.
Owen, JM with Rogers, PJ 1999, Program evaluation: forms and approaches, 2nd edn,
Allen & Unwin, St Leonards, NSW.
Patton, MQ 1997, Utilization-focused evaluation: the new century text, 3rd edn, Sage
Publications, Thousand Oaks, California.
Perloff, R 1979, ‘Evaluator intervention: the case for and against’, in R Perloff (ed.), Evaluator
interventions: pros and cons, Sage Publications, Beverly Hills, California.
