employing additional staff can improve service delivery?

out.pdf

Home >Social Science homework help >employing additional staff can improve service delivery?

143LA REVUE CANADIENNE D'ÉVALUATION DE PROGRAMME

Corresponding author: Benoît Gauthier, Réseau Circum inc., 74, rue du Val-Perché, Gatineau, PQ J8Z 2A6; <[email protected]>

RESEARCH AND PRACTICE NOTE

THE LAY OF THE LAND: EVALUATION PRACTICE IN CANADA TODAY

Benoît Gauthier Réseau Circum inc. Gatineau, Québec

Gail Barrington Barrington Research Group Inc. Calgary, Alberta

Sandra L. Bozzo Ontario Ministry of Community Family and Children’s Services Toronto, Ontario

Kaireen Chaytor Chaytor Consulting Services Ltd. Halifax, Nova Scotia

Jim Cullen Ministère de l’Éducation du Québec Montréal, Québec

Robert Lahey Treasury Board of Canada Secretariat Ottawa, Ontario

Robert Malatest R.A. Malatest and Associates Ltd. Victoria, British Columbia

Greg Mason Prairie Research Associates Ltd. Winnipeg, Manitoba

John Mayne Office of the Auditor General of Canada Ottawa, Ontario

Anita Myers University of Waterloo Waterloo, Ontario

Nancy L. Porteous Health Canada Ottawa, Ontario

Simon Roy GGI inc. Ottawa, Ontario

THE CANADIAN JOURNAL OF PROGRAM EVALUATION144

A group of 12 evaluation practitioners and observers takes stock of the state of program evaluation in Canada. Each of the con- tributors provides a personal viewpoint, based on their own ex- perience in the field. The selection of contributors constitutes a purposive sample aimed at providing depth of view and a vari- ety of perspectives. Each presentation highlights one strength of program evaluation practiced in Canada, one weakness, one threat, and one opportunity. It is concluded that evaluators pos- sess skills that other professions do not offer; they are social and economic researchers versed in using empirical data col- lection and analysis methods to provide a strong factual foun- dation for program and policy assessment. However, program evaluation has not acquired an identity of its own and, in fact, has tended to neglect key evaluation issues and to lose empha- sis on rigour. Today’s program evaluation environment is domi- nated by program monitoring, the lack of program evaluation self-identity, and insufficient connection with management needs. But evaluation is not without opportunities — results- based and outcome-based management, advocacy and partner- ship efforts, individual training and development, and bridging between program management and policy development repre- sent some. But first, evaluators must self-define to communi- cate to others what their specific contribution is likely to be. The article concludes with implications for the practice of evalu- ation in Canada and the blueprint of a workplan for evaluators individually and collectively, in their organizations and in their professional association.

Un groupe de 12 praticiens et observateurs de l’évaluation prend le pouls de l’évaluation de programme au Canada. Chaque auteur présente un point de vue personnel basé sur sa propre expérience dans le domaine. Les auteurs ont été sélectionnés pour fournir profondeur de vue et variété de perspectives. Cha- que présentation met en évidence une force de l’évaluation de programme telle que pratiquée aujourd’hui au Canada, une fai- blesse, une menace et une opportunité. On conclut que les évaluateurs possèdent des habiletés qu’aucune autre profession n’offre; ce sont des chercheurs du social et de l’économique ver- sés en cueillette de données empiriques et en méthodes d’ana- lyse permettant de fournir une base factuelle solide pour l’examen de programmes et de politiques. Toutefois, l’évalua- tion de programme n’a pas acquis d’identité propre et, en fait, a eu tendance à négliger des enjeux d’évaluation clés et à donner moins d’importance à la rigueur. L’environnement actuel de l’évaluation de programme est dominé par le suivi de program- mes, la faiblesse de l’identité propre de l’évaluation et le con- tact insuffisant avec les besoins de gestion. Mais l’évaluation

Résumé:

Abstract:

145LA REVUE CANADIENNE D'ÉVALUATION DE PROGRAMME

ne manque pas d’opportunités; on y retrouve la gestion basée sur les résultats, les efforts en partenariat et en défense des intérêts, la formation professionnelle individuelle et le rappro- chement de la gestion des programmes et du développement des politiques. Mais d’abord, les évaluateurs doivent s’auto-dé- finir pour mieux communiquer ce que pourrait être leur contri- bution particulière. L’article énonce quelques implications pour la pratique de l’évaluation au Canada et présente l’ébauche d’un plan de travail pour les évaluateurs individuellement et collec- tivement, dans leurs organisations et dans le cadre de leur as- sociation professionnelle. Un résumé en français ainsi que la version française des conclusions et implications sont présen- tés en annexe.

In 2001, the Canadian Journal of Program Evaluation devoted a special issue to provincial evaluation policy and practice in Canada. Accounts of provincial evaluation activity were provided for British Columbia (McDavid, 2001), Alberta (Bradley, 2001), Manitoba (Warrack, 2001), Ontario (Segsworth, 2001), Quebec (Cabatoff, 2001), Prince Edward Island (Mowry, Clough, MacDonald, Pranger, & Griner, 2001), Newfoundland (Ross, 2001) and the North- west Territories (Hicks, 2001). In conclusion, Seasons (2001) identi- fied some “areas of concern” flowing from the provincial profiles, including the following: considerable variation in uses and percep- tions of evaluation, the lack of depth of evaluations and the emer- gence of performance measurement, the difficulty of bringing managers to think in terms of outcomes rather than outputs and in terms of performance rather than planning, and resource constraints. Being the first ever account of evaluation activity at the provincial level, this collection of articles represented a very important contri- bution to the knowledge of evaluation practice in Canada.

By design, these articles focused on governmental program evalua- tion policies and their implementation; they were not meant to take the pulse of evaluation practice or to provide a view of the health of program evaluation itself. This article sets out to fill that gap in knowledge by providing accounts of the strategic position of evalua- tion in a variety of Canadian contexts. This article translates to- day’s challenges into ingredients for a plan for practice and action in program evaluation as a field of study.

In preparation for the 2003 Canadian Evaluation Society Conference, a group of evaluation practitioners and observers set out to take stock of the state of program evaluation in Canada. We accepted that each

THE CANADIAN JOURNAL OF PROGRAM EVALUATION146

contributor would provide a personal viewpoint, based on their own experience in the field; perspectives varied by duration (contributors’ careers having been initiated at various times over the last 25 years) and geography as well as according to the organizational role of the observer vis-à-vis the programs under study and the practice of evalu- ation (i.e., internal evaluator, consultant, arms-length analyst). This exercise was subjective by design. Moreover, our selection of contribu- tors does not constitute a random sample of practitioners but rather a purposive sample aimed at providing depth of view and a variety of perspectives. We were looking for tension, disagreement, debate; we discovered a series of complementary scans of the evaluation environ- ment that demonstrates the richness of the field.

The presentation of each point of view is structured according to a strategic analysis framework. Each presentation highlights one strength of program evaluation practiced in Canada, one weakness, one threat, and one opportunity. Strengths and weaknesses are in- ternal to program evaluation, while threats and opportunities stem from the environment in which program evaluation evolves. This framework was proposed to ensure positive views (strengths and opportunities) as well as critical concerns (weaknesses and threats) and to bring forward outside-in dimensions (opportunities and threats) as well as inside-out issues (strengths and weaknesses). Following the individual presentations, we provide a concluding syn- thesis of the observations and implications of our panel.

This document is significant in that it brings together perspectives from across Canada and from the practice of evaluation at the federal and provincial levels as well as in relation to not-for-profit organiza- tions. It also brings together the views of seasoned evaluation train- ers, managers, practitioners, and thinkers. Based on the varied points of view of 12 observers, it was possible to draw a coherent picture of the state of evaluation practice in Canada today, which led to a pro- posed blueprint of an action plan which concerns each practicing evaluator in Canada as well as their professional organization.

A Western Canadian Perspective Robert Malatest, Consultant, Victoria, British Columbia

Based on numerous program evaluation assignments in British Co- lumbia, Alberta, and Saskatchewan, I could identify several trends and issues with respect to the state of program evaluation in West- ern Canada. My focus is on public sector program evaluation.

147LA REVUE CANADIENNE D'ÉVALUATION DE PROGRAMME

A strength: the development of evaluation methodologies. During the past several years, provincial and federal agencies have recog- nized the various requirements associated with good program evalu- ation. Whereas, traditionally, many departments and agencies would conduct a client satisfaction survey as the primary program evalua- tion activity, there has been an increasing knowledge of the need to adopt quasi-experimental methods to support more rigorous program evaluation activities. Program evaluation methodologies have be- come more comprehensive and include a range of evaluation tools to demonstrate program success.

A weakness: the inadequate planning of program evaluations. Not- withstanding the enhanced awareness and use of various program evaluation tools, program evaluation occurs too often as an after- thought in the design and development of programs, rather than as an integral element of program design. The limited ability to incor- porate adequate evaluation activities into program design could well reflect the pressures for staff to design and implement new programs quickly without establishing processes that could facilitate a rigor- ous assessment of program impact. In many cases, program design and implementation could be characterized as follows: design the program first and evaluate it later.

A threat: reduced program evaluation capacity in British Colum- bia. Significant reductions in provincial government employment in British Columbia will have negative ramifications for the ability of managers to champion program evaluation activities within minis- tries. As many reductions have occurred in the corporate support functions (i.e., policy development, strategic planning, and program evaluation), the ability of ministries to design and manage complex evaluations has been compromised. Of particular concern is the loss of resources at a time when government has placed greater empha- sis on managing for outcomes and adopting an approach of continu- ous program improvement.

An opportunity: managing for outcomes. Program evaluation activi- ties in both British Columbia and Alberta have been strengthened by strong government-wide commitment to measure and report on the key outcomes for almost all ministries and/or departments. The strong emphasis on performance measurement has provided impe- tus for managers to review or evaluate programs that may not be meeting targets. Continuous performance measurement activities have also generated significant volumes of data (i.e., ongoing sur-

THE CANADIAN JOURNAL OF PROGRAM EVALUATION148

veys, administrative tracking systems) that can be used in the sup- port of program evaluation activities.

Program Evaluation in Alberta Gail V. Barrington, Consultant, Calgary, Alberta

Preparing this document was the opportunity for me to discover how vibrant the evaluation scene is in Alberta today, as the following comments suggest.1

A strength: growing sophistication. The key strength of evaluation observed in Alberta today is a growing sophistication about evalua- tion and what it entails. Evaluators have honed their skills and are better qualified, more competitive, and more methodologically sound, overall, than they were five years ago. Similarly, funders are demon- strating a commitment to, and an understanding of, evaluation that was not present before. Evaluation is now a valued and bona fide piece of the programming cycle. In particular, the widespread use of logic models as a basis for developing evaluation frameworks and perform- ance indicators provides evidence of this change. An example of the focus on outcomes is the growing adoption of the software package called Hull Outcomes Monitoring and Evaluation System (HOMES), particularly in the child welfare sector. This outcome information system is tailored for not-for-profit organizations and supports the use of common outcomes across agencies. This creates the opportunity for best practice research and other sector-wide analyses.

A weakness: dependence on performance measurement. Apart from evaluation being under-resourced (which is certainly nothing new), the key weakness to current practice in Alberta represents the other side of the outcomes coin. It is a growing dependence on perform- ance measures, in some cases to the exclusion of other, more com- plex outcomes. In government, the measures may relate more closely

1. Not feeling equipped to represent the views of Alberta evalua- tors, I did a quick consultation with six evaluators within the pri- vate, government, and not-for-profit sector in Alberta to answer the assigned questions. Special thanks to Alberta evaluators Linda Banister, Banister Research & Consulting Inc.; Kelly Ernst, Cana- dian Outcomes Institute; Irene Hoffart, Synergy Research; Peggy Howard, Howard Research Inc.; Birgitta Larsen, B.I.M. Larsen & Associates; and Bill Wong, Alberta Children’s Services.

149LA REVUE CANADIENNE D'ÉVALUATION DE PROGRAMME

to departmental activities than to programs and services. In the not-for-profit sector, they may relate more closely to organizations’ self-interest (i.e., funders’ demands for demonstrated success and agencies’ need to prove their value) than to clients’ need for good programming. Ethical and methodological issues that warrant seri- ous discussion are raised by this short-term view.

A threat: devaluation of evaluation. The focus on outcomes as an end in themselves and the avoidance of serious evaluation research into the meaning and context of program outcomes may have a sig- nificant backlash. When government and agencies eventually dis- cover that although they are accountable, their services have not improved, evaluation may be devalued. Data gathering will be seen as a waste of time because it does not result in positive change. Even now, evaluation findings rarely end up on the decision makers’ ta- ble, so any threat to the positive climate we are currently enjoying would be a serious setback.

An opportunity: linking accountability and evaluation. Despite these clouds on the horizon, the evaluators see the current environment as a real opportunity to forge the link between accountability and evaluation. The federal government’s Results-based Management and Accountability Framework (RMAF) is seen as one such approach that could be applied at other levels of government as well. Alberta evaluators believe that they can make evaluation both more rigor- ous and more useful to decision makers, that they can talk the same language as the decision makers (i.e., outcomes) and yet shift the focus to best practice and program improvement. In fact, they feel that government and funders at all levels are primed to listen to them. There has never been a better time than the present to seize this opportunity!

The Devaluation of Evaluation Greg Mason, Consultant, Winnipeg, Manitoba

A threat: short-term/short-view performance monitoring. Recent trends in evaluation, especially at the federal level, should concern Canadian evaluators. The increasing focus on performance meas- urement and auditing should be a welcome change for evaluators, but, in fact, as accountability and compliance become the order of the day, evaluators seem to be increasingly relegated to the back- ground in managing programs. Nowhere is this clearer than in the ubiquitous RMAF (results-based management framework) and

THE CANADIAN JOURNAL OF PROGRAM EVALUATION150

RBAF (risk-based audit framework). The RMAF/RBAF “industry” in Ottawa has consumed millions of dollars in consulting fees and staff time as managers ready these documents to ensure that they have spending authority. Performance monitoring, integrated risk management, and outcome reporting dominate managers’ concerns, and evaluation questions take a secondary role.

To be sure, evolution continues, and several attempts are currently being made to integrate RMAFs and RBAFs and to simplify these processes. However, the basic idea remains that everything worth knowing about a program is to be found by an audit and a few out- come measures. Given the paucity of evaluation data generated by most programs and the poor funding provided to most evaluations, many evaluation reports are anemic documents, anchored in key informant and other self-report data that never reach the impor- tant and interesting questions. It is easy to complain about the growth of the audit function, but blame for the current state of evalu- ation must be found elsewhere. Audit has filled a demand by Par- liament and the public and should be seen as valuable in its own right. If evaluation questions are not being raised, responsibility lies with evaluators.

Evaluation must inform the decision to continue, expand, or cease a program. Five questions anchor this decision:

• What problem (state of affairs) was the program/policy de- signed to address (solve or ameliorate), and does the prob- lem remain?

• Do the interventions that compose the program/policy re- flect the state of the art of what is known about the tech- niques that can address this problem?

• Has the program/policy been properly implemented? In other words, have the required outputs been produced?

• What changes in the state of affairs have occurred because of the program/policy?

• Given that the problem remains, does it make sense to con- tinue to use the current program/policy, or are there more effective approaches that should be tried?

Notice that these evaluation questions do not directly address whether risks are managed, whether the program aligns with the objectives of the sponsoring department/agency, or whether resources have been used for the purposes authorized. These issues, properly

151LA REVUE CANADIENNE D'ÉVALUATION DE PROGRAMME

the object of performance measurement and audit, emerged as re- sult of the grants and contributions (Gs&Cs) “scandal” in Human Resources Development Canada. At the time, most members of the opposition, the press, and the Canadian government concluded that the government was spending without control and welcomed proc- esses that promised to limit the excess. Managing the risks associ- ated with public spending gave rise to a series of new “risk” assessment and performance audit specialists who have created a wide array of policies related to disbursing money. The centrepiece of this new control, the Treasury Board’s policy on transfer payments, details how government may award funding to third parties. Ini- tially, Treasury Board required an RMAF and RBAF only for Gs&Cs, but now departments are applying the general framework to other policies and programs, even to the entire department.

Three factors figure in this devaluation of evaluation. First, despite the increased emphasis on measuring outcomes, few programs at the federal level actually do a very good job of this. In large meas- ure, this is because, aside from a few areas such as defence, the federal government does not deal directly with end users. This is why it often resorts to Gs&Cs to effect change. However, providing funding to a third party to deliver programming is not really policy, but an abdication of policy, reflecting a lack of jurisdiction and im- agination.

Second, evaluation units typically do not have independent budgets and often must get program managers to fund the evaluation. The budgets available for evaluation have remained fixed, and the en- tire function is underfunded. Given the resources expended on RMAFs/RBAFs and audits, this is not surprising, but the result is that the core evaluation questions remain unanswered.

A weakness: passivity in the face of threats. Third, evaluators have been passive in the face of threats to the function. Further, we have shown very little inventiveness in developing new approaches to test- ing the incremental impact of a program. Methodologies such as randomized trials and statistical techniques such as instrumental variables should be more common. When it comes to measuring the impact and cost-effectiveness of a program, focus groups and key informant interviews simply do not make the grade.

An opportunity: bridging program management and policy develop- ment. The evaluation function desperately needs rehabilitation. End-

THE CANADIAN JOURNAL OF PROGRAM EVALUATION152

less RMAF development has not supported improved policy and pro- gramming because, although audit provides essential information on how to conduct a program, it has little to say on why it should be con- ducted. Rather than a weak partner to audit, evaluation is the essen- tial bridge from program management to policy development.

A University Perspective Anita Myers, Academic, Waterloo, Ontario

A strength: grounded in reality. The key strength or what most ap- peals to students is that evaluation is applied, practical, or real-world. Universities are being called upon to demonstrate rel- evance and co-operative programs, and alumni employment data are used as key marketing strategies. Evaluation training enhances opportunities for co-op and ultimately job placements.

A weakness: lack of identity. The key weakness is the field’s iden- tity as a distinct, legitimate field of study. This issue, together with standards for practice, preparation, and certification of evaluators, was debated in the evaluation literature almost a decade ago. Re- cent articles in evaluation journals and society initiatives (such as the CES’s search for a “Core Body of Knowledge”; see Zorzi, Perrin, McGuire, Long, & Lee, 2002) indicate progress, but no resolution concerning these issues (Altschuld & Engle, 1994; Preskill, 2000; Smith, 2001). Students still ask: “What will I be if I pursue this route?” Not all instructors responsible for teaching evaluation courses in universities or colleges identify themselves first and foremost as “evaluators.” There is no agreed-upon curriculum for a single course in evaluation (which is the norm in most departments), much less the package of courses, practicum, and other learning experiences assembled to prepare students as evaluators. Whether we call it a “hybrid” (of a discipline and profession), a “near-discipline,” or a “transdiscipline,” the field of evaluation is still clarifying its scope and boundaries.

A threat: difficult integration in the academic world. The biggest threat to evaluation stemming from its environment is the tradi- tional, academic mind-set. Graduate students, particularly doctoral students, are primarily trained as future researchers and academi- cians. Part-time students are not eligible for most scholarships. It is more difficult to obtain approval and ethical clearance for evalua- tion theses. Traditional granting agencies often do not fund evalua- tion projects; contract research is charged more university overhead,

153LA REVUE CANADIENNE D'ÉVALUATION DE PROGRAMME

and reports receive little consideration (relative to scholarly publi- cations) for tenure, promotion, and merit reviews.

An opportunity: using the mentoring model. Fortunately, academia is a learning and mentoring environment that allows faculty to pur- sue projects of personal interest, encourages conceptual and meth- odological debates, and promotes a mentorship/apprenticeship model for student training. The CES student case competition is a tremen- dous vehicle for furthering the training of future evaluators, gener- ating interest among other students, and increasing the profile and legitimacy of evaluation throughout the university. For instance, at my Faculty our dean has proudly displayed the “crystal trophy,” and members of finalist and winning teams have had their accomplish- ments recognized in the campus newspaper and at annual award receptions.

Thus, on balance, the strengths of program evaluation far outweigh the weaknesses, and we will continue to address the challenges and capitalize on the numerous opportunities our field has to offer.

Getting it Started, Making it Happen Sandra L. Bozzo, Provincial Public Servant, Toronto, Ontario

My commentary presents a perspective on the state of evaluation practice in Ontario that is based on my evaluation experiences in provincial government and non-profit/voluntary sector settings and on conversations with evaluators providing consulting services in Ontario.

A strength: the commitment. There has been a growing political com- mitment in Ontario to steer government in the direction of evalua- tion. This has stemmed from the ongoing recognition that all balanced policy decisions can and should be based on results, with evaluation serving as a tool for getting there. This commitment has willingly embraced the reality that voluntary/non-profit organiza- tions have a role to play in government’s own accountability and reporting. As well, government has recognized the need to subject itself to the same level of scrutiny that voluntary/non-profit organi- zations in receipt of public funds are subject to.

A weakness: the paradigm. The foundation of evaluation policies in Ontario has been, until very recently, focused on efficiency. Public

THE CANADIAN JOURNAL OF PROGRAM EVALUATION154

and voluntary/non-profit entities face real pressures to function and perform much like businesses. The paradigm within which evalua- tion is rooted in Ontario may lead to a partial assessment of pro- grams and services. The current approach may have been implemented to help government understand how to better contain costs, identify where funds could be more appropriately invested, and determine how to redirect funding to areas of government pri- ority. This narrow perspective on program and service outcomes lim- its the ability of public or voluntary/non-profit entities to use evaluation as an organizational management and learning tool to obtain a fuller picture of what is working and what is not. A more balanced approach to evaluation — one that is wider in scope and goes well beyond a fiscal exercise — could result in more balanced, evidence-based decision making.

A threat: the limited capacity. While public and voluntary/non-profit entities are being called upon to demonstrate the effectiveness and efficiency of their programs and services, there may be limited ca- pacity within these organizations to meet the demands. Capacity here is defined broadly to include human and financial resources to undertake evaluations, evaluation skills, knowledge and understand- ing, and organizational or technological infrastructure to support evaluations. Current accountability requirements imply the need to systematize and standardize information collection over time. The shift to reporting on outcomes and conducting more rigorous evalu- ations presents a daunting task to public and voluntary/non-profit entities, already stretched to the limits resource-wise. Not only may limited data be collected or poor data systems be in place, but the skills of staff responsible for evaluations may be weak. The ques- tion is whether organizations are able to respond adequately to re- porting and accountability expectations without first addressing evaluation capacity issues. Training is a fundamental necessity to building evaluation capacity in Ontario. This, however, is only a first step, and ongoing training may be key to the sustainability of evalu- ation in the province.

An opportunity: the potential for collaboration and partnerships. Evaluation in Ontario may undoubtedly be shaped through collabo- ration and partnerships with other jurisdictions and among public and voluntary/non-profit organizations. There is an enormous po- tential to pool evaluation resources across different funders and dif- ferent funding interests. This creates an opportunity to simplify reporting practices where there are overlaps in funding, multiple

155LA REVUE CANADIENNE D'ÉVALUATION DE PROGRAMME

funding interests, or programs and services delivered in partner- ship. Joint approaches to reporting and evaluation call for greater dialogue between public or private funders and voluntary/non-profit organizations and among different levels of government. This in- volves looking at funded programs and services through a different lens: as joint investments in communities and citizens with mutu- ally beneficial outcomes.

A Global View Nancy L. Porteous, Public Servant, Ottawa, Ontario

A strength: collaboration, participation, and balance. A real strength of the evaluation field today is that many evaluators are emphasiz- ing collaboration, balancing the trade-offs between methodological rigour and meaningful participation in the evaluation process. A participatory approach involving stakeholders at key junctures in the evaluation helps in getting the rigour-to-relevance ratio right. Happily, I am encountering fewer of the old quantitative versus qualitative debates, as evaluation issues and questions are increas- ingly driving the evaluation design rather than the other way around. Both the art and science of evaluation are more and more appreci- ated. As a result, the core competencies for evaluators must include both well developed interpersonal and technical skills.

A threat: the effort required. Engaging others in the evaluation en- terprise, though, takes time. Program stakeholders involved in the process often view evaluation as labour-intensive and time-consum- ing. Evaluation projects cannot be seen exclusively as externally mandated ventures to meet the information needs or reporting re- quirements of funders or senior managers. Evaluation must strive to meet the information and decision-making needs of a variety of stakeholders whenever and wherever possible. As evaluators, we must always be aware that how we interact with stakeholders in the evaluation of any given program, policy, or initiative largely shapes their view of evaluation. We must demystify evaluation and demonstrate its usefulness.

A weakness: promotion of evaluation. Demonstrating the value of evaluation, however, is a key weakness of the field today. There is currently no clear and convincing evidence that evaluation makes a difference. As a field, we are vulnerable at a time when other speci- alities are gaining ground and encroaching on evaluation’s turf. There is, at best, a murky distinction between audit and evalua-

THE CANADIAN JOURNAL OF PROGRAM EVALUATION156

tion. There is confusion about how performance measurement re- lates to evaluation. As individual evaluation specialists, we need to do a better job of tracking how our evaluative work is used in decision-making, documenting how evaluations contribute to pro- gram and policy change. This will enable us to eventually get to a place where we can do reviews, syntheses, or meta-analyses of the value of evaluation.

An opportunity: collective action. As an evaluation association, the Canadian Evaluation Society must continue to explore and articu- late the value of evaluation through its advocacy work. Increased cooperation among other national and regional evaluation associa- tions, such as through the International Organization for Coopera- tion in Evaluation (see <http://ioce.net>), should help in creating a strong united global voice for evaluation.

Dealing with Performance Measurement Simon Roy, Consultant, Ottawa, Ontario

This presentation is rooted in my experiences with federal govern- ment departments and agencies.

For a number of years now, the federal government has emphasized the importance of conducting performance-focused evaluations to meet accountability requirements. This orientation is no doubt the result of an over-abundance of evaluation studies focusing mainly on process or delivery issues, or on poor performance measurement methodologies. The recent public attention around the monitoring gaps within one particular federal department was also a key driver. The latest transfer payment policy by the Treasury Board Secretariat Canada was perhaps the strongest response to these events and the best indicator of this performance-measurement trend. According to the policy, all federal programs involving grants and contributions must be monitored according to customized evaluation and perform- ance monitoring frameworks. The following points relate to this new policy and the emphasis on results measurement in general.

A strength: management involvement. A key strength of this trend is that more program managers and directors are now actively in- volved in the monitoring and evaluation process. The systematic development of evaluation and monitoring frameworks is involving management earlier in the evaluation process. Many managers view ongoing monitoring not only as an accountability requirement, but

157LA REVUE CANADIENNE D'ÉVALUATION DE PROGRAMME

also a tool for their own management needs. A number have suc- cessfully implemented effective measurement mechanisms to assess their programs’ performance before evaluation studies are actually conducted. Many managers also find that the tools used for moni- toring frameworks, including the logic models, help them refine and better communicate their programs’ rationale and objectives.

A weakness: neglecting key evaluation issues. While more focus on results monitoring may help departments meet accountability re- quirements, it has in many cases led to the neglect of other important evaluation issues. The strong performance measurement message sent to evaluators and managers has unfortunately encouraged some members of the community to focus most resources on this aspect of evaluation. As a result, other issues such as rationale, design, and delivery, even cost-effectiveness can be totally ignored in both summative and formative evaluations. I have witnessed a case in which extensive resources were invested in an evaluation solely to assess the impact of a program using a complex quasi-experimental design. Unfortunately, no significant impact was observed. In the absence of methodologies addressing design and delivery issues, how- ever, there were no other sources of evidence indicating explanations or solutions to the problem(s). Even relevance issues were neglected, which prevented the evaluators from truly discussing the option of abandoning the program. In my view, measuring impacts does not even make sense in some programs where impacts are either obvious or very difficult to actually measure. On the other hand, effective methodologies, such as the use of an expert panel, can be used to truly review the relevance of these programs.

A threat: biasing the evaluation process. As mentioned above, the focus on performance measurement often leads to the active involve- ment of program staff in the monitoring processes. While it may lead to better ongoing performance measurement, it may also in- crease the risk of introducing biases in the evaluation process. The ongoing performance measurement processes are typically very ob- jective measures (program outputs in most cases) that are not sub- ject to extensive data analyses or interpretation. Managers can therefore supervise and manage the monitoring process without in- troducing too many biases, especially if the monitoring framework describes the process in detail upfront. However, personal experi- ence shows that some program managers can also get directly in- volved in the evaluation of the programs for which they are responsible. In some cases, program managers will directly hire con-

THE CANADIAN JOURNAL OF PROGRAM EVALUATION158

sultants to do the work, without going through their own depart- ment’s evaluation managers. This obviously leaves the consultant in a very awkward position, as the person responsible for the pro- gram under evaluation is also his or her client. It sometimes leads to pressure on consultants to produce biased evaluation reports, al- though this is less so when evaluation managers within the depart- ment or unit manage the evaluation and play a key “intermediary” role between program managers and those who conduct the evalua- tion study.

An opportunity: setting program objectives. It should be mentioned that the focus on performance measurement and a more system- atic, upfront use of evaluation frameworks may actually encourage more program managers to set more specific objectives for their pro- grams. The development of frameworks involves the identification of key performance indicators; this can be used not only to plan meth- odologies but also for setting targets for the future. In some cases, frameworks may also propose service delivery standards against which the performance can be measured in the future. Quantitative targets, benchmarks, and service standards are more the exception than the rule in government. Using them would not only help dem- onstrate results where quasi-experimental designs are not feasible, but would also help ensure quality of service and ongoing perform- ance improvement.

Positioning Evaluation in Results-Based Management John Mayne, Public Servant, Ottawa, Ontario

A strength: rooted in practice. Much evaluation in Canada is indeed program evaluation focused on practical insight and use rather than research evaluation aimed at increasing knowledge about interven- tions. It is often done or managed internally. As a result, it is or tries to be an integral part of public administration and has a quite practical bent, all giving it a greater potential for use in public sec- tor management.

A weakness: insufficient integration with management need. Evalu- ation has not often closely aligned itself with the results-based man- agement public sector regimes being established in many Canadian jurisdictions. Most measurement resources in organizations are be- ing spent on performance measurement, not evaluation. Evaluation remains a study-based measurement tool, not well integrated with ongoing measurement efforts. At the same time, it has had limited

159LA REVUE CANADIENNE D'ÉVALUATION DE PROGRAMME

success in “speaking truth to power” — it is too often a friendly (and ultimately less useful) function rather than a challenge function. Yet objectivity and independence were original defining character- istics of evaluation.

A threat: losing disciplinary independence. There is a real danger that, in trying to be too helpful for managers, evaluation becomes de facto a management consulting function, losing its competitive advantage and its independence. At the same time, there is the real danger of evaluation being subsumed by the accounting/auditing profession and their much stronger professional associations, with evaluators becoming the occasional experts brought in as needed. Those professions are much more aggressive in building a visible interest and expertise in such areas as performance measurement and working toward measurement and reporting standards. The scenario is then of evaluators managed by management consultants or accountants/auditors with certification forced on the evaluation profession by others.

An opportunity: bringing about results-based public service. Given the strong focus on results information in public sector management reform, evaluators have the theories, tools, knowledge, and skills needed to bring about a results-based public service. They are best placed, by a considerable degree, to play a leading role if a more activist agenda is pursued seeking to align evaluation more closely with the results-based agenda of governments. Sensible results meas- urement skills are badly needed in the public sector, but are often being pursued by others without the necessary evaluation skills and knowledge.

Issues of Both Supply and Demand Robert Lahey, Public Servant, Ottawa, Ontario

This synopsis is taken from the perspective of the practice of evalu- ation in federal departments and agencies subject to the Treasury Board Evaluation Policy (some 87 federal organizations).

The evaluation function has been going through a period of revitali- zation over the last couple of years, with the infusion of new funds from the Treasury Board Secretariat, the creation of a Centre of Ex- cellence for Evaluation, and a strong desire from the Centre to rebuild capacity and reposition the function in line with the federal govern- ment’s vision of modern management and Results for Canadians (no date). The function is therefore very much in a period of transition.

THE CANADIAN JOURNAL OF PROGRAM EVALUATION160

A strength: evaluators’ skills. Evaluators possess an analytical ca- pability to address a broad range of issues critical to senior manag- ers in today’s environment. They have a unique and strong ability to synthesize a myriad of data and opinions to present a coherent analytical picture about program rationale (and continued need for a program), cost-effectiveness, alternatives for service delivery, and impacts on clients.

Related to this is the growing infrastructure and support network that evaluators within the federal government have to help deliver. This includes: an evaluation policy and standards (to help in ensur- ing a disciplined approach); a Centre of Excellence for Evaluation that serves as an advocate for the function; and an active functional community and network that can share information and advice re- garding good practices for evaluators to help deliver on this man- date.

A weakness: lack of skilled human resources. The capacity to de- liver quality evaluation in a timely fashion is sometimes hampered by too few skilled or experienced evaluators in a department or agency. There are simply too few skilled, trained, experienced evalu- ators in the federal system today. This capacity gap has been esti- mated to be in the order of 105 full-time equivalents (FTE), a number that becomes significant when one realizes how small the full com- munity of federal evaluators really is. Recent studies have shown that there are currently some 285 full-time equivalents in the sys- tem, while the need across the system is in the order of 390. Put another way, a 40% increase in trained evaluators is still needed to fully implement the federal Evaluation Policy.

It is important, though, to recognize that the issue is more than simply “too few FTEs”; there are too few people who can effectively deliver on evaluation as needed by senior managers today. By “ef- fectively delivering,” we mean to be prepared to pose and then de- liver on the tough questions (the kind being raised by the recent round of Expenditure and Management Reviews, i.e., questions fun- damental to summative evaluation); understanding both the “art” and “science” of evaluation; being able to find the right balance be- tween timeliness, objectivity, credibility, and cost in developing and delivering “just-in-time” evaluations that are valued and serve the decision-making process — evaluations that are “useful and used.”

161LA REVUE CANADIENNE D'ÉVALUATION DE PROGRAMME

The demographics of this functional community are only going to worsen this situation. Growing the type of evaluators described above takes time but is fundamental to the development of evaluators in the public service and to the long-term sustainability of evaluation in the public service.

A threat: insufficient links with senior management. Evaluation groups are not always linked, or well linked, to the needs of senior executive committees in government. As a result, they may not be fully cognizant of the expectations and priorities of senior manag- ers. Additionally, senior executives may not fully appreciate what evaluation and evaluators can deliver on. The function is not al- ways well understood. Additionally, it has an uneven history across government of meeting expectations. As a result, evaluators and the function still face issues of credibility. The net result is that, in some organizations, evaluation may be deemed to be irrelevant and may not get ingrained into the culture of the organization’s management practices. Over time, if viewed this way, it may serve as a target for spending cuts or reallocation of scarce resources.

An opportunity: need for performance information. The reallocation exercise announced in the 2003 federal budget is calling for timely, objective, and credible information on the performance of programs, as is the longer-term expenditure and management review exercise that the federal government has announced. This offers an opportu- nity to position the evaluation function as a decision-making tool for resource allocation as well as program improvement and account- ability. The framework for expenditure and management reviews (EMRs) borrows heavily from evaluation. Many of the fundamental issues of the EMRs are parallel to the kind that evaluators focus on in summative evaluation. Furthermore, these reviews are intended to be a long-term and ongoing initiative that will be fundamental to the government’s commitment to ensure Canadians know where and how their tax dollars are being used, and that they are spent wisely. The evaluation function in departments and agencies should be a part of this initiative. It will not happen by itself, though. It will require evaluators to position themselves with senior management in organizations and demonstrate in a proactive way how the prac- tice of evaluation, and the skill set of evaluators, can assist senior managers in meeting what is likely their top priority for the fore- seeable future.

THE CANADIAN JOURNAL OF PROGRAM EVALUATION162

Finding a Niche Benoît Gauthier, Consultant, Gatineau, Quebec

My point of view was shaped by being active on the federal evalua- tion scene for more than 20 years — 7 as a federal program evalua- tion officer and manager, and then 14 as an independent consultant — and consulting with the Government of Quebec.

A strength: a relevant endeavour. In 1981 and 1982, Optimum ran a series of scholarly articles debating whether program evaluation should focus on the rigour of its scientific enquiry or on the relevance of its findings within a decision-making framework. Over the years, relevance has won the battle. Based on anecdotal observation, the battle was over in 1989 or 1990 in an era when program evaluation became subsumed within a more general “review” function, along with the audit function and, sometimes, other types of intelligence gathering. The victory of relevance has meant that program evalua- tion has adapted to the needs — mainly the short-term needs — of management: it is generally timely, uses resources sparingly, and focuses on issues of interest to management — mainly middle man- agement. This is a key strength of program evaluation today in that it is likely to position evaluation as a helping hand delivering useful information when it is needed.

A weakness: a less rigorous endeavour. The victory of relevance has come at a cost: the diminishing emphasis given to the “rigour” side of the balance that contributors of Optimum were seeking 20 years ago. Program evaluation, as practiced in Canada today, more often than not, compromises its position as producer of rigorous, empiri- cal, defensible — dare I add, scientific — evidence. Program evalua- tors have collectively fallen for the easy solutions: “demonstrating” program effect based on stakeholders’ points of view, avoiding ex- perimental and quasi-experimental designs, systematically favour- ing qualitative (often softer) approaches even when quantitative (and reproducible) methods could be envisaged, and so on. Collectively, we have also been avoiding the tough questions which used to face us: we rarely tackle program relevance issues, comparative cost ben- efit questions or incremental impact concerns. Whether the slippage away from scientific rigour is a response to a lack of demand in our program evaluation market (which could be addressed by vigorous promotion efforts by our national and regional professional bodies), or to a diluted culture of empirical evaluation due to numerous inter- disciplinary rapprochements over the years, or to insufficiently fo-

163LA REVUE CANADIENNE D'ÉVALUATION DE PROGRAMME

cused evaluation training, or to some other factor, it is still a key weakness of the field today because of the threat it represents.

A threat: disappearing by lack of identity. The main threat to pro- gram evaluation today is that it is losing its individual identity as a scientific enquiry into program rationale and effectiveness. Program evaluation is more and more subsumed under parallel but essen- tially foreign sets of analytical approaches such as audit and re- view. For example, it is becoming increasingly difficult to distinguish certain forms of audits from process evaluation. Program evalua- tion has progressively lost its distinguishing character: it used to be a rigorous assessment of facts through the lens of social science aimed at improved decision-making. Today, it sometimes resembles a col- lection of opinions guided by the necessity to offer the semblance of performance information. The persistence of this vague positioning is likely to mean the disappearance of program evaluation as a field of study because, if program evaluation does not offer clear added value, other, well established domains of inquiry like auditing and accounting will take over the provision of performance information to program managers.

An opportunity: an appetite for performance information. There is a ray of hope for our trade. Since 1984 and even more so since 1994, public service managers have had to learn to work with limited pro- gram resources. One consequence of this change has been a renewed appetite for performance information. Program managers need in- telligence to make the best of available resources, to redress ineffec- tive programs, and to identify alternative approaches to addressing social problems. As evaluators, we know that we are in a particu- larly good position to offer this kind of strategic read. However, other professionals and other domains of inquiry also play on this field. In order to benefit from this opportunity, program evaluators need to demonstrate that they can provide better evidence through the cred- ible, scientific demonstration of program rationale and effectiveness — and not only through superficial description.

A New Era Jim Cullen, Public Servant, Montreal, Quebec

A strength: a professional development plan. The CES Project in Support of Advocacy and Professional Development (<http:// consultation.evaluationcanada.ca/englishhome.htm>) is an excellent opportunity to better understand evaluation practice and improve

THE CANADIAN JOURNAL OF PROGRAM EVALUATION164

program evaluation training in Canada (see Zorzi, Perrin, et al., 2002). The CES study of the benefits of evaluation has made a fun- damental contribution to our field in Canada. Kishchuk (2003) has demonstrated the importance of this CES study for framing issues of fundamental importance to program evaluation. Indeed, she pointed to the need for caution in calling for transparency in access to information, based on the overall goals of program evaluation proposed in this CES study.

A weakness: attribution and proof. The study of attribution of pro- gram effects and program evaluation based on empirical proof needs particular attention. Theoretical works as well as practical consid- erations have made evidence hard to come by, especially when seek- ing to develop a consensus on links of attribution between program operations and the apparent results. For example, some of us in edu- cation are at great pains to accept that changes in teaching and learning can be reported primarily in statistics and indicators, rather than stories and cases.

A threat: program evaluation could be left by the wayside. Without specific use of program evaluation, the new Quebec Public Adminis- tration Act will not reach its goals. The Société québécoise d’évaluation de programme (SQEP) presented a brief to the Quebec National Assembly committee in which program evaluators from Quebec clearly foresee a weakness in the new Public Administra- tion Act which makes no explicit reference to program evaluation. Results-based public administration should not only call for program evaluation but also invest public funds in the development of the qualified human resources to respond to these needs.

An opportunity: new legislative environment. The new act of the Quebec National Assembly on public administration has given pro- gram evaluation a prominent position. The enactment of this law is an opportunity for program evaluation. To the extent that the legis- lator makes full use of the aspects of the law that open up to pro- gram evaluation, the law will strengthen program evaluation, as similar laws have done federally and in other provinces.

Teaching and Learning in Evaluation Kaireen Chaytor, Consultant, Halifax, Nova Scotia

While listening to the panel presentation at the 2003 CES Confer- ence, I was struck with the number of references to the need for

165LA REVUE CANADIENNE D'ÉVALUATION DE PROGRAMME

teaching and learning on the topic of evaluation. Let me offer a few thoughts about building evaluation capacity through in-service teach- ing and learning. I use the term “teaching” to include course deliv- ery which comprises principles of evaluation and respect for many disciplines and approaches to evaluations, as well as holding rel- evance for current public policy demands. Effective learning occurs only in the presence of three ingredients: an opportunity to apply knowledge and skills based on principles, a willingness to engage in team work, and a willingness to undertake evaluation as a learning endeavour.

An opportunity: demand for skills. The demand for knowledge and skills to carry out evaluation should be viewed as a key opportunity for evaluation as a field of study. The demand is coming partly from legislation or other requirements for accountability. However, the demand speaks to the recognition of the role of evaluation and the recognition that expertise is required. The demand must be seen as an opportunity to seize.

A weakness: lack of disciplinary focus. The diversity of educational preparation to practice evaluation, while healthy in some respects, can also be viewed as a weakness. Each discipline present at any table views evaluation a bit differently. Rather than focus on the principles of evaluation, it is common to focus on the differences in a method or an approach to analysis and thus believe the differ- ences are greater than they are. The teaching and learning is ex- pected to accommodate all differences through focusing on common ground.

A threat: disconnection from management needs. Partly because of the lack of common ground, evaluation as a part of management may be under threat. Programs get their direction for evaluation from various schools of management, without a sense of the role or theory of evaluation. Program theory and the relevant public policy should be assessed using a theory of evaluation. Tying evaluation to the demands of public policy gives it relevance beyond audit or a predefined cost-benefit analysis.

A strength: a capacity to self-define. The time for evaluation to bet- ter define and present itself as a field of study having value for pub- lic sector management is now. There is an opportunity to help with accountability and reporting demands if evaluation is carried out in such a way that it addresses public sector management needs. The

THE CANADIAN JOURNAL OF PROGRAM EVALUATION166

first step is to develop teaching and learning opportunities directed at public sector need. The result should be improved capacity to carry out evaluation.

CONCLUSION

Some of the viewpoints presented here are rather pessimistic — emphasizing program evaluation weaknesses or current threats — while others are optimistic — focusing on strengths or opportuni- ties. There are a few common threads that help define the state of program evaluation in Canada in a more informed way.

Strengths. Evaluators possess skills that other professions do not offer. Evaluators are social and economic researchers who are versed at using empirical data collection and analysis methods to provide a strong foundation for program and policy assessment. Evaluators are not theoreticians; they are students of the real world, accus- tomed to dealing with the necessary compromises called for by the uncertainties of real life and focused on providing relevant informa- tion. Based on the CES Support of Advocacy and Professional De- velopment, evaluators have a collective professional development plan that can assist them to self-define. Finally, today’s evaluation practice emphasizes the involvement of program managers, the par- ticipation of stakeholders, and collaboration overall.

Weaknesses. Notwithstanding the specific skills of its practitioners, program evaluation has not acquired an identity of its own and has found it difficult to demonstrate its value, particularly in the face of short-term performance monitoring. Program evaluation has been generally unable to absorb performance measurement requirements that have surfaced in the past five years. Faced with limited re- sources, managers have made choices between evaluation and per- formance measurement that could have been made moot if program evaluators had known to position their skills and knowledge strate- gically. Where evaluators have attempted to adapt, they have some- times done so at the expense of their specificity by neglecting key evaluation issues and losing emphasis on rigour. Weak self-identity also negatively impacts the capacity of the university system to pro- duce program evaluation specialists.

Threats. The environment of program evaluation practice today presents three key interrelated threats: the dominance of program monitoring, the lack of program evaluation self-identity, and insuf-

167LA REVUE CANADIENNE D'ÉVALUATION DE PROGRAMME

ficient connection with management needs. Various events have propelled performance monitoring and short-term performance meas- urement to the front of the management scene; despite the inten- tions of the RMAF initiatives that aim to focus people on results, many managers are now content to possess a performance meas- urement framework that often focuses on the obvious outputs rather than providing a more in-depth assessment of program logic and performance to better understand why certain results are or are not observed. Positioning evaluation as an ad hoc exercise with little to no follow-up within organizations has prevented evaluators from building connections with program managers and demonstrating the value of their skills. Also, over the past 10 to 20 years, other fields of practice (e.g., auditing, accounting), with clearer identities, have encroached into what evaluators consider their program assessment territory while evaluators were less than proactive at building the professional capacity and the profile of the profession. Overall, there is a real threat to the very existence of program evaluation: lack of demonstrated value, weak self-definition as a domain of inquiry, program monitoring seen as a substitute approach by many, and lack of strategic positioning in organizations could translate into diminishing desire to fund program evaluation.

Opportunities. But there is hope. Clearly, results-based and outcome- based management are here to stay, at least for the next few years. We, evaluators, know that we can contribute significantly to making this trend a reality on the ground. We need to make that known, though, collectively through advocacy and partnership efforts and individually within our organizations by ensuring that we are ready for the challenge (getting the appropriate training, networking to learn about opportunities and difficulties, etc.). Evaluators have the opportunity to reconnect with program managers by listening to their needs and identifying how our skills can best be put to use in that context; in particular, no other field of enquiry is as poised as program evaluation is to establish the bridge between program management and policy development. But first, evaluators must clarify for them- selves who they are; they must self-define so that they can communi- cate to others what their specific contribution is and what service they provide that no other professional offers. This self-definition will also help recruit new members to the evaluation trade as would various associative actions specifically oriented to attracting students.

This analysis has a number of important implications for the prac- tice of evaluation in Canada today.

THE CANADIAN JOURNAL OF PROGRAM EVALUATION168

• Representation and advocacy: Faced with very real threats, the field of evaluation, through its professional association, must react by deploying considerable efforts in representa- tion and advocacy, making the role of evaluation in support of organizational accountability, adaptation, and learning more evident than ever. The CES has already moved in that direction (Zorzi, Perrin, et al., 2002).

• Teaching and recruitment: Universities must keep in touch with the challenges of practice and prepare students for the real world of evaluation as one of many contributors in or- ganizational dynamics. Universities are also responsible for informing students about the existence of the field and for highlighting its specific characteristics.

• Knowledge development: Researchers (mainly academics) must identify best practices in evaluation and forecast needs for adjustments to the profession to keep practitioners on track.

• Professional development: Program evaluators are respon- sible for keeping abreast of new approaches to evaluation — which goes beyond simply learning the techniques, into learning about organizational dynamics, management, and how evaluation integrates in management models.

• Leadership, locally and globally: Evaluators must exercise leadership. We must make ourselves known within our or- ganizations, particularly as experts in measurement (all measurement, including evaluation and performance moni- toring) and in the determination of results of programs and policies. We must position ourselves among the many prac- titioners who support organizations. More globally, we must be less self-centred as a field of enquiry and we must influ- ence other professionals more actively.

• Networking: Evaluators need to learn from each other. We must exchange best practices and success stories and we must support each other in practice. This emphasizes the need for local networks of evaluators (already available through CES chapters) as well as global virtual communi- ties structured around areas of interest.

• Self-valuation: Each evaluator has to come to the realiza- tion that they have something to offer that other profes- sionals don’t: evaluators are the specialists of rigorous, empirical, defensible evidence regarding results and per- formance, developed in partnership with others involved in organizational life.

169LA REVUE CANADIENNE D'ÉVALUATION DE PROGRAMME

REFERENCES

Altschuld, J.W., & Engle, M. (1994). The preparation of professional evalu- ators: Issues, perspectives and programs. New Directions for Pro- gram Evaluation, 62. San Francisco: Jossey-Bass.

Bradley, S.E. (2001). Evaluation in the Government of Alberta: Adapting to the “new way.” Canadian Journal of Program Evaluation, 16(Spe- cial Issue), 29–44.

Cabatoff, K. (2001). The long march from evaluation to accreditation: Québec’s new “government management framework.” Canadian Journal of Program Evaluation, 16(Special Issue), 73–88.

Hicks, K. (2001). Program evaluation in the Government of the Northwest Territories, 1967–2000. Canadian Journal of Program Evaluation, 16(Special Issue), 107–114.

Kishchuk, N. (2003). La transparence: Ennemi du progrès en évaluation de programmes? Retrieved on February 10, 2004 from <http:// www.sqep.ca/archives/presentations/Kishchuknatalie.pdf>.

McDavid, J.C. (2001). Program evaluation in British Columbia in a time of transition: 1995–2000. Canadian Journal of Program Evaluation, 16(Special Issue), 3–28.

Mowry, S., Clough, K., MacDonald, B., Pranger, T., & Griner, D. (2001). Evaluation policy and practice in the provincial government of Prince Edward Island. Canadian Journal of Program Evaluation, 16(Spe- cial Issue), 89–100.

Preskill, H. (2000). Coming around again: Renewing our commitment to teaching evaluation. American Journal of Program Evaluation, 21(1), 103–104.

Ross, A. (2001). Evaluation in Newfoundland: Then was then and now is now. Canadian Journal of Program Evaluation, 16(Special Issue), 101–106.

Seasons, M. (2001). Epilogue, Canadian Journal of Program Evaluation, 16(Special Issue), 115–118.

Segsworth, R. (2001). Evaluation policy and practice in Ontario. Cana- dian Journal of Program Evaluation, 16(Special Issue), 59–72.

THE CANADIAN JOURNAL OF PROGRAM EVALUATION170

Smith, M.F. (2001). Evaluation: Preview of the future #2. American Jour- nal of Program Evaluation, 22, 281–300.

Treasury Board Secretariat Canada. (no date). Results for Canadians: A management framework for the Government of Canada. Ottawa: Author.

Warrack, B. (2001). Program evaluation in the Manitoba government: Past, present and future. Canadian Journal of Program Evaluation, 16(Special Issue), 45–58.

Zorzi, R., McGuire, M., & Perrin, B. (2002, October). Projet de la Société canadienne d’évaluation visant à appuyer la défense des intérêts et le perfectionnement professionnel. Report presented to the Canadian Evaluation Society.

Zorzi, R., Perrin, B., McGuire, M., Long, B., & Lee, L. (2002). Defining the benefits, outputs, and knowledge elements of program evaluation. Canadian Journal of Program Evaluation, 17(3), 143–150.

Appendix — Sommaire

Cette annexe présente les grandes lignes des positions articulées par les collaborateurs de cet article. Elle reproduit aussi la conclu- sion et les implications de nos observations pour la pratique de l’éva- luation au Canada.

Parmi les points de vue exprimés ci-dessous, certains sont plutôt pessimistes — et mettent l’accent sur les faiblesses de l’évaluation de programme ou les menaces qui pèsent sur le champ d’étude — tandis que d’autres sont plus optimistes et mettent l’accent sur les forces et les opportunités. Il existe quelques dénominateurs com- muns qui contribuent à jeter un meilleur éclairage sur l’état de l’éva- luation de programme au Canada.

Les forces. Les évaluateurs possèdent des compétences qui leur sont propres. Ce sont des chercheurs du domaine socio-économique ver- sés dans la cueillette de données empiriques et les méthodes d’ana- lyse grâce auxquelles l’examen des programmes et des politiques repose sur de solides assises. Les évaluateurs ne sont pas des théo- riciens mais bien des observateurs de la réalité qui ont l’habitude d’affronter les compromis imposés par les incertitudes de la vie réelle

171LA REVUE CANADIENNE D'ÉVALUATION DE PROGRAMME

et qui ont à cœur de fournir une information pertinente. Avec le Soutien pour la défense des intérêts et le développement profession- nel (Zorzi, McGuire, & Perrin, 2002) de la SCÉ, les évaluateurs ont en main un plan collectif de développement professionnel pour les aider à mieux se définir. Enfin, la pratique de l’évaluation insiste de nos jours sur l’engagement des gestionnaires de programme, la participation des intéressés et la collaboration de tous et chacun.

Les faiblesses. Malgré les compétences particulières de ses prati- ciens, l’évaluation de programme n’a pas encore acquis d’identité propre, et il a été difficile d’en démontrer la valeur, notamment en regard du suivi du rendement à court terme. De façon générale, l’éva- luation de programme n’a pas pu absorber les exigences de la me- sure du rendement qui se sont profilées au cours des cinq dernières années. Aux prises avec une pénurie de ressources, les gestionnai- res ont fait, entre l’évaluation et la mesure du rendement, des choix qui auraient pu être évités si les évaluateurs de programme avaient su revêtir d’un intérêt stratégique leurs compétences et leurs con- naissances. Dans leurs tentatives d’adaptation, les évaluateurs ont parfois renoncé à leur spécificité en négligeant des enjeux clés de l’évaluation et en donnant moins d’importance à la rigueur. La fai- blesse de l’identité propre nuit à la production de spécialistes de l’évaluation de programme à l’université.

Les menaces. Le contexte dans lequel se pratique de nos jours l’éva- luation de programme présente trois grandes menaces reliées entre elles: la faveur accordée au suivi des programmes, l’absence d’une identité propre à l’évaluation de programme et une prise en consi- dération insuffisante des besoins des gestionnaires. Divers événe- ments ont permis au suivi du rendement et à la mesure du rendement à court terme de se hisser à l’avant-plan dans le domaine de la ges- tion; bon nombre de gestionnaires se satisfont d’un cadre de la me- sure du rendement qui ne mentionne souvent que les résultats les plus évidents au lieu de procéder à un examen plus approfondi de la logique d’un programme et de son rendement en vue de mieux sai- sir la raison d’être de la présence ou de l’absence de certains résul- tats. Le fait que l’évaluation soit perçue comme un exercice ponctuel sans nécessité de suivi, ou presque, au sein des organisations a em- pêché les évaluateurs de tisser des liens avec les gestionnaires de programme et de leur démontrer leur savoir-faire. En outre, au cours des 10 à 20 dernières années, d’autres champs à l’identité mieux définie (comme la vérification, la comptabilité, etc.) se sont immis- cées dans le champ de l’examen de programme que les évaluateurs

THE CANADIAN JOURNAL OF PROGRAM EVALUATION172

estiment être leur territoire, alors que les évaluateurs eux-mêmes ne se préoccupaient pas suffisamment d’affirmer leurs capacités professionnelles et de dresser le profil de leur profession. Dans l’en- semble, l’existence même de l’évaluation de programme se trouve menacée: sa valeur mal mise en évidence, sa faible autodéfinition en tant que domaine d’enquête, un suivi de programme constituant aux yeux de plusieurs un substitut et le manque de positionnement stratégique auprès des organisations risquent de se traduire par un désintérêt pour le financement de l’évaluation de programme.

Les opportunités. Il y a toutefois de l’espoir. La nécessité d’axer la gestion sur les résultats et les impacts est un impératif qui devrait persister, du moins pendant encore quelques années. Nous savons, nous, évaluateurs, que nous pouvons contribuer de façon importante à ce que cette tendance devienne une réalité sur le terrain. Mais il faut en persuader les autres, collectivement, au moyen de la promo- tion et du partenariat, et sur le plan individuel, au sein de nos orga- nisations, en nous armant pour mieux relever ce défi (par une formation adéquate, un maillage pouvant nous renseigner sur les possibilités et les difficultés, etc.). Il est possible pour les évaluateurs de renouer avec les gestionnaires de programme en se mettant à l’écoute de leurs besoins et en leur montrant comment profiter de nos compétences dans ce contexte; en particulier, aucun autre champ d’étude n’est autant en mesure que l’évaluation de programme de créer un pont entre la gestion de programmes et l’élaboration des politiques. Cependant, il faut en tout premier lieu que les évaluateurs se connaissent eux-mêmes; ils doivent s’autodéfinir pour pouvoir renseigner les autres sur la contribution qu’ils peuvent leur appor- ter et sur les services qu’aucune autre profession n’est à même de leur offrir. Cette autodéfinition aura aussi pour effet de recruter de nouveaux membres, tout comme le feront diverses activités associa- tives tendant précisément à attirer des étudiants.

La présente analyse a d’importantes incidences sur la pratique cou- rante de l’évaluation au Canada.

• Représentation et promotion: Face à des menaces très réel- les, la discipline doit, par l’intermédiaire de son association professionnelle, réagir en déployant des efforts considéra- bles en matière de représentation et de promotion afin de rendre plus évident que jamais le rôle de l’évaluation pour mieux appuyer la responsabilisation, l’adaptation et l’ap-

173LA REVUE CANADIENNE D'ÉVALUATION DE PROGRAMME

prentissage organisationnels. La SCÉ est déjà engagée dans cette voie (Zorzi, Perrin, et al., 2002).

• Enseignement et recrutement: L’université doit se tenir à l’affût des défis de la pratique et préparer les étudiants aux réalités de l’évaluation en tant que participante, avec bien d’autres champs d’étude, à la dynamique organisationnelle. L’université a aussi pour tâche de faire connaître aux étu- diants l’existence de l’évaluation et d’en souligner les ca- ractéristiques particulières.

• Développement du savoir: Les chercheurs (universitaires, surtout) doivent établir quelles sont les meilleures prati- ques d’évaluation et exposer les adaptations à apporter à la profession dans l’intérêt de ses membres.

• Perfectionnement professionnel: Les évaluateurs de pro- gramme doivent se tenir au courant des nouvelles façons d’aborder l’évaluation — ce qui ne se borne pas à appren- dre les techniques mais aussi à se renseigner sur la dyna- mique de l’organisation, sur la gestion ainsi que l’intégration de l’évaluation aux modèles de gestion.

• Leadership à l’échelle locale et globale: L’évaluateur doit faire preuve de leadership. Nous devons nous faire connaître au sein de nos organisations, notamment comme spécialistes de la mesure (de toute nature, y compris en examen et suivi du rendement) et de l’établissement des résultats des pro- grammes et des politiques. Il nous faut nous positionner parmi les nombreux intervenants qui soutiennent les orga- nisations. De manière plus globale, il faudra faire en sorte que notre champ d’étude soit moins centré sur lui-même, et exercer une véritable influence sur les membres des autres professions.

• Maillage: Les évaluateurs doivent apprendre les uns des autres. Nous devons nous communiquer nos meilleures méthodes et nos réussites ainsi que nous appuyer mutuel- lement dans la pratique. D’où la nécessité de former des réseaux d’évaluateurs à l’échelle locale (rendus possibles grâce aux sections de la SCÉ) de même que des collectivités virtuelles organisées en fonction des champs d’intérêts.

• Valorisation personnelle: Chaque évaluateur doit se persua- der que la profession a quelque chose de spécial à offrir; les évaluateurs sont les spécialistes de la démonstration rigou- reuse, empirique et défendable touchant les résultats et le rendement, mise au point en partenariat avec d’autres in- tervenants de la vie organisationnelle.

THE CANADIAN JOURNAL OF PROGRAM EVALUATION174 C

on tri

bu te

ur Fo

rc e

Fa ib

le ss

e M

en ac

e O

pp or

tu ni

té

Ro be

rt M

al at

es t,

co ns

ul ta

nt ,

Vi ct

or ia

G ai

l V . B

ar rin

gt on

, co

ns ul

ta nt

e, C

al ga

G re

g M

as on

, co

ns ul

ta nt

, W

in ni

pe g

An ita

M ye

rs ,

un iv

er si

ta ire

, W

at er

lo o

Sa nd

ra L

. B oz

zo ,

fo nc

tio nn

ai re

, To

ro nt

N an

cy L

. Po

rte ou

s, fo

nc tio

nn ai

re ,

O tta

w a

Si m

on R

oy ,

co ns

ul ta

nt , O

tta w

U n

en ga

ge m

en t à

fa ire

ra pp

or t

su r l

es ré

su lta

Li er

p lu

s ét

ro ite

m en

t i m

pu ta

bi lit

é, re

dd iti

on d

e co

m pt

es e

t év

al ua

tio n

O ffr

ir un

p on

t e nt

re la

g es

tio n

de s

pr og

ra m

m es

e t l

e dé

ve lo

pp em

en t

de s

po lit

iq ue

Le m

en to

ra t d

es é

tu di

an ts

im pl

iq ué

s en

é va

lu at

io n

Le p

ot en

tie l d

e co

lla bo

ra tio

n et

de p

ar te

na ria

t e nt

re le

s ju

rid ic

tio ns

e t e

nt re

le s

or ga

ni sa

tio ns

L’ ac

tio n

as so

ci at

iv e

pa r l

e bi

ai s

de la

S C

É et

d ’a

ut re

s as

so ci

at io

ns co

m m

e l’I

O C

D an

s le

c ad

re d

e la

p la

ni fic

at io

n d’

év al

ua tio

ns , l

’é ta

bl is

se m

en t

d’ ob

je ct

ifs p

ou r l

es p

ro gr

am m

L’ év

ol ut

io n

de s

m ét

ho de

s d’

év al

ua tio

n ve

rs p

lu s

de rig

ue ur

U n

pl us

g ra

nd n

iv ea

u de

so ph

is tic

at io

n fa

ce à

l’é va

lu at

io n

Au cu

Le fo

rt lie

n en

tre l’é

va lu

at io

n et

la ré

al ité

d u

te rr

ai n

L’ en

ga ge

m en

t p ol

iti qu

e on

ta rie

n en

fa ve

ur d

e l’é

va lu

at io

L’ ac

ce nt

m is

s ur

la co

lla bo

ra tio

n et

s ur

l’é qu

ili br

e en

tre ri

gu eu

r e t

pa rti

ci pa

tio n

L’ im

pl ic

at io

n de

s ge

st io

nn ai

re s

de pr

og ra

m m

es d

an s

La p

la ni

fic at

io n

in ad

éq ua

te de

s év

al ua

tio ns

La d

ép en

da nc

e pa

r r ap

po rt

à la

m es

ur e

du re

nd em

en t

La p

as si

vi té

d e

la co

m m

un au

té d

es é

va lu

at eu

rs fa

ce a

ux m

en ac

es a

ct ue

lle s

Le m

an qu

e d’

id en

tit é

di sc

ip lin

ai re

d is

tin ct

L’ in

si st

an ce

s ur

le s

qu es

tio ns

re lié

es à

l’ ef

fic ie

nc e

Le n

iv ea

u d’

ef fo

rt re

qu is

p ou

r m

en er

à b

ie n

un e

ét ud

e év

al ua

tiv e

La m

is e

en s

ou rd

in e

d’ en

je ux

d’ év

al ua

tio n

im po

rta nt

s as

so ci

és à

la ra

is on

d ’ê

tre ,

La ré

du ct

io n

de la

c ap

ac ité

d’ év

al ue

r a u

se in

d u

go uv

er ne

m en

t d e

C ol

om bi

e- Br

ita nn

iq ue

La d

év al

ua tio

n de

l’ év

al ua

tio n

à ca

us e

du m

an qu

e d’

im pa

ct s

ta ng

ib le

s su

r l a

pr is

e de

dé ci

si on

La tr

op g

ra nd

e ac

ce pt

at io

n pa

r le

s év

al ua

te ur

s du

m od

èl e

de la

m es

ur e

du re

nd em

en t à

c ou

rt te

rm e

La d

iff ic

ul té

d ’in

té gr

er l’é

va lu

at io

n da

ns le

m od

èl e

un iv

er si

ta ire

c la

ss iq

La c

ap ac

ité li

m ité

e de

s or

ga ni

sa tio

ns d

e liv

re r d

es év

al ua

tio ns

c ré

di bl

La d

iff ic

ul té

à d

ém on

tre r l

a va

le ur

d e

l’é va

lu at

io n

da ns

le s

or ga

ni sa

tio ns

Le s

bi ai

s po

ss ib

le s

dé co

ul an

t d e

l’i m

pl ic

at io

n ét

ro ite

d es

ge st

io nn

ai re

s de

p ro

gr am

m es

e n

175LA REVUE CANADIENNE D'ÉVALUATION DE PROGRAMME

Jo hn

M ay

ne ,

fo nc

tio nn

ai re

, O

tta w

Ro be

rt La

he y,

fo nc

tio nn

ai re

, O

tta w

Be no

ît G

au th

ie r,

co ns

ul ta

nt ,

G at

in ea

Ji m

C ul

le n,

fo nc

tio nn

ai re

, M

on tré

Ka ire

en C

ha yt

or ,

co ns

ul ta

nt e,

H al

ifa x

pr oc

es su

s év

al ua

tif

L’ ac

ce nt

s ur

la p

ra tiq

ue te

rr e

à te

rr e

pl ut

ôt q

ue s

ur l’a

cc ro

is se

m en

t d e

la co

nn ai

ss an

La c

ap ac

ité a

na ly

tiq ue

d es

év al

ua te

ur s

La p

er tin

en ce

d es

tr av

au x

po ur

le s

ge st

io nn

ai re

s vi

sé s

L’ ex

is te

nc e

d’ un

p la

n de

fo rm

at io

n pr

of es

si on

ne lle

La c

ap ac

ité d

es év

al ua

te ur

s de

s ’a

ut o-

dé fin

ir en

ta nt

q ue

c ha

m p

de c

on na

is sa

nc e

au x

pa ra

m èt

re s

de c

on ce

pt io

n et

à la

p re

st at

io n

du s

er vi

L’ in

té gr

at io

n in

su ffi

sa nt

e av

ec le

s be

so in

s de

s ge

st io

nn ai

re s

de p

ro gr

am m

Le m

an qu

e de

re ss

ou rc

es hu

m ai

ne s

qu al

ifi ée

s en

év al

ua tio

La d

im in

ut io

n de

l’ at

te nt

io n

po rté

e à

la ri

gu eu

r m

ét ho

do lo

gi qu

e de

s re

ch er

ch es

Le m

an qu

e d’

at te

nt io

n ac

co rd

ée à

la m

es ur

e de

l’a ttr

ib ut

io n

de s

ef fe

ts d

es pr

og ra

m m

Le m

an qu

e d’

ho m

og én

éi té

di sc

ip lin

ai re

d an

s la

fo rm

at io

n de

s év

al ua

te ur

m es

ur e

du re

nd em

en t

La p

er te

d e

l’i nd

ép en

da nc

e di

sc ip

lin ai

re a

lo rs

q ue

d es

pr of

es si

on s

m ie

ux o

rg an

is ée

s en

va hi

ss en

t l e

te rr

ito ire

d e

l’é va

lu at

io n

D es

li en

s té

nu s

av ec

la h

au te

di re

ct io

n de

s or

ga ni

sa tio

La d

is pa

rit io

n ca

us ée

p ar

u n

m an

qu e

d’ id

en tit

é pr

of es

si on

ne lle

d is

tin ct

iv e

L’ ab

se nc

e de

ré fé

re nc

e ex

pl ic

ite à

l’é va

lu at

io n

da ns

la ré

ce nt

e ré

fo rm

e de

l’ ad

m in

is tra

tio n

pu bl

iq ue

a u

Q ué

be c

Le m

an qu

e de

c on

na is

sa nc

e de

s be

so in

s de

s ge

st io

nn ai

re s

de pr

og ra

m m

vi sé

C on

tri bu

er à

fa ire

d u

se rv

ic e

pu bl

ic b

as é

su r l

es ré

su lta

ts u

ne ré

al ité

L’ ap

pé tit

a ct

ue l p

ou r l

’in fo

rm at

io n

su r l

e re

nd em

en t

La d

em an

de é

le vé

e po

ur l’i

nf or

m at

io n

su r l

e re

nd em

en t e

t la

c ap

ac ité

d es

é va

lu at

eu rs

d e

pr od

ui re

u n

sa vo

ir sc

ie nt

ifi qu

e cr

éd ib

L’ im

po rta

nc e

de la

p en

sé e

év al

ua tiv

e da

ns la

n ou

ve lle

L oi

su r l

’ad m

in is

tra tio

n pu

bl iq

ue d

u Q

ué be

La d

em an

de im

po rta

nt e

po ur

le s

co nn

ai ss

an ce

s et

le s

ha bi

le té

s de

s év

al ua

te ur

THE CANADIAN JOURNAL OF PROGRAM EVALUATION176

Benoît Gauthier (M.A., MPA, CMC) specializes in strategic and organizational research and intervention, market research, program evaluation, applied social research, and policy analysis. He was an evaluator and evaluation director in federal departments and agen- cies before joining a research consulting firm in Ottawa. Early in 1996, he founded Circum Network Inc. Since then, he has devel- oped an interest in organizational management issues, bridging measurement concerns and management concerns.

Dr. Gail Barrington established Barrington Research Group, Inc. in 1985 and since then has conducted or managed over 100 evalua- tion studies, many of a complex nature, for a wide variety of clients, from local not-for-profits to federal government departments. She has been a Certified Management Consultant since 1988, and fre- quently presents workshops in both Canada and the United States on evaluation topics and consulting skills.

Sandra L. Bozzo (B.Sc., M.E.S.) has over a decade of experience providing evaluation services to public and non-profit organizations. She has previously published on the topic of evaluation capacity building in the voluntary/non-profit sector. She has worked in evalu- ation with both the federal and provincial governments. As a volun- teer, she sits on the CES National Council and the Board of the CES Ontario Chapter.

Dr. Kaireen Chaytor, Chaytor Consulting Services Ltd. and Ad- junct Professor, School of Public Administration, Dalhousie Univer- sity, Halifax, conducts evaluations and teaches program evaluation in the non-profit sector and in the Government of Nova Scotia. She has prepared teaching materials for developing evaluation frame- works, and was the 2003 recipient of the CES award for contribu- tion to theory and practice of evaluation in Canada.

Jim Cullen is a policy analyst for the English services unit of the Québec Ministry of Education. He is currently acting as assistant to the Assistant Deputy Minister for the English-speaking Commu- nity, responsible for the Canada-Quebec Agreement on Minority and Second Language Education. He is a member of the Board of Direc- tors of the Société Québécoise d’évaluation de programme, and he represents CES on the Joint Committee on Standards for Educa- tional Evaluation.

Robert Lahey is the founding head of the Centre of Excellence for Evaluation (CEE), the policy centre for program and policy evalua-

177LA REVUE CANADIENNE D'ÉVALUATION DE PROGRAMME

tion in the Canadian federal government — focused on rebuilding capacity and repositioning the function in line with results-based management. His 32-year career in the federal government, where he headed the evaluation unit in five federal organizations, is now being followed up by a broad set of interests in evaluation capacity building and professional development for the community at large.

Robert Malatest (BA, MA [Economics]) is the principal of R.A. Malatest & Associates Ltd., based in Victoria. He has managed more than 100 evaluations during the past 15 years, including several major national evaluations of education, technology, and housing programs. These evaluations have included both formative and summative studies as well as research projects examining the im- pacts of selected economic and social programs on various groups.

Greg Mason, managing partner of Prairie Research Associates (PRA) Inc., is a Certified Management Consultant and economist specializing in applied policy analysis, forecasting, consumer demand analysis, survey research, and program evaluation. His consulting assignments have spanned a broad range of projects including pro- gram and policy evaluations, market research, economic and econo- metric forecasts, regulatory analysis, and feasibility studies. Trained as a quantitative researcher, he has also conducted and managed an extensive range of qualitative research.

Dr. John Mayne is an independent consultant. Until recently, he was at the Office of the Auditor General where he led efforts at de- veloping practices for effective performance measurement, manag- ing for results, and performance reporting in the Government of Canada, developing methodologies for providing audit assurance on performance reports, and leading the Office’s audit efforts in account- ability and governance. He has authored numerous articles and re- ports, and edited five books in the areas of program evaluation, public administration, and performance monitoring. In 1989 and in 1995, he was awarded the CES Award for Contribution to Evaluation in Canada.

Dr. Anita Myers is a full professor in the Department of Health Studies & Gerontology at the University of Waterloo. In addition to extensive teaching experience, she has published numerous journal articles and a book on program evaluation. She serves on the CJPE editorial board, was on the CES national council from 1993 to 1997, and received the CES Contribution to Evaluation in Canada award in 1997.

THE CANADIAN JOURNAL OF PROGRAM EVALUATION178

Nancy L. Porteous has 13 years of public and private sector ex- perience in planning, evaluation, and performance management spe- cializing in health and human services. Currently at Health Canada, she is past president of the CES and member of the executive of the Board of Trustees for the International Organization for Coopera- tion in Evaluation.

Simon Roy is principal at Goss Gilroy Inc. He has over 15 years of social research experience and has been involved in program evalu- ation for the last 8 years. Simon completed a Ph.D. in sociology at the University of Paris X in 1995.