summary 2 pages

chapter18.pdf

Home >Education homework help >summary 2 pages

490

The Future of Evaluation

Orienting Questions

1. How are future program evaluations likely to be different from current evaluations in • the way in which political considerations are handled? • the approaches that will be used? • the involvement of stakeholders? • who conducts them?

2. How is evaluation like some other activities in organizations?

3. How is evaluation viewed differently in other countries?

We have reached the last chapter of this book, but we have only begun to share what is known about program evaluation. The references we have made to other writings reflect only a fraction of the existing literature in this growing field. In choosing to focus attention on (1) alternative approaches to program evaluation, and (2) practical guidelines for planning, conducting, reporting, and using evalu- ation studies, we have tried to emphasize what we believe is most important to include in any single volume that aspires to give a broad overview of such a complex and multifaceted field. We hope we have selected well, but we encourage students and evaluation practitioners to go beyond this text to explore the richness and depth of other evaluation literature. In this final chapter, we share our perceptions and those of a few of our colleagues about evaluation’s future.

The Future of Evaluation

Hindsight is inevitably better than foresight, and ours is no exception. Yet present circumstances permit us to hazard a few predictions that we believe will hold true for program evaluation in the next few decades. History will determine whether

Chapter 18 • The Future of Evaluation 491

Predictions Concerning the Profession of Evaluation

1. Evaluation will become an increasingly useful force in our society. As noted, evaluation will have increasing impacts on programs, on organizations, and on society. Many of the movements we have discussed in this text—performance monitoring, organizational learning, and others—illustrate the increasing interest in and impact of evaluation in different sectors. Evaluative means of thinking will improve ways of planning and delivering programs and policies to achieve their intended effects and, more broadly, improve society.

2. Evaluation will increase in the United States and in other developed countries as the pressure for accountability weighs heavily on governments and nonprofit organizations that deliver vital services. The emphasis on accountability and data-based decision making has increased dramatically in the first decade of the twenty-first century. Also, virtually every trend points to more, not less, eval- uation in the public, private, and nonprofit sectors in the future. In some organi- zations, the focus is on documenting outcomes in response to external political pressures. In other organizations, evaluation is being used for organizational growth and development, which should, ultimately, improve the achievement of those outcomes. In each context, however, evaluation is in demand.

or not our predictions are accurate enough to add prophecy to the repertoire of techniques useful to evaluators.

We believe that evaluation will continue to spread rapidly around the globe, until there are few countries, territories, provinces, states, and locales in which pro- gram evaluations are not at least an occasional occurrence. As we have noted, the spreading interest in program evaluation has been evident for some years in the development of evaluation associations and activities around the world. We also be- lieve that evaluation will become an increasingly useful force in the following ways:

• Improving programs, thus improving the lot of those intended to benefit from those programs

• Improving policy making by governing boards, legislators, and congressional and parliamentary bodies

• Improving organizational learning and decision making • Improving societies through improving the social conditions programs address • Improving even itself

If these predictions seem overly optimistic, it may underscore our earlier point that evaluators may not always be completely unbiased. Yet these forecasts do not strike us as unrealistic or overdrawn; we are willing to submit them to the test of time.

Now let us move to more specific predictions concerning the profession of evaluation and its practice (Worthen, 2001).

492 Part IV • Practical Guidelines for Conducting and Using Evaluations

3. Evaluation will continue to spread rapidly around the globe until there are few countries where program evaluations are not at least an occasional occur- rence. In addition, the number of national and multinational professional societies for evaluation will burgeon.

4. The opportunity for careers in evaluation will continue to increase as the demand for evaluation skills grow. As LaVelle and Donaldson write in 2010, “Evaluation practice has grown in leaps and bounds in recent years” (2010, p. 9). The growth in evaluation in the United States is demonstrated by the dramatic growth in membership of the American Evaluation Association in the last decade (Mark, 2007).

5. Graduate programs in evaluation will increase with the growing demand. Lavelle and Donaldson (2010) found that the number of graduate train- ing programs in evaluation increased to 48 in 2008, a dramatic increase since 2006 when there were only 27 such programs following several years of decline. In- creases were particularly notable in schools of education, but programs were also found in public policy, psychology, criminal justice, and sociology. More than half of the 48 programs, however, were small ones, offering only two to three evaluation-specific courses. As the profession grows, and the market demands more professionals trained specifically in evaluation, we hope the incidence and depth of such programs grow as well. A world with growing evaluation demands requires evaluations that are conducted by people trained in the many options that evaluators can pursue.

6. Many of those conducting evaluations will need more specific evalua- tion training. Graduate programs cannot keep up with the demand. Further, since evaluation is a relatively new profession, many are not aware of the profession and the specific approaches and methodologies professional evaluators use. Many of those conducting evaluations within organizations, and as external consultants, continue to be trained in social science methodologies, or in organizational man- agement, but have not received in-depth evaluation training. As educators, public and nonprofit administrators, corporate officers, and those in a variety of other roles are asked to assume some responsibility for carrying out evaluation studies along with their other professional duties, the need for in-service education in evaluation will grow (Datta, 2006).

7. Internal evaluation will, despite its risks, become more important because of its benefits. Internal evaluators know the organizational environment. They can provide an important ongoing influence to encourage organizational learning and to use evaluation skills across the organization in many different en- deavors, from using new information technology to human resource management and traditional evaluation of programs. We predict there will be increased coopera- tion between internal and external evaluators in many evaluations.

8. Professional associations will continue to grow and to branch into new areas to expand the public presence of evaluation. In 2010, membership in the American Evaluation Association (AEA) should reach 6,000 (Kistler, 2010).

Chapter 18 • The Future of Evaluation 493

The AEA and other societies of practicing evaluators and/or evaluation theoreti- cians will continue to contribute to evaluation’s maturation. Like other profes- sional associations, the AEA has begun taking public positions on issues of relevance to evaluation and has recently begun to direct its attention to evaluation policy and working to influence such policies. (See Trochim, Mark, and Cooksy [2009]) The Canadian Evaluation Society has developed a program to recognize professional evaluators through a credentialing process that recognizes those who apply and meet the criteria as Credentialed Evaluators. As in other professions, this process is intended to help clients and stakeholders to distinguish professional evaluators from those with less direct training or experience. Through these efforts and continuing education regarding evaluation practice and standards, associa- tions will work to educate stakeholders about the potential that evaluation offers and the risks entailed in using it inappropriately.

9. Evaluation literature will increase in both quantity and quality, but relatively little of it will be based on research into the process of evaluation itself. Current funding agencies do not seem interested in supporting research con- cerning the evaluation process. Governments are investing many resources in accountability, performance monitoring, and evaluation to determine if programs work and how they work. However, our approaches to these subjects continue to rely on reasoning and intuition rather than solid evidence about how evaluators can best work with stakeholders, what forms of participation lead to what types of impacts and use, and so forth. Therefore, the empirical knowledge base in evaluation will increase very slowly. As evaluation expands, there is a critical need for more research on what occurs, what works, and what doesn’t in evaluation practice, participation, and use.

Predictions Concerning the Practice of Evaluation

As the profession grows and expands, practice will change even more dramatically.

1. Approaches to evaluation will become more eclectic and adaptive to contextual circumstances. Program evaluation will continue to be pluralistic, but will move toward greater pragmatism as evaluators work to provide valid and appropriate findings and conclusions and to improve programs, policies, and decision making in many different settings. Single-method evaluations will be viewed by professional evaluators, if not by the public and some elected officials, as simplistic and inadequate for evaluation of complex programs or those serving diverse populations. Triangulation, cross-validation, and iterative, expansive designs will be used more routinely to allow the complementarity of qualitative and quantitative approaches to enrich evaluation work. The usefulness of the different approaches will lie less in having any one of them serve as a model to be followed slavishly but rather, as House (1994a) has suggested, as collectively

494 Part IV • Practical Guidelines for Conducting and Using Evaluations

comprising the “grammar of evaluation” that evaluators must understand and be skilled in using:

[One] might see the evaluation models as something like model sentences in a gram- mar of evaluation. . . . As one progresses, . . . one does not need to think about the models consciously, except to correct particular errors or study the grammar itself.

Similarly, . . . experienced evaluators can construct evaluation designs which do not depend explicitly on particular models. Actual evaluation designs can be combinations of elements from different models, . . . just as speakers can produce novel grammatical sentences once they have learned the basic grammar of a language. (pp. 241–242)

2. Evaluation will be mainstreamed in organizations. Some of its methods, such as logic models and program theories, are already being used by program managers and staff to develop programs. As process learning from evaluation and organizational learning increase, evaluative ways of thinking in organizations will expand. Evaluation won’t always be called evaluation, but its influence will be felt through creating a culture of learning and using information and data to make decisions (Mark, 2007).

3. Evaluation will expand to evaluate programs in new arenas. In the United States and Canada, evaluators have come primarily from education and psy- chology and have evaluated programs in those areas. But, the role of evaluation in housing, social welfare, environmental programs, city planning, transportation, health, criminal justice, biotechnology, recreation, and environmental programs continues to expand. Working in these new areas will prompt evaluators to expand their repertoire of approaches and methods to adapt to these new contexts, new po- litical dynamics, and new issues to explore and investigate. In Europe, evaluators typically come from the disciplines of political science, economics, and public administration and, as a result, focus their efforts on different types of programs using somewhat different methods. Our growing awareness of these differences makes us realize that evaluators in each country can learn from practice in other countries.

4. Evaluators will become more aware of and involved in the work of planners, policy analysts, and organizational developers. Evaluation activities overlap with the work of policy analysts, planners, and organizational developers. Evaluators have approaches and qualitative methods that could contribute to the work of policy analysts. They have economic and analytic methods that could add to the repertoire of evaluators. Similarly, city planners and program planners collect information in ways that are similar to an evaluator conducting a needs assessment. Evaluators’ skills in developing logic models and program theory can help in pro- gram planning. We predict that more communication will occur across these fields, with professionals sharing approaches and methods. As noted previously, the work they do may not be called evaluation, but evaluation professionals will be bringing their evaluative skills to the tasks.

5. Evaluation (and evaluators) will become more politically sophisticated, recognizing that our goal is to encourage policymakers and program managers to use evaluative information to make decisions and to educate voters and the

Chapter 18 • The Future of Evaluation 495

public. As we move to more eclectic and adaptive evaluation practices and avoid our own infighting, we can be more successful in this venture. Greene and Henry advise us to recognize that we do not want evaluation’s disputes to “become a license for actions based entirely on ideology or the force of unconstrained rhetoric. We should unite in our commitment . . . to reclaim the conversation about the contri- butions of social science to social policies and programs” (2005, p. 350). Ideology and rhetoric are part of the political system, just as is public opinion. Information, conclusions, and judgments provided by evaluation studies become one piece of this mix of input that policymakers receive. For our activities to rise to the fore, evaluators must balance the perceived objectivity of their roles against entering the political fray to call attention to information that can improve programs and policy. Today, evaluation studies compete with information supplied by partisan, overtly political think tanks. Citizens, and some policymakers, may not know the differ- ence. Evaluators must be skilled at helping stakeholders recognize the strengths of our work.

6. Attention to ethical issues will increase as evaluators become more in- volved in political issues. The present evaluation Standards and Guiding Principles— and their descendants—provide a means for maintaining the credibility of evaluation and educating others about codes and ethics in an increasingly politicized environ- ment. Professional accountants have strengthened their ethical codes and training of practitioners in the face of public disillusionment concerning the “independent” role of accountants in reviewing the financial practices of corporations (Fitzpatrick, 1999). Evaluators can avoid the debacle that the Arthur Andersen Accounting firm faced in its auditing of Enron, and those that Standard and Poors and other bond rating firms endured as their high-rated bonds fell in the economic crisis of 2009, by strengthen- ing the ethical education of current and future evaluators.

7. Electronic and other technological advances will alter the way eval- uators collect information, draw conclusions, and report findings, en- abling broader stakeholder participation and access to evaluation reports and their findings. Today, data are routinely collected through Internet surveys and interviews. Results can be shared and discussed online with members of the evaluation team and advisory councils to consider interpretations. Databases can be shared so members can explore the data for different interpretations. Interim and final reports are routinely posted on organizational web sites with links to videos and audio depicting the program or demonstrating results. Readers are en- couraged to post comments. Such uses, and others as yet unanticipated, will in- crease as technological capacity and literacy increase.

8. Efforts will increase to democratize evaluation as part of the movement in many countries for more citizen input. Across the United States, deliberative democracy methods are being used, where local citizens serve with elected officials and others to learn about policy issues and make recommendations. Participatory evaluation is part of that movement. We will continue to democratize evaluation by involving many different stakeholders and educating them on evaluative ways of thinking and purposes.

496 Part IV • Practical Guidelines for Conducting and Using Evaluations

9. The performance measurement movement will grow in response to per- sistent demands for accountability. Performance measurement, in some form or fash- ion, is now mandated in most local, state, and federal government agencies and in nonprofit organizations led by initiatives from United Way and the World Bank. Ex- pectations from the public and from policymakers who mandated performance mea- surement are high. Yet most managers lack the expertise to collect meaningful measures of outcomes. Newcomer (2001) notes that professional evaluators will play an important role in making this process more than simply a reporting exercise. Eval- uators can help build program theory to link outcomes to program activities and, hence, make the outcome information useful for formative purposes. Further, eval- uators’ methodological expertise will be necessary to measure outcomes.

Performance measurement, however, also presents potential hazards for the evaluation field. Just as states’ testing of students on educational standards has grossly simplified learning goals and focused educational evaluation activities on just this issue, so, too, can performance measurement simplify and narrow evalu- ation activities. Many policymakers and managers underestimate the challenge of measuring program outcomes and, because of the mandated nature of perfor- mance measurement, tend to see performance measurement as all that evaluation does. Evaluators need to be active in this area to bring their expertise to bear.

A Vision for Evaluation

In addition to the predictions we have for the profession and practice of evalua- tion, we also have some visions, or goals, for it. These differ from predictions in that the evidence is not so clear as to whether these visions will be realized. Nevertheless, we would be remiss if we ended this book without describing that vision. It includes the following:

1. A global valuing of evaluation that cuts across boundaries of professional fields, job categories, sectors of society, geopolitical lines, cultures—that is, formal disciplined evaluation as a pervasive value. How will we bring about this valuing? By making others aware of evaluation and its importance. By helping those who are mandated to do evaluations to see its worth even when it is not mandated. By instilling evaluation institutions, policies and procedures, and evaluative ways of thinking in organizations (Sanders, 2001).

2. Continuing a constructive use of multiple methods and eclectic approaches to achieve the many different purposes of evaluation. The de- bates over qualitative and quantitative methods have subsided and many have moved on to the practical issues of applying their now increased methodological tools in a variety of settings. To avoid future divisive debates we should recognize the plurality of evaluation purposes, questions, and settings. An evaluator working with the U.S. government on performance monitoring issues is facing different method- ological and political challenges than the evaluator designing a special, formative

Chapter 18 • The Future of Evaluation 497

study for a nonprofit agency on its work with a new group of clients. Rather than debate the different choices these evaluators make, we should study their choices and learn more about what approaches work best in different settings. As evaluators, we should know not to judge decisions made in other evaluations without sufficient information. We need to work harder to defer that judgment and explore and col- lect information on those choices. Let’s develop thick descriptions about evaluations!

3. Increase the use of metaevaluation to improve evaluation practice. One type of publication that is regrettably rare in the evaluation journals is cri- tiques of prior evaluation reports, that is, metaevaluations. Despite the acceptance and availability of the Joint Committee’s Standards, few evaluations appear to be subjected to any closer scrutiny today than before metaevaluation standards were developed. To learn from our own work, we must be open to its review and evaluation. As others learn from our evaluations, evaluators can learn from eval- uations of their own work.

Conclusion

We leave the reader with two final thoughts. First, all our years of experience conducting and studying it convinces us that

evaluation, properly conducted, has great potential for improving programs and practices in education, human services, nonprofit organizations—virtually every area of society. Managers, policymakers, and other stakeholders have become aware that some evaluation studies are misused or ignored, with the result that some individuals have argued for decreased emphasis on the evaluative process. But that seems no more sensible than abandoning medical diagnosis because sci- ence has not yet succeeded in eliminating all disease.

The second thought we wish to leave with readers is this: Despite great strides, it is increasingly apparent how little we really do know about evaluation, compared with what we need to know. It is our earnest hope that this book has added to that knowledge and has helped to illuminate the thousand points of darkness that still constitute current processes for creating and implementing the policies and programs intended to improve the lot of humankind.