5A1-9 - Program Evaluation - Analysis of Data Collection

failing to anticipate the myriad situations inevitable in practice (Bamberger, Rugh, & Mabry, 2012)—hence the call for cultivating sound professional judgment (through reflective practice) in applying the principles and guidelines.

Like other professional judgment decisions, appropriate ethical practice occurs throughout the evaluation process. It usually falls to the evaluator to lead by example, ensuring that ethical principles are adhered to and are balanced with the goals of the stake-holders. Brandon, Smith, and Hwalek (2011), in discussing a successful private evaluation firm, describe the process this way:

Ethical matters are not easily or simply resolved but require working out viable solutions that balance professional independence with client service. These are not technical matters that can be handed over to well-trained staff or outside contractors, but require the constant, vigilant attention of seasoned evaluation leaders. (p. 306)

In contractual engagements, the evaluator has to make a decision to move forward with a contract or, as Smith (1998) describes it, to determine if an evaluation contract may be “bad for business” (p. 178). Smith goes on to recommend declining a contract if the desired work is not possible at an “acceptable level of quality” (Smith, 1998, p. 178). For internal evaluators, turning down an evaluation contract may have career implications. The case study at the end of this chapter explores this dilemma. Smith (1998) cites Mabry (1997) in describing the challenge of adhering to ethical principles for the evaluator:

Evaluation is the most ethically challenging of the approaches to research inquiry because it is the most likely to involve hidden agendas, vendettas, and serious professional and personal consequences to individuals. Because of this feature, evaluators need to exercise extraordinary circumspection before engaging in an evaluation study. (Mabry, 1997, p. 1, cited in Smith, 1998, p. 180)

Cultural Competence in Evaluation Practice

While issues of cultural sensitivity are addressed in Chapter 5, cultural sensitivity is as important for quantitative evaluation as it is for qualitative evaluation. We are including cultural competence in this section on ethics, as cultural awareness is an important feature of not only development evaluation where we explicitly work across cultures, but also virtually any evaluations conducted in our increasingly multicultural society. Evaluations in the health, education, or social sectors, for example, would commonly require that the evaluator have cultural awareness and sensitivity.

There is evidence of a growing sense of the importance and the relevance of acknowledging cultural awareness for evaluations. Schwandt (2007) notes that “the Guiding Principles (as well as most of the ethical guidelines of academic and professional associations in North America) have been developed largely against the foreground of a Western framework of moral understandings” (p. 400) and are often framed in terms of individual behaviors, largely ignoring the normative influences of social practices and institutions. The AEA Guiding Principles for Evaluators include the following caveat to address the cross- cultural limitations of their principles:

These principles were developed in the context of Western cultures, particularly the United States, and so may reflect the experiences of that context. The relevance of these

principles may vary across other cultures, and across subcultures within the United States. (AEA, 2004)

Schwandt (2007) notes that “in the Guiding Principles for evaluators, cultural competence is one dimension of a general principle (‘competence’) concerned with the idea of fitness or aptitude for the practice of evaluation” (p. 401); however, he challenges the adequacy of this dimension, asking “Can we reasonably argue for something like a cross cultural professional ethic for evaluators, and if so, what norms would it reflect?” (p. 401). Schwandt (2007) notes that the focus on cultural competence in evaluation has developed out of concern for “attending to the needs and interests of an increasingly diverse, multicultural society and the challenges of ensuring social equity in access to and quality of human service programs” (p. 401). In an imagined dialogue between two evaluators, Schwandt and Dahler-Larsen (2006) discuss resistance to evaluation and the practical implications for performing evaluation in communities. They conclude that “perhaps evaluators should listen more carefully and respond more prudently to voices in communities that are hesitant or skeptical about evaluation […] Evaluation is not only about goals and criteria, but about forms of life” (p. 504).

THE PROSPECTS FOR AN EVALUATION PROFESSION

In this chapter, we have emphasized the importance of acknowledging and cultivating sound professional judgment as part of what we believe is required to move evaluation in the direction of becoming a profession. In some professions, medicine being a good example, there is growing recognition that important parts of sound practice are tacitly learned, and that competent practitioners need to cultivate the capacity to reflect on their experience to develop an understanding of their own subjectivity and how their values, beliefs, expectations, and feelings affect the ways that they make decisions in their practice.

Some evaluation associations, the Canadian Evaluation Society (CES) being the most prominent example, have embarked on a professionalization path that has included identifying core competencies for evaluators and offering members the option of applying for a professional designation. Knowledge (formal education), experience, and professional reputation are all included in the assessment process conducted by an independent panel, and successful applicants receive a Credentialed Evaluator designation (CES, 2012b).

Other evaluation associations, with their emphasis on guidelines and standards for evaluation practice, are also embarking on a process that moves the field toward becoming more professional. Efforts are being made to identify core competencies (King et al., 2001), and discussions have outlined some of the core epistemological and methodological issues that would need to be addressed if evaluation is to move forward as a profession (Bickman, 1997; Patton, 2008). The evaluation field continues to evolve as academic and practice-based contributors offer new ideas, critique each other’s ideas, and develop new approaches. Becoming more like a profession will mean balancing the norms of professional practice (core body of knowledge, ethical standards, and perhaps even entry to practice requirements) with the ferment that continues to drive the whole field and makes it both challenging and exciting.

Although many evaluators have made contributions that suggest we are moving toward making evaluation into a profession, we are not there yet. Picciotto (2011) concludes the following:

Evaluation is not a profession today but could be in the process of becoming one. Much remains to be done to trigger the latent energies of evaluators, promote their expertise,

protect the integrity of their practice and forge effective alliances with well wishers in government, the private sector and the civil society. It will take strong and shrewd leadership within the evaluation associations to strike the right balance between autonomy and responsiveness, quality and inclusion, influence and accountability. (p. 179)

SUMMARY

Program evaluation is partly about learning methods and how to apply them. But, because most evaluation settings offer only roughly appropriate opportunities to apply tools that are often designed for social science research settings, it is essential that evaluators learn the craft of working with square pegs for round holes. Evaluators and managers have in common the fact that they are often trained in settings that idealize the applications of the tools that they learn. When they enter the world of practice, they must adapt what they have learned. What works is determined by the context and their experiences. Experience becomes the foundation not only of when and how to apply tools but, more important, the essential basis for interpreting the information that is gathered in a given situation.

Evaluators have the comparative luxury of time and resources to examine a program or policy that managers usually have to judge in situ, as it were. Even for evaluators, there are rarely sufficient resources to apply the tools that would yield the highest quality of data. That is a limitation that circumscribes what we do, but does not mean that we should stop asking whether and how programs work.

This chapter emphasizes the central role played by professional judgment in the practice of professions, including evaluation, and the importance of cultivating sound professional judgment. Michael Patton, through his alter ego Halcolm, puts it this way (Patton, 2008, p. 501):

Forget “judge not and ye shall not be judged.” The evaluator’s mantra: Judge often and well so that you get better at it.

—Halcolm

It follows that professional programs, courses in universities, and textbooks should underscore for students the importance of developing and continuously improving their professional judgment skills, as opposed to focusing only on learning methods, facts, and exemplars. Practicing the craft of evaluation necessitates developing knowledge and skills that are tacit. These are learned through experience, refined through reflective practice, and applied along with the technical and rational knowledge that typically is conveyed in books and in classrooms. Practitioners in a profession

begin to recognize that practice is much more messy than they were led to believe [in school], and worse, they see this as their own fault—they cannot have studied sufficiently well during their initial training.… This is not true. The fault, if there is one, lies in the lack of support they receive in understanding and coping with the inevitably messy world of practice. (Fish & Coles, 1998, p. 13)

Fish and Coles continue thus:

Learning to practice in a profession is an open capacity, cannot be mastered and goes on being refined forever. Arguably there is a major onus on those who teach courses of preparation for professional practice to demonstrate this and to reveal in their practice its implications. (p. 43)

The ubiquity of different kinds of judgment in evaluation practice suggests that as a nascent profession we need to do at least three things. First, we need to fully acknowledge the importance of professional judgment and the role it plays in the diverse ways we practice evaluation. Second, we need to understand how our professional judgments are made—the factors that condition our own judgments. Reflective practice is critical to reaping the potential from experience. Third, we need to work toward self-consciously improving the ways we incorporate, into the education and training of evaluators, opportunities for current and future practitioners to improve their professional judgments. Embracing professional judgment is an important step toward more mature and self-reflective evaluation practice.

Ethical evaluation practice is a part of cultivating sound judgment. Although national and international evaluation associations have developed principles and guidelines that include ethical practice, these guidelines are general and are not enforceable. Individual evaluators need to learn, through their reflective practice, how to navigate the ethical tradeoffs in situations, understanding that appropriate ethical practice will weigh the risks and benefits for those involved.

DISCUSSION QUESTIONS

1. Take a position for or against the following proposition and develop a strong one-page argument that supports your position. This is the proposition: “Be it resolved that experiments, where program and control groups are randomly assigned, are the Gold Standard in evaluating the effectiveness of programs.”

2. What do evaluators and program managers have in common? What differences can you think of as well?

3. What is tacit knowledge? How does it differ from public knowledge? 4. In this chapter, we said that learning to ride a bicycle is partly tacit. For those who want

to challenge this statement, try to describe learning how to ride a bicycle so that a person who has never before ridden a bicycle could get on one and ride it right away.

5. What is mindfulness, and how can it be used to develop sound professional judgment? 6. Why is teamwork an asset for persons who want to develop sound professional

judgment? 7. What do you think would be required to make evaluation more professional, that is, have

the characteristics of a profession?

APPENDIX

Appendix A: Fiona’s Choice: An Ethical Dilemma for a Program Evaluator

Fiona Barnes did not feel well as the deputy commissioner’s office door closed behind her. She walked back to her office wondering why bad news seems to come on Friday afternoons.

Sitting at her desk, she went over the events of the past several days and the decision that lay ahead of her. This was clearly the most difficult situation that she had encountered since her promotion to the position of director of evaluation in the Department of Human Services.

Fiona’s predicament had begun the day before, when the new commissioner, Fran Atkin, had called a meeting with Fiona and the deputy commissioner. The governor was in a difficult position: In his recent election campaign, he had made potentially conflicting campaign promises. He had promised to reduce taxes and had also promised to maintain existing health and social programs, while balancing the state budget.

The week before, a loud and lengthy meeting of the commissioners in the state government had resulted in a course of action intended to resolve the issue of conflicting election promises. Fran Atkin had been persuaded by the governor that she should meet with the senior staff in her department, and after the meeting, a major evaluation of the department’s programs would be announced. The evaluation would provide the governor with some post-election breathing space. But the evaluation results were predetermined—they would be used to justify program cuts. In sum, a “compassionate” but substantial reduction in the department’s social programs would be made to ensure the department’s contribution to a balanced budget.

As the new commissioner, Fran Atkin relied on her deputy commissioner, Elinor Ames. Elinor had been one of several deputies to continue on under the new administration and had been heavily committed to developing and implementing key programs in the department, under the previous administration. Her success in doing that had been a principal reason why she had been promoted to deputy commissioner.

On Wednesday, the day before the meeting with Fiona, Fran Atkin had met with Elinor Ames to explain the decision reached by the governor, downplaying the contentiousness of the discussion. Fran had acknowledged some discomfort with her position, but she believed her department now had a mandate. Proceeding with it was in the public’s interest.

Elinor was upset with the governor’s decision. She had fought hard over the years to build the programs in question. Now she was being told to dismantle her legacy—programs she believed in that made up a considerable part of her budget and person-year allocations.

In her meeting with Fiona on Friday afternoon, Elinor had filled Fiona in on the political rationale for the decision to cut human service programs. She also made clear what Fiona had suspected when they had met with the commissioner earlier that week—the outcomes of the evaluation were predetermined: They would show that key programs where substantial resources were tied up were not effective and would be used to justify cuts to the department’s programs.

Fiona was upset with the commissioner’s intended use of her branch. Elinor, watching Fiona’s reactions closely, had expressed some regret over the situation. After some hesitation, she suggested that she and Fiona could work on the evaluation together, “to ensure that it meets our needs and is done according to our standards.” After pausing once more, Elinor added, “Of course, Fiona, if you do not feel that the branch has the capabilities needed to undertake this project, we can contract it out. I know some good people in this area.”

Fiona was shown to the door and asked to think about it over the weekend. Fiona Barnes took pride in her growing reputation as a competent and serious director of a

good evaluation shop. Her people did good work that was viewed as being honest, and they prided themselves on being able to handle any work that came their way. Elinor Ames had appointed Fiona to the job, and now this.

Your Task

Analyze this case and offer a resolution to Fiona’s dilemma. Should Fiona undertake the evaluation project? Should she agree to have the work contracted out? Why?

In responding to this case, consider the issues on two levels: (1) look at the issues taking into account Fiona’s personal situation and the “benefits and costs” of the options available to her and (2) look at the issues from an organizational standpoint, again weighing the “benefits and the costs.” Ultimately, you will have to decide how to weigh the benefits and costs from both Fiona’s and the department’s standpoints.

REFERENCES

Abercrombie, M. L. J. (1960). The anatomy of judgment: An investigation into the processes of perception and reasoning. New York: Basic Books.

Altschuld, J. (1999). The certification of evaluators: Highlights from a report submitted to the Board of Directors of the American Evaluation Association. American Journal of Evaluation, 20(3), 481–493.

American Evaluation Association. (1995). Guiding principles for evaluators. New Directions for Program Evaluation, 66, 19–26.

American Evaluation Association. (2004). Guiding principles for evaluators. Retrieved from http://www.eval.org/Publications/GuidingPrinciples.asp

Ayton, P. (1998). How bad is human judgment? In G. Wright & P. Goodwin (Eds.), Forecasting with judgement (pp. 237–267). Chichester, West Sussex, UK: John Wiley.

Bamberger, M., Rugh, J., & Mabry, L. (2012). Real world evaluation: Working under budget, time, data, and political constraints (2nd ed.). Thousand Oaks, CA: Sage.

Basilevsky, A., & Hum, D. (1984). Experimental social programs and analytic methods: An evaluation of the U.S. income maintenance projects. Orlando, FL: Academic Press.

Berk, R. A., & Rossi, P. H. (1999). Thinking about program evaluation (2nd ed.). Thousand Oaks, CA: Sage.

Bickman, L. (1997). Evaluating evaluation: Where do we go from here? Evaluation Practice, 18(1), 1–16.

Brandon, P., Smith, N., & Hwalek, M. (2011). Aspects of successful evaluation practice at an established private evaluation firm. American Journal of Evaluation, 32(2), 295–307.

Campbell Collaboration. (2010). About us. Retrieved from http://www.campbellcollaboration.org/about_us/index.php

Campbell, D. T. (1991). Methods for the experimenting society. Evaluation Practice, 12(3), 223–260.

Canadian Evaluation Society. (2012a). CES guidelines for ethical conduct. Retrieved from http://www.evaluationcanada.ca/site.cgi?s= 5&ss=4&_lang=en

Canadian Evaluation Society. (2012b). Program evaluation standards. Retrieved from http://www.evaluationcanada.ca/site.cgi?s= 6&ss=10&_lang=EN

Canadian Institutes of Health Research, Natural Sciences and Engineering Research Council of Canada, & Social Sciences and Humanities Research Council of Canada. (2010). Tri-

council policy statement: Ethical conduct for research involving humans, December 2010. Retrieved from http://www.pre.ethics.gc.ca/pdf/eng/tcps2/ TCPS_2_FINAL_Web.pdf

Chen, H. T., Donaldson, S. I., & Mark, M. M. (2011). Validity frameworks for outcome evaluation. New Directions for Evaluation, 2011(130), 5–16.

Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation: Design & analysis issues for field settings. Chicago, IL: Rand McNally.

Cook, T. D., Scriven, M., Coryn, C. L., & Evergreen, S. D. (2010). Contemporary thinking about causation in evaluation: A dialogue with Tom Cook and Michael Scriven. American Journal of Evaluation, 31(1), 105–117.

Cooksy, L. J. (2008). Challenges and opportunities in experiential learning. American Journal of Evaluation, 29(3), 340–342.

Cronbach, L. J. (1980). Toward reform of program evaluation (1st ed.). San Francisco, CA: Jossey-Bass.

Cronbach, L. J. (1982). Designing evaluations of educational and social programs (1st ed.). San Francisco, CA: Jossey-Bass.

Epstein, R. M. (1999). Mindful practice. Journal of the American Medical Association, 282(9), 833–839.

Epstein, R. M. (2003). Mindful practice in action (I): Technical competence, evidence-based medicine, and relationship-centered care. Families, Systems & Health, 21(1), 1–9.

Epstein, R. M., Siegel, D. J., & Silberman, J. (2008). Self-monitoring in clinical practice: A challenge for medical educators. Journal of Continuing Education in the Health Professions, 28(1), 5–13.

Eraut, M. (1994). Developing professional knowledge and competence. Washington, DC: Falmer Press.

Fish, D., & Coles, C. (1998). Developing professional judgement in health care: Learning through the critical appreciation of practice. Boston, MA: Butterworth-Heinemann.

Ford, R., Gyarmati, D., Foley, K., Tattrie, D., & Jimenez, L. (2003). Can work incentives pay for themselves? Final report on the Self-Sufficiency Project for welfare applicants. Ottawa, Ontario, Canada: Social Research and Demonstration Corporation.

Garvin, D. A. (1993). Building a learning organization. Harvard Business Review, 71(4), 78 –90.

Ghere, G., King, J. A., Stevahn, L., & Minnema, J. (2006). A professional development unit for reflecting on program evaluator competencies. American Journal of Evaluation, 27(1), 108–123.

Gibbins, M., & Mason, A. K. (1988). Professional judgment in financial reporting. Toronto, Ontario, Canada: Canadian Institute of Chartered Accountants.

Gustafson, P. (2003). How random must random assignment be in random assignment experiments? Ottawa, Ontario, Canada: Social Research and Demonstration Corporation.

Henry, G. T., & Mark, M. M. (2003). Toward an agenda for research on evaluation. New Directions for Evaluation, 97, 69–80.

Higgins, J., & Green, S. (Eds.). (2011). Cochrane handbook for systematic reviews of interventions: Version 5.0.2 (updated March 2011). The Cochrane Collaboration 2011. Retrieved from www.cochrane-handbook.org

House, E. R., & Howe, K. R. (1999). Values in evaluation and social research. Thousand Oaks, CA: Sage.

Human Resources Development Canada. (1998). Quasi-experimental evaluation (Publication No. SP-AH053E-01–98). Ottawa, Ontario, Canada: Evaluation and Data Development Branch.

Hurteau, M., Houle, S., & Mongiat, S. (2009). How legitimate and justified are judgments in program evaluation? Evaluation, 15(3), 307–319.

Jewiss, J., & Clark-Keefe, K. (2007). On a personal note: Practical pedagogical activities to foster the development of “reflective practitioners.” American Journal of Evaluation, 28 (3), 334–347.

Katz, J. (1988). Why doctors don’t disclose uncertainty. In J. Dowie & A. S. Elstein (Eds.), Professional judgment: A reader in clinical decision making (pp. 544–565). Cambridge, MA: Cambridge University Press.

Kelling, G. L. (1974a). The Kansas City preventive patrol experiment: A summary report. Washington, DC: Police Foundation.

Kelling, G. L. (1974b). The Kansas City preventive patrol experiment: A technical report. Washington, DC: Police Foundation.

King, J. A., Stevahn, L., Ghere, G., & Minnema, J. (2001). Toward a taxonomy of essential evaluator competencies. American Journal of Evaluation, 22(2), 229–247.

Kitchener, K. S. (1984). Intuition, critical evaluation and ethical principles: The foundation for ethical decisions in counseling psychology. The Counseling Psychologist, 12(3), 43–55.

Krasner, M. S., Epstein, R. M., Beckman, H., Suchman, A. L., Chapman, B., Mooney, C. J., & Quill, T. E. (2009). Association of an educational program in mindful communication with burnout, empathy, and attitudes among primary care physicians. Journal of the American Medical Association, 302(12), 1284–1293.

Kuhn, T. S. (1962). The structure of scientific revolutions. Chicago, IL: University of Chicago Press.

Kundin, D. M. (2010). A conceptual framework for how evaluators make everyday practice decisions. American Journal of Evaluation, 31(3), 347–362.

Larson, R. C. (1982). Critiquing critiques: Another word on the Kansas City preventive patrol experiment. Evaluation Review, 6(2), 285–293.

Levin, H. M., & McEwan, P. J. (Eds.). (2001). Cost-effectiveness analysis: Methods and applications (2nd ed.). Thousand Oaks, CA: Sage.

Mabry, L. (1997). Ethical landmines in program evaluation. In R. E. Stakes (Chair), Grounds for turning down a handsome evaluation contract. Symposium conducted at the meeting of the AERA, Chicago, IL.

Mark, M. M., Henry, G. T., & Julnes, G. (2000). Evaluation: An integrated framework for understanding, guiding, and improving policies and programs (1st ed.). San Francisco, CA: Jossey-Bass.

Mason, J. (2002). Qualitative researching (2nd ed.). Thousand Oaks, CA: Sage. Mayne, J. (2008). Building an evaluative culture for effective evaluation and results

management. Retrieved from http://www.cgiar-ilac.org/files/publications/briefs/ ILAC_Brief20_Evaluative_Culture.pdf

Modarresi, S., Newman, D. L., & Abolafia, M. Y. (2001). Academic evaluators versus practitioners: Alternative experiences of professionalism. Evaluation and Program Planning, 24(1), 1–11.

Morris, M. (1998). Ethical challenges. American Journal of Evaluation, 19(3), 381–382. Morris, M. (Ed.). (2008). Evaluation ethics for best practice: Cases and commentaries. New

York: Guilford Press. Morris, M. (2011). The good, the bad, and the evaluator: 25 years of AJE ethics. American

Journal of Evaluation, 32(1), 134–151. Mowen, J. C. (1993). Judgment calls: High-stakes decisions in a risky world. New York:

Simon & Schuster. Newman, D. L., & Brown, R. D. (1996). Applied ethics for program evaluation. Thousand

Oaks, CA: Sage. No Child Left Behind Act of 2001, Pub. L. No. 107-110, 115 Stat. 1425. Office of Management and Budget. (2004). What constitutes strong evidence of a program’s

effectiveness? Retrieved from http://www.whitehouse.gov/omb/part/2004_program_eval.pdf

Patton, M. Q. (1997). Utilization-focused evaluation: The new century text (3rd ed.). Thousand Oaks, CA: Sage.

Patton, M. Q. (2008). Utilization-focused evaluation (4th ed.) Thousand Oaks, CA: Sage. Pawson, R., & Tilley, N. (1997). Realistic evaluation. Thousand Oaks, CA: Sage. Picciotto, R. (2011). The logic of evaluation professionalism. Evaluation, 17(2), 165–180. Polanyi, M. (1958). Personal knowledge: Towards a post-critical philosophy. Chicago, IL:

University of Chicago Press. Polanyi, M., & Grene, M. G. (1969). Knowing and being: Essays. Chicago, IL: University of

Chicago Press. Rossi, P. H., Lipsey, M. W., & Freeman, H. E. (2004). Evaluation: A systematic approach.

Thousand Oaks, CA: Sage. Sanders, J. R. (1994). Publisher description for the program evaluation standards: How to

assess evaluations of educational programs. Retrieved from http://catdir.loc.gov/catdir/enhancements/ fy0655/94001178-d.html

Schön, D. A. (1987). Educating the reflective practitioner: Toward a new design for teaching and learning in the professions (1st ed.). San Francisco, CA: Jossey-Bass.

Schön, D. A. (1988). From technical rationality to reflection-in-action. In J. Dowie & A. S. Elstein (Eds.), Professional judgment: A reader in clinical decision making (pp. 60–77). New York: Cambridge University Press.

Schwandt, T. A. (2000). Three epistemological stances for qualitative enquiry. In N. K. Denzin & Y. S. Lincoln (Eds.), Handbook of qualitative research (2nd ed., pp. 189–213). Thousand Oaks, CA: Sage.

Schwandt, T. A. (2007). Expanding the conversation on evaluation ethics. Evaluation and Program Planning, 30(4), 400–403.

Schwandt, T. A. (2008). The relevance of practical knowledge traditions to evaluation practice. In N. L. Smith & P. R. Brandon (Eds.), Fundamental issues in evaluation (pp. 29–40). New York: Guilford Press.

Schwandt, T. A., & Dahler-Larsen, P. (2006). When evaluation meets the “rough ground” in communities. Evaluation, 12(4), 496–505.

Schweigert, F. J. (2007). The priority of justice: A framework approach to ethics in program evaluation. Evaluation and Program Planning, 30(4), 394–399.

Scriven, M. (1994). The final synthesis. Evaluation Practice, 15(3), 367–382. Scriven, M. (2004). Causation. Unpublished manuscript, University of Auckland, Auckland,

New Zealand. Scriven, M. (2008). A summative evaluation of RCT methodology & an alternative approach

to causal research. Journal of Multidisciplinary Evaluation, 5(9), 11–24. Seiber, J. (2009). Planning ethically responsible research. In L. Bickman & D. Rog (Eds.), The

Sage handbook of applied social research methods (2nd ed., pp. 106–142). Thousand Oaks, CA: Sage.

Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin.

Simons, H. (2006). Ethics in evaluation. In I. Shaw, J. Greene, & M. M. Mark (Eds.), The Sage handbook of evaluation (pp. 243–265). Thousand Oaks, CA: Sage.

Skolits, G. J., Morrow, J. A., & Burr, E. M. (2009). Reconceptualizing evaluator roles. American Journal of Evaluation, 30(3), 275–295.

Smith, M. L. (1994). Qualitative plus/versus quantitative: The last word. New Directions for Program Evaluation, 61, 37–44.

Smith, N. L. (1998). Professional reasons for declining an evaluation contract. American Journal of Evaluation, 19(2), 177–190.

Smith, N. L. (2007). Empowerment evaluation as evaluation ideology. American Journal of Evaluation, 28(2), 169–178.