summary for a reading

http://www.jmde.com/ Articles

Journal of MultiDisciplinary Evaluation, Volume 8, Number 17 ISSN 1556-8180 January 2012

102

Evaluation Roots: An International Perspective Fred Carden International Development Research Centre Marvin C. Alkin University of California, Los Angeles Background: The second edition of Evaluation Roots has expanded to more global coverage, but no chapter emerged on development theory in low and middle income countries (LMICs). Purpose: The purpose of this article is initiate a conversation on filling this gap. Setting: Not applicable. Intervention: Not applicable. Research Design: Not applicable. Data Collection and Analysis: Desk review. Findings: Two important distinctions emerged in evaluation roots in LMICs. The first is that much evaluation fits effectively on the Theory Tree as it is presented, with perhaps some nuance, but we see a collective rather than individual origin of the

evaluation theory writings we have uncovered. The second is the nature of the formal evaluation work in LMICs; its practical origins are not yet formalized into full prescriptive theory. The paper notes the prominence of external actors (such as the donor community) in formalizing evaluation practice, while at the same time observes important threads are developing in different regions of the world. This paper proposes a way to thinking about evaluation in LMICs that is based on practice. The paper suggests a need for evaluation analysts, especially those in LMICs, to build a more comprehensive knowledge and documentation on development evaluation and, more broadly on building the field of evaluation in LMICs. Keywords: development evaluation; Evaluation Roots; Evaluation Theory Tree; evaluation field building __________________________________

his journal article considers evaluators or evaluation groups in the

developing world and examines how they might be placed on a conceptual framework depicting evaluation theorists. The starting point for this reflection is the Theory Tree described in Evaluation Roots (Alkin, 2004, 2012). In a new edition of the Roots publication, North American, European and Australasian theorists have been placed on an Evaluation Theory Tree consisting of

three main branches: methods, valuing and use with one branch representing his or her main emphasis among the three. This chapter considers the addition of evaluation perspectives from other parts of the world. The state of evaluation in low and middle income countries (LMICs) suggests two things. First, much evaluation fits effectively on the Theory Tree as it is presented, with perhaps some nuance, but we see a collective rather than individual origin of the limited evaluation

T

Fred Carden and Marvin C. Alkin

Journal of MultiDisciplinary Evaluation, Volume 8, Number 17 ISSN 1556-8180 January 2012

103

theory writings we have uncovered in LMICs. Second, the nature of the formal evaluation work in LMICs is such that the practical origins are not yet formalized into full prescriptive theory. As Lewin argues (best described in Marrow 1969), good theory is built from practice. While the actual building of theory is nascent in the field of evaluation in LMICs, this nascent state of evaluation suggests a first exercise in defining what constitutes evaluation for development.

Evaluation Theory/Theorists What do we mean by evaluation “theory” and by those who propound it as “theorists”? In general we have come to use the term theory as somewhat synonymous with approaches or models. Thus we are not talking about descriptive or empirical theory – that which would describe what would necessarily occur given particular sets of actions. There is a paucity of research about evaluation. We simply do not know (in a scientific way) the results of particular evaluation actions taking place within particular contextual settings. Research does not provide those answers. Perhaps the complexity of program contexts and evaluator actions are so vast that we will never be able to create a descriptive evaluation theory.

As a consequence of this research deficiency, the need for guidance on how to conduct an evaluation developed in the field. Multiple prominent evaluation writers leaped to the fore to describe their prescription of how they believe an evaluation should be conducted. Those prescriptions, which are considered fully developed, we refer to as “prescriptive theories.” Theorists whose names are attached to such theories include Michael Scriven, Robert Stake, Ernest House,

Michael Patton, Jennifer Greene, Yvonna Lincoln, David Fetterman, and Barry MacDonald. In each case, the identification of an individual as a “theorist”—and their writings as theories—implies that the work has been sufficiently developed and recognized as constituting a distinct approach.

Evaluation Theory Tree The Theory Tree classification presented by Alkin and Christie in the Evaluation Roots book (2004, 2012) provides a guiding framework for our work in this paper. They maintained that all prescriptive theories must consider: (a) issues related to the methodology being used, (b) the manner in which the data are to be judged or valued, and (c) the user focus of the evaluation effort. Relationships between theories are represented in the form of an Evaluation Theory Tree with each of the main branches representing one of the dimensions: methods, judgment/valuing, and use. Each theorist has been presented on the tree on a branch that represents his or her main emphasis among these three branches. The evaluation tree system of categorizing is based upon relative emphasis within the various models. In essence: when evaluators must make concessions, what do they most easily give up and what do they most tenaciously defend?

The central branch of the Evaluation Theory Tree (see Figure 1), the Methods branch, is one in which evaluation is primarily guided by research methodology. While most evaluation theorists have methodological concerns and view applied research as the genesis of program evaluation, some theorists have been steadfast in emphasizing that

Fred Carden and Marvin C. Alkin

Journal of MultiDisciplinary Evaluation, Volume 8, Number 17 ISSN 1556-8180 January 2012

104

orientation. Theorists on this branch are typically concerned with knowledge construction (Shadish et al., 1991) in the most rigorous manner possible given the particular constraints within that evaluation. Fundamental to these theories is the work of Donald Campbell (1957) and subsequently Campbell and Stanley (1966), which defines the conditions for appropriate experimental and quasi- experimental designs. The writings of most theorists on this branch are either grounded in or expand upon this work.

To the right of the Methods branch is the Valuing branch, which was initially inspired by the work of Michael Scriven. It was Scriven who proclaimed that evaluation is not evaluation without valuing (1967). He argues that the work of evaluation is making a value judgment about the object that is under study. Theorists on this branch also draw influence from Robert Stake’s writings (1967, 1975). This branch is split in two— objectivist and subjectivist—which distinguishes the two fundamental perspectives informing the evaluation process. The objectivist influenced sub- branch to the left is most heavily influenced by Scriven’s views. He considers that it is the role of the evaluator to do that valuing. In essence, the important valuing role belongs to the evaluator. In the subjectivist sub-branch to the right, proponents argue that reality is an ongoing, dynamic process and a truth is always relative to some particular frame of reference. Thus, valuing must take place within the context of

understanding the “subjective meaningfulness” of the evaluation information. In contrast to the objectivists, these theorists do not themselves solely make the valuing judgment.

The third branch of the tree is Use, which originally focused on an orientation toward evaluation and decision making. In essence, work by early theorists on this branch focused specifically on those empowered to use the information—most typically individuals who had contracted for the evaluation. Subsequent theorists have expanded the use concern to broader user audiences and to evaluation capacity building within the organization being evaluated.

The three tree branches are not meant to be viewed as independent from one another, but rather have been drawn in a manner that reflects their relationship to each other. Thus, if the tree were in a three dimensional space, the far right of the valuing branch would relate to (or perhaps touch) the far left side of the use branch. This is important to note because theorists are positioned on each branch in a way that reflects not only their primary emphasis, but also the other major tendency of their particular approach. Thus, the relational nature of the three branches helps to further depict theorists’ approaches.

We present the excerpted figure here for North America, Europe, Australia, and New Zealand (see Figure 1). This may be helpful as a frame of reference for the discussion on evaluation in LMICs.

Fred Carden and Marvin C. Alkin

Journal of MultiDisciplinary Evaluation, Volume 8, Number 17 ISSN 1556-8180 January 2012

105

Figure 1. Evaluation Theory Tree

Source: Alkin, M. C. (2012). Evaluation roots (2nd ed.). Thousand Oaks, CA: Sage.

Evaluation Writers in the Developing World: An Overview We address here the issue of the evaluation perspectives LMICs, sometimes called the developing world, or the ‘global South.’ First, in LMICs, the evaluation roots were initially those which were laid on by aid organizations from the North. In this sense, the roots are institutional rather than independently and contextually secured by a perspective rooted in Methods, Valuing, or Use. The approaches imposed in this process were

developed in North America and Europe and in this way build on the development of the field of evaluation in those regions rather than on any evaluation traditions in LMICs.

As evaluation became somewhat widely used in LMICs and more localized, evaluation prescriptions began to reflect local contexts. Again, these evaluation prescriptions (or descriptive “theories”) were primarily collective rather than individual. That is, they were primarily developed by groups of individuals rather than single writers. This is both cultural and political in origin. While for the most part it is therefore not possible to

Fred Carden and Marvin C. Alkin

Journal of MultiDisciplinary Evaluation, Volume 8, Number 17 ISSN 1556-8180 January 2012

106

specifically mention individual theorists as is done in the Evaluation Roots book (Alkin 2012), it is especially important to find a way to highlight the ongoing efforts at evaluation field building in the South. Thus, we will start with a framing of evaluation based on its origins in use in LMICs, and then come to a brief discussion of the links this has to the Theory Tree.

This framing of approaches is an important contribution not only for the future potential identification of theorists for the evaluation tree, but as a source of pressure for change in how the developing world deals with evaluation field building. As one of the author’s has argued elsewhere, building the field of evaluation is critical to effective use of evaluation in LMICs (Carden, 2007). It is in building the field that theorists (whether collective or individual) will begin to take a stronger place and have a stronger presence in the global evaluation community.

We are acutely aware in writing this piece that we do not represent evaluation constituencies in LMICs and hope that the article will stimulate thinking and writing by evaluators in LMICs who want to contribute to building evaluation practice (and potentially theory) in their countries and regions. We also note that as much of the development evaluation work that comes out of LMICs is locally-based, it is not well known outside its regions of application. As a result this paper is necessarily incomplete and should be thought of as a framework for thinking about development evaluation rather than as a full compendium of methodologists, or methodological approaches, from the global South. What it suggests is a need for evaluation analysts to build a more comprehensive knowledge and complete documentation on development evaluation and, more broadly on building

the field of evaluation in LMICs. It is clear that many social scientists in LMICs who work more broadly on methodology are active practitioners of evaluation and bring methodological traditions to the evaluation field, but they do not document these in evaluation use (e.g., Chilisa, 2012). There is much that could be developed to strengthen evaluation practice in LMICs through the definition and articulation of work that is done informally. This is an essential component of field building in evaluation in LMICs.

While as an informal activity, evaluation exists everywhere in many forms, development evaluation originated largely in the bilateral aid agencies and their multilateral counterparts. Evaluation was developed and expanded by these agencies as a tool to support delivery of their projects in developing countries. As aid began to develop as an industry following independence in many third world states (largely in the 1960s), it quickly became evident that some sort of support, based on the Bretton Woods model of post-war reconstruction, was needed and would be supported by countries in North America and Europe. It soon became evident that some tools would be needed to assess that aid flow. This led to the first type of development evaluation, which we refer to as adopted methodologies. Adopted Methodologies As the name implies, adopted methodologies are those that are transferred from systems developed in North America and Europe for use in LMICs, in particular in assessing aid flows. Early efforts at evaluation in the South were seen largely as technology transfer: through training activity in particular evaluation approaches at use in

Fred Carden and Marvin C. Alkin

Journal of MultiDisciplinary Evaluation, Volume 8, Number 17 ISSN 1556-8180 January 2012

107

bilateral and multilateral agencies. The larger intent was to ensure a body of evaluators who could fulfill the evaluation needs of the agencies all of which recognized the value in local consultants who understood both the local cultural and political norms and the language. (We are over simplifying here, but not by much!). This issue has been discussed elsewhere by one of the authors (Carden, 2007; Carden, 2010).

Logical Framework Analysis is one of the key adopted methodologies that has been put in place by many bilateral and multilateral agencies. It has evolved over time and is often articulated as results- based management. Much evaluation training conducted by donor agencies is built around approaches to logical framework analysis, or results based management, as used in multilateral and bilateral development assistance agencies. The ‘road to results’ (Morra-Imas & Rist, 2009) is one of the most thorough articulations of this approach and serves as a textbook in development evaluation workshops, such as the International Program for Development Evaluation Training (sponsored by the World Bank and Carleton University). The approach is also roundly criticized (Gasper, 2000; U.N. Office of Internal Oversight Services, 2008). While individuals such as Morra- Imas and Rist further develop and codify the approaches, the origin and motivation for use is fundamentally institutional. It would fall on the Use branch of the Evaluation Theory Tree with its strong orientation to use in decision making by those empowered to determine the use of development funds.

Other methods are also prescribed by agencies. More recently experimental and quasi-experimental methods have gained in popularity, notably in the call for more randomized controlled trials (RCTs). In its

2006 study, the Center for Global Development made a strong appeal for more RCTs in development evaluation as a way to strengthen understanding of what works in development (Evaluation Gap Working Group, 2006). RCTs had of course been around for a long time, but this study unleashed a flurry of activity and debate in the development evaluation community and stimulated considerable agency demand for more experimental evaluations in development. Much of the debate has centered on the relevance of RCTs for some development activities. While the proponents of RCTs clearly acknowledge their strengths as well as their limitations, agencies that set this approach as an institutional requirement are ignoring this fundamental methodological debate.

The 2006 study by the Center for Global Development led to the creation of the International Initiative for Impact Evaluation (3ie). The 3ie mandate is to support quality impact evaluation studies that can influence policy and affect many lives. 3ie strives to fund studies that produce new evidence around what works in development and disseminating that evidence. 3ie supports and fosters an evidence-based approach to decision making and increasingly works to build capacity to both produce and use impact evaluations (3ie website). Its members include 15 development agencies (bilateral, multilateral and foundations) and 4 government agencies. Associate membership is broadly based in a range of agencies based in both the global North and the global South. 3ie carries out a lot of its work through calls for proposals which are carefully screened. 3ie also sponsors the Journal for Development Effectiveness. The journal publishes results of impact studies and also encourages articles on impact evaluation

Fred Carden and Marvin C. Alkin

Journal of MultiDisciplinary Evaluation, Volume 8, Number 17 ISSN 1556-8180 January 2012

108

methods. 3ie actively supports studies with a clear counterfactual and notes that it is not focused on a single method. At present, 3ie is focusing some of its efforts on identifying “small n” methods that could be used in impact evaluation. 3ie has quickly become an important influence in development evaluation. Its work would fall on the Methods branch of Evaluation Theory Tree though tending towards use.

What adopted methodologies have in common is that they are a form of technology transfer which, for better or worse, does not see a need for adaptation to local context. A second commonality is that they tend to be presented as the singular approach for an agency’s evaluation efforts and every effort is made to use the prescribed methodology for all activities. Thirdly, they represent an organizational perspective rather than an individual methodologist’s perspective. Finally, they have their origins in the accountability needs of the agencies rather than on the learning and accountability needs of those being supported in the development assistance process. In sum, they are institutional in origin. Adapted Methodologies Partly in response to this largely colonial role of evaluation, and dissatisfaction with it, increasing attention has been given to the development of methods with engagement by development practitioners in LMICs—adapted methodologies. The dissatisfaction with donor-driven approaches, recognition of the central importance of engagement with development agents in LMICs, and the recognition by many of the need for different methods to address different questions, has led to a different approach

to the design and application of evaluation methodologies in some quarters. Many of these methodologies are still developed in large measure by evaluators and theorists in the global North, but they are explicitly concerned with application in LMICs and with the need to address, with partners from LMICs, how to adapt the approach to different socio-cultural, political, economic, and ecological settings. They start from the assumption that methodology is context-sensitive not context-neutral. In contrast with the adopted methods outlined above, they tend to cluster on the Use branch tending towards Valuing.

Adapted methodologies include many participatory methodologies, epitomized by the work of Robert Chambers (2008), such as rapid rural appraisal and participatory rural appraisal. Rapid rural appraisal emerged from practice, notably in Thailand where there was a need for rapid, timely, relevant, accurate and usable information by decision makers. This was in a context where much misleading information was available but also where extensive and detailed surveys could not be completed in a timely manner. Participatory rural appraisal “seeks and embodies participatory ways to empower local and subordinate people, enabling them to express and enhance their knowledge and take action” (Chambers, 2008, p. 85). The procedures provide opportunity for local people to play a role in determining value. While Chambers did not invent these methods, his perspective on participatory and action research has been instrumental in formalizing and disseminating these approaches. Many others, too many to enumerate here, have also been involved and have been central in participatory methods developments. The point here is that the formalization of methods has

Fred Carden and Marvin C. Alkin

Journal of MultiDisciplinary Evaluation, Volume 8, Number 17 ISSN 1556-8180 January 2012

109

been in the hands of researchers from the global North, but the articulation and application has often emerged in LMICs and has been sensitive to the contexts at play in LMICs. Adaptation has been fundamental to design. These methods situate on the Valuing branch.

Other adapted methodologies have emerged out of concerns with methodological gaps in development evaluation. Two examples follow, but there are others that also fit this category well. Space limitations confine us to a sample of approaches.

The first example, outcome mapping (Earl, Carden, & Smutylo, 2001), is concerned with explicitly trying to address the development funding conundrum: how do you assess contribution to development when you are not dealing with a singular intervention that can be randomly assigned and tested? For what can a development research project or program reasonably and fairly take credit? Outcome mapping recognizes that development is about people relating to each other and their environment. Rather than focus on products of a development program, outcome mapping focuses on changes in behaviors, activities and relationships in the individuals and organizations with which a development program works directly. It is specific about the theory of change being employed: the actors who are targeted in a program, what change it expects to see and the strategies it employs to get there. Outcome mapping provides a framework for reflection as the program evolves, thereby enhancing the potential for use. Outcome mapping has been taken up in a diverse range of programs and has been adapted in a number of ways. For example, based on their field experience in Indonesia, Deprez, Nirarita and Shatifan (2010) wrote a guide on using

outcome mapping for community development programming in Indonesia, making the adaptations they felt maintained the theoretical foundations and philosophical roots of outcome mapping, but adapted language and approach to the needs and style in their programming in Indonesia. Similarly, Loza (2010) led outcome mapping work with a very different program, carrying out research on trade issues in Latin America, and wrote up her experiences in that context, again with adaptations to the subject and geographic context. Outcome mapping sits on the Use branch of the Evaluation Theory Tree.

Another adapted methodology is most significant change. It was designed in 1996 by Rick Davies and Jess Dart (Davies & Dart, 2005) while working in South Asia. While it is a form of participatory monitoring and evaluation, it is noteworthy for its significant uptake and origins in the field. Davies and Dart were both struggling with how to evaluate large, multi-player programs that did not have clear indicators. As they both struggled with the problem in separate projects, they maintained a dialogue and built on the approach that Davies developed in his work in Bangladesh. The approach builds on stories of significant change related to a program. The analysis is carried out with the project participants and the findings are used to monitor impact and build on successes. Most significant change integrates the user directly into the evaluation process. Stories of significant change are identified at the field level and shared with a selection committee (from the program); the committee selects the most significant among the stories to share with the next level of the organization (could be a regional or a national body); this body in turn selects the most significant stories for

Fred Carden and Marvin C. Alkin

Journal of MultiDisciplinary Evaluation, Volume 8, Number 17 ISSN 1556-8180 January 2012

110

sharing with others—for example, donors, interested publics, and government agencies. The stories of change are analyzed by the players themselves and selected based on representativeness and importance for sharing with others. They are oriented not only to sharing success but to learning from what has happened— significant change can also be negative— and improving for the future. This approach is intended to help organize a large amount of diverse information that all relates to program progress. Its nature makes it useful for identifying unexpected outcomes as well as expected outcomes. The variety supports the development of a rich picture of a complex process of change. Most significant change sits on the Use branch of the Evaluation Theory Tree

Developmental evaluation as described by Patton (2010) clearly fits here as well. As defined by Michael Quinn Patton, “Developmental evaluation supports innovation development to guide adaptation to emergent and dynamic realities in complex environments” (p. 1). It situates the evaluator inside a program, supporting evaluative thinking and ongoing reflection to help programs cope in complex and changing environments. Patton was influenced in the design of developmental evaluation by his work not only in North America but also in his work with indigenous communities. Developmental evaluation sits clearly on the use branch, reflected in Patton’s existing presence on that branch, but developmental evaluation as an approach would be placed at a higher point on the branch than Patton is currently represented on the Tree.

A number of other methodologies such as realist evaluation (Pawson & Tilley, 1997) and sensemaker (Snowden, 2002) are being actively adapted for use in

development evaluation. They are not included here because they were not developed to address development evaluation gaps, but were developed in other settings. A more detailed typology could however find them a home in this discussion.

The common characteristics across adapted methodologies is that while they are still formalized in the global North, they include significant engagement with the global South and in some cases are directly built on practice in the global South. They assume context specificity and expect modifications of the methodology. Indigenous Methodologies We have identified some evaluation methodologies developed in the global South. The distinctions are not always clear cut. For example, rapid rural appraisal has strong roots in Thailand, but we have chosen to identify it as an adapted methodology because of the strong role in formalization of the method by individuals and organizations from Europe. Indigenous methodologies seem to be region specific and are usually not effectively shared across regions. Some are related to efforts in the global North but have clear design and roots linked to local theory and practice. Indeed some indigenous evaluation methods development also takes place in the global North (see, for example, Chapter 30 of the second edition of Evaluation Roots for a discussion of work employing Maori concepts developed by Wehipeihana, McKegg, Cram, and others). Here, we outline three examples, one each from Africa, South Asia and Latin America. Others surely exist; these are meant to be illustrative of the applications in use. As Alkin, Christie, and Vo note in Chapter 30

Fred Carden and Marvin C. Alkin

Journal of MultiDisciplinary Evaluation, Volume 8, Number 17 ISSN 1556-8180 January 2012

111

of Evaluation Roots (2012), placement of methods on the Evaluation Theory Tree can be extremely challenging. While each of the examples below is unique, they all reflect the close interaction between the Valuing and Use branches of the Evaluation Theory Tree and frequently express a gap in addressing local values in the methodologies developed in industrial countries that most closely reflect them.

The African Peer Review Mechanism (APRM; developed by NEPAD—the New Partnership for African Development) is one good example. It is indigenously developed, evolved, and studied. It is a specific set of procedures for country self- assessment around governance and human rights. The approach, while clearly based on the OECD Peer Review Mechanism (OECD, 1997), was developed within the NEPAD framework, by African researchers and policy makers and is applied by indigenous teams and leads to African-based assessments (inter alia Kanbur, 2004; Herbert & Gruzd, 2008; Mbelle, 2010; Turianskyi, 2010). About half of all African countries have joined this voluntary scheme which focuses on governance (political, economic and corporate) and socio-economic development in its peer review.

A peer review mechanism is, as it sounds, a formal process whereby your peers review your policy and practice in a field such as governance, development policy, etc., and provide feedback that builds on a set of principles and reflects their experience and observations about improvement that could be made in your approach. OECD introduced a system of peer review more than forty years ago and the system is both widely used within the OCED and has been adopted and adapted in many other settings. The African mechanism is one of the most significant and is considered indigenous here

because of the significant engagement of African agents and agencies in its design, implementation, critique, and evolution. It is a voluntary process and builds on a country self-assessment report. The self- assessment is developed as a national dialogue that is open and expects input from all sectors, including non- governmental and the private sector. The review includes not only an assessment mission, but a follow-on National Plan of Action for immediate and longer term changes that are intended based on the findings of the peer review. National assessments are peer reviewed by heads of state and senior officials. Processes are sometimes criticized such as the South African peer review (Mbelle, 2010; Turianskyi, 2010), and some of these criticisms extend to the continental APRM processes. This in itself is an important signal of the importance attached to the process and concerns to ensure progress in what is a relatively young and inexperienced process. A comment on the mechanism by Cilliers in 2003 remains relevant today: “Many questions remain…the debate on peer review has opened up considerable space for African civil society organizations to seek out and establish parallel processes to hold African governments and leaders accountable to their stated commitments and decisions. Perhaps it is here where most hope for accountability and effective review could be found.” (Cilliers, 2003, p. 14). APRM is situated on the Use branch of the Evaluation Theory Tree, leaning towards Valuing

Sistematización, or systematization, is a Latin-American methodology, developed by a small team in the late 1980s. By contrast with APRM, sistematización is on the Valuing branch, leaning towards Use. It is an approach rooted in the work of Paulo Freire and

Fred Carden and Marvin C. Alkin

Journal of MultiDisciplinary Evaluation, Volume 8, Number 17 ISSN 1556-8180 January 2012

112

participatory action research but adapted over time within the Latin American context. As with many indigenous methods and much development evaluation more generally, the design roots are multiple and it will be extremely difficult to ascribe to one person (Oscar Jara, Sergio Martinic, Daniel Selener, Mariluz Morgan, and Esteban Tapella would be a significant core group). Much of the documentation in English on the methodology (León, n.d.) was supported by ActionAid, a non-governmental British development agency that is active in Latin America. The authors indicated above continue to publish in Latin America on experience with systematization (Tapella, 2011).

Systematization is the “participatory and thoughtful process of reflecting about different aspects of a specific project or general intervention: its actors, actors’ interaction, outcomes, results, impact, problems and processes.” (Tapella, n.d., p. 24). The basic concept is for participants to reconstruct, analyze and interpret an experience in order to learn for future interventions. It is often used as an ongoing process of reflection and learning in a project or intervention. As with most significant change and outcome mapping, systematization tends to support the explication of unexpected outcomes and impacts. It is built on the assumption that many interventions do not happen as planned so a systematic documentation and a formal and documented reflection on the actual path an intervention takes helps to both identify the path and learn for improvement.

In a brief article, Hansford, Santana, and Reinoso (2007) outline an experience using systematization with a network of community-based, non-governmental organizations focused on strengthening food security in Latin America. The

network emerged from a series of projects amongst a group who wanted to keep the reflection and learning that had been important to the projects’ success alive for future initiatives. With support from one of the international donors, a core of people were trained as facilitators in systematization. The confidence this built among the development practitioners gave them the impetus to both improve programs but also to disseminate their experience. In one case, the Association for the Sustainable and Supportive Development of the Sisal Region in Brazil, key elements of the micro-finance and technical assistance programs that were developed have been incorporated into national policies to strengthen small-scale farming. The Citizen Report Card was designed and implemented in India by the Public Affairs Centre (see http://www.citizenreportcard.com/). It was designed to permit citizens to get feedback to public agencies and was designed with local use in mind in the Indian context. The Citizen Report Card sits on the Use branch of the Evaluation Theory Tree, with a strong orientation to Valuing because of the importance it attaches to citizen empowerment in its design and processes. Through participatory surveys, the Citizen Report Card is intended to improve access to and delivery of public services. The documentation that is available is instrumental, reports such as those found on project websites (such as http://www.adb.org/projects/e-toolkit/ Naawan-Citizen-Report.pdf) and on knowledge exchange sites such as that of Civicus (see http://pgexchange.org /index.php?option=com_content&view=a rticle&id=164&Itemid=160). Some reports have been more formally published (Sekhar, Nair, & Prabhakar,

Fred Carden and Marvin C. Alkin

Journal of MultiDisciplinary Evaluation, Volume 8, Number 17 ISSN 1556-8180 January 2012

113

2008). This document reports on the ranking quality of public services provided by Gram Panchayats (local governance bodies) in two districts of Chattisgarh state, India. The findings were openly discussed and debated within the districts but there is no follow-up data readily available on whether or not sustainable changes have been made since the work was reported.

The Public Affairs Centre (http://www.pacindia.org/index.html) in India maintains some minimal documentation on Citizen Report Cards, but as in the case of many action oriented agencies, engages in limited academic publication of its work. This is an important, indigenously developed approach that is well known in South Asia and amongst the small core of people with a strong interest in community based surveys. It merits more attention in the evaluation research community. It would be important, for example, to document follow up where Citizen Report Cards have been presented (such as in the example above) and through that process, potentially improve the report cards and their use. This would permit expanding the scope of the experience and use of the approach beyond the experience of the Public Affairs Centre.

Placement on the Evaluation Theory Tree Clearly, we have not found a strong focus on evaluators in the South as individual theorists. There are, however, systemic influences on evaluation through organizations and institutions. All of this is interesting and we think worth pursuing in some form even though it does not fit so neatly with the framework we were attempting to work with in linking to the Roots book. We still see linkages, but of necessity in trying to show a relationship to evaluation theory in the North we will need to be accommodating. Clearly some indigenous methods can be included. In that regard, we will attempt to place on the Evaluation Theory Tree contributions to theory development that were group developed (so designated) with some of the names of those identified with the group noted in the text (see Figure 2). Moreover, adaptations to Northern evaluation procedures (as we have designated them) will be included when they show promise for potentially evolving into a unique theory reflecting local values and contexts. Again, so designated.

Fred Carden and Marvin C. Alkin

Journal of MultiDisciplinary Evaluation, Volume 8, Number 17 ISSN 1556-8180 January 2012

114

Figure 2. Modified Evaluation Theory Tree In Table 1 we have elaborated on the LMIC approaches found in Figure 2 by presenting them in columns for each of the branches of the tree and have indicated which of the identified approaches in LMICs is to be found in each. Moreover, these approaches had been placed in the table in a manner

roughly corresponding to their placement on the tree branch (top to bottom). Where possible, organizations or individuals identified with an approach have been specified. Finally, as a reminder we have indicated whether the approach is adopted, adapted, or indigenous.

Fred Carden and Marvin C. Alkin

Journal of MultiDisciplinary Evaluation, Volume 8, Number 17 ISSN 1556-8180 January 2012

115

Table 1 Development Evaluation Approaches and the Theory Tree

Use Methods Valuing

Developmental Evaluation

Patton

(Adapted)

Rapid Rural Appraisal, Participatory Rural Appraisal

Chambers

(Adapted)

Outcome Mapping

Earl, Carden & Smutylo

(Adapted)

Systematization

Selener, Tapella et al.

(Indigenous)

Most Significant Change

Davies and Dart

(Adapted)

3ie: International Initiative on Impact Evaluation

Center for Global Development

(Adopted)

Citizen Report Card

Public Affairs Centre, India

(Indigenous)

African Peer Review Mechanism

New Partnership for Africa’s Development

(Indigenous)

Logical Framework Analysis/Results-Based Management

Many bilateral development agencies as well as multilateral development banks (e.g., World Bank)

(Adopted)

An Aside An unanticipated consequence of this investigation is a concern about the applicability of the tree branches for considering evaluation in the LMICs. As noted early in this article, there are relationships among the branches in the Theory Tree with some connection

between the Use and Valuing branches— they are not opposite extremes, but rather bending towards each other. Based on his experience in the LMICs, one of the authors (Carden) is proposing here that there is a new branch growing that bridges Use and Valuing. As systems concepts take more root in evaluation and as an increasing number of authors

Fred Carden and Marvin C. Alkin

Journal of MultiDisciplinary Evaluation, Volume 8, Number 17 ISSN 1556-8180 January 2012

116

address systems thinking as a key influence in the field, complexity is emerging as a driving influence, as what some methodologies would be least likely to give up in any compromise. Neither outcome mapping nor developmental evaluation sit completely comfortably on the Use branch. A key driving influence in these (and other new approaches) is their focus on emergence as a key factor. They are built on the premise that we are often working with things that cannot be known in advance. Therefore our starting point is not a clear set of objectives and actions for an intervention but rather a high level outcome and the early steps that can be taken to move in that direction. As context intervenes through political, social, economic and ecologic changes, the path changes and the programs must necessarily change to take account of these changes. The other author (Alkin) suggests that on the one hand, complexity may not be a new branch but may simply be an extension of the Use branch. That is, it is a concern considered in seeking to attain use. Thus complexity as a potential defining characteristic is much like a further stage of the Use branch which at its base, looked at evaluation and decision making, but subsequently had a broadened definition to users more generally and to increased organizational capacity building. On the other hand, there may be a need, given the diversity in the LMIC world, to create a new branch called “Context,” attesting to the complexity and uncertainty in these contexts as the defining characteristic. Clearly, there are innumerable and extremely diverse contexts that need to be attended to. This author has recognized the context issue by his Chapter in Evaluation Roots (2nd ed.)

entitled “Context Sensitive Evaluation” (Alkin, 2012). The authors of this piece, while recognizing that they are both from North America, welcome input from those more familiar with evaluation in LMICs. We hope that you will reflect on these musings.

Authors’ Note The views expressed are those of the authors. The authors would like to thank Tarek Azzam for his assistance with the Evaluation Theory Tree figures.

References Alkin, M. C. (Ed.). (2004). Evaluation

roots: tracing theorists’ views and influences. Thousand Oaks, CA: Sage.

Alkin, M. C. (Ed.). (2012). Evaluation roots (2nd ed.). Thousand Oaks, CA: Sage.

Campbell, D. T. (1957). Factors relevant to the validity of experiments in social settings. Psychological Bulletin, 54, 297-312.

Campbell, D. T., & Stanley J. C. (1966). Experimental and quasi-experimental designs for research. Chicago. IL: Rand McNally.

Carden, F. (2007). The real evaluation gap. Alliance, 12(4), 53-54.

Carden, F. (2010). Introduction to the Forum on Evaluation Field Building in South Asia. American Journal of Evaluation, 31(2), 219-221.

Center for Global Development. Evaluation Gap Working Group. (2006). When will we ever learn: Improving lives through impact evaluation. Washington, DC: Center for Global Development.

Fred Carden and Marvin C. Alkin

Journal of MultiDisciplinary Evaluation, Volume 8, Number 17 ISSN 1556-8180 January 2012

117

Chambers, R. (2008). Revolutions in development inquiry. London: Earthscan.

Chilisa, B. (2012). Indigenous research methodologies. London: Sage.

Cilliers, J. (2003). Peace and security through good governance: a guide to the NEPAD African Peer Review Mechanism (Paper no. 70). Pretoria, South Africa: Institute for Security Studies.

Davies, R., & Dart, J. (2005). The ‘Most Significant Change’ technique: A guide to its use. Available at http://mande.co.uk/docs/MSCGuide. pdf

Deprez, S., Nirarita, E., & Shatifan, N. (2010). Outcome mapping: Jejak perubahan menuju kerberhasilan. Denpasar: VECO Indonesia.

Earl, S., Carden, F., & Smutylo, T. (2001). Outcome mapping: Building learning and reflection into development programs. Ottawa: International Development Research Centre.

Gasper, D. (2000). Evaluating the ‘logical framework approach’ towards learning-oriented development evaluation. Public Administration and Development, 20, 17-28.

Hansford, F., Arauứjo, V., Santana, L., & Reinoso, G. H. (2007). From participatory systematization to a regional network for policy change. Participatory Learning and Action, 56, 33-40.

Herbert, R., & Gruzd, S. (2008). The African Peer Review Mechanism: Lessons from the pioneers. Johannesburg, South Africa: South African Institute of International Affairs.

International Initiative for Impact Evaluation (3ie). (n.d.). International Initiative for Impact Evaluation

website. Available at http://www.3ieimpact.org/.

Kanbur, R. (2004). The African Peer Review Mechanism: An assessment of concept and design. Politikon, 31, 157- 166.

León, R. (n.d.). Systematizing and learning from our experience on advocacy and campaign in the Americas. In R. León (ed.), Advocacy for change. London: ActionAid.

Loza, J. (2010). Monitoring and evaluating research outputs: The use of outcome mapping in the Latin American Trade Network. Saarbrücken: Lambert Academic Publishing.

Marrow, A. (1969). The practical theorist: The life and work of Kurt Lewin. New York, NY: Basic Books.

Mbelle, N. (2010). The APRM process in South Africa. Rosebank, South Africa: Open Society Initiative for Southern Africa.

Morra-Imas, L., & Rist, R. (2009). The road to results: Designing and conducting effective development evaluations. Washington DC: The World Bank.

Organization for Economic Co-operation and Development (OECD). (1997). Peer review: A tool for co-operation and change. Paris, Organization for Economic Co-operation and Development.

Pawson, R., & Tilley, N. (1997). Realistic evaluation. London: Sage.

Patton, M. Q. (2011). Developmental evaluation: Applying complexity concepts to enhance innovation and use. New York, NY: Guildford.

Scriven, M. (1967). The methodology of evaluation. In R. E. Stake (Ed.), Curriculum evaluation (American Educational Research Association

Fred Carden and Marvin C. Alkin

Journal of MultiDisciplinary Evaluation, Volume 8, Number 17 ISSN 1556-8180 January 2012

118

Monograph Series on Evaluation No. 1). Chicago, IL: Rand McNally.

Sekhar, S., Nair, M., & Prabhakar, K. (2008). Public services provided by Gram Panchayats in Chattisgarh. Chattisgarh & Bangalore: Smarthan and Public Affairs Centre.

Selener, D., Purdy, C., & Zapata, G. (1996). A participatory systemization workbook: documenting, evaluating and learning from our development projects. Quito, Ecuador: International Institute for Rural Reconstruction.

Shadish, W. R., Cook, T. D., & Leviton, L. C. (1991). Foundations of program evaluation: Theories of practice. Newbury Park, CA: Sage.

Snowden, D. (2002). Complex acts of knowing: Paradox and descriptive self- awareness. Journal of Knowledge Management, 6(2), 100-111.

Stake, R. E. (1967). The countenance of educational evaluation. Teachers College Record, 68, 523–540.

Stake, R. E. (1975). Program evaluation particularly responsive evaluation (Occasional Paper No. 5). Kalamazoo, MI: The Evaluation Center.

Tapella, E. (2011) ‘¿Cómo aprender desde la práctica? Aproximaciones conceptúales y metodológicas para la sistematización de experiencias de desarrollo’, en Perspectivas em Políticas Públicas, Vol. II, Nº 4 (in press).

Tapellan, E. (n.d.). Systematization: Basic concepts and methodological considerations. In R. León (Ed.), Advocacy for change. London: ActionAid.

Turianskyi, Y. (2010). Off track? Findings from South Africa’s first APRM implementation report. Johannesburg, South Africa: South African Institute of International Affairs

United Nations Office of Internal Oversight Services. (2008). Review of results-based management at the United Nations. New York, NY: United Nations.