Assignment4

The Logic Model: A Tool for Incorporating Theory in Development and Evaluation

of Programs

Riki Savaya, PhD Mark Waysman, PhD

ABSTRACT. The role of program theory in developing, operating and evaluating programs is gaining increased emphasis in recent years. A clear grasp of a program’s theory can contribute to the successful perfor- mance of many important developmental and evaluative tasks along its life span. Although there is growing recognition of the importance of program theory in the development and evaluation of programs, it should be noted that this is not a simple task. Programs are often com- plex, comprising many different types of interlinking components. What is needed is a relatively simple instrument that can help the practitioner explicate and present program theory, by guiding and structuring the process. The logic model is such a tool, whose purpose is to describe and articulate program theory. Drawing upon examples from the authors’

Riki Savaya is affiliated with the Bob Shapell School of Social Work, Tel Aviv Univer- sity, Israel, and the Center for Evaluation of Human Services, Rishon Lezion, Israel. Mark Waysman is affiliated with the Center for Evaluation of Human Services, Rishon Lezion, Israel.

Address correspondence to: Dr. Riki Savaya, Bob Shapell School of Social Work, Tel Aviv University, P.O.B. 39040, Ramat Aviv 69978, Israel (E-mail: savaya@post. tau.ac.il).

This article is based on a paper presented at the International Conference on Evalua- tion for Practice, University of Huddersfield, England, 12-14 July, 2000.

Administration in Social Work, Vol. 29(2) 2005 http://www.haworthpress.com/web/ASW

© 2005 by The Haworth Press, Inc. All rights reserved. Digital Object Identifier: 10.1300/J147v29n02_06 85

work with both nonprofit and governmental organizations, this paper presents potential uses of the logic model tool in explicating program theory for a variety of purposes throughout the life span of programs: for assessing the feasibility of proposed programs and their readiness for evaluation, for program development, for developing performance mon- itoring systems, and for building knowledge. [Article copies available for a fee from The Haworth Document Delivery Service: 1-800- HAWORTH. E-mail ad- dress: <[email protected]> Website: <http: //www.HaworthPress. com> © 2005 by The Haworth Press, Inc. All rights reserved.]

KEYWORDS. Logic model, program theory, program evaluation, per- formance monitoring systems

Over the last decades the literature on programs and interventions in the field of social work reflect a growing emphasis on incorporating theoretical frame- works in practice and in program evaluation (Alter & Egan, 1997; Alter & Murty, 1997; Mizrahi, 1992; Reid, 1994; Rosen, 1993). This implies a clear definition of the population, problems and outcomes that are the focus of any program, a clear presentation of the theoretical assumptions that guide the choice of intervention, and systematic assessment of effectiveness. Current professional standards for judging the quality of a program and assessing its readiness for dissemination thus require a high level of clarity in conceptual- ization and presentation of programs.

Concurrently, literature in the field of program evaluation shows a growing emphasis on the role of program theory in developing, operating and evaluat- ing programs (Bickman, 1987, 1990; Chen, 1990; Weiss, 1997). Wholey (1987) states that program theory identifies “program resources, program ac- tivities, and intended program outcomes, and specifies a chain of causal as- sumptions linking program resources, activities, intermediate outcomes, and ultimate goals” (p. 88). As such, program theory lays out the assumptions that are held regarding the transformative mechanisms that are considered to be ac- tive in creating change and growth (Lipsey, 1987; Pawson & Tilly, 1997).

Several authors have pointed to the fact that, in many cases, program theo- ries exist but are never made explicit (Cook, 2000; Leeuw, 2003). Schon (1983, 1995), for example, points out that even experienced practitioners may well be unable to transcend their concrete actions and translate their tacit knowledge into abstract concepts that may be examined and applied beyond their own setting. Although their activities may reveal an inner logic, in which the outcome of one action sets in motion a process of reflection which prompts

86 ADMINISTRATION IN SOCIAL WORK

the next action, they may well be unable to explain the connections. The knowledge they demonstrate in their actions remains implicit. The series of activities that unfolded on the basis of their tacit considerations and knowl- edge is difficult to transfer to others, so that they can learn from them, and dif- ficult to replicate elsewhere.

Thus, while there is growing recognition of the importance of program the- ory in the development and evaluation of programs, and there is often an ex- pectation that theory be addressed, this is clearly not a simple task. Programs are often complex, comprising many different types of components. Attempt- ing to integrate them all into a single coherent framework may prove daunting.

As such, what is needed is a relatively simple instrument that can help the practitioner by guiding and structuring the process. The logic model frame- work is an ideal tool for this purpose. The logic model is a diagram that describes “how the program theoretically works to achieve benefits for participants. It is the ‘If-Then’ sequence of changes that the program intends to set in motion through its inputs, activities, and outputs” (United Way of America, 1996, p. 38). The logic model captures the logical flow and linkages that exist in any program. Even in cases where the theory of a program has never been made explicit, the logic model approach can help to uncover, articulate, present and examine a program’s theory.

Although there are several ways to represent the logic model, it is usually set forth as a diagram, resembling a flow chart, that contains a series of boxes linked via connecting one-way arrows, as presented in detail, for example, in recent manuals published by the United Way (1996) and the W. K. Kellogg Foundation (2000). The basic logic model includes the following components:

Inputs: the human, financial, organizational, and community resources that need to be invested in a program so that it will be able to perform its planned activities.

Activities: what the program does with the inputs; the processes, events, and actions that are an intentional part of the program implementation.

Outputs: the direct products of program activities, usually measured in terms of the volume of work accomplished (e.g., the number of classes taught, number of group meetings held, number of pamphlets distributed, number of announcements broadcast on radio or TV) and the number of people reached (number who attended each meeting, number who received written materials, etc.).

Outcomes: the benefits or changes in the program’s target population; for example, changes in knowledge, perceptions, attitudes, behavior,

Riki Savaya and Mark Waysman 87

or status. Programs often posit a chain of outcomes that are linked to each other in a logical sequence over time, with immediate outcomes leading to intermediate outcomes, which in turn lead to long term-outcomes. For instance, it may be expected that new knowl- edge and increased skills (immediate outcomes) will lead to modi- fied behavior (intermediate outcomes), which will lead, in turn, to improved condition (long-term outcome).

Inputs or resources to the program usually appear in the first box at the left of the model, and the longer-term outcomes are presented on the far right. In between these ends, the major program activities are boxed, followed by in- tended outputs and outcomes from each activity (McLaughlin & Jordan, 1999). Figure 1 presents the basic format of a generic logic model. Actual models are usually much more complex, since programs often contain a vari- ety of activities, each of which may require its own inputs and also may lead to specific outputs, that are directed at either common or separate outcomes. Graphic presentation of this complexity may thus require more boxes to rep- resent the different components and more arrows to depict the linkages be- tween them (for examples, see Bickman, 1990; Wandersman et al., 1997; Weiss, 1997).

The logic model is a tool that practitioners and evaluators are finding increasingly useful in explicating and presenting program theory for many purposes, as witnessed by the publication of user manuals (e.g., W. K. Kellogg Foundation, 2000; United Way of America, 1996) and profes- sional papers (for example, Alter & Egan, 1997; Cooksy, Gill, & Kelly, 2001; Cozzens, 1997; Julian, 1997; McLaughlin & Jordan, 1999). The structured nature of the end product (the logic model diagram) helps to structure and focus the process of articulating the program theory by di- viding it up into discrete units (inputs, outputs, outcomes, etc.) that are connected via links that can be readily examined for logic and feasibility.

Most existing publications, however, have focused on a limited range of uses. In this paper, we demonstrate potential uses of the logic model tool in explicating program theory for a variety of purposes, via an inte-

88 ADMINISTRATION IN SOCIAL WORK

Inputs Activities

Situation: Target Population & Needs

Outputs Initial Outcomes

Intermediate Outcomes

Long-Term Outcomes

FIGURE 1. Basic generic logic model.

grative framework based on the life span of programs (see Figure 2). The paper consists of a series of case studies, taken from our work, that illus- trate the different uses of logic models at different points in time through- out the life cycle of programs.

PROGRAM INCEPTION STAGE: USING LOGIC MODELS FOR ASSESSING PROGRAM FEASIBILITY

AND READINESS FOR EVALUATION

Purpose

A clear and concise presentation of program theory can be extremely useful prior to program implementation and prior even to the detailed planning of programs, at the early stage, when the idea for a program is first being consid- ered. When programs are first being conceived and examined, feasibility and evaluability assessment help us to determine (a) whether there is a good chance that the program can, in fact, be implemented with the given resources, (b) whether the program has a reasonable chance of achieving its intended out- comes, and (c) whether the program has been presented in a sufficiently clear and detailed manner to allow for the planning of its evaluation. A proper as- sessment can prevent investment of resources on programs that, with some forethought, could have been seen in advance to be impractical or ineffective. For this purpose, program planners need to clarify several things: What does the program intend to achieve (outcomes)? For whom (target population)? Via which activities? and With what resources? The resources available must be sufficient to implement all the planned activities, these activities must be ap- propriate to the target population (age-appropriate, culturally sensitive, etc.), and they must also logically have a reasonable chance of leading to the desired outcomes. In the following example, the authors demonstrate the use of the

Riki Savaya and Mark Waysman 89

Program Inception Program Modification Program Operation

LM for assessing program feasibility and its readiness for evaluation

LM for program development

LM for developing performance monitoring system

Program Dissemination

LM for knowledge building

FIGURE 2. Uses of the logic model (LM) along the life span of a program.

logic model to assess the evaluability of a program, prior to its commence- ment.

Context

The Special Projects (SP) department within the Planning and Research di- vision of a large Israeli government agency funds pilot testing of innovative new social welfare programs that are subsequently to be disseminated by other public agencies. One of the conditions for funding any program is that its initi- ators and managers agree for it to be accompanied by external evaluation. As each program is approved for funding, qualified evaluators are approached and asked to submit evaluation proposals.

Example

In this case, we were sent a copy of the program proposal that a government agency had submitted to the SP department for running a program geared to helping a specific group of parents, who had been separated from their chil- dren. The program’s aim was to help these parents prepare for resuming con- tact with their children and to help them develop their parenting skills. We were asked to study the document and submit a proposal of our own for evalu- ation of this program that had been approved for funding and was about to commence operation.

Review of this agency’s proposal raised a number of questions about the program’s readiness for both implementation and evaluation. We therefore decided that prior to preparing a formal evaluation proposal, we would attempt to systematically examine its feasibility and evaluability (Wholey, 1987) by assessing the program theory with the help of a logic model. Using the compo- nents of the logic model described above (target populations, problems, re- sources, activities, outputs, outcomes), we read through the proposal and marked any statement that could be classified as pertaining to any one of them. We then assembled all the statements in each category and tried to see if we could arrive at a clear picture of each component. Finally, we examined whether there appeared to be logical linkages connecting these components to each other. By the end of this process, we had some serious concerns that this program, at least as it appeared in writing, may not be ready for implementa- tion or for evaluation: Some of the activities appeared to be very difficult to implement and the intervention itself did not seem to have a reasonable chance of leading to its stated outcomes.

For example, the program’s target population was defined in different ways throughout the proposal: “mothers or fathers of children up to age 17, who

90 ADMINISTRATION IN SOCIAL WORK

have not lived with them for at least six months”; “any father who has at least one child under the age of 17 or whose wife is pregnant”; “any father about to return home [no specification of children’s ages].” It is apparent that the target population was defined differently in different places, such that it was not clear to us whether the program was intended for fathers only or for both fa- thers and mothers; whether a minimum period of separation was required; and whether or not the program was limited to actual parents or was also to include parents-to-be.

The program’s planned activities were not specified in the proposal, be- yond a general statement that they will include 12 weekly group meetings, each lasting two hours. It was, therefore, not possible to infer from the pro- posal what experiences these parents were going to undergo that were ex- pected to lead to the desired outcomes, which, in turn, were also not specified beyond general statements about preparing the parents for their return home and enhancing their parenting skills. Figure 3 illustrates how the logic model helped us to identify and clarify the problems in this program proposal and to present them in a succinct and clear visual schema.

While the reader may suspect that we have singled out here an extreme ex- ample of poor or vague program planning, from our experience, this is unfor- tunately not the case. Programs are usually planned by practitioners, who have little experience in planning and developing new interventions and are not usually aware of the importance of explicating the program theory when de- veloping a program proposal. Without the aid of a simple and structured tool for reviewing programs, such as the logic model, it is easy to “get lost” and end up with a plan that is full of holes or contradictions.

Although we were unable in this case to specify a clear logic model, our at- tempt to do so guided us in converting a long, diffuse proposal into a clear, concise, structured and focused document that highlighted the lacunae in the theory underlying this program. This document was sent to the SP department, along with our reply, in which we declined their offer to submit an evaluation

Riki Savaya and Mark Waysman 91

Situation Inputs Activities Outputs Outcomes

different and contradictory definitions of the target population: no specification of needs

OK group meetings contents unspecified

− 12 weekly meeetings, each lasting two hours

general, non-specific statements, not operationally defined

FIGURE 3. Example of logic model to help assess program feasibility and readiness of evaluation.

proposal at this point in time, since it appeared to us that the program was not yet ready for evaluation. Instead, we recommended that the agency should work on clarifying and specifying the program theory prior to its implementa- tion and prior to planning any evaluation.

Although we do not know whether this recommendation was accepted, in- terestingly, the SP department later decided to contract with evaluation con- sultants to help them assess program proposals, prior to their final approval. Previously, evaluators had been included in the process only after approval of funding, at which point it can be very difficult to make major changes in a pro- gram.

PROGRAM MODIFICATION STAGE: USING LOGIC MODELS FOR IMPROVEMENT

OF EXISTING PROGRAMS

Purpose

In many cases, initial program planning does not include a clear specifica- tion of all program components and a systematic examination of the logical linkages between them. Although this may not necessarily be a problem, there may be cases where difficulties become apparent after a program has been in operation for a while. Logic models can also be beneficial in these cases, where they may focus formative efforts at program improvement. In the fol- lowing example, we constructed and elaborated the logic model of an existing program with management and staff of the Department of Social Welfare in a large Israeli city, in a collaborative process that lasted almost a year, based on the findings of an implementation evaluation.

Context

The Welfare Department established a network of Family Aid Centers (FACs), aimed at improving services to those segments of the population that had not been considered a high priority in the past. In particular: (a) clients who had applied only for material or instrumental assistance and (b) those for whom traditional approaches (such as psychotherapy and counseling) had been exhausted but who were still in need of other forms of assistance. These neighborhood-based FACs were to provide instrumental and material help, in a fast, efficient and courteous manner, without requiring an in-depth psychosocial assessment and without the prior prerequisite, that receipt of instrumental aid would be dependent on the client’s willingness to participate in counseling.

92 ADMINISTRATION IN SOCIAL WORK

The FACs were also supposed to develop new and innovative services for these groups and, in particular, were to promote the use of advocacy strategies to help these clients exercise their rights, within both the social welfare system and other public agencies.

Example

After the FACs had been operating in six neighborhoods within the city for a period of about two years, it was decided to evaluate their implementation and generate recommendations for enhancing the model prior to expanding the program to additional sites throughout the city. The evaluation had four stages:

1. to uncover and elucidate the FACs’ existing conceptual model (mission, policies, and practices);

2. to assess the implementation and effectiveness of this model; 3. to identify gaps between the conceptual model and its implementation

and results in practice; and 4. to facilitate development, operationalization and consolidation of a re-

vised model geared to narrow the above gaps.

Construction of a logic model was the tool that we used in the fourth stage. First, we presented our findings from the previous stages, regarding the FACs’ existing conceptual model and its implementation, which revealed disagree- ments and lack of clarity among stakeholders regarding program components, such as target populations and required inputs, as well as discrepancies be- tween the program plan and its implementation in practice.

Based on these findings, we initiated an interactive, participatory process involving ourselves and two separate groups within the department (adminis- tration and program staff). Over a period of eight months, we held a total of 15 meetings, each lasting approximately three hours, during which we utilized the logic model tool to develop and refine the FAC program. The components of the logic model guided and structured this group process. Each meeting was devoted to defining and elaborating one component of the logic model, which in this case began with the program’s “mission.” Only after the two groups had reached agreement on the written formulation of a component did we progress to the next one. Since each component in the logic model stems logically from the previous component, the process flowed in a systematic and organized manner.

For example, the program’s intended outcomes stemmed clearly from the activities that are to be implemented by the FACs, which in turn were targeted

Riki Savaya and Mark Waysman 93

at addressing specific client problems. One problem that the group agreed should be addressed by the FACs was difficulties that some clients have in re- alizing their rights for services and benefits that they are entitled to receive from government agencies. Several types of activities were planned to address this problem, including provision of information about rights and entitlements, representing clients before governmental agencies, physical accompaniment of clients to agencies, and development of advocacy campaigns to promote utilization and expansion of rights. These activities were expected to lead to an increase in utilization of benefits and rights among clients that failed to do so in the past.

The discussions were often very heated, suggesting a high degree of in- volvement and interest on the part of group members. The end result of this group process was a written program plan that differed from the original con- ception in several ways: It was more explicit, more detailed and specific, more operational, more consensual and, most of all, much tighter–all components were clearly and logically connected and integrated into a coherent program theory.

This product served a number of purposes within the organization and also between the organization and external agencies. First, the existence of a clear and consensual logic model facilitated communication, cooperation, and shared understanding among different levels of management and staff within the Welfare Department and also promoted greater clarity in role division (see Coffman, 2000). It also specified the inputs that each FAC needed and was en- titled to for implementing the planned activities and who was responsible for allocating these resources. Previously, there had been considerable confusion and disagreement regarding key elements in the program that created tension and frustration among the stakeholders, limited the scope of activities, and prevented the FACs from realizing their full potential to help their target cli- ents.

Furthermore, the existence of a clearly articulated document presenting the program model helped the Welfare Department to present the program to ex- ternal agencies, such as government ministries and foundations, in order to raise funds for running and evaluating the program.

Finally, this program was developed in a large multicultural city, and was to be implemented in a variety of different neighborhoods, each with different sociodemographic characteristics: different ethnic groups, with different reli- gious affiliations and levels of religiosity (from secular to ultra-orthodox), as well as immigrants from many different countries. As such, it was clear that any model developed might have to be adapted to suit the characteristics of the local residents. The existence of the logic model helped to make decisions

94 ADMINISTRATION IN SOCIAL WORK

about the components and elements that were core to the program versus those that could be adapted (for more details on this application, see Hacsi, 2000).

PROGRAM OPERATION STAGE: USING LOGIC MODELS

FOR DEVELOPING PERFORMANCE MONITORING SYSTEMS

Purpose

Performance monitoring (PM) has been widely promoted as an effective way to ensure a focus on program results and accountability, to determine what publicly funded programs are actually doing, and to provide for control over expenditures (Perrin, 1998). Performance monitoring systems are cur- rently being implemented, and sometimes even built into legislative require- ments by governments in developed countries throughout the world (e.g., the United States 1993 Government Performance and Results Act; see Wholey, 1997). In the private and nonprofit sectors, performance monitoring is also gaining popularity as a means for keeping programs on track and improving the quality of program implementation and results by providing ongoing peri- odic measurement of key indicators (Newcomer, 1997).

Along with this, critics have claimed that performance monitoring is often implemented inappropriately, focusing on trivial indicators, providing data that are irrelevant to stakeholders (and especially program staff) and not focus- ing sufficiently on outcomes (Bernstein, 1999; Greene, 1999; Winston, 1999). Many of these shortcomings may be prevented, when the measures are se- lected and developed after the program theory has been formulated in collabo- ration with the system’s intended users.

Context

The example that we present here stems from our work with the New Israel Fund (NIF), a philanthropic foundation that helps to fund local NGOs active in achieving social change. The NIF also offers its grantees two additional ser- vices via its capacity-building branch, SHATIL: (1) technical assistance and training in a number of areas (such as organizational development and re- source development); and (2) help in establishing and maintaining coalitions of NGOs to promote common goals. While SHATIL is an extension of the NIF, it also receives independent funding from other sources, some of whom have begun to require greater accountability and, in particular, assessment of the results of SHATIL’s work. The director of SHATIL approached us to help

Riki Savaya and Mark Waysman 95

the organization develop a PM system, with the specific requirement that the development process be in accord with SHATIL’s organizational culture, which tends to be more egalitarian and less hierarchical, thus involving a high degree of staff participation in many processes. When the decision to develop the PM system was presented to staff, with the understanding that they would be actively involved in the development process, they saw it also as an oppor- tunity to obtain a set of tools that would help them to learn from their accumu- lated experience in a systematic manner so as to improve practice.

Example

We present here the work that we did in developing a PM system for one of the SHATIL teams: consultants in organizational development (OD). In ap- proaching this task, we decided that the first step should be to develop a clear and comprehensive picture of their work. The rationale was that the PM sys- tem should be based on a set of indicators that truly reflect the core aspects of this team’s work–the program theory that stems from the practice wisdom they have developed over the years–and not necessarily the indicators that are most sa- lient or easiest to measure. We thought that the ideal way to arrive at this picture was with the aid of a logic model.

In developing the model, we decided to engage in a group dialogue with the team, not only because management had requested this, but also because we believed that in order to promote future utilization of the system, it should have broad support from within and reflect a consensus that the system is rele- vant and useful. The information to be provided should be generally perceived as bearing the potential to help the OD consultants learn from their experience and derive ideas for improving the quality of the consultation that they provide their clientele.

Although the group process with this team proved to be considerably more difficult and more prolonged than we had anticipated, it was clearly justified. It required the consultants to explicate and formulate their activi- ties, their working assumptions and the outcomes that they strive to achieve with their clients. As often occurs in other participatory processes with peo- ple that are invested in their work, here, too, the process involved a number of very heated discussions. Consensus was, however, reached eventually, leading to a clear model depicting all the components of their work. For ex- ample, it included a detailed statement of six major outcome dimensions, formulated as areas of organizational health: It was conceptualized that any NGO that functions well in all of these dimensions can be deemed to be healthy and not in need of help from SHATIL. These outcomes cover all the

96 ADMINISTRATION IN SOCIAL WORK

main target areas to which SHATIL OD consultants direct their interven- tions, such as establishing “a clear and coherent organizational identity,” “a clear and functional organizational structure,” or “an organizational culture that promotes growth and effectiveness.” Each outcome dimen- sion was further broken down into several more specific outcomes. For example, the outcome dimension “existence of a clear and coherent orga- nizational identity” comprises the following three specific outcomes: “the organization has a clear and coherent vision and mission,” “the orga- nization’s target populations are clearly defined,” and “the organiza- tion’s goals and objectives are clearly specified and linked to its mission.”

Once agreement was reached on all the components of the logic model, the next step was to determine which components and which parts within each component would be routinely monitored as part of the PM system, since it is neither feasible nor advisable to monitor them all. This was achieved by asking the team to review the entire model and to prioritize and choose the areas of most value for reporting to external sources, such as donors, and for internal purposes, such as organizational learning. Thus, they were asked to decide whether or not they wish to monitor in- puts and, if so, which inputs. They were then asked the same question re- garding activities, outputs, and outcomes. The existence of a written, clearly conceptualized and consensual model, depicting the consultants’ work as a whole and the theory underlying it, made this step of the process relatively easy, and agreement on this was thus reached rapidly, leading to the following step: development of indicators and measures.

In this example, we were invited to help an organization develop a PM system to monitor its work. We used a clearly articulated logic model as the basis for selecting the aspects of program performance that would be monitored and thereby ensured that the aspects most important for ac- countability and for organizational learning were selected, rather than those that are easiest to measure, as is often the case.

We believe that the direct work on the development of the system went smoothly and quickly because of our previous investment of time and ef- fort in constructing the logic model. We further believe that this PM sys- tem has a greater chance of being accepted within the organization and utilized over time, due to the fact that staff were part of the process, that it focuses on aspects of their work that are meaningful and relevant to them, and that are clearly linked to their overall mission.

Riki Savaya and Mark Waysman 97

PROGRAM DISSEMINATION STAGE: USING LOGIC MODELS

FOR KNOWLEDGE BUILDING

Purpose

Since development of a new program usually involves a large investment of effort, time and money, program funders, initiators and implementers often want their investments to yield benefits beyond the immediate context. In ad- dition, prominent figures in the evaluation field (e.g., Cronbach et al., 1980) have suggested that evaluation should “facilitate the transfer of knowledge from some programs or sites to other programs or sites, through explaining the processes that cause or prevent the outcomes achieved” (Shadish, Cook, & Leviton, 1991, p. 323). Logic models provide one of the most efficient ways to examine the applicability and generalizability of programs to other settings and populations.

Context

This example also stems from our work with SHATIL. Whereas the previ- ous example illustrated the work of SHATIL’s consultants that work with NGOs on capacity-building, this example pertains to the second main service that SHATIL provides its clients: assistance in establishing and maintaining coalitions.

From its consulting work with Arab organizations active in the area of edu- cation, it became increasingly clear to SHATIL that attempts to solve prob- lems of inequality in allocation of government resources by the traditional method of providing alternative services had many limitations. A different ap- proach is embodied in programs that aim to change government policy rather than to provide substitutes for its areas of neglect. These programs employ a variety of social change strategies, such as advocacy, lobbying, pressure groups, public education, etc. Based on these ideas, SHATIL in 1991 launched its Equal Access Initiative to empower Arab leadership and community-based organizations to achieve concrete policy changes in the area of early child- hood education. The project based its activities on a coalition task force com- prised of representatives from existing Arab NGOs, active in the field of early childhood education, that was established, coordinated and supported by SHATIL.

The evaluation had two components. The first component (see Savaya & Waysman, 1999), which assessed the program’s effectiveness, revealed that it was highly effective in achieving its intended outcomes. This success

98 ADMINISTRATION IN SOCIAL WORK

prompted the second component of this evaluation, aimed at conceptualizing a program model that could be examined to determine its generalizability to other populations and problems and, where appropriate, taught and dissemi- nated to other settings and organizations. This conceptualization was needed since the program was essentially experimental, and many of its interventions were developed during its implementation, in accord with changing circum- stances and responses from stakeholders to prior interventions. In this exam- ple, we describe our application of the logic model tool to conceptualize this advocacy program and its use in determining the program’s generalizability to other settings.

In this case, the logic model was developed retrospectively, via an attempt to induce the principles underlying this program from a study of its products and activities. To this end, we collected data from three sources: written mate- rials generated by the project, in-depth interviews with project stakeholders, and observations of task force meetings. These data were the basis for a collab- orative, interactive effort, in which the authors used the logic model frame- work to conceptualized the program’s theory, followed by feedback from stakeholders and revisions (this process is described in detail in Waysman & Savaya, 2003).

Once the model was complete, and we had a clear and concise representa- tion of this complex program, we were able to utilize the logic model for as- sessment of the program’s generalizability to other settings.

To achieve this aim, we first identified, via professional literature and per- sonal acquaintance, a group of five expert consultants with sufficient knowledge and experience, from different perspectives, to evaluate the generalizability of the program. The experts were from different fields (education, psychology, social work, and sociology) and ethnic backgrounds (Arabs and Jews), from univer- sities in Israel and the U.S., and each is well-known for his or her expertise in one or more areas relevant to this endeavor. The five experts all received a re- port, which presented the methods employed to develop the model, the graphic diagram of the model and the narrative text describing it. They were asked to read these materials and to respond in writing to a set of questions re- garding the generalizability of this model to other settings. We thus received five written expert opinions that we reviewed and analyzed to identify their main themes.

Interestingly, although the five experts were from different disciplines, cul- tures and countries, they all agreed that the SHATIL model has great potential for solving social problems in many different areas. They also specified addi- tional prerequisite conditions for its success, particularly highlighting the role of the wider sociopolitical context in enabling a project of this kind to occur

Riki Savaya and Mark Waysman 99

and succeed (e.g., they suggested that a program of this kind requires an open democratic political system in order to work).

Although it was not feasible in this case, it is possible to add an additional component to this process: direct interaction among the experts, to enable them to relate to each other’s opinions, either via a group meeting, video con- ference or a real-time Internet discussion. Such interaction among the experts could help to focus and enrich the ideas that emerge from this process.

Use of the logic model served here as a powerful tool for building knowl- edge about a successful program: It helped us to organize and structure a very large amount of information from a variety of sources into a clearly conceptu- alized model, presented graphically and summarized in writing, that accu- rately reflected the program from the point of view of its initiators and implementers. In addition, due to its concise and concrete nature and its visual layout, the model was very easy to comprehend, thus also making it very easy to elicit feedback from stakeholders, who could readily identify gaps and mis- takes requiring revision, as well as from experts, who were able to use the model for judging program generalizability.

This model was also subsequently used by SHATIL for an additional pur- pose: to teach and disseminate their methods and strategies for establishing and maintaining coalitions for social change.

CONCLUSIONS

This paper has presented a series of examples illustrating how the process of explicating program theory can contribute to the planning of programs and to their development, implementation, evaluation and dissemination. The the- orizing process was demonstrated with the aid of the logic model framework, which serves to guide, structure and simplify the activity. Our examples dem- onstrate that this framework can be useful in helping us to think about pro- grams at various points throughout their life cycle.

From the examples presented here, spanning from the earliest stage of pro- gram inception through program modification, operation, and dissemination, it is clear that both the process of developing a logic model and the existence of the product itself, can contribute in many ways to program success. For ex- ample, it can be used to

• assess the feasibility of a program idea; • assess the readiness of a program for evaluation; • facilitate communication, cooperation, and shared understanding among

stakeholders;

100 ADMINISTRATION IN SOCIAL WORK

• promote greater clarity in role division among staff; • specify the resources needed to run the program; • present the program to external agencies for different purposes, such as

fundraising; • aid in making decisions about adaptations and modifications of a pro-

gram to fit with the needs and characteristics of different implementation sites;

• help select the aspects of program performance to be monitored, when developing a performance measurement system, so as to ensure that the aspects most important for accountability and for organizational learn- ing are selected (rather than those that are easiest to measure);

• generate knowledge about successful programs; • elicit feedback from stakeholders and experts, regarding different issues,

such as relevance, innovation, and generalizability to other settings; and • teach and disseminate the program.

Despite its many advantages, some limitations of the logic model approach should be noted. First, developing a logic model can, at times, be a lengthy and costly endeavor. From our experience the process can take anywhere from a single sitting lasting just a few hours to many sittings over a period of months. Sometimes, depending on the purpose, it is enough for just one person who knows the program well to be involved in the process; at other times it is essen- tial that many people and groups, both from within and outside the organiza- tion take part. At times, discussions regarding a logic model may reveal conflict or power struggles within the organization that had been dormant as long as things were left vague. The process of specifying a logic model may also direct participants to the conclusion that, to ruin the program, changes will need to be made in organizational structure (e.g., changes in role defini- tions and hierarchies). These issues require considerable investment of time and effort on the part of the organization and, as such, should be kept in mind when deciding to embark on this process. Finally, it should be noted that logic models are merely conceptual representations of programs that should always be attentive to changes in their environment and clientele. As such, we need to be aware of the risk of rigidly adhering to a model as it was originally con- ceived, when external circumstances have changed and may require rethink- ing of the model.

Although other frameworks are available for conceptualizing programs, from our experience the logic model is the most comprehensive and easy to use. Cooksy and her colleagues (2001) have reached the same conclusion, af- ter comparing the logic model to other approaches, including path diagrams, program templates, concept maps and textual descriptions.

Riki Savaya and Mark Waysman 101

REFERENCES

Alter, C., & Egan, M. (1997). Logic modeling: A tool for teaching critical thinking in social work practice. Journal of Social Work Education, 33, 85-102.

Alter, C., & Murty, S. (1997). Logic modeling: A tool for teaching practice evaluation. Journal of Social Work Education, 33, 103-117.

Bernstein, D. J. (1999). Comments on Perrin’s “Effective use and misuse of perfor- mance measurement.” American Journal of Evaluation, 20, 85-93.

Bickman, L. (1987) (Ed.) Using program theory in evaluation. New Directions for Evalu- ation, Number 33. San Francisco: Jossey-Bass.

Bickman, L. (1990) (Ed.) Advances in program theory. New Directions for Evaluation, Number 47. San Francisco: Jossey-Bass.

Chen, H. (1990). Theory-driven evaluations. Newbury Park, CA: Sage Publications. Coffman, J. (2000). Simplifying complex initiative evaluation. The Evaluation Ex-

change (Harvard Family Research Project Quarterly Newsletter), 5(2/3), 2-3. Cook, T. D. (2000). The false choice between theory-based evaluation and experimen-

tation. In P. J. Rogers, T. A. Hacsi, A. Petrosino, & T. A. Huebner (Eds.), Program theory in evaluation: Challenges and opportunities. New directions in evaluation (Vol. 87, pp. 27-35). San Francisco: Jossey Bass.

Cooksy, L. J., Gill, P., & Kelly, P. A. (2001). The program logic model as an integrative framework for a multimethod evaluation. Evaluation and Program Planning, 24, 119-128.

Cozzens, S. E. (1997). The knowledge pool: Measurement challenges in evaluating fundamental research programs. Evaluation and Program Planning, 20, 77-89.

Cronbach, L. J., Ambron, S., Dornbusch, S. M., Hess, R. D., Hornik, R.C., Phillips, D. C., Walker, D. F., & Weiner, S. S. (1980). Toward reform of program evalua- tion. San Francisco: Jossey-Bass Publishers.

Greene, J. C. (1999). The inequality of performance measurements. Evaluation, 5, 160-172.

Hacsi, T. A. (200). Using program theory to replicate successful programs. In P. J. Rogers, T. A. Hacsi, A. Petrosino, & T. A. Huebner (Eds.), Program theory in evalu- ation: Challenges and opportunities. New directions in evaluation (Vol. 87, pp. 71-78). San Francisco: Jossey Bass.

Julian, D. A. (1997). The utilization of the logic model as a system-level planning and evaluation device. Evaluation and Program Planning, 20, 251-257.

Leeuw, F. L. (2003). Reconstructing program theories: Methods available and prob- lems to be solved. American Journal of Evaluation, 24, 5-21.

Lipsey, M. W. (1987). Theory as method: Small theories of treatments. Paper pre- sented at the National Center for Health Services Research conference: Strengthen- ing causal interpretations of non-experimental data, Tucson, Arizona.

McLaughlin, J. A., & Jordan, G. B. (1999). Logic models: A tool for telling your pro- gram’s performance story. Evaluation and Program Planning, 22, 65-72.

Mizrahi, T. (1992). The Future of Research Utilization in Community Practice. In A. J. Grasso & I. Epstein (Eds.), Research utilization in social services. New York, London and Norwood: The Haworth Press, Inc.

102 ADMINISTRATION IN SOCIAL WORK

Newcomer, K. E. (1997) (Ed.) Using performance monitoring to improve public and nonprofit programs. New Directions for Evaluation, Number 75. San Francisco: Jossey-Bass Publishers.

Pawson, R., & Tilley, N. (1997). Realistic evaluation. Thousand Oaks, CA: Sage Publica- tions.

Perrin, B. (1998). Effective use and misuse of performance measurement. American Journal of Evaluation, 19, 367-379.

Reid, W. J. (1994). The Empirical Practice Movement. Social Service Review, 68(2) 165-184.

Rosen, A. (1993). Systematic Planned Practice. Social Service Review, 67(1) 84-100. Savaya, R., & Waysman, M. (1999). Outcome evaluation of an advocacy program to

promote early childhood education for Israeli Arabs. Evaluation Review, 23(3), 281-303.

Schon, D. A. (1983). The reflective practitioner: How professionals think in action. New York: Basic Books.

Schon, D. A. (1995). Reflective inquiry in social work practice. In P. M. Hess and E. J. Mullen (Eds.), Practitioner-researcher partnerships, building knowledge from in, and for practice. Washington, DC: NASW Press.

Shadish, W. R., Cook, T. D., & Leviton, L. C. (1991). Foundations of program evalua- tion: Theories of practice. Newbury Park, CA: Sage Publications.

United Way of America. (1996). Measuring program outcomes: A practical approach. Alexandria, VA: United Way of America.

Wandersman, A., Goodman, R. M., & Butterfoss, F. D. (1997). Understanding coalitions and how they operate: An “open systems” organizational framework. In M. Minkler (Ed.), Community organizing and community building for health (pp. 261-277). New Brunswick, NJ: Rutgers University Press.

Waysman, M., & Savaya, R. (2003, under review). A Program Model for Coali- tion-Based Social Change Initiatives: Case Study of the SHATIL Equal Access Pro- ject in Early Childhood Education.

Weiss, C. H. (1997). Theory-based evaluation: Past, present, and future. In D. J. Rog (Ed.), Progress and future directions in evaluation: Perspectives on theory, practice, and methods. New Directions for Evaluation, Number 76. San Francisco: Jossey-Bass.

Wholey, J. S. (1987). Evaluability assessment: Developing program theory. In L. Bickman (Ed.), Using program theory in evaluation. New Directions for Evalua- tion, Number 33. San Francisco: Jossey-Bass.

Wholey, J. S. (1997). Trends in performance measurement. In E. Chelimsky and W. R. Shadish (Eds.), Evaluation for the 21st century, pp. 124-133. Thousand Oaks, CA: Sage Publications.

Winston, J. A. (1999). Performance indicators–Promises unmet: A response to Perrin. American Journal of Evaluation, 20, 95-99.

W. K. Kellogg Foundation. (2000). Logic model development guide. Battle Creek, Michi- gan: Author.

Riki Savaya and Mark Waysman 103