6w1

BibliU-Print-B1BGTXDIJJSYB1.pdf

38187753 - Pearson Education Limited ©

1. 2. Describe the gap between the desired and current state of affairs. 3. Describe the restraining forces: those forces that resist movement toward

the desired state. 4. Describe the driving forces: those forces that can assist movement

toward the desired state. 5. Select from the restraining forces those that will be addressed during

change efforts. Restraining forces that both are important and capable of being weakened are chosen.

6. Select from the driving forces those that will be addressed during change efforts. Driving forces that both are important and capable of being strengthened are chosen.

7. For each restraining force chosen in phase 6, identify actions that will weaken the restraining force.

8. For each driving force chosen in phase 7, identify actions that will strengthen the driving force.

9. Integrate and sequence the actions chosen in phases 8 and 9 to create a comprehensive action plan for moving from the current state of affairs to the desired state of affairs. Include a plan for evaluating the effects of the action plan and a timeline for implementing the action plan and evaluation.

Figure 13.15 symbolizes the first five phases of a force field analysis. The analysis is most effective if the descriptions of the desired and current state of affairs as well as restraining and driving forces are based on data gathered for the force field analysis, rather than on the planners’ perceptions alone.

Figure 13.15 Overview of Initial Phases of Force Field Analysis

38187753 - Pearson Education Limited ©

Current State of Affairs

Driving Forces

Resisting Forces

Desired State of Affairs

Description of what is

Description of what should be

38187753 - Pearson Education Limited ©

Cautions Concerning Planning

Postmodern theory warns us that the rational analyses, predictions, controls, and measurements that make up conventional planning often are inconsistent with the multiplicity of realities and competing interests that make up the real world. Chaos theory reminds us that, unlike traditional plans, complex systems such as schools do not consist of simple cause-and-effect relationships, are affected by seemingly unrelated variables inside and outside the system, and are subject to unpredictable events and changes. (A full discussion of chaos theory and its implications for schools is presented in Chapter 21 .)

Do these cautions mean that we throw out the planning tools discussed in this chapter and simply “wing it” regarding instructional improvement? No, but they do mean that we change our traditional concept of planning to provide for the following:

Planning that is centered on people—students, teachers, families, and community members—not on planning tools and documents (the tools and documents should be used in the service of people) Decentralized planning in which the supervisor becomes the coordinator of stakeholder teams actively involved in the planning process Diversity, in terms of both the groups represented and the ideas considered in the planning process Regular interaction among planners and between planners and those who will be affected by the plan A recognition by planners of the complexity, multiplicity, and nonlinearity inherent in the school organization The consideration of alternative futures for the school community Improvement plans with built-in flexibility, allowing a fair degree of spontaneity and exploration during implementation

List attributes of planning that is better suited for schools than conventional planning. Learning Objective 13.6

38187753 - Pearson Education Limited ©

Provisions for continuous and critical assessment of the plan during its implementation An understanding that the important thing is the goal the plan is intended to reach, not the plan itself, and a willingness to revise the plan when appropriate

Check Your Understanding 13.0 Click here to gauge your understanding of the key concepts presented in this chapter.

38187753 - Pearson Education Limited ©

Field Exercise

Through discussions with members of a small group of teachers who work closely with one another and who are willing to participate in the exercise, choose a broad area that all or most of the teachers in the group identify as a general professional need. Design a brief survey to help determine specific needs within the broad area you have chosen to investigate. The survey can be a multiple-choice, open-ended, or ranking survey. Administer the survey to the group you are working with. Analyze survey results and identify specific needs indicated by those results. Share the survey results with the teachers and ask for their reactions to the survey results. Write a report on the exercise. In your report, discuss (1) the broad area of need selected by the teachers, (2) the survey you created, (3) the results of the survey, (4) the teachers’ reactions to survey results, (5) how the survey you designed could be improved, and (6) other ways of gathering data that would contribute to the needs assessment. Do not reveal the names of any of the teachers in your report.

38187753 - Pearson Education Limited ©

Chapter 14 Implementation and Evaluation Skills

Learning Objectives

for this chapter:

Questions to Reflect on

as you read this chapter:

1. Think of an innovation that was attempted at a school you work in or are familiar with. Did the innovation go through each of the five stages of implementation described in this chapter? If so, what was taking place at each stage, and how did moving through that stage affect the innovation? If the innovation did not move through each stage, which stage or stages were left out, and how, if at all, did omitting the missing stage or stages affect the innovation?

2. Reflect on the implementation of an innovation currently taking place in a school you work in or are familiar with. How are the five factors that affect implementation discussed in this chapter impacting implementation, either positively or negatively?

Describe the five stages of implementation. Identify five factors that affect implementation. List stages of concern at the self, task, and impact level. Discuss why teachers should be involved in program evaluation. Describe the difference between formative and summative program evaluation. Explain the purpose of empowerment evaluation.

14.1 14.2 14.3 14.4 14.5

14.6

38187753 - Pearson Education Limited ©

3. This chapter reviews Wolfe’s five typical methods of program evaluation. Although Wolfe presents the methods in a humorous light, many schools actually use methods approximating Wolfe’s. What program evaluations have you participated in or observed that have relied on one or more of Wolfe’s methods? Which method or methods were used?

4. Consider an instructional program you are familiar with for which a summative evaluation is warranted. What questions about the program should the evaluation focus on? From what data sources (persons, places, things, events, or processes) should data needed to answer your evaluation questions be collected? What data-gathering methods (observing, testing, surveying, interviewing, and so on) should be used to collect data from the sources you have identified?

5. In what ways would a school that you work at or are familiar with benefit from empowerment evaluation? What concerns would you have about bringing empowerment evaluation to the school?

In Chapter 13 we dealt with the first two components of instructional improvement at the school level, needs assessment and planning. In this chapter we discuss the third and fourth components, implementation and evaluation. Our treatment of the third component focuses on the implementation of an innovation: a new curriculum, instructional strategy, model of teaching, or instructional program. Here we describe the stages of implementing an innovation, factors affecting the quality of implementation, and implementation at the individual level. In the second part of this chapter we discuss formative and summative evaluation as well as empowerment evaluation. In Chapter 21 we discuss the broader topic of school culture and change.

38187753 - Pearson Education Limited ©

Stages of Implementation

Assuming that a needs assessment has been carried out and a plan developed, major innovations still need to proceed through several stages of implementation if they are to have a reasonable chance of success. A number of scholars have described the stages of implementation (Arrowsmith et al., 2021 ; Frank et al., 2022 ; McIntosh et al., 2018 ; Nielsen et al., 2018 ; Swain-Bradway et al., 2015 ; Ward et al., 2021 ) and, although they do not all agree on the number of stages or names of each stage, there is enough consistency among the scholars to identify five broad stages: (1) readiness, (2) installation, (3) initiation, (4) full operation, and (5) continuation and renewal. Let’s consider a school’s implementation of a new program on service learning to illustrate these five stages.

Readiness

The most important type of readiness is that of those who will be responsible for the success of the program. In the case of service learning, this includes teachers, families, and community members. Initial professional development for teachers assists them to understand the value of service learning to the personal, social, and academic development of students as well as to the school and outside community. Any new program needs the support of families, and the school can share the benefits of service learning through online communication, meetings of the school–family council and family–teacher association, and informal conversations between educators and family members. Although it can be carried out at school, much service learning takes place in the community and often requires community partners to work with teachers and students. Therefore, community members—both those who will assist students with and be assisted by community service learning—are helped to understand and to prepare to participate in such learning. Communication with the school-community council, community organizations, service agencies, and community leaders prepares the community for a service-learning program.

Describe the five stages of implementation.Learning Objective 14.1

38187753 - Pearson Education Limited ©

Community publications, educator visits to community events, and town hall meetings ready the community for service leaning. Beyond stakeholder readiness, the school establishes a clear vision of what it wants service learning to accomplish and ensures that it can provide the resources necessary to move toward that vision.

Installation

A leadership team, with a majority of teachers, is established to assist with the implementation of service learning. Guidelines for service learning, with input from all stakeholder groups, are developed. Resources and materials are procured. Support structures for service learning such as professional development days, peer observation, peer coaching, co-teaching, and professional learning communities are planned. Formal service-learning partnerships are established with community organizations, agencies, and stakeholder groups.

Initiation

Professional development utilizes the aforementioned structures to assist teachers to plan and facilitate service learning and integrate that learning with the existing curriculum. Those teachers who are more confident about delivering service-learning pilot service-learning projects, then assist others with their early application. Implementation is incremental. A few service-learning projects are carried out, with teachers learning from those projects how to improve future projects. The first projects may be school based, with service learning gradually moving into community settings.

Full Operation

Service learning has become standard practice throughout the school. Most teachers have become proficient at planning and facilitating service learning. Professional development has shifted to sharing variations of service learning and applying service learning to interdisciplinary projects and varied learning environments. Service learning has been fully integrated with the rest of the instructional program.

Continuation and Renewal

38187753 - Pearson Education Limited ©

Time and energy must be expended to sustain service learning. Over time, formal and informal leaders of service learning will leave the school, and new leadership committed to service learning will be necessary. New teachers must be oriented to and assisted in developing skills for facilitating service learning. Resources and materials used for service learning become depleted and must be replaced. Changes in the student population, school staff, and community mean that service learning must be periodically renewed to remain congruent with the educational environment. Existing school–community partnerships for service learning need to be sustained, and new partnerships need to be developed. Renewal of service learning is based on data gathered on the effects of service learning, including feedback from all stakeholder groups.

38187753 - Pearson Education Limited ©

Factors That Affect Implementation

Fixsen et al. (2005) describe three degrees of implementation:

1. Paper implementation: the policy calling for the innovation and procedures for implementation are in print and available.

2. Process implementation: training on the innovation is provided, manuals on implementation are distributed, coaches are assigned to assist with implementation, and so forth.

3. Performance implementation: the innovation is actually being applied in a way that benefits those who it is intended to benefit.

The problem with innovations in schools (and other organizations as well) is that many innovations never progress beyond the first or second degree of implementation. To illustrate, one of the authors once attended a state-level meeting of educators at which a school district was given an award for an innovative beginning teacher support program. A few days later, the author, who knew teachers in the district, congratulated them on the award. The teachers responded that they had never heard of the support program in question, and that new teachers in their schools were not receiving any new types of support. The award-winning support program was in fact a “paper program” that, at least at that point in time, had never been implemented at the school level.

As major innovations proceed through the previously discussed stages of implementation, a variety of other factors affect the quality of implementation (Gale et al., 2020 ; Genlott et al., 2019 ; Kafyulilo et al., 2016 ; Martinez et al., 2022 ). These factors can be grouped under five broad categories:

External Factors. These include any factors outside of the school that impact implementation, including families and the community, the district

Identify five factors that affect implementation.Learning Objective 14.2

38187753 - Pearson Education Limited ©

office and other schools in the district, and state and federal policies. The School Organization. The school culture, including its levels of collegiality, collaboration, and openness to change, is one important factor within this category. School leadership also is critical, especially the leaders’ change-facilitation skills and commitment to the innovation. The innovation, if it is to be successful, must also have support from teachers who are trusted and respected by their colleagues. The school’s level of shared decision making is an important factor, including the extent to which the decision to adopt the innovation was a shared one. Teachers. Important factors regarding teachers include whether they have a clear understanding of the innovation and whether they see benefits of the innovation for themselves and their students. The motivation and capacity of teachers to develop the skills necessary to implement the innovation also are key factors. The Innovation. Stiegelbauer (2008) notes that the size and complexity of an innovation are salient factors in whether implementation will be successful. If the innovation is too small or too simple, teachers are not likely to take it seriously. If the change is too large and complex, teachers may resist it. Another crucial factor is whether the innovation is consistent with the school’s vision and mission, and whether it is compatible with school and classroom practices valued by the school community. The flexibility of the innovation is another relevant factor: Is the innovation capable of mutual adaptation with current school structures, processes, and practices? Support. Initial training in how to implement a major innovation is usually necessary but never sufficient. With most innovations, several support structures increase the probability of success. Early on, teachers benefit from seeing the innovation modeled in visits to other schools, workshops, or demonstration lessons. As they begin to apply a new curriculum or teaching model in their own classrooms, teachers can receive classroom assistance through co-teaching, peer coaching, or collaborative walkthroughs. Problems with implementation are bound to occur, and problem solving can take place in professional learning groups or conversations with critical friends. The presence and quality of support structures are significant factors in all stages of implementation.

38187753 - Pearson Education Limited ©

Our discussion of factors that affect implementation is closely related to the concept of fidelity (Harn et al., 2017 ; Helsabeck et al., 2022 ; Khoury et al., 2019 ; Lombardi et al., 2022 ; Thierry et al., 2022 ). Simply stated, fidelity means that the innovation is implemented as designed. On the one hand, Lendrum and Humphrey (2012) report that the lower the fidelity, the greater the risk that the innovation will fail to bring about the intended outcomes. On the other hand, the same authors state, “local changes are inevitable and surface-level adaptations may contribute to the effectiveness and sustainability of an intervention” (p. 643). The key to the fidelity–adaptation debate, according to Lendrum and Humphrey, is to strike a balance between the two, maintaining the core features of the innovation while allowing teachers to adapt other aspects of the innovation to their own students and teaching environment.

The desire for fidelity also needs to be tempered with flexibility in the support structures provided during implementation. For example, a peer coach’s assistance should not be focused entirely at ensuring fidelity. Rather, the coach and teacher receiving assistance should engage in reflective inquiry about how the innovation can be adapted to the teacher’s strengths, students’ needs, and content being taught without losing the core qualities of the innovation.

38187753 - Pearson Education Limited ©

Implementation at the Individual Level

Teachers have different concerns about innovations and go through the implementation process at different rates as their concerns are addressed. Hall and Hord (2006) state, “Even when the change is introduced to every member of the organization at the same time, the rate of making the change and of developing skill and competence in using it will vary individually” (p. 7). In short, teachers and others need individualized assistance when implementing an innovation. By recognizing and supporting teachers as individuals having their own unique implementation experiences, there is “the opportunity for administrators, teachers, and teacher educators to collaborate and address identified concerns in order to gauge fidelity of implementation to realize substantive change” (Rogers, 2021 , p. 10). Hall and Hord have identified seven stages (0–6) of concern about the innovation. Figure 14.1 describes the seven stages.

Reprinted and Electronically reproduced by permission of Pearson Education, Inc., Upper Saddle River, New Jersey.

Impact

6 Refocusing The focus is on exploration of more universal benefits from the innovation, including the

possibility of major changes or replacement with a more powerful alternative. Individual has

definite ideas about alternatives to the proposed or existing form of the innovation.

List stages of concern at the self, task, and impact level. Learning Objective 14.3

38187753 - Pearson Education Limited ©

5 Collaboration The focus is on coordination and cooperation

with others regarding use of the innovation.

4 Consequence Attention focuses on impact of the innovation on

students in the individual’s immediate sphere of influence. The focus is on relevance of the

innovation for students; evaluation of student outcomes, including performance and

competencies; and changes needed to increase student outcomes.

Task

3 Management Attention is focused on the processes and tasks

of using the innovation and the best use of information and resources. Issues related to

efficiency, organizing, managing, scheduling, and time demands are utmost.

Self

2 Personal Individual is uncertain about the demands of the

innovation, their adequacy to meet those demands, and their role with the innovation.

This includes analysis of the individual’s role in relation to the reward structure of the

organization, decision making, and consideration of potential conflicts with existing

structures or personal commitment. Financial or status implications of the program for self and

colleagues may also be reflected.

38187753 - Pearson Education Limited ©

1 Informational A general awareness of the innovation and

interest in learning more detail about it is indicated. The person seems to be unworried

about themselves in relation to the innovation. They are interested in substantive aspects of the innovation in a selfless manner such as general

characteristics, effects, and requirements for use.

Unrelated

0 Awareness Little concern about or involvement with the innovation is indicated.

Individuals may have concerns at several stages at any given time, but their most intense concerns often are at a particular stage. The stages of concern are developmental, moving from stage 0 (lowest stage) to stage 6 (highest stage). Individuals need to have their intense concerns at a particular stage met before they can concentrate on the next highest stage. The specific concerns of a teacher result not just from the innovation in isolation but from the interplay of the innovation, the teacher’s personal and professional characteristics, and the educational context (Kwok, 2014 ). Hall and Hord (2006) describe three ways supervisors and other change leaders can assess educators’ stages of concern about an innovation: (1) informal conversations, (2) an open-ended question asking educators to write about their concerns, and (3) the 35-item Stages of Concern Questionnaire. Figure 14.2 describes expressions voiced at each stage of concern.

Figure 14.2 Stages of Concern: Typical Expressions of Concern About the Innovation Source: Adapted from Hall, Gene E.; Hord, Shirley M., Implementing Change: Patterns, Principles, and Potholes, 3rd Ed., © 2011.

Reprinted and Electronically reproduced by permission of Pearson Education, Inc., Upper Saddle River, New Jersey. p. 140.

38187753 - Pearson Education Limited ©

Stages of Concern

Expressions of Concern

Impact

6 Refocusing I have some ideas about something that would work even better.

5 Collaboration I am concerned about relating what I am doing with it to what other instructors are doing with it.

4 Consequence How is my use of it affecting kids?

Task

3 Management I seem to be spending all my time getting materials ready for it.

38187753 - Pearson Education Limited ©

Self

2 Personal How will using it affect me?

1 Informational I would like to know more about it.

Unrelated

0 Awareness I am not concerned about it (the innovation).

Once teachers’ stages of concern have been identified, supervisors and other change leaders can assist change by addressing those concerns and fostering teacher development toward higher stages of concern. Bullard et al. (2017) recommend specific responses or actions for each stage of concern. For example, if a teacher is at the personal stage (stage 2) and is beginning to wonder how the change might affect them, Bullard et al. (2017) suggest that the supervisor continue to provide the teacher with information about the innovation and offer words of encouragement regarding implementation. For a

38187753 - Pearson Education Limited ©

teacher at the consequence stage (stage 4), they suggest sharing examples with the teacher of how the innovation has worked in other situations as well as information about how the change impacted students. An innovation will have a much greater chance for success once a critical mass of teachers reaches the impact stages of concern (consequence, collaboration, and refocusing). Teachers with these higher stages of concern are a worthwhile innovation’s best friend!

38187753 - Pearson Education Limited ©

Evaluation of Instructional Improvement Efforts

Perhaps no area in education has gone through such a dramatic expansion in knowledge, techniques, and attention than the field of educational evaluation. What once seemed the province of experts, consultants, and “university types” now has become a part of day-to-day operations in schools. The national accountability and testing movement certainly has been a contributing cause. More important has been the role of the literature on successful schools, which has made clear that decisions about instructional changes should be made from a base of comprehensive and credible data about students, and that those affected most directly by instructional change (i.e., teachers) should be involved in defining, implementing, and interpreting the evaluation agenda.

In schools and districts that do improve instruction, evaluation is not a perfunctory paper assignment done by a particular person or division to fulfill district or state requirements. Rather, evaluation is seen as the basis for determining professional actions as to the “what and how” of improving learning for students. Knowledgeable decision making about instruction comes from intense and critical study of the consequences of the common work of teachers: teaching. Including teachers in determining the criteria, procedures, and use of evaluative data in schools is not simply a nice thing to do—it is essential to do. If teachers are to extend their own thinking and commitment about collective instruction, they have to be part of the evaluation process. Without their involvement, policymakers have denied teachers the intellectual engagement of viewing teaching as a collective activity—“a cause beyond oneself.”

The Intersection of Instructional Supervision and Program Evaluation

What types of program evaluation are within the arena of instructional supervision? The criterion for answering this question is whether the program directly affects teaching and learning. For example, a district’s transportation,

Discuss why teachers should be involved in program evaluation. Learning Objective 14.4

38187753 - Pearson Education Limited ©

facilities maintenance, and athletic programs all are important, but evaluating these programs is an administrative responsibility, not a function of instructional supervision. Traditional programs that directly affect teaching and learning include curricula, instructional programs, professional development, student assessment, and so on. Additionally, in recent years, instructional supervision has expanded to include programs in areas like teacher leadership, action research, social justice, and community building, as the relationship of these areas with teaching and learning has been acknowledged.

Because program evaluation within the arena of instructional supervision relates directly to teaching and learning, it makes sense for teachers to be involved in instructional program evaluation. They are the professionals who have the responsibility for implementing instructional programs, have a firsthand view of how the programs are working, and are in the best position to gather and analyze relevant evaluation data. Later in this chapter we discuss empowerment evaluation as a vehicle for teacher participation in program evaluation.

Judgments

How do we know our instructional programs are successful? Should we continue with the same curriculum, instructional methods, scheduling, and grouping practices, or should changes be made? Evaluating is the act of making such a judgment. How do we decide whether something is good or bad? Frequently, we make judgments with statements such as “What a great reading program,” “What a lousy classroom,” and “What wonderful students.” How do we really know if something is great, lousy, or wonderful? Many years ago, Wolfe (1969) offered a tongue-in-cheek classification of five typical methods by which we make such judgments:

1. Cosmetic method. You examine the program, and if it looks good it is good. Does everybody look busy? The key is attractive and full bulletin boards covered with projects emanating from the program.

2. Cardiac method. No matter what the data say, you know in your heart that the program was a success. This is similar to the use in medical research of subclinical findings.

38187753 - Pearson Education Limited ©

3. Colloquial method. After a brief meeting, preferably at a local watering hole, a group of project staff members conclude that success was achieved. No one can refute a group decision.

4. Curricular method. A successful program is one that can be installed with the least disruption of the ongoing school program. Programs that are truly different are to be eschewed at all costs.

5. Computational method. If you have to have data, analyze them to death. Whatever the nature of the statistics, use the most sophisticated multivariate regression discontinuity procedures known to humans.

Wolfe’s humor aside, let’s look at reasonable, valid ways of evaluating. In the turbulence of instructional change, it is useful to know whether the new practice is going to be any better than the old. If not, then we may be investing large amounts of energy without a justifiable increase in instructional benefits to students. If we are to make a commitment to instructional change, we also must make a commitment to evaluating that instructional change.

38187753 - Pearson Education Limited ©

Two Types of Program Evaluation: Formative and Summative

There are two broad purposes of educational evaluation:

Formative evaluation is intended to improve a program. It is carried out while the program is in progress and can be ongoing throughout the life of the program. Summative evaluation results in a definitive judgment about the value of a program. It is carried out after a program has been in existence for a period of time. A summative evaluation is usually the basis for a decision about whether the program will continue, undergo major revisions, or be terminated.

Formative and summative program evaluations are not always mutually exclusive. For instance, data gathered for formative evaluations might be reanalyzed later as part of a summative evaluation.

Formative Program Evaluation

Formative evaluation is appropriate at any stage of implementation but is best initiated at the earliest stage. Evaluation data can be gathered both on how well the program is being implemented and on the effects of the program. Several of the ways of assessing need presented in Chapter 13 can be adapted for formative evaluation and are included in the example below. In this example, a school was in the beginning stages of their efforts to integrate explicit teaching of academic vocabulary into classroom instruction for all grades and subject areas. For each data source, examples of questions one might ask when evaluating the instructional program are listed, as well as a brief example of what that evaluation method might look like in practice:

Describe the difference between formative and summative program evaluation. Learning Objective 14.5

38187753 - Pearson Education Limited ©

Schoolwide observation. Are teachers modeling the use of academic vocabulary in their teaching? Are students using academic vocabulary in their classroom interactions? With the teacher? With their peers? Supervisors and teachers work together to create an observation tool tailored to assess implementation of the academic vocabulary program, and then that observation tool is used to gather data from each classroom. The schoolwide observation data that are collected should lead to dialogue across the campus about the meaning of the data, the status of the implementation, and any individual teacher or schoolwide support that is needed to further program development. Review of student performance data. Are students demonstrating understanding of the academic vocabulary in their classroom assignments and assessments? Are students using the academic vocabulary correctly when participating in class discussions? Grade-level or departmental teams meet to discuss student performance data. Teachers bring examples of student work to the meeting to share and review with each other, looking for evidence of how their explicit teaching of academic vocabulary is affecting student learning. From this collaborative dialogue, teachers decide on possible next steps for instruction. Curriculum and instructional resources and materials. Does the curriculum guide include the academic vocabulary deemed most important for students to know, understand, and be able to utilize to communicate their thoughts and ideas? Do teachers have a toolbox of strategies and activities, and the necessary materials, for explicitly teaching academic vocabulary? Supervisors and teachers together analyze the curriculum guide to determine whether it is serving the intended purpose and what needs to be revamped. They look at lesson plans, instructional technology, and other instructional strategies teachers are using to determine which academic vocabulary teaching methods are best supporting students to gain a better understanding of related academic concepts. Third-party review. What are the strengths and weaknesses of the academic vocabulary program? Are teachers implementing the program with fidelity and consistently from class to class? The supervisor solicits help from a neutral third-party, maybe a fellow supervisor from another school in the district who is familiar with explicit teaching of academic vocabulary. The

38187753 - Pearson Education Limited ©

supervisor explains the goals of the instructional program and the data they would like the third-party to gather. The fellow supervisor writes a report of their findings, and then the school uses that data to help inform future instructional decision making. Multiple-choice survey. Are teachers incorporating academic vocabulary into their instructional planning? Are teachers receiving adequate support to implement explicit teaching of academic vocabulary? Are teachers collaborating with colleagues about strategies for teaching academic vocabulary? Are students receiving multiple exposures to the selected academic vocabulary? Are students learning both definitional and contextual information about the vocabulary words? A multiple-choice survey asks respondents to agree or disagree with statements regarding the instructional program. Such a survey can be designed to elicit specific feedback from a specific stakeholder group. Written open-ended survey. What do teachers, students, families, and administrators think about the academic vocabulary program? The written open-ended survey is a brief questionnaire that asks various stakeholder groups to share their thoughts about the instructional program.

The two groups that tend to be most involved in instructional programs are teachers and students, so it makes sense to gather data from those populations. In some cases, it is also wise to gather data from families and other community members. In addition to the data collection methods just listed, interviews or informal discussions with small, representative groups of stakeholders also can be very helpful.

The best types of formative assessments are ongoing, enabling periodic adjustments to the program. Two methods for informal and continuous data gathering are the feedback window (also called the parking lot) and the progress board. Although both of these tools, as originally conceived, were wall charts in locations where educators visited on a regular basis (e.g., the teachers’ lounge, the teachers’ workroom), electronic versions of both can be used. The feedback window, shown in Figure 14.3 , allows any stakeholder to provide feedback on any or all of four areas, each area represented by a “window pane.” In our example, a supervisor met with a group of teachers who were implementing the Understanding by Design (UbD) instructional framework. To guide discussion

and gather data during the meeting, they used the feedback window as part of a formative evaluation of their new instructional program. Included in Figure 14.3 are the main points made by the teachers, including their most-often expressed comments about the UbD program and their experiences with UbD implementation.

Figure 14.3 Feedback Window

What aspects of the program are working well?

Identifying desired results is beginning to guide lesson planning

Students are responding positively to authentic performance tasks

UbD keeps our focus on student learning and understanding

What questions do you have about the program?

Will we have structured planning time for UbD?

How do we create essential questions that do not have

prescribed outcomes? Will we continue UbD next school

year?

What types of assistance do you

need to better implement the program?

Workshops for each of the UbD components

Peer study groups Individualized mentoring

Opportunities to observe other classes

What suggestions do you have for

improving the program? More common planning time

Consistent and ongoing schoolwide focus on the initiative

The progress board is completed at the end of each month, individually or in a group meeting. The collaborative dialogue that takes place when the entries are made in a group meeting makes that our preferred option. A portion of a progress board (for January through May) is illustrated in Figure 14.4 . The top two entries in the left-hand column of the board ask the teachers to evaluate the

program over the previous month. The middle three entries call for an assessment of the major effects of the program since its inception, allowing any changes in effects to be tracked from month to month. The bottom two entries request that teachers set goals for the next month aimed at improving the program and identify resources and other types of support they will need to meet those goals. The project board provides for short-term formative evaluations of the program as well as a long-term record of positive and negative aspects of program implementation, effects, and modifications.

Figure 14.4 Portion of a Progress Board

January February March April May

What went well this month

Problems identified this month

Effects thus far on teachers

Effects thus far on school

culture

Effects thus far on students

Improvement goals for next

month

New resources and support needed

Summative Program Evaluation

As we noted earlier, the purpose of summative program evaluation is to decide whether a program will be continued, restructured, or eliminated. The stakes are higher in summative program evaluation, so it tends to be more formal, comprehensive, and rigorous than formative evaluation. A number of key decisions must be made in a summative program evaluation. We examine each of these decisions next.

Who Will Evaluate? Whether the supervisor, a team of faculty members, central office personnel, or private consultants should be responsible for coordinating an evaluation depends on the particular school’s resources and the purpose of the evaluation. However, it is critical that teachers be involved in evaluation of instructional programs. All stakeholders should be not only subjects of the study but co- investigators of the study as well. As Greene (1986) has noted, there exists a “consensus on the need for stakeholder participation” (p. 1), and such

participation is defined as “shared decision making, rather than just advising or providing input” (p. 9).

What Questions Need to Be Answered? Evaluation questions will depend on the nature of the program and what members of the evaluation’s audience wish to learn about the program. Let’s say that a new social studies curriculum is to be evaluated. An implementation question might be, To what extent has the new curriculum been implemented at the classroom level? An outcomes question might be, What changes in students’ knowledge, skills, and attitudes have resulted from the new curriculum? Once the evaluation questions have been formulated, they become the basis for the remainder of the evaluation.

What Data Will Be Gathered and How? Data sources are persons, places, things, events, or processes from which data needed to answer evaluation questions can be gathered. Examples of data sources are students, teachers, principals, families, teaching episodes, student products, and school records. Data-gathering methods are ways to collect data from sources. Examples include testing, observations, content analysis, case studies, review of records, administration of rating scales and surveys, and interviewing. Our bias is toward using multiple sources of data and multiple data-collection methods for each evaluation question. Multiple sources and methods increase the likelihood that the evaluation results will be valid.

How Will the Data Be Analyzed? Data analysis is largely determined by the evaluation questions and types of data. Decisions to be made include how to organize, summarize, and display data and how to reach conclusions based on the data. Central office, university, or private experts may be necessary to assist with complex quantitative or qualitative analysis. Stakeholders, however, can make valuable contributions to data analysis, especially by reviewing results and suggesting explanations, implications, and conclusions.

How Will the Evaluation Be Reported?

After collecting and analyzing the results of tests, observations, surveys, interviews, and testimonials, how should the evaluation be reported? The answer is largely determined by the audience. Most school board members and superintendents will not read a 200-page technical report on the raw data, statistical treatments, and evaluation methodologies. They are interested in the results and conclusions. The technical report should be available to decision makers and referenced in an executive summary. Any reader of the executive summary who is confused or desires more information about certain parts of the paper can check the complete technical report. On the other hand, if the audience for the evaluation report consists of people with sophisticated evaluation skills, a complete technical report would be in order.

Regardless of the audience, certain types of information are included in most evaluation reports. (Again, these components will vary in length and technical sophistication depending on the audience.) Typical evaluation reports include discussions of the following:

1. The purpose of the evaluation 2. A description of the program being evaluated 3. Evaluation questions or objectives 4. Methodology, including data sources, data-gathering methods, and data

analysis methods 5. Results and conclusions, including strengths and weaknesses of the

program 6. Recommendations for the future

As we said earlier, the evaluation questions are the basis for decisions regarding other components of the evaluation. Figure 14.5 provides an evaluation- planning matrix with a column for writing evaluation questions. For each question, there are cells on the matrix for identifying the coordinator, evaluation team, data sources, data-gathering methods, data analysis methods, timeline, and resources needed for addressing the question.

Figure 14.5 Chart for Planning Data Gathering and Analysis

Figure 14.5 Full Alternative Text

Program Evaluation and Teacher Empowerment

David Fetterman and Abraham Wandersman (2007) , leading proponents of empowerment evaluation (EE), define it as “the use of evaluation concepts, techniques, and findings to foster improvement and self-determination” (p. 186). EE has been used in health care, community development, business, government, and education (Fetterman & Wandersman, 2005 ). Donaldson (2017) argues that “one of empowerment evaluation’s most significant contributions to the [evaluation] field has been to improving evaluation use and knowledge utilization” (p. 136). Applied to instructional supervision, EE involves supervisors facilitating teachers as they perform program evaluations. Herein, we describe the purposes of EE, necessary supervisory support, and the EE process.

Purposes

EE seeks to improve the program being evaluated, while also improving the school culture. The premise is that inviting teachers to work together evaluating a program they are part of will improve communication, increase collaboration, foster collegiality, and “ensure that programs are adapted in an ecologically valid and consumer-approved manner” (Brann et al., 2020 ). The EE process seeks to increase teachers’ understanding of program evaluation and their confidence as evaluators. EE also attempts to free teachers from narrow views of their professional roles and help them become more self-directed. A final purpose of EE is to build both individual and school capacity to carry out future program evaluations.

Support

Supervisory support for empowerment evaluation begins with professional development to provide teachers with an understanding of EE and the skills to gather and analyze evaluation data. Teachers need to learn how to develop and

Explain the purpose of empowerment evaluation.Learning Objective 14.6

administer simple classroom observation instruments, surveys, and interviews; how to mine existing data from school archives; and how to perform basic data analysis. The supervisor must also provide teachers with the resources to carry out EE, including time, space, and material resources.

A supervisor cannot simply assign teachers to start doing EE, even after professional development on the EE process. Fetterman (2019) emphasizes the important role a critical friend plays in the EE process. Teachers engaged in program evaluation need technical and moral support in each phase of the evaluation. Teacher-evaluators need technical support in deciding on evaluation questions, choosing or creating data collection instruments, organizing and analyzing data, and planning for change. Teachers also need moral support to gain confidence in performing evaluations, including assurance that their conclusions and recommendations make sense. Most importantly, teachers need to know they will be supported in carrying the evaluation through to completion and implementing program improvement. Nothing can destroy teachers’ commitment to EE as quickly as the supervisor suspending EE to focus on other initiatives or failing to support follow-up for program improvement.

Process

The steps in EE are deceptively simple: (1) decide on the evaluation team’s mission, (2) evaluate the current program, and (3) plan for the future (Fetterman & Wandersman, 2007 ). Underlying the three-step process are a number of principles of EE that not only increase its complexity but also make it a powerful vehicle for both program and teacher development. Principles of EE include improvement, community ownership, inclusion, democratic participation, social justice, community knowledge, evidence-based strategies, capacity building, organizational learning, and accountability (Fetterman & Wandersman, 2005 ).

We take the liberty here of adapting EE’s principles to teachers evaluating instructional programs. One principle is that the school’s professional community should take ownership of the evaluation. The evaluation should include all members of the professional community who wish to be involved,

and all other stakeholders—including students, families, and community members—should be consulted and kept informed throughout the evaluation.

The evaluation should reflect teachers’ knowledge of (1) their students, (2) the students’ cultures, and (3) how the program has been adapted to the local context. The teacher-evaluators, in turn, should seek out other stakeholders’ perspectives on the program. EE is a democratic process characterized by transparency, collaboration, and shared decision making. Decisions are informed by data gathered by the teachers from the school and community and from outside sources like research literature and comparable schools with similar programs.