Leading a Vision for Learning

Murnane, K.P.B.A.C. J. (2020). Data Wise, Revised and Expanded Edition. [VitalSource Bookshelf]. Retrieved from https://bookshelf.vitalsource.com/#/books/9781612505237/

5

EXAMINING INSTRUCTION

Elizabeth A. City, Melissa Kagle, and Mark B. Teoh

FRANKLIN MATH DEPARTMENT HEAD MALLORY GOLDEN BEGAN THE MEETING BY acknowledging the department’s work: “Well, we’ve made a lot of progress so far. We’ve decided that our learner-centered problem is that students are not able to solve multistep problems very well. Now, our next step is to understand why they’re having so much trouble with multistep problems.” “Are we really going to talk about this for another meeting?” interrupted Eddie. “All we

do is talk. Students are going to fail the state test again while we sit around and talk.” “I hear you,” replied Mallory. “But my question is, what’s happening—or not happening—

in our teaching that’s leading our students to struggle with multistep problems?” “Look, it’s not as if we haven’t taught multistep problems,” responded Eddie. “They’re in

every book I’ve used, not to mention on the state test. It would help if kids would do their homework and come prepared to class, but I don’t see that happening anytime soon, so I’ll give them more multistep problems to work on in class.”

Educators are constantly solving problems. These problems range from simple (a student doesn’t have a pencil) to complex (a student doesn’t understand an assignment or two students aren’t getting along). To manage the steady stream of problems, we tend to leap to solutions. However, many of the problems we face are too complicated for us to solve quickly on our own.

The learner-centered problem you have articulated by digging into data is a complicated problem—if it were an

easy one, you would have solved it by now. To solve it, you need to understand its teaching dimensions as well. While many factors outside of school influence children’s learning, these are outside the reach of most teachers. What educators can control is teaching. Teaching, therefore, will be the focus of the action plan. You need a process that allows teachers to take responsibility for solving the problem, instead of backing away from it because they feel it’s not their problem, or that they can’t do anything about it anyway, or that they’re being blamed for it.

To do this, you reframe the learner-centered problem as a “problem of practice” that, if solved, will mean progress toward your larger goals for students.1 The problem of practice is:

Directly related to the learner-centered problem

Based on evidence found when examining instruction

Within the school’s control

A statement about practice, not a question

Specific and small

Not only does identifying the problem of practice lay important groundwork for future action, it also saves time. Even though it may feel like one more thing to do, remember that this investment will likely keep you from spending months or years on something that won’t work because it doesn’t address the actual problem of practice that is at the heart of student learning difficulties.

Four main tasks will help you investigate instruction and articulate a problem of practice:

1. Link learning and teaching: With this particular learner-centered problem, how does instruction have an impact on what students learn?

2. Develop the skill of observing practice: How do we look at instructional data? 3. Develop a shared understanding of effective practice: What does effective instruction for

our learner-centered problem look like and what makes it effective? 4. Analyze current practice: What is actually happening in the classroom in terms of the

learner-centered problem, and how does it relate to our understanding of effective practice?

Because learning and teaching are so intertwined, you may already have partially completed one or more of these tasks when your school was digging into data. While schools that are new to the improvement process may find it easiest to proceed through these tasks in the sequence above, many schools find that examining instruction involves doing them more or less simultaneously. More important than the order in which the tasks are done is the necessity that they all be addressed.

LINK LEARNING AND TEACHING

Mallory continued the math department meeting by asking teachers to use their experience in the classroom to brainstorm why Franklin students were struggling with multistep problems. Teachers wrote their responses on sticky notes:

The first step in articulating a problem of practice is to establish a link between learning and teaching. This may sound surprising, since we presumably wouldn’t be teachers if we didn’t think our efforts mattered for learning. However, in the context of accountability policies and the day-to-day pressures of school, it can be easy to forget that teaching makes a difference in student learning. If teachers don’t fundamentally believe this, then it’s going to be difficult to convince them to change their teaching.

Linking learning and teaching is also about helping teachers take responsibility for student learning. “Responsibility” doesn’t mean “it’s my fault”—it means, “I can and will do something about the learner-centered problem.” Poor test results and external pressures can lead educators to try to shift responsibility to others through finger-pointing and blame. For school-based educators, however, the primary focus has to be on what we have control over—what happens at school. This is not an easy task. Despite their hard work, teachers don’t often see great improvements on state tests, and they don’t think it’s possible to work any harder. They see lots of big issues that affect student learning that they can’t readily fix, like poverty, previous learning experiences, and parental education. The school leader must keep the conversation focused on what teachers can do in the classroom.

What we can do is teach well. To improve the quality of teaching in a school, leaders must push the conversation about the learning problem past the level of what students are and aren’t doing to look at what teachers are and aren’t doing. Additionally, school leaders have to help teachers link learning and teaching in a way that doesn’t make them defensive but does get them thinking about their own practice. When planning opportunities for teachers to link learning and teaching, consider these points:

How will you move the conversation from “students” (or “parents” or “community,” etc.) to “teachers”?

How will you frame the work as an opportunity to improve instruction, rather than as a failure (proactive vs. reactive)?

How will you help teachers have a questioning rather than a defensive stance?

How will you reinforce a collective conviction that teaching matters for learning?

Many school leaders we have worked with use structured protocols to address these questions and make the conversation both safer and closer to instruction for teachers. At Franklin, Mallory used the Affinity Protocol (see Selected Protocols at the end of the book) to have teachers brainstorm hypotheses about the learner-centered problem. After Mallory saw that most of the sticky notes started with “Students,” she encouraged teachers to consider other reasons students might be struggling with multistep problems, and to try starting some ideas with “I” or “Teachers.” Teachers’ responses included:

Although many of their sticky notes still didn’t focus specifically on teachers, when they organized all of their notes into categories, they labeled the categories “Curriculum,” “Instruction,” and “Motivation/Expectations,” and added a “Parking Lot” category for things over which they had no control. The process of working through the protocol helped teachers decide that ideas like “Students aren’t able to think abstractly” should be in the category of “Instruction,” even though they hadn’t thought of it that way originally. The categories, which focus on what teachers are doing rather than on students, reflected the faculty’s evolving understanding of their role in students’ learning. We have seen many schools use the Affinity Protocol with great success because it’s anonymous (thus letting participants write down things they might not say out loud), it levels the speaking field (thus addressing the

potential for one person to dominate or for everyone to wait to see what the principal says), and it’s fun (people appreciate the hands-on experience of using sticky notes and moving them around, rather than sitting and talking).

Other schools use a process of asking “why?” repeatedly to peel away the layers of the learning problem. They start with their learner-centered problem and then ask why they have that problem. For each answer they come up with, they ask why again, and repeat that process several times. One middle school’s “why-why-why” diagram about its special education students who were not answering math questions beyond the first step is displayed in Exhibit 5.1.2

Exhibit 5.1

Why-Why-Why Special Education Subgroup—Mathematics

The initial step of linking learning and teaching need not take a long time— most schools we know devote one meeting to it. However, it lays the groundwork for closer examination of practice by focusing attention squarely on

the instructional core, which is defined as the interaction between teachers, students, and content.3 The data used in the protocols above are teachers’ own experiences, including their assumptions and beliefs. This is a good place to start because this type of data is easily accessible and influences how teachers approach the problem of practice. To truly understand the problem of practice, however, you will have to get closer to what actually happens in classrooms.

DEVELOP THE SKILL OF OBSERVING PRACTICE

At Clark, teachers had spent the previous few weeks doing peer observations, with each teacher visiting one colleague. In team meetings, teachers shared what they had seen. “I enjoyed visiting Anita’s classroom. I saw students who were really engaged,” began

Kristina, a third-grade teacher. “That was true in Jae’s classroom, too,” said Vivian, a fourth-grade teacher. “Students

were well-behaved, they worked well in groups, and they looked like they were really on task.” “Good,” replied Sandy, the principal. “Engagement is important. The next question is,

what evidence did you see that students were engaged, and what task were students engaged in?”

Examining instruction is a complex undertaking. There are many potential sources of evidence to explore, ranging from artifacts (such as assignments or assessments), to self-reports (such as surveys or interviews of teachers), to observations (in person or via video). When weighing this evidence, educators need to be able to recognize,

understand, and describe what they’re seeing. This is not as easy as it sounds, especially when the evidence under consideration is actual classroom practice. Understanding the Key Elements of Observing Practice

Most teachers have not had much experience with examining teaching, which means they have neither the skills to describe teaching in a fine-grained, evidence-based way, nor the collegial culture in which examining practice feels supportive rather than threatening. If you plunge right into examining instruction without developing skills for doing so, you will find the conversation is awash in compliments and generalities like “students are engaged” and “the lesson was well planned.” One principal we know calls this “happy talk”—our tendency as educators to be overly nice to colleagues, especially when we’re just beginning to examine practice. Although it is often helpful to point out what is working well, that level of generality and abstraction does little to help us understand how our teaching links to the learning in the classroom.

Developing skills for examining practice takes time, and dogged determination. In response to consistent feedback from educators who find this work particularly challenging, the Data Wise Project team developed Key Elements of Observing Practice: A Data Wise DVD and Facilitator’s

Guide. This resource offers a set of tools for developing teachers’ capacity to observe practice, including videos, agendas, protocols, and handouts to support professional development on this topic. The main message is that observation of practice is most effective when embedded in a formal protocol. The specific format of the protocol is less important than that it be designed to support an ongoing conversation, based on evidence, about what practice looks like. In particular, we found that effective protocols contain five key elements, and when the purpose of the observation is to identify a problem of practice, the elements involve the following tasks:

Key Elem ent

Tasks Associated with This Element (when the purpose of the observation is to identify a problem of practice)

Focu s

Review the learner-centered problem, provide context for the lesson(s) to be observed, discuss how observers will focus their attention during the observation.

Obse rve

View one or more classrooms, taking notes that capture details about what the teachers and students are saying

and doing and what tasks students are asked to complete.

Debri ef

Discuss the teaching and learning observed using descriptions (not inferences or judgments) and commit to next steps.

Adju st

Carry out the next steps agreed on during debriefing.

Follo w Up

Discuss what was learned during adjustment and plan future work.

When debriefing an observation, it is important that teachers be able to describe what they see using precise, shared vocabulary. For example, a teacher who has not had much practice analyzing instruction might observe a teacher introducing a new math concept and respond with the comment, “I noticed that students seem confused.” “Confused,” like “engaged,” could mean many things to different teachers. In contrast, a more precise observation might be, “I noticed that several students didn’t start the assignment immediately. One student was looking at other students. Two students were talking to each other while looking at the assignment, and four students raised their hands after the teacher gave directions for the assignment.” Cultivating the habit of maintaining a

relentless focus on evidence helps build the precise, shared vocabulary that will allow you to identify a meaningful problem of practice.

Cultivating the habit of intentional collaboration is also essential. It is more powerful to observe and analyze practice collaboratively because each person brings his or her own set of beliefs and assumptions to the observation. Hearing others’ responses to the same lesson helps challenge individual assumptions, allows everyone to notice different things and see the same things in a new way, and leads to a better understanding of the practice observed. For collaboration to be productive and safe, your team will need to adhere to the norms you set when organizing for collaborative work (see chapter 1) and perhaps revisit and expand upon those ground rules for discussion so that discussing teaching openly is possible. An important norm that many teams have is confidentiality, meaning that what is discussed about an observation will not be shared outside the group doing the analysis. Learning to See

To develop the skills of observing practice, you need not jump right into having teachers observe one another. Schools we have worked with often start by having teachers watch videos of teaching from outside the school

(such as the videos available in the Key Elements DVD mentioned above). Observing and discussing the practice of teachers they don’t know gives teachers the opportunity to build the skills necessary for safe and insightful conversations about practice. One principal we know called it “learning to see.” She gave teachers time to watch videos and make note of what they saw. She modeled using language like “I noticed that …,” “I saw that …,” and “I heard that …” with examples of what she saw and heard. With her persistent reminders to cite evidence rather than rush to judgments, teachers developed the habit of doing so, and then used this skill to examine their own practice when they observed each other teach.

We have also seen teams develop these skills when one brave soul, such as a teacher or instructional coach, volunteers to have other teachers observe his teaching. Group members then discuss their observations, practicing using evidence, precise vocabulary, and norms. It can be especially compelling when the school leader volunteers to have teachers observe and discuss her teaching, thus opening his or her own practice to faculty for examination.

This step of “learning to see” instruction focuses on description rather than evaluation. The distinction is important because our normative judgments tend to cloud our ability to see what’s happening—for example, students

look “engaged,” which we think is good, but when we look at what they’re “engaged” with, we see that they’re doing work that’s several grade levels below where they should be. There are limits, though, to what describing can do.

DEVELOP A SHARED UNDERSTANDING OF EFFECTIVE PRACTICE

At Clark, a seventh-grade teacher had volunteered to have colleagues observe her teaching the concept of inference. At the next faculty meeting, teachers discussed what they had seen and were thinking about teaching inference. “The kids were doing inference through drama,” said Vivian, a fourth-grade teacher. “First,

they were taking lines from everyday interactions, like ‘I’m fine’ or ‘Excuse me,’ and they were saying the lines in all different ways, and then the rest of the class was guessing—or inferring—what they were actually thinking when they said it. One student said ‘Excuse me’ like she was really embarrassed, and another said it like he was really annoyed. Then they moved on to reading lines from A Midsummer Night’s Dream, and they worked on saying those lines with meaning that was appropriate, based on the text.”

“It was great, but I hope there are other ways to teach inferring—I’m just not the drama type and I can’t see using Shakespeare with third graders,” Kristina said, referring to her students. “Do we have any other ideas about teaching inference?”

A critical step in articulating a problem of practice is to develop a shared understanding of instruction that will effectively address the learner-centered problem that you have identified. You need a vision for what this effective teaching looks like so you can assess whether what you’re doing now fits or doesn’t fit that vision. The problem of practice is in the gap between current practice and effective practice for addressing the learner-centered problem. If an understanding of effective practice for addressing the problem already existed among your faculty, you probably wouldn’t have the learner-centered problem, as teachers would know how to successfully teach inferring or solving multistep problems, or whatever your learner-centered problem is. Thus, building teacher knowledge is important in articulating the problem of practice. The faculty will continue to develop and refine their understanding of effective practice as they develop the action plan, implement the plan, and assess progress, but it’s important to lay the groundwork at this stage. Without a vision of what’s possible, we limit our expectations and goals for addressing the learner-centered problem, as well as our ability to examine our own practice with an informed eye.

Drawing on Internal Resources

In developing a shared understanding of effective practice, the essential question is, Based on student and teacher data, what does instruction that addresses the learner-centered problem look like? To answer this question, schools can look internally at instruction in their own buildings and look externally to other practitioners and to research. Looking internally has advantages: it honors the work of teachers in the building, which can build a sense of confidence and competence, and the practice is very specific to the context of the school, thus requiring less translation or adaptation than an external practice might. Looking internally also has potential disadvantages: the scope of ideas may be narrow because we don’t have all the answers; singling out particular teachers as examples of “effective practice” may promote a sense of competition or comparison among teachers; and our assumptions about what’s possible may be limited. A school that chooses to use examples of effective practice from inside the school can mitigate these pitfalls in several ways. First, introduce a wider range of practices by bringing in outside resources, such as journal articles that address the learner-centered problem. Second, involve the whole staff in identifying effective practice to cut down on feelings of competition.

And third, facilitate conversations carefully to get teachers thinking more widely about practice.

When looking internally to develop ideas of effective practice, the key is to ground the discussion in evidence. We know many schools where teachers share instructional strategies that they call “best practices.” This is an important way to develop craft knowledge, but is often disconnected from data and from the learner-centered problem. Some schools we work with address this by inviting teachers to share practices related to the learner-centered problem and to support the belief that these are “best” practices with evidence of student learning. In these schools, teachers use the following protocol: “This is a ‘best practice’ because when I did this, the learning looked like this.” They then present their instructional strategy and evidence from student work. Connecting best practices to data serves multiple purposes: it increases the likelihood that the practice is effective rather than simply congenial; it reinforces the discipline of grounding all conversations about teaching and learning in evidence rather than generalities or assumptions; it’s more persuasive—teachers are more likely to try something for which there’s evidence that it works; and it reinforces the link between learning and teaching. Drawing on External Resources

At Franklin, Sasha, the math coach, showed the math department videos of algebra lessons from Japan, Germany, and the United States from the TIMSS (Trends in International Math and Science Study). “Well, sure, it would be great if our lessons looked like the Japanese one, but our

classrooms are really different,” Eddie said. “First of all, we’re dealing with a much more diverse population than those Japanese teachers are. Second, I’ve tried that approach of putting a problem on the board and having kids try to solve it, and it just doesn’t work. Some of them don’t even start it, and just talk to their neighbors. Others start it, and then stop as soon as they’re stuck. If I’m not helping them through each step of the problem, they just give up.”

“What do you see in these videos that relates to the article that I gave you about differences in teaching practice across countries at different performance levels on international math tests?” asked Sasha.

“The article said that countries in the middle of the pack, like the U.S. and Germany, tend to spend lots of class time on reviewing homework and applying formulas. We definitely saw that in the video,” said Mallory.

“The article also said that the countries that score higher, like Japan, have students develop some understanding of the theory behind an operation, rather than just applying an algorithm. And there was the piece in the article about breadth versus depth—U.S. textbooks are much fatter than Japanese and Korean textbooks and cover a lot more concepts. We saw some of both of those things. In the U.S. classroom, students were doing many problems that drew on a number of different math concepts, whereas in the Japanese classroom, students did two problems in the class period, both on the same math concepts.”

“Okay,” said Eddie. “Like I said, I’d love to teach just like those Japanese teachers. But how would this work at Franklin?”

In schools, internal resources often are not adequate for developing a shared understanding of effective practice related to the learner-centered problem. This is when school leaders should use external resources to seed the conversation about effective practice and build teachers’ knowledge. You can go to the source, by visiting another school or attending a professional conference, or you can bring it in, by learning from consultants or reviewing research. Looking externally brings its own advantages and disadvantages. Some advantages include bringing in a range of ideas and expertise that is beyond that of your staff, and making it easier to have an “objective” conversation. Many school leaders we know bring in articles and videos to start conversations with their faculty, which helps teachers get perspective on their own practice and talk about effective practice. External resources also can challenge assumptions about what’s possible by showing evidence of other practitioners succeeding where we have not; providing access to ideas that have potentially been tested more systematically or for a longer period of

time (e.g., research); and helping triangulate teachers’ hunches and experiences of good practice with external ideas.

Drawing on outside resources also brings potential disadvantages. For some teachers, external resources challenge their professionalism and suggest that they’re not good teachers or don’t know what good teaching is. There is also the “But they’re different! That wouldn’t work here!” problem, as math teacher Eddie exemplified at Franklin. In this view, educators assume that whatever success the external resource had won’t hold when applied to their own unique setting. With external resources, it’s not always clear which elements are essential and which can be adapted (or how) to another context.

Adopting an inquiry stance, in which any resource—internal or external—is questioned and investigated, addresses these disadvantages. Inquiry is essential in developing a shared understanding of effective practice because you want everyone to understand not only what effective practice for the learner-centered problem looks like but why it is effective. In the Clark example about using Shakespeare to teach inferring, teachers need to move beyond simply

deciding that drama is a good tool for teaching inferring to inquire why drama seems to be effective. Is it because students have the opportunity to use nontext clues like body language and intonation to interpret text? Or perhaps because the Shakespearean text is so difficult that students have to figure out ways to make sense of it? Or is it because of the kinds of questions the teacher asked? Similarly, in the Franklin example, is the Japanese math teacher’s practice effective because he is helping students understand theory instead of applying an algorithm, because he allows students to solve the problem in different ways, or because he is going into more depth and covering less content? As teachers ask and answer these questions, they’ll form a vision of effective practice that they can adapt to fit their own setting. This depth of understanding will support both their examination of practice and their implementation of it later in the improvement process.

Some schools form inquiry groups in which teachers frame a question about teaching related to their learner-centered problem and then investigate resources to help them answer the question. At Clark, teachers inquired about Think Alouds, the process of teachers making their own thinking explicit for students as they modeled a strategy or skill, such as inferring.4 In one high school, teachers wanted to know how to help their students who were several years below grade level in reading catch up. They particularly wanted strategies that were both effective and age-appropriate for their students, and no one on the faculty knew what to do. The teachers formed an inquiry group in which they found, read, and discussed several articles and books on the subject. After a lengthy discussion, they generated a list of effective practices for helping students significantly behind in reading.

It often helps to draw on both internal and external resources. Some people are more persuaded by research, others by a colleague’s success, while some people have to see the effectiveness of a practice to both believe and understand it. Very often, we need a combination of research, practices, and experience to develop our understanding of effective practice. At one elementary school, the faculty thought that differentiated instruction might be a good strategy for meeting needs of various

learners, but didn’t know what it would look like. They read articles, watched videos, listened to their special education teachers describe their practice, and discussed all of the ideas to develop a common understanding of differentiated instruction as it applied in their particular context.

ANALYZE CURRENT PRACTICE

At Franklin, the math teachers had decided to focus their next faculty meeting on exploring two questions generated by both their brainstorm about why students were struggling with multistep problems and their investigation into effective practice: (1) Are we teaching a consistent strategy for solving multistep problems? (2) Do we give students sufficient practice with multistep problems? Now they had to decide how they were going to answer those questions. Sasha, the math coach, suggested that they observe each other teaching to get a sense

of what was being done in different classrooms. Eddie disagreed. “I’m not giving up my planning period to observe people teaching. I need

my planning time.” Mallory countered, “Well, Roger might be able to get subs to cover so we can see each

other.” “I don’t want to miss teaching a class so that I can go observe other people, either. I’m

behind where I need to be with the kids as it is. Besides, do we really need to see each other teach to answer these questions? I think it’s safe to say we’re not teaching a consistent strategy for solving multistep problems—I have no idea how anyone else teaches them. Do we give students sufficient practice? Well, I don’t know what ‘sufficient’ is, but apparently it’s not enough or they’d be doing better. And we tell students they can do the problems if they’ll try.”

To articulate a problem of practice, faculty must be able to describe what is going on currently in the school, with a shared understanding of effective practice as the referent point. Gathering data about teaching to examine what’s happening in classrooms helps move the conversation away from an emotional blame game and toward identifying the teaching dimensions of a problem of practice. Just as we can think students have learned something until we look at their work and see they haven’t, as teachers, so can we think we’ve taught something until we look at our work and see otherwise.

As with any improvement effort, when examining instruction, school leaders face many decisions about what data to look at and how to examine them. These decisions come with trade-offs and depend heavily on the context of particular schools. The data that may help educators at one school understand their teaching practice as it relates to their learner-centered problem may not help another school at all, or may take so long to collect and analyze that the energy for improvement dwindles and no action is ever taken. Consider these three questions when making decisions about how to examine instruction:

What data will answer your questions about teaching practice in your school?

What are teachers ready for and willing to do?

What are your resources, including time?

What Data Will Answer Your Questions About Teaching Practice in Your School?

As discussed in chapter 3, decisions about what data to look at should start with questions. In the case of teaching practice, it’s critical to frame what questions you have about what’s happening in classrooms in terms of the learner-centered problem. If you don’t, you’re likely to collect lots of data that don’t help you. This can waste precious time, as well as goodwill about engaging in the improvement process, particularly around something as sensitive as teaching practice. The questions should draw on your inquiry up to this point, including your understanding of effective practice. We know one school that approaches describing current practice as an instructional audit. The faculty develops a list of questions related to things they would expect to see in classrooms effectively teaching to the learner-centered problem, and then observe each other teaching, taking notes on the answers to those questions. We have also seen schools use the Question Formulation Technique mentioned in chapter 3, and we have seen schools fill out a simple form, as shown in Exhibit 5.2.

Exhibit 5.2

Exploring the Learner-Centered Problem

The questions will determine what the relevant data are. For example, if the prevailing chorus from teachers is, “I’m teaching it—they’re just not learning it,” a question might be whether teachers are actually teaching whatever “it” is. The data needed to answer this question will most likely involve seeing teachers in action. If the question is what kinds of tasks we ask students to do related to the learner-centered problem, the data involved might be homework and classwork assignments, tests, and other assessments. If the question is about what teachers think they’re doing, helpful data would come from asking teachers to describe their practice through surveys, focus groups, or interviews.

One elementary school investigating problem solving in math met in cross-grade groups to try to understand their

lowest-performing students’ experience of a math curriculum that relied heavily on having students construct their own meaning of math concepts. Teachers wondered if more structure was needed in the lessons for the most struggling students. If so, what was the best method, what would it look like, and when should they introduce it? After sharing lesson plans and student work, teachers decided to observe in each other’s classrooms to get more data on the classroom performance of this neediest group of students and the instruction in the classrooms. Each teacher arranged to observe a peer before the next meeting. They then shared their observations to determine patterns of response and brainstorm the kinds of support they felt their students needed, based on the observation data they’d collected from their own and their peers’ students.

Another school devoted several staff meetings to something called triads. The principal divided the staff into groups of three, mixing up specialists and teachers of different grade levels. Each triad came up with a focused question it wanted to investigate, such as a question about teaching inference in reading. Teachers were then released from team meetings in order to observe their colleagues and collect data in response to the question posed by their colleagues. At the next meeting, faculty met

in their triads to share observation data and reflect on what they’d learned.

What Are Teachers Ready for and Willing to Do?

The next step is to figure out what sorts of data teachers are ready and willing to examine about their own practice. Two helpful questions to consider: Are teachers accustomed to being in each other’s classrooms, having people watch them teach, and discussing their practice? Is there a culture of inquiry where talking about teaching practice is seen as an opportunity to improve rather than an evaluation? If the answer to these questions is yes, then all data about teaching practice is a possibility, including data that come from directly observing teaching, like video and class visits. If the answer is no, then direct observation may be too threatening to teachers and they may refuse to do it, or be so general in their observations that the data aren’t helpful. In this case, data that rely on self-reports from teachers, such as surveys, focus groups, or interviews, can be less threatening, provide information, and begin to get teachers accustomed to the idea of examining their practice. If the answer is somewhere in between, then examining artifacts of teaching such as student or teacher work may be appropriate.

At an elementary school where math scores on the state test were low and teachers weren’t comfortable

having someone watch them teach, the instructional leadership team designed a survey based on their questions about what was happening with math. The teachers on the team made a special point of distributing and collecting the survey themselves without involvement from the principal, because they hoped teachers would answer more honestly if the questions came from other teachers. The team learned several important things when they analyzed the results from the survey: teachers weren’t spending the 60 minutes per day they were supposed to on math, teachers generally didn’t use the hands-on materials that were part of the curriculum, and teachers wanted more support in how to teach math well.

In many schools, custom or contract will dictate the extent of and the guidelines for classroom observation. We have found that schools that use questions and data to establish a culture of inquiry in which teachers are participants rather than targets find ways to examine practice. Teachers want to know the answers to the questions they have generated, and thus are more likely to participate in collecting the data and examining their own practice.

What Are Your Resources, Including Time?

Finally, resources—especially time—play a role in determining what data you decide to examine. Again, two

questions: How much time do you have to collect and analyze the data about instruction? What other resources are available to you? The answers to these questions influence how many data sources you can examine and which ones you choose. If, like Franklin, you are under intense pressure to take action and you have only a couple of meetings available in which to examine instruction, you might choose to look at artifacts of teaching and hear self-reports through a focus group. If, like Clark, you have less urgency and schedule more regular meetings, you might choose a more lengthy process, such as teacher and administrator observations in classrooms. Other resources will influence your choices as well. Do you have someone who can collect the data and do an initial analysis? If not, you may not want to do a survey, which is quick for teachers to complete but takes time to compile and analyze. Do you have a way to free teachers to visit each other’s classrooms? Some schools use combinations of substitute teachers, administrators, and teachers who have a planning period to cover for teachers who are visiting colleagues’ classrooms. Other schools have limited resources in terms of time and people, but use technology to support their data efforts. We’ve seen schools use electronic survey and polling tools to do a survey of their teachers (and their students, too).

If you are a school leader, you are an important resource for examining instruction because you have the distinct advantages of flexibility and time that allow for examination of practice across the whole school. Though it may not seem that you can get into classrooms as much as you would like, you do not have the responsibility of being in a single classroom in fixed periods for most of the day. Thus, you have the opportunity to visit multiple classrooms, attend different grade-level and content-area meetings, and get a sense of the bigger picture of what’s happening across classrooms.

Trade-Offs

With all of these decisions there are trade-offs. If you examine instruction more quickly with limited data sources, you will get to designing and implementing solutions faster, but you may sacrifice some accuracy in framing the problem of practice. If you take your time and examine several data sources, you may be more accurate, but you may lose a sense of urgency and momentum for improvement. If you are an administrator, you might have a more flexible schedule but teachers might worry that you will use some of the observation data in their final evaluation. If you tread carefully and don’t push too hard on teachers’ comfort level about examining practice, you may get willing participants, but not the level of precision

and depth you want in the problem of practice. Push too hard, and you may get resistance when you go to implement solutions. If you have a few people do most of the work of examining instruction, it may be done more quickly and at greater depth, but you may not get the level of understanding and buy-in you’ll want from the rest of the teachers whose practice you ultimately want to improve. As a leader, you have to balance your questions, your teachers’ skills and readiness, and your resources with the goal of answering your questions as thoroughly as possible in the time frame you have.

Articulating the Problem of Practice

Once you have linked learning and teaching, developed the skill of examining practice, developed a shared understanding of effective practice, and analyzed current practice in your school, you can articulate the problem of practice that will be the focus of your improvement efforts. At Clark, the problem of practice is “As teachers, we do not teach inference explicitly, and do not help students make connections between the inferences they make in their lives and the inferences they need to make from texts.” At Franklin, the problem of practice is “As teachers, we do not consistently teach a process for solving multistep problems, and we don’t give students enough opportunities to work with multistep problems.”

After articulating your problem of practice, like the teams at Clark and Franklin, your team will be ready to design an action plan to address it.

Integrating the ACE Habits of Mind into Step 5: Examining Instruction

SHARED COMMITMENT TO ACTION, ASSESSMENT, AND ADJUSTMENT

The main deliverable that you produce in Step 5 is your problem of practice statement. For the improvement process to be effective, it is important that you phrase this problem in a way that sets you up for—but does not lock you into—action. One thing that we have found to be effective is encouraging educators to start the problem of practice with the phrase, “As teachers, we ….” This helps ensure that the problem is truly focused on teachers, and that teachers are taking ownership of it. Another caveat is to be sure that you haven’t snuck a solution into your statement. Doing this can be tempting, but it means you are skipping the important conversation about choosing instructional strategies that is described in chapter 6.

When a team tells us that their problem of practice is “As teachers, we do not showcase student work on the bulletin board outside our classrooms,” we ask them to pause. Is this really the problem of practice, or is it an idea for how to solve a more fundamental problem? Rephrasing the problem—“As teachers, we do not provide incentives for students to do their best work”—gets at a much deeper issue. And it opens the door for much more creative instructional strategies than the one they jumped to in their original statement.

INTENTIONAL COLLABORATION

Observing and discussing instruction is difficult because it goes against longstanding practices in education in which teachers work behind closed doors, with observers in their classrooms only for evaluative purposes. As mentioned in this chapter, one way to lessen anxiety about observations is to begin by looking at videos of instruction, rather than visiting classrooms directly, so teachers can simply practice taking observation notes and describing what they see. (The Data Wise Project website provides links to publicly available websites containing example videos of instruction.) Practicing with such videos allows educators

to develop the necessary skills for observing instruction in a relatively neutral context. We have found that some teachers prefer to start examining instruction at their school by videotaping their own lesson, and then choosing what part of the video to show and how it is used. A teacher can watch the video alone and pick a five- or ten-minute segment to show others, along with specific questions to ask the group. Another suggestion is to be thoughtful about who is doing the videotaping. Encourage teachers to work with another teacher they trust, or to videotape the lesson themselves using a tripod or setting the camera at a good angle to capture the whole room. It is also worth being thoughtful about what is recorded. Many discussions after the observation focus on what the students were doing, so ideally the video would capture the students and not be focused exclusively on the teacher. Some teachers may find this helps remove the impression that video puts them personally under the microscope. Intentional collaboration is all about building trust. In our experience, having colleagues give one another rich, descriptive information on their practice can be a more powerful strategy for building trust than any team-building activity done outside the context of teaching.

RELENTLESS FOCUS ON EVIDENCE

There are a few ways to support colleagues in engaging in a productive conversation that is grounded in evidence from the classroom observation. First, ensuring the effectiveness of a conversation after observing practice actually begins with thinking about the conversation that occurs before entering the classroom. In that conversation (which we refer to as a focus meeting in the Key Elements of Observing Practice DVD and Facilitator’s Guide), the host teacher actively enlists colleagues in naming what evidence to collect from the observation that will help shed light on his or her practice. Then, when it is time for the debriefmeeting, the objective is for colleagues to share the evidence that they were asked to collect.

Another effective way of sorting through the evidence is to use the four-quadrant diagram shown in Exhibit 5.3 during the Affinity Protocol. We have found that using this diagram and the Affinity Protocol (for example, by sorting each sticky note into the quadrant that best matches the type of evidence on the note) keeps teachers to specific, non-normative statements, “depersonalizes” the observation data, and allows teams to make sense of

what they see. The diagram provides guidance about what type of evidence to collect during the observation—as much as possible, evidence should be both descriptive and specific, instead of general and judgmental. It will take practice before most educators are good at writing down specific, descriptive statements from an observation of instruction—but in the end, the practice will pay off when you have rich, detailed data from each observation.

Exhibit 5.3 Objectivity/Specificity Matrix

6

DEVELOPING AN ACTION PLAN

Tom Buffett, Mark B. Teoh, and Gerardo Martinez

CLARK’S THIRD- AND FOURTH-GRADE TEAM LEADER, ANITA SUAREZ, FOUND IT hard to interrupt her colleagues’ animated conversation. “I used that current events activity that we read about last week,” Vivian was saying. “The kids loved it. I often have a hard time getting them interested in a nonfiction reading exercise, but this time they were really into it.” “Well, I tried that ‘Think Aloud’ thing we saw you do in your classroom,” Jae told

Vivian. “You know, when you projected that poem on the screen and then you walked the kids through what was going on in your head as you read? I don’t know how you made it look so easy … I could only keep it going for a few minutes before it kind of started feeling like a monologue.”

It was encouraging to see the teachers trying out some of the new ideas that had come out of their work together. But so far, the efforts were individualized and uncoordinated. As leader of her team, Anita knew that she would be responsible for helping them come up with a coherent plan for making instructional changes together. But she hadn’t led this kind of work before, and she wondered how she could channel her team’s energy into getting a formal plan down on paper.

The third- and fourth-grade team at Clark K–8 School has deepened its knowledge about effective reading instruction by observing team members teach, discussing professional literature on this subject, and watching and debriefing a video related to literacy instruction. In the process, the team has identified some promising strategies for helping students learn how to make text-based inferences. Like Anita at Clark K–8 School, you may find that teachers at your school begin to

improve their instruction as a result of participating in these kinds of activities.

Nevertheless, explicitly committing to a particular strategy or set of strategies for instructional improvement and writing up a formal action plan is important. Creating an action plan will increase the clarity and transparency of your work. You can use the document for communication within your school as well as between your school and key external constituents such as families, the district office, and partner organizations. By committing your team’s thinking to paper, you create a process through which team members can raise and address the different understandings that naturally develop when discussing practice. And perhaps most important, action planning is a way to translate what you learn through analyzing a broad swath of data—from state test scores to current instructional practices—into concrete strategies for improving what is happening in classrooms.

Successful action planning typically includes the following four tasks:

1. Decide on an instructional strategy or strategies that will solve the problem of practice you identified through your analysis of student and

teacher data. The instructional strategy your team commits to is the heart of the action plan.

2. Agree on what your plan will look like in classrooms. Your team can reach a shared understanding of the strategy by carefully describing what team members would expect to see teachers and students doing if the plan were implemented well.

3. Put the plan in writing. By documenting team members’ roles and responsibilities and specifying the concrete steps that need to occur, you build internal accountability for making the plan work. Identifying the professional development your team will need and including it in your action plan lets teachers know they will be supported throughout the process of instructional improvement.

4. Decide how you will know if the plan is working. Before implementing your plan, it is important to determine what type of student outcome data you will need to collect in order to understand whether students are indeed learning more. Because this task represents a substantial undertaking that is often overlooked, we discuss it as the sole topic of chapter 7, “Planning to Assess Progress.”

Working together to create an instructional action plan that you will collaboratively implement can help build a professional community at your school. But it isn’t easy. This chapter highlights key aspects of each of the first three action-planning tasks listed above and identifies some of the opportunities and tensions that may arise during this phase of the improvement process.

DECIDE ON AN INSTRUCTIONAL STRATEGY

Solutions at last! If you are new to the improvement process recommended by this book, it may seem as though you have had to work through quite a number of steps before turning to the task of deciding what to do about the student learning difficulties you unearthed in data analysis. Now that you have reached this task, however, it is worth the time to deliberately consider possible solutions by first clarifying the scope of your plan, then brainstorming a list of ideas, and, finally, deciding which of the ideas makes the most sense to implement. Clarify the Scope of Your Plan

The scope of the plan your team develops will depend on the unit of improvement the team seeks to address. The term “unit” in this case refers to the group that is the subject of your proposed improvement, whether that is a content area, a grade level, the whole building, or some

other grouping at your school. Which unit you choose depends on the scope of the problem of practice. When the data show that the problem of practice is consistent across the school, a schoolwide solution makes sense. On the other hand, when the data indicate that the problem of practice is particular to a content area, grade level, or group of students and teachers, a more directed solution will be appropriate. Note, however, that if the unit of improvement you select is smaller than the whole school, your action plan must fit within the context of your schoolwide strategy for improvement. In other words, there should be a connection between the smaller unit’s action plan and your school’s overall approach to improvement so that any success you achieve also serves a larger schoolwide goal.1

At Clark K–8 School, the principal asked each grade-level team to create its own action plan that would support the schoolwide goal of improving student reading performance. At Franklin High School, the principal asked the entire math department to work together to develop a plan that would be implemented in all math classes across all grade levels. The instructional leadership team at one school we have worked with identified the problem of practice as the fact that teachers weren’t adequately supporting students in becoming critical thinkers and

independent learners. Rather than choose schoolwide strategies that would apply across all classrooms, they had the literacy and math teams prepare their own action plans. Each plan was content-specific, yet addressed the schoolwide goal of teaching students to be critical thinkers and independent learners.

School leaders should also consider other factors, such as the availability of resources and faculty capacity, when determining the appropriate scope for their action plans. Some schools choose to do a schoolwide plan because they do not have the resources to support multiple teams, each pursuing a different plan. Other schools conduct a pilot action plan with a single grade-level or content-area team to test the plan before investing in bringing it to the whole school. Still others choose the whole school as the unit of improvement precisely because they want to send a clear signal that they are all responsible for all children, and their collective problem of practice is not just the concern of the willing.

School leaders must weigh all of these considerations. As you think about the scope of action planning in the context of your school, be aware of the potential tension involved in identifying a strategy that is both broad enough to be relevant to teachers who teach different content or students in different grades, and specific

enough to ground instructional conversations and improvement efforts in concrete classroom practices. Brainstorm Solutions to the Problem

Even schools we know that are very skilled at examining data and identifying the problem of practice can get stuck at the point of figuring out how to solve the problem. As one principal said to us, “Now what do I do? We know what the problem is, but if we knew what to do about it, we’d have done it already.” Looking for solutions can involve engaging teachers in a conversation about how to address a problem, identifying and making creative use of in-house expertise, and being honest about when it is time to seek guidance from outside sources. It is also an opportunity to give teachers time and space to create something collaboratively that is larger and more powerful than anything they could do individually. Developing a shared understanding of effective practice among your faculty, as described in chapter 5, will help generate ideas about appropriate solutions to your problem of practice.

As with all steps of the improvement process, generating solutions is both an end and a means—it is important to come up with the good solutions for your problem, but how you come up with those solutions matters, too. Some schools choose to spend more time

than others in developing solutions in order to help faculty “buy in” to those solutions. After all, teachers are the ones who will be implementing the solutions, so it is essential that they understand them and appreciate their potential to improve student learning.

There are a number of approaches that you can use to identify possible solutions. The simplest approach is to assemble a group and ask people to brainstorm ideas, which you can keep track of on poster paper or projected on a screen. You can also use a variation of the Affinity Protocol discussed in the previous chapter, where individuals use sticky notes to capture their proposed solutions and then work together to group them into logical categories. One school leader we know used a Café Protocol in which faculty discussed possible solutions in small groups at a number of tables in the library.2 The protocol called for teachers to switch groups twice, each time bringing the solutions from the previous conversation to their new group. At the end of the protocol, the faculty compiled the list of suggestions their conversations had generated. Although the protocol took a full hour, the conversations went much deeper into discussing possible solutions than a brainstorming session would have and allowed faculty to have extended, substantive conversations with one another, a welcome

change of pace from their usual focus on protocols with tight time limits.

A few days after the November math department meeting, department chair Mallory Golden met with Sasha Chang, the district math coach assigned to Franklin High School. “So,” Mallory explained, “we talked about designing a formal process for solving problems, and teachers got really excited thinking about how we could ask students in all classes to create posters demonstrating their use of this process in solving complex problems. People seemed pretty enthusiastic about teaching a consistent method across classes, and we are going to talk about it more at our next meeting.” Mallory paused for a moment. “Frankly,” she continued, “I have to say I’m a bit worried we’re going to get bogged down in agreeing on the wording of the method and we’ll never quite get to using it with the kids.”

Like Mallory, many school leaders find that after taking the first important steps toward defining a solution, their faculty may need some guidance about how to implement it.

Exhibit 6.1

Problem-Solving Approach

After hearing Mallory’s concern, the math coach showed her a diagram of a process called the Problem-Solving Approach adapted from George Polya’s book, How to Solve It (see Exhibit 6.1).3 The coach, Sasha, had used

this particular approach successfully at other schools. Mallory felt it was just what the team needed to make progress on defining their solution, so she asked Sasha to bring her diagram of the Problem-Solving Approach to the next department meeting and talk about how she had used it with other schools. When Sasha did this, the faculty was quite interested in the approach and by the end of the meeting had decided to adopt it instead of spending any more time trying to reinvent the proverbial wheel.

It is doubtful that the teachers at Franklin High would have embraced the Problem-Solving Approach if someone from the district’s central office had mandated it at the beginning of the school year. But because the teachers learned about the approach after they had identified the problem of practice and begun to look for solutions, they received the idea enthusiastically. Many school leaders find that a successful search for solutions involves allowing faculty to offer their own suggestions while at the same time planting good ideas from outside sources.

Select a Solution to Implement

Anita posted the list of teaching strategies the Clark third- and fourth-grade team had brainstormed at their previous meeting and told the team they had to decide

what solutions they were going to implement. “It looks like a great list to me,” said Jae. “I think we should do all of them!” “I don’t know about that,” Kristina countered. “That list looks overwhelming to me. Why don’t we just do one thing well—maybe something straightforward, like working on vocabulary words related to inferring?”

Successful teams are clear about why they select particular strategies. Two important criteria for selecting strategies are the feasibility of implementing the strategy and its likely impact. The feasibility of a particular approach depends on the availability of resources. Commonly required resources for implementing improvement strategies include professional development materials (such as videos relating to specific content areas or instructional techniques), support (including workshops and internal or external people who can teach and provide ongoing guidance about the strategy), and time (which can involve deciding how much of the team’s precious collaborative time to devote to this effort). The feasibility of any approach also depends on teachers’ existing skills and capacities. The degree of trust that exists among the faculty members is also important. Simply put, some of the most promising strategies for instructional improvement are feasible only if the faculty members trust each other enough to open their classrooms and learn from one another.

Teams can assess the likely impact of potential strategies in a variety of ways, including reviewing academic research and examining student performance

data from other places that are already implementing the strategy in question. As a practical matter, however, teams most often rely on their members’ professional knowledge and experience. One useful way of thinking about the impact of a particular strategy is to challenge your faculty to identify “high-leverage” solutions. High-leverage solutions are those that, because of their intensity or the sheer number of students they affect, are likely to make the biggest difference in what children learn.

Clark’s third- and fourth-grade team assessed both the feasibility and potential impact of their potential solutions on the chart shown in Exhibit 6.2. Though teachers agreed that the new curriculum solution might have the greatest potential impact among all the items on the chart, the team concluded that this option was not feasible because there were not enough funds in the school’s budget. Instead, Clark K–8 School selected from among the strategies that were both feasible and likely to have an impact, and they decided to adopt the strategy of using regular Think Alouds as part of their reading instruction.

Exhibit 6.2

Clark K–8 School Assessment of Inference Strategies

At schools that have not developed a collaborative orientation toward continuous instructional improvement, teachers will often identify “solutions” that do not involve a change in their instruction. For example, it is common for teachers at schools that lack such a professional community to propose an afterschool program as the key strategy for addressing student weaknesses. While we have seen schools use afterschool programs very effectively as part of their improvement strategy, such programs are effective only in combination with—not in lieu of—instructional improvements during the school day. It is easy to achieve consensus on solutions that do not require teachers to make changes in their day-to-day practice, even when data show that such practices are consistently ineffective. As a school leader, when it comes time to choose a solution from your brainstormed list, you may need to remind teachers that the goal is a solution that changes how students learn in their

classrooms. You may find that the more involved teachers are in selecting the solution, the more committed they will be to doing the hard work of implementing it.

A final component of choosing an instructional strategy is for school leaders to determine how much change to tackle in the action plan. While some schools will not take on enough—like schools that decide an afterschool program will solve all their problems—most schools actually attempt too much in their action plans. While such enthusiasm is laudable, it is also dangerous. We have seen too many schools come up with a fine list of strategies in their action plan and then not implement any of them well because they are stretching their resources too thin and pushing beyond teachers’ capacity to learn and apply new instructional approaches. Under these conditions, most teachers ultimately retreat to their previous practices, which the data showed were not achieving the desired student learning outcomes. School leaders need to be sure that the strategies selected not only are focused on day-to-day instruction in classrooms and are likely to solve the problem of practice, but also can be reasonably implemented in the action plan’s time frame. Listening to both the “we can do it all” teachers (who are always easier to listen to) and the “we can’t do this” teachers (who are always more tempting to ignore)

is important in gauging what strategies will effectively and manageably accomplish important changes in practice.

AGREE ON WHAT YOUR PLAN WILL LOOK LIKE IN CLASSROOMS

Anita was concerned. After talking informally with a number of third- and fourth-grade teachers, she began to realize that although teachers had expressed a lot of enthusiasm for using Think Alouds in their reading instruction, each individual seemed to have his or her own concept of what this would actually look like in the classroom.

How your team defines an instructional strategy to address weaknesses in students’ skills is a critical part of implementing that strategy. A faculty gains much of its understanding about what the strategy is and why it is the focus of the action plan in the process of selecting the strategy. However, even if your team is faced with the challenge of implementing an instructional strategy that was determined unilaterally (perhaps by district mandate), your faculty can still deepen their understanding and appreciation of the strategy by participating in a discussion of what it actually looks like in practice. We have found that it is important for faculty discussions on this topic to accomplish two things: develop a common vision for implementing the strategy and lay out a theory of how the strategy will affect learning.

Develop a Common Vision for Implementation

It is often not enough to select an instructional strategy by name because it is too easy for people to attach different meanings to commonly used names or words. For example, if you select a strategy that calls for peer editing of draft writing assignments, teachers may have a variety of ideas about what this means. After deciding on an improvement strategy, successful teams develop a shared understanding of what effective instruction looks like by establishing implementation indicators. These

include descriptions of what teachers will be doing, what students will be doing, and what the classroom environment will look like when the instructional strategy is in place. The process of creating these indicators allows teams to develop the capacity to have increasingly specific instructional conversations. It also facilitates implementation by making the team’s expectations clear. After two team meetings at Clark K–8 School, the third- and fourth-grade teachers produced the list of implementation indicators for their Think Aloud strategy shown in Exhibit 6.3.

Exhibit 6.3

Implementation Indicators

CLARK K–8 SCHOOL Implementing Think Alouds About

Inferring: What We Will See in Classrooms

Teache rs

• Model inferring by walking students through their own thought processes as they read a piece of text

• Project a text on a screen and mark it up while thinking aloud about inferring

• Conference with students, asking them to think aloud while reading

• Move throughout room to monitor students’ inferring

Studen ts

• Know what the word “infer” means and use it accurately to describe their reading process

• Use Think Aloud strategies to infer in whole-class, small-group, and conference discussions of text

• Answer questions that show their ability to infer

Classr ooms

• Display evidence of students inferring from text

• Include class-generated poster of what good readers do when they infer (for students to reference while reading)

• Allow flexible arrangement of furniture to enable paired reading, group activities, and sustained independent work

Studen t Work

• Demonstrates students’ ability to infer, predict, connect, and evaluate in their reading journals and other reading assessments

Lay Out the Theory Behind Your Solution

The Franklin High School math department had invited principal Roger Bolton to join their December meeting. They described with enthusiasm the Problem-Solving Approach and how they planned to implement it across all math classes. “You’ve got yourself a plan here, and it feels like a good one,” said Roger. “Now, can you walk me through how teaching this approach is going to lead our students to become stronger mathematical thinkers?”

Roger’s question is deceptively powerful. At this stage in the game, it can be very helpful to push your team members to put into words exactly how they believe their plan is going to achieve its desired effect. Chances are, there are a number of ways the plan could produce positive results, and not all teachers agree on the theory behind the solution. For example, do teachers believe that the Problem-Solving Approach will lead students to become more independent problem solvers? Will it increase students’ confidence and make them more willing to try problems that they already have the math knowledge to solve but have been too disorganized to tackle? Will it make them pay more attention in math class, thus getting more out of lessons they might ordinarily have tuned out and making them more capable of solving problems themselves? Will it help teachers give students responsibility for deciding what to do rather than guide them through problems step by step? Alternatively, is the real strength of this strategy that students will be able to use the same approach from class to class and from year to year, thus allowing them to build experience with problem solving so that it eventually becomes second nature? When pressing teachers to articulate how they think the plan will work, it is also worth asking them if they can imagine any unintended negative consequences. For example, will requiring students to use the Problem-Solving Approach squelch the

creativity of real mathematical thinkers by reducing math to following a recipe? 4

The answers to these questions are not mutually exclusive—in all likelihood, some students will benefit

most from consistent instruction from year to year, while others will find it helpful to have a clear strategy for tackling particular assignments. Answering these questions collaboratively will help your team develop a shared understanding of the theory or theories that drive its approach to improving instruction and student learning. Articulating the theory also helps with implementation, as we will see in chapter 8. If teachers understand and agree how the action plan should address the problem of practice, they are more likely to implement the action plan and be able to adapt it to meet students’ needs. For example, if the theory behind Franklin’s Problem-Solving Approach is that breaking the process into steps will help make it less overwhelming for students so they are more likely to persevere through each step, then a teacher might adapt the steps for her students. If, on the other hand, the theory is that having a consistent strategy from class to class will help students, she should not adapt the steps, but instead make a point of using the exact terminology laid out in the model.

PUT THE PLAN IN WRITING

Mallory stood up from her desk and took a long stretch. She had offered to convert the scribbles from Franklin High School’s last math department meeting into a draft action plan for implementing the Problem-Solving Approach. She was amazed at

how many steps there were when she typed it all out. Was she making it more complicated than it needed to be?

After choosing an instructional strategy and agreeing on what it will look like in practice, it is time to identify the specific tasks that need to be completed for successful implementation. This step involves assigning responsibilities and time frames, as well as planning for how your school will support teachers as they carry out this new work.

Franklin High School’s action plan (see Exhibit 6.4) shows one way of summarizing this kind of information, but there are many others. Whatever display you choose, it is important to be very clear about what the tasks are, who is going do them, and when they are going to be accomplished. You will find that your action plan contains two different types of tasks. One type is the instructional strategies that teachers will use in their classrooms when working with students. The other type is professional development activities that are designed to support teachers’ implementation of the selected strategy or strategies. Since these two kinds of activity are likely to occur simultaneously, it is helpful to capture both in a single action plan. Assign Responsibilities and Time Frames

Writing down what you and your colleagues agree to accomplish and when is an important step in developing the internal accountability that allows you to hold one another responsible for following through on your actions and creates a shared written history that your school can refer to over time. This document can help teachers think

outside their classrooms and consider their larger roles within the school, and it can be an important avenue for strengthening and developing shared expectations among teachers and between teachers and students, parents or administrators.

Exhibit 6.4

Draft Action Plan for Improving Learning and Teaching in Franklin High School Math Classes

LEARNE R-CENT ERED PROBLE M:

Students have trouble solving multistep problems independently.

PROBLE M OF PRACTI CE:

As teachers, we do not consistently teach a process for solving multistep problems, and we don’t give students enough opportunities to work with multistep problems.

INSTRU CTIONA L

As teachers, we will integrate the Problem-Solving Approach (PSA) into our daily lessons, class assignments, and assessments. We will explain the

STRATE GY:

rubric for evaluating problem-solving to students and will use the rubric regularly in grading assignments.

TASK WHE N

Math department chair and math coach create materials for professional development (PD), including poster rubric and anchor chart listing steps of the problem-solving process to post in the classroom for student reference

Janu ary,

week 1

Math coach models teaching PSA in four classes; all math teachers attend at least one modeling session

Janu ary,

week 2

Math coach leads PD session with math department, debriefs model lessons, offers list of poster problems with a range of difficulty

Janu ary 18,

dept. meeti

ng

Math teachers work in small teams to design PSA lessons for their classes and choose

Janu ary,

which problems to use for Poster #1 assignment

week 3

Math teachers model PSA in classes, share poster rubric with students, and assign Poster #1

Janu ary,

week 4

Math teachers integrate PSA into regular lessons; students work on posters (Due Feb. 15)

Febru ary,

week s 1

and 2

Math teachers work in small teams to assess student work using rubric

Febru ary,

week s 3

and 4

Math coach observes four math teachers (volunteers) integrating PSA into teaching and provides over-the-shoulder coaching

Febru ary,

week s 3

and 4

Math coach leads PD session with math department, discussing student work and

Marc h 7,

offering guidance for the next round of teaching and Poster #2

dept. meeti

ng

Math teachers continue to integrate PSA in classes; students work on Poster #2 (Due April 25)

Marc h and April

Math teachers meet weekly in small groups to discuss student work on Poster #2 and other assignments, and administer district math final exams on June 15

May and

June

When assigning responsibilities and creating a timeline for implementation, you’ll want to be sensitive to the other demands and duties placed on your teachers, as well as the scope of the plan. The level of detail in the plan will depend on the time frame in which you intend to implement it. If the timeline is relatively short, then you may want your action plan to be very specific and indicate what will happen each week. If the timeline is longer, like Clark K–8 School’s plan (see Exhibit 6.5) that is designed to be implemented over an entire school year, then the plan may be a bit more broad.

If the time line takes you across an upcoming academic year, one option is to prepare an action plan that is more specific for the fall semester and less so for the spring,

with the expectation that the spring portion of the plan will be laid out in more detail once implementation is under way and the team can assess its progress. Short-term plans can be helpful for schools that are just beginning the improvement process because they keep the implementation steps concrete, manageable, and immediate.

Exhibit 6.5

Clark Third- and Fourth-Grade Team Reading Action Plan

Plan How to Support Teachers in Their New Work

What can school leaders do to support the action plan? The first and generally easiest step is to provide the resources teachers need to implement the action plan, which might include books, curriculum materials, chart paper, or instructional technologies. The second and far more challenging step is to help teachers develop the skills and knowledge they need to engage in the heart of the work. The school leaders we know who do the best job of supporting teachers in improving their practice think of themselves as “teachers of teachers” who are committed to supporting adult learning.

Support for teachers is most effective if it is coherent, content focused, and frequent.5 “Coherent” means that the support should align with the action plan; avoid professional development that doesn’t directly support your schoolwide instructional strategy. “Content focused” means that the support should be specifically grounded in what teachers teach, and it may need to provide opportunities for teachers to develop their content knowledge and pedagogy. In one school district we know, elementary teachers took math courses to complement their learning of a new curriculum. The courses were specifically designed to provide the deep level of mathematical knowledge they needed to successfully implement the new curriculum. “Frequent” means that

support must be sustained and substantial. Small amounts of time and one-shot workshops will not suffice. The schools we know that are most successful at the work of instructional improvement support teachers at least on a biweekly basis through common planning time or at meetings before and after school. These schools also often dedicate larger blocks of time to significant learning experiences, such as full days during the summer or school year.

Your school culture, which is a reflection of the nature and level of expectations shared among you and your colleagues, will have implications for how best to provide teacher support and how ambitious you can be in your planning efforts. At Clark K–8 School, where teachers have a strong history of collaborative work, the action plan calls for teachers to regularly observe in each other’s classrooms and to offer one another feedback for improving instruction. The same support strategy would probably not work as well at Franklin High School, where the math department has a deeply rooted tradition of teacher autonomy and isolation.

Just as strong teachers are expected to differentiate their methods of instruction according to the needs of their students, so should a thoughtful system of support account for teachers’ different professional needs. How

do you know what your faculty needs to be supported most effectively? As with all learners, sometimes it’s best just to ask them. We have found that involving teachers in deciding what kinds of support they will be offered goes a long way toward developing the buy-in necessary for successful implementation. One school we know spent two months on extensive plans for the following year’s professional development. The instructional leadership team concluded that involving all the teachers in conversations about their professional development needs was critical. The end product was a detailed plan that mapped out how grade-level meetings, content-area meetings, and afterschool sessions would be used together to support teachers.

In sum, developing an action plan is an opportunity to identify relevant and focused professional development for your teachers. Remember, if teachers already had the skills and knowledge they needed to teach differently, they’d probably be doing it. Like all learners, teachers need explicit, specific support in order to improve their practice. Real improvement in student learning takes real time, real resources, and a real commitment to improve. Planning for how to measure whether student learning is in fact improving is the subject of the next chapter.

Integrating the ACE Habits of Mind into Step 6: Developing an Action Plan

SHARED COMMITMENT TO ACTION, ASSESSMENT, AND ADJUSTMENT

Since we first published Data Wise, we’ve developed a real appreciation for shorter action plans. This chapter’s two examples—Franklin High School’s six-month plan and Clark K–8 School’s yearlong one—took quite a while to enact. But action plans spanning as little as two to six weeks and targeting very sharply defined problems of practice can also be quite effective. This is especially true if the team enacting the plan meets weekly, with regular collaboration allowing for quick midcourse corrections that can lead to rapid progress.

The thing to watch out for when designing shorter plans is a tight connection between a rich problem of practice and your plan for addressing it. With interim assessments becoming widely available, many teachers have frequent updates on student skills and knowledge, usually as measured by multiple-choice tests. It can be quite tempting to look at these results at the end of the

unit and then use common planning time to identify the items students struggled with most and create a plan for reteaching the content measured by those items. But is that really the most powerful action that the team can take? Will reteaching truly help teachers to become better at their craft?

One thing you can do to make sure that the action plan serves the purpose of making real and lasting improvements in teaching is to ask yourself whether the plan calls for changing what you are teaching or how you are teaching it. If the answer is how, you are generally in better shape. Another idea is to design your action plan template so that it forces you to write at the top of the page both the specific learner-centered problem you found in the data and the problem of practice that you believe underlies it. This will keep you from either skipping the step of examining instruction, or from doing that step and then ignoring the problem of practice that you found. Too often, we’ve seen teams do the hard work of coming up with a nice juicy problem of practice—for example, “As teachers, we give students tasks that reward summarizing more than evaluation”—and then proceed to design an action plan that calls for something that does not address the problem, such as giving students “more practice” with a particular skill. Having the problem of practice statement as a key component of

the action plan document can be a helpful reminder to make your actions directly connected to what you learned through inquiry.

INTENTIONAL COLLABORATION

The collective effort that got you through the Prepare and Inquire phases is every bit as important when you hit the Act phase. When designing your action plan, it is essential to get the input of everyone on the team. It is equally important to make sure that everyone has a role in enacting it. Putting names and due dates right on the action plan is a good way of making it clear who is involved and exactly what they are responsible for. Another idea is to suggest that team members say out loud what their personal next steps are—we call this “making commitments.” Presenting your commitments in front of others is a public act that builds internal accountability for getting the work done.

When designing your plan, look for opportunities to break the work into small pieces that individuals or pairs can do either between meetings or even during some of the collaborative time that you have set aside. Not everything needs to be done by committee! Sometimes it

is more efficient to allow pairs to go off and draft a lesson plan, a rubric, or an assignment description and then bring a draft back to the group for feedback (the Tuning Protocol can be particularly effective for this).6 The secret is making it clear to all that a draft is only a work in progress, and that the descriptive and specific feedback of everyone at the table is needed to make the plan worthy of your collective effort.

RELENTLESS FOCUS ON EVIDENCE

As discussed earlier this chapter, deciding what to do to address your problem of practice is a critical part of action planning. After brainstorming a number of solutions, what do you use to help you decide which one to implement? Evidence! For each idea on the brainstormed list, you can have each team member provide a piece of evidence supporting the argument that this idea would indeed address your problem of practice. (Team members can provide evidence in favor of an idea whether they first proposed it or not.) You then go around the circle again and have each member provide evidence that might argue against the idea. The advantage of this approach is that it allows you to keep ideas separate from

the person who introduced them, and it challenges people to look for evidence on both sides of the argument. Warning: this activity is bound to leave you feeling you need to collect a bit more evidence to support your decision. But hopefully, it gives you a pretty good idea of exactly what evidence you need, and this extra step may save you time in the long run.

e7

PLANNING TO ASSESS PROGRESS

Jennifer L. Steele and Jane E. King

ANITA, THE TEAM LEADER, DISTRIBUTED THE AGENDA AT CLARK K–8 SCHOOL’S third- and fourth-grade team meeting. “Are we still planning?” Vivian asked, sounding a bit exasperated. “I thought we finalized the action plan last week. We’re doing Think Alouds, right?” She looked around at her colleagues. “When are we supposed to take action with our action plan?” “I’m glad we’re all so excited to get started,” Anita replied. “But remember that we

haven’t yet made a plan for how we’re going to measure student progress.” “I think the state and the feds will tell us whether or not we’re making progress all

right,” Jae replied. “To be worth all of the time and energy we are investing,” Kristina responded, “this

can’t all be just about raising test scores. I didn’t become a teacher to spend my days doing test prep. I mean, I’m trying to help my students become avid readers and critical thinkers. These are complex skills … no standardized test is going to really capture what my kids can do.”

Clark K–8 School is struggling with an essential part of designing a strong action plan: deciding how to measure progress. Public schools annually confront the question of how well students are performing on state assessments. However, systematically measuring students’ progress isn’t an activity that needs to be put on hold until the state test results arrive. Assessing progress is an integral part of the improvement process, through which educators build their internal accountability to one another and find better ways to meet all students’ learning needs. Too often, educators begin implementing action plans without thinking about how they will assess progress. As a result, they aren’t likely to know whether

they’re making progress, and they may not even agree on what making progress means in the context of the plan. Schools engaged in improvement work benefit from setting clear goals for student improvement and proficiency, and from deciding in advance how and when they will measure progress toward those goals. This chapter is a guide to developing an assessment plan as an essential complement to your action plan. Assessment plans address the following questions:

What assessments will be used to measure progress?

When will each type of assessment data be collected?

Who is responsible for collecting and keeping track of the data?

How will the data be shared among faculty and administrators?

What are the goals for student improvement and proficiency?

Of these, the two most challenging decisions are identifying the kinds of assessments to use to measure progress and deciding how to set appropriate goals for student progress.

CHOOSE ASSESSMENTS TO MEASURE PROGRESS

Just as teachers at Clark K–8 School and Franklin High School used multiple data sources to identify a learner-centered problem (see chapter 4), so should schools plan to use multiple sources to measure progress in students’ learning. We have found it helpful to begin such planning by categorizing the available data sources by three time frames: short term, medium term, and long term. Organizing data sources by time frame encourages the use of multiple data sources and helps make the assessment of progress an ongoing part of the school’s professional culture. In general, “short-term data” refers to information that can be collected daily or weekly from students’ work and classroom interactions. “Medium-term data” is gathered systematically within a school, grade, or department at periodic intervals during the year. “Long-term data” is gathered annually and

includes students’ performance on statewide tests. Each type of data can provide important information about what students know and are able to do.

Planning to Use Short-Term Data

Because short-term data are generated continuously within a school, teachers can use them to make assessments of students’ progress a regular, embedded part of their practice. Good sources of short-term data that educators can plan to use to measure progress include students’ classwork and homework, classroom observations of student performance, and conferences with students about their learning. In answer to the questions of who will collect the data, when to collect it, and how it will be shared, we’ve found that responsibility for short-term data collection and analysis is often best delegated to individual classroom teachers because they can examine student work, observe students’ participation, and confer with their students on a regular basis. Teachers have primary responsibility for keeping track of this information, and schools and districts can support the effort by making available data tracking systems that are easy to use.

Looking at Classwork and Homework

The advantage of using students’ classwork and homework to gauge learning is that teachers have access

to a constant stream of data. What’s more, these data are closely aligned with classroom instruction, so teachers can keep track, in real time, of which students seemed to learn the critical skills of a lesson. Teachers can then use these short-term data to inform their instruction so that it responds to students’ learning needs.

In planning to use student work to assess progress, teachers should realize that the task of assessing progress goes far beyond checking completion of classwork and homework. The holistic impressions a teacher can glean from skimming the work of an entire class are helpful to a point, but real use of students’ work to measure progress entails systematically gathering concrete data. For example, a teacher may prepare a small number of yes-or-no questions that she would like to answer about students’ work on a particular assignment. If she has asked students to mark up a short text to show their inferences, her list might include these questions: Does the student list three or more inferences in the margin of this text? Are at least two of the listed inferences plausible? For an open-response math question, she might ask: Did the student get the question right? Did he begin the question correctly? Did he omit subsequent steps? She could then review each student’s work to determine the answers to her questions, which

she would record in a spreadsheet where she maintains short-term assessment data.

After collecting the information described above, a teacher should consider its implications for her instruction and look for areas of performance in which groups or individual students continue to struggle. Realistically, teachers would not engage in this level of specific analysis every day. However, the assessment plan should include a strategy for permitting teachers to examine performance in the action plan’s priority area (e.g., inferences or multistep problems) at regular intervals.

One advantage of collecting short-term data is that teachers can bring samples of the data to discuss at meetings of their departments or grade-level teams. The assessment plan should include time for teachers to share and discuss the short-term data they’ve collected in their classrooms. Ideally, the work teachers share with one another includes direct evidence of the skills that are the focus of the action plan. For instance, each of the math teachers at Franklin High School might bring to their department meeting a sample of student work that demonstrates their students’ ability to solve multistep mathematics problems. Sharing student work in this manner enables teachers to gain insights from their

colleagues about individual students’ work, while also encouraging teachers to develop common norms of what high-quality student work should look like.

Observing Students’ Participation

Another way schools can plan to measure student learning in the short term is by observing students’ participation and demonstrated understanding of a particular concept or skill in the classroom. This kind of observation may be conducted by teachers in their own classrooms or by classroom visitors. Observations can provide rich data on what tasks students are engaged in and how they talk about texts, concepts, and problems they are working on. However, using classroom observations to measure student learning has two limitations. First, it provides more information about the skill levels of the students who participate most vocally than it does about students who are quieter. Second, students may be less willing to take intellectual risks by participating when they are being observed by people other than their teacher. The first limitation is readily addressed by drawing on additional data sources, such as individual conferences or examination of students’ written work to develop a more complete picture of student performance. The second limitation can be mitigated by creating a classroom culture in which

teachers and administrators frequently observe classes, so that having multiple adults in the room becomes less unusual to students. In our experience, students adjust to observers more readily than teachers do. Talking with students about why visitors are coming can also provide a good opportunity to involve them in discussions about their own learning and the school’s efforts to support it.

Asking Students About Their Learning

A third and often overlooked strategy for measuring students’ progress in the short term is asking the students themselves to talk about their learning. Insights from students can be gleaned through individual conferences, small focus groups, surveys, or written reflections. Conversations can be conducted by teachers in their own classrooms or by colleagues who are visiting classes.

There are several ways to gather information from students about their learning. One approach is to have students share their thinking about a particular academic problem or question while the interviewer takes notes on a conference sheet, as a Franklin teacher did on a form her team designed (see Exhibit 7.1).

The interviewer can ask students what they’re doing, why, and what they’re learning. When students talk through problems out loud, it is easy to see where their

understanding of a concept is strong and where it falters. This insight can help teachers correct students’ misconceptions, either through large-group instruction (when the misconception is common) or through individual instruction (when it is unique to that student).

Another approach to these conversations is to simply ask students to describe how they think about a particular concept in the abstract. For instance, a math teacher at Franklin High School could ask her student what the term “multistep problem” means to him and how he would know when multiple steps are required to solve a problem. She might also ask him about what he learned from particular lessons she had delivered in the past. In situations where classwork and homework show stronger growth for some students than for others, it may be useful to ask students whose learning grew considerably about how they learned the material. At the same time, conversations with struggling learners may provide insights into why these students are not benefiting from the instructional strategies listed in the action plan. The interviewers should share the information gained from these interviews with other teachers so that the faculty can revise the action plan to better meet the needs of all students.

Exhibit 7.1

Franklin High School Individual Student Conference Sheet

Stu den t’s Na me:

Michael Serrano

Dat e:

March 18

Cla ss and Per iod:

Ms. McGovern’ s 4th period Intensive Math

Skil l to Foc us on:

Using the Problem-S olving Approach

Teacher’s Questions(Sum

I asked Michael to talk through a word problem in the text that asked

marize what you ask the student to talk about.)

him to solve for the rate given the distance and the time.

Student’s Responses(Su mmarize key points in the student’s response.)

Michael seemed uncertain and said he would need to know “how fast the guy was going.” Asked if it was supposed to be in miles. I said, “What do you think?” He stalled, and I asked if there was a formula he might apply. Then a lightbulb went on, and he recalled the formula (rate)(time) = (distance). At this point, he plugged in the numbers correctly and solved for the rate.

Student’s Understanding( What does the response reveal about the student’s understanding of the concept?)

Michael still needs prompting to know when to use a formula he obviously knows how to use. He just doesn’t know when to apply the formula. This seems to be a common problem in several conferences I’ve had with students this week, and it’s consistent with

the math poster evaluations we did in January.

Implications for Instruction(Wh at does the student’s response suggest about future instructional approaches?)

I need to give students more word problems out of context, so they don’t rely on the lecture preceding the assignment to determine how to attack the problem. We need to spend more time in class on the “understanding” and “planning” phases of problem-solving, just having kids strategize based on what is given and what is asked for in the problem.

As is the case with all forms of data, having a consistent method for collecting and recording the data from these conversations can support the subsequent analysis because it keeps the data focused and facilitates looking for patterns. In some schools we know, the instructional leadership team generates and provides teachers with templates for recording information from student conferences; in other schools, teacher teams generate the templates to meet their specific needs.

As an alternative to interviewing students individually, some schools have conducted focus groups and online

surveys of students about their learning. In many cases, these were conducted following a test administration to detect areas where students felt well prepared for the test as well as areas where they perceived gaps in their own learning. Another useful technique is to ask students to write reflections about their learning on a particular topic. The advantage of focus groups, surveys, and written reflections is that these methods allow teachers to collect information from a large number of students much more rapidly than is possible with individual conferences. The trade-off is that the information may not be as individualized or as detailed as information obtained through conferences.

Planning to Use Medium-Term Data

Almost from the moment the Franklin math department had agreed that they would all teach a consistent approach to problem-solving, they had been talking about how students could demonstrate their understanding of the approach. Someone had thrown out the idea of asking students to create posters that showed the steps they had taken to solve complex problems, and before they knew it, Poster #1 and Poster #2 had been written into the action plan. Math department chair Mallory Golden liked the poster idea and wanted to make it work. But if they really were going to be able to use the posters to measure student progress, there was still a lot they would need to think through.

Unlike short-term data, which the school generates internally through students’ regular work and classroom performance, medium-term data are gathered

systematically at wider intervals throughout the school year. Medium-term data are helpful for tracking students’ progress within a single school year. Sources of medium-term data include commercial tests and assessments developed by a school or district. Some people use the term “benchmark” or “interim” assessments to describe tests that help teachers measure how close students have come to mastery of a particular group of skills.1 The real power in benchmark assessments lies in administering them several times over the year to track students’ progress. Using Benchmark or Interim Assessments

Some schools use commercially prepared tests to measure students’ performance several times a year. These tests are particularly used in the core content areas and are often in a multiple-choice format. Because some commercial tests provide multiple forms of the test for each grade level, it is possible for a school to administer these tests many times a year and track students’ progress. Districts often decide on the types of medium-term assessments their schools will use and when the assessments will be administered. Schools without such district requirements, however, can choose which tests they want to administer throughout the year.

In selecting benchmark assessments, leaders must consider the kinds of diagnostic information they want the tests to provide. For instance, some measures of student performance yield rich information about the processes students use to complete tasks and are useful in planning instruction. Also, many assessments are designed so that analyzing which wrong answer a student selects can provide useful information. When teachers analyze medium-term data together they can learn a great deal about their students’ skills. Later, they can confer with individual students about their responses and about where they encountered difficulty, and thus get an even better sense of how to best help students. When choosing which assessment to use, it is important to keep in mind the amount of effort needed to administer the assessment (some of the most valuable instruments require teachers to sit with students individually), the speed with which results can be obtained, and the richness of the information they provide.

Using In-House Assessments

“Okay,” said Mallory as she opened the December math department meeting. “We agreed we would use a rubric to assess the problem-solving posters. Today we are going to figure out what goes in that rubric.” Adelina asked, “How about if we measure how well students complete each of the

four steps of the Problem-Solving Approach?”

Will replied, “I think we should start by asking what we want to measure: students’ ability to get the right answer, or their ability to use the four-step approach?”

“Well, the PSA is a high-leverage skill that we want them to internalize,” said Adelina. “Um, aren’t we then prioritizing the means over the ends?” asked Eddie. “Isn’t it most

important that they get the question right?”

Developing in-house assessments offers schools tremendous flexibility. Schools can design in-house assessments that are closely aligned to the goals of the action plan, and they can use these assessments to track student progress. As we see at Franklin High School, when you design an open-ended assessment, you will need to agree on how to score it. Taking the time to develop a common rubric is an important part of this process.

We also know of schools that have developed their own schoolwide quarterly writing assessments and created their own rubrics (see Exhibit 7.2 for an example from Franklin) or adapted widely available writing rubrics like the 6+1 Trait Writing model created by the Northwest Regional Educational Laboratory.2

The trade-off in creating internal assessments is that these in-house tests face many of the validity and reliability challenges that were discussed in chapter 2. There are four particular challenges that schools should attempt to address when developing their own assessments. First, each version of the assessment must measure the same skills. If, for instance, Franklin

teachers were to give a permutation problem in January that students could solve through trial and error, and a different permutation problem in March that required students to know the permutation formula, then these assessments would measure largely different skills, albeit unintentionally. Schools can minimize the risk of this problem by having teachers who are well versed in their content work together to develop the assessment.

Exhibit 7.2

Franklin High School’s Problem-Solving Poster Evaluation Rubric

Second, difficulty levels should be consistent from one version of the assessment to the next. If a school administers a literary analysis writing prompt using an F. Scott Fitzgerald passage in September and another using a John Milton passage in December, the scores will not necessarily be comparable because there may be a significant difference in difficulty between the two

passages. Again, schools can minimize this problem by developing all prompts in advance and carefully considering the difficulty of each. When schools borrow questions from released versions of standardized tests, they may be able to obtain difficulty ratings for each question. Schools can then use these ratings to ensure that assessments are of comparable difficulty from one version to the next.

Third, tests must be administered under standardized conditions. For example, if some teachers allowed students to converse during one assessment and others did not, the scores would not be comparable between classrooms. Faculty can minimize these problems by deciding collectively on test-administration procedures.

Fourth, a consistent scoring system must be established. If teachers are left to use rubrics in whatever way they see fit when they assess students’ math posters, essays, or other work, their scores will not necessarily be comparable. Schools can also minimize variations in scoring approaches by training teachers to calibrate their scoring as a group and by removing students’ names from the work to minimize the impact of preexisting assumptions about individual students. In addition, the school can have two teachers score each assessment, and if the two differ considerably in their scoring, a third scorer can help resolve the disagreement.

As with all assessments, schools should do their best to address these concerns and should not rely too heavily on any single measure. Because medium-term assessments allow teachers to track students over the course of a year, they offer an excellent opportunity for teachers to monitor how individuals are progressing and diagnose what each student needs. In addition, these data are valuable for identifying patterns by class, grade level, or other categories. Planning to Use Long-Term Data

Long-term data such as statewide assessments are data collected on an annual basis and present both challenges and opportunities. Long-term data are often generated by an external accountability system connected to state and federal mandates that relies on statewide tests to measure whether schools are meeting the needs of all students. As a result, long-term data are the data that seem to “count” and that schools are under the most pressure to improve. Although these pressures can create an important urgency for improvement, they can unfortunately lead schools to adopt a tunnel-vision focus on long-term data as the sole measure of success. Like any single measure of performance, long-term data give one slice of information that needs to be compared with other data to create a complete picture. The following assessment plan developed by Clark K–8 School shows

how multiple measures of student performance can be incorporated (see Exhibit 7.3).

Exhibit 7.3

Clark K–8 School Progress Assessment Plan, Third- and Fourth-Grade Team

Long-term data can be very helpful for evaluating progress over a longer time frame, such as several years. When the state test remains consistent and the school’s

scores are averaged over multiple years and compared over time, end-of-year test data can reveal useful information about a school’s performance trajectory. While statewide tests may also be used longitudinally to track the performance of individual students from one year to the next, their usefulness for that purpose depends on having comparable forms across grade levels. Such tests are generally more valuable for examining trends over time than for focusing on individual performance. State tests are also limited in their diagnostic usefulness because in many (but not all) states, the results arrive in the school year after the test is administered.

SET STUDENT LEARNING GOALS

Anita was pleased with the plan that she and her grade-level team at Clark had come up with. She liked the way they would be looking at data throughout the year to see how students’ reading skills were coming along. The only problem now was that she had included a “Goals” column on the plan, and it was looking pretty empty. How should the team think about setting goals for student learning? It seemed like any target they would come up with would be, in some sense, arbitrary. What amount of progress was reasonable to expect?

It is important to establish short-term, medium-term, and long-term goals so teachers have targets to aim for and benchmarks by which to assess their students’ progress. Setting goals is part of establishing a culture of internal

accountability and high expectations, and of envisioning what’s possible. While you may not initially know what constitutes reasonable goals, having a candid conversation about how much progress you expect to achieve is essential. Goal setting is a recursive process that becomes more precise as you learn more about your students’ learning trajectories.

We often encourage teams to set audacious goals, while reminding them that the improvement process will not necessarily be a steady climb to the summit. School improvement takes a long time, and proceeds in dips, plateaus, and surges as teachers learn new skills and incorporate them into their practice. Schools sometimes even see declines in student performance at the beginning of the improvement process because teachers are struggling to master unfamiliar strategies. However, as teachers become more confident in using new instructional strategies and as students internalize teachers’ heightened expectations, setting—and reaching—ambitious goals may become easier and easier.

Improvement Goals and Proficiency Goals

Your goals for short-term, medium-term, and long-term assessments should include goals for both improvement and proficiency. An improvement goal is a target for students’ growth on a given assessment within a

specified period of time—for example, to have all students improve their scores by 25 percent on the district mathematics interim assessments between September and May.

By contrast, a proficiency goal is a target for how many students will achieve a level of performance that is considered reasonable and appropriate for students in their grade within a specified period of time. Proficiency goals are not used to measure growth so much as to measure the number of students who have met the performance benchmark. An example of a proficiency goal would be for 90 percent of a school’s tenth graders to perform at or above grade level on a district math assessment by the end of the school year.

Establishing both improvement and proficiency goals keeps schools focused on two distinct and important objectives—growth and competence. A school that is pursuing an improvement goal must concentrate on advancing the learning of all students, even those who are currently far below proficient or considerably above proficient. We have worked with schools where most or all of the students are meeting the proficiency mark, but few students are improving markedly. In these schools, improvement goals can be the more relevant measure and a more meaningful motivator for change. However,

we’ve also worked with a school that managed to improve the performance of a large number of students from reading three or more grades below level to reading only one grade below level. This school met its improvement goal because a large number of its students significantly raised their reading scores. However, these students’ progress did not help the school meet its proficiency goal because the students remained below the appropriate level for their grade.

Both growth and competence are important for students’ real-world success. If students are to be ready to compete in today’s economy, it is not enough for their academic performance to show improvement over time. They need to be proficient in problem solving and communication, among other skills. For students whose performance is less than proficient in either area, improvement isn’t enough. Schools must insist on educating all of their students for proficiency, and that means regularly setting progress goals for both proficiency and improvement.

In addition to setting aggregate improvement goals, some schools find it useful to set improvement targets for individual students throughout the year. Setting individual student goals can focus teachers’, students’, and even parents’ attention on what needs to be

accomplished. We know schools that design individual learning plans with targets and action steps for all students, as well as schools that take a less time-intensive approach of simply setting goals for each student. In schools that involve students and parents in conversations about goal setting, students often surprise teachers by setting goals for themselves that are higher than what the teachers would have set for them.

When the Franklin math department began deliberating about target scores on their poster assessment, they ran into a problem. “It seems like we’re shooting in the dark here,” said one geometry teacher. “I mean, we’ve never given this assessment before, so we have no idea how kids are going to do on it. We could sit here and say we want to have 100 percent of the students exceeding expectations by the end of the year, but that’s just talk. I don’t see how we can set goals for performance on this assessment until we have some kind of a baseline for where we are now. Without that, what’s the point?”

Setting realistic and appropriate goals involves first analyzing baseline data to determine where your school currently stands. You may already have done this as part of the data overview or when digging into data that helped you identify your learner-centered problem. Next you need to think about what long-term success would look like. The geometry teacher in the vignette above raises an issue that we have often heard: How do you decide what you are shooting for? We have seen schools that use other schools as a benchmark by which to set

their goals. Sometimes schools look to other schools that are having more success with similar student populations for an idea of what they should aim to achieve. We have also seen schools look to schools with different student populations (for example, urban schools looking at suburban schools … or vice versa) for the same purpose. This approach of looking externally for benchmarks can minimize the potential to underestimate the level of achievement that is possible based on current data.

The next step is to decide how long it should take to move from the current performance level to a level that would constitute success. With this time frame in mind, you can set intermediate goals that will lead to meeting that long-term success target.

The Goldilocks Problem

Just as Goldilocks found herself trying out beds that were too hard and too soft before settling into one that was just right, so do schools face the difficulty of setting goals that are neither so hard that faculty become demoralized nor so soft that there is little to strive for. It’s useful for teachers to set ambitious goals so they are constantly pushing themselves and each other to improve their practice and promote student learning. However, because the improvement process involves ongoing inquiry and learning, schools must keep in mind that

progress goals are a means to an end rather than an end in themselves. In other words, missing a lofty target does not necessarily mean complete failure, just as attaining a very conservative goal does not mean you have achieved raging success.

Setting goals offers schools the opportunity to discuss the big picture of what matters to them—like helping students become learners and thinkers, readers and problem solvers, and kind and contributing citizens. Setting goals also offers schools the opportunity to establish high expectations for what students can achieve with the support of teachers. Goals that are set internally are merely benchmarks for measuring progress, keeping faculty members focused, and holding team members accountable to one another, and they should be seen as such. Attaching extraordinarily high stakes to these goals creates incentives to game the system, which (as chapter 2 explains) does not support student learning. Thus, schools should strive to set ambitious but achievable goals, and maintain the perspective that what matters most is not meeting a particular target but constantly self-assessing to determine what is working and what can be done better to meet all students’ learning needs. Exhibit 7.4 from Franklin High School shows how goals can be included in an assessment plan. After deciding how, when, and by

what standard students’ progress will be measured, you are ready to begin implementing your action and assessment plans.

Exhibit 7.4

Franklin High School Math Department Progress Assessment Plan

ntegrating the ACE Habits of Mind into Step 7: Planning to Assess Progress

SHARED COMMITMENT TO ACTION, ASSESSMENT, AND ADJUSTMENT

Chapter 7 addresses the importance of planning to assess the extent to which your action plan improved student achievement; chapter 6stresses the need to develop implementation indicators that allow you to assess the extent to which the plan improved teaching practice. Taken together, these two types of assessment give your team the guidelines you need to make a collective impact. You can make this work more personal—and more immediate—by giving teachers time to set their own goals for how they would like to see their practice change over time. To formalize this, you can hand out index cards and ask teachers to each write down a short-, medium-, and long-term goal that they have for their own practice. This can help to reinforce the idea that the team’s collective success hinges on each individual making a commitment to improving his or her ability to help students meet ambitious learning goals.

INTENTIONAL COLLABORATION

If you have given teachers an opportunity to write down their personal improvement goals, you can then facilitate intentional collaboration by pairing teachers and having

them share their goals and brainstorm ways they can support one another in reaching them. Over the course of implementing the action plan, building in time to allow pairs to reconnect around their goals may help to foster the kind of mutual support that will allow teachers to make meaningful changes in practice. This kind of “buddying up” can even work across schools or districts. Specialists (such as art or music teachers) who do not have a person who shares their role in their building can find it very useful to discuss their goals and their progress toward them with someone who understands well the pedagogical challenges associated with a particular content area.

RELENTLESS FOCUS ON EVIDENCE

Don’t overlook the value of the student voice as a potential source of evidence. So far in this book, we have mentioned the importance of conferencing with students about their learning. But we have not yet discussed the possibility of asking students directly about how well they think a teacher’s use of the instructional strategies supports that learning. As always, it is important to find ways to elicit responses from students that are

descriptive and specific so that you will be able to act on them. For example, “Working in small groups feels like a waste of time to me” is not nearly as useful as, “When we work in small groups, I find that I usually know how to do the problem and so it really slows me down to have to get everyone else to understand it.”

Interviews and surveys are two ways of collecting student data; which you use depends on your priorities. If you would like detailed information with the possibility of follow-up questions, interviews can be the way to go. In this case, you will need to decide who will do the interviewing. Will teachers ask their own students? Their colleagues’ students? Will the information be collected by someone that the students do not know? Or is it possible to have the students themselves collect the information as part of a research project? For any of these, you will need to think about the reliability of the data and the validity of the inferences you can draw from it. Nevertheless, when taken as part of a larger effort to assess the action plan, students can provide an important piece of the puzzle.

Surveys can also be an effective means of learning about student perceptions of teaching. A questionnaire can be as low-tech as a piece of paper with checkboxes or as high-tech as an online survey. The Data Wise

Project website provides links to organizations that have produced surveys designed to elicit student views on a wide range of topics pertaining to their level of engagement with content, their peers, and teachers. If properly administered, they can provide an opportunity for allowing students to share their feedback anonymously. Teachers have told us that this information is extremely valuable because it provides them with more information about the target of instruction: the students!

8

ACTING AND ASSESSING

Liane Moody, Mary Russo, and Jonna Sullivan Casey

THE BACK-TO-SCHOOL BUZZ WAS UNMISTAKABLE. CLARK K–8 SCHOOL PRINCIPAL Sandy Jenkins welcomed many returning teachers and a few new ones as she opened the first faculty meeting of the year. “What I’m passing out is a one-page summary of the instructional strategies we developed last spring to improve students’ skill at making inferences in a variety of literary genres,” she explained. “Since we spent a lot of our time working in grade-level teams, I thought it would be helpful for us all to see how our work in developing student reading skills connects across the school and gets built from grade level to grade level.” On paper, the summary of the action plans for each team made a lot of sense. Yet

Sandy had a nagging feeling that bringing these plans to life was going to be a lot harder than anyone expected. What if the focus on the action plan distracted teachers from all the other important things they needed to do? What if teachers struggled with the strategies and gave up quickly? What if teachers didn’t implement them with skill and consistency? And, worst of all, what if the strategies didn’t make a difference for student learning?

When teachers have worked collaboratively to develop an action plan for improving instruction, you may find that they have a great deal of momentum behind their ideas for change. Actively engaging in the improvement process can empower teachers, inspiring in them the confidence that they play the critical role in improving

student achievement and the commitment to fulfill this role to the best of their abilities. Nevertheless, as a school leader, you may find that when it comes time to implement the action plan, your faculty needs you more than ever. Teachers will be center stage as they begin implementing instructional improvements in their classrooms, and they will look to you to orchestrate a smooth flow among the various components of the plan and help them regroup when the going gets tough.

The reality of school improvement work is that even with the best planning, when you begin implementation you will inevitably encounter surprises and challenges. Instructional strategies will look different in practice than they did on paper. Students will respond in different ways. As teachers develop more sophisticated knowledge about their practice, new and more complex problems of student learning will emerge. Thus far in the improvement cycle you have been guiding your faculty in an inquiry process; implementation is simply the next logical step in training teachers to look deeply at problems of student learning and develop a disciplined way to respond to them. To capture this spirit, some school leaders describe the action plan as an experiment, a form of action research in which a school tests its theories of how instructional strategies lead to student learning.

Your school teams worked hard to put their action plan ideas in writing. Now that it is time to put the ideas into practice, four questions can guide your work as a school leader:

Are we all on the same page?

Are we doing what we said we’d do?

Are our students learning more?

Where do we go from here?

ARE WE ALL ON THE SAME PAGE?

While you may wish teachers could put all of their energies into implementing the action plan, the reality is that any plan for instructional improvement will need to occur in the context of your school’s ongoing work. For example, Clark K–8 School’s new focus on using Think Alouds with third- and fourth-grade students must occur alongside their district’s requirement that all schools implement workshop-style instruction. At Franklin High School, teachers will need to integrate the new

problem-solving strategy they’ve developed with a very explicit and detailed pacing guide for the math curriculum. To help teachers bring the action plan alive, school leaders need to communicate the action plan clearly, integrate the plan into the ongoing work of the school, and use teams for support and internal accountability.

Communicate the Action Plan Clearly

How would you like to lead a school where teachers are so focused on student learning that a visitor could ask any one of them about your school’s instructional goals and get the same thoughtful answer? Communication prior to and during implementation can help ensure that the goals and strategies of the action plan are well understood and that expectations for teachers and students are clear and consistent across classrooms.

One effective means of communication is the creative use of school documents. While the action plan document itself may comprehensively lay out the steps teachers will follow to implement an instructional strategy, that document is most likely not the best way to communicate the plan to faculty and staff who were not actively involved in drafting it. Offering instead a one-page summary that sets out the key components of school improvement work can be an important step in

helping create an environment in which all teachers are able to describe in specific and concrete language how the school is working to improve student performance. A document that distills the important points from a school action plan, like the Clark K–8 School Improvement Plan at a Glance shown in Exhibit 8.1, can be effectively shared with everyone in the school community, including students and parents.

At Franklin High School, the entire math department was involved in the process of identifying problem solving as a major student weakness and in choosing the Problem-Solving Approach as a promising way to deal with it. However, not all math teachers participated in every step of writing up the action plan. Franklin experienced a common problem of communication that arises when schools attempt to move from having a representative team develop an action plan to having a broader group implement it. To deal with this problem, the math department chair worked to bring all faculty on board by making sure that all math teachers understood the goals for student work, the changes in practice designed to help achieve these goals, and how progress toward these goals would be measured. Communicating this information in a large school is an important and ongoing challenge.

Exhibit 8.1

Clark K–8 School Improvement Plan at a Glance

ntegrate the Action Plan into Ongoing School Work

Being clear about what the action plan entails is an important task. But it can be equally important to make sure that your school’s other instructional goals are not

eclipsed by the new plan. A good way to do this is to work with your faculty to develop a schoolwide curriculum map and describe how your new strategies fit into this overall plan.

At Clark K–8 School, principal Sandy Jenkins and her leadership team had created a schoolwide English language arts curriculum map based on the state curriculum frameworks, district guidelines, and their own action planning. For each English language arts learning standard, the team compared what each grade level would be covering each week of the school year to ensure that teachers would teach all standards to all students over the course of a year. Anyone who consulted this map, which was prominently displayed outside of the school office and posted in individual classrooms, could see what topics would be featured when at each grade level. The map proved extremely helpful when grade-level teams met to figure out how to integrate the instructional strategies from their action plans into their ongoing work.

Use Teams for Support and Internal Accountability

As he looked around at the department meeting, principal Roger Bolton could see that Anne McGovern and Jean Louis felt lost. As teachers of “intensive math” at Franklin High School, their classes were filled with the weakest math students in the building. Both new teachers, they were pleased to be included in the math department’s process for analyzing data and brainstorming instructional improvements. They were enthusiastic about the Problem-Solving Approach that the

department had chosen. But now that it was time to actually start teaching it, the plan seemed out of reach. “I don’t see how we can teach this method,” Roger heard Jean mutter at the end of the meeting, “if our students can’t actually read the problems we give them or do the most basic calculations that they ask for.”

When you first began organizing for collaborative data analysis, as a school leader you worked to build a system of interlocking teacher teams. Now that it is time to implement your action plan, these teams can become an important source of support and inspiration for teachers. As teachers become accustomed to checking in with their colleagues regularly, they make a greater effort to be prepared for meetings. When teachers see their teammates working hard to implement the strategies, they are motivated to do their best to make changes in their own classrooms. And as teachers realize that they eventually will be turning over their students to the next year’s teachers—who will expect students to have experienced the teaching practices agreed on in the action plan—they develop an increased sense of personal responsibility for their students’ performance.

There may be times, however, when you discover that the team structures you put in place are not enough, and that you may need to facilitate the development of new collaborations. Principal Roger Bolton guessed that one of the reasons intensive math teachers were frustrated was that they needed guidance about how to make the Problem-Solving Approach more accessible to their

students. So Roger arranged to have the math department meet with English as a Second Language specialists and special education teachers to discuss how to provide more instructional scaffolding for student learning, such as breaking the steps of the approach into smaller units that could be taught over a longer period of time. Because Roger had involved all faculty in exploring math performance at the beginning of the year and then later shared the math department’s action plan with the whole school, all teachers were quite familiar with the task at hand. Encouraging faculty to work across disciplines to figure out how to connect the instructional goals of the school to their daily lessons can be a powerful way of making sure implementation gets off the ground, especially when considering the learning of at-risk or struggling students.

ARE WE DOING WHAT WE SAID WE’D DO?

As a school leader you can begin to keep track of how things are going almost from the moment teachers start implementing the action plan. Monitoring is critical to ensure a smooth launch and to clearly signal to teachers that this work is so important to you as the school leader that you intend to give it your personal attention. There is no point in waiting until the implementation period is well under way to get a read on whether teachers really

integrated the instructional strategies into their practice. If you wait that long and then find out that things did not go well, it’s like looking in a rearview mirror—the action is past, and it is often too late to change it. If you stay involved from the beginning, you can become a helpful partner in making sure that instruction really changes. To do this, you may find it helpful to visit classrooms frequently, promote consistency rather than conformity in instructional practices, and adapt professional development plans to meet ongoing needs that emerge from the work.

Visit Classrooms Frequently

Observing practice was something your team needed to do in order to identify the problem of practice that you were committed to addressing with your action plan. Once you start implementing that plan, you will want to get right back into classrooms to see how things are going. In fact, these visits should be part of your plan to assess progress.

As mentioned in chapter 5, the Data Wise Project Team created Key Elements of Observing Practice: A Data Wise DVD and Facilitator’s Guide in response to feedback from practitioners about the challenge of creating a culture where observing practice is a comfortable part of all teachers’ experience.1 That resource argues that

effective classroom-visit routines contain five key elements:

Focus

Observe

Debrief

Adjust

Follow Up

When the purpose of classroom visits is to implement and fine-tune instructional improvements assessing progress, then the task associated with the focus element is reviewing your problem of practice and understanding how the lessons you will be observing will attempt to address that problem. When you observe, you will be collecting evidence about what teaching and learning looks like when the action plan is being implemented, and when you debrief you will discuss the evidence that you heard and saw and commit to how you will adjust

instruction based on what you learned. When you follow up, you will discuss what happened when you adjusted instruction and plan new steps.

During your focus meeting, you may want to distribute note-taking sheets that remind observers about what the learner-centered problem and problem of practice are and give people space to keep track of what the teacher is doing, what the students are doing, and what task students are actually working on.2

One school we know developed a very detailed protocol and schedule for conducting classroom visits. Recognizing that it would be impossible for an individual to observe all aspects of classroom practice at once, the school’s protocol involved having four classroom visitors at a time, each charged with making observations about a specific area of practice. Because their action plan focused on improving the use of questioning and conversations in teaching, the four areas of practice targeted by the protocol looked at what types of questions teachers ask, what kind of evidence there is for conversations about learning, physical evidence of improvement strategies, and students’ perspectives on their work. Each observer was given a note-taking form unique to the area he or she was observing. The entire classroom visit schedule called for a pre-meeting to give

an overview about the day, which consisted of two classroom visits of 15 minutes each, and a post-meeting to debrief the experience.

Promote Consistency Rather Than Conformity

When you visit classrooms, you are bound to find as many variations on the action plan as you have teachers. The challenge for school leaders involves distinguishing between skillful adaptations that move teachers to more sophisticated ways of supporting student learning and unskillful ones that water down or distort the spirit or intent of the original plan.

Clark principal Sandy Jenkins had worked hard to encourage teachers to faithfully implement their action plans. But on one visit to Vivian’s fourth-grade classroom, she began to question the value of her efforts. As Sandy observed the teacher leading the class in a Think Aloud exercise, she could see that two students seemed to hold back from the activity. When the teacher called on one of these students, he was clearly uncomfortable and struggled to put his ideas into words. When Sandy expressed her concerns to the teacher after the visit, Vivian knew just

what she was talking about. “Jeffrey struggles with activities that put pressure on him to speak up,” she explained. “But if I don’t call on him he just tunes out and misses the lesson. I used to work with Jeffrey and several students with similar difficulties separately to show them how to underline key words that helped them figure out the deeper meaning of what they were reading, but with the focus on whole-class Think Alouds, I stopped doing that.”

Sandy is faced with a leadership challenge that arises for many school leaders when they work to change instructional practice: the difficulty of promoting consistency while supporting teacher creativity. In

communicating the action plan to teachers, Sandy may have overemphasized the importance of consistency in implementing the plan without highlighting the equal importance of allowing room for teachers to employ their knowledge and creativity in responding to the needs of diverse learners, particularly struggling learners. Vivian will not be able to truly improve her practice if she strives simply for conformity—faithful implementation of a plan that may wind up leaving behind some of the students who most need her individualized attention.

Striving for consistency involves getting to the deeper “why” behind instructional strategies, just as you did when you first examined instruction (see chapter 5). For Clark K–8 School the goal of the action plan is to use teaching strategies that encourage students to “think about how they think” as a means of developing inference skills in reading, not to make sure all students participate in a particular activity such as Think Alouds. Consistency involves the deeper question: How can we achieve our instructional goals with all students?

Although achieving consistency may be difficult, it is a worthwhile goal for several reasons. One is that if you think of the action plan as an experiment, the only way to truly test your hypothesis that your new instructional strategies will improve student learning is to faithfully

implement those strategies and then see what happens. Another reason is that in striving for consistency, teachers develop a shared understanding of what constitutes effective instruction, and this understanding is essential in making ongoing improvement part of the everyday work of your school. The most important reason, however, is that students can benefit enormously from consistent instruction. It helps them integrate what they learn as they move from teacher to teacher during the school day or from one grade to the next, and it ensures that all students in all classrooms throughout the school have similar opportunities to learn.

Despite the virtues of consistency, it is important to recognize that, given the needs of diverse learners, it is a rare instance when a single strategy or set of strategies will meet the needs of all students. Classroom teachers are perfectly positioned to explore the chosen instructional strategies and try to improve them in the context of solving learning problems. When they report on their adaptations, the whole school can learn what it takes to elevate the learning of all students.

Some examples we’ve observed of helpful adaptations involved an action plan in which elementary school teachers used a math routine called “number of the day.” During a 10-minute segment of instructional time,

teachers asked students to brainstorm strategies for computing that day’s number and then displayed the strategies on large chart paper in the classroom. The theory behind the plan was that if students have regular opportunities to explore numbers in an open-ended way, guided by the teacher and with other students so they build their knowledge in a social context, they will become more comfortable with numbers and develop an intuitive sense of the many ways numbers can be used. Teachers worked with the class to record the ideas their students generated, and the principal reviewed them during her classrooms visits. One first-grade teacher noted that some students in her class were not as forthcoming with their strategies as others. To inspire these children to see that they were capable of generating strategies that were just as good as those of students who spoke up more readily, the teacher decided to designate certain students the day before to be the first to share their ideas the next day for the number-of-the-day task.

Another teacher made a different modification. When he saw that his students were not producing as many math strategies as he would have liked, he decided to leave the 10-minute math chart paper up all day long and let students add strategies during free times throughout the day, such as lunch or recess. The result was a

dramatic increase in the number and quality of strategies students generated. In modifying the instructional strategy, these teachers were able to make changes that made it more likely children would experience the heart of the strategy: stress-free exploration as a way to build comfort and confidence in manipulating numbers.

Adapt Professional Development Plans to Meet Ongoing Needs That Emerge from the Work

Franklin High School math teachers had been using the Problem-Solving Approach in their classes for six weeks. The action plan called for their March professional development session to offer guidance about assigning and grading Poster #2. As math coach Sasha Chang and department head Mallory Golden met with the principal to plan the session, they began by sharing some concerns. “I’m not so sure we should focus our professional development on the rubric again,”

began Sasha. “I’ve been in a lot of classrooms, and I’ll tell you what I’m seeing: teachers are using the Problem-Solving Approach, that’s for sure. But it seems like it’s kind of taken on a life of its own. Instead of focusing on helping students get good at thinking for themselves, they are focusing on helping them get good at following the steps of the process. Teachers seem to be placing more emphasis on evaluating whether the students follow the steps than on evaluating the thinking that actually happens at each step.”

“My sense is that quite a few teachers realize this is happening,” Mallory responded. “Some people have come up to me and say that they like the approach, but they feel the training on it just hasn’t gone deep enough. They find themselves just walking students through the ‘understand’ phase of the problem, doing much of the work for the kids themselves because they’re just not sure how you teach that part of the process.”

When you first created your action plan, you took your best guess at what kinds of professional development your faculty would need to support them in the work. Once you begin implementing it, however, you may find that you have to modify your professional development

plan, either by changing the content of scheduled sessions or by adding additional ones to make sure teachers get the help they need. One school we know used a “just-in-time” approach to ensure that teachers were thoroughly prepared to teach each unit of a new math program. They arranged for a workshop to be held just before the teaching of key units at each grade level so that teachers could review the lesson, materials, and teaching moves required immediately before they taught them.

For teachers to make meaningful change in their practice, they may need to make multiple attempts to integrate new ideas. Instead of being discouraged when things do not go perfectly the first time around, as a school leader you can focus your efforts on understanding why teachers are having trouble with certain aspects of the plan. If you structure professional development so it is flexible and provides opportunities for teachers to learn, try out, analyze, get feedback, and reflect on new practices, you can help teachers feel that they will be supported during this hard work.

For example, an action plan for improving writing skills might call for the use of one-on-one conferencing with students about their writing. For teachers unfamiliar with this practice, incorporating it meaningfully into their

teaching might require that they have an the opportunity to see another skilled teacher demonstrate key components of conferencing, such as asking appropriate questions, giving specific feedback, and guiding students in editing. After that, teachers would most likely need ongoing opportunities to try out this practice in their classrooms and meet with other teachers to discuss, analyze, and reflect on their implementation and the results they were getting. You can demonstrate the depth of your belief in professional development by participating in sessions along with teachers; you model the importance of professional learning by being a learner yourself. In creating and supporting ongoing opportunities for professional development, a school leader reinforces the idea that the goal of school improvement work is to create a school culture in which everyone is learning.

ARE OUR STUDENTS LEARNING MORE?

Although supporting teachers in improving their practice is an important part of your job during implementation of the action plan, as a school leader your ultimate responsibility is to keep your faculty focused on student learning. After all, the main reason your school developed an action plan was to increase student learning, and the main reason you developed an assessment plan was to

be able to measure progress toward this goal. You can keep the focus on learning in the short term by regularly checking in with teachers about learning outcomes. In the longer term, you can help teachers see the big picture and be honest in evaluating what is or is not working.

Regularly Check in with Teachers About Learning Outcomes

Sandy Jenkins had learned a lot by visiting classrooms throughout the fall to observe how teachers were implementing the strategies outlined in their action plans. As part of her visits, she tried to make a point of looking through student work and talking with the children themselves about what they were learning. A couple of days after the winter results of the district’s reading assessment were in, Sandy stopped by third-grade teacher Jae Kim’s classroom to have a more concrete discussion about how things were going. “I noticed that your class made some real progress on the December assessment,”

Sandy began. “Can you tell me more about what you saw in the data and how it affects the way you will approach the next few months?” Jae showed Sandy a spreadsheet that displayed the results of the district test alongside his own assessment of each student’s performance from conferencing notes, then explained his findings: “Well, although most students reached or exceeded their targets on the district assessment, you can see that there are three kids who fell pretty far short. For Danny and Keisha, I saw this coming. You can see from my notes that Danny is still at the decoding stage and Keisha just can’t seem to get past the literal meaning of the words. As for Marissa, I’m really surprised at her score. If you just sit with her as she reads and talks about a book she’s read, she seems to know what’s going on. Maybe she has trouble transferring her knowledge into a testing situation? Anyway, I’m a little concerned about how to reach these kids. They all need something different and there just doesn’t seem time in the day to give it to them.”

In the days before your school was engaged in a process of continuous improvement, it might have been awkward to sit down with a teacher in the middle of the year and ask him what he could tell you about how much his

students were learning. But once you have action and assessment plans in place, initiating these kinds of discussions can become much more comfortable. When teachers begin to understand that you are an ally in their quest to help each child reach his or her potential, you may find that they look forward to these encounters and actively solicit your advice in finding solutions.

Sandy and Jae had their conversation about the performance of struggling learners in his class before it was too late. By the end of their discussion, Sandy had offered to have a literacy specialist work with Jae’s neediest students intensively over the next several weeks right in the classroom, so that Jae could see some of her teaching moves and begin to use them himself. She offered to join the next third- and fourth-grade team meeting to participate in their discussion about whether Think Alouds were making a difference for students who were still working to master the most basic reading skills. Another strategy she came up with was to ask the school’s parent liaison to make a particular effort to reach out to the families of these three students and make sure that they understood the extra support their children would be getting.

Collecting short-term and medium-term data on student learning in the way described in chapter 7 provides direct

and immediate evidence about whether students are learning and gives you an opportunity to make midcourse adjustments in the plan if they aren’t. When you focus on what multiple data sources reveal about student learning, you can remain true to the goals of the plan while modifying the instructional approach when needed. Maintaining the ability to change midstream requires treating the action plan as a dynamic document, so that teachers don’t see it as a straitjacket but as a living, organic process tailored to the school’s mission. Help Teachers See the Big Picture

Math department chair Mallory Golden collapsed into the chair in principal Roger Bolton’s office and let out a long sigh. “Take a look at this,” she said as she handed over a chart showing Franklin High School’s performance on the most recent state math test. “All that work on the Problem-Solving Approach and the percentage of our students scoring in the lowest two proficiency levels has barely budged. According to the rubrics we’ve been using, our students have made big strides in tackling multistep problems. How do I tell my teachers that all that hard work was for nothing?”

After leading your school through a difficult process of change, it can be frustrating when you don’t see results right away. During his regular visits to classrooms, Roger observed firsthand the visible changes in math instruction at Franklin High School that seemed to engage more students in learning. He felt instruction in math was finally on the right track. They still had a long way to go, but he knew it was up to him to make sure that

they didn’t allow a quick look at the data to throw cold water on their efforts.

Helping teachers see the big picture means encouraging them to think about whether they have really explored the data completely and whether they have taken into account the many other indicators of student learning available to them as a school, as well as the many other ways to present data. When Mallory and Roger looked closer at the numbers, they realized that although the percentage of tenth graders scoring in the “Failing” and “Needs Improvement” categories of the test had not changed much since the previous year, more students were now in the higher of these two categories (see Exhibit 8.2). Students had higher raw scores within each proficiency level than they had in previous years, but because of the way the results were reported, these differences were not readily seen. They also noticed that the percentage of students taking the test was noticeably higher than it had been in previous years. This could mean that some of the weakest students, who traditionally had avoided school and testing days, this year had the confidence to at least give the test a try. By looking at the same data in a more detailed way, they had a more complete picture of the effects of implementing their action plan for math.

Exhibit 8.2

Franklin High School Distribution of Student Proficiency over Time

Helping teachers see the big picture also means managing expectations around the question of how soon it is reasonable to expect to see improvements in scores on tests used for accountability purposes. When Roger saw that the math faculty’s assessment plan set a target that Franklin High School would meet accountability goals on the state math test, he let them do it. After all, he reasoned, meeting this target was a measure by which

many people would judge the school. Nevertheless, Roger knew that it would take time for teachers to develop skill and competence in teaching the Problem-Solving Approach. Also, the plan had only been in effect for the second half of the school year, and there had been some glitches along the way. This meant that in a number of classrooms students had only a month or two of solid training in the methodology. Given these factors, Roger felt that it was too soon to use the state test results to judge whether the plan was successful or not, so he encouraged his faculty to try the approach for another year. He also shared with teachers his own evidence and observations from his classroom visits that instruction in math classes was improving.

Be Honest in Evaluating What Is and Is Not Working

Sandy carefully reviewed the charts that the instructional leadership team had prepared showing changes in student reading performance over the previous school year. Although all grades had made progress, she noticed an interesting phenomenon: the improvement among the fifth and sixth graders was much stronger than that among third and fourth graders. To build each team’s commitment to the plan, she’d made the conscious decision to let each grade-level team choose its own instructional strategies for achieving the schoolwide goal of helping students become more skilled readers. But now she wondered: was there a chance that the strategy adopted by the third- and fourth-grade team was not as effective as the strategy adopted by the fifth- and sixth-grade team?

Sandy faces a delicate situation. The third- and fourth-grade team implemented their Think Aloud

strategy with zeal all year. Yet student performance on the district-mandated benchmark reading test was not that impressive. As a school leader, you may find that once the data come in you need to help your teachers take a hard look at whether the strategies they have worked so hard to implement really made a difference. This can be difficult, because when people believe strongly in their ideas they often become “pet projects” in which a lot of energy and time is invested, and teachers may become unwilling to look at signs that those strategies are not working with some or all of their students. While we do not want to encourage schools to simply abandon strategies that do not produce results across the board, it is important to take a careful look when learning outcomes are not as robust as you had hoped. As a school leader, your responsibility will sometimes be to point out when a well-designed and well-executed strategy just isn’t doing the job.

Exhibit 8.3

Clark K–8 School Reading Test Performance

In the case of Clark K–8 School, the graph of reading test performance (see Exhibit 8.3) would not be enough to tell Sandy whether the third- and fourth-grade strategy was problematic, and certainly would not be enough to tell her why. There are plenty of reasons why the average growth patterns might be different for different grade levels. Sandy would need to consider this graph in combination with other data, such as her observations during her classroom visits that the Think Aloud approach did not seem to work well with certain students, or her observations during classroom visits of the upper grades that might have shown those teachers having particular success with Reader’s Response notebooks and Literature Circles. When Sandy begins a conversation with her faculty about whether they might need to

supplement or replace their chosen strategy, she can raise questions such as: Are we truly implementing the strategy to the extent we planned? Have we been doing it long enough? Is it working for all learners? Is this strategy powerful enough to be our primary focus?

Viewing implementation as an experiment from which the school will learn new things about effective instruction can help school staff approach their own work honestly and critically. As you evaluate the effectiveness of implementation (the extent to which we are doing what we said we’d do) as well as the effectiveness of the strategies themselves (the extent to which students are learning more), it helps if you remain open to the possibility that change may be required. In this way you help your school become invested in student learning rather than in any particular strategy. Sometimes the data will show that it is time to change course.

WHERE DO WE GO FROM HERE?

Once you have implemented your action plan and the ideas and practices embodied in it have been tried out with consistency, it is time to regroup. No doubt you and your faculty have learned a lot—and probably determined that you have a lot more to learn. It’s important to ground the next level of work solidly in evidence of student learning. Three things you can do once you have made it

around the steps of the improvement cycle are to celebrate success, revisit your criteria and raise the bar, and plan how to keep the work fresh and ongoing.

Celebrate Success

At the final faculty meeting of the year, Sandy surprised her teachers by not passing out a series of data charts. “We’re going to talk about a different kind of data today,” Sandy began. “We’ve worked hard all year to try to improve reading instruction in our school. We’ve had our ups and our downs, but I think every one of us has learned something. On the paper I am distributing, I’d like you to take a few minutes to describe something in your teaching this year with which you were successful. Be sure to write down evidence of its success and the reasons why you think your effort worked. After you’ve had a chance to jot down your ideas, evidence, and reasons, we’re going to divide into cross-grade groups and use a protocol that allows us to really get at the heart of what success looks like, sounds like, and feels like.”

So often teachers are asked to look at test scores or their own practice and talk about what went wrong. We have found that tempering this by giving teachers a chance to think deeply about what makes for successful practice can be a very positive learning experience. Using a formal protocol such as the Success Analysis Protocol that Sandy used at Clark K–8 School can help ensure that the conversation gets beyond simple platitudes and continues the practice of talking about teaching in a fine-grained way.3 Giving teachers a chance to talk together about their own success and reflect on the reasons for that success does more than give them a well-earned moment of appreciation. It also allows them

to begin to internalize the qualities at the heart of best practices, such as evidence, analysis, and reflection, and to think about how to bring these qualities more strongly into their work.

Another way to celebrate success involves simply showing your faculty how far they have come. Sandy used the table in Exhibit 8.4 to show her faculty how they had worked through the eight steps of the improvement cycle, often managing a number of different tasks at once, in order to make progress.

Exhibit 8.4

Clark K–8 School Time Line for Completing the Steps of the Improvement Process

There are, of course, many other ways of celebrating success that may rely more heavily on making public the results of your student assessments. Success can be defined collectively or individually, and progress as well as achievement can be acknowledged for both students and teachers. What is important is that a collective sense of accountability for results is built among teachers. Acknowledgments of progress are especially powerful for students when they can be broadly distributed across all student groups, and not just among those who are accustomed to recognition. An authentic way of celebrating success involves thoughtful displays of the student work that meets or exceeds clearly articulated standards. In one school we worked with, each teacher put up a “Wall of Work” in his or her classroom and the principal created schoolwide displays showing the work of every student in the school, together with a statement of the standard the work was intended to demonstrate. In another school, to reward students’ efforts to improve their writing, teachers displayed not just the final pieces of writing, but the brainstorming sheets, outlines, first drafts, and edited drafts as well. They did this to show concretely the hard work that it takes to get to standards and to remind students that, with effort, everyone can improve.

Revisit Your Criteria and “Raise the Bar”

Roger met with the Franklin High School math department and talked with them about the disappointing performance of tenth graders on the state math exam. “Let’s not let this get us down, folks,” he told them. “Some of the posters the kids created, especially for the most recent assignment, are quite impressive. You seem to have hit on a type of assignment that captures their imagination and an approach to tackling word problems that gives kids something to hold on to.” “The second round of posters did go quite well,” Mallory added. “In fact, for most

steps of our rubric, we met our target of having 75 percent of our students score ‘Meets Expectations’ or better.”

“I wonder,” Roger replied, “whether it’s time to take another look at that rubric and the way we apply it. Let’s be sure that when we say a piece of work ‘Meets Expectations’ we are talking about high expectations. When we first got into this problem-solving thing we weren’t quite sure what it was reasonable to ask of kids. Now that we’ve seen a little of what they can do, I think we can start asking for more. How about taking a look at the scoring rubric used in the state exam? We want to be sure there is alignment between how we assess students and how the state does.”

As teachers gain understanding of the strategies they are implementing, as a school leader, you can help them adjust the criteria by which success is measured to promote the deeper understanding of the material that comes with time. By revisiting the conversations they had when they were first developing shared understandings and expectations for the work, teachers can begin to redefine what is possible. At Franklin, teachers had very little experience using a systematic approach to evaluate students’ demonstrations of their problem-solving strategies. At first, they may have made their rubric too easy or made their grading too lenient. By sticking with the strategies and refining them, they can collectively

come to better and more nuanced understandings of what good work really looks like.

For a school that is implementing a new set of instructional techniques, very visible signs of implementation—like the prominent display of student work—may be one way of demonstrating success. For a school in which teachers are refining their practice, the criteria will need to become more refined and more focused on complex teaching and learning skills. For example, a more refined approach could include the student self-assessment on the rubric with each piece of work displayed. As the criteria change, the ongoing feedback teachers receive from their colleagues evolves to push their practice toward higher goals. By continuing the conversations in which teachers reflect on their practice and their goals for student learning, the criteria for success will continue to deepen along with teacher learning.

Plan How to Keep the Work Fresh and Ongoing

The improvement cycle curves back on itself for a reason: once you get to the “end,” you continue back around the cycle, but each time you use it at a higher level and apply it to a more complex problem of student learning. As using the improvement cycle to change instruction becomes ingrained in the practice of your school, you

may find it becomes easier to know what questions to ask, how to examine the data, and how to support teachers and students. You will also be able to go deeper into the work, asking tougher questions, setting higher goals, and involving more people in the process. The first time around it might have been all you could do to get your faculty to begin thinking about addressing student learning in a proactive way. As you continue, you may find that you need to involve other constituents who have a stake in student learning outcomes, such as families and community members, in the process. By the time you have been doing this kind of work for several years, you may notice that you quite naturally find ways to enlist the support of even more partners in making a difference for student learning.

As you keep the work going, you may also find that you are able to distribute leadership responsibilities more efficiently than you originally could. When your school has developed a solid culture of inquiry, it becomes natural for large numbers of teachers to take active roles in making sure that the school focuses on continuous improvement. Teachers increase their span of interest and responsibility from their own classrooms and grade levels to concern for the school as a whole. The roles of instructional leadership team members—in particular those of data managers—may shift as well. Whereas at

the beginning only a few individuals might have been interested in or capable of analyzing data, as more teachers in your building become comfortable with this work, the skills become more widely spread throughout the faculty. The instructional leadership team can then graduate to a role of thinking more broadly about what kinds of assessments and professional development opportunities would best support the school’s vision for improving student learning, and the team can engage in evidence-based conversations with your district about what kinds of central support would be most helpful. With that in mind, the next chapter of this book lays out the most effective ways we have seen districts support schools in using data wisely.

Integrating the ACE Habits of Mind into Step 8: Acting and Assessing

SHARED COMMITMENT TO ACTION, ASSESSMENT, AND ADJUSTMENT

We’ve saved our single most important insight about building this particular habit of mind for this last step of

the Data Wise Improvement Process. Here it is: every time you have a meeting or hold a professional development session, set aside five minutes at the end for participants to provide specific, descriptive feedback about how it went. If you have taken one of our courses or read Key Elements of Observing Practice: A Data Wise DVD and Facilitator’s Guide, you know that the Data Wise Project team’s favorite way to do this is to use the Plus/Delta Protocol (see Selected Protocols at the end of the book). You begin by reminding people of what the objectives and the agenda of the session were. Then give them a few minutes to reflect on “pluses.” These are things that worked well. To prompt for these, you can ask, “What contributed to our learning and productivity today? What should we keep doing?” Then allow time for people to reflect on the things they wish they could have changed about the session. These are called “deltas” (and often noted with the symbol Δ, which denotes change). Here you can ask: “What detracted from today’s experience? What should we do differently next time?”

This protocol nicely connects an action (the meeting) and your assessment (asking for feedback). The power of the Plus/Delta Protocol lies in your commitment to using the feedback you obtain to adjust what happens the next time your group is together. For example, during one meeting, you may learn that people felt uncomfortable

that an important discussion about the learning needs of a particular group of students was cut short because you didn’t leave enough time for it on the agenda. If you don’t ask for feedback about the meeting, some people may walk away with a sense of unfinished business or disillusionment with the process. If you allow space for this comment to arise, but do nothing to address it, you are arguably worse off than if you didn’t ask for feedback—people will learn that nothing comes of raising a “delta.” But if you take the feedback to heart and directly address it before or during the next meeting, participants will see that when they provide a candid assessment, it does indeed lead to action. Time after time, we’ve seen how this reflective practice generates a virtuous cycle of continuous improvement, in which a school or team can gracefully recover when something doesn’t work.

INTENTIONAL COLLABORATION

Throughout this book, we’ve argued that protocols can be extremely useful tools for building this habit, and we’ve pointed you to protocol instructions so that you know how to get started. But protocols become truly powerful when

you adapt them to meet your needs and your context. Believe it or not, there is a protocol out there that is designed specifically to help you reflect on the effectiveness of a protocol! It is called “SUMI,” and it involves writing the following four letters on a piece of chart paper (or whiteboard, or shared document):4

S What surprised you as you engaged in this protocol?

U How could you use this protocol in your teaching and other meetings?

M How would you modify this protocol?

I What is the impact of this protocol?

Then you ask participants to reflect out loud on these four questions. Taking time to debrief each protocol has two important benefits. First, it helps participants to construct meaning for themselves from what they experienced. We’ve been interested to see that sometimes the person in the room who was most vocally against using protocols ends up having the greatest insights. One principal we worked with told us that the impact of one particular protocol was that “it forced me to listen, which was hard, so it made me wonder if I am not

doing enough of it.” Second, it gives teachers an opportunity to think about how they might adapt the protocol for use in their classrooms, and teachers can never resist an opportunity to think of new strategies to use with their students. When students see the adults in your building modeling intentional collaboration, they get a glimpse at what team-work looks like. But when students regularly experience the rich peer interactions that protocols allow, they can make great strides in developing the kinds of expert thinking and complex communication skills that they will need to thrive.

RELENTLESS FOCUS ON EVIDENCE

One way to capture evidence of team learning is to create “living documents” that keep track of decisions over time. In our own work on the Data Wise Project team, we create one electronic document for each of our major work streams and store all documents online so everyone on the team can access and edit them. Each time we plan a meeting for a particular work stream, we add a new agenda to the top of the appropriate document, and as each meeting unfolds, we take notes directly into the agenda. At the end of the year, the documents contain an

easily searchable archive of our previous thinking, reflections, and actions. We strongly recommend that you consider this approach for documenting all the good work you and your colleagues do to use data wisely. You may find it is a rich source of evidence for collective reflection on what you have achieved—and what remains to be done.

9

ROLES FOR THE DISTRICT CENTRAL OFFICE

Nancy S. Sharkey and Richard J. Murnane

THE PURPOSE OF THIS CHAPTER is to provide school superintendents and their leadership teams with an understanding of the roles that district central offices can play, and in most districts must play, if schools are to make constructive use of student assessment results. These roles, which we describe below, include providing a data system and software, incentives, skills, and time to do the work. We also argue that it is important for central office teams to model the work.

THE DATA SYSTEM

Districts and schools must consider numerous factors in establishing and maintaining a data system—what is worth including, how the data should be organized and stored, what software to use and whether to create or buy it, and who has access to what data.

What Should Be Included?

An important first step districts can take to support schools’ efforts to use data constructively is to create and maintain a student information system that is accurate and kept up to date. Of course, it makes sense

for the data system to contain student-specific, detailed results from assessments used for accountability purposes—the results schools are under pressure to improve. However, since analysis of these scores always raises more questions than it answers (see chapter 4), the potential to examine multiple data sources is important. For that reason, many districts have found it useful to ask schools to administer interim assessments in core subjects over the school year and to store results from these tests in the district’s central database. Other potentially valuable information to store and make available to schools electronically includes results on midyear and end-of-year course-specific tests. As explained in chapter 4, teacher teams often find it valuable to examine not only students’ total scores on these examinations, but also skill-specific subscores and responses to individual items. Consequently, it is important that the district data system make this easy to do.

Given that time spent testing children is time taken from instruction, it is important to verify that test results are useful in diagnosing the strengths and limitations of children’s knowledge and the extent to which they have mastered skills included in state learning standards and in the district’s curriculum. For example, if the curriculum and standards emphasize skill at responding to

open-ended questions, it makes sense for the assessments to include open-ended questions. If the curriculum and standards focus on multistep questions, the assessment should as well. We have seen many schools analyze student assessment data only to realize that the district-required assessments were not aligned with the district-required curriculum. For example, an eighth-grade math test assesses algebra skills not covered in the district’s curriculum until ninth grade. If the assessments do not match the curriculum and standards, teachers and administrators are unlikely to believe that the assessment results are accurate measures of what students know and can do and, consequently, that looking carefully at those results is a valuable use of their scarce time.

As we explained in chapter 3, school-based teams often want to examine the relative performance of students with different characteristics. For that reason, it makes sense for the districtwide database to contain information about individual students, including gender, race, and ethnicity, special education status, language-minority status, and free or reduced-price lunch eligibility. Some school teams also want to examine the performance of groups that participate in special programs, such as tutoring or afterschool programs. Having this information recorded on the district database

makes these tasks much easier to accomplish. Since enrollment in programs like tutoring changes rapidly, it is important to implement a strategy for keeping the information up to date. How Should the Data Be Organized?

It makes sense to store student data at the individual level so that school teams have the flexibility to examine assessment results for students grouped in a variety of ways. It is also important that the data system track students over time, as they switch schools and complete additional assessments. To do this, each student must have a personal identification number. Many states assign identification numbers to students when they first enter a public school in the state. When districts use these same numbers, the tasks of following students from district to district and merging results from state- and district-administered assessments become easier. Updating the district’s data system regularly so that records accurately reflect the school each student is attending and the classes each student is taking is critical. School teams will persevere in making sense of student assessment results only if they find that the information in the district’s database is accurate and up to date.

Districts with relatively small numbers of students may find it sufficient to store student data in a database management program. However, to provide support for a large number of data teams making multiple varied queries about student data, many districts find it necessary to purchase a data warehouse, and some state education agencies are developing their own statewide data warehouses.

What Software Should Be Used?

Once all pertinent information is located in a well-supported data warehouse or database, the district must give schools software to analyze the data. Many districts we have worked with provide all schools with customized software that allows teams to easily create the types of graphs illustrated in chapter 3. It is also becoming more common for districts and states to provide online tools for displaying, charting, and analyzing student data directly through the data warehouse websites.

It is important to recognize that the capacity to analyze student assessment data will vary widely across schools. This creates the challenge of providing tools that satisfy the needs of school teams doing quite sophisticated work and those that are just beginning. This is a challenge because, typically, the more sophisticated the analyses a

software system will support, the more complex its design and the more difficult it will be to use.

We have seen districts deal with the “power versus ease of use” trade-off in a variety of ways. Some provide all schools with basic reports that school staffs will likely start with when creating simple data overviews of their schools. These basic reports can be preprogrammed in the district’s data system, enabling school staffs to retrieve them easily for their own schools. One district calls these reports the “Top Ten” and distributes paper copies to school administrators as well as making them available online. Once these overviews have been discussed by administrators, members of the instructional leadership team at each school can engage in the deeper data work that discussion of these reports will prompt. A variant of this strategy used by other districts is to create an easy-to-use interface that allows schools to address an important but limited set of questions. Once school teams have mastered this software and demand more powerful tools, the district provides training in accessing more sophisticated features.

As school teams become good at the work, they make greater demands of the data. For example, one school team we worked with wanted to know how students who

had been in their building for at least two years had fared on the state assessments. Another wanted to investigate whether students who had math for their first-period class did less well on the state math exam than students who had math later in the day. Designing data and software systems that allow teams to answer such detailed questions efficiently and also engage the novice user after only a little training is a significant design challenge. District staff need to be prepared to support schools at all stages in their learning and to help schools to share novel display strategies and creative and informative analyses.

Make or Buy?

Districts that want to give schools the tools to do the work described in this book need to decide whether to create data and software systems in-house or purchase them from an external vendor. Each alternative has advantages and disadvantages, the importance of which will depend on local circumstances, including in-house technical capabilities. One district that we have worked with created its own web-based database system, which enabled schools to combine student characteristics and state assessment data with benchmark data generated at the school level. An advantage of this choice was that because many central office departments and school

representatives were involved in the design and implementation of the data system, many clients were ready to use the product. A disadvantage was that the process was slow. It took more than a year to provide schools the tools for basic analyses like those described in chapter 3, and the development process was ongoing three years later.

Another district we have worked with contracted with a vendor to provide reports and analyses of student assessment data to administrators, teachers, parents, and students. Some vendors provide data warehouse and software systems that will support very sophisticated data analyses. Some also provide benchmark or interim assessments and teacher professional development tailored to meet needs identified in the assessment results. An advantage of purchasing products and services from external vendors is that they are usually prepared to act quickly. Of course, these products come at a cost, which is typically assessed on a per-student basis.

Who Has Access to What Data?

Because student information is confidential, every district with a central data system must have a process for determining who has access to what data for which students. In several districts we work with, teachers could

not access the assessment results of their former students. Since results from the state tests administered in May were not available until September, this meant that they could not learn whether students they had taught the previous year were able to demonstrate proficiency on the skills they had emphasized. The importance of remedying this flaw in the access system seems straightforward. However, other access questions are more difficult. For example, should teachers be able to access test scores of the students taught by other teachers in their schools? Should math teachers be able to access reading scores for the students to whom they teach math? These are two examples of the many access questions that arise when school faculties attempt to examine student assessment results. However, as norms in the education sector slowly shift toward giving teachers more access to student data, these questions may become less salient.

INCENTIVES AND SKILLS

We have seen that schools make more progress with this work when they feel that all district policies—especially those that pertain to planning, professional development, and evaluation—support rather than undermine their efforts to use data to improve teaching and learning. An instructional leadership team that has dedicated itself to

this practice wants to know that its efforts are being recognized and valued by the central office. When district personnel request to see the improvement process in action during their visits, ask what they can do to make things go more smoothly, and find ways to connect schools that can learn from one another, so school-based educators have strong incentives to deepen the work of improvement.

While incentives are important, they are not enough. To use student assessment results constructively, school faculty need a variety of skills, including the ability to (1) understand, interpret, and use assessment data correctly (assessment literacy); (2) use software or online tools to access data and create and understand graphic displays of assessment results; (3) participate productively in group conversations and decisions where they have to be open to sharing practice and debating ideas; and (4) develop, implement, and assess action plans to improve instruction. The district central office has a major role to play in organizing and supporting the requisite professional development.

Assessment Literacy

To interpret score reports from different tests, teachers and administrators need to understand scales, benchmarks, and percentile ranks. To appreciate what

inferences are appropriate to make from assessment results, educators also need to understand other concepts explained in chapter 2, such as measurement and sampling errors, validity, and reliability. In our work with schools, we have observed two strategies that appear to help school teams acquire elements of assessment literacy: modeling good practice and providing expertise on an as-needed basis. One district we have worked with created a sample whole-school improvement plan that modeled the appropriate use of student assessment results. Since every school was required to create such a plan, the model plan attracted considerable attention. This district also had a research office staffed by people with the skills to answer the many questions that arise as school-based teams work to make sense of test score reports. Using the Data System

In chapter 3, we described the types of data displays that school staffs may find useful in catalyzing conversations about student assessment results. We have seen district central offices engage in several activities to help school-based educators learn to use the district’s data system and create effective graphic displays. One is professional development specifically directed to teaching these skills. A second is creating guidelines with

instructions for school-based users to easily produce commonly requested graphs for specific groups of students. A third is creating an online help system and having district staff available to answer any questions that arise. Many educators refer to this process as “pulling the data” from the system. They also speak of the need to help other teachers learn to pull the data on their own, rather than relying on the principal or a data manager to help them. Group Processes and Collaboration

A theme of this book is that to contribute to instructional improvements, the work of analyzing student assessment results needs to be done collaboratively. Some teachers feel uncomfortable participating in many of the activities described in this book, including brainstorming about potential explanations for patterns in student assessment results and discussing their students’ work with other teachers. Some teachers are reluctant to be observed by other teachers because this runs so contrary to the closed-door culture of their schools. It is worth remembering that many building and district administrators come out of schools with closed-door cultures and have no experience in creating a culture in which teachers work together as described in this book.

We have suggested that using protocols to structure conversations about student performance and instructional strategies can help create a “data wise” school culture. In our experience, many school leaders are not aware of the value of protocols and how to use them. We see this as a potentially valuable focus of professional development for school leaders and other educators.

As we explained in chapter 3, having all teachers and administrators take the assessments that states require of students has been an eye-opening experience in some schools we have worked with, one that catalyzed constructive conversations about teaching and learning. Districts can facilitate this activity by ensuring that school staffs have copies of previously administered tests and answers available.

Collaboratively creating scoring rubrics and then scoring student papers in groups also builds collegiality and teacher knowledge. Teachers have told us that scoring state assessments was a powerful form of professional development. It not only taught them to understand what the assessment was asking, but also increased their understanding of the curriculum standards and what different levels of student performance actually looked like. This activity can help

teachers develop a common language of assessment and instruction. Typically, however, the teachers who have experience grading state tests are the “specialists,” or teachers who are already recognized for their skills. Districts might expand the cadre of teachers who participate in grading state tests so that more teachers have access to these learning opportunities and can share what they have learned with other teachers at their schools. Developing, Implementing, and Assessing Action Plans

Another recurring theme of this book is that analyzing student assessment results is a critical part of effective instructional improvement strategies. Chapter 5, for example, presented ways school staffs can identify current teaching practices at their schools, and chapter 6 discussed how teachers and administrators can create instructional action plans that move teachers from analyzing data to developing strategies for improvement. In our experience, however, we have found that many school faculties need help in moving from understanding the learner-centered problem to framing a problem of practice to designing and implementing an effective action plan for improving instruction.

Districts can support the work proposed in both chapters by providing professional development in

content and pedagogy. In some districts we have worked with, professional development is provided through coaches (also known as specialists), who receive professional development from the district’s central office and then share what they have learned at the schools where they work. These coaches typically spend part of their time working with groups of teachers (for example, leading a session on exploring assessment results). They also spend part of their time in classrooms, modeling, assisting with, and observing lessons. An important and often missing step in using coaches to support instructional improvement is teaching them the skills described in this book. Unless coaches understand how to do the steps of the improvement cycle, they will not be able to help schools carry them out.

Employing coaches is not the only way to increase teachers’ instructional capacity. In most schools, there are teachers with the expertise to help other teachers develop greater pedagogical and content knowledge. By creating opportunities for teachers to learn from each other (for example, by visiting one another’s classrooms and having structured and safe conversations about teaching and learning), districts can take advantage of the expertise that already exists within schools and the district while increasing capacity still further.

TIME

The school day in most districts does not afford teachers and administrators enough time to work together to systematically examine student assessment results and then translate their acquired knowledge into instructional improvements. One district we worked with negotiated a contract with its teachers union that called for slightly longer school days four days per week and an early release day for students once each week. Teachers meet for professional development after students leave on the early release day. Half of this time each month is dedicated to district-selected professional development topics, and half is set aside for topics selected by school-based decision-making teams. Teachers reported that they had used some of their time creating rubrics together and scoring student work based on the rubrics. Professional development time also was allocated to look at external assessment results and to plan instruction accordingly.

Teachers often need time not only to talk with each other, but also to talk with students. As explained in chapter 4, conversations with students can be an important data source for understanding what they know and don’t know and, perhaps more importantly, why. Systematically gathering this type of data from students,

however, can be difficult in the typical instructional schedule. If districts value the information gleaned from conversations between students and teachers about student work, they could encourage schools to adopt schedules that build in time for such conversations.

Observing colleagues in the same school can provide teachers with practical instructional ideas, can help foster mentoring relationships, and can further discussions about instructional strategies. For these reasons, many schools we have worked with see this activity as central to their strategy for creating consistent, effective instructional programs. Some school districts facilitate this activity by paying for substitutes to cover the classrooms of visiting instructors.

It is quite possible that there isn’t enough common planning time, even if the district has created additional shared professional development time. In these cases, the district might compensate teachers if they spend their own time on professional development activities pertaining to improving teaching and learning. For example, one district negotiated a stipend for teachers who participated in a set number of hours of professional development activities that exceeded the number of hours required by contract. This extra pay provided teachers with the incentive to improve their instructional

practice, even when the district could not provide additional time in the school day for this endeavor.

MODELING THE WORK

Perhaps the most important way a school district’s leaders can support schools’ efforts to improve is to model the process, engaging in each step so that they have first-hand experience in what is involved. A central office leadership team that did this started by setting norms and expectations around how team members would use their collaborative time. They took time to build their own assessment literacy, admitting to one another when they did not know how to read some of the reports they were expecting a typical teacher to understand. Then they analyzed overview data of student performance on state assessments and dug into a wide range of other sources of evidence to help them identify a learner-centered problem that they were particularly passionate about. Then came the tricky part: examining what they were doing as a district that was contributing to the learner-centered problem. With this problem of practice in hand, they created an action plan that called for them to make changes in district policy. They enacted the plan and assessed the extent to which their actions may have addressed the learner-centered problem. Throughout the process, they tried to support one

another in cultivating the habits of mind of action, collaboration, and evidence. The work was hard, but they knew that they needed to face it—just as they were asking school teams to do. Their dedication sent a strong message to schools throughout the district about how much the central office leadership valued the work of improvement.

After trying out the process, the district can show its commitment by sticking with it. This includes celebrating the successes, setting new goals, and returning to the improvement cycle again and again. It also includes looking through all of the ACE insights at the end of the preceding eight chapters and thinking about how to apply them to the district context. In short, district central offices that want schools to become data wise need to become data wise themselves.