Using Data to Improve Schools

vgirldesir

UsingDataToImproveSchools.pdf

Home >Education homework help >Using Data to Improve Schools

What’s Working What’s

Working

Using Data to Improve Schools: What’s Working

This publication was created with editorial assistance from KSA-Plus Communications in Arlington, Va.

This report was produced in whole or part with funds from the Office of Educational Research and Improvement, U.S. Department of Education, under award # R215 U99 0019. Its contents do not necessarily reflect the views or policies of the Department of Education.

About AASA

The American Association of School Administrators, founded in 1865, is the professional organization for over 14,000 educational leaders across the United States and in other countries. AASA's mission is to support and develop effective school system leaders who are dedicated to the highest quality public education for all children.

iii

Foreword School system leaders are discovering the power of data for promoting school improvement.

With recent advances in technology and the increased demand for assessing student learning,

an unprecedented amount of data are available to educators. School districts across America are

beginning to use the tools necessary to make effective use of the data. In addition to test scores,

many educators are collecting data about citizenship, character, healthy lifestyles, school climate

and parental and community involvement.

One superintendent reflected that “We spend a lot of time on testing but not much time on

what to do with the test results.” As educators shift their focus from simply reporting test results

to using the data to improve instruction, data become essential ingredients in school improve-

ment. Educators know that the effective use of data can measure student progress, evaluate

program and instructional effectiveness, guide curriculum development and resource allocation,

promote accountability and, most importantly, ensure that every child learns.

Using Data to Improve Schools: What’s Working is an easy-to-read guide to using data to drive

school improvement. School system leaders and their staffs can learn from this book how to

build a districtwide culture of inquiry that values the use of data for sound decision-making.

School board members, parents and community members interested in helping improve schools

will find tools for their work as well in this guide. It describes the challenges and the successes of

educators from districts both large and small committed to using data.

We are sure that you will find this guide useful in your ongoing efforts to provide leadership

to your schools and communities.

Paul D. Houston, Ph.D.

Executive Director

American Association of School Administrators

Using Data to Improve Schools: What’s Working

Bill Adams

Superintendent

Salem County Vocational Technical Schools

Woodstown, N.J.

Lance Alwin

Superintendent

Antigo Unified School District

Antigo, Wis.

Mary Barter

Superintendent

Durango School District 9-R

Durango, Colo.

Richard P. Fragale

Superintendent

Central Union High School District

El Centro, Calif.

David E. Gee

Superintendent

Western Suffolk BOCES

Dix Hills, N.Y.

John Lacy

Superintendent

Billings R-IV School District

Billings, Mo.

Peg Portscheller

Executive Director

Colorado Association of School Executives

Englewood, Colo.

Roland Smit

Superintendent

Mobridge School District

Mobridge, S.D.

Linda Dawson

Project Director

National School Boards Foundation

Aspen Group International

Castle Rock, Colo.

Acknowledgments Development Advisory Team

Acknowledgments

AASA Project Staff

Judy Seltz

Associate Executive Director

Constituent Relations and Services

Geannie Wells

Director

Center for Accountability Solutions

Mike Parker

Assistant Director

Center for Accountability Solutions

Aleck Johnson

Program Manager

Center for Accountability Solutions

Sarah Wayne

Program Assistant

Center for Accountability Solutions

AASA Executive Committee

Don W. Hooper

President

Superintendent

Fort Bend Independent School District

Sugar Land, Texas

Bill Hill

President-Elect

Superintendent

Deer Valley Unified School District

Phoenix, Ariz.

Benjamin O. Canada

Immediate Past President

Educational Consultant

Atlanta, Ga.

Mary F. Barter

Superintendent

Durango School District 9-R

Durango, Colo.

Barbara F. Erwin

Superintendent

Scottsdale Unified School District 48

Phoenix, Ariz.

Richard P. Fragale

Superintendent

Central Union High School District

El Centro, Calif.

David E. Gee

Superintendent

Western Suffolk BOCES

Dix Hills, N.Y.

Donald L. Kussmaul

Superintendent

East Dubuque Community Unit School District 119

East Dubuque, Ill.

John R. Lawrence

Superintendent

Lincoln County R-III School District

Troy, Mo.

Estanislado Y. Paz

Superintendent

Tucson Unified School District

Tucson, Ariz.

Kay E. Royster

Deputy Chief Executive Officer

Detroit Public School District

Detroit, Mich.

Using Data to Improve Schools: What’s Working Foreword .............................................................. iii

Acknowledgments .............................................. iv

Chapter 1: Why Data Matter ........................................ 1

Chapter 2:

Using Data to Make Smart Decisions ...... 13

Chapter 3: Data, Public Engagement and Strategic Communications........................ 27

Chapter 4: Strategies for Success ................................ 37

Appendix A: Accountability Measures ............ 53

Appendix B: Working with the Media ............ 54

Appendix C: Resources ...................................... 56

Appendix D: Glossary ........................................ 60

M any superintendents have a powerful ally on their side: data. Increasingly,

superintendents are using data to make smarter decisions, and they are getting results.

Instead of responding defensively to critics, they are armed with facts and figures that tell a

more complete story and help critics understand the root causes of the challenges schools face.

“Data-driven decision-making is about gathering

data to understand if a school or district is meeting its

purpose and vision,” says Victoria Bernhardt, author

of Data Analysis for Comprehensive Schoolwide

Improvement. (See A Closer Look on page 2.) “If we do

not have a target, we could make decisions that essen-

tially lead to ‘random acts of improvement.’” Instead,

Bernhardt says, superintendents should strive for

“focused acts of improvement,” which occur when

schools are clear about their purpose, about what they expect students to know, and about what

they expect students to be able to do.

In data-driven districts, superintendents work side by side with other administrators, teachers,

principals and parents to ensure all children achieve. Everyone strives toward common goals.

Data provide quantifiable proof, taking the emotion and rancor out of what can be tough calls

for superintendents and school boards (e.g., dismantling a popular but ineffective program or

closing a school). Data also provide the substance for meaningful, ongoing dialogue within the

educational community.

For the past several years, Superintendent Gerrita Postlewait and the Horry County (S.C.)

Public School Board have made data-driven decision-making the cornerstone of school

improvement. “This is now a school system that focuses on results,” Postlewait says. “If I, as the

Chapter 1: Why Data Matter • What goals has your school district set for the next three years?

• What data will help judge whether the district is meeting its goals?

• How are superintendents using data they currently collect to improve student achievement over time?

• What additional data must be collected and why?

• In what ways are teachers, principals, district staff and the community involved in data collection and analysis?

If I, as the superintendent, cannot talk about how much learning has occurred, then I’m not achieving what I had hoped to as superintendent.

— Gerrita Postlewait, superintendent,

Horry County (S.C.) Public Schools

superintendent, cannot talk about how much learning has occurred, then I’m not

achieving what I had hoped to as superintendent.” (See Getting Results on page 3.)

Postlewait and other superintendents acknowledge that data were not always at

the forefront of their decision-making. Until recently, they collected data only

because gathering this information was mandated by the state in return for

funding. Rarely were data used to analyze whether children were learning at

grade level, teachers were using sound instructional practices or parents were

pleased with the quality of schools.

“Many school improvement plans are hit or miss,” observes Superintendent

Roland Smit, who heads the 600-student Mobridge (S.D.) School District. “We may

think we should work on a specific area but we don’t always look at the indicators

that may tell us this is, in fact, something we should work on. With the help of

data, we can zero in on our weaknesses.” Data help superintendents like Smit

make decisions with far greater precision and clarity.

Data help district and school leaders craft a sound blueprint with measurable

results for continuously improving schools so decisions are no longer based on

incomplete or biased information. The challenge for superintendents is to know

specifically what they are evaluating. For example, is a particular math program

effective? Which students are showing the great-

est gains and why? How can we target those

who are showing the least improvement?

Superintendents can focus on a specific grade

level or measure overall progress toward meeting

math standards, or they can use data to see how

many students are at levels of proficiency or

excellence in math.

A Closer Look Victoria Bernhardt

Author, Data Analysis for Comprehensive Schoolwide Improvement

For many, data are confusing, even intimidating, but

in your book you say data are logical if we think

about what we need to know and why. Tell us more

about that.

If questions can be created, the data that are

needed to answer the questions are very logical.

For example, for the question “How well are we

Using Data to Improve Schools: What’s Working

What is data-driven decision-making? • Collecting data

• Analyzing data

• Reporting data

• Using data for school

improvement

• Communicating

through data

Data-driven decision- making requires a cultural shift in thinking that must be nurtured so all stake- holders are committed to this effort.

Data help: • Measure student progress

• Make sure students don’t fall through the cracks

• Measure program effectiveness

• Assess instructional effectiveness

• Guide curriculum development

• Allocate resources wisely

• Promote accountability

• Report to the community

• Meet state and federal reporting requirements

• Maintain educational focus

• Show trends (but not necessarily solutions)

Data do not help: • If the data are not valid and reliable

• If appropriate questions are not asked after reviewing

the data

• If data analysis is not used for making wise decisions

doing?” one would probably want to

look at student achievement results

on standardized tests and state or dis-

trict assessments to get an idea of

how students in the district are scor-

ing right now. For the question “Are

all students learning?” one might

want to take the general student

achievement analysis to a deeper

level, looking at the distribution of

scores to understand which students

are scoring below mastery, and how

far they are scoring below mastery.

You also write that data can help uncov-

er solutions to some of the toughest chal-

lenges schools face today. How so?

I think data can help us see things

we might not see otherwise. We

might have processes or programs in

operation for years. Once we look at

the data from all angles, we might

find that a program is not helping all

students learn. Data help us get to

the root causes of a problem so we

solve the problem and not just the

symptom.

In what ways can superintendents use

data to improve student achievement?

Student achievement data, for exam-

ple, can help superintendents under-

stand which instructional strategies

are creating the best results and see

where additional training might be

needed. Perceptions data can tell

superintendents about parent, stu-

dent and staff satisfaction with the

learning environment, which also

could reveal areas in need of

improvement. Staff perceptions can

Chapter 1: Why Data Matter

In the early 1990s, the Horry County (S.C.) School Board was

granted the authority to raise taxes. School board members,

parents and community members in the racially and economi-

cally diverse school district of 29,000 students wanted to know

the district was spending their taxpayer dollars wisely.

Gerrita Postlewait, who was an instructional specialist with

the district at the time, says the district staff began by re-

examining their expectations. “For example, what kind of

achievement results did we want to see,” Postlewait says.

“What kind of activities did we want? Those questions and

others kick-started the data-driven decision-making process.”

District leaders, principals and teachers learned a lot along

the way, including that at-risk four-year-olds enrolled in the

district’s structured academic prekindergarten program scored

consistently higher in reading, math and writing in first,

second and third grades than students who participated in

preschool programs sponsored by other governmental agen-

cies. In fact, by third grade, 75 percent of the students who

attended the district’s preschool program scored higher than

their counterparts in reading and writing.

After collecting and analyzing the data, the district

expanded the prekindergarten program by an additional 200

children. “As we prepared our budget, we had a way to talk

with our public about why this proposed change was impor-

tant,” Postlewait says. “We could show a reduction in the

amount of time we have to spend with these students in reme-

dial education a little later in grades one, two and three.”

Postlewait became superintendent of the district in 1997.

“It’s chaotic and bone-wearying work, but it’s worth it because

we are now seeing gains in student achievement. This is the

first year we have had results to celebrate.”

Getting Results Horry County, S.C.

Using Data to Improve Schools: What’s Working

tell superintendents what is possible. Demographic data can provide valuable information about

meeting the learning needs of students in the future, including: How might the makeup and size

of the school population change? Or, how many new teachers with specialties will be needed in

the near future?

If you are a superintendent using data for the first time, where do you begin? How do you get started?

I would start by understanding what data exist in the district

and see what data I can get easily elsewhere, like the state edu-

cation department or other sources. Undoubtedly, I could get

demographic data, such as enrollment and attendance figures.

I also believe that perceptions data is very important for under-

standing schools and the health of the district. I would admin-

ister online questionnaires to students, staff and parents. Then,

if I were really on the ball with the data analysis, I would set up

a comprehensive database to store and analyze the data at the district, school, classroom,

teacher and student levels.

What sort of leadership must a superintendent provide as more and more teachers, district staff, board

members and others use data?

I believe that superintendents must create a safe environment for data-driven decision-making.

Superintendents must lead, model and encourage staffs to use different types of data and exam-

ine the numbers systematically to avoid knee-jerk reactions to single, independent pieces of

data. It will require training and trust so that every staff member can have access to, and look at,

data on an ongoing basis. It requires a vision that is truly shared, so staff members are all contin-

uously improving in the same direction.

Getting Started One of the biggest challenges for any district implementing data-driven decision-making is

knowing where to begin. Often, there are so many data to choose from that the process can be

overwhelming for district staff, teachers and principals. The best advice is to start with the

basics. Too often, school administrators and staff collect data without first figuring out what it is

they need to know.

“We recommend not fishing blindly as you review data,” says Mike Parker, assistant director

of the Center for Accountability Solutions at the American Association of School Administrators

(AASA). “If you don’t have a purpose in mind, it’s easy to get off track.”

A district should begin data collection by defining what it wants to know. Is the strategic

plan the district put in place three years ago improving student performance? Why are high

school freshmen in your district garnering low grades in English? Did the district make good on

its promise to the community that test scores would increase 5 percent over last year’s scores?

Not everything that is faced can be changed, but nothing can be changed until it is faced.

— James Baldwin,

author

Chapter 1: Why Data Matter

In Data Analysis for Comprehensive Schoolwide Improvement, Bernhardt lists seven questions to

help focus the early stages of data-driven decision-making:

• What is the purpose of the school or district?

• What do you expect students to know and be able to do by the time they leave school?

(Standards)

• What do you expect students to know and be able to do by the end of each year?

(Benchmarks)

• How well will students be able to do what they want to do with the knowledge and

skills they acquire by the time they leave school? (Performance)

• Do you know why you are getting the results you get?

• What would your school and educational processes look like if your school were

achieving its purpose, goals and expectations for student learning?

• How do you want to use the data you will gather?

Chapter 2 takes a closer look at different types of data that can help answer these

questions. Of course, the best questions are the ones superintendents, principals, teachers and

school board members develop together.

Making decisions based on data is a little like being a detective. Good data analysis requires

asking lots of questions, uncovering more and more information and revisiting hypotheses

along the way until a complete picture — supported by the facts — unfolds.

Challenging Assumptions Data-driven school improvement requires

administrators to challenge their own assump-

tions. Almost every district has common

beliefs about a school or groups of students.

One school may be known for its nationally

ranked reading program. Another may be clos-

ing the achievement gap among different

groups of students. But are these schools really

accomplishing what they think they are? How

do they know for sure?

Data help district leaders determine

whether their perceptions match reality.

Running a longitudinal analysis, for example,

will show whether a reading program is

You use data to inform the doctor of the progress of the patient. You determine whether the patient is progressing in a good direction — or is there additional assistance that the patient needs? That analogy works for me, because that’s really what you’re asking for: You want data that are more diagnostic, that permit you to monitor progress on a regular basis and that provide you with the student’s vital signs of learning.

— James Parsley, superintendent,

Vancouver (Wash.) School District

Using Data to Improve Schools: What’s Working

sustaining its impact over time. Disaggregating data by different student popula-

tions will show which students are excelling and which are falling behind. These

clues begin to form a picture of what is really happening in schools.

In the rural community of Antigo, Wis., 90 miles northwest of Green Bay, data

helped Superintendent Lance Alwin bridge the divide between educators and fami-

lies that homeschool their children. The work began when the Antigo Unified

School District reviewed progress toward its goals, one of which was to become “an

inclusive educational community.”

“One of the questions that I asked was why we weren’t providing services to

families of homeschoolers,” says Alwin. “The response was that homeschoolers are

on the religious fringes. They want to do their own thing and they don’t like what

we have to offer.”

It was a stand off, Alwin recalls. But if the district truly was to be an inclusive

community, it would have to tackle this issue head-on.

Data about the district’s homeschool population was collected and analyzed. The

district surveyed its homeschool population, asking parents why they home-

schooled their children.

What the district learned challenged long-held perceptions. First, data revealed

the number of families homeschooling their children was much higher than origi-

nally thought. The data also showed that religion was not the number one reason

parents homeschooled their children. Instead, parents said they felt they could do a

better job than the public schools. District leaders also were surprised to learn that

80 percent of homeschool families were interested in accessing instructional

resources provided by the district, as long as no strings were attached.

In response, district leaders created a unique charter school designed to meet the

needs of homeschool families based on the information and data they had gath-

ered. The school opened during the 1997–98 school year with four students. In

2000–01, 62 children enrolled in the school. “We’ve had to turn students away,”

says Alwin. “Now, we’re getting calls from homeschool families several counties

away because they have heard that this school district is willing to work with

them.”

The district is meeting its goal of becoming more inclusive. During the 1997–98

school year, 178 children who were eligible to attend the schools in the Antigo

Unified School District were homeschooled. During the 2000–01 school year, the

number dropped to 119, bucking national trends. Alwin says confronting the data

forced the district to examine new relationships. “This was rich data for us to

mine,” says Alwin. “It helped us determine how to create a charter school that was

of use to them, not us.”

Lessons Learned from Seattle Spokane (Wash.)

Superintendent Brian

Benzel learned the follow-

ing lessons using data to

drive decision-making in

the Seattle (Wash.)

Public Schools:

• Start small; don’t

overwhelm staff with

a “data dump.”

• Begin with the

core issues, such as

student achievement

in reading or

mathe-matics.

• Listen to what the

data tell about the

big picture; don’t

get lost in too

many details.

• Work to create trust

and build support by

laying data on the

table without fear of

recrimination by staff.

• Provide training

opportunities for staff

on how to use data.

• Be patient, working

with what is possible

in the district.

Chapter 1: Why Data Matter

Asking the Right Questions The more administrators dig for answers, the more questions emerge. As they analyze the data,

they begin to see patterns and trends. They may notice a cluster of schools doing better than the

rest of the district. Still more questions surface. What are these schools doing differently? Can

these best practices apply to other low-performing schools in the district? Why or why not?

Those who have experience using data say some questions will elicit more substantive infor-

mation than others. In Using Data for School Improvement, a 1998 report by the Annenberg

Institute for School Reform, Kate Jamentz, WestEd’s director of programs in professional and

organizational learning, says:

For example, questions that ask “which” are better than those that ask “how

many?” Asking which students are not meeting the standards in reading is bet-

ter than asking how many students are meeting the standards. It is important

to look at the types of questions we’re asking because the nature of a question

determines next steps in data collection and analysis.

Phil Streifer, associate professor of educational leadership at the University of Connecticut

and a data specialist, calls an in-depth line of questioning the “drill down” process. (See Chapter

2 for more on the “drill down” process.) Streifer says the drill down process “starts with a

global question or issue, which is then broken down into its component parts for analysis.”

Streifer says once the data analysis is completed, the team can make “a reasoned decision on

next steps.”

Newark (N.Y.) Central School District Superintendent Robert Christmann agrees. “Data are

only as good as the questions you are asking. You need your questions to be as specific as possi-

ble so you can know whether your efforts are successful.”

Key considerations when formulating questions include:

• How should student achievement be measured in the district?

• Are goals for student achievement based on data elements aligned with what the

teachers teach?

• What are the best indicators of student achievement upon which the district should

base its decisions?

• What indicators of student achievement are collected regularly throughout the year so

that informed decision-making can occur?

Computer Software and Data-Driven Decision-Making Companies are creating software tools for superintendents and staff who are engaging in data-

driven decision-making. AASA’s Center for Accountability Solutions advises consumers to look

for software that offers flexibility to customize district data reports. These reports should allow

district leaders to review data for different student groups over time.

Using Data to Improve Schools: What’s Working

“Through our experience, we have found that each district’s unique needs will determine

which tool is best,” says Geannie Wells, the center’s director. “There is no one-size-fits-all

solution.”

Be sure to compare different software products. Talk with district staff, teachers and others

about what capability the software should have to meet their needs.

Districts should ask two sets of questions when purchasing data software — one of themselves

and one of the vendor.

District leaders should ask themselves:

• What types of data should be included?

• Who will need access to which data elements?

• Is a product accessible through a web browser attractive?

• Is the infrastructure available to house a web-based product?

• Is it feasible to use an application service provider (ASP)?

• Is it acceptable to house the data outside the district on nonschool-system servers?

• How many records and fields are needed?

• How often will the district need to refresh data?

• Should the software allow the district to create groups, select assessments and generate

its own reports or should the district select software with existing templates?

After deciding what the district’s needs are, administrators can shop for software armed with

a list of consumer-savvy questions for the software vendor, such as:

• How will district data be transferred into the new software system?

• How many data updates are included in the software cost?

• Who is responsible for importing new data?

• What security features are built into the software?

• What are the limitations on fields and records?

• What type of, and how much, technical support will be provided by the vendor?

• How much on-site training does the vendor provide?

• When new software features are developed, are they provided free of charge or will

districts have to pay for an updated version?

• What type of database is required?

• In what format are the data stored?

• Can districts receive all of their data upon request?

Chapter 1: Why Data Matter

What’s Available? Increasingly, manufacturers of traditional student information systems are adding decision sup-

port services. It is wise to research what features the district’s software vendor might include in

the future before purchasing new software or completely overhauling existing technology.

Lots of data-specific software options also are on the market. Some packages offer a suite of

software tools designed to help districts collect and analyze disaggregated data. These tools can

collect and report longitudinal data on individual students, including demographic and testing

information, or track different areas, such as safety and security, curriculum and instruction

or parent involvement.

Other web-enabled software allows districts to

collect and analyze data that help them report on

student performance and identify solutions. Still other

software offers an analysis of questions tied to equity,

as well as comparisons that highlight the effectiveness

of programs, grade-level groupings, technologies

in the classroom and more.

Software prices vary widely. Some companies

charge a fee based on enrollment, while others charge per school or site. Some districts require

more customization than others, which drives the price up. Some software packages, such as

Quality School Portfolio (QSP), are free at this time.

A listing of many such products is available from AASA’s Center for Accountability Solutions

at www.aasa.org/data.

Homegrown Efforts While some districts prefer to use commercial software, other districts are developing home-

grown models using existing software tools to store and access data. Like their commercial coun-

terparts, these tools disaggregate data and conduct statistical analyses. Typically, they can store

multiple indicators of student achievement, everything from grades to dropout rates to annual

assessments and more.

As chief operating officer for the Seattle (Wash.) Public Schools before becoming superintendent

in Spokane, Wash., Brian Benzel and his staff worked to create a “data decision support system.”

“First, we worked to create a profile of a Seattle Public Schools graduate,” Benzel says. “The pro-

file reflects a series of statements about what we wanted our graduates to accomplish, including

the importance of communicating clearly, thinking critically and working productively in teams.”

The district also examined its grade-level standards in mathematics, reading and writing to

make sure students were learning the skills and knowledge they needed to reflect the Seattle

Public Schools Graduate Profile.

Time is money in a school district. Using a computer software program that helps collect and analyze data allows us to spend that time and money more wisely.

— Roland Smit, superintendent,

Mobridge (S.D.) School District

Using Data to Improve Schools: What’s Working

Benzel and his staff used multiple indicators to gauge whether teachers were using the most

effective instructional practices and what impact those practices were having on student achieve-

ment. The indicators are grouped into five categories, including academic achievement and stu-

dent behavior. Test scores, attendance rates, truancy rates, student climate surveys, staff surveys

focusing on areas such as teamwork, leadership and curriculum, and other data are used to

measure overall progress.

Benzel began developing a “value-added school indicators profile” that uses longitudinal data

to predict student achievement. “We wanted to make sure that a year’s worth of instruction was

resulting in a year’s worth of gains,” Benzel says.

The stakes are high in Seattle, where parents decide

which school their child will attend. Funding follows the

student. “In a choice system like [Seattle’s], customers

need to know how schools are doing so they can make

wise choices,” Benzel says.

In Poway, Calif., just north of San Diego, the school

district has taken another “homegrown” approach to

data-driven decision-making. This suburban, upper-income district of 32,000 students began its

foray into data-driven decision-making in 1996. The district had completed an intensive strate-

gic planning process, but was struggling with how to measure whether it was achieving its

vision, mission and core values.

Director of Learning Support Services Ray Wilson says data analysis began with identifying

what the district wanted to measure. Charting student progress was at the top of the list. “For

example, there’s research that shows that students who aren’t reading at grade level by third

grade will have problems later in school,” Wilson says. “So, one benchmark we set was that all

third graders would be reading at grade level before going to fourth grade.”

Decisions also were made about who would collect what data. The district agreed to collect

benchmark data (e.g., third grade reading scores, fourth grade writing scores, fifth grade mathe-

matics assessments, etc.) and provided school leaders with training and tools to help them col-

lect and analyze their own annual data. A part-time data analyst was assigned to every school.

“A principal does not have time in her day to sit and analyze data,” Wilson says. “You can look

at the data and see a problem, but investigating the ‘whys’ behind it can take hours and hours.”

Data-driven decision-making has prompted Wilson and his colleagues to redefine what quali-

fies as data. Early on, they largely thought of it as student performance data — state and local

test scores. It wasn’t long before they realized they needed to broaden their definition.

“For example, in a district like ours with a large white population, it’s easy for the majority

student population to mask the problems of minority students,” Wilson says. “Our students

score at the 80th percentile on norm-referenced tests, but when you disaggregate the data by

race, gender, ethnicity and the number of years a student has been enrolled in the district, you

We wanted to make sure that a year’s worth of instruction was resulting in a year’s worth of gains.

—Brian Benzel, superintendent,

Spokane (Wash.) Public Schools

find small pockets of students performing at the 30th and 40th percentiles.” In other words, it’s

not just about tracking how well students are performing on tests, but determining which

students are not doing well and why, and then seeking improvements.

Using data wisely requires clarity of purpose, time and a desire to seek and understand

improved educational achievement. Ultimately, superintendents say that data have made their

job easier because data help them determine whether the policies approved at the district level

are having the intended impact in the classroom.

Summary Key points in Chapter 1 include:

• Data provide quantifiable proof, taking the emotion and rancor out of the decision-

making process.

• Determining what data to collect is based largely on first figuring out what is important

to know about student performance, teacher quality, parent and community satisfac-

tion and other issues tied to district goals.

• Data-driven school improvement helps superintendents and others know whether the

district and its schools are realizing their vision and purpose.

Chapter 1: Why Data Matter

Often, the term data is confused with the term statistics, but the two are not interchange-

able. Data go beyond numbers, averages and percentages. They are the raw materials for

effective decision-making. But to make the right decisions, superintendents need the right tools.

No single set of data or collection method applies to all school districts. While districts might

share common goals, demographics and achievement

levels, they are not identical. Therefore, it is essential

that districts consider their unique traits when devel-

oping their data systems. Such consideration will

invariably help their efforts to improve student

achievement.

There are some commonalities in all data collec-

tion. Successful integration of data-driven decision-

making into educational strategy requires a team

approach. That teamwork starts with the relationship between the board of education and the

superintendent. The district’s expectations must be clearly articulated, measurable — and attain-

able. The roles of the school board and school staff must be clearly defined. And the superin-

tendent — who ultimately is responsible for coordinating the data collection, regulation and

reporting — must work collaboratively with the board, staff and community to lead the district

toward improved student performance.

Chapter 2: Using Data to Make Smart Decisions

• What different types of data should superintendents use when assessing student performance?

• How can data analysis effectively target student achievement gaps?

• Which methods work best for efficient — and accurate — data collection?

• What analytical methods can school administrators employ so they are confident that they are interpreting data correctly?

• How can superintendents develop an effective accountability program that is supported by staff, parents and the community?

Collecting data without purpose is meaningless. — Theodore B. Creighton, author,

Schools and Data: The Educator’s Guide for Using Data to

Improve Decision Making

Data Types and Applications Before student performance can be improved, it must be defined and measured. Indicators of

student performance include:

• Test scores

• Rigor of coursework

• Graduation rates

• Attendance rates

• Promotion rates

• Rates of participation in co-curricular activities (including community service)

Data can be measured on many levels. By looking at aggre-

gate data, superintendents can form a general assessment of a

particular curriculum, class or school. But breaking down, or

disaggregating, data provides a more complete view. It goes

beyond the “what” and tries to discover the “why” and “how.”

For example, 32 percent of ninth graders at Elmwood High

School passed advanced algebra. Aggregated, the data do not

bode well for the advanced algebra program. When the data are

disaggregated, a more complete picture comes into focus.

By asking more questions — What percentage of ninth graders took advanced algebra? What is

the program’s gender breakdown? Are any of the students in an English as a Second Language (ESL)

program? Is this the first honors course for any of the students? — superintendents get more com-

plete answers. Time, staff capacity and cost factors prohibit unlimited disaggregation, but asking a

few, specific, thoughtful questions provides better data. And better data lead to better decisions.

Collecting Existing Data Data collection starts with this question: What in particular does the district want to learn?

Once this question has been answered, Phil Streifer, associate professor of educational leadership

at the University of Connecticut, recommends conducting a data audit to determine if the need-

ed data are available. When beginning to search for the answers, superintendents should first

look at their own information. Districts already keep track of certain data in different ways and

in a variety of places such as teachers’ files on students, individual school offices, district offices,

even board member files.

Existing data sometimes are found in digital format on spreadsheets. This is often the easiest

and most efficient way to maintain data. While many spreadsheet applications are available,

superintendents should consider using a program that incorporates or easily converts this data

to graphics. The ability to present data graphically is essential — especially when dealing with

community stakeholders and the media. (See sidebar on p. 34.)

Using Data to Improve Schools: What’s Working

Learning is not attained by chance, it must be sought for with ardor and attended to with diligence.

— Abigail Adams,

letter to John Quincy Adams

In many cases, if the data are not immediately at hand, superintendents

do not have to look far to find answers. “A lot of valuable data can be found

if districts look at their own practices,” says Theodore B. Creighton, associ-

ate professor of educational leadership at Sam Houston State University.

“There are a lot of data that we should collect that we don’t collect.” For

example, tracking parental involvement — hours spent at school, Parent-

Teacher Association participation, and so on — can easily be accomplished.

That leads to another question: Are the data important for school

improvement? Perhaps the parental involvement data are essential for one

school, but the issue is not important for another. Even in the same district,

schools have different data needs — and different ways of collecting data.

Collecting New Data Although some data will already be available, not everything is easily acces-

sible, and Streifer advises superintendents to consider the cost benefits of

collecting new data before proceeding. “Many districts often want to add

student tests to match state mastery testing in the ‘off’ years for longitudi-

nal analyses. A decision should be made concerning the cost benefit of

adding these tests in terms of lost instructional time and the cost of pur-

chasing, administering, scoring and analyzing these tests.”

How do superintendents know what new data are worth generating?

“This depends on the mission-critical nature of the question under consid-

eration,” says Streifer. “If the data needed are the only way to address the

problem, and the question is mission-critical, then, yes, they would be

worth generating.”

There are other considerations beyond cost:

• Are the data being used to achieve a district goal?

• Are the data the only way to address the problem?

• Given that it generally takes six months to three years to generate,

collect and analyze data, can the district wait? Will the data still be

relevant to the question?

Chapter 2: Using Data to Make Smart Decisions

Examples of Targeted New Data Sources Every district collects and ana-

lyzes data on student test scores

and grades. But there are other

ways to measure student per-

formance, such as:

• surveys and questionnaires

(of teachers, students, par-

ents, employers, community

members, etc.);

• interviews or focus groups

(with the same groups

mentioned above);

• teacher logs/diaries;

• classroom observations of

actual instructional practices

and student responses;

• alternative assessments (e.g.,

work samples, portfolios,

senior projects and perform-

ance tasks); and

• locally developed pretests

and posttests.

Qualitative information, which

describes what people actually

say, do, think or feel, puts a

human face on quantitative data.

It provides depth and detail and

may increase a person’s under-

standing of a situation.

Source: At Your Fingertips: Using Everyday Data to Improve Schools, 1998

Testing Results Are More Than Grades The most widely used — and publicized —

method of assessing student performance is

testing. But testing is not a one-size-fits-all

endeavor. The different kinds of tests —

each with their own guidelines — can be

subject to different interpretation.

• Norm-referenced tests, the most com-

mon form of educational achieve-

ment tests used, compare individual

student performance with a national

sample of others in the same grade.

Results often are described as “stu-

dents scored in the 62nd percentile,”

meaning that they scored higher than

61 percent of the students who took

the test (Iowa Tests of Basic Skills,

Stanford Achievement Test, etc.).

• Criterion-referenced tests measure an

individual’s performance against a

well-specified set of standards. Results

often are described as “62 percent of

students met or exceeded the stan-

dards” (National Assessment of

Educational Progress, state proficiency

tests, Advanced Placement exams,

etc.).

• Performance tests require students to

demonstrate their abilities (via portfo-

lios, presentations, experiments, etc.).

• International tests, though not tech-

nically a test “type,” have gained

prominence by comparing student

performance by country (Third

International Mathematics and

Science Study, etc.).

Using Data to Improve Schools: What’s Working

Data Collection to Support Student Achievement Plainview, Okla.

Data collection is not a one-

size-fits-all process. And it is

not limited to those districts

where student performance is

below standard. High-

performing schools are

finding that data collection

and analysis can help them

build upon an already solid

foundation.

Plainview School in south-

west Okla. is a rural PreK–12

school with approximately

1,200 students. The school

began extensive data collec-

tion in 2000.

“The most difficult part

was finding what we wanted

to analyze — and working

with others on resistance to

change,” says Lisa Moore,

Plainview’s principal. “The

easiest part was making the

decision to look at our educa-

tion process in order to

improve.”

Plainview’s accomplish-

ments are impressive. In a

ranking of Oklahoma’s 547

public schools, the school

came in 11th in efficiency and

18th in student performance.

Several nationally certified

teachers are on staff and, in

1999, Plainview was recog-

nized by Expansion Management

magazine as a “Blue Ribbon

Scholar School.”

Moore thinks that, in

some ways, rural school dis-

tricts have data collection

advantages over larger dis-

tricts. “Rural schools typically

are smaller, thereby making it

easier to obtain clear, close-

to-the-student data,” she

says. “We have unique oppor-

tunities — like close commu-

nications — to facilitate data

collection and usage.”

Plainview uses observa-

tions, surveys and standard-

ized test score analysis to

collect data. The data are

presented through reports

on individuals, classes, grade

levels and school buildings.

Moore emphasizes that

the purpose of the data col-

lection and analysis is to

enable the school to make

appropriate decisions about

curriculum structure and

alignment. “The school

board, with the help of the

administration and faculty,

uses results to ensure that we

are meeting the needs of our

students and steadily increas-

ing our school improvement

process to the maximum

potential,” she says.

It is important that tests be reliable and valid. Reliable tests are consistent; when different

individuals review their results over time they arrive at the same conclusion. Defining validity in

a test is not as precise. In essence, a test is valid if it measures what it claims to measure.

Here’s another way to think about it. Let’s say a man steps on his bathroom scale every

morning, and the scale reads 240 pounds. Each morning the result is the same, meaning the

scale is reliable. When the man heads to the doctor for his annual check-up, he is weighed on

the doctor’s more accurate scale,

and the results are different. This

scale shows that the man actual-

ly weighs 245 pounds, 5 pounds

more than on his scale at home.

Although the man’s scale at

home is reliable, it is not valid

because it consistently underesti-

mates his weight. All measures

are subject to the same types of

errors. Verifying both reliability

and validity is critical to ensuring

that tests actually are measuring

what they purport to measure.

When data show gaps in stu-

dent performance, they show

more than where a school or dis-

trict did not meet its goal. When

analyzed, these data show which

student groups need more

improvement, thereby enabling

districts to address students’

needs. For example, Elmwood

High School’s goal was for ninth

graders to score in the 60th per-

centile on the state mathematics

test. That school year, the stu-

dents scored in the 50th per-

centile. While the school did not

meet its goal, the results were

not considered a failure, and the

district set about to improve

overall performance. Further

Chapter 2: Using Data to Make Smart Decisions

The Annenberg Institute for School Reform has developed the Inquiry Cycle, six activities that attempt to capture an ongoing, nonlinear process that involves the interaction of reflection and action. Each activity is essential; none may be omitted. The inquiry process is not over after completing one cycle. The six activities are:

Establish Desired Outcomes

Make Meaning of the DataTa

ke A

ct io

D efine the

Q uestions

As se

ss &

E va

lu at

e Ac

tio ns

Collect and O

rganize D ata

The Inquiry Cycle

Source: Annenberg Institute for School Reform

analysis showed that female students scored in the 40th percentile, while

male students scored in the 60th. The question for the district changed:

Why is there a difference in performance between male and female

students, and how can this gap be closed?

Indicators of student achievement also go beyond academic attainment.

They include “softer” measurements. Qualitative measurements,

in a sense, humanize the data by going beyond the numbers. These “quality-

of-life” indicators include:

• job skills and preparation;

• citizenship (volunteerism, voting, etc.);

• appreciation of the arts;

• development of character and values (integrity, patriot-

ism, work ethic, etc.); and

• healthy lifestyle.

Interpreting Data Education reform is — and will continue to be — scrutinized by

the media, politicians and community stakeholders. Therefore,

it is vital that any data collected be interpreted accurately and

fairly. “Achievement gains tend to occur when a school ana-

lyzes student performance data and develops a plan of action based on that

analysis,” says Daniel Domenech, superintendent of Fairfax County (Va.)

Public Schools. “Conversely, schools that select programs based on ‘per-

ceived’ needs almost always meet

failure.”

The Vermont Department of Education recommends the following

when analyzing student performance data:

• Whenever possible, use data from the same sources from year

to year.

• Collect and analyze as much data as possible. Use multiple measures.

• Always consider the nature, size and characteristics of the group

being tested.

• Be sure to know what is being measured.

• Remember that large-scale assessment data are about the school’s

program — not just about the grade at which the assessment is

administered.

Using Data to Improve Schools: What’s Working

Even rich assessment tools are relatively impotent if not embedded in systems committed to the standards they address, hungry for the data they produce, and willing to challenge and redesign traditional practices and organizational structures that get in the way of each student [meeting] those standards.

— Kate Jamentz, director of programs

in professional and organizational

learning, WestEd

• Compare performance from different groups of students (disaggregated data). The

goal is to reduce differences in performance among groups while increasing excellence

for all.

Reacting to a single test score is perhaps the most common

mistake made when interpreting data. Longitudinal measure-

ment — conducted consistently from year to year — is neces-

sary to properly measure progress, growth and change. The con-

sistency of the data elements — necessary for any long-term

data collection and analysis — reduces confusion.

The level of analysis also can diffuse misinterpretation. The

“drill down” process is an effective method of disaggregating data. It begins with a general

question, then "drills down" the question into smaller and smaller parts. In Data Analysis

for Comprehensive Schoolwide Improvement, Victoria Bernhardt describes four layers of

disaggregation:

First-Layer Disaggregations

How many students are there?

Male vs. female

Limited English Proficiency (LEP) vs. non-LEP

Ethnicities

Lunch codes

Second-Layer Disaggregations

How have the demographics changed over time?

Increases vs. decreases in categorical variables

Third-Layer Disaggregations

What percentage of students are gifted, and are they equally distributed among genders and

ethnicities?

Fourth-Layer Disaggregations

How has the enrollment of LEP students entering the building changed over the years?

Do students with higher attendance get better grades?

Chapter 2: Using Data to Make Smart Decisions

If your data set is only test scores, you are doomed.

-— Suzanne Bailey, restructuring issues consultant,

Tools for Schools, August/September 2000

Using Data to Improve Schools: What’s Working

The Palisades School District is a 100-square-mile area in Upper Bucks County, Pa., with a population of approximately 15,300. It serves 2,200 students in three elementary schools, a middle school and a high school.

When Francis Barnes took the job as superin- tendent, he was charged with finding out why stu- dents received good grades on report cards but performed poorly on standardized tests. Then Barnes had to develop a plan to improve student performance.

The first task at hand was to collect existing data. “There were a variety of pieces of data in a number of places,” Barnes says. “The problem was that the data were not organized sufficiently to permit any analysis. Nor did the administrators or staff have training in the analysis of data for the purpose of improving student performance.”

Barnes and his staff developed an action plan that was aggressive, yet thoughtful and realistic. “We generated, by consensus agreement, a data- driven decision-making model,” he says. “Our data- driven process engages stakeholders in decision- making to attain district goals. This data-based decision-making vision required us to gather all the available data and analyze the information, seeing what we actually had and also what was missing.” This included mainly state test and standardized national test data.

The district then participated in a pilot project with the AASA Center for Accountability Solutions to use Quality School Portfolio (QSP). This project provided software for staff to input and organize existing data, and to develop other assessment measures to fill existing gaps.

“We believe that both qualitative and quantita- tive data are essential,” Barnes says. With this in

mind, the district applied a hospital model for assessing its educational health. “We used quantita- tive measures, such as test score results, as indica- tors of academic health in the same way physicians use numbers, such as temperatures and blood pres- sure readings, as measures of physical health. We also used softer qualitative measures, such as the result of interviews with students, in the same way physicians ask questions during physical examina- tions.”

Some of these softer qualitative measures are unique to Palisades. “Walk Through” data, a term coined by the district, are gathered by interviewing each student once each year using a faculty-devel- oped protocol. (See the list of questions boxed on p. 21.) “The interview results give us qualitative data as to whether students know what we expect of them in writing, reading analysis and math; whether they know how their work is scored; and whether they know how to improve it,” Barnes says. Responses are analyzed at the building level, thereby enabling faculty to assess whether students are learning what is expected. The data then are used to make necessary changes in curriculum and instruction.

As Palisades’ data-driven decision-making efforts continue, its data-collection methods have expanded. Barnes notes districtwide assessments in kindergarten through eighth grade are the direct result of data analysis. The district currently is for- malizing assessments in 9th through 12th grade.

The effort is producing results; Palisades School District students are scoring higher on standardized tests. For example, in 1998, Palisades High School students scored an average of 1270 on the Pennsylvania System of School Assessment (PSSA) reading test and 1250 on the mathematics test. (The state’s “proficient” score is 1300.) By 2001,

“Walking Through” Data Collection Palisades School District, Pa.

Chapter 2: Using Data to Make Smart Decisions

the students’ average scores increased to 1380 on both tests.

The district does not rely on just one data resource, however. “We use the state assessments in reading and math in grades 5, 8 and 11, and writ- ing in grades 6 and 9,” says Barnes. “While these are ‘good’ assessments… we were somewhat frus- trated in the 1996–97 school year that, while these tests measured student performance, we did not receive any data that would help us improve performance.”

The district subsequently adopted the New Standards Reference exams, which were adminis- tered for the first time in spring 1998 to students in grades 4, 8 and 10 in reading and mathematics. “These tests provide specific information by student, class and district for areas that need to be improved,” says Barnes. “We have taken this data seriously and provided the appropriate training to prepare our teachers to use the data to improve instruction.”

Barnes says the data indicated that students needed to work on writing conventions, reading analysis and interpretation, and math problem solv- ing. So he focused the first year on writing, the sec- ond year on math problem solving and the third year on reading analysis and interpretation. “Focus means that each year we provide training and time for administrators and faculty to work on imple- menting changes in their instruction based on what they have learned in the focus area,” says Barnes.

Faculty also have created quarterly assessments in math and language arts at each grade level to provide benchmark data to use for improvement. When students do not achieve on quarterly assess- ments, they participate in after-school and even summer-school tutoring sessions. The results are impressive. For example, in spring 1998, 33 percent of the high school students taking the New Stand- ards exams for reading analysis and interpretation scored at or above the standards. In 2001, that fig- ure was 64 percent. For mathematics problem solv-

ing, the number of high school students scoring at or above the standard increased from 18 per- cent to 43 percent.

Results also are evident in stu- dents’ accomplishments after they graduate high school. They are entering better postsecondary institutions; students from Palisades High School’s Class of 2001 now attend such prestigious schools as Boston University, The College of William and Mary, Franklin and Marshall College, Temple University and Le Cordon Bleu. And Barnes notes that the community is showing more interest in and support of stu- dents, staff and the district’s efforts to help all students fulfill their potential.

Elementary Reading Analysis and Interpretation Walk Through Questions Durham Nockamixon Elementary School, Palisades School District, Pa.

• Can you share a connection you have made with text this year?

• How did that connection help you understand the story?

• What clues do you use to make predictions when you read a story?

• What kinds of questions do you ask yourself before, during and after reading a book or story? Can you share a question you’ve written down?

• What do you do when you do not understand what you are reading? How would you help a friend who didn’t understand what he was reading?

Accountability Plans School districts have been mandated to improve student performance. Domenech, whose dis-

trict is responsible for more than 158,000 students, shares his experience: “To blunt the criticism

of a community that implied the schools constantly asked for more money but were never will-

ing to be held accountable, we devised a plan that allowed for objective measurement of our

performance and we agreed to be held accountable for our results, or lack of.”

Fairfax County’s plan includes a process for determining a biennial objective and measurable

targets that are part of a system that provides each school with specific and measurable targets

for student achievement. Rewards and sanctions apply to schools and students. For schools, the

rewards include bonuses; the sanctions include reconstitution. For students, there are promotion

and retention guidelines by subject and grade level. Students who fail to meet the benchmarks

are required to attend remediation programs and summer school.

Implementing a full-scale accountability plan is not easy. “It is an education process and a

change in culture,” says Domenech. “We had the immediate support of the school board and

the community. Getting the support of the administrators and staff was more difficult. Everyone

had an excuse as to why objective measurement of our performance was not fair.”

Armed with the right data and a full understanding of what the data show, superintendents

can effect change as well as measure progress. Says Domenech, “The use of gain scores tended to

neutralize the socioeconomic factors and did away with most objections. The reward system also

helped, although the sanctions created a great deal of angst.”

Ownership and Responsibility It is important to get community ownership and buy-in at the front end of developing an

accountability plan — a method by which districts, schools and students are evaluated and held

responsible for performance. Stakeholders should have input into deciding what counts and

what districts should measure. By including all stakeholders in

these decisions from the start, school districts benefit from their

support throughout the process.

In 1995, the Plainfield (N.J.) Board of Education adopted six

goals focusing on school reform. Those goals included setting

standards, identifying and establishing benchmarks, improving

evaluation of personnel and providing skills development, and

completing a community planning process that would result in

a comprehensive district improvement plan.

The district spent the next two to three years laying the

groundwork for standards-based reforms, according to Joshua P. Starr, Plainfield’s director of

accountability. Most of the efforts were aimed at building community engagement. District lead-

ers explained to parents, business leaders, and other community members what the reforms

Using Data to Improve Schools: What’s Working

All stakeholders take ownership and respon- sibility for our children’s achievement. We are convinced that community engagement is necessary in order to progress.

— Inez P. Durham, superintendent,

Plainfield (N.J.) Public Schools

were and how the reforms would improve student achievement.

They also listened closely to community concerns.

The district’s efforts were effective: In 1998, voters approved

on the first ballot a $33.9 million education referendum that pro-

vided the necessary resources for expanding the district’s educa-

tion reforms. That same year, the board adopted a specific goal to

deal with reforms based on research. “This meant that resources

had to be put into place to ensure the efficient, effective gather-

ing and analysis of data,” says Inez P. Durham, superintendent of

the district, which serves more than 7,200 students. In 1999, the

board established an office of accountability. “From 1995, it was

clear that all stakeholders had a responsibility for student

achievement and school reform — and would be accountable.”

Plainfield is required by a court case to have a districtwide

plan. The plan’s acceptance and success is contingent upon sup-

port from all stakeholders. Therefore, members representing the

community, administration, parents and other groups were given

the opportunity to participate in the plan’s development. The

resulting plan, approved by the board, has been well-received.

“The plan is designed to promote local ownership and collec-

tive responsibility,” Durham says. “It is locally driven and

identifies the outcomes and indicators that the Plainfield

community values and expects. The system is designed to sustain good practices, continual eval-

uation and continuous improvement based on reliable and comprehensive data. It is a system of

rewards and consequences.”

Plainfield’s accountability plan is implemented throughout the district via its Leadership

Innovation and Change Councils (LINCCs), which are site-based management teams that

include representatives from administration, teaching staff, support staff, parents, community,

business and corporations and, when appropriate, students.

In addition to a districtwide LINCC, each school has its own LINCC. The school-level LINCCs

measure progress toward achieving six non-negotiable goals. There are six scoring guidelines for

each goal. “It has been a challenge for all to become comfortable with following the procedures

for collecting and reporting accountability data and to score utilizing rubrics,” Durham says. The

district is helping LINCC members overcome those obstacles with dry runs, inservice programs

and workshops.

The plan has widespread support and interest. “All stakeholders take ownership and responsi-

bility for our children’s achievement,” Durham says. “We are convinced that community

engagement is necessary in order to progress.” (See Chapter 3 for more about community

engagement.)

Chapter 2: Using Data to Make Smart Decisions

Establishing Performance Targets Four main sources provide helpful informa-

tion when establishing performance targets.

Superintendents can:

• Review relevant state and national

performance standards.

• Discuss expectations with important

stakeholder groups.

• Compare students’ performance

with that of an exemplary school

or program.

• Compare students’ performance with

state and national averages.

Source: At Your Fingertips: Using Everyday Data

to Improve Schools, 1998

A Closer Look Phil Streifer Associate Professor of Educational Leadership, University of Connecticut

What is the first thing superintendents should consider when they develop a data-collection program?

Data collection should start with the questions they most want to address. Ask: What data do

I need to answer these questions? This will quickly lead to a list of the data needed —

something I call a data audit.

An issue of concern always is “What will it cost to collect the data?” I encourage superin-

tendents to consider the cost benefits of collecting data in terms of the value they will provide

in answering their questions. In some cases, questions may

need to be modified because the data will be too costly to col-

lect. For example, surveying large groups of constituents can

be time-consuming and costly. It may be appropriate to do so,

but only after thoughtful review of whether it’s absolutely

necessary or whether some other sampling might be equally

valuable.

There are different types of data. How do you distinguish

among them?

Data can be separated into three broad categories for analysis

and interpretation: input, progress and outcome variables.

Input variables are variables that students come to school with, such as their background, pre-

learning, socioeconomic status and so on. Input variables also can include teacher variables,

such as their level of education or previous in-service training experiences. Process variables

are those that currently are at play on the issue under consideration, such as the curriculum

in use, instructional techniques and materials. The levels of teacher and parent participation

in decision-making also are process variables. Outcome variables are the results or measures of

both input and process variables, typically student tests, observation data and survey results.

A number of school districts do not have a districtwide accountability plan. Where should they begin?

I would start with the district goals and build an accountability plan around those issues.

Second, I would make sure that there is a high degree of match between the data needed and

the data collected. Generally, this issue of match is first measured by the data’s reliability (abil-

ity to consistently yield the same result over time) and validity (the degree to which the data

measure what they purport to measure). Finally, match is determined by the level of appropri-

ateness of the data to your population.

Using Data to Improve Schools: What’s Working

I think it’s also important to desensitize the annual reporting of high-profile data by having regular reports to the board, staff and parents on student performance covering a wide range of variables.

— Phil Streifer, associate professor of

educational leadership, University of Connecticut

What types of data are necessary to create an effective accountability plan?

I would focus first on district/board goals or mission-critical problems. Next, I would generate

data that allow for longitudinal analyses — over several cohorts (groups of individuals having

a statistical factor, such as grade level, race or age) if at all possible. Thus, if mastery tests are

administered in the fall, I would look to a comparative assessment in the spring, or one that

would allow for fall-to-fall analyses. Overall, longitudinal analyses yield the most useful infor-

mation. Beyond this, I would ensure the data you have or are about to collect will provide

you with the information needed. Poorly constructed surveys, for example, yield no useful

information at all. Tests that are too easy or too hard also have limited utility.

Board, staff, parent and public support are essential. How can school administrators get community

buy-in for such a plan?

I don’t think you can get support after the fact — that is, when the results are in. Early on, we

need to engage these folks in why these data are needed and in helping them to understand the

data’s limitations and value. I think it’s also important to desensitize the annual reporting of

high-profile data by having regular reports to the board, staff and parents on student perform-

ance covering a wide range of variables. With consistent, regular updates from the district, the

once-a-year state report may not take on the degree of importance it sometimes does now. I

have found all of these constituencies to be thoughtful and reasonable once we take the time to

explain the complexities and how the data are used to gauge progress and design improvement.

Summary Key points in Chapter 2 include:

• Districts should use multiple assessment and nonassessment measures when assessing

student performance.

• Disaggregated data is a powerful tool to help superintendents identify which students are

achieving, which students need additional assistance and how best to target limited

resources.

• Methods for data collection vary; many superintendents start with the question: What

do I want to know and what data will help answer this question?

• A districtwide accountability plan provides for objective measurement of performance

and holds boards, superintendents, principals, teachers, parents, students and others

accountable for results.

Chapter 2: Using Data to Make Smart Decisions

Superintendents understand the importance of parent and community engagement, strategic

communications and working with the media. They know that, unless they communicate

clearly and honestly about data and what the data reveal about student learning, their efforts

will fail and their credibility could be damaged

irreparably.

“School improvement requires a constant conver-

sation with our community,” says Beaverton (Ore.)

School District Superintendent Yvonne Katz. “The

community’s input informs our processes, procedures

and what we do. I’m convinced I wouldn’t be here if I

didn’t gather and use community input.”

Success in promoting and sustaining data-driven

decision-making as a tool for school improvement depends on:

• Educating parents and community members about what different data convey and

how to use data to measure the success of school-improvement efforts, budget deci-

sions, teacher quality initiatives and other practices.

• Asking teachers, parents and community stakeholders what their concerns are for

schools and what data they need to see to know schools are improving.

Chapter 3: Data, Public Engagement and Strategic Communications

• What data matter most to your community?

• In what ways is the community involved in discussing data, such as test results, school safety reports and teacher quality ratings?

• How is community input on how to improve schools gathered?

• How do teachers, staff and board members communicate with parents in the district?

• How is jargon-free data reported to the community, both the successes and the challenges?

• Who are the most credible messengers in the district?

• How can the district build an effective working relationship with the media?

To depend on the local newspaper editor or reporter to do all of your public relations for you is like depending on the hospital orderly to do your open-heart surgery.

— Brian Hale, member, Neb. School Boards Association

• Listening to the insights of these different stakeholders and reporting back to them

how their input may shape decision-making.

• Sharing the district’s successes and failures along the way.

• Speaking in clear, easy-to-understand language that is free of “educationese.”

• Understanding that the public schools belong to the public, and school leaders need

the public’s permission to do the work they are charged with doing.

This chapter explores three cornerstones of effective communication: public engagement,

strategic communications and media relations.

Public Engagement: Listening to Your Community How do superintendents proactively involve their communities in understanding and asking

informed questions about data? How do they determine which indicators matter most to various

stakeholders — parents, taxpayers, teachers, business leaders, senior citizens and others? How do

district leaders build support early on so that when controversial data are released, the public

says, “How can we help?” rather than assigning blame or finding fault?

“What’s really important is to clarify what the community’s role is,” says Superintendent Jane

Hammond of Jefferson County (Colo.) Public Schools. “We gather input from the community

and then we get together with educators to decide what we want to do to get the results the

community wants.”

Engaging the public begins with listening. It’s not

uncommon for an educator’s view of success to differ

significantly from the views of a taxpayer, parent or

business leader. (See Communicating Data to the

Public on page 32.) Yet each of these constituents has

a stake in public schools.

One of the most effective ways superintendents

and district staff can engage their public is by asking:

What would you need to see that would indicate our schools are improving? Other questions

include: How do we as a community define success? What elements are important in defining

student success? How do we as a community work together to ensure success for our young peo-

ple? The answers to these questions not only give superintendents a clearer sense of the public’s

priorities, but they also identify what data are most meaningful.

“When asked what their priorities are for spending money, our local community said ‘we

want a quality teacher in every classroom, including giving teachers the coaching they need to

improve,’” Hammond says. “Professional development emerged as a higher priority than reduc-

ing class sizes. I have never before seen the public be willing to support staff development.”

Using Data to Improve Schools: What’s Working

The fundamental purpose of any public engagement initiative is to channel a community’s concern, apa- thy or anger into informed and constructive action.

— Annenberg Institute for School Reform, Reasons for Hope,

Voices for Change, 1998

The most effective strategies for community listening are face-to-face meetings, such as:

• Focus groups, typically involving 8 to 12 people. Some focus groups are conducted with a

cross-section of people representing the community or a specific subgroup, such as parents

with children in public schools or recent high school graduates. A skilled, neutral modera-

tor leads the discussion using a script and predetermined questions. The moderator listens

for key themes, new ideas, differences of opinion and common ground.

• Community forums/town hall meetings,

which may attract anywhere from 60 to 200 peo-

ple. Typically, these meetings focus on one issue or

pose a question such as, “What should students

know and be able to do?” or “What does a good

school look like?” The information gathered at the

forum can be used to help inform the district’s

strategic-planning process, goal-setting or data-

gathering efforts.

• A study circle involving parents, educators, cler-

gy, students and other publics. Together, they use

dialogue to arrive at a decision about an issue or a problem. A study circle typically is made

up of 10–15 people who meet weekly for two hours over four weeks. A facilitator leads the

discussion, often drawing from a discussion guide shared with participants. During the final

session, the group decides how to address the issue and offers solutions.

• High-level advisory committees or superintendent leadership councils made up

of a cross-section of business people and community leaders who periodically meet with the

district superintendent. These committees offer guidance on key issues, provide a heads-up

about what they are hearing in the community and help brainstorm alternative methods of

public engagement. The councils can be issue-specific, addressing an issue such as how to

win support for an upcoming bond issue, or they can be more general and focus on broadly

defined areas such as leadership.

As school leaders begin to implement these strategies,

the challenge becomes how well they connect their

work to the priorities of the community. Lackluster pub-

lic engagement efforts can result in better packaging of

more of the same rather than increasing student learn-

ing by using good data, gathering community input and

making strategic decisions based on both.

Chapter 3: Data, Public Engagement and Strategic Communications

If you don’t listen to your community, you may make some assumptions that you are meeting their needs, but you are not. I find public engagement makes our decision-making easier, otherwise you are just guessing what the public wants.

— Jane Hammond, superintendent,

Jefferson County (Colo.) Public Schools

I don’t think you’ll survive without engaging your public. The public has to be in public education.

—Yvonne Katz, superintendent,

Beaverton (Ore.) School District

Using Data to Improve Schools: What’s Working

Since Superintendent Clifford Janey arrived in

Rochester, N.Y., in 1995, his work has been guided

by a simple but powerful idea: If community

members and educators have a shared under-

standing of what schools should do, they can work

together to make that happen. Realizing the

promise of that idea has not been easy, but the

payoff has been enormous for the district’s nearly

37,000 students.

At the start, Janey found a community demor-

alized by a history of low achievement. Experiment

after experiment had been tried, but no “silver

bullet” of school reform had pulled the communi-

ty together to improve schools. Janey believed that

Rochester was “idea-rich and focus-poor.”

To help build a common understanding of

what the community wanted from its schools and

how that vision could be made a reality, Janey

invited the community to help set specific, meas-

urable goals for schools. He met with scores of

Rochester residents and community leaders to

learn their priorities for the district.

Next, Janey worked with the board of educa-

tion and district staff to formulate a set of per-

formance benchmarks — goals for the district to

meet by 1999 — based on the public’s concerns.

Detailed in the plan were 40 specific indicators of

success in the four major areas that community

members had highlighted:

• Student achievement and citizenship

• Safe and supportive schools

• Community-school partnerships

• District accountability

But the community’s participation didn’t stop

there. As part of this plan, the district secured

commitments from community partners, including

city and county government, social service agen-

cies, colleges and universities and the private

sector. A coalition of local groups sponsored an

annual event to chart progress, raise awareness

about these efforts and share information on

district performance.

During the next couple of years, Rochester

made progress toward nearly all of the bench-

marks, and public sentiment toward the district

improved. When Janey decided it was time to

revisit the district’s strategic plan, he stuck with

the strategy of community involvement that had

worked in the past.

To assess whether the original benchmarks

were still relevant, the school conducted focus

groups and interviews with more than 200 teach-

ers, administrators, parents, civic leaders, employ-

ers, religious leaders and local politicians in 1997.

Community members voiced support for the

benchmarking initiative, but they said the original

benchmarks were not ambitious enough, not

streamlined enough and not focused enough on

core academic skills.

Rochester education leaders spent much of

the next few years processing this feedback,

retooling the strategic plan and gauging the

district’s progress. The district also conducted

public opinion surveys to evaluate the attitudes

of parents, students and school staff about

district efforts.

Engaging the Public in Strategic Planning Rochester (N.Y.) City School District

Chapter 3: Data, Public Engagement and Strategic Communications

Based largely on community input, the district

unveiled revised benchmarks in the 2000–01

school year. District officials had cut the number of

benchmarks and honed their focus. These revised

benchmarks revolve around two goals:

• All students will meet high standards in

reading, writing, mathematics, science and

social studies.

• All students will graduate prepared for

civic participation, higher education and

the world of work.

Buttressing these goals are two new, specific

benchmarks that describe the progress the district

is expected to make between the 1999–2000 and

2004–05 school years:

• Within a given group of students who

enter high school together in ninth grade,

the percentage of students receiving a

diploma within four years will increase

from 44 percent to 90 percent.

• The percentage of students receiving a

Regents diploma with honors (indicating

those students passed the New York

Regents exams with an average of 90

percent or higher) within four years will

increase from 3 percent to 15 percent.

Rather than erecting walls to shut out the

community, Janey and other Rochester school offi-

cials repeatedly invited the public to join in the

work of strengthening education. The results have

been sweeping changes and steady improvements.

The district has not yet met all its goals, but it has

made progress toward every benchmark set by the

community — and it remains committed to com-

munity engagement as a way of guaranteeing that

public schools serve the public.

Excerpted from Danforth Foundation Report,

Rochester City School District, N.Y., 2001.

Crafting a Communications Plan The following are vital elements of a communications plan:

• Messages: What do we hope to achieve as a result of communicating?

• Audiences: With whom do we want to communicate?

• Results: What measurable effect do we hope to achieve?

• Medium: What communications method do we want to employ?

• Timeline: By what date do we want to be successful?

• Expense: How much are we willing to spend on communications?

• District and school staff: Who will be responsible for successful communications?

Engaging Business Leaders Who knows a community better than the community itself? The Florida Chamber of Commerce is exploring new ways of listening to its community. One idea is to use volun- teers as “listening posts” who keep their ears close to the ground. Bus drivers, hairdressers and others in the community could make it their job to pick up rumors circulating about the school district. When needed, they could stop the rumor before it can spread by offer- ing the “real story” about student achieve- ment, teacher recruitment, a bond issue or whatever the hot-button topic might be. These “listening posts” could also be con- vened periodically to discuss issues as a group or report to each other as needed.

Communicating Data to the Public An effective way to build public support and increase

community confidence in public schools is to show the pub-

lic how schools are being held accountable for results. Sharing

data in easy-to-read charts and short, jargon-free reports helps

deepen community understanding of the issues facing public

education.

A common concern voiced by superintendents is that the

community reads limited coverage in the local newspaper

about its school district or that reporters only cover the “bad”

news. But too often educators make the mistake of placing

too much stock in media reports. Polls show parents most

often get their information from teachers, the principal, other

parents and their children. Districts play an important role in

disseminating factual, timely information to parents about stu-

dent achievement — the information parents care about most.

Increasingly, districts are reporting information directly to

their communities with the help of a school report card or

annual performance report. Many successful districts use the

report to guide discussions on performance and education

priorities at school forums, parent meetings, staff meetings

and board meetings. Training teachers and principals to help

facilitate these conversations ensures everyone is focused on

the data, what the data reveal about performance and how to

improve instruction in the classroom.

Using Data to Improve Schools: What’s Working

If we are interested in community building, then we, along with members of the proposed community, are going to have to invent our own practice of community. Despite the difficulties, if we are success- ful, our community will not be counterfeit but real.

— Thomas Sergiovanni,

Building Community in Schools, 1994

Who Has Credibility in Your District? Too often, superintendents count on the media to share district news and information with the public. While the media play an important role, surveys show teachers and other staff members have more clout with parents seeking information.

A 1996 survey conducted by the Denver-based Education Commission of the States shows teachers have the most credibility among parents. Here’s how the survey ranked who parents rely on “heavily” or “somewhat heavily” as sources of information about education issues:

Source: Listen, Discuss, and Act: Parents’ and

Teachers’ Views on Education Reform, 1996

88% Teachers

83% Children

82% Other parents

72% School officials

68% Print media

45% TV and radio

“We have spent countless hours and thousands of dollars training administrators on how to use

data,” Katz says. “We know who is doing well in what area and who is not doing so well. We talk

about the data and don’t push the numbers under the rug. You cannot be a smoke and mirrors

superintendent anymore in this country. People are very smart about wanting to see the evidence.”

A critical first step of reporting data to the community is finding out which data different

stakeholders care about most. Often there are discrepancies among constituencies. For example,

educators may want to see more demographic data, while parents tend to rank school safety and

achievement high on their list.

A national research project published in Education Week in 1999 (see Appendix A) asked par-

ents, taxpayers and teachers across the country what indicators they would use to hold their

schools accountable. Here is a sample of how parents, taxpayers and educators ranked their top

five indicators:

Parents Taxpayers Educators

School safety School safety School safety

Teacher qualifications Teacher qualifications Class size

Class size Graduation rates Teacher qualifications

Graduation rates Dropout rates Graduation rates

Dropout rates Percentage of students Per-pupil spending promoted to the next grade

Chapter 3: Data, Public Engagement and Strategic Communications

More and more school districts are using the Internet as a tool to communicate district data to their communities. Achievement data for stu- dents in Orange County (Fla.) Public Schools, Palisades (Pa.) School District and Plano (Texas) Independent School District are only a click away for parents, business leaders, teachers and any- one else who has access to the World Wide Web.

Orange County's website — www.ocps.k12.fl.us — boasts detailed school-by- school enrollment data, scores from state and voluntary tests, an online parents guide and detailed frameworks of higher achievement in all subjects at every age level.

As comprehensive is Palisades School District’s site — www.palisadessd.org — which posts results by grade and school for the stan- dards in reading, writing, mathematical skills, mathematical concepts and problem solving, as well as the percentages of graduates attending two- and four-year colleges.

District data also is a main highlight of the Plano School District website — www.pisd.edu. The district’s Department of Research, Assessment and Evaluation posts assessment results, the yearly district accountability report and detailed explanations of all tests adminis- tered in the county.

Websites as a Communications Tool

Strategic Communications: Getting the Most for Your Effort Districts have a wealth of data to share with their communities: dropout

rates, test scores, attendance rates, school safety data, budget figures and

other information. One of the biggest

challenges facing districts is determining where to begin. What informa-

tion is most useful to communicate? Superintendents do not have the

time or the resources to communicate with everyone about all of the dis-

trict’s initiatives. That is why it is important to be strategic. Simply said:

Set priorities, gather input early on, plan ahead, be proactive and always

evaluate the results of communication efforts.

Because so many different initiatives and programs compete for atten-

tion, it can be challenging for superintendents to keep their communica-

tion efforts focused on improving student performance. District leaders

who have turned around poorly performing schools say they worked

hard to keep their communication with staff, parents and the public

focused on achievement.

For example, if a district’s emphasis is on helping all students meet

standards, school visitors should be greeted with student work that

meets the standard, scoring guidelines outlining academic expectations,

and students who can explain what they need to know and be able to do

Using Data to Improve Schools: What’s Working

User-Friendly Report Cards Here are some helpful guidelines to ensure that a school’s report card is read and used.

• Keep school report cards short, such as a six-panel brochure. Have a more detailed version available for peo- ple who want more data.

• Add short narrative descriptions. School data are not self- explanatory to nonexperts.

• Advise parents and others how they can use the data. It’s not obvious to most.

• Spend as much time working on distribution as on pro- duction. Research shows that even most teachers say they’ve never seen district report cards.

Source: Reporting Results: What the Public Wants to Know About Schools, 1999

Graphical Representation “Graphics reveal data,” notes Edward R. Tufte in The Visual Display of Information. According to Tufte, graphics communicate complex ideas with clarity, precision and efficiency. Graphical displays should:

• show the data

• induce the viewer to think

about the substance rather than

about methodology, graphic

design, the technology of

graphic production or some-

thing else

• avoid distorting what the data

have to say

• present many numbers in a

small space

• make large data sets coherent

• encourage the eye to compare

different pieces of data

• reveal the data at several levels

of detail, from a broad overview

to the fine structure

• serve a reasonably clear pur-

pose: description, exploration,

tabulation or decoration

• be closely integrated with

the statistical and verbal

descriptions of a data set

at each grade level. Schools can illustrate the progress students are making on tests that measure

whether they are meeting the standards with the help of easy-to-read bar graphs and charts that

help explain what data the district is using to measure progress. This approach helps parents and

community members understand how these different efforts are connected. It also shows them

that they are receiving important and timely information on school quality.

Taking the Pulse of the Community A poll is a structured method to determine the level of support or intensity of opinion on an

issue, product or philosophy. Public opinion polling is one way to gauge the community’s views

on education issues. For example, educators can use a poll to quantify the level of support for a

proposed ballot issue or to test different options for a new start time for school. Results of

polling should be used to inform decision-making, not as the sole basis for decisions.

When considering whether to use polling, superintendents should think about the resources

required (it can be expensive), the importance of the information to the decision-making

process and the specific information needed so they’ll know more than they did before they

tested the waters.

Here are some key points to remember when

writing a poll:

• The way a question is asked can determine

the answer.

• It’s important to look for obvious questions

that are not asked by the pollster.

• Respondents are more likely to tell pollsters

they support plans favoring change,

even if they do not fully understand how

the proposed change will affect them.

• Not all pollsters make sure respondents

know enough about the subject of the poll

questions to give meaningful answers.

• There are many diverse publics, as well as

variety in thinking among specific sub-

groups of a given population.

Superintendents must research what each of

these constituencies thinks, feels and wants

before making major decisions.

Chapter 3: Data, Public Engagement and Strategic Communications

School Ambassadors The National School Public Relations Association

says the following school employees, in order,

are best known to the public:

• School secretaries

• Custodians

• Bus drivers

• School nurses

• Teachers

• Principals

• Superintendents

• School board members

Think of each employee as a school ambassador

to the community. Provide them with communi-

cation skills and training. They are among the

district’s most important messengers.

Working with the Media: Making Sure Your Message Is Heard Think back to the last interview you did with a reporter. How did what you said during the

interview translate to the printed page or television? Was your message heard? Working with the

media can be challenging, especially when you are trying to communicate complex data and

information that doesn’t neatly fit into a 10-second sound bite.

Before any media interview, know the opposing side’s point of view as well as you know your

own. Decide what the goals are for the interview. For example, do you want to raise awareness

about trend data that reveal improvement in student test scores? Are you sharing district strate-

gies for turning around low-perform-

ing schools? Identify your audiences

and which messages resonate best

with them. Stay positive, succinct,

solution-driven and in control of the

interview.

For more on media interview tech-

niques and related materials, see

Appendix B on page 54.

Summary Key points in Chapter 3 include:

• Teachers, principals, district staff and the community should be involved in gathering,

analyzing and discussing data.

• Success in promoting data-driven school improvement depends on educating parents

and community members about what information different data convey.

• Understanding what data the community uses to measure whether schools are improv-

ing helps superintendents avoid major disconnects in communicating information

with taxpayers, business leaders, parents and other community members.

• The most effective strategy for listening to community concerns is a face-to-face

meeting with different or multiple constituencies, such as a focus group, a community

forum, a study circle or a superintendent leadership council.

• Crafting a comprehensive strategic communications plan helps superintendents

identify key messages, audiences, results, effective tools (e.g., web-based data charts)

and timelines.

Using Data to Improve Schools: What’s Working

The public schools really are the public’s schools. If they are given the impression that they are welcome to participate only if they can do something educators think worthwhile, this puts the cart before the horse (i.e., treats citizens as a means) and disconnects the public from schools.

— David Mathews,

Is There a Public for Public Schools?, 1996

Capitalizing on recent advances in technology and research regarding educational improve-

ment, superintendents nationwide have developed countless strategies for using data to

drive decisions at the district and school levels. Whether they are data experts or novices,

however, they probably could be doing more to make this invaluable resource — data — work

for them.

How? Listen to the stories of superintendents and other education leaders from across the

country who have been using data to drive decision-making. More than anything else, their

experiences underscore the importance of team build-

ing. After all, leaders are only as good as their teams.

Superintendents illustrate the power of building a

data-friendly culture, ensuring that school board

members and staff understand their roles and respon-

sibilities, providing the training needed to foster new

data skills and establishing a system focused on

continuous improvement.

Changing the Culture Data traditionally have not been a major factor in the ways schools and districts make

decisions. The intuition of principals and teachers, advocacy by parents and political interests

often have guided decision-making. But all that is changing.

Chapter 4: Strategies for Success • How can the school district’s culture shift to one that encourages using data

to drive decisions?

• What roles should a superintendent, school board members, central office staff, principals, teachers and others play in collecting, processing, interpreting and using data?

• How will key players be trained to effectively participate in data-driven decision-making?

• Is the data system being built able to foster continuous improvement in the school district?

• What can be learned from other successful districts?

This is the critical point of data: It’s not necessarily the answers that are most important; it’s the discussion that occurs and the questions that are asked because of the data.

— Karen Bates, superintendent,

Lake Washington (Wash.) School District

The call for data-based accountability, trumpeted first by state education leaders, has been

met with skepticism by many of the people in schools who are being asked to collect and use

data. Teachers and principals sometimes react initially with fear, distrust and resistance. Their

first questions, if they dare to ask, concern how the data will be used against them.

“Teachers are afraid of data because they think their jobs may hang in the balance, based on

how their students are doing. Principals are worried about where they will get the time to ana-

lyze data,” says Superintendent Ray Yeagley, who has spent years easing the data-driven fears of

educators in the Rochester (N.H.) School District. “The superintendent has to help people see

the global picture about what data can and can’t do. I need to help my principals find the time

to make this a priority. I need to reassure my teachers about how it will and won’t be used, so

they feel it won’t be used unfairly.”

To overcome these hurdles, education leaders recommend:

• Taking every opportunity to show principals and teachers that data are not being

used to “get” them, but to improve student learning. Tom Glenn, superintendent of

the Leander (Texas) Independent School District, says he used every outlet he could

find — including the district newsletter, meetings, workshops and private conversations

— to get that message out. “We still encounter resistance, from time to time, to laying

the data on the table. But that happens less and less as our culture helps people under-

stand that we’re not asking these questions to harm people,” he explains. “You have to

act appropriately with the data and make sure people understand: We’re working on

the system.”

• Helping district faculty and staff understand that using data, much of which already is

public, is a shared responsibility. The Poway (Calif.) Unified School District has adopted

the philosophy that it is made up of “professional learning communities,” says Ray

Wilson, director of learning support services. “We all work together for the common

good. If one of us has poor test scores, it’s our job as a department to figure out why and

help that person get test scores up,” he says. To help people become more comfortable

and familiar with data, Wilson’s department devised a scavenger hunt for the district’s

part-time school-based data staff. “They

had to create a school profile that included

teacher credentialling rates, mobility rates,

test data and the number of years that the

teachers at each grade level had been

teaching,” he says. “All this information is

available on the Internet. Many people

were surprised how much information

about their school, which they struggle

every year to collect, is public.”

Using Data to Improve Schools: What’s Working

The importance of data for administrators, policymakers and teachers in the classroom — to be able to break data down and know where strengths and weaknesses reside — is crucial if you want to make any kind of improvement.

— Monte Moses, superintendent,

Cherry Creek (Colo.) School District

Chapter 4: Strategies for Success

Collecting the Right Kinds of Data Leander (Texas) Independent School District

The need for data has been felt acutely, even among school systems in Texas where the Texas Assessment of Academic Skills (TAAS) has set a national standard for data collection, says Tom Glenn, superintendent of the Leander (Texas) Independent School District.

In the early 1990s, as TAAS was taking hold, Glenn and other district officials realized that their teachers and principals needed something the state test did not provide: frequent updates on student progress that could be used to guide instruction and address problems throughout the school year.

Despite its strengths, TAAS tests students only once annually, near the end of each school year. Results are reported in the summer, when educa- tors are powerless to address achievement short- falls. What those shortfalls might be remains some- what murky, because TAAS reports on student per- formance in terms of broad categories rather than specific skills.

So Leander began exploring an additional test- ing option. Principals and teachers worked togeth- er for two years to revise language arts and mathe- matics curricula, then worked for another year to help develop a test based on those curricula.

The resulting Student Profile System tests stu- dents from kindergarten through high school in language arts and mathematics three times a year — at the start of the school year, in the middle and at the end. Principals, teachers and students receive reports, including easy-to-understand charts that show specifically where students excel and where they need to focus more attention.

“The first time we used the test, on the first day of the school year, we had elementary students in tears,” recalls Glenn. “They said, ‘I’m in the fourth grade, and I’m taking the fourth-grade test

at the beginning of the year — I don’t know any of

this stuff!’ It took us a while to help students under-

stand that they were not expected to know any of

this stuff yet — that we were just trying to find out

how much of it they did know, so we didn’t waste

time at the beginning of the year teaching them

things they already knew.”

Teachers and principals were concerned, too.

“We promised them: ‘We will never use this assess-

ment to evaluate you; it is for your use,’” Glenn

says. “We had to prove it over and over again. We

told people that the system is the problem; individ-

ual people are not necessarily the problem. We

called it ‘failing forward’ or ‘learning from our mis-

takes’ or ‘working on the system’ — we called it

lots of different things. We had to help people trust

and understand that, as we collected data and

found things that weren’t always positive, they

were not going to be hung out to dry.”

Glenn says the 13,200-student district contin- ues to use the system because it:

• focuses attention on learning, not testing;

• generates data that can be used to improve education, not just categorize students and schools; and

• allows students, as well as teachers, to manage the learning process better.

Principals, teachers and students report that the system is helping them work more effectively than ever before to improve learning — which is the purpose of testing, Glenn notes. “Once people understand that you will not use data to beat them up,” he says, “then they will help you collect that data and be very honest about it.”

Ultimately, to shift the culture of your district to not only tolerate but embrace data, you

must not rely on data so much that you discount the training, experience and expertise of the

people who you want to use the data, says David Benson, superintendent of Blue Valley (Kan.)

School District.

In the early stages of Blue Valley’s data-collection efforts, a respected administrator, now a

deputy superintendent, fumed visibly for several weeks before finding the words to voice his

frustration, Benson recalls. “He came out of a meeting and said, ‘Dave, you’re ignoring my intu-

ition. I feel this way about this — and because I feel this way, you have to listen to me.‘“ Benson

says, “It was such an epiphany for me to understand the importance of that point: You cannot

take a human institution and just crunch it to data points. You have to listen to people.”

Benson listened to the administrator, as well as to

other critics. As a result, the district’s data system has

been improved. Moreover, it has survived and thrived

as principals and teachers throughout the district have

made way for data in their work. Today, the 17,000-

student district uses local, state and national test data

in its accreditation process, school profiles and strate-

gic planning. But that did not happen overnight.

“It’s a gradual process of proof,” says Benson. “If

you can demonstrate that what you’re doing can

make a difference in the lives of the students we serve, then it will gain acceptance. If it is data

for data’s sake only, and is not related to our primary mission of providing an academic educa-

tion to children in a safe and orderly climate, then it will not gain acceptance.”

A Closer Look Spence Korte

Superintendent, Milwaukee (Wis.) Public Schools

Why do school districts need to be able to make decisions based on data today?

We have a history of going after money for a hot new project and running it for three or four

years, and then it becomes part of what we do. Nobody is really getting at the critical ques-

tions: We just spent over a billion dollars this year — is that money put in the right places? Is

it getting results?

In Milwaukee, we’ve been building a sophisticated network to collect student informa-

tion. Each of our schools gets a report on how their students are doing. It’s pretty practical

from a principal’s point of view. For example, how many ninth graders are coming in below

grade level, and do I need to change my hiring pattern so I have more remediation for kids

coming into my school? The information is online. People in schools can access all their

records 24 hours a day.

Using Data to Improve Schools: What’s Working

One of the primary responsibilities of a leader is to help people confront problems. The way you confront problems is you identify them, and the way you iden- tify problems is by looking at data.

— Terry Grier, superintendent,

Guilford (N.C.) County Schools

What has worked well for your district in terms of setting up a sound data system?

Driving our whole technology design and infrastructure is an ex-high school principal. He

comes with 24 years of field experience in knowing what information has to look like to add

value for schools. Instead of recruiting an information systems technician to manage this,

we’ve gone the other way and drafted a practitioner. The result is that our data tend to be

more user-friendly and more school-oriented than they might be otherwise. That’s something

to think about when you’re out looking for an IT person. You can always surround that per-

son with technicians, but if he or she doesn’t know what schools need, the fit you get might

not be what you want.

How important is data training for teachers, principals and

other district employees?

It’s absolutely critical. In retrospect, one of the things

that we did right was take an existing building and set

it up as a training center. We cycle people through

every day.

All our best technology teachers are there. Almost

all of them have a background in schools, so it isn’t

“techspeak” as much as it is, “When I was a teacher at

such-and-such a school, this was a big problem for me.

I bet you have that problem at your school. Let me show you how we solved it with technolo-

gy.” We have good credibility with the teachers.

We do our own training about 90 percent of the time. Sometimes we’ll have a vendor

contract where they may do the installation and 500 hours of training or something. But we

don’t rely heavily on vendors. In the technology world, the vendor can be on the top of the

heap at the beginning of the year, and out of business at the end of the year. We want to fig-

ure out how we can institutionalize technology as a major part of the way we do business.

How do you accomplish that? Who needs to be involved in the process?

You need to start with the end users, and they need to be educated about what the district- level folks need to make those decisions. You’ve got to get buy-in at each of those points along the way.

You always go through that part where you’re running the old system and you have a

new system ready to go. Ten percent of the people are ready to dump the old system and go

to the new system. You need to find a way to make it politically palatable to say, by a certain

date, “We can’t afford to run two systems. We don’t have the people for it. How can we help

you get to where you need to be?” It really does take a driver somewhere in the organization

who understands that the discomfort quickly gives way to comfort if you do it well.

Chapter 4: Strategies for Success

Businesses don’t keep data that’s useless, that doesn’t inform them of anything, yet in education we have data that just runs all over us. We have to target it and organize it in such a way that it informs us so that we can make better decisions.

— David Benson, superintendent,

Blue Valley (Kan.) School District

How do you get the teachers and principals to buy into data-driven decision-making?

You have to show how it adds value. And at some point, I just had to say, “I understand your

problem, but we’re going to do this. Get to it.” Part of leadership is that, at some point, you

have to be able to see where it’s going and have the courage to stay with it until it actually

works in all of your schools. A lot of people are not willing to pay the political price. You have

to be willing.

Unraveling Roles and Responsibilities The culture of a school district will not smile favorably on data-driven decision-making for long

if the members of that culture continually step on each others’ toes or fumble the ball when it’s

their turn to run with the data. That is why the superintendent must draw the game plan in

clear, bold strokes — to help board members, district staff, principals and teachers understand

their important roles in the data system.

“It all has to work in coordination,” says

Douglas Otto, superintendent of the Plano (Texas)

Independent School District. “Everybody has to

have the same vision of what we’re doing and

why. Everybody has to understand what kinds of

data are important to collect and how they’re

used. From the school board and superintendent

on down, there has to be that common sense

of purpose.”

The 47,000-student district’s data efforts appear to be paying off. Plano educators monitor

student performance on the Iowa Tests of Basic Skills, the Cognitive Abilities Test, the Texas Assess-

ment of Basic Skills and the Scholastic Aptitude Test. The district also collects and reports data in

areas such as staffing, spending and enrollment patterns to inform decision-making continually.

To keep such a data system humming along smoothly, Otto says, team players must under-

stand their special roles and responsibilities:

• Superintendent and School Board. They share leadership of the district's data

efforts but play significantly different roles. In Plano, the division of labor has been

clear: The board adopts policies to govern the system; the superintendent manages

daily operations. “The board made sure we had the appropriate policies in place to

make sure we were going to collect data, analyze it and use it to improve instruc-

tion,” says Otto. “Board leadership is extremely necessary.”

To make effective policy decisions, the board must understand data. For exam-

ple, the board needs to know that some data trends are the result of a “common

cause” (a regular feature of the school system, such as normal student attendance

patterns) and some are the result of a “special cause” (an unusual circumstance, such

Using Data to Improve Schools: What’s Working

There are no secrets to success. It is the result of preparation, hard work and learning from failure.

— Colin Powell,

U.S. Secretary of State

as a flu bug that causes many students to miss school on a big test day). Otherwise, the

district might spend an inappropriate amount of time trying to solve a problem that was

a fluke.

The superintendent’s role, then, is to empower central office and school staff to use

data as effectively as possible to meet the board’s mandates. Max Riley, superintendent of

the Lawrence Township (N.J.) Public Schools, says the superintendent should send staff a

clear message: “This is not a ‘gotcha.’ I’m going to give you every tool I can think of to

help you succeed.”

• District Staff. Central office and other

staff at the district level must be pre-

pared to discard longstanding ways of

doing business if data dictate that a

new approach is needed. In 1996, for

example, Plano brought in consultants

to conduct a curriculum audit to com-

pare district practices with established

standards. Based on the audit, Plano

revamped its curriculum management

team and expanded its research and

assessment team to help “make wise decisions about what data we’re going to collect,

who’s going to collect it, who’s going to analyze it, who’s going to be trained, and how it

gets put back into the curriculum and strategies used in the classroom,”says Otto.

• School Site Councils. In many districts, school-based management teams — often called

councils or committees — are taking up data along with the other tools they use to help

make decisions for their campuses. “We’d had site-based improvement committees in

place for many years,” Otto says, recalling the changes that followed Plano’s curriculum

audit. “All of a sudden, one of their chief roles was to monitor areas of improvement in

school, make recommendations, work with staff, and use the same data that principals

and teachers were getting so everyone was on the same wavelength.”

• Principals and Teachers. Though principals bear ultimate responsibility for what hap-

pens in their schools, they must work with teachers to use data in ways that best serve

their unique student populations and school needs. In Plano, principals receive assis-

tance — and are held accountable — for this work. “Principals receive data training. It’s

not voluntary; it’s mandatory. They have to know how to use data on their school-based

improvement committees. It’s part of their performance appraisal,” says Otto. “Principals

also need to play a monitoring role to make sure that teachers are implementing the cur-

riculum, using effective strategies and improving student achievement.”

Chapter 4: Strategies for Success

Data-driven decision-making is not someplace we’ve arrived; it’s still a journey and always will be. But it’s a journey where people understand how you buy the ticket and get on the train. And if you’re not on the train, you need to find another place to be, because this is the way we have to operate.

— Yvonne Katz, superintendent,

Beaverton (Ore.) School District

Yvonne Katz, superintendent of the Beaverton (Ore.) School District, says data must be dis-

tributed across grade levels, so elementary school teachers know how to prepare their students

for middle school level work, and high school principals know how well their incoming fresh-

men will be prepared to perform. "Everyone can see what the data are showing" in Beaverton,

says Katz. “As you look at groups coming through, you can plan what your focus will need to be

in future years, when the kids hit your level.”

The Evolving State Role As school districts become more sophisticated in their use of data, they are starting to look

beyond their own borders to appreciate the helpful role that state departments of education can

play. This outlook is reversing years of mistrust and resentment, begun when states started col-

lecting data that district officials worried would be used against them. As local school systems

have become more confident and comfortable with data-driven decision-making, they have

looked to the states for data training and other forms of technical support.

Katz retains a unique perspective on the state role, based on her experiences as associate com-

missioner for general education of Texas in the early 1980s. “We had a tremendous training pro-

gram in Texas, bringing in principals and vice principals to be trained for a whole week in data

usage, then visiting campuses around the state,” she says.

New York provides a different kind of support for districts. State assessment scores not only

are distributed widely, but are broken down to allow comparisons among similar school districts,

such as those that serve high percentages of students eligible for federal free or reduced-price

lunches.

“Some fairly helpful comparisons can be made for follow-up for the purposes of instructional

development for classroom teachers and principals,” says Kevin McGuire, who served as a super-

intendent for 18 years before becoming director of the New York State Center for School Leadership.

“There are extraordinarily talented educators in every corner of the state. By linking people

together, we think we can help everyone grow, the adults as well as the children,” McGuire says.

“For example, if people are having difficulty with students who are remedial in nature, they can

talk to a similar school district that seems to be meeting the needs of similar students in a more

effective way.”

Using Data to Improve Schools: What’s Working

Chapter 4: Strategies for Success

The superintendent generally:

• Translates the board’s vision for the

school district into measurable goals

based on data.

• Works with district faculty, staff,

parents and other community stake-

holders to craft plans for meeting

goals by certain dates.

• Collects data to show clear, steady

progress.

• Celebrates successes, evaluates

shortcomings and revises plans for

improvement based on data, along

with the board.

The school board generally:

• Establishes a vision for the school

district based on data showing what

has been achieved so far and what

progress is necessary.

• Spells out — for the superintendent

and other employees and stake-

holders — how district performance

will be evaluated.

• Reviews relevant data to evaluate

district progress toward identified

goals.

• Revises goals and plans for

improvement based on data.

Broad participation in improvement efforts serves to:

• Promote a high level of support for those efforts.

• Generate sound solutions by expanding the discussion.

• Motivate participants and their associates.

• Increase the likelihood that the effort will lead to constructive action.

• Prepare participants for their role in implementing improvements.

• Increase ownership of and commitment to specific strategies.

• Empower important stakeholder groups.

• Foster lasting, rather than temporary, change.

Who Does What? Data-driven decision-making, especially in the early stages, demands that district

leaders point the way. Superintendents and school boards both must play important

but distinct roles.

Source: At Your Fingertips: Using Everyday Data to Improve Schools, 1998

Training the Players Superintendents know that data-driven decision-making requires new knowledge and skills. In a

2000 AASA membership survey, superintendents overwhelmingly called for training on using

data. They also knew their staffs needed a crash course in collecting and analyzing data, using

data-processing technology, and operating the decision support system.

“I can sit up here in the superintendent’s office and look at the data and issue directives, but

then people don’t understand what I’m doing or why I’m doing it,” says Glenn. “You’ve got to

empower people to use data. They’ll help you identify problems — and often, through team-

work and looking at other ways of doing things, they will come up with solutions.”

To help schools use Leander’s new Student Profile System, which generates data on student

progress throughout the year, the district provided teachers with lots of training, much of which

directly involved them in helping develop the district’s student profile test. A specially trained

“data group” gathered data and reported it back to schools in user-friendly formats. Principals

received training in helping teachers use data individually, in grade-level teams and as a school.

“We did a lot of modeling with people,” says Glenn. “I met with principals three or four times a

year on their campuses to talk about their data: What’s going on? Where are the peaks? Where

are the valleys? What are they doing about the valleys?”

But Glenn does not advocate trying to make a data technician of every school employee.

More important is training that emphasizes the conceptual underpinnings of data-driven

decision-making — the core principles of how educators can use data to improve education.

“I can hire all kinds of people to come in and help us do the grunt work of collecting the

data, putting it into charts and those kinds of things,” says Glenn. “But as many people as possi-

ble in the system have to understand the philosophical basis of what you’re trying to accom-

plish. Sometimes that’s where we fail in education — in not helping people understand what

we’re trying to accomplish with all this paperwork we’re asking them to do.”

Implementation Models Depending on what type of training is needed, superintendents might want to explore

various ways of providing training. Should an outside vendor be hired to train staff, or should

this be handled internally? Should one person spearhead the effort? Or would it be best to

build a team?

Only you can answer such questions for your district. But successful superintendents and

other education leaders have learned the following lessons:

• Not all employees need to receive training, but employees at all levels need to learn

about data-driven decision-making. This helps build buy-in at all levels and ensures

that the system works the way it should.

Using Data to Improve Schools: What’s Working

Chapter 4: Strategies for Success

• Perhaps because data can be intimidating at first, districts often rely too heavily on

outside consultants. The result can be generic training that is poorly matched to the

specific needs or problems of a district. If a data effort is to be sustained over the long

term, the district must cultivate in-house trainers who can help colleagues use the

local data system.

• Actual district data should be used in training only when appropriate. If training focus-

es on data analysis, then using local data can provide a compelling context for partici-

pants. But if training is intended to help participants master new data software, local

data can be a distraction.

When in doubt, remember: Data also can help tailor training to the unique needs of a dis-

trict. Consider the technology self-assessment used by teachers in Blue Valley. Teachers complete

a 43-item questionnaire to determine how they use technology and how they might use it

better. The district uses that data to design professional development in technology. “Because

we have used data, we have targeted inservice to specific, identified needs, rather than just using

broad-brush technology inservice,” says Superintendent David Benson.

The return on investment for data training is incalculable. “No matter whether you’re a huge

district or a very small district, if teachers are trained and have some knowledge of how to col-

lect and analyze data, they can do some remarkable things in the classroom — without needing

some decision from on high passed down telling them what to do,” says Otto.

Creating a System for Continuous Improvement Increasingly, leading school districts are using data-driven decision-making to ensure continuous

improvement. They do so by moving beyond the initial work of assessing their own schools’

performance, comparing those schools against each other and monitoring trends over time.

These districts look beyond their borders — to benchmark their schools’ performance against

that of other top-performing schools across the country.

Benchmarking is more than comparing test

scores. Done well, benchmarking helps dis-

tricts learn what it takes to be effective, how

well they are using key strategies, where

improvements are possible or necessary, and

how they can learn from the best practices of

other successful districts.

A recognized leader in district benchmarking practice is the Western States Benchmarking

Consortium. Founded in 1997, when a group of education leaders from high-achieving districts

gathered to devise strategies for improvement, the consortium continually has brought together

superintendents and others from member districts to develop research-based benchmarks for

district success.

Meaningful school reform is probably more about using data effectively than about any other tool you can use.

— Douglas Otto, superintendent,

Plano (Texas) Independent School District

Using Data to Improve Schools: What’s Working

Looking for Value Added Plainville (Conn.) School District

In the three years he headed the Plainville (Conn.) School District, then-Superintendent Max Riley raised student scores on Connecticut state exams to record levels — but he didn’t do it alone. Riley recruited a ”stat jock” from Central Connecticut State University to help principals and teachers use data for improvement.

James Conway, a psychology researcher and professor, was contracted by the district to “act as our coach for thinking about data,” says Riley. Conway combed through reams of available data and analyzed performance trends over time and differences among schools. Then the researcher made himself available for confidential consulta- tions with principals about their schools’ data.

“I wanted principals to have access to some- one with whom they could have conversations without necessarily having to go through the superintendent,” Riley says. “It was understood from square one that these were confidential conversations and that Jim was just there to help principals in whatever they wanted to do, as long as it was focused on data and student learning.”

Providing that sense of security made a dif- ference. Principals latched onto Conway. They called with questions. They explored the data. They started seeing their schools in new ways.

For example, most people in Plainville tradi- tionally had focused on raw test scores — and ignored relevant data about the student popula- tion — in evaluating the success of the district’s three elementary schools. The school that served mainly children of white-collar parents routinely earned top marks. Two other schools, both of which served children of comparatively low- income families, fared less well.

But Conway toppled those perceptions with a value-added analysis. Essentially, Conway col- lected data on how well students performed when they entered each school, then examined how much progress that school enabled students to make. According to this analysis, the two schools with the lowest test scores added value to children’s education at double the rate of the school with the highest test scores.

“Our highest-performing school was under- performing in terms of moving kids along,” says Riley. “It sent shock waves through the organiza- tion. It turned everything upside-down in terms of people’s mythological understanding about who’s doing a good job.” The value-added analysis also helped the principals at all three schools better understand where they were pro- viding effective instruction and where they need- ed to work harder, he adds.

Now superintendent of the Lawrence Township (N.J.) Public Schools, Riley is setting up a similar system. Working with a researcher from a nearby university, Riley encourages principals to sift through data to see how they might increase their value added for students.

Working with college and university researchers can be particularly advantageous for relatively small districts like Plainville and Lawrence because districts can pay relatively little and get data services tailor-made to meet their needs. “Of course, the board loves it, because they feel like they’re making focused, informed decisions about how to spend the money,” says Riley. “Data also give principals the ammunition to come in at budget time and say, ‘Look, it’s your own darn data system showing where I need help. Now, I need help.’”

Field-testing those benchmarks during the 2000–01 school year, the consortium’s seven mem-

ber districts conducted self-assessments and shared results and insights. Now, member

districts are using the benchmarks in everything they do, from setting goals and strategic

planning to designing professional development and establishing accountability systems.

“In Plano, we are developing strategies for our school-based improvement committees to use

the benchmarks to drive improvements in buildings,” says Otto, whose district has been at the

forefront of the consortium’s work. “One of our weaknesses is that we’re still not at the point

where every teacher is using the data to improve instruction. We know that from our own obser-

vation and from talking with groups of principals and curriculum coordinators. That tells us that

more training is needed.”

How does it work? Based on recent research showing links between specific organizational

policies and student achievement increases, the consortium has adopted four strategic areas of

focus for benchmarking: student learning, capacity development, learning community develop-

ment and data-driven decision-making.

Under each benchmark, a district’s performance shows it to be at one of four stages of devel-

opment: emergent, islands, integrated or exemplary. For example, the consortium has developed

a data-driven decision-making benchmark for “relating investments, outcomes and improve-

ment strategies.” A district with “emergent” performance in this area collects data that are not

maintained consistently over time and are not linked well enough to key outcomes to inform

critical decisions. By contrast, the data needs of an “exemplary” district are clearly articulated

and met in all stages of planning and implementation.

For each stage of development, the benchmarks include “behavioral indicators” (beliefs and

actions evident in the system) and “support indicators” (things that support desired behaviors,

such as professional development, policies and resources). These indicators not only let districts

know how well they are doing, but also give them sound, research-proven ideas about how they

can do better.

The process of developing the consortium’s benchmarks has been more valuable, in some

ways, than the benchmarks themselves. “It really generates a lot of professional opportunities,”

says Otto. “Having superintendents sitting around the table talking about these benchmarks —

what we’re doing and the strategies we’re employing — really raises the level of conversation

with regard to school improvement.”

“The consortium holds great promise for us in terms of continuous improvement based

on objective standards,” agrees Benson, whose Blue Valley School District also participates in

benchmarking. “You want to make continuous improvement in those areas where you can say

objectively: ‘This will make a difference for kids.’”

Chapter 4: Strategies for Success

Data-Driven Decision-Making: Using Information to Improve Instructional Practice

Using Data to Improve Schools: What’s Working

Benchmarks for School District Success The Western States Benchmarking Consortium’s benchmarks for school district performance allow a district’s

performance to be judged to be at the “emergent,” “islands,” “integrated” or “exemplary” stage of development.

The table below describes indicators at different levels of performance for how well educators use data to personalize

education or students and profile and monitor student performance.

Data for Personalization

The instructional approach is “one-size- fits-all,” with underper- forming and accelerated students “pulled out.” As a result, there is little or no integration among remedial, accelerated and ongoing instruction. The concept of personal- ized instruction for all students based upon individualized perform- ance data is foreign.

Data for Personalization

Some schools provide examples of new approaches to accelerat- ed or extended learning that are driven directly by examination of data.

Data for Personalization

Evidence shows that the programmatic use of data in most areas such as accelerated and extended learning pro- grams flows directly from examinations of performance data.

Data for Personalization

Teachers routinely use performance data to truly personalize learning for all students.

Profiling and Monitoring Student Performance

The district has not organized a formal student performance database. As a result, schools are left to their own devices to track performance over time. Teachers have great difficulty in determining the peform-ance level of students who move from school to school. Traditional grade-based report cards provide parents with information on student progress.

Profiling and Monitoring Student Performance

The student perform- ance database is still nonexistent, but more people are demanding that it be designed and implemented to improve curriculum and instruc- tion. Schools are strug- gling with the inadequa- cy of the performance information base regard- ing timeliness, validity, reliability and compre- hensiveness. The formal report card is still grade- based and remains the primary mode of com- munication with parents regarding student progress.

Profiling and Monitoring Student Performance

The district provides a profile of student performance for each school. All schools use these profiles to set annual and longer range improvement targets. Student progress reporting is based on a continuum of progress in achieving well-defined content and perform- ance standards. More schools are providing parents with specific suggestions regarding techniques for family reinforcement of these standards.

Profiling and Monitoring Student Performance

Data are aggregated/ disaggregated to class- room/school/district levels to determine necessary improvements in instruc- tional practice. Heavy evidence of reliance on student performance data from multiple sources exists. The system main- tains a performance profile for every student, integrat- ing all information about the student’s performance over time. Parents receive frequent progress reports of growth on the standards continuum, accompanied by suggestions for home reinforcement.

Emergent Islands Integrated Exemplary

Source: Western States Benchmarking Consortium

The following school district performance benchmarks developed by the Western States

Benchmarking Consortium describe indicators of district success in four major areas of

educational performance.

Student Learning

• Ensuring learning for all students

• Integrating standards

• Incorporating innovative practice

• Integrating technology

• Developing a coherent curriculum

Capacity Development

• Expanding organizational effectiveness

• Adopting a curriculum management system

• Promoting innovation

• Improving professional/organizational learning

Learning Community Development

• Understanding and using state academic results

• Providing community-based learning opportunities

• Building community partnerships

• Building community development

Data-Driven Decision-Making

• Using a variety of data effectively

• Using information to improve instructional practice

• Using data to affect student performance

• Relating investments, outcomes and improvement strategies

Chapter 4: Strategies for Success

Danger Zones Superintendents charging into the unfamiliar territory of data-driven decision-making

sometimes see the edge of the cliff only just before they reach it. Here are some lessons learned

by district leaders who have been there:

• Don’t rush headlong down the path to collect data and enforce accountability before

bringing on board all the people needed to make the new system work. Talk. Listen.

Build consensus about what is most important to do. Otherwise, you might find your-

self out in front — with no one behind you.

• Don’t fish blindly for meaning in data. If you do not know what questions need to

be asked — before you even collect the data — you are not likely to find the answers

you need.

• Don’t bombard people with data for its own sake. Don’t report it just because it is

available. Stick with data points that can drive decisions for improvement.

• Don’t lose track of what is most important — such as key indicators of student learn-

ing progress. If you place equal priority on many data points, some of which might

have little to do with your district’s core mission, then your team will not know where

to focus its energy.

• Don’t forget to draw the "big picture" for the people collecting, interpreting and using

data. They need to understand why they are being asked to do this and what it all

means. Explain it again and again and again.

Summary Key points in Chapter 4 include:

• Successful data-driven decision-making requires a shift in the culture of a school

district that encourages the use and analysis of data without fear of reprisal.

• The superintendent, school board members, principals and teachers must be clear on

the purposes and goals of data collection and analysis.

• To help educators use data effectively, it is important to provide training and

assistance, because data-driven decision-making requires new knowledge and skills.

• School districts using data-driven decision-making ensure continuous improvement by

benchmarking their schools’ performance against that of other top-performing schools

across the country; benchmarking helps districts learn what it takes to be effective.

Using Data to Improve Schools: What’s Working

Source: Reporting Results: What the Public Wants to Know, 1999

Appendix A: Accountability Measures

Parents, taxpayers and educators were asked to rate the following items in importance for

school accountability.

Category Parents Taxpayers Educators

School safety 9.6 9.4 9.3

Teacher qualifications 9.3 9.2 8.3

Class size 8.9 7.9 8.8

Graduation rates 8.7 8.2 8.3

Dropout rates 8.3 8.1 7.4

Statewide test scores 8.2 8.0 7.1

Parental satisfaction survey data 8.1 8.0 7.0

SAT/ACT scores 8.1 7.9 6.9

Percent of students promoted to next grade 8.0 8.1 7.0

Course offerings 7.8 7.9 7.3

Attendance rates 7.8 8.0 7.6

Per-pupil spending 7.6 7.6 8.0

Student satisfaction survey data 7.5 7.0 7.1

Teacher salaries 7.3 7.8 7.6

Hours of homework per week 7.2 7.3 6.3

Number of students 7.2 7.2 6.7

Percent of students who go on to a four-year college 7.0 6.9 6.8

Percent of students with an "A" or a "B" average 7.0 6.5 5.8

Number of students per computer 6.9 6.4 6.1

Percent of parents who attend parent-teacher conferences 6.4 6.6 6.3

Demographics of students 4.5 4.6 5.0

Using Data to Improve Schools: What’s Working

Appendix B: Working with the Media

Media Dos and Don’ts on Data

• Do make sure your district has a comprehensive communications plan that addresses

the release of potentially controversial data such as test scores, dropout rates, teacher

quality ratings, school safety reports and other indicators.

• Don’t wait until the day test scores are released to deepen the media’s understanding

about what the tests measure, how the district will use the test scores and what the

results tell parents and community members. This should be an ongoing process. Be

proactive. Keep the media informed. Make an occasional courtesy call even when you

don’t have to.

• Do make sure staff is apprised of test results and other sensitive data before the media,

so everyone is focused on the same messages. For example, if the state releases test

scores one day before the scores are released publicly, hold meetings with teachers,

principals and other district staff to inform them of the results.

• Don’t hide data from reporters. It’s public information.

• Do talk to the reporter first when inaccuracies surface in a story, such as factually incor-

rect data or scores. Make sure you have documentation to back up your claim. If prob-

lems continue, contact the reporter’s editor.

• Don’t neglect to share what is working in the district. Suggest feature stories that show

trends in school improvement backed by data.

Interview Techniques: How to Handle Tough Media Interviews on Data

Needling Reporter: “Oh, come on now, you don’t believe trend data show improvements

in teacher quality do you?”

Superintendent: “Yes, I do, Beth, and… .” Go on to your key message.

Twisted Facts Reporter: “So your test scores are down again.” (When, in fact, the test has

changed or a new population of students is tested.)

Superintendent: “Let’s be clear about what the data tell us… .”

False Assumption Reporter: “So I assume you are going to spend taxpayer money to purchase a

new test since this one shows bad results… .”

Superintendent: “Well, Mike, I wouldn’t agree with your conclusion.” Steer the interview back on message.

Baiting

Reporter: “So why does this community keep attacking your initiatives?”

Superintendent: “You’ll have to talk directly to those who raise concerns. What I can tell you is that our district is dedicated to… .” (Discuss public engagement efforts to gather and respond to community input.)

Pregnant Pause

Reporter: The reporter remains silent after you answer a question.

Superintendent: Do not rush to fill in the silence. Wait for the reporter to follow-up or ask, “Does that answer your question?”

Loaded Question Reporter: “You don’t deny that the district knew all along about sagging test

scores and refused to do anything about it.”

Superintendent: “Here’s what we have done to improve low-performing schools… . Here’s what has worked and where we need to improve… .”

Adapted from a Michigan School Boards Association media slide presentation.

Using Data to Improve Schools: What’s Working

Appendix C: Resources

Reports “Building an Automated Student Record System.” National Forum on Education Statistics, 2000.

This booklet is designed to lead education organization decision-makers through the process of

making the best and most cost-effective decisions about information management systems devoted

to individual student records. Available online at http://nces.ed.gov/pubs2000/building/

“Evaluation of California’s Standards-Based Accountability System.” WestEd, November 1999.

This evaluation examines the status and impact of local accountability systems, content stan-

dards, assessment measures, use of data, consequences and incentives, and challenges and assis-

tance in school districts across the state. The document reports the findings of the study and its

implications for policy. Available online at http://web.wested.org/online_pubs/accountability/

“Reporting Results: What the Public Wants to Know.” A companion report to Education Week’s

Quality Counts ’99.

A report of what parents, taxpayers and educators think they need to know about schools in

order to hold them accountable. Available online at

http://www.edweek.org/sreports/qc99/opinion/edweekresults.pdf

“Student Data Handbook for Elementary, Secondary, and Early Childhood Education.” National

Center for Education Statistics, 2000.

This handbook offers guidelines, developed by the U.S. Department of Education’s National

Center for Education Statistics, for current and consistent terms, definitions and classification codes

to maintain, collect, report and exchange comparable information about students. Available online

at http://nces.ed.gov/

“Using Data for School Improvement: Report on the Second Practitioners’ Conference for

Annenberg Challenge Sites.” Annenberg Institute for School Reform at Brown University Tools for

Accountability Project, 1998.

This report, developed after the second practitioners conference for Annenberg Challenge Sites,

provides valuable examples of data collection, ways of thinking about accountability and resources.

Real-life examples of schools implementing data-collection efforts to improve student achievement

appear throughout the report. Available online at

http://www.annenberginstitute.org/images/using_data4.pdf

“Vermont Department of Education. Equity and Excellence Action Planning Guide “

http://www.state.vt.us/educ/actplan/apcover.htm

This resource provides a look at each step of the action planning process for educators.

Articles “Data-Driven Decisions.” Articles by Theodore B. Creighton, Raymond Yeagley, Philip A. Streifer,

Francis Barnes, Marilyn Miller and George A. Goens. The School Administrator, April 2001.

This issue of The School Administrator includes a number of articles on data-driven decision-mak-

ing. Available online at http://www.aasa.org

“On the Job: Data Analysts Focus School Improvement Efforts.” Joellen Killian and

G. Thomas Bellamy. Journal of Staff Development, Winter 2000.

This article presents one school district’s model for building-level disaggregation and data analy-

sis. Available online at http://www.nsdc.org/library/jsd/killion211.html

“Smart Moves: Achieving Your Vision Depends on Follow-Through.” Joan Richardson. Tools for

Schools, National Staff Development Council, August/September 2000.

This article discusses how school districts move beyond publishing and publicizing a school’s

newly developed vision statement and begin the real work of using the vision as a guide and a

measure for school improvement. Available online at

http://www.nsdc.org/library/tools/tools9-00rich.html

“This Goes on Your Permanent Record.” Stewart Deck. CIO Magazine, Nov. 1, 2000.

This article examines how various school districts are building and implementing data

warehouses to support districtwide, building-and classroom-level decision-making.

Available online at http://www.cio.com/archive/110100/permanent.html

“Using Data to Improve Instruction.” Articles by Nancy Protheroe; Jay Feldman and Rosann

Tung; Yi Du and Larry Fuglesten; Thomas Glass; Brett Geis, Terrell Donicht and Steven Smith; and

Thomas Fowler-Finn. ERS Spectrum, Summer 2001.

This issue of the ERS Spectrum includes several articles about the use of data in school

improvement efforts.

Using Data to Improve Schools: What’s Working

Books Accountability Dialogues: School Communities Creating Demand from Within.

Kate Jamentz. WestEd, 2001.

This book explains some of the common misperceptions about school accountability and

provides a strong rationale for including accountability dialogues in any accountability system.

Examples from schools that use accountability dialogues provide a real sense of what can

happen when responsibility for school improvement is shared among all the stakeholders in

a school community.

At Your Fingertips — Using Everyday Data to Improve Schools. Karen Levesque, Denise Bradby,

Kristi Rossi and Peter Teitelbaum. MPR Associates Inc. and American Association of School

Administrators and National Center for Research in Vocational Education, 1998.

This handbook teaches educators new and productive ways of using data. Through step-by- step instruction, this book focuses on using available data to improve teaching and learning. Information available online at http://www.mprinc.com

Effectively Using Data to Drill Down and Make Better Educational Decisions.

Philip A. Streifer. Scarecrow Press, Inc., 2002.

This book is a guide on how to use data in decision-making, with a strong emphasis on

asking the right questions in a cyclical process designed to narrow the focus of inquiry.

Data Analysis for Comprehensive Schoolwide Improvement. Victoria L. Bernhardt. Eye on Education, Larchmont, N.Y., 1998.

This book teaches the layperson how to gather, interpret and use data to improve schools. Educators are given practical tools so they can make better data-based decisions.

Getting Excited About Data: How to Combine People, Passion and Proof. Edie L. Holcomb. Corwin Press, Thousand Oaks, Calif., 1999.

This book outlines a process for showing how well a school or district meets its primary goal: sustained student learning. The author offers tips on finding answers to questions about data, such as: What data do we need and how do we collect it?

Improving School Board Decision-Making: The Data Connection. National School Boards Foundation, 2001

This guidebook helps school board members understand what data are, ask for the appropri- ate data to inform decision-making, work with the superintendent to understand what data reveal, use data to support decisions on policy and budgets and inform the community.

Is There a Public for Public Schools? David F. Mathews. Kettering Foundation Press, 1997.

This book provides school leaders with insights for engaging the public when undertaking

school improvement efforts.

Thinking About Tests and Testing: A Short Primer in Assessment Literacy. Gerald Bracey. American Youth Policy Forum, Washington, D.C., 2000.

This book is a nonpartisan, objective discussion that helps readers understand the arguments now raging about "high-stakes tests" and their consequences. The book is simple and straightfor- ward. Available online at http://www.aypf.org/BraceyRep.pdf

The School Portfolio: A Comprehensive Framework for School Improvement, Second Edition. Victoria L. Bernhardt. Education for the Future Initiative, 1999.

This book shows how to develop a school portfolio tailored to a particular school and vision. It explains that school portfolio is the most effective way to ensure a school’s success at systemic reform. Extensively tested, it is a nonthreatening self-assessment tool that exhibits a school’s goals, achievements and vision for improvement.

Schools and Data: The Educator’s Guide to Using Data to Improve Decision Making. Theodore B. Creighton. Corwin Press, Thousand Oaks, Calif., 2000.

This book focuses on the relevance of statistics in the day-to-day lives of principals and teach- ers. This is an essential resource for any educator who wants to break through the statistical confusion to improve skills in problem analysis, program evaluation, data-driven decision-making and report preparation.

Using Data / Getting Results: A Practical Guide for School Improvement in Mathematics and Science. Nancy Love. Christopher-Gordon Publishers, 2002.

This book is a guide for inquiry into improving mathematics and science teaching and learning. It examines ways to engage school communities and produce powerful learning.

Internet Resources

Center for Accountability Solutions, American Association of School Administrators. http://www.aasa.org/data

AASA has created this website to help school leaders gather, use and report meaningful data on student, school and district performance. It includes a comprehensive listing of data resources that is updated regularly.

Using Data to Improve Schools: What’s Working

Appendix D: Glossary

A Accountability — A process by which educators are held responsible for performance or

outcomes.

Application service provider (ASP) — A third-party entity (generally for-profit) that manages

and distributes software-based services and solutions to customers across a wide area network

from one or more off-site locations.

B Benchmark — A standard by which something can be measured or judged. Benchmarks can

be used to monitor progress toward meeting goals.

C Cohort – A group of students or others of the same age moving through a system together

(e.g., all of the students who enter kindergarten the same year are part of the same cohort).

Cohorts typically are used to measure progress over time and compare that progress with

other cohorts.

Common cause — A regular feature of the school system (e.g., normal student attendance

patterns).

Comparative analysis report — A report that contrasts two or more districts, students or

other groups that have similar characteristics and can be compared to each other.

Criterion-referenced test (CRT) — A test that measures an individual’s performance against a

well-specified set of standards (distinguished from tests that compare students in relation to

the performance of other students, known as norm-referenced tests).

Cut score — A specified point on a score scale, such that scores at or above that point are

interpreted or acted on differently from scores below that point. In standards-based assess-

ments, cut scores typically delineate passing from failing, proficiency from mastery, and so on.

D Data — Factual information (such as measurements or statistics) used as a basis for reasoning,

discussion or calculation. Data can be qualitative or quantitative. Good data must be both

reliable and valid.

Data mining — The analysis of data for relationships that previously have not been discovered.

Data warehouse — A central repository for all or significant parts of the data that a system

collects. A data warehouse typically is a database or collection of databases existing in virtual,

rather than physical, space.

Decision support system (DSS) — A computer program application that analyzes and presents

data so users can make decisions more easily.

Demographic data — Data that focus on the gender, socioeconomic background, race and

ethnicity of students in a school or district.

Disaggregated data — Data broken down by specific student subgroups, such as current

grade, race, previous achievements, gender, socioeconomic status, and so on.

Drill down process — A method of disaggregating data that begins with a general question,

followed by increasingly specific questions that focus on smaller subsets of data.

F Formative assessment — Assessment in which learning is measured at several points during a

teaching/learning phase, with the primary intention of obtaining information to guide further

teaching or learning steps. Formative assessments take a variety of forms, including question-

ing, comment on a presentation or interviewing.

I International test — A test administered uniformly in a number of countries that compares

student performance country by country (e.g., Third International Mathematics and Science

Study).

L Longitudinal data — Data measured consistently from year to year to track progress, growth

and change over time. True longitudinal studies eliminate any students who were not present

and tested in each of the years of the study.

M Mission — A statement that defines what an organization is created to do and reflects its core

values and beliefs to guide it toward its goals.

Using Data to Improve Schools: What’s Working

N Normal curve equivalent (NCE) — Normalized standard scores on an equal interval scale

from 1 to 99, with a mean of 50 and a standard deviation of about 21.

Norm-referenced test (NRT) — A test that compares individual student performance with a

larger group, usually a national sample representing a diverse cross-section of students. (e.g.,

Iowa Tests of Basic Skills, Stanford Achievement Test). NRT results typically are measured in

percentile ranks. (Norm-referenced tests should be contrasted with criterion-referenced tests,

which measure performance compared to a standard or benchmark.)

P Percentile score — A score that designates what percent of a norm-referenced group earned

raw scores lower than a particular score. Percentiles often are divided into quartiles (groupings

of 25 percent).

Perception data — Data that inform educators about parent, student and staff perceptions

about the learning environment, which could also reveal areas in need of improvement.

Performance test — A test that requires students to demonstrate their abilities by providing

examples of their work (e.g., portfolios, presentations, experiments).

Portfolio assessments — Assessments of a collection of a student’s educational work that

shows growth, self-reflection and achievement over time.

Q Qualitative data — Data based on information gathered from one-on-one interviews, focus

groups or general observations over time (as opposed to quantitative data).

Quantifiable proof — Proof that can be precisely measured.

Quantitative data — Data based on "hard numbers" such as enrollment figures, dropout rates

and test scores (as opposed to qualitative data).

R Reliability — The consistency of test scores over different test administrations, multiple raters

or different test questions. Reliability answers the question "how likely is it that a student

would obtain the same score if they took the same test a second time?"

S Special cause — An unusual circumstance in a school system (e.g., fluctuations in attendance

pattern due to a flu bug or a snow day).

Stakeholders — The students, parents, taxpayers, community members, business leaders,

educators, board members and all others who have a share or interest in the school or district.

Standard — Something established for use as a rule or basis of comparison in measuring or

judging capacity, quantity, content, extent, value, quality, and so on.

Stanine — A standard score of nine units in which 1, 2 or 3 indicates below-average

performance; 4, 5 or 6 indicates average performance; and 7, 8, or 9 indicates above-

average performance. Stanines are still used in some standardized tests.

Summative assessment — An assessment at the end of a period of education or training,

which sums up how a student has performed.

V Validity — The degree to which tests measure what they purport to measure.

Value-added — A measurement of the learning that a student achieves through participation

in a program.

Vision — A future-focused statement about what an organization wants to be, where it wants

to go and what kind of system it wants to create.