Using Data to Improve Schools
Using Data to Improve Schools
Using Data to Improve Schools
What’s Working What’s
Working
Using Data to Improve Schools: What’s Working
ii
This publication was created with editorial assistance from KSA-Plus Communications in Arlington, Va.
This report was produced in whole or part with funds from the Office of Educational Research and Improvement, U.S. Department of Education, under award # R215 U99 0019. Its contents do not necessarily reflect the views or policies of the Department of Education.
About AASA
The American Association of School Administrators, founded in 1865, is the professional organization for over 14,000 educational leaders across the United States and in other countries. AASA's mission is to support and develop effective school system leaders who are dedicated to the highest quality public education for all children.
iii
Foreword School system leaders are discovering the power of data for promoting school improvement.
With recent advances in technology and the increased demand for assessing student learning,
an unprecedented amount of data are available to educators. School districts across America are
beginning to use the tools necessary to make effective use of the data. In addition to test scores,
many educators are collecting data about citizenship, character, healthy lifestyles, school climate
and parental and community involvement.
One superintendent reflected that “We spend a lot of time on testing but not much time on
what to do with the test results.” As educators shift their focus from simply reporting test results
to using the data to improve instruction, data become essential ingredients in school improve-
ment. Educators know that the effective use of data can measure student progress, evaluate
program and instructional effectiveness, guide curriculum development and resource allocation,
promote accountability and, most importantly, ensure that every child learns.
Using Data to Improve Schools: What’s Working is an easy-to-read guide to using data to drive
school improvement. School system leaders and their staffs can learn from this book how to
build a districtwide culture of inquiry that values the use of data for sound decision-making.
School board members, parents and community members interested in helping improve schools
will find tools for their work as well in this guide. It describes the challenges and the successes of
educators from districts both large and small committed to using data.
We are sure that you will find this guide useful in your ongoing efforts to provide leadership
to your schools and communities.
Paul D. Houston, Ph.D.
Executive Director
American Association of School Administrators
Using Data to Improve Schools: What’s Working
iv
Bill Adams
Superintendent
Salem County Vocational Technical Schools
Woodstown, N.J.
Lance Alwin
Superintendent
Antigo Unified School District
Antigo, Wis.
Mary Barter
Superintendent
Durango School District 9-R
Durango, Colo.
Richard P. Fragale
Superintendent
Central Union High School District
El Centro, Calif.
David E. Gee
Superintendent
Western Suffolk BOCES
Dix Hills, N.Y.
John Lacy
Superintendent
Billings R-IV School District
Billings, Mo.
Peg Portscheller
Executive Director
Colorado Association of School Executives
Englewood, Colo.
Roland Smit
Superintendent
Mobridge School District
Mobridge, S.D.
Linda Dawson
Project Director
National School Boards Foundation
Aspen Group International
Castle Rock, Colo.
Acknowledgments Development Advisory Team
Acknowledgments
v
AASA Project Staff
Judy Seltz
Associate Executive Director
Constituent Relations and Services
Geannie Wells
Director
Center for Accountability Solutions
Mike Parker
Assistant Director
Center for Accountability Solutions
Aleck Johnson
Program Manager
Center for Accountability Solutions
Sarah Wayne
Program Assistant
Center for Accountability Solutions
AASA Executive Committee
Don W. Hooper
President
Superintendent
Fort Bend Independent School District
Sugar Land, Texas
Bill Hill
President-Elect
Superintendent
Deer Valley Unified School District
Phoenix, Ariz.
Benjamin O. Canada
Immediate Past President
Educational Consultant
Atlanta, Ga.
Mary F. Barter
Superintendent
Durango School District 9-R
Durango, Colo.
Barbara F. Erwin
Superintendent
Scottsdale Unified School District 48
Phoenix, Ariz.
Richard P. Fragale
Superintendent
Central Union High School District
El Centro, Calif.
David E. Gee
Superintendent
Western Suffolk BOCES
Dix Hills, N.Y.
Donald L. Kussmaul
Superintendent
East Dubuque Community Unit School District 119
East Dubuque, Ill.
John R. Lawrence
Superintendent
Lincoln County R-III School District
Troy, Mo.
Estanislado Y. Paz
Superintendent
Tucson Unified School District
Tucson, Ariz.
Kay E. Royster
Deputy Chief Executive Officer
Detroit Public School District
Detroit, Mich.
Using Data to Improve Schools: What’s Working Foreword .............................................................. iii
Acknowledgments .............................................. iv
Chapter 1: Why Data Matter ........................................ 1
Chapter 2:
Using Data to Make Smart Decisions ...... 13
Chapter 3: Data, Public Engagement and Strategic Communications........................ 27
Chapter 4: Strategies for Success ................................ 37
Appendix A: Accountability Measures ............ 53
Appendix B: Working with the Media ............ 54
Appendix C: Resources ...................................... 56
Appendix D: Glossary ........................................ 60
vi
M any superintendents have a powerful ally on their side: data. Increasingly,
superintendents are using data to make smarter decisions, and they are getting results.
Instead of responding defensively to critics, they are armed with facts and figures that tell a
more complete story and help critics understand the root causes of the challenges schools face.
“Data-driven decision-making is about gathering
data to understand if a school or district is meeting its
purpose and vision,” says Victoria Bernhardt, author
of Data Analysis for Comprehensive Schoolwide
Improvement. (See A Closer Look on page 2.) “If we do
not have a target, we could make decisions that essen-
tially lead to ‘random acts of improvement.’” Instead,
Bernhardt says, superintendents should strive for
“focused acts of improvement,” which occur when
schools are clear about their purpose, about what they expect students to know, and about what
they expect students to be able to do.
In data-driven districts, superintendents work side by side with other administrators, teachers,
principals and parents to ensure all children achieve. Everyone strives toward common goals.
Data provide quantifiable proof, taking the emotion and rancor out of what can be tough calls
for superintendents and school boards (e.g., dismantling a popular but ineffective program or
closing a school). Data also provide the substance for meaningful, ongoing dialogue within the
educational community.
For the past several years, Superintendent Gerrita Postlewait and the Horry County (S.C.)
Public School Board have made data-driven decision-making the cornerstone of school
improvement. “This is now a school system that focuses on results,” Postlewait says. “If I, as the
1
Chapter 1: Why Data Matter • What goals has your school district set for the next three years?
• What data will help judge whether the district is meeting its goals?
• How are superintendents using data they currently collect to improve student achievement over time?
• What additional data must be collected and why?
• In what ways are teachers, principals, district staff and the community involved in data collection and analysis?
If I, as the superintendent, cannot talk about how much learning has occurred, then I’m not achieving what I had hoped to as superintendent.
— Gerrita Postlewait, superintendent,
Horry County (S.C.) Public Schools
superintendent, cannot talk about how much learning has occurred, then I’m not
achieving what I had hoped to as superintendent.” (See Getting Results on page 3.)
Postlewait and other superintendents acknowledge that data were not always at
the forefront of their decision-making. Until recently, they collected data only
because gathering this information was mandated by the state in return for
funding. Rarely were data used to analyze whether children were learning at
grade level, teachers were using sound instructional practices or parents were
pleased with the quality of schools.
“Many school improvement plans are hit or miss,” observes Superintendent
Roland Smit, who heads the 600-student Mobridge (S.D.) School District. “We may
think we should work on a specific area but we don’t always look at the indicators
that may tell us this is, in fact, something we should work on. With the help of
data, we can zero in on our weaknesses.” Data help superintendents like Smit
make decisions with far greater precision and clarity.
Data help district and school leaders craft a sound blueprint with measurable
results for continuously improving schools so decisions are no longer based on
incomplete or biased information. The challenge for superintendents is to know
specifically what they are evaluating. For example, is a particular math program
effective? Which students are showing the great-
est gains and why? How can we target those
who are showing the least improvement?
Superintendents can focus on a specific grade
level or measure overall progress toward meeting
math standards, or they can use data to see how
many students are at levels of proficiency or
excellence in math.
A Closer Look Victoria Bernhardt
Author, Data Analysis for Comprehensive Schoolwide Improvement
For many, data are confusing, even intimidating, but
in your book you say data are logical if we think
about what we need to know and why. Tell us more
about that.
If questions can be created, the data that are
needed to answer the questions are very logical.
For example, for the question “How well are we
Using Data to Improve Schools: What’s Working
2
What is data-driven decision-making? • Collecting data
• Analyzing data
• Reporting data
• Using data for school
improvement
• Communicating
through data
Data-driven decision- making requires a cultural shift in thinking that must be nurtured so all stake- holders are committed to this effort.
Data help: • Measure student progress
• Make sure students don’t fall through the cracks
• Measure program effectiveness
• Assess instructional effectiveness
• Guide curriculum development
• Allocate resources wisely
• Promote accountability
• Report to the community
• Meet state and federal reporting requirements
• Maintain educational focus
• Show trends (but not necessarily solutions)
Data do not help: • If the data are not valid and reliable
• If appropriate questions are not asked after reviewing
the data
• If data analysis is not used for making wise decisions
doing?” one would probably want to
look at student achievement results
on standardized tests and state or dis-
trict assessments to get an idea of
how students in the district are scor-
ing right now. For the question “Are
all students learning?” one might
want to take the general student
achievement analysis to a deeper
level, looking at the distribution of
scores to understand which students
are scoring below mastery, and how
far they are scoring below mastery.
You also write that data can help uncov-
er solutions to some of the toughest chal-
lenges schools face today. How so?
I think data can help us see things
we might not see otherwise. We
might have processes or programs in
operation for years. Once we look at
the data from all angles, we might
find that a program is not helping all
students learn. Data help us get to
the root causes of a problem so we
solve the problem and not just the
symptom.
In what ways can superintendents use
data to improve student achievement?
Student achievement data, for exam-
ple, can help superintendents under-
stand which instructional strategies
are creating the best results and see
where additional training might be
needed. Perceptions data can tell
superintendents about parent, stu-
dent and staff satisfaction with the
learning environment, which also
could reveal areas in need of
improvement. Staff perceptions can
Chapter 1: Why Data Matter
3
In the early 1990s, the Horry County (S.C.) School Board was
granted the authority to raise taxes. School board members,
parents and community members in the racially and economi-
cally diverse school district of 29,000 students wanted to know
the district was spending their taxpayer dollars wisely.
Gerrita Postlewait, who was an instructional specialist with
the district at the time, says the district staff began by re-
examining their expectations. “For example, what kind of
achievement results did we want to see,” Postlewait says.
“What kind of activities did we want? Those questions and
others kick-started the data-driven decision-making process.”
District leaders, principals and teachers learned a lot along
the way, including that at-risk four-year-olds enrolled in the
district’s structured academic prekindergarten program scored
consistently higher in reading, math and writing in first,
second and third grades than students who participated in
preschool programs sponsored by other governmental agen-
cies. In fact, by third grade, 75 percent of the students who
attended the district’s preschool program scored higher than
their counterparts in reading and writing.
After collecting and analyzing the data, the district
expanded the prekindergarten program by an additional 200
children. “As we prepared our budget, we had a way to talk
with our public about why this proposed change was impor-
tant,” Postlewait says. “We could show a reduction in the
amount of time we have to spend with these students in reme-
dial education a little later in grades one, two and three.”
Postlewait became superintendent of the district in 1997.
“It’s chaotic and bone-wearying work, but it’s worth it because
we are now seeing gains in student achievement. This is the
first year we have had results to celebrate.”
Getting Results Horry County, S.C.
Using Data to Improve Schools: What’s Working
4
tell superintendents what is possible. Demographic data can provide valuable information about
meeting the learning needs of students in the future, including: How might the makeup and size
of the school population change? Or, how many new teachers with specialties will be needed in
the near future?
If you are a superintendent using data for the first time, where do you begin? How do you get started?
I would start by understanding what data exist in the district
and see what data I can get easily elsewhere, like the state edu-
cation department or other sources. Undoubtedly, I could get
demographic data, such as enrollment and attendance figures.
I also believe that perceptions data is very important for under-
standing schools and the health of the district. I would admin-
ister online questionnaires to students, staff and parents. Then,
if I were really on the ball with the data analysis, I would set up
a comprehensive database to store and analyze the data at the district, school, classroom,
teacher and student levels.
What sort of leadership must a superintendent provide as more and more teachers, district staff, board
members and others use data?
I believe that superintendents must create a safe environment for data-driven decision-making.
Superintendents must lead, model and encourage staffs to use different types of data and exam-
ine the numbers systematically to avoid knee-jerk reactions to single, independent pieces of
data. It will require training and trust so that every staff member can have access to, and look at,
data on an ongoing basis. It requires a vision that is truly shared, so staff members are all contin-
uously improving in the same direction.
Getting Started One of the biggest challenges for any district implementing data-driven decision-making is
knowing where to begin. Often, there are so many data to choose from that the process can be
overwhelming for district staff, teachers and principals. The best advice is to start with the
basics. Too often, school administrators and staff collect data without first figuring out what it is
they need to know.
“We recommend not fishing blindly as you review data,” says Mike Parker, assistant director
of the Center for Accountability Solutions at the American Association of School Administrators
(AASA). “If you don’t have a purpose in mind, it’s easy to get off track.”
A district should begin data collection by defining what it wants to know. Is the strategic
plan the district put in place three years ago improving student performance? Why are high
school freshmen in your district garnering low grades in English? Did the district make good on
its promise to the community that test scores would increase 5 percent over last year’s scores?
Not everything that is faced can be changed, but nothing can be changed until it is faced.
— James Baldwin,
author
Chapter 1: Why Data Matter
5
In Data Analysis for Comprehensive Schoolwide Improvement, Bernhardt lists seven questions to
help focus the early stages of data-driven decision-making:
• What is the purpose of the school or district?
• What do you expect students to know and be able to do by the time they leave school?
(Standards)
• What do you expect students to know and be able to do by the end of each year?
(Benchmarks)
• How well will students be able to do what they want to do with the knowledge and
skills they acquire by the time they leave school? (Performance)
• Do you know why you are getting the results you get?
• What would your school and educational processes look like if your school were
achieving its purpose, goals and expectations for student learning?
• How do you want to use the data you will gather?
Chapter 2 takes a closer look at different types of data that can help answer these
questions. Of course, the best questions are the ones superintendents, principals, teachers and
school board members develop together.
Making decisions based on data is a little like being a detective. Good data analysis requires
asking lots of questions, uncovering more and more information and revisiting hypotheses
along the way until a complete picture — supported by the facts — unfolds.
Challenging Assumptions Data-driven school improvement requires
administrators to challenge their own assump-
tions. Almost every district has common
beliefs about a school or groups of students.
One school may be known for its nationally
ranked reading program. Another may be clos-
ing the achievement gap among different
groups of students. But are these schools really
accomplishing what they think they are? How
do they know for sure?
Data help district leaders determine
whether their perceptions match reality.
Running a longitudinal analysis, for example,
will show whether a reading program is
You use data to inform the doctor of the progress of the patient. You determine whether the patient is progressing in a good direction — or is there additional assistance that the patient needs? That analogy works for me, because that’s really what you’re asking for: You want data that are more diagnostic, that permit you to monitor progress on a regular basis and that provide you with the student’s vital signs of learning.
— James Parsley, superintendent,
Vancouver (Wash.) School District
Using Data to Improve Schools: What’s Working
6
sustaining its impact over time. Disaggregating data by different student popula-
tions will show which students are excelling and which are falling behind. These
clues begin to form a picture of what is really happening in schools.
In the rural community of Antigo, Wis., 90 miles northwest of Green Bay, data
helped Superintendent Lance Alwin bridge the divide between educators and fami-
lies that homeschool their children. The work began when the Antigo Unified
School District reviewed progress toward its goals, one of which was to become “an
inclusive educational community.”
“One of the questions that I asked was why we weren’t providing services to
families of homeschoolers,” says Alwin. “The response was that homeschoolers are
on the religious fringes. They want to do their own thing and they don’t like what
we have to offer.”
It was a stand off, Alwin recalls. But if the district truly was to be an inclusive
community, it would have to tackle this issue head-on.
Data about the district’s homeschool population was collected and analyzed. The
district surveyed its homeschool population, asking parents why they home-
schooled their children.
What the district learned challenged long-held perceptions. First, data revealed
the number of families homeschooling their children was much higher than origi-
nally thought. The data also showed that religion was not the number one reason
parents homeschooled their children. Instead, parents said they felt they could do a
better job than the public schools. District leaders also were surprised to learn that
80 percent of homeschool families were interested in accessing instructional
resources provided by the district, as long as no strings were attached.
In response, district leaders created a unique charter school designed to meet the
needs of homeschool families based on the information and data they had gath-
ered. The school opened during the 1997–98 school year with four students. In
2000–01, 62 children enrolled in the school. “We’ve had to turn students away,”
says Alwin. “Now, we’re getting calls from homeschool families several counties
away because they have heard that this school district is willing to work with
them.”
The district is meeting its goal of becoming more inclusive. During the 1997–98
school year, 178 children who were eligible to attend the schools in the Antigo
Unified School District were homeschooled. During the 2000–01 school year, the
number dropped to 119, bucking national trends. Alwin says confronting the data
forced the district to examine new relationships. “This was rich data for us to
mine,” says Alwin. “It helped us determine how to create a charter school that was
of use to them, not us.”
Lessons Learned from Seattle Spokane (Wash.)
Superintendent Brian
Benzel learned the follow-
ing lessons using data to
drive decision-making in
the Seattle (Wash.)
Public Schools:
• Start small; don’t
overwhelm staff with
a “data dump.”
• Begin with the
core issues, such as
student achievement
in reading or
mathe-matics.
• Listen to what the
data tell about the
big picture; don’t
get lost in too
many details.
• Work to create trust
and build support by
laying data on the
table without fear of
recrimination by staff.
• Provide training
opportunities for staff
on how to use data.
• Be patient, working
with what is possible
in the district.
Chapter 1: Why Data Matter
7
Asking the Right Questions The more administrators dig for answers, the more questions emerge. As they analyze the data,
they begin to see patterns and trends. They may notice a cluster of schools doing better than the
rest of the district. Still more questions surface. What are these schools doing differently? Can
these best practices apply to other low-performing schools in the district? Why or why not?
Those who have experience using data say some questions will elicit more substantive infor-
mation than others. In Using Data for School Improvement, a 1998 report by the Annenberg
Institute for School Reform, Kate Jamentz, WestEd’s director of programs in professional and
organizational learning, says:
For example, questions that ask “which” are better than those that ask “how
many?” Asking which students are not meeting the standards in reading is bet-
ter than asking how many students are meeting the standards. It is important
to look at the types of questions we’re asking because the nature of a question
determines next steps in data collection and analysis.
Phil Streifer, associate professor of educational leadership at the University of Connecticut
and a data specialist, calls an in-depth line of questioning the “drill down” process. (See Chapter
2 for more on the “drill down” process.) Streifer says the drill down process “starts with a
global question or issue, which is then broken down into its component parts for analysis.”
Streifer says once the data analysis is completed, the team can make “a reasoned decision on
next steps.”
Newark (N.Y.) Central School District Superintendent Robert Christmann agrees. “Data are
only as good as the questions you are asking. You need your questions to be as specific as possi-
ble so you can know whether your efforts are successful.”
Key considerations when formulating questions include:
• How should student achievement be measured in the district?
• Are goals for student achievement based on data elements aligned with what the
teachers teach?
• What are the best indicators of student achievement upon which the district should
base its decisions?
• What indicators of student achievement are collected regularly throughout the year so
that informed decision-making can occur?
Computer Software and Data-Driven Decision-Making Companies are creating software tools for superintendents and staff who are engaging in data-
driven decision-making. AASA’s Center for Accountability Solutions advises consumers to look
for software that offers flexibility to customize district data reports. These reports should allow
district leaders to review data for different student groups over time.
Using Data to Improve Schools: What’s Working
8
“Through our experience, we have found that each district’s unique needs will determine
which tool is best,” says Geannie Wells, the center’s director. “There is no one-size-fits-all
solution.”
Be sure to compare different software products. Talk with district staff, teachers and others
about what capability the software should have to meet their needs.
Districts should ask two sets of questions when purchasing data software — one of themselves
and one of the vendor.
District leaders should ask themselves:
• What types of data should be included?
• Who will need access to which data elements?
• Is a product accessible through a web browser attractive?
• Is the infrastructure available to house a web-based product?
• Is it feasible to use an application service provider (ASP)?
• Is it acceptable to house the data outside the district on nonschool-system servers?
• How many records and fields are needed?
• How often will the district need to refresh data?
• Should the software allow the district to create groups, select assessments and generate
its own reports or should the district select software with existing templates?
After deciding what the district’s needs are, administrators can shop for software armed with
a list of consumer-savvy questions for the software vendor, such as:
• How will district data be transferred into the new software system?
• How many data updates are included in the software cost?
• Who is responsible for importing new data?
• What security features are built into the software?
• What are the limitations on fields and records?
• What type of, and how much, technical support will be provided by the vendor?
• How much on-site training does the vendor provide?
• When new software features are developed, are they provided free of charge or will
districts have to pay for an updated version?
• What type of database is required?
• In what format are the data stored?
• Can districts receive all of their data upon request?
Chapter 1: Why Data Matter
9
What’s Available? Increasingly, manufacturers of traditional student information systems are adding decision sup-
port services. It is wise to research what features the district’s software vendor might include in
the future before purchasing new software or completely overhauling existing technology.
Lots of data-specific software options also are on the market. Some packages offer a suite of
software tools designed to help districts collect and analyze disaggregated data. These tools can
collect and report longitudinal data on individual students, including demographic and testing
information, or track different areas, such as safety and security, curriculum and instruction
or parent involvement.
Other web-enabled software allows districts to
collect and analyze data that help them report on
student performance and identify solutions. Still other
software offers an analysis of questions tied to equity,
as well as comparisons that highlight the effectiveness
of programs, grade-level groupings, technologies
in the classroom and more.
Software prices vary widely. Some companies
charge a fee based on enrollment, while others charge per school or site. Some districts require
more customization than others, which drives the price up. Some software packages, such as
Quality School Portfolio (QSP), are free at this time.
A listing of many such products is available from AASA’s Center for Accountability Solutions
at www.aasa.org/data.
Homegrown Efforts While some districts prefer to use commercial software, other districts are developing home-
grown models using existing software tools to store and access data. Like their commercial coun-
terparts, these tools disaggregate data and conduct statistical analyses. Typically, they can store
multiple indicators of student achievement, everything from grades to dropout rates to annual
assessments and more.
As chief operating officer for the Seattle (Wash.) Public Schools before becoming superintendent
in Spokane, Wash., Brian Benzel and his staff worked to create a “data decision support system.”
“First, we worked to create a profile of a Seattle Public Schools graduate,” Benzel says. “The pro-
file reflects a series of statements about what we wanted our graduates to accomplish, including
the importance of communicating clearly, thinking critically and working productively in teams.”
The district also examined its grade-level standards in mathematics, reading and writing to
make sure students were learning the skills and knowledge they needed to reflect the Seattle
Public Schools Graduate Profile.
Time is money in a school district. Using a computer software program that helps collect and analyze data allows us to spend that time and money more wisely.
— Roland Smit, superintendent,
Mobridge (S.D.) School District
Using Data to Improve Schools: What’s Working
10
Benzel and his staff used multiple indicators to gauge whether teachers were using the most
effective instructional practices and what impact those practices were having on student achieve-
ment. The indicators are grouped into five categories, including academic achievement and stu-
dent behavior. Test scores, attendance rates, truancy rates, student climate surveys, staff surveys
focusing on areas such as teamwork, leadership and curriculum, and other data are used to
measure overall progress.
Benzel began developing a “value-added school indicators profile” that uses longitudinal data
to predict student achievement. “We wanted to make sure that a year’s worth of instruction was
resulting in a year’s worth of gains,” Benzel says.
The stakes are high in Seattle, where parents decide
which school their child will attend. Funding follows the
student. “In a choice system like [Seattle’s], customers
need to know how schools are doing so they can make
wise choices,” Benzel says.
In Poway, Calif., just north of San Diego, the school
district has taken another “homegrown” approach to
data-driven decision-making. This suburban, upper-income district of 32,000 students began its
foray into data-driven decision-making in 1996. The district had completed an intensive strate-
gic planning process, but was struggling with how to measure whether it was achieving its
vision, mission and core values.
Director of Learning Support Services Ray Wilson says data analysis began with identifying
what the district wanted to measure. Charting student progress was at the top of the list. “For
example, there’s research that shows that students who aren’t reading at grade level by third
grade will have problems later in school,” Wilson says. “So, one benchmark we set was that all
third graders would be reading at grade level before going to fourth grade.”
Decisions also were made about who would collect what data. The district agreed to collect
benchmark data (e.g., third grade reading scores, fourth grade writing scores, fifth grade mathe-
matics assessments, etc.) and provided school leaders with training and tools to help them col-
lect and analyze their own annual data. A part-time data analyst was assigned to every school.
“A principal does not have time in her day to sit and analyze data,” Wilson says. “You can look
at the data and see a problem, but investigating the ‘whys’ behind it can take hours and hours.”
Data-driven decision-making has prompted Wilson and his colleagues to redefine what quali-
fies as data. Early on, they largely thought of it as student performance data — state and local
test scores. It wasn’t long before they realized they needed to broaden their definition.
“For example, in a district like ours with a large white population, it’s easy for the majority
student population to mask the problems of minority students,” Wilson says. “Our students
score at the 80th percentile on norm-referenced tests, but when you disaggregate the data by
race, gender, ethnicity and the number of years a student has been enrolled in the district, you
We wanted to make sure that a year’s worth of instruction was resulting in a year’s worth of gains.
—Brian Benzel, superintendent,
Spokane (Wash.) Public Schools
find small pockets of students performing at the 30th and 40th percentiles.” In other words, it’s
not just about tracking how well students are performing on tests, but determining which
students are not doing well and why, and then seeking improvements.
Using data wisely requires clarity of purpose, time and a desire to seek and understand
improved educational achievement. Ultimately, superintendents say that data have made their
job easier because data help them determine whether the policies approved at the district level
are having the intended impact in the classroom.
Summary Key points in Chapter 1 include:
• Data provide quantifiable proof, taking the emotion and rancor out of the decision-
making process.
• Determining what data to collect is based largely on first figuring out what is important
to know about student performance, teacher quality, parent and community satisfac-
tion and other issues tied to district goals.
• Data-driven school improvement helps superintendents and others know whether the
district and its schools are realizing their vision and purpose.
Chapter 1: Why Data Matter
11
Often, the term data is confused with the term statistics, but the two are not interchange-
able. Data go beyond numbers, averages and percentages. They are the raw materials for
effective decision-making. But to make the right decisions, superintendents need the right tools.
No single set of data or collection method applies to all school districts. While districts might
share common goals, demographics and achievement
levels, they are not identical. Therefore, it is essential
that districts consider their unique traits when devel-
oping their data systems. Such consideration will
invariably help their efforts to improve student
achievement.
There are some commonalities in all data collec-
tion. Successful integration of data-driven decision-
making into educational strategy requires a team
approach. That teamwork starts with the relationship between the board of education and the
superintendent. The district’s expectations must be clearly articulated, measurable — and attain-
able. The roles of the school board and school staff must be clearly defined. And the superin-
tendent — who ultimately is responsible for coordinating the data collection, regulation and
reporting — must work collaboratively with the board, staff and community to lead the district
toward improved student performance.
13
Chapter 2: Using Data to Make Smart Decisions
• What different types of data should superintendents use when assessing student performance?
• How can data analysis effectively target student achievement gaps?
• Which methods work best for efficient — and accurate — data collection?
• What analytical methods can school administrators employ so they are confident that they are interpreting data correctly?
• How can superintendents develop an effective accountability program that is supported by staff, parents and the community?
Collecting data without purpose is meaningless. — Theodore B. Creighton, author,
Schools and Data: The Educator’s Guide for Using Data to
Improve Decision Making
Data Types and Applications Before student performance can be improved, it must be defined and measured. Indicators of
student performance include:
• Test scores
• Rigor of coursework
• Graduation rates
• Attendance rates
• Promotion rates
• Rates of participation in co-curricular activities (including community service)
Data can be measured on many levels. By looking at aggre-
gate data, superintendents can form a general assessment of a
particular curriculum, class or school. But breaking down, or
disaggregating, data provides a more complete view. It goes
beyond the “what” and tries to discover the “why” and “how.”
For example, 32 percent of ninth graders at Elmwood High
School passed advanced algebra. Aggregated, the data do not
bode well for the advanced algebra program. When the data are
disaggregated, a more complete picture comes into focus.
By asking more questions — What percentage of ninth graders took advanced algebra? What is
the program’s gender breakdown? Are any of the students in an English as a Second Language (ESL)
program? Is this the first honors course for any of the students? — superintendents get more com-
plete answers. Time, staff capacity and cost factors prohibit unlimited disaggregation, but asking a
few, specific, thoughtful questions provides better data. And better data lead to better decisions.
Collecting Existing Data Data collection starts with this question: What in particular does the district want to learn?
Once this question has been answered, Phil Streifer, associate professor of educational leadership
at the University of Connecticut, recommends conducting a data audit to determine if the need-
ed data are available. When beginning to search for the answers, superintendents should first
look at their own information. Districts already keep track of certain data in different ways and
in a variety of places such as teachers’ files on students, individual school offices, district offices,
even board member files.
Existing data sometimes are found in digital format on spreadsheets. This is often the easiest
and most efficient way to maintain data. While many spreadsheet applications are available,
superintendents should consider using a program that incorporates or easily converts this data
to graphics. The ability to present data graphically is essential — especially when dealing with
community stakeholders and the media. (See sidebar on p. 34.)
Using Data to Improve Schools: What’s Working
14
Learning is not attained by chance, it must be sought for with ardor and attended to with diligence.
— Abigail Adams,
letter to John Quincy Adams
In many cases, if the data are not immediately at hand, superintendents
do not have to look far to find answers. “A lot of valuable data can be found
if districts look at their own practices,” says Theodore B. Creighton, associ-
ate professor of educational leadership at Sam Houston State University.
“There are a lot of data that we should collect that we don’t collect.” For
example, tracking parental involvement — hours spent at school, Parent-
Teacher Association participation, and so on — can easily be accomplished.
That leads to another question: Are the data important for school
improvement? Perhaps the parental involvement data are essential for one
school, but the issue is not important for another. Even in the same district,
schools have different data needs — and different ways of collecting data.
Collecting New Data Although some data will already be available, not everything is easily acces-
sible, and Streifer advises superintendents to consider the cost benefits of
collecting new data before proceeding. “Many districts often want to add
student tests to match state mastery testing in the ‘off’ years for longitudi-
nal analyses. A decision should be made concerning the cost benefit of
adding these tests in terms of lost instructional time and the cost of pur-
chasing, administering, scoring and analyzing these tests.”
How do superintendents know what new data are worth generating?
“This depends on the mission-critical nature of the question under consid-
eration,” says Streifer. “If the data needed are the only way to address the
problem, and the question is mission-critical, then, yes, they would be
worth generating.”
There are other considerations beyond cost:
• Are the data being used to achieve a district goal?
• Are the data the only way to address the problem?
• Given that it generally takes six months to three years to generate,
collect and analyze data, can the district wait? Will the data still be
relevant to the question?
Chapter 2: Using Data to Make Smart Decisions
15
Examples of Targeted New Data Sources Every district collects and ana-
lyzes data on student test scores
and grades. But there are other
ways to measure student per-
formance, such as:
• surveys and questionnaires
(of teachers, students, par-
ents, employers, community
members, etc.);
• interviews or focus groups
(with the same groups
mentioned above);
• teacher logs/diaries;
• classroom observations of
actual instructional practices
and student responses;
• alternative assessments (e.g.,
work samples, portfolios,
senior projects and perform-
ance tasks); and
• locally developed pretests
and posttests.
Qualitative information, which
describes what people actually
say, do, think or feel, puts a
human face on quantitative data.
It provides depth and detail and
may increase a person’s under-
standing of a situation.
Source: At Your Fingertips: Using Everyday Data to Improve Schools, 1998
Testing Results Are More Than Grades The most widely used — and publicized —
method of assessing student performance is
testing. But testing is not a one-size-fits-all
endeavor. The different kinds of tests —
each with their own guidelines — can be
subject to different interpretation.
• Norm-referenced tests, the most com-
mon form of educational achieve-
ment tests used, compare individual
student performance with a national
sample of others in the same grade.
Results often are described as “stu-
dents scored in the 62nd percentile,”
meaning that they scored higher than
61 percent of the students who took
the test (Iowa Tests of Basic Skills,
Stanford Achievement Test, etc.).
• Criterion-referenced tests measure an
individual’s performance against a
well-specified set of standards. Results
often are described as “62 percent of
students met or exceeded the stan-
dards” (National Assessment of
Educational Progress, state proficiency
tests, Advanced Placement exams,
etc.).
• Performance tests require students to
demonstrate their abilities (via portfo-
lios, presentations, experiments, etc.).
• International tests, though not tech-
nically a test “type,” have gained
prominence by comparing student
performance by country (Third
International Mathematics and
Science Study, etc.).
Using Data to Improve Schools: What’s Working
16
Data Collection to Support Student Achievement Plainview, Okla.
Data collection is not a one-
size-fits-all process. And it is
not limited to those districts
where student performance is
below standard. High-
performing schools are
finding that data collection
and analysis can help them
build upon an already solid
foundation.
Plainview School in south-
west Okla. is a rural PreK–12
school with approximately
1,200 students. The school
began extensive data collec-
tion in 2000.
“The most difficult part
was finding what we wanted
to analyze — and working
with others on resistance to
change,” says Lisa Moore,
Plainview’s principal. “The
easiest part was making the
decision to look at our educa-
tion process in order to
improve.”
Plainview’s accomplish-
ments are impressive. In a
ranking of Oklahoma’s 547
public schools, the school
came in 11th in efficiency and
18th in student performance.
Several nationally certified
teachers are on staff and, in
1999, Plainview was recog-
nized by Expansion Management
magazine as a “Blue Ribbon
Scholar School.”
Moore thinks that, in
some ways, rural school dis-
tricts have data collection
advantages over larger dis-
tricts. “Rural schools typically
are smaller, thereby making it
easier to obtain clear, close-
to-the-student data,” she
says. “We have unique oppor-
tunities — like close commu-
nications — to facilitate data
collection and usage.”
Plainview uses observa-
tions, surveys and standard-
ized test score analysis to
collect data. The data are
presented through reports
on individuals, classes, grade
levels and school buildings.
Moore emphasizes that
the purpose of the data col-
lection and analysis is to
enable the school to make
appropriate decisions about
curriculum structure and
alignment. “The school
board, with the help of the
administration and faculty,
uses results to ensure that we
are meeting the needs of our
students and steadily increas-
ing our school improvement
process to the maximum
potential,” she says.
It is important that tests be reliable and valid. Reliable tests are consistent; when different
individuals review their results over time they arrive at the same conclusion. Defining validity in
a test is not as precise. In essence, a test is valid if it measures what it claims to measure.
Here’s another way to think about it. Let’s say a man steps on his bathroom scale every
morning, and the scale reads 240 pounds. Each morning the result is the same, meaning the
scale is reliable. When the man heads to the doctor for his annual check-up, he is weighed on
the doctor’s more accurate scale,
and the results are different. This
scale shows that the man actual-
ly weighs 245 pounds, 5 pounds
more than on his scale at home.
Although the man’s scale at
home is reliable, it is not valid
because it consistently underesti-
mates his weight. All measures
are subject to the same types of
errors. Verifying both reliability
and validity is critical to ensuring
that tests actually are measuring
what they purport to measure.
When data show gaps in stu-
dent performance, they show
more than where a school or dis-
trict did not meet its goal. When
analyzed, these data show which
student groups need more
improvement, thereby enabling
districts to address students’
needs. For example, Elmwood
High School’s goal was for ninth
graders to score in the 60th per-
centile on the state mathematics
test. That school year, the stu-
dents scored in the 50th per-
centile. While the school did not
meet its goal, the results were
not considered a failure, and the
district set about to improve
overall performance. Further
Chapter 2: Using Data to Make Smart Decisions
17
The Annenberg Institute for School Reform has developed the Inquiry Cycle, six activities that attempt to capture an ongoing, nonlinear process that involves the interaction of reflection and action. Each activity is essential; none may be omitted. The inquiry process is not over after completing one cycle. The six activities are:
Establish Desired Outcomes
Make Meaning of the DataTa
ke A
ct io
n
D efine the
Q uestions
As se
ss &
E va
lu at
e Ac
tio ns
Collect and O
rganize D ata
The Inquiry Cycle
Source: Annenberg Institute for School Reform
analysis showed that female students scored in the 40th percentile, while
male students scored in the 60th. The question for the district changed:
Why is there a difference in performance between male and female
students, and how can this gap be closed?
Indicators of student achievement also go beyond academic attainment.
They include “softer” measurements. Qualitative measurements,
in a sense, humanize the data by going beyond the numbers. These “quality-
of-life” indicators include:
• job skills and preparation;
• citizenship (volunteerism, voting, etc.);
• appreciation of the arts;
• development of character and values (integrity, patriot-
ism, work ethic, etc.); and
• healthy lifestyle.
Interpreting Data Education reform is — and will continue to be — scrutinized by
the media, politicians and community stakeholders. Therefore,
it is vital that any data collected be interpreted accurately and
fairly. “Achievement gains tend to occur when a school ana-
lyzes student performance data and develops a plan of action based on that
analysis,” says Daniel Domenech, superintendent of Fairfax County (Va.)
Public Schools. “Conversely, schools that select programs based on ‘per-
ceived’ needs almost always meet
failure.”
The Vermont Department of Education recommends the following
when analyzing student performance data:
• Whenever possible, use data from the same sources from year
to year.
• Collect and analyze as much data as possible. Use multiple measures.
• Always consider the nature, size and characteristics of the group
being tested.
• Be sure to know what is being measured.
• Remember that large-scale assessment data are about the school’s
program — not just about the grade at which the assessment is
administered.
Using Data to Improve Schools: What’s Working
18
Even rich assessment tools are relatively impotent if not embedded in systems committed to the standards they address, hungry for the data they produce, and willing to challenge and redesign traditional practices and organizational structures that get in the way of each student [meeting] those standards.
— Kate Jamentz, director of programs
in professional and organizational
learning, WestEd
• Compare performance from different groups of students (disaggregated data). The
goal is to reduce differences in performance among groups while increasing excellence
for all.
Reacting to a single test score is perhaps the most common
mistake made when interpreting data. Longitudinal measure-
ment — conducted consistently from year to year — is neces-
sary to properly measure progress, growth and change. The con-
sistency of the data elements — necessary for any long-term
data collection and analysis — reduces confusion.
The level of analysis also can diffuse misinterpretation. The
“drill down” process is an effective method of disaggregating data. It begins with a general
question, then "drills down" the question into smaller and smaller parts. In Data Analysis
for Comprehensive Schoolwide Improvement, Victoria Bernhardt describes four layers of
disaggregation:
First-Layer Disaggregations
How many students are there?
Male vs. female
Limited English Proficiency (LEP) vs. non-LEP
Ethnicities
Lunch codes
Second-Layer Disaggregations
How have the demographics changed over time?
Increases vs. decreases in categorical variables
Third-Layer Disaggregations
What percentage of students are gifted, and are they equally distributed among genders and
ethnicities?
Fourth-Layer Disaggregations
How has the enrollment of LEP students entering the building changed over the years?
Do students with higher attendance get better grades?
Chapter 2: Using Data to Make Smart Decisions
If your data set is only test scores, you are doomed.
-— Suzanne Bailey, restructuring issues consultant,
Tools for Schools, August/September 2000
19
Using Data to Improve Schools: What’s Working
20
The Palisades School District is a 100-square-mile area in Upper Bucks County, Pa., with a population of approximately 15,300. It serves 2,200 students in three elementary schools, a middle school and a high school.
When Francis Barnes took the job as superin- tendent, he was charged with finding out why stu- dents received good grades on report cards but performed poorly on standardized tests. Then Barnes had to develop a plan to improve student performance.
The first task at hand was to collect existing data. “There were a variety of pieces of data in a number of places,” Barnes says. “The problem was that the data were not organized sufficiently to permit any analysis. Nor did the administrators or staff have training in the analysis of data for the purpose of improving student performance.”
Barnes and his staff developed an action plan that was aggressive, yet thoughtful and realistic. “We generated, by consensus agreement, a data- driven decision-making model,” he says. “Our data- driven process engages stakeholders in decision- making to attain district goals. This data-based decision-making vision required us to gather all the available data and analyze the information, seeing what we actually had and also what was missing.” This included mainly state test and standardized national test data.
The district then participated in a pilot project with the AASA Center for Accountability Solutions to use Quality School Portfolio (QSP). This project provided software for staff to input and organize existing data, and to develop other assessment measures to fill existing gaps.
“We believe that both qualitative and quantita- tive data are essential,” Barnes says. With this in
mind, the district applied a hospital model for assessing its educational health. “We used quantita- tive measures, such as test score results, as indica- tors of academic health in the same way physicians use numbers, such as temperatures and blood pres- sure readings, as measures of physical health. We also used softer qualitative measures, such as the result of interviews with students, in the same way physicians ask questions during physical examina- tions.”
Some of these softer qualitative measures are unique to Palisades. “Walk Through” data, a term coined by the district, are gathered by interviewing each student once each year using a faculty-devel- oped protocol. (See the list of questions boxed on p. 21.) “The interview results give us qualitative data as to whether students know what we expect of them in writing, reading analysis and math; whether they know how their work is scored; and whether they know how to improve it,” Barnes says. Responses are analyzed at the building level, thereby enabling faculty to assess whether students are learning what is expected. The data then are used to make necessary changes in curriculum and instruction.
As Palisades’ data-driven decision-making efforts continue, its data-collection methods have expanded. Barnes notes districtwide assessments in kindergarten through eighth grade are the direct result of data analysis. The district currently is for- malizing assessments in 9th through 12th grade.
The effort is producing results; Palisades School District students are scoring higher on standardized tests. For example, in 1998, Palisades High School students scored an average of 1270 on the Pennsylvania System of School Assessment (PSSA) reading test and 1250 on the mathematics test. (The state’s “proficient” score is 1300.) By 2001,
“Walking Through” Data Collection Palisades School District, Pa.
Chapter 2: Using Data to Make Smart Decisions
21
the students’ average scores increased to 1380 on both tests.
The district does not rely on just one data resource, however. “We use the state assessments in reading and math in grades 5, 8 and 11, and writ- ing in grades 6 and 9,” says Barnes. “While these are ‘good’ assessments… we were somewhat frus- trated in the 1996–97 school year that, while these tests measured student performance, we did not receive any data that would help us improve performance.”
The district subsequently adopted the New Standards Reference exams, which were adminis- tered for the first time in spring 1998 to students in grades 4, 8 and 10 in reading and mathematics. “These tests provide specific information by student, class and district for areas that need to be improved,” says Barnes. “We have taken this data seriously and provided the appropriate training to prepare our teachers to use the data to improve instruction.”
Barnes says the data indicated that students needed to work on writing conventions, reading analysis and interpretation, and math problem solv- ing. So he focused the first year on writing, the sec- ond year on math problem solving and the third year on reading analysis and interpretation. “Focus means that each year we provide training and time for administrators and faculty to work on imple- menting changes in their instruction based on what they have learned in the focus area,” says Barnes.
Faculty also have created quarterly assessments in math and language arts at each grade level to provide benchmark data to use for improvement. When students do not achieve on quarterly assess- ments, they participate in after-school and even summer-school tutoring sessions. The results are impressive. For example, in spring 1998, 33 percent of the high school students taking the New Stand- ards exams for reading analysis and interpretation scored at or above the standards. In 2001, that fig- ure was 64 percent. For mathematics problem solv-
ing, the number of high school students scoring at or above the standard increased from 18 per- cent to 43 percent.
Results also are evident in stu- dents’ accomplishments after they graduate high school. They are entering better postsecondary institutions; students from Palisades High School’s Class of 2001 now attend such prestigious schools as Boston University, The College of William and Mary, Franklin and Marshall College, Temple University and Le Cordon Bleu. And Barnes notes that the community is showing more interest in and support of stu- dents, staff and the district’s efforts to help all students fulfill their potential.
Elementary Reading Analysis and Interpretation Walk Through Questions Durham Nockamixon Elementary School, Palisades School District, Pa.
• Can you share a connection you have made with text this year?
• How did that connection help you understand the story?
• What clues do you use to make predictions when you read a story?
• What kinds of questions do you ask yourself before, during and after reading a book or story? Can you share a question you’ve written down?
• What do you do when you do not understand what you are reading? How would you help a friend who didn’t understand what he was reading?
Accountability Plans School districts have been mandated to improve student performance. Domenech, whose dis-
trict is responsible for more than 158,000 students, shares his experience: “To blunt the criticism
of a community that implied the schools constantly asked for more money but were never will-
ing to be held accountable, we devised a plan that allowed for objective measurement of our
performance and we agreed to be held accountable for our results, or lack of.”
Fairfax County’s plan includes a process for determining a biennial objective and measurable
targets that are part of a system that provides each school with specific and measurable targets
for student achievement. Rewards and sanctions apply to schools and students. For schools, the
rewards include bonuses; the sanctions include reconstitution. For students, there are promotion
and retention guidelines by subject and grade level. Students who fail to meet the benchmarks
are required to attend remediation programs and summer school.
Implementing a full-scale accountability plan is not easy. “It is an education process and a
change in culture,” says Domenech. “We had the immediate support of the school board and
the community. Getting the support of the administrators and staff was more difficult. Everyone
had an excuse as to why objective measurement of our performance was not fair.”
Armed with the right data and a full understanding of what the data show, superintendents
can effect change as well as measure progress. Says Domenech, “The use of gain scores tended to
neutralize the socioeconomic factors and did away with most objections. The reward system also
helped, although the sanctions created a great deal of angst.”
Ownership and Responsibility It is important to get community ownership and buy-in at the front end of developing an
accountability plan — a method by which districts, schools and students are evaluated and held
responsible for performance. Stakeholders should have input into deciding what counts and
what districts should measure. By including all stakeholders in
these decisions from the start, school districts benefit from their
support throughout the process.
In 1995, the Plainfield (N.J.) Board of Education adopted six
goals focusing on school reform. Those goals included setting
standards, identifying and establishing benchmarks, improving
evaluation of personnel and providing skills development, and
completing a community planning process that would result in
a comprehensive district improvement plan.
The district spent the next two to three years laying the
groundwork for standards-based reforms, according to Joshua P. Starr, Plainfield’s director of
accountability. Most of the efforts were aimed at building community engagement. District lead-
ers explained to parents, business leaders, and other community members what the reforms
Using Data to Improve Schools: What’s Working
22
All stakeholders take ownership and respon- sibility for our children’s achievement. We are convinced that community engagement is necessary in order to progress.
— Inez P. Durham, superintendent,
Plainfield (N.J.) Public Schools
were and how the reforms would improve student achievement.
They also listened closely to community concerns.
The district’s efforts were effective: In 1998, voters approved
on the first ballot a $33.9 million education referendum that pro-
vided the necessary resources for expanding the district’s educa-
tion reforms. That same year, the board adopted a specific goal to
deal with reforms based on research. “This meant that resources
had to be put into place to ensure the efficient, effective gather-
ing and analysis of data,” says Inez P. Durham, superintendent of
the district, which serves more than 7,200 students. In 1999, the
board established an office of accountability. “From 1995, it was
clear that all stakeholders had a responsibility for student
achievement and school reform — and would be accountable.”
Plainfield is required by a court case to have a districtwide
plan. The plan’s acceptance and success is contingent upon sup-
port from all stakeholders. Therefore, members representing the
community, administration, parents and other groups were given
the opportunity to participate in the plan’s development. The
resulting plan, approved by the board, has been well-received.
“The plan is designed to promote local ownership and collec-
tive responsibility,” Durham says. “It is locally driven and
identifies the outcomes and indicators that the Plainfield
community values and expects. The system is designed to sustain good practices, continual eval-
uation and continuous improvement based on reliable and comprehensive data. It is a system of
rewards and consequences.”
Plainfield’s accountability plan is implemented throughout the district via its Leadership
Innovation and Change Councils (LINCCs), which are site-based management teams that
include representatives from administration, teaching staff, support staff, parents, community,
business and corporations and, when appropriate, students.
In addition to a districtwide LINCC, each school has its own LINCC. The school-level LINCCs
measure progress toward achieving six non-negotiable goals. There are six scoring guidelines for
each goal. “It has been a challenge for all to become comfortable with following the procedures
for collecting and reporting accountability data and to score utilizing rubrics,” Durham says. The
district is helping LINCC members overcome those obstacles with dry runs, inservice programs
and workshops.
The plan has widespread support and interest. “All stakeholders take ownership and responsi-
bility for our children’s achievement,” Durham says. “We are convinced that community
engagement is necessary in order to progress.” (See Chapter 3 for more about community
engagement.)
Chapter 2: Using Data to Make Smart Decisions
23
Establishing Performance Targets Four main sources provide helpful informa-
tion when establishing performance targets.
Superintendents can:
• Review relevant state and national
performance standards.
• Discuss expectations with important
stakeholder groups.
• Compare students’ performance
with that of an exemplary school
or program.
• Compare students’ performance with
state and national averages.
Source: At Your Fingertips: Using Everyday Data
to Improve Schools, 1998
A Closer Look Phil Streifer Associate Professor of Educational Leadership, University of Connecticut
What is the first thing superintendents should consider when they develop a data-collection program?
Data collection should start with the questions they most want to address. Ask: What data do
I need to answer these questions? This will quickly lead to a list of the data needed —
something I call a data audit.
An issue of concern always is “What will it cost to collect the data?” I encourage superin-
tendents to consider the cost benefits of collecting data in terms of the value they will provide
in answering their questions. In some cases, questions may
need to be modified because the data will be too costly to col-
lect. For example, surveying large groups of constituents can
be time-consuming and costly. It may be appropriate to do so,
but only after thoughtful review of whether it’s absolutely
necessary or whether some other sampling might be equally
valuable.
There are different types of data. How do you distinguish
among them?
Data can be separated into three broad categories for analysis
and interpretation: input, progress and outcome variables.
Input variables are variables that students come to school with, such as their background, pre-
learning, socioeconomic status and so on. Input variables also can include teacher variables,
such as their level of education or previous in-service training experiences. Process variables
are those that currently are at play on the issue under consideration, such as the curriculum
in use, instructional techniques and materials. The levels of teacher and parent participation
in decision-making also are process variables. Outcome variables are the results or measures of
both input and process variables, typically student tests, observation data and survey results.
A number of school districts do not have a districtwide accountability plan. Where should they begin?
I would start with the district goals and build an accountability plan around those issues.
Second, I would make sure that there is a high degree of match between the data needed and
the data collected. Generally, this issue of match is first measured by the data’s reliability (abil-
ity to consistently yield the same result over time) and validity (the degree to which the data
measure what they purport to measure). Finally, match is determined by the level of appropri-
ateness of the data to your population.
Using Data to Improve Schools: What’s Working
24
I think it’s also important to desensitize the annual reporting of high-profile data by having regular reports to the board, staff and parents on student performance covering a wide range of variables.
— Phil Streifer, associate professor of
educational leadership, University of Connecticut
What types of data are necessary to create an effective accountability plan?
I would focus first on district/board goals or mission-critical problems. Next, I would generate
data that allow for longitudinal analyses — over several cohorts (groups of individuals having
a statistical factor, such as grade level, race or age) if at all possible. Thus, if mastery tests are
administered in the fall, I would look to a comparative assessment in the spring, or one that
would allow for fall-to-fall analyses. Overall, longitudinal analyses yield the most useful infor-
mation. Beyond this, I would ensure the data you have or are about to collect will provide
you with the information needed. Poorly constructed surveys, for example, yield no useful
information at all. Tests that are too easy or too hard also have limited utility.
Board, staff, parent and public support are essential. How can school administrators get community
buy-in for such a plan?
I don’t think you can get support after the fact — that is, when the results are in. Early on, we
need to engage these folks in why these data are needed and in helping them to understand the
data’s limitations and value. I think it’s also important to desensitize the annual reporting of
high-profile data by having regular reports to the board, staff and parents on student perform-
ance covering a wide range of variables. With consistent, regular updates from the district, the
once-a-year state report may not take on the degree of importance it sometimes does now. I
have found all of these constituencies to be thoughtful and reasonable once we take the time to
explain the complexities and how the data are used to gauge progress and design improvement.
Summary Key points in Chapter 2 include:
• Districts should use multiple assessment and nonassessment measures when assessing
student performance.
• Disaggregated data is a powerful tool to help superintendents identify which students are
achieving, which students need additional assistance and how best to target limited
resources.
• Methods for data collection vary; many superintendents start with the question: What
do I want to know and what data will help answer this question?
• A districtwide accountability plan provides for objective measurement of performance
and holds boards, superintendents, principals, teachers, parents, students and others
accountable for results.
Chapter 2: Using Data to Make Smart Decisions
25
Superintendents understand the importance of parent and community engagement, strategic
communications and working with the media. They know that, unless they communicate
clearly and honestly about data and what the data reveal about student learning, their efforts
will fail and their credibility could be damaged
irreparably.
“School improvement requires a constant conver-
sation with our community,” says Beaverton (Ore.)
School District Superintendent Yvonne Katz. “The
community’s input informs our processes, procedures
and what we do. I’m convinced I wouldn’t be here if I
didn’t gather and use community input.”
Success in promoting and sustaining data-driven
decision-making as a tool for school improvement depends on:
• Educating parents and community members about what different data convey and
how to use data to measure the success of school-improvement efforts, budget deci-
sions, teacher quality initiatives and other practices.
• Asking teachers, parents and community stakeholders what their concerns are for
schools and what data they need to see to know schools are improving.
27
Chapter 3: Data, Public Engagement and Strategic Communications
• What data matter most to your community?
• In what ways is the community involved in discussing data, such as test results, school safety reports and teacher quality ratings?
• How is community input on how to improve schools gathered?
• How do teachers, staff and board members communicate with parents in the district?
• How is jargon-free data reported to the community, both the successes and the challenges?
• Who are the most credible messengers in the district?
• How can the district build an effective working relationship with the media?
To depend on the local newspaper editor or reporter to do all of your public relations for you is like depending on the hospital orderly to do your open-heart surgery.
— Brian Hale, member, Neb. School Boards Association
• Listening to the insights of these different stakeholders and reporting back to them
how their input may shape decision-making.
• Sharing the district’s successes and failures along the way.
• Speaking in clear, easy-to-understand language that is free of “educationese.”
• Understanding that the public schools belong to the public, and school leaders need
the public’s permission to do the work they are charged with doing.
This chapter explores three cornerstones of effective communication: public engagement,
strategic communications and media relations.
Public Engagement: Listening to Your Community How do superintendents proactively involve their communities in understanding and asking
informed questions about data? How do they determine which indicators matter most to various
stakeholders — parents, taxpayers, teachers, business leaders, senior citizens and others? How do
district leaders build support early on so that when controversial data are released, the public
says, “How can we help?” rather than assigning blame or finding fault?
“What’s really important is to clarify what the community’s role is,” says Superintendent Jane
Hammond of Jefferson County (Colo.) Public Schools. “We gather input from the community
and then we get together with educators to decide what we want to do to get the results the
community wants.”
Engaging the public begins with listening. It’s not
uncommon for an educator’s view of success to differ
significantly from the views of a taxpayer, parent or
business leader. (See Communicating Data to the
Public on page 32.) Yet each of these constituents has
a stake in public schools.
One of the most effective ways superintendents
and district staff can engage their public is by asking:
What would you need to see that would indicate our schools are improving? Other questions
include: How do we as a community define success? What elements are important in defining
student success? How do we as a community work together to ensure success for our young peo-
ple? The answers to these questions not only give superintendents a clearer sense of the public’s
priorities, but they also identify what data are most meaningful.
“When asked what their priorities are for spending money, our local community said ‘we
want a quality teacher in every classroom, including giving teachers the coaching they need to
improve,’” Hammond says. “Professional development emerged as a higher priority than reduc-
ing class sizes. I have never before seen the public be willing to support staff development.”
Using Data to Improve Schools: What’s Working
28
The fundamental purpose of any public engagement initiative is to channel a community’s concern, apa- thy or anger into informed and constructive action.
— Annenberg Institute for School Reform, Reasons for Hope,
Voices for Change, 1998
The most effective strategies for community listening are face-to-face meetings, such as:
• Focus groups, typically involving 8 to 12 people. Some focus groups are conducted with a
cross-section of people representing the community or a specific subgroup, such as parents
with children in public schools or recent high school graduates. A skilled, neutral modera-
tor leads the discussion using a script and predetermined questions. The moderator listens
for key themes, new ideas, differences of opinion and common ground.
• Community forums/town hall meetings,
which may attract anywhere from 60 to 200 peo-
ple. Typically, these meetings focus on one issue or
pose a question such as, “What should students
know and be able to do?” or “What does a good
school look like?” The information gathered at the
forum can be used to help inform the district’s
strategic-planning process, goal-setting or data-
gathering efforts.
• A study circle involving parents, educators, cler-
gy, students and other publics. Together, they use
dialogue to arrive at a decision about an issue or a problem. A study circle typically is made
up of 10–15 people who meet weekly for two hours over four weeks. A facilitator leads the
discussion, often drawing from a discussion guide shared with participants. During the final
session, the group decides how to address the issue and offers solutions.
• High-level advisory committees or superintendent leadership councils made up
of a cross-section of business people and community leaders who periodically meet with the
district superintendent. These committees offer guidance on key issues, provide a heads-up
about what they are hearing in the community and help brainstorm alternative methods of
public engagement. The councils can be issue-specific, addressing an issue such as how to
win support for an upcoming bond issue, or they can be more general and focus on broadly
defined areas such as leadership.
As school leaders begin to implement these strategies,
the challenge becomes how well they connect their
work to the priorities of the community. Lackluster pub-
lic engagement efforts can result in better packaging of
more of the same rather than increasing student learn-
ing by using good data, gathering community input and
making strategic decisions based on both.
Chapter 3: Data, Public Engagement and Strategic Communications
29
If you don’t listen to your community, you may make some assumptions that you are meeting their needs, but you are not. I find public engagement makes our decision-making easier, otherwise you are just guessing what the public wants.
— Jane Hammond, superintendent,
Jefferson County (Colo.) Public Schools
I don’t think you’ll survive without engaging your public. The public has to be in public education.
—Yvonne Katz, superintendent,
Beaverton (Ore.) School District
Using Data to Improve Schools: What’s Working
30
Since Superintendent Clifford Janey arrived in
Rochester, N.Y., in 1995, his work has been guided
by a simple but powerful idea: If community
members and educators have a shared under-
standing of what schools should do, they can work
together to make that happen. Realizing the
promise of that idea has not been easy, but the
payoff has been enormous for the district’s nearly
37,000 students.
At the start, Janey found a community demor-
alized by a history of low achievement. Experiment
after experiment had been tried, but no “silver
bullet” of school reform had pulled the communi-
ty together to improve schools. Janey believed that
Rochester was “idea-rich and focus-poor.”
To help build a common understanding of
what the community wanted from its schools and
how that vision could be made a reality, Janey
invited the community to help set specific, meas-
urable goals for schools. He met with scores of
Rochester residents and community leaders to
learn their priorities for the district.
Next, Janey worked with the board of educa-
tion and district staff to formulate a set of per-
formance benchmarks — goals for the district to
meet by 1999 — based on the public’s concerns.
Detailed in the plan were 40 specific indicators of
success in the four major areas that community
members had highlighted:
• Student achievement and citizenship
• Safe and supportive schools
• Community-school partnerships
• District accountability
But the community’s participation didn’t stop
there. As part of this plan, the district secured
commitments from community partners, including
city and county government, social service agen-
cies, colleges and universities and the private
sector. A coalition of local groups sponsored an
annual event to chart progress, raise awareness
about these efforts and share information on
district performance.
During the next couple of years, Rochester
made progress toward nearly all of the bench-
marks, and public sentiment toward the district
improved. When Janey decided it was time to
revisit the district’s strategic plan, he stuck with
the strategy of community involvement that had
worked in the past.
To assess whether the original benchmarks
were still relevant, the school conducted focus
groups and interviews with more than 200 teach-
ers, administrators, parents, civic leaders, employ-
ers, religious leaders and local politicians in 1997.
Community members voiced support for the
benchmarking initiative, but they said the original
benchmarks were not ambitious enough, not
streamlined enough and not focused enough on
core academic skills.
Rochester education leaders spent much of
the next few years processing this feedback,
retooling the strategic plan and gauging the
district’s progress. The district also conducted
public opinion surveys to evaluate the attitudes
of parents, students and school staff about
district efforts.
Engaging the Public in Strategic Planning Rochester (N.Y.) City School District
Chapter 3: Data, Public Engagement and Strategic Communications
31
Based largely on community input, the district
unveiled revised benchmarks in the 2000–01
school year. District officials had cut the number of
benchmarks and honed their focus. These revised
benchmarks revolve around two goals:
• All students will meet high standards in
reading, writing, mathematics, science and
social studies.
• All students will graduate prepared for
civic participation, higher education and
the world of work.
Buttressing these goals are two new, specific
benchmarks that describe the progress the district
is expected to make between the 1999–2000 and
2004–05 school years:
• Within a given group of students who
enter high school together in ninth grade,
the percentage of students receiving a
diploma within four years will increase
from 44 percent to 90 percent.
• The percentage of students receiving a
Regents diploma with honors (indicating
those students passed the New York
Regents exams with an average of 90
percent or higher) within four years will
increase from 3 percent to 15 percent.
Rather than erecting walls to shut out the
community, Janey and other Rochester school offi-
cials repeatedly invited the public to join in the
work of strengthening education. The results have
been sweeping changes and steady improvements.
The district has not yet met all its goals, but it has
made progress toward every benchmark set by the
community — and it remains committed to com-
munity engagement as a way of guaranteeing that
public schools serve the public.
Excerpted from Danforth Foundation Report,
Rochester City School District, N.Y., 2001.
Crafting a Communications Plan The following are vital elements of a communications plan:
• Messages: What do we hope to achieve as a result of communicating?
• Audiences: With whom do we want to communicate?
• Results: What measurable effect do we hope to achieve?
• Medium: What communications method do we want to employ?
• Timeline: By what date do we want to be successful?
• Expense: How much are we willing to spend on communications?
• District and school staff: Who will be responsible for successful communications?
Engaging Business Leaders Who knows a community better than the community itself? The Florida Chamber of Commerce is exploring new ways of listening to its community. One idea is to use volun- teers as “listening posts” who keep their ears close to the ground. Bus drivers, hairdressers and others in the community could make it their job to pick up rumors circulating about the school district. When needed, they could stop the rumor before it can spread by offer- ing the “real story” about student achieve- ment, teacher recruitment, a bond issue or whatever the hot-button topic might be. These “listening posts” could also be con- vened periodically to discuss issues as a group or report to each other as needed.
Communicating Data to the Public An effective way to build public support and increase
community confidence in public schools is to show the pub-
lic how schools are being held accountable for results. Sharing
data in easy-to-read charts and short, jargon-free reports helps
deepen community understanding of the issues facing public
education.
A common concern voiced by superintendents is that the
community reads limited coverage in the local newspaper
about its school district or that reporters only cover the “bad”
news. But too often educators make the mistake of placing
too much stock in media reports. Polls show parents most
often get their information from teachers, the principal, other
parents and their children. Districts play an important role in
disseminating factual, timely information to parents about stu-
dent achievement — the information parents care about most.
Increasingly, districts are reporting information directly to
their communities with the help of a school report card or
annual performance report. Many successful districts use the
report to guide discussions on performance and education
priorities at school forums, parent meetings, staff meetings
and board meetings. Training teachers and principals to help
facilitate these conversations ensures everyone is focused on
the data, what the data reveal about performance and how to
improve instruction in the classroom.
Using Data to Improve Schools: What’s Working
32
If we are interested in community building, then we, along with members of the proposed community, are going to have to invent our own practice of community. Despite the difficulties, if we are success- ful, our community will not be counterfeit but real.
— Thomas Sergiovanni,
Building Community in Schools, 1994
Who Has Credibility in Your District? Too often, superintendents count on the media to share district news and information with the public. While the media play an important role, surveys show teachers and other staff members have more clout with parents seeking information.
A 1996 survey conducted by the Denver-based Education Commission of the States shows teachers have the most credibility among parents. Here’s how the survey ranked who parents rely on “heavily” or “somewhat heavily” as sources of information about education issues:
Source: Listen, Discuss, and Act: Parents’ and
Teachers’ Views on Education Reform, 1996
88% Teachers
83% Children
82% Other parents
72% School officials
68% Print media
45% TV and radio
“We have spent countless hours and thousands of dollars training administrators on how to use
data,” Katz says. “We know who is doing well in what area and who is not doing so well. We talk
about the data and don’t push the numbers under the rug. You cannot be a smoke and mirrors
superintendent anymore in this country. People are very smart about wanting to see the evidence.”
A critical first step of reporting data to the community is finding out which data different
stakeholders care about most. Often there are discrepancies among constituencies. For example,
educators may want to see more demographic data, while parents tend to rank school safety and
achievement high on their list.
A national research project published in Education Week in 1999 (see Appendix A) asked par-
ents, taxpayers and teachers across the country what indicators they would use to hold their
schools accountable. Here is a sample of how parents, taxpayers and educators ranked their top
five indicators:
Parents Taxpayers Educators
School safety School safety School safety
Teacher qualifications Teacher qualifications Class size
Class size Graduation rates Teacher qualifications
Graduation rates Dropout rates Graduation rates
Dropout rates Percentage of students Per-pupil spending promoted to the next grade
Chapter 3: Data, Public Engagement and Strategic Communications
33
More and more school districts are using the Internet as a tool to communicate district data to their communities. Achievement data for stu- dents in Orange County (Fla.) Public Schools, Palisades (Pa.) School District and Plano (Texas) Independent School District are only a click away for parents, business leaders, teachers and any- one else who has access to the World Wide Web.
Orange County's website — www.ocps.k12.fl.us — boasts detailed school-by- school enrollment data, scores from state and voluntary tests, an online parents guide and detailed frameworks of higher achievement in all subjects at every age level.
As comprehensive is Palisades School District’s site — www.palisadessd.org — which posts results by grade and school for the stan- dards in reading, writing, mathematical skills, mathematical concepts and problem solving, as well as the percentages of graduates attending two- and four-year colleges.
District data also is a main highlight of the Plano School District website — www.pisd.edu. The district’s Department of Research, Assessment and Evaluation posts assessment results, the yearly district accountability report and detailed explanations of all tests adminis- tered in the county.
Websites as a Communications Tool
Strategic Communications: Getting the Most for Your Effort Districts have a wealth of data to share with their communities: dropout
rates, test scores, attendance rates, school safety data, budget figures and
other information. One of the biggest
challenges facing districts is determining where to begin. What informa-
tion is most useful to communicate? Superintendents do not have the
time or the resources to communicate with everyone about all of the dis-
trict’s initiatives. That is why it is important to be strategic. Simply said:
Set priorities, gather input early on, plan ahead, be proactive and always
evaluate the results of communication efforts.
Because so many different initiatives and programs compete for atten-
tion, it can be challenging for superintendents to keep their communica-
tion efforts focused on improving student performance. District leaders
who have turned around poorly performing schools say they worked
hard to keep their communication with staff, parents and the public
focused on achievement.
For example, if a district’s emphasis is on helping all students meet
standards, school visitors should be greeted with student work that
meets the standard, scoring guidelines outlining academic expectations,
and students who can explain what they need to know and be able to do
Using Data to Improve Schools: What’s Working
34
User-Friendly Report Cards Here are some helpful guidelines to ensure that a school’s report card is read and used.
• Keep school report cards short, such as a six-panel brochure. Have a more detailed version available for peo- ple who want more data.
• Add short narrative descriptions. School data are not self- explanatory to nonexperts.
• Advise parents and others how they can use the data. It’s not obvious to most.
• Spend as much time working on distribution as on pro- duction. Research shows that even most teachers say they’ve never seen district report cards.
Source: Reporting Results: What the Public Wants to Know About Schools, 1999
Graphical Representation “Graphics reveal data,” notes Edward R. Tufte in The Visual Display of Information. According to Tufte, graphics communicate complex ideas with clarity, precision and efficiency. Graphical displays should:
• show the data
• induce the viewer to think
about the substance rather than
about methodology, graphic
design, the technology of
graphic production or some-
thing else
• avoid distorting what the data
have to say
• present many numbers in a
small space
• make large data sets coherent
• encourage the eye to compare
different pieces of data
• reveal the data at several levels
of detail, from a broad overview
to the fine structure
• serve a reasonably clear pur-
pose: description, exploration,
tabulation or decoration
• be closely integrated with
the statistical and verbal
descriptions of a data set
at each grade level. Schools can illustrate the progress students are making on tests that measure
whether they are meeting the standards with the help of easy-to-read bar graphs and charts that
help explain what data the district is using to measure progress. This approach helps parents and
community members understand how these different efforts are connected. It also shows them
that they are receiving important and timely information on school quality.
Taking the Pulse of the Community A poll is a structured method to determine the level of support or intensity of opinion on an
issue, product or philosophy. Public opinion polling is one way to gauge the community’s views
on education issues. For example, educators can use a poll to quantify the level of support for a
proposed ballot issue or to test different options for a new start time for school. Results of
polling should be used to inform decision-making, not as the sole basis for decisions.
When considering whether to use polling, superintendents should think about the resources
required (it can be expensive), the importance of the information to the decision-making
process and the specific information needed so they’ll know more than they did before they
tested the waters.
Here are some key points to remember when
writing a poll:
• The way a question is asked can determine
the answer.
• It’s important to look for obvious questions
that are not asked by the pollster.
• Respondents are more likely to tell pollsters
they support plans favoring change,
even if they do not fully understand how
the proposed change will affect them.
• Not all pollsters make sure respondents
know enough about the subject of the poll
questions to give meaningful answers.
• There are many diverse publics, as well as
variety in thinking among specific sub-
groups of a given population.
Superintendents must research what each of
these constituencies thinks, feels and wants
before making major decisions.
Chapter 3: Data, Public Engagement and Strategic Communications
35
School Ambassadors The National School Public Relations Association
says the following school employees, in order,
are best known to the public:
• School secretaries
• Custodians
• Bus drivers
• School nurses
• Teachers
• Principals
• Superintendents
• School board members
Think of each employee as a school ambassador
to the community. Provide them with communi-
cation skills and training. They are among the
district’s most important messengers.
Working with the Media: Making Sure Your Message Is Heard Think back to the last interview you did with a reporter. How did what you said during the
interview translate to the printed page or television? Was your message heard? Working with the
media can be challenging, especially when you are trying to communicate complex data and
information that doesn’t neatly fit into a 10-second sound bite.
Before any media interview, know the opposing side’s point of view as well as you know your
own. Decide what the goals are for the interview. For example, do you want to raise awareness
about trend data that reveal improvement in student test scores? Are you sharing district strate-
gies for turning around low-perform-
ing schools? Identify your audiences
and which messages resonate best
with them. Stay positive, succinct,
solution-driven and in control of the
interview.
For more on media interview tech-
niques and related materials, see
Appendix B on page 54.
Summary Key points in Chapter 3 include:
• Teachers, principals, district staff and the community should be involved in gathering,
analyzing and discussing data.
• Success in promoting data-driven school improvement depends on educating parents
and community members about what information different data convey.
• Understanding what data the community uses to measure whether schools are improv-
ing helps superintendents avoid major disconnects in communicating information
with taxpayers, business leaders, parents and other community members.
• The most effective strategy for listening to community concerns is a face-to-face
meeting with different or multiple constituencies, such as a focus group, a community
forum, a study circle or a superintendent leadership council.
• Crafting a comprehensive strategic communications plan helps superintendents
identify key messages, audiences, results, effective tools (e.g., web-based data charts)
and timelines.
Using Data to Improve Schools: What’s Working
36
The public schools really are the public’s schools. If they are given the impression that they are welcome to participate only if they can do something educators think worthwhile, this puts the cart before the horse (i.e., treats citizens as a means) and disconnects the public from schools.
— David Mathews,
Is There a Public for Public Schools?, 1996
Capitalizing on recent advances in technology and research regarding educational improve-
ment, superintendents nationwide have developed countless strategies for using data to
drive decisions at the district and school levels. Whether they are data experts or novices,
however, they probably could be doing more to make this invaluable resource — data — work
for them.
How? Listen to the stories of superintendents and other education leaders from across the
country who have been using data to drive decision-making. More than anything else, their
experiences underscore the importance of team build-
ing. After all, leaders are only as good as their teams.
Superintendents illustrate the power of building a
data-friendly culture, ensuring that school board
members and staff understand their roles and respon-
sibilities, providing the training needed to foster new
data skills and establishing a system focused on
continuous improvement.
Changing the Culture Data traditionally have not been a major factor in the ways schools and districts make
decisions. The intuition of principals and teachers, advocacy by parents and political interests
often have guided decision-making. But all that is changing.
37
Chapter 4: Strategies for Success • How can the school district’s culture shift to one that encourages using data
to drive decisions?
• What roles should a superintendent, school board members, central office staff, principals, teachers and others play in collecting, processing, interpreting and using data?
• How will key players be trained to effectively participate in data-driven decision-making?
• Is the data system being built able to foster continuous improvement in the school district?
• What can be learned from other successful districts?
This is the critical point of data: It’s not necessarily the answers that are most important; it’s the discussion that occurs and the questions that are asked because of the data.
— Karen Bates, superintendent,
Lake Washington (Wash.) School District
The call for data-based accountability, trumpeted first by state education leaders, has been
met with skepticism by many of the people in schools who are being asked to collect and use
data. Teachers and principals sometimes react initially with fear, distrust and resistance. Their
first questions, if they dare to ask, concern how the data will be used against them.
“Teachers are afraid of data because they think their jobs may hang in the balance, based on
how their students are doing. Principals are worried about where they will get the time to ana-
lyze data,” says Superintendent Ray Yeagley, who has spent years easing the data-driven fears of
educators in the Rochester (N.H.) School District. “The superintendent has to help people see
the global picture about what data can and can’t do. I need to help my principals find the time
to make this a priority. I need to reassure my teachers about how it will and won’t be used, so
they feel it won’t be used unfairly.”
To overcome these hurdles, education leaders recommend:
• Taking every opportunity to show principals and teachers that data are not being
used to “get” them, but to improve student learning. Tom Glenn, superintendent of
the Leander (Texas) Independent School District, says he used every outlet he could
find — including the district newsletter, meetings, workshops and private conversations
— to get that message out. “We still encounter resistance, from time to time, to laying
the data on the table. But that happens less and less as our culture helps people under-
stand that we’re not asking these questions to harm people,” he explains. “You have to
act appropriately with the data and make sure people understand: We’re working on
the system.”
• Helping district faculty and staff understand that using data, much of which already is
public, is a shared responsibility. The Poway (Calif.) Unified School District has adopted
the philosophy that it is made up of “professional learning communities,” says Ray
Wilson, director of learning support services. “We all work together for the common
good. If one of us has poor test scores, it’s our job as a department to figure out why and
help that person get test scores up,” he says. To help people become more comfortable
and familiar with data, Wilson’s department devised a scavenger hunt for the district’s
part-time school-based data staff. “They
had to create a school profile that included
teacher credentialling rates, mobility rates,
test data and the number of years that the
teachers at each grade level had been
teaching,” he says. “All this information is
available on the Internet. Many people
were surprised how much information
about their school, which they struggle
every year to collect, is public.”
Using Data to Improve Schools: What’s Working
38
The importance of data for administrators, policymakers and teachers in the classroom — to be able to break data down and know where strengths and weaknesses reside — is crucial if you want to make any kind of improvement.
— Monte Moses, superintendent,
Cherry Creek (Colo.) School District
Chapter 4: Strategies for Success
39
Collecting the Right Kinds of Data Leander (Texas) Independent School District
The need for data has been felt acutely, even among school systems in Texas where the Texas Assessment of Academic Skills (TAAS) has set a national standard for data collection, says Tom Glenn, superintendent of the Leander (Texas) Independent School District.
In the early 1990s, as TAAS was taking hold, Glenn and other district officials realized that their teachers and principals needed something the state test did not provide: frequent updates on student progress that could be used to guide instruction and address problems throughout the school year.
Despite its strengths, TAAS tests students only once annually, near the end of each school year. Results are reported in the summer, when educa- tors are powerless to address achievement short- falls. What those shortfalls might be remains some- what murky, because TAAS reports on student per- formance in terms of broad categories rather than specific skills.
So Leander began exploring an additional test- ing option. Principals and teachers worked togeth- er for two years to revise language arts and mathe- matics curricula, then worked for another year to help develop a test based on those curricula.
The resulting Student Profile System tests stu- dents from kindergarten through high school in language arts and mathematics three times a year — at the start of the school year, in the middle and at the end. Principals, teachers and students receive reports, including easy-to-understand charts that show specifically where students excel and where they need to focus more attention.
“The first time we used the test, on the first day of the school year, we had elementary students in tears,” recalls Glenn. “They said, ‘I’m in the fourth grade, and I’m taking the fourth-grade test
at the beginning of the year — I don’t know any of
this stuff!’ It took us a while to help students under-
stand that they were not expected to know any of
this stuff yet — that we were just trying to find out
how much of it they did know, so we didn’t waste
time at the beginning of the year teaching them
things they already knew.”
Teachers and principals were concerned, too.
“We promised them: ‘We will never use this assess-
ment to evaluate you; it is for your use,’” Glenn
says. “We had to prove it over and over again. We
told people that the system is the problem; individ-
ual people are not necessarily the problem. We
called it ‘failing forward’ or ‘learning from our mis-
takes’ or ‘working on the system’ — we called it
lots of different things. We had to help people trust
and understand that, as we collected data and
found things that weren’t always positive, they
were not going to be hung out to dry.”
Glenn says the 13,200-student district contin- ues to use the system because it:
• focuses attention on learning, not testing;
• generates data that can be used to improve education, not just categorize students and schools; and
• allows students, as well as teachers, to manage the learning process better.
Principals, teachers and students report that the system is helping them work more effectively than ever before to improve learning — which is the purpose of testing, Glenn notes. “Once people understand that you will not use data to beat them up,” he says, “then they will help you collect that data and be very honest about it.”
Ultimately, to shift the culture of your district to not only tolerate but embrace data, you
must not rely on data so much that you discount the training, experience and expertise of the
people who you want to use the data, says David Benson, superintendent of Blue Valley (Kan.)
School District.
In the early stages of Blue Valley’s data-collection efforts, a respected administrator, now a
deputy superintendent, fumed visibly for several weeks before finding the words to voice his
frustration, Benson recalls. “He came out of a meeting and said, ‘Dave, you’re ignoring my intu-
ition. I feel this way about this — and because I feel this way, you have to listen to me.‘“ Benson
says, “It was such an epiphany for me to understand the importance of that point: You cannot
take a human institution and just crunch it to data points. You have to listen to people.”
Benson listened to the administrator, as well as to
other critics. As a result, the district’s data system has
been improved. Moreover, it has survived and thrived
as principals and teachers throughout the district have
made way for data in their work. Today, the 17,000-
student district uses local, state and national test data
in its accreditation process, school profiles and strate-
gic planning. But that did not happen overnight.
“It’s a gradual process of proof,” says Benson. “If
you can demonstrate that what you’re doing can
make a difference in the lives of the students we serve, then it will gain acceptance. If it is data
for data’s sake only, and is not related to our primary mission of providing an academic educa-
tion to children in a safe and orderly climate, then it will not gain acceptance.”
A Closer Look Spence Korte
Superintendent, Milwaukee (Wis.) Public Schools
Why do school districts need to be able to make decisions based on data today?
We have a history of going after money for a hot new project and running it for three or four
years, and then it becomes part of what we do. Nobody is really getting at the critical ques-
tions: We just spent over a billion dollars this year — is that money put in the right places? Is
it getting results?
In Milwaukee, we’ve been building a sophisticated network to collect student informa-
tion. Each of our schools gets a report on how their students are doing. It’s pretty practical
from a principal’s point of view. For example, how many ninth graders are coming in below
grade level, and do I need to change my hiring pattern so I have more remediation for kids
coming into my school? The information is online. People in schools can access all their
records 24 hours a day.
Using Data to Improve Schools: What’s Working
40
One of the primary responsibilities of a leader is to help people confront problems. The way you confront problems is you identify them, and the way you iden- tify problems is by looking at data.
— Terry Grier, superintendent,
Guilford (N.C.) County Schools
What has worked well for your district in terms of setting up a sound data system?
Driving our whole technology design and infrastructure is an ex-high school principal. He
comes with 24 years of field experience in knowing what information has to look like to add
value for schools. Instead of recruiting an information systems technician to manage this,
we’ve gone the other way and drafted a practitioner. The result is that our data tend to be
more user-friendly and more school-oriented than they might be otherwise. That’s something
to think about when you’re out looking for an IT person. You can always surround that per-
son with technicians, but if he or she doesn’t know what schools need, the fit you get might
not be what you want.
How important is data training for teachers, principals and
other district employees?
It’s absolutely critical. In retrospect, one of the things
that we did right was take an existing building and set
it up as a training center. We cycle people through
every day.
All our best technology teachers are there. Almost
all of them have a background in schools, so it isn’t
“techspeak” as much as it is, “When I was a teacher at
such-and-such a school, this was a big problem for me.
I bet you have that problem at your school. Let me show you how we solved it with technolo-
gy.” We have good credibility with the teachers.
We do our own training about 90 percent of the time. Sometimes we’ll have a vendor
contract where they may do the installation and 500 hours of training or something. But we
don’t rely heavily on vendors. In the technology world, the vendor can be on the top of the
heap at the beginning of the year, and out of business at the end of the year. We want to fig-
ure out how we can institutionalize technology as a major part of the way we do business.
How do you accomplish that? Who needs to be involved in the process?
You need to start with the end users, and they need to be educated about what the district- level folks need to make those decisions. You’ve got to get buy-in at each of those points along the way.
You always go through that part where you’re running the old system and you have a
new system ready to go. Ten percent of the people are ready to dump the old system and go
to the new system. You need to find a way to make it politically palatable to say, by a certain
date, “We can’t afford to run two systems. We don’t have the people for it. How can we help
you get to where you need to be?” It really does take a driver somewhere in the organization
who understands that the discomfort quickly gives way to comfort if you do it well.
Chapter 4: Strategies for Success
41
Businesses don’t keep data that’s useless, that doesn’t inform them of anything, yet in education we have data that just runs all over us. We have to target it and organize it in such a way that it informs us so that we can make better decisions.
— David Benson, superintendent,
Blue Valley (Kan.) School District
How do you get the teachers and principals to buy into data-driven decision-making?
You have to show how it adds value. And at some point, I just had to say, “I understand your
problem, but we’re going to do this. Get to it.” Part of leadership is that, at some point, you
have to be able to see where it’s going and have the courage to stay with it until it actually
works in all of your schools. A lot of people are not willing to pay the political price. You have
to be willing.
Unraveling Roles and Responsibilities The culture of a school district will not smile favorably on data-driven decision-making for long
if the members of that culture continually step on each others’ toes or fumble the ball when it’s
their turn to run with the data. That is why the superintendent must draw the game plan in
clear, bold strokes — to help board members, district staff, principals and teachers understand
their important roles in the data system.
“It all has to work in coordination,” says
Douglas Otto, superintendent of the Plano (Texas)
Independent School District. “Everybody has to
have the same vision of what we’re doing and
why. Everybody has to understand what kinds of
data are important to collect and how they’re
used. From the school board and superintendent
on down, there has to be that common sense
of purpose.”
The 47,000-student district’s data efforts appear to be paying off. Plano educators monitor
student performance on the Iowa Tests of Basic Skills, the Cognitive Abilities Test, the Texas Assess-
ment of Basic Skills and the Scholastic Aptitude Test. The district also collects and reports data in
areas such as staffing, spending and enrollment patterns to inform decision-making continually.
To keep such a data system humming along smoothly, Otto says, team players must under-
stand their special roles and responsibilities:
• Superintendent and School Board. They share leadership of the district's data
efforts but play significantly different roles. In Plano, the division of labor has been
clear: The board adopts policies to govern the system; the superintendent manages
daily operations. “The board made sure we had the appropriate policies in place to
make sure we were going to collect data, analyze it and use it to improve instruc-
tion,” says Otto. “Board leadership is extremely necessary.”
To make effective policy decisions, the board must understand data. For exam-
ple, the board needs to know that some data trends are the result of a “common
cause” (a regular feature of the school system, such as normal student attendance
patterns) and some are the result of a “special cause” (an unusual circumstance, such
Using Data to Improve Schools: What’s Working
42
There are no secrets to success. It is the result of preparation, hard work and learning from failure.
— Colin Powell,
U.S. Secretary of State
as a flu bug that causes many students to miss school on a big test day). Otherwise, the
district might spend an inappropriate amount of time trying to solve a problem that was
a fluke.
The superintendent’s role, then, is to empower central office and school staff to use
data as effectively as possible to meet the board’s mandates. Max Riley, superintendent of
the Lawrence Township (N.J.) Public Schools, says the superintendent should send staff a
clear message: “This is not a ‘gotcha.’ I’m going to give you every tool I can think of to
help you succeed.”
• District Staff. Central office and other
staff at the district level must be pre-
pared to discard longstanding ways of
doing business if data dictate that a
new approach is needed. In 1996, for
example, Plano brought in consultants
to conduct a curriculum audit to com-
pare district practices with established
standards. Based on the audit, Plano
revamped its curriculum management
team and expanded its research and
assessment team to help “make wise decisions about what data we’re going to collect,
who’s going to collect it, who’s going to analyze it, who’s going to be trained, and how it
gets put back into the curriculum and strategies used in the classroom,”says Otto.
• School Site Councils. In many districts, school-based management teams — often called
councils or committees — are taking up data along with the other tools they use to help
make decisions for their campuses. “We’d had site-based improvement committees in
place for many years,” Otto says, recalling the changes that followed Plano’s curriculum
audit. “All of a sudden, one of their chief roles was to monitor areas of improvement in
school, make recommendations, work with staff, and use the same data that principals
and teachers were getting so everyone was on the same wavelength.”
• Principals and Teachers. Though principals bear ultimate responsibility for what hap-
pens in their schools, they must work with teachers to use data in ways that best serve
their unique student populations and school needs. In Plano, principals receive assis-
tance — and are held accountable — for this work. “Principals receive data training. It’s
not voluntary; it’s mandatory. They have to know how to use data on their school-based
improvement committees. It’s part of their performance appraisal,” says Otto. “Principals
also need to play a monitoring role to make sure that teachers are implementing the cur-
riculum, using effective strategies and improving student achievement.”
Chapter 4: Strategies for Success
43
Data-driven decision-making is not someplace we’ve arrived; it’s still a journey and always will be. But it’s a journey where people understand how you buy the ticket and get on the train. And if you’re not on the train, you need to find another place to be, because this is the way we have to operate.
— Yvonne Katz, superintendent,
Beaverton (Ore.) School District
Yvonne Katz, superintendent of the Beaverton (Ore.) School District, says data must be dis-
tributed across grade levels, so elementary school teachers know how to prepare their students
for middle school level work, and high school principals know how well their incoming fresh-
men will be prepared to perform. "Everyone can see what the data are showing" in Beaverton,
says Katz. “As you look at groups coming through, you can plan what your focus will need to be
in future years, when the kids hit your level.”
The Evolving State Role As school districts become more sophisticated in their use of data, they are starting to look
beyond their own borders to appreciate the helpful role that state departments of education can
play. This outlook is reversing years of mistrust and resentment, begun when states started col-
lecting data that district officials worried would be used against them. As local school systems
have become more confident and comfortable with data-driven decision-making, they have
looked to the states for data training and other forms of technical support.
Katz retains a unique perspective on the state role, based on her experiences as associate com-
missioner for general education of Texas in the early 1980s. “We had a tremendous training pro-
gram in Texas, bringing in principals and vice principals to be trained for a whole week in data
usage, then visiting campuses around the state,” she says.
New York provides a different kind of support for districts. State assessment scores not only
are distributed widely, but are broken down to allow comparisons among similar school districts,
such as those that serve high percentages of students eligible for federal free or reduced-price
lunches.
“Some fairly helpful comparisons can be made for follow-up for the purposes of instructional
development for classroom teachers and principals,” says Kevin McGuire, who served as a super-
intendent for 18 years before becoming director of the New York State Center for School Leadership.
“There are extraordinarily talented educators in every corner of the state. By linking people
together, we think we can help everyone grow, the adults as well as the children,” McGuire says.
“For example, if people are having difficulty with students who are remedial in nature, they can
talk to a similar school district that seems to be meeting the needs of similar students in a more
effective way.”
Using Data to Improve Schools: What’s Working
44
Chapter 4: Strategies for Success
45
The superintendent generally:
• Translates the board’s vision for the
school district into measurable goals
based on data.
• Works with district faculty, staff,
parents and other community stake-
holders to craft plans for meeting
goals by certain dates.
• Collects data to show clear, steady
progress.
• Celebrates successes, evaluates
shortcomings and revises plans for
improvement based on data, along
with the board.
The school board generally:
• Establishes a vision for the school
district based on data showing what
has been achieved so far and what
progress is necessary.
• Spells out — for the superintendent
and other employees and stake-
holders — how district performance
will be evaluated.
• Reviews relevant data to evaluate
district progress toward identified
goals.
• Revises goals and plans for
improvement based on data.
Broad participation in improvement efforts serves to:
• Promote a high level of support for those efforts.
• Generate sound solutions by expanding the discussion.
• Motivate participants and their associates.
• Increase the likelihood that the effort will lead to constructive action.
• Prepare participants for their role in implementing improvements.
• Increase ownership of and commitment to specific strategies.
• Empower important stakeholder groups.
• Foster lasting, rather than temporary, change.
Who Does What? Data-driven decision-making, especially in the early stages, demands that district
leaders point the way. Superintendents and school boards both must play important
but distinct roles.
Source: At Your Fingertips: Using Everyday Data to Improve Schools, 1998
Training the Players Superintendents know that data-driven decision-making requires new knowledge and skills. In a
2000 AASA membership survey, superintendents overwhelmingly called for training on using
data. They also knew their staffs needed a crash course in collecting and analyzing data, using
data-processing technology, and operating the decision support system.
“I can sit up here in the superintendent’s office and look at the data and issue directives, but
then people don’t understand what I’m doing or why I’m doing it,” says Glenn. “You’ve got to
empower people to use data. They’ll help you identify problems — and often, through team-
work and looking at other ways of doing things, they will come up with solutions.”
To help schools use Leander’s new Student Profile System, which generates data on student
progress throughout the year, the district provided teachers with lots of training, much of which
directly involved them in helping develop the district’s student profile test. A specially trained
“data group” gathered data and reported it back to schools in user-friendly formats. Principals
received training in helping teachers use data individually, in grade-level teams and as a school.
“We did a lot of modeling with people,” says Glenn. “I met with principals three or four times a
year on their campuses to talk about their data: What’s going on? Where are the peaks? Where
are the valleys? What are they doing about the valleys?”
But Glenn does not advocate trying to make a data technician of every school employee.
More important is training that emphasizes the conceptual underpinnings of data-driven
decision-making — the core principles of how educators can use data to improve education.
“I can hire all kinds of people to come in and help us do the grunt work of collecting the
data, putting it into charts and those kinds of things,” says Glenn. “But as many people as possi-
ble in the system have to understand the philosophical basis of what you’re trying to accom-
plish. Sometimes that’s where we fail in education — in not helping people understand what
we’re trying to accomplish with all this paperwork we’re asking them to do.”
Implementation Models Depending on what type of training is needed, superintendents might want to explore
various ways of providing training. Should an outside vendor be hired to train staff, or should
this be handled internally? Should one person spearhead the effort? Or would it be best to
build a team?
Only you can answer such questions for your district. But successful superintendents and
other education leaders have learned the following lessons:
• Not all employees need to receive training, but employees at all levels need to learn
about data-driven decision-making. This helps build buy-in at all levels and ensures
that the system works the way it should.
Using Data to Improve Schools: What’s Working
46
Chapter 4: Strategies for Success
47
• Perhaps because data can be intimidating at first, districts often rely too heavily on
outside consultants. The result can be generic training that is poorly matched to the
specific needs or problems of a district. If a data effort is to be sustained over the long
term, the district must cultivate in-house trainers who can help colleagues use the
local data system.
• Actual district data should be used in training only when appropriate. If training focus-
es on data analysis, then using local data can provide a compelling context for partici-
pants. But if training is intended to help participants master new data software, local
data can be a distraction.
When in doubt, remember: Data also can help tailor training to the unique needs of a dis-
trict. Consider the technology self-assessment used by teachers in Blue Valley. Teachers complete
a 43-item questionnaire to determine how they use technology and how they might use it
better. The district uses that data to design professional development in technology. “Because
we have used data, we have targeted inservice to specific, identified needs, rather than just using
broad-brush technology inservice,” says Superintendent David Benson.
The return on investment for data training is incalculable. “No matter whether you’re a huge
district or a very small district, if teachers are trained and have some knowledge of how to col-
lect and analyze data, they can do some remarkable things in the classroom — without needing
some decision from on high passed down telling them what to do,” says Otto.
Creating a System for Continuous Improvement Increasingly, leading school districts are using data-driven decision-making to ensure continuous
improvement. They do so by moving beyond the initial work of assessing their own schools’
performance, comparing those schools against each other and monitoring trends over time.
These districts look beyond their borders — to benchmark their schools’ performance against
that of other top-performing schools across the country.
Benchmarking is more than comparing test
scores. Done well, benchmarking helps dis-
tricts learn what it takes to be effective, how
well they are using key strategies, where
improvements are possible or necessary, and
how they can learn from the best practices of
other successful districts.
A recognized leader in district benchmarking practice is the Western States Benchmarking
Consortium. Founded in 1997, when a group of education leaders from high-achieving districts
gathered to devise strategies for improvement, the consortium continually has brought together
superintendents and others from member districts to develop research-based benchmarks for
district success.
Meaningful school reform is probably more about using data effectively than about any other tool you can use.
— Douglas Otto, superintendent,
Plano (Texas) Independent School District
Using Data to Improve Schools: What’s Working
48
Looking for Value Added Plainville (Conn.) School District
In the three years he headed the Plainville (Conn.) School District, then-Superintendent Max Riley raised student scores on Connecticut state exams to record levels — but he didn’t do it alone. Riley recruited a ”stat jock” from Central Connecticut State University to help principals and teachers use data for improvement.
James Conway, a psychology researcher and professor, was contracted by the district to “act as our coach for thinking about data,” says Riley. Conway combed through reams of available data and analyzed performance trends over time and differences among schools. Then the researcher made himself available for confidential consulta- tions with principals about their schools’ data.
“I wanted principals to have access to some- one with whom they could have conversations without necessarily having to go through the superintendent,” Riley says. “It was understood from square one that these were confidential conversations and that Jim was just there to help principals in whatever they wanted to do, as long as it was focused on data and student learning.”
Providing that sense of security made a dif- ference. Principals latched onto Conway. They called with questions. They explored the data. They started seeing their schools in new ways.
For example, most people in Plainville tradi- tionally had focused on raw test scores — and ignored relevant data about the student popula- tion — in evaluating the success of the district’s three elementary schools. The school that served mainly children of white-collar parents routinely earned top marks. Two other schools, both of which served children of comparatively low- income families, fared less well.
But Conway toppled those perceptions with a value-added analysis. Essentially, Conway col- lected data on how well students performed when they entered each school, then examined how much progress that school enabled students to make. According to this analysis, the two schools with the lowest test scores added value to children’s education at double the rate of the school with the highest test scores.
“Our highest-performing school was under- performing in terms of moving kids along,” says Riley. “It sent shock waves through the organiza- tion. It turned everything upside-down in terms of people’s mythological understanding about who’s doing a good job.” The value-added analysis also helped the principals at all three schools better understand where they were pro- viding effective instruction and where they need- ed to work harder, he adds.
Now superintendent of the Lawrence Township (N.J.) Public Schools, Riley is setting up a similar system. Working with a researcher from a nearby university, Riley encourages principals to sift through data to see how they might increase their value added for students.
Working with college and university researchers can be particularly advantageous for relatively small districts like Plainville and Lawrence because districts can pay relatively little and get data services tailor-made to meet their needs. “Of course, the board loves it, because they feel like they’re making focused, informed decisions about how to spend the money,” says Riley. “Data also give principals the ammunition to come in at budget time and say, ‘Look, it’s your own darn data system showing where I need help. Now, I need help.’”
Field-testing those benchmarks during the 2000–01 school year, the consortium’s seven mem-
ber districts conducted self-assessments and shared results and insights. Now, member
districts are using the benchmarks in everything they do, from setting goals and strategic
planning to designing professional development and establishing accountability systems.
“In Plano, we are developing strategies for our school-based improvement committees to use
the benchmarks to drive improvements in buildings,” says Otto, whose district has been at the
forefront of the consortium’s work. “One of our weaknesses is that we’re still not at the point
where every teacher is using the data to improve instruction. We know that from our own obser-
vation and from talking with groups of principals and curriculum coordinators. That tells us that
more training is needed.”
How does it work? Based on recent research showing links between specific organizational
policies and student achievement increases, the consortium has adopted four strategic areas of
focus for benchmarking: student learning, capacity development, learning community develop-
ment and data-driven decision-making.
Under each benchmark, a district’s performance shows it to be at one of four stages of devel-
opment: emergent, islands, integrated or exemplary. For example, the consortium has developed
a data-driven decision-making benchmark for “relating investments, outcomes and improve-
ment strategies.” A district with “emergent” performance in this area collects data that are not
maintained consistently over time and are not linked well enough to key outcomes to inform
critical decisions. By contrast, the data needs of an “exemplary” district are clearly articulated
and met in all stages of planning and implementation.
For each stage of development, the benchmarks include “behavioral indicators” (beliefs and
actions evident in the system) and “support indicators” (things that support desired behaviors,
such as professional development, policies and resources). These indicators not only let districts
know how well they are doing, but also give them sound, research-proven ideas about how they
can do better.
The process of developing the consortium’s benchmarks has been more valuable, in some
ways, than the benchmarks themselves. “It really generates a lot of professional opportunities,”
says Otto. “Having superintendents sitting around the table talking about these benchmarks —
what we’re doing and the strategies we’re employing — really raises the level of conversation
with regard to school improvement.”
“The consortium holds great promise for us in terms of continuous improvement based
on objective standards,” agrees Benson, whose Blue Valley School District also participates in
benchmarking. “You want to make continuous improvement in those areas where you can say
objectively: ‘This will make a difference for kids.’”
Chapter 4: Strategies for Success
49
Data-Driven Decision-Making: Using Information to Improve Instructional Practice
Using Data to Improve Schools: What’s Working
50
Benchmarks for School District Success The Western States Benchmarking Consortium’s benchmarks for school district performance allow a district’s
performance to be judged to be at the “emergent,” “islands,” “integrated” or “exemplary” stage of development.
The table below describes indicators at different levels of performance for how well educators use data to personalize
education or students and profile and monitor student performance.
Data for Personalization
The instructional approach is “one-size- fits-all,” with underper- forming and accelerated students “pulled out.” As a result, there is little or no integration among remedial, accelerated and ongoing instruction. The concept of personal- ized instruction for all students based upon individualized perform- ance data is foreign.
Data for Personalization
Some schools provide examples of new approaches to accelerat- ed or extended learning that are driven directly by examination of data.
Data for Personalization
Evidence shows that the programmatic use of data in most areas such as accelerated and extended learning pro- grams flows directly from examinations of performance data.
Data for Personalization
Teachers routinely use performance data to truly personalize learning for all students.
Profiling and Monitoring Student Performance
The district has not organized a formal student performance database. As a result, schools are left to their own devices to track performance over time. Teachers have great difficulty in determining the peform-ance level of students who move from school to school. Traditional grade-based report cards provide parents with information on student progress.
Profiling and Monitoring Student Performance
The student perform- ance database is still nonexistent, but more people are demanding that it be designed and implemented to improve curriculum and instruc- tion. Schools are strug- gling with the inadequa- cy of the performance information base regard- ing timeliness, validity, reliability and compre- hensiveness. The formal report card is still grade- based and remains the primary mode of com- munication with parents regarding student progress.
Profiling and Monitoring Student Performance
The district provides a profile of student performance for each school. All schools use these profiles to set annual and longer range improvement targets. Student progress reporting is based on a continuum of progress in achieving well-defined content and perform- ance standards. More schools are providing parents with specific suggestions regarding techniques for family reinforcement of these standards.
Profiling and Monitoring Student Performance
Data are aggregated/ disaggregated to class- room/school/district levels to determine necessary improvements in instruc- tional practice. Heavy evidence of reliance on student performance data from multiple sources exists. The system main- tains a performance profile for every student, integrat- ing all information about the student’s performance over time. Parents receive frequent progress reports of growth on the standards continuum, accompanied by suggestions for home reinforcement.
Emergent Islands Integrated Exemplary
Source: Western States Benchmarking Consortium
The following school district performance benchmarks developed by the Western States
Benchmarking Consortium describe indicators of district success in four major areas of
educational performance.
Student Learning
• Ensuring learning for all students
• Integrating standards
• Incorporating innovative practice
• Integrating technology
• Developing a coherent curriculum
Capacity Development
• Expanding organizational effectiveness
• Adopting a curriculum management system
• Promoting innovation
• Improving professional/organizational learning
Learning Community Development
• Understanding and using state academic results
• Providing community-based learning opportunities
• Building community partnerships
• Building community development
Data-Driven Decision-Making
• Using a variety of data effectively
• Using information to improve instructional practice
• Using data to affect student performance
• Relating investments, outcomes and improvement strategies
Chapter 4: Strategies for Success
51
Danger Zones Superintendents charging into the unfamiliar territory of data-driven decision-making
sometimes see the edge of the cliff only just before they reach it. Here are some lessons learned
by district leaders who have been there:
• Don’t rush headlong down the path to collect data and enforce accountability before
bringing on board all the people needed to make the new system work. Talk. Listen.
Build consensus about what is most important to do. Otherwise, you might find your-
self out in front — with no one behind you.
• Don’t fish blindly for meaning in data. If you do not know what questions need to
be asked — before you even collect the data — you are not likely to find the answers
you need.
• Don’t bombard people with data for its own sake. Don’t report it just because it is
available. Stick with data points that can drive decisions for improvement.
• Don’t lose track of what is most important — such as key indicators of student learn-
ing progress. If you place equal priority on many data points, some of which might
have little to do with your district’s core mission, then your team will not know where
to focus its energy.
• Don’t forget to draw the "big picture" for the people collecting, interpreting and using
data. They need to understand why they are being asked to do this and what it all
means. Explain it again and again and again.
Summary Key points in Chapter 4 include:
• Successful data-driven decision-making requires a shift in the culture of a school
district that encourages the use and analysis of data without fear of reprisal.
• The superintendent, school board members, principals and teachers must be clear on
the purposes and goals of data collection and analysis.
• To help educators use data effectively, it is important to provide training and
assistance, because data-driven decision-making requires new knowledge and skills.
• School districts using data-driven decision-making ensure continuous improvement by
benchmarking their schools’ performance against that of other top-performing schools
across the country; benchmarking helps districts learn what it takes to be effective.
Using Data to Improve Schools: What’s Working
52
53
Source: Reporting Results: What the Public Wants to Know, 1999
Appendix A: Accountability Measures
Parents, taxpayers and educators were asked to rate the following items in importance for
school accountability.
Category Parents Taxpayers Educators
School safety 9.6 9.4 9.3
Teacher qualifications 9.3 9.2 8.3
Class size 8.9 7.9 8.8
Graduation rates 8.7 8.2 8.3
Dropout rates 8.3 8.1 7.4
Statewide test scores 8.2 8.0 7.1
Parental satisfaction survey data 8.1 8.0 7.0
SAT/ACT scores 8.1 7.9 6.9
Percent of students promoted to next grade 8.0 8.1 7.0
Course offerings 7.8 7.9 7.3
Attendance rates 7.8 8.0 7.6
Per-pupil spending 7.6 7.6 8.0
Student satisfaction survey data 7.5 7.0 7.1
Teacher salaries 7.3 7.8 7.6
Hours of homework per week 7.2 7.3 6.3
Number of students 7.2 7.2 6.7
Percent of students who go on to a four-year college 7.0 6.9 6.8
Percent of students with an "A" or a "B" average 7.0 6.5 5.8
Number of students per computer 6.9 6.4 6.1
Percent of parents who attend parent-teacher conferences 6.4 6.6 6.3
Demographics of students 4.5 4.6 5.0
Using Data to Improve Schools: What’s Working
54
Appendix B: Working with the Media
Media Dos and Don’ts on Data
• Do make sure your district has a comprehensive communications plan that addresses
the release of potentially controversial data such as test scores, dropout rates, teacher
quality ratings, school safety reports and other indicators.
• Don’t wait until the day test scores are released to deepen the media’s understanding
about what the tests measure, how the district will use the test scores and what the
results tell parents and community members. This should be an ongoing process. Be
proactive. Keep the media informed. Make an occasional courtesy call even when you
don’t have to.
• Do make sure staff is apprised of test results and other sensitive data before the media,
so everyone is focused on the same messages. For example, if the state releases test
scores one day before the scores are released publicly, hold meetings with teachers,
principals and other district staff to inform them of the results.
• Don’t hide data from reporters. It’s public information.
• Do talk to the reporter first when inaccuracies surface in a story, such as factually incor-
rect data or scores. Make sure you have documentation to back up your claim. If prob-
lems continue, contact the reporter’s editor.
• Don’t neglect to share what is working in the district. Suggest feature stories that show
trends in school improvement backed by data.
55
Interview Techniques: How to Handle Tough Media Interviews on Data
Needling Reporter: “Oh, come on now, you don’t believe trend data show improvements
in teacher quality do you?”
Superintendent: “Yes, I do, Beth, and… .” Go on to your key message.
Twisted Facts Reporter: “So your test scores are down again.” (When, in fact, the test has
changed or a new population of students is tested.)
Superintendent: “Let’s be clear about what the data tell us… .”
False Assumption Reporter: “So I assume you are going to spend taxpayer money to purchase a
new test since this one shows bad results… .”
Superintendent: “Well, Mike, I wouldn’t agree with your conclusion.” Steer the interview back on message.
Baiting
Reporter: “So why does this community keep attacking your initiatives?”
Superintendent: “You’ll have to talk directly to those who raise concerns. What I can tell you is that our district is dedicated to… .” (Discuss public engagement efforts to gather and respond to community input.)
Pregnant Pause
Reporter: The reporter remains silent after you answer a question.
Superintendent: Do not rush to fill in the silence. Wait for the reporter to follow-up or ask, “Does that answer your question?”
Loaded Question Reporter: “You don’t deny that the district knew all along about sagging test
scores and refused to do anything about it.”
Superintendent: “Here’s what we have done to improve low-performing schools… . Here’s what has worked and where we need to improve… .”
Adapted from a Michigan School Boards Association media slide presentation.
Using Data to Improve Schools: What’s Working
56
Appendix C: Resources
Reports “Building an Automated Student Record System.” National Forum on Education Statistics, 2000.
This booklet is designed to lead education organization decision-makers through the process of
making the best and most cost-effective decisions about information management systems devoted
to individual student records. Available online at http://nces.ed.gov/pubs2000/building/
“Evaluation of California’s Standards-Based Accountability System.” WestEd, November 1999.
This evaluation examines the status and impact of local accountability systems, content stan-
dards, assessment measures, use of data, consequences and incentives, and challenges and assis-
tance in school districts across the state. The document reports the findings of the study and its
implications for policy. Available online at http://web.wested.org/online_pubs/accountability/
“Reporting Results: What the Public Wants to Know.” A companion report to Education Week’s
Quality Counts ’99.
A report of what parents, taxpayers and educators think they need to know about schools in
order to hold them accountable. Available online at
http://www.edweek.org/sreports/qc99/opinion/edweekresults.pdf
“Student Data Handbook for Elementary, Secondary, and Early Childhood Education.” National
Center for Education Statistics, 2000.
This handbook offers guidelines, developed by the U.S. Department of Education’s National
Center for Education Statistics, for current and consistent terms, definitions and classification codes
to maintain, collect, report and exchange comparable information about students. Available online
at http://nces.ed.gov/
“Using Data for School Improvement: Report on the Second Practitioners’ Conference for
Annenberg Challenge Sites.” Annenberg Institute for School Reform at Brown University Tools for
Accountability Project, 1998.
This report, developed after the second practitioners conference for Annenberg Challenge Sites,
provides valuable examples of data collection, ways of thinking about accountability and resources.
Real-life examples of schools implementing data-collection efforts to improve student achievement
appear throughout the report. Available online at
http://www.annenberginstitute.org/images/using_data4.pdf
57
“Vermont Department of Education. Equity and Excellence Action Planning Guide “
http://www.state.vt.us/educ/actplan/apcover.htm
This resource provides a look at each step of the action planning process for educators.
Articles “Data-Driven Decisions.” Articles by Theodore B. Creighton, Raymond Yeagley, Philip A. Streifer,
Francis Barnes, Marilyn Miller and George A. Goens. The School Administrator, April 2001.
This issue of The School Administrator includes a number of articles on data-driven decision-mak-
ing. Available online at http://www.aasa.org
“On the Job: Data Analysts Focus School Improvement Efforts.” Joellen Killian and
G. Thomas Bellamy. Journal of Staff Development, Winter 2000.
This article presents one school district’s model for building-level disaggregation and data analy-
sis. Available online at http://www.nsdc.org/library/jsd/killion211.html
“Smart Moves: Achieving Your Vision Depends on Follow-Through.” Joan Richardson. Tools for
Schools, National Staff Development Council, August/September 2000.
This article discusses how school districts move beyond publishing and publicizing a school’s
newly developed vision statement and begin the real work of using the vision as a guide and a
measure for school improvement. Available online at
http://www.nsdc.org/library/tools/tools9-00rich.html
“This Goes on Your Permanent Record.” Stewart Deck. CIO Magazine, Nov. 1, 2000.
This article examines how various school districts are building and implementing data
warehouses to support districtwide, building-and classroom-level decision-making.
Available online at http://www.cio.com/archive/110100/permanent.html
“Using Data to Improve Instruction.” Articles by Nancy Protheroe; Jay Feldman and Rosann
Tung; Yi Du and Larry Fuglesten; Thomas Glass; Brett Geis, Terrell Donicht and Steven Smith; and
Thomas Fowler-Finn. ERS Spectrum, Summer 2001.
This issue of the ERS Spectrum includes several articles about the use of data in school
improvement efforts.
Using Data to Improve Schools: What’s Working
58
Books Accountability Dialogues: School Communities Creating Demand from Within.
Kate Jamentz. WestEd, 2001.
This book explains some of the common misperceptions about school accountability and
provides a strong rationale for including accountability dialogues in any accountability system.
Examples from schools that use accountability dialogues provide a real sense of what can
happen when responsibility for school improvement is shared among all the stakeholders in
a school community.
At Your Fingertips — Using Everyday Data to Improve Schools. Karen Levesque, Denise Bradby,
Kristi Rossi and Peter Teitelbaum. MPR Associates Inc. and American Association of School
Administrators and National Center for Research in Vocational Education, 1998.
This handbook teaches educators new and productive ways of using data. Through step-by- step instruction, this book focuses on using available data to improve teaching and learning. Information available online at http://www.mprinc.com
Effectively Using Data to Drill Down and Make Better Educational Decisions.
Philip A. Streifer. Scarecrow Press, Inc., 2002.
This book is a guide on how to use data in decision-making, with a strong emphasis on
asking the right questions in a cyclical process designed to narrow the focus of inquiry.
Data Analysis for Comprehensive Schoolwide Improvement. Victoria L. Bernhardt. Eye on Education, Larchmont, N.Y., 1998.
This book teaches the layperson how to gather, interpret and use data to improve schools. Educators are given practical tools so they can make better data-based decisions.
Getting Excited About Data: How to Combine People, Passion and Proof. Edie L. Holcomb. Corwin Press, Thousand Oaks, Calif., 1999.
This book outlines a process for showing how well a school or district meets its primary goal: sustained student learning. The author offers tips on finding answers to questions about data, such as: What data do we need and how do we collect it?
Improving School Board Decision-Making: The Data Connection. National School Boards Foundation, 2001
This guidebook helps school board members understand what data are, ask for the appropri- ate data to inform decision-making, work with the superintendent to understand what data reveal, use data to support decisions on policy and budgets and inform the community.
59
Is There a Public for Public Schools? David F. Mathews. Kettering Foundation Press, 1997.
This book provides school leaders with insights for engaging the public when undertaking
school improvement efforts.
Thinking About Tests and Testing: A Short Primer in Assessment Literacy. Gerald Bracey. American Youth Policy Forum, Washington, D.C., 2000.
This book is a nonpartisan, objective discussion that helps readers understand the arguments now raging about "high-stakes tests" and their consequences. The book is simple and straightfor- ward. Available online at http://www.aypf.org/BraceyRep.pdf
The School Portfolio: A Comprehensive Framework for School Improvement, Second Edition. Victoria L. Bernhardt. Education for the Future Initiative, 1999.
This book shows how to develop a school portfolio tailored to a particular school and vision. It explains that school portfolio is the most effective way to ensure a school’s success at systemic reform. Extensively tested, it is a nonthreatening self-assessment tool that exhibits a school’s goals, achievements and vision for improvement.
Schools and Data: The Educator’s Guide to Using Data to Improve Decision Making. Theodore B. Creighton. Corwin Press, Thousand Oaks, Calif., 2000.
This book focuses on the relevance of statistics in the day-to-day lives of principals and teach- ers. This is an essential resource for any educator who wants to break through the statistical confusion to improve skills in problem analysis, program evaluation, data-driven decision-making and report preparation.
Using Data / Getting Results: A Practical Guide for School Improvement in Mathematics and Science. Nancy Love. Christopher-Gordon Publishers, 2002.
This book is a guide for inquiry into improving mathematics and science teaching and learning. It examines ways to engage school communities and produce powerful learning.
Internet Resources
Center for Accountability Solutions, American Association of School Administrators. http://www.aasa.org/data
AASA has created this website to help school leaders gather, use and report meaningful data on student, school and district performance. It includes a comprehensive listing of data resources that is updated regularly.
Using Data to Improve Schools: What’s Working
60
Appendix D: Glossary
A Accountability — A process by which educators are held responsible for performance or
outcomes.
Application service provider (ASP) — A third-party entity (generally for-profit) that manages
and distributes software-based services and solutions to customers across a wide area network
from one or more off-site locations.
B Benchmark — A standard by which something can be measured or judged. Benchmarks can
be used to monitor progress toward meeting goals.
C Cohort – A group of students or others of the same age moving through a system together
(e.g., all of the students who enter kindergarten the same year are part of the same cohort).
Cohorts typically are used to measure progress over time and compare that progress with
other cohorts.
Common cause — A regular feature of the school system (e.g., normal student attendance
patterns).
Comparative analysis report — A report that contrasts two or more districts, students or
other groups that have similar characteristics and can be compared to each other.
Criterion-referenced test (CRT) — A test that measures an individual’s performance against a
well-specified set of standards (distinguished from tests that compare students in relation to
the performance of other students, known as norm-referenced tests).
Cut score — A specified point on a score scale, such that scores at or above that point are
interpreted or acted on differently from scores below that point. In standards-based assess-
ments, cut scores typically delineate passing from failing, proficiency from mastery, and so on.
D Data — Factual information (such as measurements or statistics) used as a basis for reasoning,
discussion or calculation. Data can be qualitative or quantitative. Good data must be both
reliable and valid.
61
Data mining — The analysis of data for relationships that previously have not been discovered.
Data warehouse — A central repository for all or significant parts of the data that a system
collects. A data warehouse typically is a database or collection of databases existing in virtual,
rather than physical, space.
Decision support system (DSS) — A computer program application that analyzes and presents
data so users can make decisions more easily.
Demographic data — Data that focus on the gender, socioeconomic background, race and
ethnicity of students in a school or district.
Disaggregated data — Data broken down by specific student subgroups, such as current
grade, race, previous achievements, gender, socioeconomic status, and so on.
Drill down process — A method of disaggregating data that begins with a general question,
followed by increasingly specific questions that focus on smaller subsets of data.
F Formative assessment — Assessment in which learning is measured at several points during a
teaching/learning phase, with the primary intention of obtaining information to guide further
teaching or learning steps. Formative assessments take a variety of forms, including question-
ing, comment on a presentation or interviewing.
I International test — A test administered uniformly in a number of countries that compares
student performance country by country (e.g., Third International Mathematics and Science
Study).
L Longitudinal data — Data measured consistently from year to year to track progress, growth
and change over time. True longitudinal studies eliminate any students who were not present
and tested in each of the years of the study.
M Mission — A statement that defines what an organization is created to do and reflects its core
values and beliefs to guide it toward its goals.
Using Data to Improve Schools: What’s Working
62
N Normal curve equivalent (NCE) — Normalized standard scores on an equal interval scale
from 1 to 99, with a mean of 50 and a standard deviation of about 21.
Norm-referenced test (NRT) — A test that compares individual student performance with a
larger group, usually a national sample representing a diverse cross-section of students. (e.g.,
Iowa Tests of Basic Skills, Stanford Achievement Test). NRT results typically are measured in
percentile ranks. (Norm-referenced tests should be contrasted with criterion-referenced tests,
which measure performance compared to a standard or benchmark.)
P Percentile score — A score that designates what percent of a norm-referenced group earned
raw scores lower than a particular score. Percentiles often are divided into quartiles (groupings
of 25 percent).
Perception data — Data that inform educators about parent, student and staff perceptions
about the learning environment, which could also reveal areas in need of improvement.
Performance test — A test that requires students to demonstrate their abilities by providing
examples of their work (e.g., portfolios, presentations, experiments).
Portfolio assessments — Assessments of a collection of a student’s educational work that
shows growth, self-reflection and achievement over time.
Q Qualitative data — Data based on information gathered from one-on-one interviews, focus
groups or general observations over time (as opposed to quantitative data).
Quantifiable proof — Proof that can be precisely measured.
Quantitative data — Data based on "hard numbers" such as enrollment figures, dropout rates
and test scores (as opposed to qualitative data).
R Reliability — The consistency of test scores over different test administrations, multiple raters
or different test questions. Reliability answers the question "how likely is it that a student
would obtain the same score if they took the same test a second time?"
63
S Special cause — An unusual circumstance in a school system (e.g., fluctuations in attendance
pattern due to a flu bug or a snow day).
Stakeholders — The students, parents, taxpayers, community members, business leaders,
educators, board members and all others who have a share or interest in the school or district.
Standard — Something established for use as a rule or basis of comparison in measuring or
judging capacity, quantity, content, extent, value, quality, and so on.
Stanine — A standard score of nine units in which 1, 2 or 3 indicates below-average
performance; 4, 5 or 6 indicates average performance; and 7, 8, or 9 indicates above-
average performance. Stanines are still used in some standardized tests.
Summative assessment — An assessment at the end of a period of education or training,
which sums up how a student has performed.
V Validity — The degree to which tests measure what they purport to measure.
Value-added — A measurement of the learning that a student achieves through participation
in a program.
Vision — A future-focused statement about what an organization wants to be, where it wants
to go and what kind of system it wants to create.