An Evaluation of Math Assessment Policy Process in a Southwestern School District.

Section 1: The Problem

Introduction

The purpose of this study is to evaluate the policy process that led to a math

assessment policy in a local school district. This evaluation of the policy process marks

the first instance in this district of the collection of empirical data on the implementation

of a mathematics assessment policy and its effect on teachers and administrators charged

with carrying out its procedures. The setting was a small school district in the

southwestern United States. The absence of empirical data on the implementation of the

math policy contributed to gaps in practices related to K-12 math instruction prior to the

onset of this study, as the data analysis later shows. The existence of the problem was

verified by district leadership. In a letter, the superintendent stated, “The math exit

assessment policy has not been evaluated since its creation” (personal communication,

July 26, 2012). The absence of any policy analysis data available to the public prior to

this study also supported the need for the study. A review of various district resources,

such as the website, school board minutes, and the district’s policy manual, had not

revealed evidence of any policy evaluation or analysis, which highlighted the importance

of this project study. Evidence of the problem was supported by the local district’s

research director, who confirmed that no evidence of a math assessment policy evaluation

existed within the district’s archives (personal communication, February 21, 2013). The

district superintendent also saw a need to evaluate the math assessment policy and stated

in a letter, “I am in agreement that there is a need for analyzing the district’s exit

assessment policy for math. The math policy has not been evaluated since its creation”

(personal communication, July 26, 2012).

The local school district studied was small, consisting of fewer than 10

elementary, middle, and high schools, and served nearly 1,200 students. There were

fewer than 100 teachers in the district when the study was conducted. The district had a

history of declining enrollment, with approximately 1% of the student population moving

out on an average annual basis (district school board minutes, 2014; Hudson, B., 2014b).

The populations that were primarily affected by the absence of empirical

evaluation data on the mathematics assessment policy were district-level administrators

and K-12 math teachers. The math assessment policy had an effect on teachers by

requiring them to use formative common assessments to inform their instruction and end-

of-year exit assessments to determine student readiness for the next grade level.

Administrators were involved in the math assessment policy because they were initially

required to use the policy as a means to improve teacher instruction (senior administrator,

personal communication, 2010). Later, as state teacher evaluation mandates came into

effect, these assessments were used as a portion of teacher evaluations. Analyzing the

results of the changes brought about by the mathematics assessment policy in reference to

teacher and administrator behaviors was important because of the potential influence

these behaviors had on professional collaboration and student achievement.

Students are also affected by the absence of policy evaluation data. According to

Marzano (2007), the level of achievement and learning students can attain is directly

influenced by how schools are run and how teachers manage their classrooms and

instruction. The former superintendent of curriculum and instruction at the study district

was interviewed by the State Department of Education in 2010. In this interview, the

superintendent of curriculum stated that the mathematics assessment policy was intended

to change the way the school was run and how teachers delivered instruction:

The assessments took place at different intervals throughout the year. Each team

developed the assessments based on any indication there was a need to determine

the effectiveness of the instruction based on student needs and strengths during

any grading period. We did not want specific dates, except each team had to

develop exit assessments given the last three weeks of a course. All of the results

were used to determine student growth and curriculum alignment and passed on to

next year’s teachers to help guide instructional planning.

Now that an evaluation of the mathematics assessment policy’s creation,

implementation, and outcomes has been conducted, the local district has information

about how students, teachers, and administrators were affected. An investigation of

whether or not the mathematics assessment policy goals were attained helped to uncover

the effects the policy had on students and their achievement, collaborative teacher

practices, the means by which teachers informed their instruction, the degree to which the

goals of the policy were met, and teacher and administrator perceptions of the policy.

In the spring semester of 2005, the superintendent provided an implementation

plan to all mathematics teachers for the mathematics assessment policy (senior

administrator, personal communication, 2005). The local policy required mathematics

teachers to create common assessments at all grade levels. Mathematics teachers were

required to document that each question on the assessment met state standards. Teachers

were required to administer these assessments a minimum of three times per year in K-8

and three times per semester in high school. Mathematics teachers were required to

record the mathematics assessment results of each student in a spreadsheet. Each item

within the spreadsheet was tagged with a state standard code. Finally, the spreadsheets

compiled by mathematics teachers were submitted to the assistant superintendent in

charge of curriculum.

From the standpoint of larger educational populations and settings, Webb (1995)

warned of the harm that local assessment policies can have on students when the policies

themselves are not regularly evaluated for the quality indicators of purpose, methods of

standard setting, level setting, determining final standards, effect on what is taught

because of the assessment, and school management versus central control. Milton Town

School District (2011) in Vermont has an assessment policy that describes the need for

collaboration among teachers, school board members, administrators, community

members, and students. Collaboration among teachers, school board members,

administrators, community members, and students could be a possible avenue not only

for carrying out Milton’s assessment policy, but also for evaluation of the policy itself.

Rationale

According to the district’s mission statement, “[the district] is dedicated to

providing a superior learning environment for students of all ages and providing a

springboard to success in higher education and the world at large.” The local district is

often seeking solutions that will improve student achievement, classroom instruction, and

school culture. The creation of the local math assessment policy is one such solution that

has now been in place for several years in different forms, as will be addressed later.

Before the implementation of the math assessment policy, there had been no prior

research on math assessments in the local district or in districts with similar

demographics. Furthermore, the math assessment policy might have continued

indefinitely without any meaningful thought given to its continued effect on students,

teachers, and schools had this study not been conducted. An indication of the previous

lack of policy evaluation was the fact that searches of all district documents including the

district website, school board minutes, policy handbook, and personnel handbook

demonstrated no record of analysis of the math assessment policy.

The local district had received very little media coverage within the county or

state, with the exception of the local print newspaper and a separate electronic

newspaper. The district has strong ties with the State Department of Education and

federal education initiatives, with school employees serving on both state curriculum and

standards committees, as well as the Southern Regional Board under the Gates

Foundation educational initiatives.

Data from the State Department of Education in 2012 showed that the local

district in question was in need of remediation and had been on corrective action due to

declining student achievement since 2004. The purpose of the math assessment policy

was to address these five issues:

(1) What are the student group’s needs and/or have they already learned? (2)

Based on the data, what are the contributing factors? (3) Develop a plan to

address the student needs and/or strengths. (4) Implement the plan with fidelity.

(5) Use common assessments and NWEA [Northwest Evaluation Association] to

determine the effectiveness of the plan in regard to SSAP [State Student

Assessment Program] proficiency levels. In addition, we required all teacher

teams to develop formative assessments (common assessments) as content area

teams after reviewing all SSAP data. Furthermore, math teachers assessed

standards to get students to grade-level proficiencies.

Despite use of the math assessment policy, it is unclear whether or not the local school

district has closed the achievement gap for special education students and minorities. The

achievement gap issue will be discussed more thoroughly in later sections. Closer

examination and analysis of the mathematics assessment policy served to explain why it

is difficult to tease out achievement gains.

Definitions

During an evaluation of the policy process, it is important to clearly define the

terminology being used. Certain terms have specific meanings to people who study

education policy and policy evaluation at large. Below are the terms typically related to

educational assessment policies that are used in this evaluation of the policy process.

Administrators: Both the principals in a school who manage or have executive

charge of the school and responsibility to see that the math assessment policy is

implemented, and those having responsibility for the general organization, direction, and

supervision of the school district, its programs, and its staff, possessing accountability to

parent groups and the school board.

Assessment: An ongoing process aimed at understanding and improving student

learning through feedback. Assessment helps to inform students of what their future

learning should be and informs teachers of how their instruction can improve (Black &

William, 1998; Marzano, 2010; Walvoord, 2010).

State Student Assessment Program (SSAP): A state achievement test delivered to

students in the spring of each school year to determine student levels of achievement in

mathematics.

Effective Math Assessment Policy: According to the superintendent in charge

during implementation of the math assessment policy, the goal of the policy was “to

allow teachers to look at specific and detailed levels of learning for each student and

allow teams to focus their instruction by groups of students and/or individual students”.

An effective math assessment policy will help to determine what students learned,

identify factors contributing to a lack of learning, and address student needs and

strengths.

Evaluation: The process by which a policy is determined to have accomplished

the goals for which it was designed (Bardach, 2008, 2011; Theodoulou & Kofinis, 2004).

Interpretive theory: A theoretical framework that describes human and

organizational behaviors in abundant detail that allows individuals outside the culture or

organization to understand the context of these behaviors (Bastedo, 2005; Geertz, 1973;

Hamman & Hopson, 2013; Humphrey, 2008; Miller, 2002).

Math assessment policy: The policy and practice of math teacher teams creating

common assessments, analyzing student growth measures, and reporting these results to

the superintendent in charge of instruction.

Policy: Formal strategic decision-making processes engaged in by a governing

body. Individuals and communities are categorized and assigned positions in different

social, political, and power contexts (Hamman & Hopson, 2012).

Policy analysis: The process of uncovering complex information about how a

policy was created, the policy’s expected outcomes, the policy’s results, or how the

policy was put into place (Fischer, 1995; Patton & Sawicki, 1993; Yanow, 2000).

Policy implementation: The act of carrying out the math assessment policy, by

either creating the assessments for the policy, administering the assessments, collecting

the data from the assessments, or a combination of these actions (Lane & Hamman,

2003).

Policy outcomes: The results that come from use of the math assessment policy

(Lane & Hamman, 2003).

Policy process theory: The theoretical framework that explains the processes

individuals within an organization go through when attempting to implement a policy that

is put in place by superiors in that organization (Klein, 2001; Lane & Hamman, 2003;

Sabatier, 2007; Spillane, Reiser, & Reimer, 2002; Weick, 1995).

Significance

The analysis of the local assessment policy process was meaningful because it

provided important insights regarding the effectiveness of the policy and suggestions to

improve the policy. Searching through district literature confirmed that no formal

analysis of the math assessment policy had taken place prior to this study. The goal of the

study was to evaluate the local mathematics assessment policy. Conducting a narrative

case study related to the math assessment policy of this southwestern school district has

provided stakeholders in the educational community the opportunity to be informed about

the math assessment policy and how the policy has affected teacher instructional focus,

teacher responsibility to the policy, teacher collaboration, and student achievement in

mathematics. The results of a qualitative narrative study can assist educators in

understanding the complex issues and challenges that come about when a policy is

implemented.

The more urgent significance of the project study involves helping the local

district with effective assessment policies. A more effective assessment policy could lead

to higher student achievement and help educators uncover areas in which students need to

improve and grow in mathematics. According to Ceneviva and Farah (2007), the manner

in which assessment policy is disseminated and later evaluated has a direct effect on

overall school achievement. Assessment policy evaluation should be transparent to the

parents and community of a school.

Implications of Not Evaluating the Policy Process

Policy process demands investigating information about how a policy was created

or put into place, the policy’s expected outcomes, and the policy’s results (Fischer, 1995;

Yanow, 2000). Effective policies evolve over time and only improve if given adequate

analysis and regular evaluation. Part of policy process theory is the systematic breaking

down of a broader issue into more manageable categories, which provides an opportunity

to gain a deeper understanding regarding how the particular element contributes to the

broader problem. A cognitive bias may be formed by anchoring created policies to

judgments and substantiations other than policy theory.

Social science theory demonstrates that bias is furthered when one interprets

information about a policy based on preconceived notions not founded in theory;

therefore, a deeper understanding of policy and practice cannot be achieved without

analyzing policy. If the policy contains gaps in practice or implementation, then the

development of a solution to these gaps and a resolution of these shortcomings can never

be realized.

Guiding/Research Question

Past research related to the evaluation of assessment policies beyond the local

district is wide ranging and varied. Cho and Kingston (2011) conducted a policy analysis

of assessments created to meet the requirements of No Child Left Behind (NCLB) for

students with disabilities and found that data on these students are not reliable enough to

be used by teachers to improve their instruction. Woessman (2011) recommended

analyzing policies on merit pay that involve student assessment scores from the Program

for International Student Assessment (PISA). Pickowsky (2012) used assessment to

evaluate student learning outcomes at the college level and advised registrars to use these

results to properly place students in their future courses.

Block (2012) evaluated a policy involving open- versus closed-book math

assessments at the college level and concluded that a policy supporting open-book

assessments resulted in deeper learning. Ward (2012) recommended that governments

create evidence-based policies targeting math assessments. This study is significant

because although policy evaluations regarding assessment exist within the literature,

there is a lack of literature specific to local math assessment policies, demonstrating both

a gap in practice and in the literature. This study has potential to assist with better

assessment policy formation and adds to the literature on math education policies.

I addressed a local problem that represented a gap in practice, which was that the

math assessment policy had not been evaluated for effectiveness, teacher perceptions, or

outcomes preceding this study. The district superintendent confirmed that the math

assessment policy had never been evaluated. The research questions addressed in the

study centered on the effectiveness of the policy, the nature of the policy, the outcomes of

the policy, and teacher perceptions of the policy. These research questions were as

follows:

What organizational context led to the creation of the math assessment policy?

a. How was the math assessment policy implemented?

b. What actors were involved with the implementation of the math

assessment policy?

1. How was math instruction conducted before and after the implementation of

the math assessment policy?

2. What are the perceived outcomes associated with the implementation of the

math assessment policy, and what is the basis for the perceived outcomes?

Relationship to the Local Problem and Research Questions

The Common Core Standards require that all schools examine their educational

practices and local policies and ensure that they lead to college and career readiness for

students (Common Core, 2014). Etherington (2011) defined narrative research as

collecting stories of participants’ experiences in order to collect data capturing complex,

deep, and nuanced understandings of the participants’ involvement in organizational

phenomena. I conducted a qualitative narrative evaluation of the value of the local math

assessment policy and its processes as they evolved from stakeholders in the local

education community. This study can inform decision makers in other districts in similar

situations about the quality of the local math assessment policy. This policy evaluation

uncovered approaches that may improve the policy, thereby enhancing student

mathematical knowledge and achievement.

Limitations

There are several possible limitations inherent within this study. The first

limitation specific to this policy evaluation is that the period of time in which this study

was conducted, 2 to 4 weeks for data collection, was relatively short. Longitudinal effects

that can occur in qualitative research may mean that conclusions drawn during a short

period of time could be different from data analyzed during a long period of time

(Houghton, Casey, Shaw, & Murphy, 2013; King, & Horrocks, 2010; Maxwell, 2004).

Because data were collected during a period of less than 2 months, the study may not

have captured the full effect of the math assessment policy over time.

The second limitation was that all participants self-reported data. Self-reporting

data can be problematic because the researcher must rely on the honesty and the accurate

memory of the participants (Attride-Stirling, 2001; Brutus, Aguinis, & Wassmer, 2012;

Simera, Moher, Hoey, Schulz, & Altman, 2010). Even when participants are intending to

report with complete honesty, they may lack the ability to maintain impartiality. This

impartiality will affect the ability of respondents to accurately report what they

experienced. Participants may lack the understanding necessary to answer questions

accurately. Finally, participants who self-report may possess response bias and be more

likely to respond either positively or negatively depending on their disposition or

personality traits.

The third limitation is that the results of a qualitative case study cannot be

generalized to the larger population of school districts in the same manner it is possible to

do with a quantitative statistical analysis (Merriam, 2009; Patton, 2001; Rubin & Rubin,

2012; Tracy, 2010). Quantitative studies in general sample a large number of participants.

The samples studied are analyzed numerically, and measurements are used to determine

the level of reliability and validity of the study. In contrast, studying only one location

with seven participants in a case study may limit the transferability of the findings of this

analysis. No previous evaluation of the math assessment policy and its development

process had been conducted to which the results of this study can be compared; therefore,

theories that resulted from the policy analysis will have no prior empirical findings to

which they connect (Hajer, 2003). As a final point, difficulties in predicting effective

improvements of the policy create their own limitations (Saint-Germain, 2002).

Review of the Literature Addressing the Problem

During a literature search on the math assessment policy in the winter of 2013,

only three newspaper articles and no academic articles were found. The first article

involved lagging test scores within the state’s high schools (Fincher, 2013). The second

article involved changing school schedules and cutting academic programs to save money

due to state budget cuts (McQuiggin, 2010). The final article addressed common

assessment results and their decline (Hudson, 2008). The common assessment results

mentioned in Hudson’s article were propagated through the math assessment policy.

However, no literature evaluating the math assessment was found.

Governmental databases that were used to identify educational reform practices

were those of the U.S. Department of Education and the State Department of Education.

Evidence related to the relevance of analyzing the math assessment policy was identified

through Boolean searches of policy analysis, assessment policy, assessment, and

education policy reform in Google Scholar, Education Research Complete, ProQuest,

Sage, and ERIC. All searches were conducted in English. The local district website was

also reviewed for any mention of analysis of the math assessment policy and directions to

staff about the math assessment policy. Public data obtained from federal and state

education department websites, the district website, school board minutes, the policy

handbook, and the personnel handbook also demonstrated that the math assessment

policy had not been evaluated.

Conceptual Framework

Evaluation has the potential to provide data regarding a policy’s effectiveness

(Foster, McBeth, & Clemons, 2010; Quinn, Dunn, McAdam, McKitterick, & Patterson,

2014). If a policy is not examined to determine whether or not its outcomes have been

achieved, then additional policy changes may be made through uninformed processes that

lack efficiency and effectiveness.

The conceptual framework of the study was rooted in policy process theory

(Ciolan, 2013; Hartley, 2009; Sabatier, 2007). Policy process theory consists of

institutional analysis (Hardy & Koontz, 2009; Nowlin, 2011; Ostrom, 2011; Sabatier,

2007; Thomas & Jorgensen, 2009; Weible, Heikkla, deLeon, & Sabatier, 2011).

According to policy process theorists, the level of government plays a vital role: The

closer a bureaucratic body is to the problem, the more likely individuals in the

organization may be to use the policy evaluation process and change a current policy

(Howlett, Ramish, & Perl, 2009; Karger & Stoesz, 2013; Knoepfel, Larrue, & Varone,

2011; Sabatier, 2007). Policy process theory involves democratic empowerment and a

common purpose among members of an organization (Hartley, 2009). The policy process

framework also breaks the policy-making process into smaller, more manageable pieces;

examines the role of structural barriers and policy makers’ responses to public opinion;

and considers how the policy is used by the organization.

Finally, policy process theory consists of the cultural, societal, and political

characteristics of a local setting. These frameworks explain how policy makers use

financial incentives and disincentives, documents, mandates, persuasion, influence, and

capacity building to compel members of an organization to follow the requirements of a

policy (Sabatier, 2007; Thomas & Jorgensen, 2009; Veselý, 2012, 2013). The policy

process framework was chosen because it explains the processes individuals within an

organization go through when attempting to implement a policy and understand the goals

of the policy, as well as the behaviors and environment that result from the policy that is

put in place by superiors in that organization (Klein, 2001; Lane & Hamman, 2003;

Spillane et al., 2002; Weick, 1995). The stages of policy process theory are explained

below.

The Stages of Policy Process

Problem identification. The beginning stage of policy process is problem

identification. In the first stage, the problem is defined and articulated by stakeholders

such as school board members and school administrators (Chan & Seddon, 2012;

DiNitto, 2010; Howlett, 2010; Howlett, Ramesh, & Perl, 2009; Petersen, 2009; Sabatier,

2007). The problem identification stage also involves identifying problems that warrant a

policy, freeing up space on the agenda by removing old policies that have lost their

relevance, and deciding what problems have come to the forefront. An example of

identifying a policy problem is NCLB, which gave legitimacy to a federal education

policy by forcing the public school system to test students on a state-by-state basis and

report those results to the U.S. Department of Education in order to receive federal

funding (Brill, 2011; Ravitch, 2010).

Once policy problems have been identified, nondecisions may become a form of

policy making itself (DiNitto, 2010; Hu, Xu, Dinev, & Ling, 2011; Marr & Huang,

2014). Special interest groups that are resistant to change are often behind policy push

back. Nondecision based policies also occur when administrators at state and local levels

avoid policy or action that they know their constituents will not support. The federal Race

to the Top initiative exemplifies policy making by avoiding policy decisions. Many states

have avoided Race to the Top funds because feedback from voters in their state shows

that their constituents oppose its requirements (Marr & Huang, 2014).

Agenda setting. The second stage of policy process is agenda setting. When a

policy is formulated, other alternatives should be considered and discussed by

stakeholders and interest groups (Baumgartner, 2013; Chaisse & Matsushita, 2012;

Farouk & Husin, 2011; Mason & Brown, 2013). Policy proposals are often crafted at this

stage, and in some bureaucratic circles, involve educational lobbyists from state

departments, policy entrepreneurs such as students and teachers advocating for systemic

change in schools, and private funding groups such as the Gates Foundation (Brill, 2011;

Ravitch, 2010; Zhao, 2012).

Policymaking. The third stage of policy process is the policy-making stage. Once

problems have been identified, policies must be made that address the problems

identified in the previous stage. Those policies must be adopted by the appropriate

administrative body. At the policy-making stage, not making a policy decision creates its

own policy (Farough, 2013; Hahn, 1990; Pinn, 2014). To persevere past this stage of

policy-making, the policy must be legitimized (Stone, 2012). Some of the ways policies

are legitimized are public statements made by elected officials, executive orders, budgets

that allocate money to support the policy, laws, rules, regulations, and administrative

decisions. If legitimacy is not achieved, then the policy will not be developed, and the

status quo is maintained (DiNitto, 2010). In certain cases, it is necessary to legitimize a

policy by analyzing the amount of money required to carry out a policy. Regarding the

math assessment policy, more time was required from teachers to implement the policy,

but there were no additional financial costs identified by the local district in the policy’s

implementation (senior administrator, personal communication, 2010).

Implementation. The next stage of policy process is implementation. In many

cases, participants fulfill the policy’s requirements after a law or rule is passed.

Implementation in terms of the math assessment policy includes teacher-created

assessments, scoring, and reporting those scores to the superintendent’s office (Crews,

Crews, & Burton, 2013; Lee & Faugher, 2013; Peterson, 2009; Yoon, Song, & Lee,

2013). In other instances, a policy is passed, and nothing happens or changes because of

the policy. The implementation stage can also be a continuation of the policy-making

process. Other issues that may arise are administrators’ unsuccessful attempts to separate

themselves from the politics of the policy. The policy may have more effect on the

organization in the implementation stage than policy makers intended. At the same time,

opponents of the policy may continue to prevent the policy’s implementation. Finally,

conflict may continue at this stage between policy makers and those opposed to the

policy.

Evaluation. The final stage of policy process is evaluation. There was a need for

the evaluative stage prior to the onset of this study, making it a highly significant tool for

filling in gaps in practice at the local level. An overall policy evaluation was the ultimate

purpose of this study as it aligned to the theoretical framework of policy process theory

(Mott, 2013; Murchan, Loxley, & Johnston, 2009). Policy evaluation consists of many

steps and variations to the process (Mears, 2010; United Nations General Assembly,

2007). These steps include checking the effects of the policy on the organization or

bureaucracy that it has been implemented in, making formal and informal evaluations,

guaranteeing the coherence and completeness of the policy, ensuring that the policy has

met its objectives, checking if the policy is being followed with fidelity by the institution,

verifying that the leadership behind the policy is of high quality, and ensuring that the

policy is performing as it was designed to be accountable to taxpayers and stakeholders.

Specific variations within education involve acquisition and interpretation of reform

policies by teachers and taking the interests of stakeholders into account (Chikritzhs,

2009; Creemers & Kyriakides, 2010).

Viewing educational policy evaluation from a local perspective was essential to

the study; however, there is much to learn from the literature involving policy process

theory and policy analysis from a national and global perspective. Policy analysis

research can provide guidance when evaluating local education policy. The literature in

this section is an outline of national and local assessment policy studies and their

findings. Strategies to gather the research for the literature review included the

examination of professional journals emphasizing policy analysis, noted books on

education policy and its theoretical frameworks, and scholarly websites. Pertinent data

were taken from these sources to create the review of the literature.

Chitty (2009) and Naidu (2011) indicated that teachers interact with policy on a

regular basis and must make choices based on policy in their daily lessons. Stein,

Kaufman, Sherman, and Hillen (2011) found that many school districts have policies that

specifically outline what teachers must teach in mathematics at particular grade levels.

The research from those who investigate teachers’ interactions with policy can best be

described as a policy analysis (Naidu, 2011; Rizvi & Lingard, 2013; Weimer & Vining,

2005). Within the context of policy process theory, teachers’ perceptions of their

students, classrooms, and schools that have been influenced by the math assessment

policy were explored in order to evaluate the assessment policy.

Challenges Associated With Policy Implementation

Walsha and Anthony (2009) identified the importance of implementing

instructional policies at the local level regarding mathematics. The authors recommended

that teachers and administrators collaborate on changes in math instructional policy and

fulfill these changes within the classroom. They advised that math faculties, teams,

departments, and teacher education programs should be given adequate resources.

Walsha and Anthony concluded that teachers, principals, coaches, mentors, researchers,

parents, students, and policy-makers must play a part in student math achievement.

Policy process theorists investigate the effect policies have on participants in their natural

setting to explain why and how individuals react to policy (Geertz, 1973; Klein, 2001;

Lane & Hamman, 2003; Leedom, 2001; Spillane et al., 2002; Weick, 1995).

Poverty predicts achievement better than reforms in education policy (Brill, 2011;

Hochschild, 2003; Ladd, 2011; Ravitch, 2010). In an atmosphere of constant policy

change and implementation, individual teacher attempts to meet student needs and the

socioeconomic backgrounds of students are often neglected. NCLB legislation requires

that all public schools assess student achievement in mathematics in third through 10th

grade using tests that measure performance against state-adopted academic content

standards (U.S. Department of Education, 2011; Zucker, 2004). Adopting assessment

policies that are based on state standards does not promise student achievement gains,

and attempts within the Chicago schools have shown mixed results (Luppescu,

Allensworth, Moore, de la Torre, & Murphy, 2012).

Utility of Conceptual Framework

Policy process theory was used to explain how individuals may have understood

the math assessment policy and the process they went through to implement the policy

within the local district as well as the final stage of the policy process, which requires that

the policy be evaluated. With the process of implementation in mind, the research

questions described in the previous guiding questions section were central to the study

and helped to explore problems with the policy, teachers’ and administrators’ perceptions

of the policy, and ways in which teachers and administrators believed the policy could be

improved. Analysis and interpretation of the data collected from these open-ended

questions were closely examined to uncover how teachers’ understandings of the goals of

the policy affected the way that teachers taught, the professional change experienced by

teachers and administrators, and the transformation of the school environment

(Gigerenzer & Selten, 2002; Gueudet & Trouche, 2009; Leedom, 2001; Simon, 1990;

Weick, 1995).

The Policies of a Nation at Risk

The National Commission on Excellence in Education (1983) report titled “A

Nation at Risk” (ANAR) was the definitive policy document for the 1980s. ANAR called

for higher standards with an emphasis in mathematics and science to make U.S. students

career and workforce ready and globally competitive. The Center for the Study of

Mathematics Curriculum (CSMC, 2005) summarized the educational report from ANAR,

the participants, the findings of the report, and the concluding recommendations. The

commission expressed concern about how many teachers were teaching in the science,

technology, engineering, and math (STEM) fields without any experience or credentials

and how few schools required an adequate amount of course work in these fields for

graduation.

Policy recommendations from ANAR have yet to be realized in U.S. schools.

Many schools still do not require high school students to complete three or more credits

in mathematics and science to receive a diploma (Anderson & Chang, 2011; Carol Morris

Consulting & The Lee Institute, 2011; CSMC, 2005). Brill (2011) noted that the advice

made by the ANAR commission was largely ignored by public schools when the

document was issued. ANAR is the origin of the assessment policies surrounding NCLB

and Race to the Top educational reform initiatives (Brill, 2011). Without the concerns

addressed in ANAR, education policy would likely not have taken on the shape it has in

the present day. Many at the time saw ANAR as a social emergency in the making.

Monumental, catastrophic thinking has created an environment of conspiracy theories

surrounding teachers and their unions and has been counterproductive in the search for

policies that create more effective schools and better modes of instruction (Ravitch,

2010).

Goldberg and Harvey (1983) discussed their participation on the National

Commission on Excellence in Education and their perspective on the commission’s

findings. They reacted to the public’s response to ANAR and emphasized reasons to be

optimistic about the gains public schools can make with students. They urged readers not

to overreact to their committee’s findings and to seek a positive avenue in responding to

these findings. Goldberg and Harvey called for positive proactivity among educators, not

reprimanding educators through outside authority and political pressure.

Rothstein (2008) analyzed the recommendations made in ANAR more than 2

decades later and commended the commission’s attempt to make a coherent, systematic,

and comprehensive national curriculum, but they noted the regrettable the lack of U.S.

educators who had accomplished this task. Reaction to ANAR can be seen in light of the

Cold War and the political urgency generated from it. Educating U.S. students was not

only important; excellent education was seen as the patriotic duty of the citizens and

government. Ansary (2010) found that ANAR was in large part a political tool for the

Reagan administration to take the subject of education from Reagan’s political

competitor in the 1984 election and make it his own.

Outcome and Standards Based Education Policies

Hood (2011) documented education policy measures that took place in Chicago

public schools in the late 1990s and early 2000s. Despite education reform, Chicago’s

schools have failed to close the achievement gap between minority students and their

nonminority peers.

One of the major teachers’ unions outlined nine components for creating quality

education policy. These components are:

1. Standards must focus on academics.

2. Standards must be grade-by-grade or clustered for selected grade spans in

elementary, middle, and high school.

3. Standards must be clear and specific enough to lead to a Common Core

curriculum.

4. Standards must include particular content in each of the four content areas—

English, math, science, and social studies.

5. Standards must attend to both content and skills.

6. Standards must be manageable, given time constraints.

7. Standards must not dictate how material should be taught.

8. Standards must be rigorous and “world class.”

9. Standards must be written clearly enough for all stakeholders to understand

(American Federation of Teachers, 2003).

Recommendations such as these foreshadowed education policies that led to the

Common Core Standards that states have recently adopted to qualify for Race to the Top

funds (U.S. Department of Education, 2012).

The No Child Left Behind Act and the Consequence of Its Policies

Ackley (2011) discussed the consequences of NCLB policy and asserted that the

goal to have 100 percent of U.S. students proficient in mathematics and language arts by

2014 is unrealistic, problematic, and will soon result in all schools in the U.S. being

labeled as failing. Finn (2008) documented his individual experience as an education

policy maker in addition to outlining the history of U.S. education during the last 60

years. One consequence of the assessment policies implemented by NCLB was schools,

teachers, and administrators rigging the testing system to show false gains in student test

data (Finn, 2008). Schools that have cheated during NCLB reform (Baltimore, New

York, Washington D. C., Atlanta, and Philadelphia) were contrasted with hardworking

school districts making true gains with their students. Honest school districts have been

overlooked by the Adequate Yearly Progress (AYP) policies of NCLB (Roth, 2011).

Ravitch (2010) identified further evidence of schools cheating under NCLB

policies in her chapter on “The Trouble with Accountability” (p. 155). Ravitch cited

many instances of top-performing school districts gaming the system only to be

discovered later during state department education auditing as having cheated on high

stakes tests. She concludes by stating that the more teacher performance hinges on these

scores, the more cheating will occur.

Simpson, Kite, and Gable (2006) conducted cluster analyses of 113 districts in

Massachusetts from 2003 to 2005. Simpson et al. demonstrated that variations among

schools participating in NCLB are a result of how resources are used and allocated in

those school districts. It is difficult to determine which resources contribute to increased

student achievement because there are so many confounding variables such as student

economic status present within each system at one time.

Brown (2008) studied the interplay of education policy related to accountability

and local control. Brown demonstrated prior to NCLB policies, National Assessment of

Educational Progress (NAEP) documented Wisconsin students were high achieving and

not in need of reform. He argued local control has been effective in keeping NCLB

reforms in check at fairly low cost to school districts in Wisconsin. In Brown’s opinion,

NCLB policies could potentially lower student achievement in Wisconsin.

Rotherham and Dillon (2007) reported on all 50 states’ AYP measurements

required by NCLB policy and the unrealistic idea that 100% of these states’ students can

become proficient in mathematics and reading. Rotherham and Dillon concluded there

exists wide variability and inconsistencies in how each state calculates its own AYP. The

researchers called into question the validity of the very measuring tools required by

federal assessment policies.

Brown (2010) also discussed how growing up in the specter of NCLB policies has

affected college students enrolled in teacher education programs. Potential educators

have shaped their idea of what it means to be a teacher around the environment NCLB

created in each classroom they participated in prior to college. Students growing up

during the NCLB period of education policy are more skeptical about the high stakes

tests than previous generations and feel they limit what could be a more enriching

educational experience.

Goldstein (2011) charted the effect of media on education policies during the

Bush Administration era leading up to and including the inception of NCLB legislation.

Goldstein cited numerous examples of negative media portrayal of teachers and their

unions that have resulted in a lack of respect for the profession of teaching as a whole in

the United States. The reported cases of ineffective teachers are disproportionate to the

actual makeup of the profession. Poor teachers and low scores are over-reported;

whereas, high scores and good teachers are often underreported or not mentioned at all in

the media. Nevertheless, negative reporting of public schools and their teachers has

significant influence on education policies enacted at the local, state, and federal level.

Particularly related to this study are federal policies that result in adoption of local

policies. Grissom (2009) scrutinized the factors that determine how public school boards

implement local policies in the era of NCLB. Grissom concluded that professional

decision-making practices and racial homogeneity of members are highly predictive of

board success in implementing policy.

School boards are the primary local influence on education policy and their

professionalism directly affects teachers’ daily practices. McIntosh (2011) described new

high school assessment trends resulting in more testing of freshmen, sophomore, and

junior students and fewer exit assessments for seniors as a requirement for graduation. To

view assessment data collected as a requirement of NCLB policies, schoolview.org is a

log of public records involving all student performance records for all public schools in

the local district’s state and showed their classifications in terms of both growth and

NCLB requirements for AYP.

From a federal policy perspective, NAEP is the assessment used to nationally

rank state educational systems in the U.S. (Institute for Education Sciences, 2011; “The

Nation’s Report Card”, 2010). Lane et al. (2009) described how NAEP scores are

calculated and calls validity based on equity into question. Klein (2010) discussed the

vague objectives of the NAEP mathematics assessment. Klein claimed that it is

unreasonable to expect gains in the NAEP because the test is meant to measure IQ.

Educators should proceed with caution when using NAEP data to determine the quality

of instruction in the United States. Klein further stated policies on assessment, as

mandated through NCLB, tied directly to state standards, are more reliable. In recent

years equity has improved as inclusivity of students requiring special accommodations

have increased (Kitmitto, 2011; Maxwell & Shah, 2011).

On the question of how U.S. schools compete with foreign schools, Sahlberg

(2007, 2012a, 2012b, 2012c) pointed out stark differences between Finnish and U.S.

school leadership, instruction, and testing practices. Finnish teachers are the overall

leaders of their schools, whereas in the U.S. a management top-heavy with a top-down

administrative style remains the predominate norm in school leadership. The ability for

teachers to oversee their schools and have the ultimate say in how their job is conducted

and their subjects are taught makes teaching one of the most popular careers in Finland.

Mondom (2011) outlined the educational history leading up to the writing of

NCLB and discussed problems that have arisen in schools that implement the policy at

the local level. These issues stem from the unintended consequence of NCLB, resulting

in most schools emphasizing only mathematics and reading, with the idea that students

will learn everything they need when only concentrating on these two subject areas. The

difficulty with only concentrating on math and reading is that many districts often use the

NCLB agenda to do away with their fine arts programs, including music and theater

programs. Less of an emphasis on history and civics resulted, creating a group of citizens

that does not know their rights and are intellectually narrow. The largest injustice is that

U.S. children may have no knowledge of how the U.S. came to be or major historical

events that were important to its founding. A thin exposure to history and the arts resulted

in high school graduates who are not well rounded and cannot think critically.

Stein, Kaufman, Sherman, and Hillen (2011) discussed the results of local district

policies that require mandatory algebra for all students and their effects on student

achievement. NAEP data were used to study these effects. They demonstrated algebra for

all only works when effective interventions for at-risk students are available and in

frequent use. The researchers recommended education policies that require algebra for all

must also contain a well-thought-out intervention policy.

Walker and Mohammed (2008) recommended changing NCLB policy by using

the NAEP rather than state assessments. Rather than look for 100% of all students to be

advanced and proficient by the year 2014, it would better serve schools and students to

look at score growth as a measure of effective teaching and take into account socio-

economic status factors as part of the calculation. Many states have currently enacted

some form of a growth policy rather than using AYP calculations, but as of yet, have not

used the NAEP assessment (U.S. Department of Education, 2012).

Race to the Top Policies

Brill (2011) told the stories that have affected the nation’s current education

policies, including funding of education by private organizations such as the Bill and

Melinda Gates Foundation, social attitudes of Americans toward education and the

relationship between unions and federal policy makers. He chronicled Race to the Top as

it affected public schools in New York’s Knowledge is Power Program (KIPP) schools in

Texas and high-performing teachers in the Teach for America program. He noted burnout

is a significant part both of administrator and teacher turnover in America’s high-

performing charter schools and contended reformers and public officials will have to

develop new policies when working with teachers’ unions in order to create sustainable

lasting improvements in public education.

Through Race to the Top the U.S. Department of Education asks public schools to

advance school reform by:

 Adopting standards and assessments that prepare students to succeed in

college and the workplace and to compete in the global economy.

 Building data systems that measure student growth and success, and inform

teachers and principals about how they can improve instruction.

 Recruiting, developing, rewarding, and retaining effective teachers and

principals, especially where they are needed most.

 Turning around our lowest achieving schools (U.S. Department of Education,

2014).

Congruent with the requirements outlined in the aforementioned legislation,

Varela (2011) documented factors that contribute to teachers in a district being

unqualified and noncertified for their positions. These factors include teacher shortages

and student socio-economic status. Sometimes schools cannot meet the requirements of

federal policy at the local level. For instance, students who are poor are far less likely to

have classroom teachers who are highly qualified, despite federal education policy

requiring it since the 2005-2006 school year (U.S. Department of Education, 2012).

Sewlyn (2007) investigated teacher opinions of NCLB and the effects on their

classroom instruction, professional development, and hiring pool. Sewlyn found NCLB

has decreased the number of qualified teachers graduating from universities and the

number of new students entering the profession. NCLB has also decreased the number of

minority teachers and males entering the profession. The aforementioned research

findings are important for policy makers to consider if increasing the number of qualified

male and minority teachers is a future goal. Cohn (2007) researched education policy that

gives classroom teachers the authority over their curriculum and assessments—important

issues to consider in the creation or analysis of policies involving teacher autonomy.

Teacher autonomy improved the quality of instruction and increased student achievement

gains.

Implications for Possible Project Directions

Possible directions that could have been taken other than evaluating the policy

process are using an impact evaluation, an organizational review, and process monitoring

(Aguinis, 2011; Bergman, Ellingsen, Johannesson, & Svensson, 2010; Janiszewski & Uy,

2008; Kepes, McDaniel, Brannick, & Banks, 2013; Lichfield Diocesan Board of

Education, 2011; World Bank Group, 2013). An impact evaluation would have assessed

changes in the well-being of teachers, administrators, and students attributed to the math

assessment policy. The well-being of participants was not the focus of this study.

Information on well-being alone does not tell stakeholders the level of effectiveness of

the math assessment policy, nor does it answer the questions posed in the previous

sections of this study.

An organizational review involves obtaining objective and well informed

feedback about the math assessment policy (Council of the Great City Schools, 2012;

Sitek, Seifert, & Klaus-Dieter, 2010; Swan, Bresnan, Newell, Robertson, & Dopson

2010; Vithessonthi & Thoumrungroje, 2011). The purpose of an organizational review is

to improve an organization’s internal communication and shared understanding of the

math assessment policy by providing an opportunity for teachers and administrators to

provide anonymous and thorough feedback about the math assessment policy. As is the

case with impact evaluations, an organizational review will not provide empirical

evidence regarding whether or not the math assessment policy was effective. Improved

communication and understanding were desired outcomes of the policy analysis.

However, this was not a sufficient outcome to determine the effectiveness of the math

assessment policy.

Process monitoring is a formalized system for measuring the performance of an

organization or service (Rodriguez, Neussbaum, Lopez, & Sepulveda, 2010; Thornton,

Ocasio, & Lounsbury, 2012; Van Gestel & Hillebrand, 2011). Process monitoring

measures how well the school district is performing and uses these measures as a

management tool. Process monitoring also assesses how outcomes are attained over time.

The credibility of the policy is reported to the public systematically to identify effective

practices within the policy. Process monitoring may be a lengthy process because it

requires continuous annual collecting and analyzing of information to compare expected

results to yearly performance of the outcomes of the math assessment policy. A lengthy

time frame for data collection is beyond the scope and sequence of a doctoral study

project because of the length of several years required for data collection.

Transition Statement

Section 1 was an explanation of the problem and its significance. Section 2

contains the methodology that documents the broader problem. The broader problem was

that this study marks the first time the local math assessment’s policy process has ever

been evaluated. Section 3 includes the research design and methodology. This section

includes the qualitative research design and approach, participants, data collection,

analysis, and the limitations associated with the research.

Section 3 consists of an analysis of the math assessment policy process based on

the findings from the interviews of teachers and administrators (Jitendra, Griffin, & Xin,

2010). This section is made up of an introduction that describes the project, goals of the

project, rationale of the chosen project genre, and how the problem will be addressed.

Another review of the literature addressing the project is included in Section 3 along with

discussion of needed resources, supports and barriers, and roles and responsibilities of

stakeholders. Finally, project implications are discussed including social change

implications, and the importance of the policy analysis to local and national or global

stakeholders.

Section 4 is an explanation of the advantages and disadvantages of the math

assessment policy, its importance, implications, applications, and directions for future

research. This section involves a reflection on what I have learned as a scholar,

practitioner, and project developer. Recommendations to address the problem differently,

analysis of scholarship, project development and evaluation, and leadership and change

are also discussed.

Section 2: The Methodology

Introduction

Description of the Project

The project is an evaluation of the math assessment policy process in a

southwestern school district. In order to bring about effective district policies, all

stakeholders must understand the level of effectiveness of current policies. The means by

which to determine efficacy is a policy evaluation. For this evaluation of the processes

that led to the math assessment policy and its continuation, I developed an evaluation

report, using both Sabatier’s (2007) five components of the policy process and Bardach’s

(2011) eightfold path of policy analysis. In this project, I identified the strengths and

weaknesses of the math assessment’s policy process. I used contributions from teachers

and administrators via participant interviews to evaluate the policy process’s positive

outcomes and drawbacks. The project concludes with recommendations for improvement

of the math assessment policy and future development of policy within the local district.

Goal of the Project

The goal of the project was to evaluate the processes used to create the math

assessment policy and to determine whether the policy had accomplished its intended

outcomes. In this section, the project, which was an evaluation of a local math assessment

policy, is described. The study is framed by the theoretical frameworks of Sabatier’s

(2007) policy process theories and Bardach’s (2008, 2011) eightfold path, which was the

framework that assisted in evaluation of the policy.

Selection of Qualitative Analysis Research Design

I evaluated the policy process concerning math assessment through qualitative

research. According to Creswell (2008) and Lodico, Spaulding, and Voegtle (2010),

qualitative researchers explore relationships among people and problems. I focused on

teachers and administrators, evaluating the successes and shortcomings of a math

assessment policy in a local setting in the southwest. Teachers and administrators were

interviewed about their experiences with a local math assessment policy. Using

qualitative research allowed me to delve deep into participants’ thoughts. The selection of

interviewing as the means of inquiry supported a focus on exposure and understanding

thick descriptions of experience.

Thick description is the process of focusing on contextual features when

observing and interpreting social meaning (Creswell, 2008; Ponterotto & Grieger, 2007;

Rankin, 2011). I noted not only what was happening among people within an

organization in a literal sense, but also what might be causing a particular event.

Qualitative research involves a variety of research approaches and allows a researcher the

opportunity to inquire about the natural environment without establishing or testing

predetermined hypotheses (Denzin & Lincoln, 2011; Newman & Benz, 1998). According

to Jacob (1988), qualitative research is aimed at investigating the realities of everyday

circumstances.

Selection and Justification of the Type of Analysis Conducted

A case study evaluation of a math assessment policy was conducted in a rural

southwestern school district. The justification for using a case study evaluation was that

in discussing the math assessment policy and reviewing district resources including the

personnel handbook, it was evident that an analysis of the process that was used to

develop the policy had not been conducted prior to this study. The lack of analysis of the

math assessment policy was confirmed by the senior administration (senior administrator,

personal communication, 2012). The math assessment had not been evaluated before the

onset of this study; therefore, a gap in practice was filled through this study.

Organizations such as school districts need to address possible gaps in local practice to

determine whether money and time should continue to be spent on carrying out a policy.

In addition, organizations need to determine whether policies are being carried out in the

way in which they were intended.

Overall Analysis Goal

The overall analysis goal was to explore the successes and/or weaknesses of the

math assessment policy as perceived by the administrators and teachers who created and

administered the assessment mandated by the local policy. A secondary goal of the

analysis was to determine whether the policy was effective in reaching the goal of greater

student achievement in math. The final goal of the analysis was to determine whether or

not future changes should be made to the math assessment policy.

Participants

Criteria for Selecting Participants

The math assessment policy was implemented in the local district in the fall of

2005 (senior administrator, personal communication, 2014). Those invited to participate

were K-12 mathematics teachers and administrators who had been working with the math

assessment policy for between 1 and 9 years. Teachers and administrators who had used

the policy for 5 or more years had the most depth and breadth of experiences to share and

were ideal participants for the evaluative study. Teachers and administrators who had

fewer than 5 years of experience with the policy demonstrated a different frame of

reference with regard to the policy. Including a cross section of experienced and

inexperienced teachers helped to bring depth and objectivity to analysis of the policy

process. Participants were interviewed for 1 hour or less and took time after the interview

to read through the transcribed interview, review the findings, and confirm correct

interpretations. Creswell (2008) and Lodico, Spaulding, and Voegtle (2010) referred to

this process as triangulation.

Number and Justification of Participants

Three teachers, three administrators who had been teachers at the onset of the

policy and then became principals, and two administrators who had been using the policy

since its inception were interviewed for this study. The small number of participants

allowed for deep inquiry during interviews to uncover each participant’s detailed

experiences with the math assessment policy (Creswell, 2008; Lodico et al., 2010;

Merriam, 2014; Patton, 2002). Quantitative instruments restrict responses to

predetermined classifications by using standardized instrumentation and analysis.

Consequently, quantitative researchers are able to gauge reactions of many participants,

which can increase the amount of data and therefore breadth. Conversely, qualitative

studies usually investigate only a few cases, with great depth, exhaustive descriptions,

and context enhancing the study with depth of knowledge from participants, rather than

scanning the surface with breadth of knowledge as is the case with quantitative analysis.

A qualitative study was ideal for achieving this level of depth with the small number of

participants who were included in the study.

Procedures Used to Gain Access to Participants

Data collection took place when Institutional Review Board (IRB) approval was

granted and approval from the local district’s superintendent was verified. Requests for

interviews were presented to six teachers and six administrators. Of these 12, three

teachers and five administrators agreed to participate in this study. These participants

signed consent forms demonstrating their voluntary cooperation in the study prior to data

collection. I ensured that vulnerable populations were protected, and all requirements for

the IRB’s research on human subjects were met. I completed NIH certification (729287),

and received an approval reference number from IRB: 04-23-14-0168413.

Methods for Establishing a Researcher-Participant Relationship

According to Balthasar (2011), transparency in the researcher-participant

relationship is paramount in establishing trust. To establish a trusting relationship with

participants and ensure that participants gave fully informed consent, an explanation of

the effect of an analysis of the math assessment policy was disclosed. The participants

were informed, their interviews were recorded, audio recordings were destroyed within

60 days, and transcriptions are locked and password protected and will continue to be

until May 10, 2019, as required.

Measures for Ethical Protection of Participants

Three teachers and five administrators were interviewed based on the assurance of

confidentiality of participation. Lodico et al. (2010) pointed out that informed consent is

required for all participants in all cases. In the analysis of the math assessment policy,

participants were provided consent forms and asked to provide signatures prior to data

collection. Necessary components of the informed consent form that cited the risks,

benefits, and procedures involved in the study were provided. The district’s

superintendent was solicited for approval of the study prior to data collection. Each

participant was identified with alphanumeric coding to preserve participants’

confidentiality. All data collected from interviews are securely locked and password

protected.

Data Collection

Methods for the Collection of Qualitative Data

The conceptual framework of policy process theory guided data collection (Klein,

2001; Lane & Hamman, 2003; Sabatier, 2007; Spillane, Reiser, & Reimer, 2002; Weick,

1995). Throughout data collection, identifying markers of the policy process were

investigated. The distinct steps of problem identification, agenda building, policy

formation, policy implementation, and policy evaluation were noted during data

collection and incorporated into the interview questionnaire.

First, invitations to participate, which included the nature and purpose of the study

and statement of IRB approval, were emailed to all prospective teachers and

administrators. One follow-up email was necessary for participants who did not respond

to the first email. Some teachers and administrators who were approached declined to

participate in the study, citing too many obligations during the last month of school

taking up their time. Participants were asked to sign an informed consent document prior

to the interview. Once participants were determined, interview dates and times were

scheduled at the convenience of the participants. All interviews were conducted onsite at

the local setting unless otherwise requested by the participant. The interviews were

completed within a 2-week time frame and scheduled at the convenience of the

participants.

A researcher-designed instrument was used to guide open-ended interviews. To

ensure confidentiality, each participant was assigned a number for identification during

and following the interview. These interviews were intended to last between 60 and 90

minutes per interview. Each interview was audio recorded using my password-protected

iPod Voice Memo app. I transcribed the data into printed form and coded for data

analysis of emergent themes that were used to create thick descriptions (Creswell, 2008).

All participants were given the opportunity to member check by reviewing the findings

and interpretations.

Instrumentation

A researcher-created interview instrument was used in the study. The interview

instrument consisted of the interview questions (see Appendices C and D). According to

Turner (2010), pilot studies are a very helpful means to refine interview questions;

however, according to Lodico et al. (2010), pilot samples are used primarily with

populations in which there are a large number of people that a sample can be collected

from. In this case, there were only 12 people making up the population who had used and

were using the math assessment policy within the local district, and only eight members

of this population agreed to participate in this study. Conducting a pilot study would

likely have resulted in even fewer people taking part in the final study. The number of

participants within the study was maximized to the extent possible by avoiding a pilot

study.

The interview questions were aligned with the guiding questions present within

the previous section, as demonstrated through the Template for Aligning Research

Questions with Interview Protocol (see Appendix D). The interview protocol was used to

discover teacher and administrator perceptions of the math assessment policy in terms of

meeting its original outcomes, its effect on math instruction, teacher and administrator

perceptions of the policy, and its effect on student achievement (see Appendices B and

C).

Implementation Plan

The researcher-created interview instrument was intended to give insight into the

benefits and costs to teachers of conducting the assessment policy as required by

administration. Responses identified points for future improvement of the assessment

policy. Each interview took less than 1 hour. Nunnery, Ross, and Bol (2008)

substantiated the validity of using data on teacher perceptions as a tool for analyzing

educational policies. Therefore, eight personal interviews were used in gathering data for

the study to explore educator perceptions of the successes or failures of the mathematics

assessment policy. The interviews were conducted in a location requested by participants

and audio recorded for transcription post interview. Scripting notes were also taken

during the recorded interview for comparison during analysis.

System for Keeping Track of Data

I used qualitative research with interviews as the strategy of narrative data

collection. I used face-to-face interviews as the data collection method. King and

Horrocks (2010) noted that the use of interviews for narrative research designs allows the

participants and researcher to interact in a demonstrable way that results in deep

exploration of the experiences of the participants. I recorded, transcribed, and coded the

data for emergent themes to organize and keep track of the data. Data are currently stored

securely. Five years from May 9, 2014, when data collection ended, the data will be

destroyed so that study participants may remain confidential (Creswell, 2008).

I conducted qualitative interviews with eight participants. Interviews consisting of

11 questions were conducted with math K-12 teachers and administrators familiar with

the math assessment policy; these interviews were audio recorded. The audio-recorded

interviews were transcribed, member checked for accuracy, and peer debriefed for

validity and reliability. Initially, participants were interviewed based on the order in

which they responded to the email requesting participation and when they requested to be

scheduled. All transcripts were reviewed line by line for each participant. Once a possible

theme was identified, an identifying phrase was created, and this phrase was assigned a

color. Discrepant themes were assigned a color different from common themes for ease

of identification.

The Process for Generating, Gathering, and Recording Data

The conceptual framework of policy process theory guided data collection (Klein,

2001; Lane & Hamman, 2003; Sabatier, 2007; Spillane, Reiser, & Reimer, 2002; Weick,

1995). Throughout data collection, identifying markers of the policy process were

investigated. The distinct steps of problem identification, agenda building, policy

formation, policy implementation, and policy evaluation were noted during data

collection and were incorporated into the interview instrument.

I e-mailed invitations to participate, including the nature and purpose of the study

and statement of IRB approval, to all participating teachers and administrators. Follow-up

e-mails were sent to those who did not respond to the first e-mail. In response to these

two requests, eight out of 12 participants agreed to be interviewed. Participants were

asked to sign an informed consent document prior to the interview. Administrators

granted permission for teachers from their school to be interviewed. Participants

determined which interview dates and times were most convenient. All interviews were

conducted onsite at the local setting, except in two cases in which the participants

requested to be interviewed at home. All interviews were completed within a 2-week time

frame.

Finally, a researcher-designed instrument was used to guide open-ended

interviews. To ensure confidentiality, each participant was assigned an alphanumeric

code for identification during and following the interview. These interviews lasted 45 to

60 minutes. I audio recorded each interview using my password-protected iPod Voice

Memo app, transcribed the data into printed form, and coded for data analysis of

emergent themes that could be used to create thick descriptions (Creswell, 2008). All

participants member checked the transcription to review the findings and interpretations

and found them satisfactory in depicting their interview. A peer debriefer was also used

and stated agreement with the interpretations and findings, indicating that they were

reasonable and reliable by stating, “This all looks good to me” (personal communication,

2014).

A researcher-created interview instrument was utilized in the study. The interview

instrument consisted of the interview questions (See Appendices C and D). The interview

questions aligned with the guiding questions present within the previous section as

demonstrated through the Template for Aligning Research Questions with Interview

Protocol (See Appendix D). The interview protocol sought to discover teacher and

administrator perceptions of the math assessment policy in terms of meeting its original

outcomes, its effect on math instruction, and teacher and administrator perceptions of the

policy, and its effect on student achievement. Explanation of the instrument has been

included in the following section.

The self-created interview instrument was intended to give insight into the

benefits and costs to teachers of conducting the assessment policy as required by

administration. Responses identified points for future improvement of the assessment

policy. Nunnery, Ross, and Bol (2008) substantiated the validity of using data on teacher

perceptions as a tool for analyzing educational policies. Eight personal interviews were

used in gathering data for the study to explore teacher perceptions of the successes or

failures of the mathematics assessment policy. The interviews were conducted in the

participants’ classrooms or offices, with the exception of two participants who requested

to be interviewed at home, and audio recorded for transcription post interview.

Role of the Researcher

The role of the researcher in the local setting was teacher and mathematics

department chairperson, coach, and mentor of the district’s only high school. It was

important for the researcher to be mindful of the possibility of power asymmetry and

avoid it wherever possible (Humphrey, 2008). Participants were selected who did not

have a subordinate role to me, the researcher. The largest number of participants worked

in other schools. Participants were my acquaintances and had very little professional

influence from me, as I was not a supervisor or administrator to them. To decrease power

asymmetry, as the researcher, I restated interviewers’ responses and included my

interpretation of what the interviewee stated (Van der Vegt, deJong, Bunderson, &

Molleman, 2010).

Data Analysis

Participant Interviews

The purpose of data analysis was to analyze a local mathematics assessment

policy in a southwestern school district. The process for organizing the data for analysis

included the review of interview data from multiple points of view and interpretation of

that data through understanding teacher and administrator perceptions in-depth (Creswell,

2008; Denzin & Lincoln, 2011). Analysis consisted of data obtained from open-ended

interviews. The theoretical framework of policy process theory was intimately connected

to the analysis of data because during analysis, answers to participants’ questions about

problem identification, agenda building, policy formulation, policy implementation, and

policy evaluation were discerned (Ceneviva & Farah, 2007; Chikritzhs, 2009; Cho &

Kingston, 2011; Petersen, 2009).

During the open-ended interview process, different characteristics of a qualitative

case study design were used (Creswell, 2008; Janesick, 2004; Mills, 2003; Seidman,

2012). First, the exploration of how the math assessment policy worked K-12 and what

administrators and teachers thought about the policy during its implementation and

continuation were studied. In addition, the discovery of how the math assessment policy

evolved since its creation and whether its intended outcomes were realized were pursued

and evaluated.

Inductive reasoning was used to carry out the interview process. The concepts

described by teachers and administrators about their instructional practices in relation to

the math assessment policy were revealed. Likewise, concepts described by

administrators about school-wide factors that were affected by the math assessment

policy were explained. Finally, the purpose of dialogue with interview participants was to

discover points of view, to organize experiences, and make sense of the effect on

instruction and student achievement.

Thick Description

An important step in the data analysis process included thick descriptions of the

routine use of the math assessment policy by faculty and administrators that express the

perception of the respondents (Geertz, 1973; Leslie, Paradis, Gropper, Reeves, & Kitto,

2014; Redding, 2005). Creswell (2008) indicated this data analysis step entails an

environmental account of observable facts organized into categories. Quotes from

participants were used to capture the fundamental nature of respondents’ perceptions and

help describe and identify themes of the math assessment and its policy process.

Thematic Analysis

The use of thematic analysis was used to determine the opinions of teachers and

administrators using the math assessment policy in the local district (Halverson, Graham,

Spring, Drysdale, & Henrie, 2014; McLean & Griffiths, 2013; Tate, Elliam, & Kirchoff,

2009). This analysis provided insight into the themes surrounding policy process theory

and what respondents considered important to discuss regarding issues of policy

formation, implementation, and evaluation within the theoretical framework. As part of

the thematic analysis the context revealed the reasons administrators and teachers found

the assessment policy a useful tool to inform instruction. Conversely, part of the thematic

analysis also found that administrators and teachers found parts of the math assessment

policy problematic as a tool to inform instruction.

Coding for Themes

Coding used in the data analysis process included searching for patterns from

participant responses and coding for themes. Thematic coding helped to link together

common themes that occurred among interviews to show the connections, similarities,

and differences between participant experiences (Attride-Stirling, 2001; Denzin &

Lincoln, 2011; Rubin & Rubin, 2012). The following steps were used in the coding

process:

1. Define coding categories.

2. Assign code labels to the categories.

3. Classify relevant information into the categories.

4. Test the reliability of the coding.

5. Measure the reliability of the coding.

6. Locate the sources of unreliability in the coding. (Raymond, 1992)

Using these steps, specific theme codes were determined after interviews were

conducted. The major coding categories align with the theoretical framework of policy

process theory including problem identification, agenda setting, policy formation, policy

implementation, and policy analysis (Stone, 2011; Vesely, 2012; Weimer, & Vining,

2005; Yanow, 2000). Coding was explored with subcategories from these major themes.

Originally, themes emerged from the 10 findings mentioned in the previous

section on outcomes. These themes were generated from both the math teachers’ and

administrators’ data analysis. After continuing the color-coding process and getting

feedback from my committee chair, I combined the data into five themes. The themes

identified were: (a) Uncertainty in the ranks, (b) Sharing power, (c) Collaborating among

the mathematics disciplines, (d) Policy evolution, and (e) Policy outcomes.

Uncertainty in the ranks referred to the different perceptions teachers and

administrators had regarding the math assessment policy. Administrators in every case

expressed a belief that all applicable teachers were using the policy. Conversely, in each

case for the participants that were teachers none of them expressed a belief that all

teachers were currently or had in the past used the math assessment policy.

Sharing power related to how teachers were accessed for their buy-in with the

policy. Both teachers and administrators asserted that top-down administration was used

to instill the policy in the beginning. Administrators expressed a belief that teachers

currently own the policy and that it is no longer simply an administrative requirement.

Teachers expressed a belief that the policy was no longer widely used by all schools in

the district.

Collaborating among the mathematics disciplines was associated with the fact that

several participants indicated that the math assessment policy resulted in a change in

practice. That change in practice was that teachers had to write assessments in group

leading to collaboration. The change in practice also related to teachers helping each

other with student achievement when teachers found they were weak in a standard

another teacher that was stronger in that standard could assist that teacher in improving

instruction in that area.

Policy evolution was connected to sharing power in that the policy started as a

top-down mandate and then over time moved to teachers owning their students’

assessment outcomes so that they can determine what to teach. Policy evolution was also

linked to how assessments were revised, edited, and changed from the onset of the policy

to present day due to changing from state standards to common core state standards.

Policy outcomes showed that to some extent, although difficult to determine, the

policy improved student achievement and helped to close achievement gaps. Participants

expressed a belief that the policy either in the past or at the time of data collection met the

outcomes of improving student achievement. The participants also expressed that since

the standards have changed the degree to which this can be determined is complex.

Each of the five themes was connected to one of the three research questions:

1. What organizational context led to the creation of the math assessment policy?

How was the math assessment policy implemented? Theme five provided

relevant data for this question.

2. How was math instruction conducted before and after implementation of the

math assessment policy? Theme five provided relevant for this question.

3. What are the perceived outcomes associated with the implementation of the

math assessment policy and what is the basis for the perceived outcomes?

Themes one through four provided relevant data for the final question.

The participants agreed that math assessment policy originated from a need to make sure

all standards were being taught in a consistent manner K-12. Participants also discussed a

more collaborative nature amongst educators, adapting assessments to new standards, and

informing instruction as the means by which the math assessment policy changed teacher

instruction over time.

Perspective and Story

Gubrium and Holstein (2009) urged researchers go beyond the data collected

from interview transcripts and consider the context from which participants tell their

stories. For instance, the way a subordinate employee might tell her story may be very

different than the way a supervisor might tell the same story, even though they have

experienced the same assessment policy. Perspectives differ from participant to

participant and those perspectives were considered as part of the analysis because all

participants had their own set of circumstances that affected the telling of the same

experience.

Procedures for Dealing With Discrepant Cases

To increase the validity of the proposed policy evaluation, it was essential to

determine during analysis where discrepant data were present (Cohen, Manion, &

Morrison, 2000; Maxwell, 2004; Merriam, 2009). Asking participants to verify the

interpretation of their interview was the first step in identifying possible discrepant cases

within the data. When discrepancy arose it was thoroughly and accurately reported.

Eliminating bias and statements that may lead the reader allows the reader to make his

own conclusions about discrepant cases. When data differed from the major themes of the

study discrepant cases or rival explanations have been include with fidelity in the data

analysis.

Rival explanations are data that are contrary to findings from similar sources of

data (Patton, 2001; Yilmaz, 2013; Yin, 2011). It was important to examine data that do

not coincide with major findings because doing so increases the credibility of the study

by demonstrating that measures were taken to search for alternative ways of seeing and

understanding the environment in which data were collected. Searching for rival

explanations also aided theory development during data analysis because these

explanations, along with mainstream explanations, offered a full description of how and

why events came into being.

Discrepant Data

Discrepant data were also found that demonstrates a lack of consistency in

maintaining educational policy within the district and negative aspects of the policy.

Further discrepant data showed that the policy poorly fits schools that do not have more

than one math teacher per grade level, and a complete lack of uniform math curriculum

existed in the district prior to the math assessment policy.

Evidence supporting a lack of consistency maintaining the policy can be found in

Teacher T1’s statement, “I mean, I guess I didn’t use the actual exit assessment, but I

used ideas that it’s built upon.” Teacher T1 went on to say, “Well, I think there should be

more clarity about it. From what I know I’m not sure anyone does an exit assessment.”

Teacher T1 also stated, “I never used an exit assessment as required.” Teacher T2 added

to this finding, “When we moved away from [the policy] I think it definitely hurt, so to

speak, student achievement.”

Evidence displaying negative aspects of the policy can be found in Teacher T1’s

statement, “The problem I see is that there are all of these overlapping assessments so I

think maybe there is too many, and it would be great if we could have one thing that was

integrated all together.” Administrator A3 reiterated this idea, “Our state assessment,

NWEA, classroom, common formative assessments…we’ve just got too much assessing

going on.”

Evidence that a lack of fit for some schools that do not have more than one math

teacher per grade level was seen in Administrator A3’s comments:

I think [the senior administrator] having a very high school perspective and

multiple math teachers had a very clear vision of wanting to have this math be

taught and the same assessment be taught to help improve math instruction and

math learning overall. And it applied differently here because we had one teacher.

We didn’t have to do a lot of collaboration and things that I think you do if you’re

in a bigger school.

Finally, evidence that prior to the math assessment policy, there existed no

uniform math curriculum comes from Administrator A4 who stated, “[Prior to the math

assessment policy] we’d never had a strong curriculum in our district; we’d never had

any uniformity in any of our classes. It was a complete free-for-all really.”

Data that differed most dramatically from the major themes identified in the

previous sections came from Administrators A5, A4, and A3. Administrator A5 was the

only participant to request the district limit its changes when answering the question,

“What future recommendations do you have, if any, for the math assessment policy and

why do you make those recommendations?” Adminstrator A5 stated, “I recommend that

we stay with the assessments rather than jumping into something totally brand new

because it looks intriguing.”

Administrator A4 pointed out the difficulties inherent in implementing the policy

when asked the question, “Do you have anything to add?”

That has moved us, but I think accountability has also moved us. It wasn’t a fun

move, and I don’t like the way the move happened, you know having to be held

dangling in front of the public “boo math teachers,” you know, “boo high school.”

It was hard; it was really hard.

Administrator A4 discussed the extent to which math educators were

disconnected from the standards within the district prior to implementation of the policy

when asked, “Do you have anything to add?”

I’ll never forget [a junior administrator] handed me, he just brought me a box

filled with [a teacher] stuff ‘cause [a teacher] was moving to counselor, and I was

taking [a teacher]’s position so he brought me this box of like lesson plans and

books, and I said ‘alright.’ ‘These are the classes you’re teaching,’ and I’m like

‘what do I need to teach?’ and he’s like, uh, ‘just whatever,’ and I was like

‘really?’ I mean, it was no one ever, I mean I could have taught two chapters and

no one would have known. All semester I could have done a whole semester and

taught two chapters and no one would have known.

Administrator A3 discussed how the policy really didn’t fit the school where he

worked:

I think [the senior administrator] having a very high school perspective and

multiple math teachers had a very clear vision of wanting to have this math be

taught and the same assessment be taught to help improve math instruction and

math learning overall. And it applied differently here because we had one teacher;

we didn’t have to do a lot of the collaborations and things that I think you do if

you’re in a bigger school or in a dynamic system where you have multiple

teachers teaching multiple or the same things.

Evidence of Quality

To ensure quality data collection, evidence of triangulation, peer debriefing, and

member checking were used in this study (Barusch, Gringeii, & George, 2011; Cooper,

Brandon, & Lindberg, 1998; Creswell, 2008; Marshall & Rossman, 2011; Nguyen, 2008;

Spall, 1998). Triangulation consists of using different sources of information to increase

the validity of a study (Guion, Diehl, & McDonald, 2011; Rapport et al., 2013; Spall,

1998). Triangulation was guaranteed by corroborating evidence from interviewing eight

different sources. Within these eight sources, two types of information were collected:

one from three teachers a second from five administrators.

Peer debriefing consists of having a trusted colleague read the researcher’s

analysis of the data and interpretations and check the validity of the researcher’s

paradigm in relation to the data analysis or call into question the way in which data were

interpreted (Given, 2008; Wertz, Charmaz, & McMullen, 2011). Peer debriefing has

further established triangulation by utilizing a colleague outside of the math assessment

policy to investigate the method of data collection, interpretations of the researcher, and

the conclusions that stemmed from data analysis. I interpreted the raw data after each

interview was conducted. The peer debriefer was given interpreted data to read. After the

peer debriefer read the interpreted data, he confirmed the data were plausible and

credible, “This all looks good. This is an interesting study.”

Finally, member checking synonymous with informant feedback and respondent

or participant validation improves the accuracy, credibility, and transferability or external

validity of a study (Houghton, Casey, Shaw, & Murphy, 2013; Krefting, 1991; Tracy,

2010). According to Tanggaard (2008) member checking can be done during the

interview process, at the conclusion of the study, or both. Specific to this study, member

checking occurred after each participant interview was transcribed. This served as the

final indicator of quality. Participants were asked to validate the findings of the data

analysis after transcription or correct the transcription. Participants all stated that

transcriptions were reliable, accurate, and valid and did not request any changes to the

transcription.

Limitations in Addressing the Problem

The evaluation of the math assessment policy in a local district in the southwest

has its limitations. The limitations of the evaluation in addressing the problem include the

number of participants studied. All data collected were restricted to interview data, and

the theoretical frameworks used to align the study possess their own limitations. These

limitations are outlined below.

Reliance on Interview Data

The initial limitation in evaluating the math assessment policy was the fact that

when using interview data personal narratives introduce bias and retelling limits findings

usefulness and value, hence the importance of peer debriefing and member checking.

Creswell (2008) suggested that “participants misinterpreting complex questions” have led

to failures in supplying the essential findings or information needed to allow for informed

recommendations. Bardach (2011) also agreed that improper semantics can weaken a

policy evaluation. Adams and Cox (2008) went on to state that assuming an evaluation is

the answer to organizational or academic problems lessens the worth of the evaluation.

Hartley (2009) indicated that policy evaluations are not intended to be the solution of

their own accord, but rather to determine the outcomes and future possible directions.

Accordingly, the intent of this policy evaluation was to inform and demonstrate personal

narratives to assist with suggestions for modifications of the math assessment policy.

Number of Participants

One particular limitation of this study that was pointed out to me by potential

participants was that it was conducted near the end of a school year. Two potential

participants declined to be part of the policy evaluation citing too many obligations

during the end of the year, thus limiting the number of participants in the study. A second

limitation was that participants may not have understood what the policy evaluation was

seeking to understand and what kind of a commitment participating involved. In order for

the policy evaluation to benefit the local district, it was important that all members have a

thorough understanding of what was involved. It may have been the case that both the

senior administrator and I misjudged the population involved in using the math

assessment policy and, therefore, participants who did not apply to the math assessment

policy may have been solicited. The next potential limitation was that of the 12

participants solicited for inclusion in the math assessment policy evaluation only eight

agreed to participate. The final limitation was that many participants did not understand

the difference between a program, procedure, and a policy. This misunderstanding may

have also lead to the self-selected exclusion of two other participants from the data

collection process.

Researcher Bias

A teacher in the same district conducting a policy evaluation may have had an

unrealized effect on both the administrators and teachers who participated. To minimize

this effect, teachers within the researcher’s own department were avoided, which may

have created a further limitation by lowering the number of potential participants.

Keeping participants confidential further assisted in minimizing this limitation, but the

limitation still exists.

The policy evaluation and its design are limited. The math assessment policy

evaluation and the data collected were based on a small, rural district in the southwest.

This evaluation may be difficult for large, urban school districts to replicate because the

time of data collection would need to be expanded. There are many factors that may limit

the findings to other settings. The first limitation identified during this study was the

small sample size, which consisted of eight participants. The second limitation was that

all participants worked in the same small rural district. The eight-person sample was

adequate in providing answers to the interview and research questions. Finally, only three

teachers and five administrators provided data for the math assessment policy evaluation.

Data Analysis Results

Interview data were collected from 3 K-12 math teachers and 5 administrators

about the process that led to a local math assessment policy. I recorded the interviews and

transcribed the recordings by hand. Documentary data from local media resources were

also reviewed and are cited within the following findings.

Integral pieces of Bardach’s (2008, 2011) policy analysis are the techniques for

evaluating effectiveness of communication between policy makers and analysts. Policy

descriptions must be written in the vernacular and clear enough that people in and out of

academia will be able to understand their intent without misinterpretation or different

factions of parties using the policies having wide variation in understanding of how to use

the policy (National Education Policy Center, 2014). Evidence of a gap in practice was

discovered when senior administration stated, “The math assessment policy has never

been evaluated since its creation” (senior administrator, personal communication, 2012).

The framework for interpreting the results of this study was the evaluation of

policy process. The evaluation of policy process includes both understanding policy

analysis and policy process. Sabatier’s (2007) five characteristics of the policy process

framework were used to frame the initial stages of this study prior to data collection.

Sabatier (2007) notes the five components of the policy process: agenda setting, policy

formation, legitimation, implementation, and evaluation. Bardach’s (2008, 2011) policy

analysis framework helped to complete the evaluative process, as it provided guidelines

for judging the fitness of a policy. According to Bardach, there are eight components to

evaluating policy: define the problem, assemble evidence, construct alternatives, select

criteria, project outcomes, confront tradeoffs, make decisions, and story-telling.

Most components of Sabatier’s (2007) policy process theories and Bardach’s

(2011) eightfold path to policy analysis were found in the setting. However, in a few

cases, components of the theoretical frameworks had to be addressed through district,

community, or state documentary data because they were not revealed during qualitative

data collection. During the process of advancing the math assessment policy through data

collection, analysis, and interpretation, it appears that agenda setting, policy formation,

and legitimation all occurred concurrently via an authoritarian mandate from upper

leadership. This authoritarian process may be seen as a marginal weakness of the policy.

Teacher participants in this study expressed a belief that there may have been a lack of

teacher adherence to the policy as the policy continued over time. Administrator A2

mentioned that teacher complaints at the onset of the policy were not uncommon.

Conversely, a strength of the policy was increased teacher collaboration when prior to the

policy, there had been very little or none at all. Some participants noted an increase in

cohesiveness among similar content areas and throughout the grade levels. This policy

evaluation concludes the evaluative stage of Sabatier’s policy process.

Bardach’s (2011) eightfold path for policy analysis can be seen in this case study

narrative. During the progression of evaluating the math assessment policy, the problem

was identified from the fact that the policy had never before been evaluated. Evidence

was collected through interview data from three teachers and five administrators.

Alternatives to the policy evaluation were considered. These included a program

evaluation and longitudinal analysis. The criteria for evaluating the math assessment

policy were determined through the use of three research questions involving origins,

development, and results of the policy. Results of the policy demonstrated that it had met

its intended goals to an extent as perceived by interview participants, an authoritarian

means of transmitting the policy was used, and collaborative practices among teacher

practitioners increased. Work still exists for the district regarding the math assessment

policy. The district must confront tradeoffs in terms of the policy’s effect on excessive

testing versus the positive collaboration provided by the policy. The local district may

have to decide which of these tradeoffs is of greater importance or whether they are of

equal importance. If discovered they are equally important then the tradeoff of decreased

instructional time resulting from more testing will have to be accepted by district

personnel. The concluding step of the eightfold path, policy evaluation, is addressed in

the following section.

Findings: Case Narrative

This case study narrative examines how a school district implemented a math

assessment policy to shape and inform math curriculum and instruction in order to bolster

student achievement on state mandated exams. Three teachers and five administrators

were interviewed to assess their experience and perceptions of implementing common

assessments. Throughout the formation, implementation and execution of the policy,

some problems arose, but many of the participants also expressed numerous positive

aspects of the policy. The case study narrative demonstrates how the state requirements

for student achievement are often at odds with the teachers’ philosophies and community

values.

The purpose of the case study narrative was to tell the story of the people

involved in the local district’s math assessment policy. Case study narratives combine

two types of qualitative designs: the case study and narrative research. Cases might be

people, school districts, companies, actions, or measures (Creswell, 2008; Fritz, 2008;

Merriam, 2009). In this study, the case at hand was the evaluation of policy process that

led to the math assessment policy in a local southwestern school district. In order to

inform stakeholders about the sentiments of participants and community members

regarding the policy, the case study narrative was critical. The presence of both empirical

evidence and opinions about the math assessment policy provided a holistic

understanding of the policy. Case study narratives may consist of varied data including

intercultural development inventories, review of archival and documentary data,

observations, and interviews. This case study narrative’s data was collected through

participant interviews, documentary data from media outlets, and archival data from the

state department of education.

The critical case narrative was chosen to uncover the why and how an event came

into being (Creswell, 2008; Fritz, 2008; Holley, 2006). The evaluation of the math

assessment policy process was a critical case that sought to understand how and why the

math assessment policy came about and evolved over time. It also seeks to show what the

results and reactions of the larger community were in response to the math assessment

policy. Understanding how the policy changed chronologically may help stakeholders

appreciate the processes needed to develop new policies and improve the current ones.

The narrative in many studies can vary in purpose (Fritz, 2009; Heber, 2011; Yin, 2009).

Those purposes may consist of constructing individual or group identities, persuade the

reader, rationalize the meaning behind an argument, teach a lesson, and make sense of

events. In this study the narrative offered both the perspectives of teachers and

administrators regarding the math assessment policy with the intention of determining

how the policy came to be, was used, and impacted teachers and principals.

There are two theoretical frameworks that are used to guide this case study

narrative. The first framework was Sabatier’s (2007) policy process theories. The second

framework was Bardach’s (2008, 2011) eightfold path that guides policy evaluation. Both

frameworks were essential to the project study because in order to complete evaluation of

policy process the policy must be analyzed. Furthermore, the case study narrative

evaluated the process that was followed when the math assessment policy in the local

district was established, implemented, developed, and how it affected the local

community’s attitudes toward the district. This case study narrative further filled a gap in

practice in which the math assessment policy had not been evaluated prior to

development of the project study.

Policy Evaluation Framework

Sabatier’s (2007) stages of the policy process included agenda setting, policy

formulation, policy adoption, policy implementation, and policy evaluation. The stages of

the policy evaluation process and the steps to policy analysis were embedded in the above

findings; and to emphasize clarity, their presence and significance were illustrated in the

following sections.

Agenda Setting

According to Green-Pedersen and Walgrave (2014) agenda setting occurs when

the “problem” that needs addressing is put on the formal policy agenda of issues to be

addressed by organization personnel. There is no indication from the data that district

personnel were convened to discuss an agenda of issues prior to the mandate that the

policy be implemented. The only agenda setting that appears to have occurred was with

one isolated individual, the senior administrator in office at the time the policy began.

The district’s situation in the spring of 2005 was that the previous three years

(2002, 2003, and 2004) of state assessment data showed that students’ state math

assessment scores were on the decline. The assistant superintendent at this time saw a

need to correct this problem by mandating that teachers write and administer a series of

common assessments and exit assessments throughout the district K-12. These

assessments would help create uniformity among grade levels in mathematics and inform

the mathematics teachers of the following school years of the standards their students

needed to learn. However, teachers and administrators were not unified on the common

assessments. Administrator A2 indicated this lack of unity stemmed from teachers’

beliefs that adding new assessments, as well as writing them, was just one more thing to

add to their to do list: “many of the perceptions were this is something I have to do now.”

The purpose of the mandate of the math assessment policy was to improve student

achievement in mathematics based on data presented to the district by the state

department of education (Walter, 2004). This was further supported by the data collected

through Administrator A4. This participant was in the district when the policy was

formulated and was particularly vocal in stating that numerous teachers in the same

subject area could be teaching completely different content in the course of a school year.

Furthermore, this participant stated that teachers were not teaching to the state standards

as a whole. Lack of uniformity and failure to teach the state standards were thought to be

the cause of poor state test scores. Moreover, the math assessment policy was thought to

be the tool that would create uniformity and guarantee teachers taught the state standards

thereby improving state assessment results.

Six administrators and twelve math teachers were tasked with identifying the

standards students needed to learn to improve their mathematics achievement and write

both formative and summative assessments that matched these standards. From the onset,

difficulties arose with selecting the content to use to write these assessments. Teacher T3

stated that the teachers in collaborative meetings were required to identify standards and

write assessments that were aligned to textbooks rather than the state standards: “Our

math assessment was created from the book we were using at the time.”

Furthermore, problems were also evident in the absence of consistency in

instruction and content, a lack of adequate assessments, instruction that was not being

informed, and immediate feedback. Administrator A1 stated, “They wanted that policy in

place so that those kinds of teacher practices were consistent across all grade levels.” The

types of practices Administrator A1 referred to were uniformity in content and

instruction. This statement implies that there was a problem recognized within the district

in which instructional and content consistency did not exist prior to the math assessment

policy. Further, data from Administrator A1 indicates that administrators and the school

board also recognized a problem in which a lack of accountability existed prior to the

policy. Administrator A1 explains, “Expectations of administrators and probably the

board of education supporting those expectations that we create common assessments and

used exit exams.” Teacher T3 also stated, “I believe [the senior administrator] thought we

needed more data to, you know, pass on standards to the next teachers the following

year.”

It was clear teachers needed data to inform their instruction. Teacher T1

suggested that teachers needed to inform their instruction: “Knowing what the kids need

to know, creating a way of assessing that, and then actually assessing how they did as a

way to inform, evaluate your instruction.” Administrator A3 also stated that

Part of it was just to make sure that we knew what we were doing. That the kids

were really learning and growing and that we were actually able to help them be

successful by the end of the year or so. I think the biggest motivation was for us to

guide our instruction. The second piece was, of course, the accountability to make

sure that all teachers were indeed teaching what needs to be taught.

Additionally, not having immediate feedback was an issue because teachers could

not inform their instruction in a timely manner for the students they were currently

teaching. Teacher T2 stated that prior to the math assessment policy, absence of

immediate feedback was a problem: “We wanted to make sure we were teaching to the

standards and that our kids were growing throughout the year in what they were actually

taught not just our state assessments so that we could get immediate feedback.”

Administrators were also receiving pushback from the media, superintendent, and

school board. Documentary evidence from local media sources also point to a concern at

the district level. A 2004 newspaper from the town of the district’s locale reported that

“in general, said [the senior administrator], ‘positive trends are noted in reading scores,

especially in the fourth, ninth and 10th grades. Writing scores indicate a stable trend line.

But math scores,’ he said, ‘remain a concern for district officials’” (Walter, 2004).

Policy Formulation

The policy formulation period was a semester long period in which central office

had one meeting per school with teachers to tell them that the math assessment policy

needed to occur. Teachers were not given the option to opt out of the math assessment

policy, and there was little discussion devoted to other options between teachers and

administrators. While there seemed to be some unity in the need for common assessment,

as demonstrated in the data analyzed for this study, teachers and administrators largely

felt that the process of policy formation was exclusionary. By all accounts within the data

collected, formulating the math assessment policy was an event that was directed through

top-down management.

Teacher T3 stated, “I had no part of creating the math assessment policy.”

Teacher T2 said, “I didn’t have anything to do with actually creating the policy, but

following through with and creating the assessments that were outlined by it.”

Administrator A1 declared, “I don’t believe I had a role creating the policy.” In reference

to who formulated the policy A1 asserted, “I would say from our assistant superintendent

at the time, who was the director of curriculum and instruction.” Teacher T1 also said

they had, “none or no role” in formulating the policy. Administrator A4 declared, “I had

very little involvement in creating the policy.” These statements all provide evidence that

the math assessment policy’s agenda and formulation were set into play without many of

the district’s stakeholders having taken part in these two portions of the policy process.

Policy Adoption and Implementation

Participants recalled that the policy was adopted in the spring of 2005 as a

directive from central office. Administrators A4 and A5 both cited a professional

development event the fall of 2003 by Richard and Rebecca Dufour as being the catalyst

that resulted in adoption of the math assessment policy. According to administrators A4

and A5 the assistant superintendent in charge of instruction at the time followed Dufour

and Eaker’s (1998) data driven models to develop the policy. The adoption process was

clearly authoritative and given little discussion so participants had little recollection

regarding adoption other than when it occurred and where it came from.

By contrast, participants had many recollections of how they were required to

implement the math assessment policy. Many teachers and administrators expressed

feelings about policy adoption and implementation that were similar to how they felt

about policy formulation. They felt that they were not included in the formation process,

but were expected to fully participate in implementation. Administrator A4 said, “I was

employed by the district, and I was doing the assessment policy,” thus indicating that A4

saw the adoption and implementation of the math assessment policy as something that

they were now responsible for as an employee of the district. The problem with this was

that there was no discussion about what to do and why to do it, just that it needed to be

done. Data collected from all participants shows that teachers understood that student test

scores needed to improve, but no discussion of whether there was an alternative to the

math assessment policy was never mentioned by participants.

Other participants indicated the part they played in implementing the policy

mainly involved writing and administering the assessments the policy required rather

than helping to formulation the policy and what it contained. Administrator A1 declared,

“We created common assessments for each unit of instruction. We also had an exit exam

which was a common assessment for all students at the end of the year.” Administrators

A2 averred:

We worked really hard with each team to look at creating an end of the year

assessment that would help the following year’s teacher get a better sense of what

type of learner and how well the student was doing with math and then we did

some backwards planning from that and formative assessments to ensure that we

were not using it as an autopsy but as a formative assessment to see if we needed

to determine if we needed to change instruction.

Teacher T2 reiterated the same sentiment Administrator A2 had expressed, “We

definitely did a backwards design. We created our common assessments which tied to the

exit assessments and then did lesson planning from there, so that we knew each

requirement was specifically taught.” Teacher T2 liked the math assessment policy and

writing the assessments for the policy. She felt that the policy strongly informed her

instruction.

Policy Evaluation

The final stage of the policy process, policy evaluation, had not preceded this

study. The goals for this policy evaluation required an in-depth inquiry into reasons for

the local district’s math assessment policy. Inquiry included the discovery of successes

and downsides from administrators and teachers about the math assessment policy as well

as successes of the policy. This policy evaluation was relevant because no prior

evaluation conducted regarding the policy had occurred prior to the onset of this study

(senior administrator, personal communication, 2012). All participants expressed a belief

in the efficacy of the math assessment policy. According to all interviewed participants,

with the exception of Teacher T1, the policy met its outcomes in terms of improved

student math achievement, improved teacher collaboration, and informed instruction. The

complexity of new Common Core State Standards and the participants not being in

possession of numerical state test data when interviewed makes it difficult to pinpoint to

what extent these outcomes were met.

Policy Process Framework

Bardach’s eightfold path for policy analysis includes defining the problem,

assembling evidence, constructing alternatives, selecting the criteria, projecting

outcomes, confronting tradeoffs, deciding, and telling the story. The following sections

outline the path of policy analysis for the local math assessment policy.

Defining the Problem

The problem identified by the math assessment policy was lack of student

achievement in math. In the spring of 2005 the local district found that state assessment

data showed students’ state math assessment scores were waning. The assistant

superintendent at this time recognized the district could use a means of remedying this

problem by issuing a directive that a series of common assessments and exit assessments

be written and administered throughout the district K-12. These assessments would

generate consistency within grade levels in mathematics and apprise the following school

years instructional leaders about what math standards students were lacking. Therefore,

the problem was defined as improving student achievement in mathematics based on the

state level data presented to the district by the State Department of Education.

Furthermore, evidence of the problem was thought to have been rooted in the

absence of consistency in instruction and content, a lack of adequate assessments,

absence of informed instruction, and state feedback that took an excessive amount of time

to reach the district’s schools. Administrator A1 stated, “They wanted that policy in place

so that those kinds of teacher practices were consistent across all grade levels.” This

statement implied that there was a problem recognized within the district in which

instructional consistency and use of state standards among teachers did not exist prior to

the math assessment policy. Additional data from Administrator A1 indicated that

stakeholders also recognized a problem with student math achievement being minimally

evaluated or not evaluated at all: “Expectations of administrators and probably the board

of education supporting those expectations that we create common assessments and used

exit exams.” Teacher T3 also stated, “I believe [the senior administrator] thought we

needed more data to, you know, pass on standards to the next teachers the following

year.”

Teacher T1 suggested that prior to the math assessment policy teachers had no

means of informing their instruction in order to improve instruction and target

intervention: “Knowing what the kids need to know, creating a way of assessing that and

then actually assessing how they did as a way to inform, evaluate your instruction.”

Administrator A3 also stated that:

Part of it was just to make sure that we knew what we were doing. That the kids

were really learning and growing and that we were actually able to help them be

successful by the end of the year or so. I think the biggest motivation was for us to

guide our instruction. The second piece was of course the accountability to make

sure that all teachers were indeed teaching what needs to be taught.

Teacher T2 made an interesting statement about the relationship between what

teachers taught and what was presented on the standardized state assessment: “We

wanted to make sure we were teaching to the standards and that our kids were growing

throughout the year in what they were actually taught, not just our state assessments so

that we could get immediate feedback.” This implied that what was being tested on the

state assessment was not necessarily what teachers were teaching in their classrooms. The

lack of consistency among teachers in terms of using the standards pointed to another

problem that likely affected student achievement declines in math at the time.

Assembling Evidence

Assembling evidence was part of the process of evaluating the math assessment

policy. Evidence in this study was assembled in three ways. First, evidence that the

policy needed to be evaluated was confirmed by the senior administrator (personal

communication, 2012). Next, a literature review was conducted establishing evidence

that policy evaluation was appropriate for the local educational setting. Finally, data were

collected from participants, using a case study research design, regarding the math

assessment policy and its outcomes. Evidence assembled that describes the top-down

nature of the policy’s implementation was perhaps most striking. Teacher T3 stated, “I

had no part of creating the math assessment policy.” Teacher T2 said, “I didn’t have

anything to do with actually creating the policy, but following through with and creating

the assessments that were outlined by it.” Administrator A1 declared, “I don’t believe I

had a role creating the policy.” In reference to who formulated the policy A1 asserted, “I

would say from our assistant superintendent at the time who was the director of

curriculum and instruction.” Teacher T1 also said they had, “none or no role” in

formulating the policy. Administrator A4 declared, “I had very little involvement in

creating the policy.” These statements all provide evidence that the math assessment

policy’s agenda and formulation were set into play without many of the district’s

stakeholders having taken part in these two portions of the policy process.

Constructing Alternatives

Alternatives to the policy evaluation that were considered were process

monitoring, program evaluation, and organizational review. These alternatives were

ultimately not considered because a short time frame was required for collecting data in

this study, these methods do not determine the math assessment policy’s effectiveness,

and the wellbeing of participants was not a central focus of this study. Therefore, the

criterion selected for this study, evaluation of the math assessment policy, was selected

substantiating the relevance of the theoretical framework. When executing one

alternative, some positive outcome could be traded for a negative out come and vice

versa. With this in mind decisions will have to be made that determine what tradeoffs will

yield the greatest benefit (Bardach, 2011; Patashnik, 2014).

The local school district had several options regarding alternatives to the math

assessment policy. The first option was to maintain the status quo and continue to have

teachers administer, write, revise, and analyze the data from classroom math assessment.

Another option that had been offered by local community members, as illustrated by local

media outlets, was to completely ignore federal and state guidelines for assessment and

evaluation of public school teachers and quit assessing students altogether (Hudson, B.,

2014a). This alternative could be disastrous because it would mean the district would be

operating on a budget dependent solely on corporate donors. There was little evidence

that the local community and business entities have the funding and interest to support a

budget the size of the local district’s $11.7 million (Fincher, 2014; Hudson, B., 2014b).

A third alternative was to wait for the current political climate in education to run

its course and see what new policies are mandated by state and federal governments.

Finally, teachers and administrators interviewed in this study recommended the district

eliminate extra testing and use only the required state mandated assessments. This was a

simple idea, but confounded by the language in the current educator effectiveness bill that

has been enacted by the state legislature. The educator effectiveness bill specifies that

educators must have a body of assessment evidence that includes classroom assessment

data.

From the data collected, it was not possible to determine whether viable

alternatives to the math assessment policy were considered at or prior to the time the

policy was formulated. In fact it seems from the data collected that the district’s senior

administrators saw the math assessment policy as the only option to improve the district’s

declining test scores. Moreover, the question, “Were alternatives to the math assessment

policy looked at prior to implementing the policy?” was never explicitly asked during

data collection. However, had there been investigation of alternative practices other than

the math assessment policy, this still may have been mentioned by participants when they

were asked the following three questions during the interview process:

1. What forces were behind the policy during its inception?

2. Why was the policy created?

3. Who did the policy originate from?

The alternatives sought or the lack thereof at the time of the math assessment policy’s

creation cannot be ascertained within this study. Future research should be conducted in

order to tease this out. However, alternatives that can be pursued in the future have been

outlined herein.

Projecting Outcomes

Future policy directions were determined by analyzing policy outcomes thus far.

Some participants discussed an improvement in student achievement including gaps in

student achievement that had been an issue in the past. Overall, participants were fairly

general about their perceptions that the math assessment policy had led to increased

student achievement in math rather than citing state assessment data. Administrator A1

indicated that student achievement was positively affected:

One benefit that I see for students is that it narrowed the focus of what teachers

were actually instructing on the topics, I guess you could say. And so just in doing

a student would tend to be more successful instead of being taught all these

scattered random acts of great things that don’t necessarily relate. So, I think just

by having the common assessments and the exit exams clarified narrowed the

teaching focus, and that certainly would help to close learning gaps. But like I

said, I can’t give you stats or data on that.

State assessment data showed mixed results regarding math achievement in the

local district as demonstrated through SchoolView (2015) assessment reports. Data stored

in the state’s archival data base showed that from 2007 to 2014 the district’s elementary

school performance declined with 52% of students proficient in mathematics and ended

with 41% proficient. During this same period of time, the district’s middle school began

with 39.75% of students proficient in math and ended with 34.58%. The only school

displaying achievement gains in the state archives was the district’s high school

beginning with 22.5% proficient in math and ending with 34.33% proficient. It should be

noted that the state assessment tool and standards changed significantly from 2011 to

2014, which may have played a part in student achievement declines.

Increased teacher collaboration. Participants suggested that the policy has been

the vehicle for a more collaborative culture (Garrett, 2012; Hudson, 2013; Kitagawa,

2011; Teague & Anfara, 2012; Wilhelm, 2009). This finding suggests that future

continuation of the math assessment policy would continue this trend. This finding was

expressed most articulately by Administrator A1 and Teacher T2. Administrator A1

emphasized that collaboration was brought about by the policy:

I think that the common assessments become the product that pushed teachers to

move in the direction that they need to move. You know, if you expect them to sit

down around the table and create an assessment together, which is what we did,

then we’re all on the same page looking at the same standards expecting the

students to know the same things working together to get that done, and a lot of

teachers had to change their ways in order to be a part of that collaboration and

have that product in the end.

Teacher T2 confirmed this.

I really feel like it did a good job of when we got together as a fifth grade team

(because when we were using these assessments we all taught math, I wasn’t the

only math teacher), and so we would get together we would give a common

assessment. We would enter our data into a template and then we would talk

together about what we did well and see where one class in one standard did

really well and where one class didn’t.

Increased collaboration among teachers was one strong point of the policy that many

participants pointed out during data collection. Administrator A4 discussed the

environment in the local district before creation of the math assessment policy and

pointed out that many teachers at the high school were teaching from different sources

even when teaching the same course. Administrator A4 also revealed that the increased

collaboration that generated through the policy brought this absence of uniformity to an

end.

Informed instruction. Data analysis suggested that informed instruction would

likely continue in the future if the math assessment policy continues. A common practice

nationwide prior to NCLB was teachers addressing only content they wanted to teach

rather than all of the state standards for their particular grade level (Main, 2012; Powers,

2013). This was emphasized in the data by Administrator A1:

I believe the policy was created in order to make sure that teachers were teaching

all expected standards to students and to get feedback on you know how confident

I guess the students were on each of those standards. We used our exit exams

more to decide on what we need to do a better job of teaching next year, then we

didn’t really use the exit exams as much for say placing a student or because I

taught fifth grade, it was really reflecting on our practice and using it to inform

our instruction. And I would say that they wanted that policy in place so that those

kinds of teacher practices were consistent across all grade levels, all departments,

all subjects. That kind of thing.

Administrator A3 supported Administrator A1’s claims emphasizing the consistency in

addressing the standards that resulted from the math assessment policy:

So, it was kind of a measure to make sure that the same math was being taught

and the same level of math was being taught. So, I think it was to improve math

learning and assessment procedures to make sure everyone was being taught the

same thing at the same level.

The math assessment policy also resulted in consistency among teachers of the same

course, as demonstrated by the findings that stemmed from Administrator A4’s interview:

I think it was also we’d never had a strong curriculum in our district; we’d never

had any uniformity in any of our classes. It was a complete free-for-all really. I

mean algebra 2 from two different teachers, it was kind of a recognized as a

district and I think as administrators and I think as teachers that we had a

uniformity issue, so one that is part of that uniformity, but that idea of pushing the

assessment, the exit assessment especially, from the assistant superintendent.

The district continues to see benefits with teachers informing their instruction that came

from the math assessment policy, as indicated by Administrator A5:

We’ve tweaked it into that and you know we’re using that information for, you

know, and I feel like we’ve gotten to a better place looking at and saying we can

look at it and make adjustments. You know, we’ve gotten away from the drilling

down to every little tiny thing on the spreadsheet that was, you know. It was

impossible I think to make any educational decisions looking at it that way. It was

way overwhelming—too much information and there’s too much there to, you

know.

Confronting Tradeoffs

During qualitative data collection, participants mentioned tradeoffs that the

district had not yet confronted. Additional tradeoffs that may need to be confronted by

the local district were also uncovered through analysis of documentary data. Tradeoffs in

policy making were widely defined as the costs incurred when a mandate was carried

forth by an organization (Andrews et al., 2013; Halpern et al., 2013; Rizvi & Lingard,

2013). The tradeoffs for implementing assessment policies in local school districts are

that any assessment policy can be thought of as teaching to the test and local

communities often view this as a negative aspect of public education (Bhattacharyya,

Junot, & Clark, 2013; Rebora, 2012).

One example of the negative community perception regarding assessment policy

in the local district was detailed in a submission to an electronic periodical that targets the

local district’s community members. The opinion piece author wrote that in her belief,

teachers looked down on her because she had no interest in college. This perception was

a result of the overemphasis her teachers gave to testing. She contended and that many

students in rural communities have no interest or need for college and because of this,

should not be tested to a large degree (Hudson, U., 2014). This highlights the lack of

value the larger community the local district resides in sees in higher education, but more

importantly, it demonstrates the frustration and lack of value the rural community has

toward testing, assessment, and policies geared toward assessment.

A second article in the same periodical outlined how the complexity of numerous

education policies have, in the author’s opinion, made the local public schools worse not

better (Hudson, B., 2014a). The only local print news source in the local district’s

community has similar examples of disgruntled community members’ attitudes toward

assessment in the district. Two articles in this publication indicated that testing in the

local district was looked upon negatively and thought to be excessive by many parents

and community members (Feazel, 2004; McQuiggin, 2010).

A national website, Great Schools (2015), collects anonymous data about local

school districts from the public. In reviewing this site there was only one parent comment

that addressed assessment or testing. This comment indicated that the local district was

only interested in high test scores and students that score higher are disciplined more

leniently than students that do not score well on the state assessments, “One child

expelled (low scoring student) one threat ignored (high scoring student) test scores are

what they strive for. 4-6 weeks of practice for the [state assessments].”

This narrative was certainly not only about community perceptions; outside of

assessment, there are many community members that make overall positive comments

about the local district in person; however, these comments could not be found in

searches of local, state, and national media sources. In conclusion, the previous section

seeks to illustrate the battle the school district finds itself in, trying to serve two masters:

(1) state and federal agencies and (2) the local community.

Deciding

In the decision-making stage of Bardach’s policy analysis framework it is the job

of researcher to put themselves in the shoes of the decision-makers. In this case, the

decision makers are both the school board and the senior administration of the local

district’s central office. This school district has many decisions to consider. As previous

evidence demonstrates, the district needs to consider what decisions about future

assessment policy will do to the already fractured relationship it has with the community

at large while at the same time giving consideration to teachers and administrators that

value the results of the current math assessment policy and have worked hard to

implement it over the past 10 years.

From the above evidence it would seem there were two basic decisions to make:

(1) continue the math assessment policy in one form or another at the risk of continued

alienation of the community and families at large, but with funds from state and federal

entities available; (2) abandon the federal regulations and hope the at-risk local economy

can pick up the tab for school funding. These decisions may place the school district in an

uncomfortable position: there is much to lose in making either of these two choices.

Telling the Story

The story of the local school district in the southwest at large was one that

outlines the disconnectedness of the school district and the larger community. On one end

of the spectrum, the district personnel overwhelmingly see the math assessment policy as

a positive advancement of the school district’s students’ math achievement. On the other

end, there was overwhelming evidence that the policy and policies like it are not

supported by numerous and vocal community members. This puts the school district in a

conundrum. On the one hand, the district, in order to procure money to continue and

operate, was beholden to state and federal authorities to follow assessment policy in the

form of state assessments, ACT, and was also required by the teacher evaluation system

to continue the math assessment policy in some form. On the other hand, the same

teacher effectiveness requirements demanded schools work with the larger community to

address its wants and needs. The question remains, can local school districts both serve

their community members and continue to follow federal and state testing requirements?

To reconcile these opposing forces the local district may need to further study the

implications of conducting two contradictory forms of business and determine whether it

can be an institution of integrity.

Findings: Answers to Research Questions

The purpose of this study was to evaluate the math assessment policy of a local

district in the southwest in terms of its outcomes, student achievement, and teacher and

administrator perceptions of the policy. The adherence to the policy by staff, excessive

testing making evaluation of all data problematic, top-down administration of the policy,

consistent uniformity in math standards and teaching, evolution of the policy over time,

current ownership of the policy by district teachers in order to inform their instruction,

changes in standards affecting the nature and use of the policy, and the policy’s

relationship to a collaborative culture were all explored as themes in this evaluation. The

goal was to gain an understanding about how teachers and administrators perceived the

policy, whether the policy has met its goals, and whether or not the policy has improved

student achievement.

Participants interviewed in this study consisted of three math teachers and five

administrators. Among these participants were three middle school math educators, one

of whom had taught kindergarten when the policy was implemented; two middle school

administrators, two elementary principals, and one high school administrator were also

included. Data were collected using semi-structured interview questions that allowed the

teachers and the administrators to voice their perceptions of the math assessment policy

in relation to the research and interview questions.

Interviews were analyzed to uncover eight essential findings that answered the

research questions. Each finding is discussed below.

Research Question 1: What organizational context led to the creation of the math

assessment policy?

a. How was the math assessment policy implemented?

b. What actors were involved with the implementation of the math assessment

policy?

Finding 1. The first finding that addressed research question one was the assistant

superintendent’s top-down approach to policy adoption at the onset. This was alluded to

by teacher T3 in answer to the question “what, if any, was your role in creating the math

assessment policy?” Teacher T3 explains, “I had no part of creating the math assessment

policy.” Administrator T2 expressed this, as well, in answer to the question “do you have

anything else to add?” by saying, “You know, certainly at the beginning it was very top-

down. I’m not sure that it could have been anything else at that time. And so, many of the

perceptions were this is something I have to do now.” Administrator A3 also implied this

in answer to the question, “Who did the policy originate from?” stating, “We’ve always

been fairly collaborative as an administrative team, and so, I would say the admin team,

um high school upper admin, and I don’t know how much the elementary did on this

can’t recall.” Administrator A2 also answered the same question by saying

Well, it was from, I guess it was from [the senior administrator], but it was from

administration, but I mean, I think we’ve always had the SSAP and then we were

doing NWEA, and it just drilled down from that point. I think the biggest push

was when [the senior administrator]. That’s definitely NCLB going down to the

state level with the state test and you know, using the NWEA has helped us really

get that national norm data that we need.

The data from administrator A2 not only indicates a top-down approach from the

district’s administration, but also a direct cause and effect relationship between the math

assessment policy and the No Child Left Behind Act of 2001.

Finding 2. The second finding was that the policy ensured teachers were teaching

all of the math standards consistently throughout the district. This finding was uncovered

in response to the question “Why was the policy created?” Administrator A1 stated,

I believe the policy was created in order to make sure that teachers were teaching

all expected standards to students and to get feedback on, you know, how

confident I guess the students were on each of those standards. We used our exit

exams more to decide on what we need to do a better job of teaching next year,

then we didn’t really use the exit exams as much for, say, placing a student or

because I taught fifth grade it was really reflecting on our practice and using it to

inform our instruction. And I would say that they wanted that policy in place so

that those kinds of teacher practices were consistent across all grade levels, all

departments, all subjects. That kind of thing.

Administrator A3 added to this by stating in answer to the question “Why was the policy

created?”

So, it was kind of a measure to make sure that the same math was being taught

and the same level of math was being taught. So, I think it was to improve math

learning and assessment procedures to make sure everyone was being taught the

same thing at the same level.

Administrator A4 emphasized the importance of the policy based on issues of

instructional and consistency among curriculum and standards when responding to the

question “Who did the policy originate from?”

I think it was also uh we’d never had a strong curriculum in our district, we’d

never had any uniformity in any of our classes. It was a complete free-for-all,

really. I mean Algebra 2 from two different teachers, it was kind of a recognized

as a district and I think as administrators and I think as teachers, that we had a

uniformity issue so one that is part of that uniformity, but that idea of pushing the

assessment the exit assessment especially from the assistant superintendent.

Administrator A3 responded to the question “Why was the policy created?”

I think to improve math instruction and math learning and to make sure that we

knew that was happening. So, it was kind of a measure to make sure that the same

math was being taught and the same level of math was being taught. So, I think it

was to improve math learning and assessment procedures to make sure everyone

was being taught the same thing at the same level.

Finding 3. Finding 3 centered around evolution of the policy over time. Teacher

T3 confirmed this finding in answer to the question “Why was the policy created?”

saying, “the math standards have changed so much and because the math standards have

changed... They’ve evolved, they’ve changed and I know that [the senior administrator]

based on the new standards has created tests to addressed those standards.” Administrator

A5 affirmed regarding the same question,

We’ve tweaked it into that and, you know, we’re using that information for, you

know. And I feel like we’ve gotten to a better place looking at and saying we can

look at it and make adjustments, you know we’ve gotten away from the drilling

down to every little tiny thing on the spreadsheet that was, you know. It was

impossible I think to make any educational decisions looking at it that way. It was

way overwhelming. Too much information and there’s too much there to you

know.

Teacher T3 stated that the policy and teachers’ attitudes about it have changed

dramatically since it first began:

Since it’s gone away, the couple of years that we still had five teachers, there was

huge discrepancies from one classroom to another in their exit assessment kind of

stuff or their state testing. So, I think it really when we moved away from doing it

I think it definitely hurt, so to speak, student achievement because people were

doing their own thing, we weren’t all on the same page.

Teacher T3 also stated in response to the question “Has the math assessment

policy impacted student achievement?”

I’d love to say it’s affected it positively, but once again when you change the

standards, we’re using different standards now. We saw a lot of improvement for

a while with common assessments and now it’s kind of taken a dip again as far as

our results and scores go. I would say probably just the nature of problem solving

more of the nature of the kinds of questions the kids are being asked more than

just the math like facts and things like that. Just how they’re retraining the kids’

minds to be able to figure things out more and stick with it, and I think that’s why

we’re seeing a little bit of a dip because the expectation is higher.

Administrator A2 responded to the same question by stating, “I do believe it has

impacted student achievement, and I think that again the data is a little bit difficult to

tease out because not only have we been doing a transition test to help gauge that, but

we’ve also transitioned into whole new standards and new targets.”

In response to the question “Do you have anything else to add?” Teacher T2

indicated that evolution of the policy accompanied a change in administration:

I feel like and maybe this isn’t where your research is going, so I’ll say it and you

can take it or not. I think, you know, when [the senior administrator] came in, she

was overwhelmed with it and the job that she was doing kind of changed, and I’m

not saying that’s a bad thing. She did some great things for our district, but as far

as curriculum and asking teachers to really look at data, that wasn’t her strength,

and it definitely when it wasn’t an emphasis coming from the admin office I think

people were like, “well we don’t have to do it.” It’s not policy. Then they didn’t

see it as a policy, so people have definitely quit doing it. Or it’s not consistently

being done say across K-12.

Teacher T2’s comments point out the important aspects in interpretive theory for

this policy evaluation. This teacher and the two other teachers that took part in the study

showed that teachers were interpreting the use of the policy differently from

administrators. Administrators who participated in this study did not indicate in any way

that there was a deviation from the policy on the part of teachers. However, teacher T2,

clearly stated above that this was the case. Using thematic analysis, this indicates that

there may be possible issues with communication between teachers and administrators at

the local district that need to be addressed (Bastedo, 2005; Geertz, 1973; Hamman &

Hopson, 2013; Humphrey, 2008; Miller, 2002).

Relationship to the literature. Finding 1 was similar to findings that Rizvi and

Lingard (2011) articulated. The authors found that the sample they studied expressed

fatigue with top-down policies. They also found that the globalization paradigm requires

a bottom-up philosophy and procedures that may be difficult for management in the short

term, but results in more institutional efficiency in the long run. Amaral, Tavares, and

Santos (2013) found that the 1988 University Autonomy Act gave Portuguese

universities a bottom-up policy of autonomy. Policy decisions were made using academic

staff and students with democratic elections of governing bodies. A recent reform

movement in Portugal’s educational system replaces bottom up democracy with top-

down directives. These studies illustrated the constant struggle between administrative

and governing bodies authority in relation to the people that these entities serve.

Finding 2 was consistent with Main (2012) and Powers (2013), who both

contended that policies surrounding the Common Core will ensure that all teachers are

teaching the same content to students regardless of age. Main was concerned about this

instructional rigidity because the Common Core had been seen as developmentally

inappropriate particularly for elementary students. The Common Core required teaching

all students exactly the same material, regardless of whether or not a student was

cognitively ready. Policies like these often overemphasized content over the emotional

needs of children. Powers pointed out that, Common Core policies contained their own

assessment policies embedded within them. These assessments were intended to

determine whether or not the Common Core standards are being followed. Those teachers

failing to adhere to the standards were required to conduct more professional

development that ensures the teacher will teach Common Core standards in the future.

Finding 3 illustrated the manner in which policies can evolve for better or worse.

On the negative side of policy evolution Holley (2006) outlined how increased

competition to be admitted to U.S. universities caused a change in university entrance

policies through applicant abuse of the policy. The abuse of the policy changed the

meaning and function of the policy leading to loss of desirable candidates that, highly

competitive departments could have benefitted from. Selahattin and Kevin (2001)

showed that policies on economic growth change the intended outcome of the policy

depending on the size of a community’s technology sector. Roberge (2012) explained

that policies involving school bullying have changed from a focus in paradigm on strict

discipline and punishment to a focus on the way bullies interact with their peers and vice

versa.

Dietz and Rogers (2012) argued for a completely different approach to policy in

which policies themselves are considered experiments. Evolution of policies stems from

failure of the policy which leads to a better set of procedures to reach the goals the policy

intended from the beginning. The change from NCLB to Race to the Top policies may be

seen as education reform evolving in a similar way. Perhaps a better set of procedures for

a new math assessment policy can be realized from the results of this evaluation.

Relationship to conceptual framework. The three findings in this section were

consistent with Bardach’s (2011) eightfold path in confronting tradeoffs, defining the

problem, and assembling evidence. Finding 1 illustrated the tradeoffs between who was

given authority to make a policy. This finding also served to illustrate the many complex

layers of policy from the federal, state, to local levels of the policy. This was similar in

theory to Sabatier’s (2007) institutional rational choice that makes up this framework of

the policy process. Institutional rational choice occurs in part when decisions about rules

are made at one level of authority within a structure of rules existing at a different level.

In this policy analysis one level of authority was the local school district and the fact that

it was bound by state and federal rules of conduct.

Finding 2 paralleled Bardach’s (2008, 2011) first component of the eightfold path,

problem definition. The problem in this instance was that not all teachers were teaching

content that matched the math standards before the policy was put into place. This

finding also paralleled Sabatier’s (2007) multiple streams framework in which agenda

setting and decision making occur. Although none of the participants made direct

reference to agenda setting, it can be inferred from state archival data that improvement

of state assessment results was the agenda that ultimately led to the math assessment

policy. These archival data were addressed in subsequent sections of this manuscript. The

push and pull of the federal NCLB act and the district’s failure to meet AYP in the early

years of the policy may have been the catalyst for the creation of the math assessment

policy. Lack of AYP can indicate that teachers are not attending to the standards that are

tested (Rotherham & Dillon, 2007).

Finally, finding 3, evolution of the policy, differed from Sabatier’s (2007)

innovation and diffusion models. Innovation occurred when a new policy was adopted

from an outside organization. This process appeared in education reform when one state,

district, or school observes that another state, district, or school has used a policy. The

state, district, or school observing the policy then adopted the policy and other states,

districts, or schools observing also follow suit in a domino type dynamic. Juxtaposed

against this was the evolutionary nature of policy analysis. Bardach (2008, 2011)

recommended that the eightfold path be repeated for policy refinement. This may have

occurred with the math assessment policy, although no data collected guarantees that this

was the means by which the policy changed from its original utility. However, there was

solid data from respondents that the policy became something different when

administration changed hands, particularly in Teacher T2’s comments. This matched

closely with Sabatier’s (2007) social construct framework. When the district’s social

construct surrounding administration changed the policy began to change.

A finding embedded within finding 3 was that the policy had changed because

state standards changed to the Common Core. This finding matched the literature to

varying degrees (Partnership for Assessment of Readiness for College Careers, 2014;

Smarter Balanced, 2014). Goldstein (2009) warned that Common Core standards were

vague and had uncertain outcomes. Moreover, policy differences between educational

reformers remained an issue in two ways. First, cosigners of the standards were members

of testing groups and this represented a conflict of interest. Second, standardized tests

were the underlying goal of the Common Core and by its very nature discouraged

creative teaching. Polikoff and Porter (2014) stated that the Common Core was the driver

behind value added assessment policies, teacher quality policies, instructional alignment

to the standards, and policies on instructional measures. The authors also found that

surprisingly instructional alignment to students has a weak correlation. Instructional

alignment and pedagogical quality were not as important to learning as policy makers

emphasizing standards based reforms asserted. Educational policies have yet to be able to

determine what really matters in quality instruction. Value added measures failed to

detect differences in content and instructional quality (Rothman, 2014; Zancarrella &

Moore, 2014).

Research Question 2: How was math instruction conducted before and after the

implementation of the math assessment policy?

Finding 1. The first finding regarding instruction was that the policy has been the

vehicle for a more collaborative culture. This finding was expressed primarily by

administrators. Administrator A1, who was a teacher when the policy began, emphasized

that collaboration was brought about by the policy when responding to the question “In

what ways, if any, has the math assessment policy helped teachers identify what teachers

must do and change in their practice to improve student learning?”

I think that the common assessments become the product that pushed teachers to

move in the direction that they need to move. You know, if you expect them to sit

down around the table and create an assessment together, which is what we did,

then we’re all on the same page, looking at the same standards, expecting the

students to know the same things, working together to get that done, and a lot of

teachers had to change their ways in order to be a part of that collaboration and

have that product in the end.

Teacher T2 confirmed this when answering the question “In what ways, if any,

has the math assessment policy met the goal of determining what students have learned?”

I really feel like it did a good job of when we got together as a fifth grade team

(because when we were using these assessments we all taught math, I wasn’t the

only math teacher) and so we would get together, we would give a common

assessment, we would enter our data into a template and then we would talk

together about what we did well, and see where one class in one standard did

really well and where one class didn’t.

Teacher 3 insisted collaboration was the cornerstone of the policy when answering the

same question:

We all were involved in common formative assessments. We sat down. We made

the test together as a fifth grade team. Every test we would write, we would look

at and then we would decide what we needed to go back and teach the next time.

In answer to the question “What future recommendations do you have, if any, for

the math assessment policy and why do you make those recommendations?”

Administrator A2 emphasized the importance of the collaborative nature of the policy

and its impact on instruction:

That they’re reviewing data together and certainly they don’t necessarily have to

have inter-rater reliability as much in math as in some of the other content areas,

but that collaborative scoring together and making sure that everybody is indeed

on the same pace, that they’re there for the problem solving when they’re hitting a

wall ups the ante of the level of instruction.

Finding 2. The second finding with regard to research question two was that

ownership of the policy had shifted from administration to teachers wanting to know

what students need to learn in order to inform their instruction. In response to the

question, “In what ways, if any, has the math assessment policy met the goal of

determining what students have learned?” Adminstrator A2 stated, “Drilling down to the

classrooms all the way down to the student and how else will we know what they know?

What they need help with? It guides instruction every day.”

Relationship to the literature. Finding 1, policies can foster collaboration,

complemented Teague and Anfara’s (2012) findings that high levels of collaboration

among teachers results in better performance among students. Kitagawa (2010) found

that inter-organizational collaboration arrangements, rooted in Scottish educational

policy, between universities at international, national, and subnational levels resulted in a

pooling of initiatives that save all collaborators resources, time, and money. Sharing

research facilities also resulted in greater collaboration between institutions. Garrett

(2012) found that collaboration between communities and schools, embedded in school

evaluation policies, help to produce a more educated workforce.

Most similar to finding 1 was Hudson’s (2013) research, which found that a

policy in higher education implemented to create a place-based system of family and

education services for youth provided direct evidence that policies contribute to

collaborative practices. The “neighborhood” policy fostered collaboration by college

instruction, research, teacher training, community service, community leadership, and

community grant management. Wilhelm (2009) found that school policies can critically

impact whether or not collaborative cultures were fostered and maintained within

schools.

The finding that top-down policies over time were owned from the bottom up,

finding 2, depended on the nature of what the policy intended to address (Girdwood,

2013). Top-down demands a great deal of command and control to institute a policy. In

order to succeed, a top-down policy must have clear goals, knowledge of what the result

will be from the policy, a hierarchy of authority in an institution, policy aligned with the

institution’s rules, and a bottom rung of the hierarchy that is tapped as a resource. There

were many criticisms of the top-down method of policy creation in the literature. One

such criticism was that upper administration often failed to consider the full ramifications

that result from the policy (Amaral, Tavares, & Santos, 2013; Spillane, Reiser, & Reimer,

2002).

A second criticism of top down implementation was that administrators often

ignored the political aspects of policy creation. A lack of awareness of the policy’s

impact resulted in policy failure and more politicking in the organization. Policy makers

may assume that statute framers were important actors in the making of the policy, when

local community members were often the most taken for granted and most important

policy players. Rapid top-down institution of a policy almost always resulted in negative

consequences (Easterly, 2008; Thomas & Wan-Lin, 2013; Ying Chieh, Yu-An, & Chad,

2012). This was similar to Villiers’ (2011) finding that people responded to policies with

undesired behavior and policies should be created based on people’s motivations. This

finding was explored in greater detail in the next section under Research Question 3.

Relationship to the conceptual framework. Finding 1, that the policy instituted

more collaborative practices, was difficult to match up to Bardach’s (2008, 2011)

framework, but it loosely fit into the fifth part of the framework, “project the outcomes”.

From the data collected, it was apparent that many participants were in agreement that the

policy did have the positive outcome of fostering collaboration among teachers.

Sabatier’s (2007) policy process model matched finding 1 in regard to positive social

constructs. A collaborative school culture is a social construct. In the case of this policy

analysis, it was vital to understand what social constructs would be successful in

improving the math assessment policy.

Regarding finding 2, Bardach’s (2008) framework did not specifically address

top-down models of policy because it was a more generalized framework for analysis of

any policy that may have been created. However, top-down policy creation was part of

Bardach’s analyzing the tradeoffs inherent in the agenda setting of policy. There were

always tradeoffs in top-down policy creation, as shown in the next section. Often the

tradeoffs were that the people associated with the bottom rung of the institution did not

buy in to the policy and either did not carry it out or half-heartedly did so. Sabatier’s

(2007) institutional rational choice also fit with finding 2 because it was the choice of

hierarchical institutions to determine whether or not a top-down model of policy making

should be instituted. In this case it was likely that at the time of the creation the

administration believed the only rational means of instituting the policy was with a top-

down approach. It is interesting to note that the findings stated by administrators in this

study differed greatly from the literature. Findings specified by teachers showed that

teachers were not using the policy and indicated a lack of buy in from faculty. As a final

point, the interpretive theory framework supported the finding that administrators and

teachers viewed the policy differently (Bastedo, 2005; Geertz, 1973; Hamman & Hopson,

2013; Humphrey, 2008; Miller, 2002).

Research Question 3: What are the perceived outcomes associated with the

implementation of the math assessment policy, and what is the basis for the perceived

outcomes?

Finding 1. Regarding research question three, it was found that not all math

teachers in the district were actually using the policy. This belief was voiced by two

teachers, and a third teacher said they never used the assessments that were initiated by

the policy. In answer to the question “What future recommendations do you have, if any,

for the math assessment policy and why do you make those recommendations?” Teacher

T2 stated,

And so my recommendation would be that we still do use exit assessments and

talk about them for vertical articulation to pass on, you know, like as a fifth grade

teacher to the sixth grade teacher to say, “my kids did great in these areas from

100

the exit assessments, but in fractions they are still really limited.” To use it as a

tool to pass on information from grade level to grade level. I don’t think teachers

are using them in the lower grades.

Teacher T3 said in partial answer to the question “In what ways, if any, has the

math assessment policy helped teachers identify what teachers must do and change in

their practice to improve student learning?” “I know the math teacher right now doesn’t

use the assessments as we’d created.” Finally, Teacher T2 disclosed in answer to the

question “How did teachers integrate the math assessment policy into their daily

instructional plans and course goals?” “Um, well, I didn’t really because I didn’t have an

exit assessment. I mean, I guess I didn’t use the actual exit assessment, but I used the

ideas that it’s built upon to think, you know, ‘what do kids need to know’ and what’s the

best way for kids to be understanding this when they leave.”

Finding 2. The second finding, too many tests being administered all with the

same goal as the math assessment policy leading to a lack of time for evaluation of all

data, was expressed by both an administrator and a teacher in response to the question

“What future recommendations do you have, if any, for the math assessment policy and

why do you make those recommendations?” Administrator A3 stated,

We need to be smarter; we need to assess less and be smarter with the

assessments that we give, and that’s the big picture, and everybody is saying that.

Our state assessment, NWEA, classroom CFAs, they’re we’ve just got too much

assessing going on, and we need to be smarter about, you know, using data, and I

101

think the student clickers helped, but to add that all up is kind of arduous, you

know, a system that’s a little bit.

Teacher A1 stated in response to the same question,

so the problem I see is that there are all of these overlapping assessments, so I

think maybe there is too many, and it would be great if we could have one thing

that was integrated all together. In fact, I think we should just get rid of TCAP and

just use NWEA the whole way through, or TCAP should be broken up into three

pieces like NWEA is, well, or the PARCC, or whatever we’re going to be having.

Finding 3. The idea that the policy had met its intended goals to a certain degree

was finding 3 in regard to research question three. Administrator A2 stated in response to

the question “In what ways, if any, has the math assessment policy met the goal of

determine what students have learned?” “I think it’s just been critical.” In response to the

same question Administrator A1 affirmed, “I would say that it did meet the goal to a

certain extent at least it certainly was much better than what we were doing before we

had those common assessments and exit exams.” Administrator A2 added to this by

saying,

So, if we were to just say, oh, compared to the old standards, I think our kids

would have surpassed so many skills and scores than kids in the past have because

we’ve upped the ante in the level of instruction and have to delve in so deeply

with major math concepts with the transition, it would be difficult to find great

data to prove that, but I would have to say that the level of instruction and the fact

102

that we’re meeting new standards that are much more rigorous indicates to me

that it has been helping.

By contrast, in response to the question “In what ways, if any, has the math

assessment policy closed the student learning gaps that existed when the policy began?”

Teacher T1, in response said, “I have no idea.” indicating a different awareness of the

impact of the policy versus administrators. Teacher T2 responded to the question “Has

the math assessment policy impacted student achievement?”

I definitely think it impacted it positively. When we had everybody teaching the

same thing, it was consistent and when you’ve got five different, very different

teachers and styles it kept us consistent; it kept us on the same page; there was a

lot of fidelity with how we all taught.

Teacher T3 detailed in response to the question “In what ways, if any has the

math assessment policy helped teachers identify what teachers must do and change in

their practice to improve student learning?”

It was great at determining what they had learned because we would send them to

[the senior administrator], and he would put this little graph together, and we

would know exactly which questions and which standards we were to address and

which ones the kids really didn’t know.

Administrator A2 confirmed this, as well, when asked “Has the math assessment

policy impacted student achievement?”

I do believe it has impacted student achievement, and I think that again, the data

is a little bit difficult to tease out because not only have we been doing a transition

103

test to help gauge that, but we’ve also transitioned into whole new standards and

new targets.

Administrator A3 supported Administrator A2’s assertions to the same question

stating, “Kids are scoring more successfully on the assessments we use.” In response to

the same question Administrator A4 stated, “I would say it has because I would say

we’ve obviously made a lot of decisions based on results. You know, we’ve changed

curriculum in our courses.”

Relationship to the literature. Finding 1, that not all persons charged with

carrying out a policy actually followed it, was similar in function and form to Balls, Eury,

and King’s findings (2011) that showed when leaders give up their own power, persons

working under them often accepted their own responsibility over adherence to rules and

policy. Because of the top-down nature of the policy, the teachers at the local district

were not given the power that normally leads to ownership of a policy. In extreme cases,

outside of the local district, NCLB assessments were cheated on by those charged to

carry out the policy correctly. Desimone (2013) found this happened because NCLB

removed teacher discretion and ownership.

According to Bhattacharyya, Junot, and Clark (2013) unintended consequences

carried out by schools regarding NCLB included teaching to the test in order to make

higher scores at the expense of critical thinking and authentic learning. Rebora (2012)

studied 10,000 teachers and found most teachers did not value high stakes testing mainly

because students did not take them seriously. Perhaps this was at play in the local district

with teachers who decided not to follow the policy. Other researchers suggested that

104

novice teachers encounter more experienced teachers who instill bad habits in them, such

as ignoring essential policies. When novice teachers saw there was no consequence for

not following a policy, they were not motivated to follow it (Keogh, Garvis, Pendergast,

& Diamond, 2012; Wallace & Irons, 2010).

When policies created more obstacles to meeting organizational goals, users

questioned or defied the policy. The Robert Schuman Centre for Advanced Studies

(2010) recommended eliminating these types of policies. Lack of oversight to ensure the

policy was followed may have been the result of vulnerabilities inherent in the policy and

not addressed by the policy (Thoumi, 2012). Wendel (2014) found that there are

strategies that people use when new policies are put into place: cheat, make or change

habits, or support conscious action. This can be avoided by structuring the policy to make

it inviting to use, constructing an environment that supports the policy, and helping

people prepare to carry out the policy. Beaver (2014) asserted that policies were often not

followed by staff members because they were either not aware of them or their wording

left the impression they were suggestions that could be ignored.

Finding 2, too many assessments resulted from the policy which made analyzing

data overwhelming for teachers, was strongly supported in the literature. Schuster (2012)

used multiple regression to examine the effects of numerous types of assessments. The

author found that exit exams did little to improve students’ math achievement, GPA, or

dropout rates. Standards based exams were positive predictors of increased dropout rates.

End of course exams correlated with a decline in math achievement, and the more at risk

students were the higher their decline in math. The only students that benefitted from

105

these assessments were students in the top quartile. Schuster concluded that using a large

variety of assessments to measure student academic growth resulted in confusing,

inefficient, and ineffective quantities of data .

Amrein-Beardsley and Collins (2012) found that types of student groups can bias

a teacher’s ability to test well on value added assessments. Districts were using

inconsistent assessment results to determine how their teachers were evaluated and which

teachers were fired. In fact, teachers of higher achieving students had the most difficulty

showing that they added any value to students’ learning within a given school year.

Papay (2010) concurred, finding that multiple assessments’ variations in content, scaling,

student samples, timing, and measurement error made assessment results intrinsically

unstable. Senge, Cambron-McCabe, Lucas, Smith, and Dutton (2012) established that the

number of assessments students took in a given year increased dramatically since the start

of NCLB and continued to increase because of involvement by business leaders in

funding school reform initiatives. Excessive testing limited the amount of time students

spent learning and demoralized teachers endeavors to reach all students socially and

emotionally (Au, 2011; Daun, 2014; Elwood, 2013; Isaacs, 2010; Santoro, 2010).

Conversely, Grissom, Nicholson-Crotty, and Harrington (2014) found that assessments

from NCLB had no impact in terms of overall school accountability.

Finding 3, that the math assessment policy had met its intended goals, differed

greatly from findings on school reform before and including NCLB. Dworkin and Tobe

(2014) discovered that one of the unintended results of NCLB was higher teacher burnout

and less trust in their supervisors. Administrators that gave teachers more ownership and

106

decision making in school policies experienced less of an ill effect from this. Gisbond,

Neill, and Schaeffer (2012) found that NCLB did not meet its intended goals, damaged

educational equity between students, failed to improve student academic performance,

created over testing, and failed to meet its intended goals.

Pallas (2011) demonstrated there was no student test data that showed gaps in

achievement, beyond what public schools have already attained, can be closed further.

Dahill-Brown and Lavery (2012) asserted that the ability of a policy to achieve its

intended goals was based on the district’s capacity and political will to carry out federal

policy. This was determined to a great extent by the political leanings of the state the

district was based in. More conservative states were more likely to administer less

rigorous tests yielding outcomes that appear more closely aligned to the goals in NCLB.

Relationship to the conceptual framework. Finding 1 related to Bardach’s (2008,

2011) “project the outcomes.” Policy makers should admit the reality that faculty may not

follow policy and make a plan in advance to address this when it arises. Finding 1’s

relationship to the conceptual framework was associated with Sabatier’s (2007) findings

on punctuated equilibrium. Punctuated equilibrium states that policies accompany periods

of rapid change preceded or followed by no change. In this instance the adoption of the

math assessment policy marked a time of rapid change followed by little or no change as

evidenced by several years in which the policy was not revised or replaced.

Finding 2 related to Bardach’s (2008, 2011) policy analysis issues when excesses

are the cause of policy. These excesses often stemmed from the bureaucracy inherent in

all organizations which tied with Sabatier’s (2007) multiple streams framework. Policy

107

can both originate from multiple sources and generate multiple outcomes. The rationing

of attention in this case was that a great deal of attention was devoted to testing,

increasing the number of tests administered. The search for alternatives led to another

instrument which increased the amount of testing. Exit assessments were used, but so

were other forms of assessment during the time the policy was implemented and

throughout its duration.

Finding 3 connected to Bardach’s (2008, 2011) framework in which there was a

danger that extrapolating from successes observed in demonstrative context and timing a

policy analysis during a period that was likely to create the impression that success was

inevitable. Whenever an organization’s policies were met with success, Bardach

recommended that concerns about high level or extremely below level results be explored

and unwarranted optimism avoided before the outcomes are thoroughly tested.

Simultaneously, the idea that the assessment policy had met its intended goals was

present in Sabatier’s (2007) framework that positive predictions about policy outcomes

should be tested to the extent that policy makers are able to avoid periods of study and

areas that are too convenient.

The idea that the math assessment policy had changed because the standards

changed fits with the evolutionary model (Bardach 2008, 2011). A common process of

change may be instigated by other policies connected to the policy under analysis.

Moreover, Sabatier (2007) demonstrated that policy change can have a variety of

presentations. A common way a policy may change was when the policy degenerates or

goes beyond degeneration and fails. Another way policies may change was that they may

108

improve upon their original intent and goals. The Common Core was only two years in

use by the local district. Therefore, it will take time to see whether this change in policy

is positive, negative, or neutral.

The problem was that a math assessment policy had not been evaluated in the

local district. By evaluating the policy process in the local district the problem was

resolved. Recommendations made from this policy evaluation have the potential to create

ongoing and global impact on administrators and teachers who want to improve in the

area of math assessment or even assessment in general. As teachers work together to

strive for excellence in education, the school setting will be able to make adjustments in

the curriculum for diverse groups of students. In the next section, reflections and

conclusions that were rooted in the literature were discussed in order to further support

the recommendations made in this policy evaluation (Mears, 2010; Murchan et al., 2009;

Polikoff & Porter, 2014).

The purpose of this study was to evaluate a local math assessment policy and

describe how teachers and administrators implemented and continued a math assessment

policy in a local school district. The themes identified were (a) uncertainty in the ranks,

(b) sharing power, (c) collaborating among the mathematics disciplines, (d) policy

evolution, and (e) policy outcomes. The peer debriefer agreed that these were common

themes and stated, “This all looks good. This is an interesting study.” In the current

section, I have attempted to explain these emergent themes.

Uncertainty in the ranks. Uncertainty in the ranks refers to the different

perceptions teachers and administrators had regarding the math assessment policy.

109

Administrators in every case expressed a belief that all applicable teachers were using the

policy. Conversely, in each case for the participants that were teachers, none of them

expressed a belief that all teachers were currently or had in the past used the math

assessment policy.

Sharing power. Sharing power relates to how teachers were accessed for their

buy in with the policy. Both teachers and administrators asserted that top-down

administration was used to instill the policy in the beginning. Administrators expressed a

belief that teachers owned the policy and that it was no longer simply an administrative

requirement.

Collaborating. Collaborating among the mathematics disciplines was associated

with the fact that several participants indicated that the math assessment policy resulted

in a change in practice. That change in practice was that teachers had to write

assessments in a group leading to collaboration. The change in practice also related to

teachers helping each other with student achievement when teachers found they were

weak in a standard another teacher that was stronger in that standard could assist that

teacher in improving instruction in that area.

Policy evolution. Policy evolution was connected to sharing power in that the

policy started as a top-down mandate and then over time moved to teachers owning their

students’ assessment outcomes so that they can determine what to teach. Policy evolution

was also linked to how assessments were revised, edited, and changed from the onset of

the policy to present day due to changing from state standards to Common Core state

standards.

110

Policy outcomes showed that to some extent, based on participant perceptions, the

policy improved student achievement and helped to close achievement gaps. Participants

expressed a belief that the policy, either in the past or currently, met the outcomes of

improving student achievement. The participants also expressed that since the standards

have changed, the degree to which this can be determined was complex. State level

archival data showed an increase in student math achievement for some schools and a

decrease in other schools.

Evidence of Quality

Participants were asked to review their transcribed data to ensure quality data

collection (Barusch, Gringeii, & George, 2011; Cooper, Brandon, & Lindberg, 1998;

Creswell, 2008; Marshall & Rossman, 2011; Nguyen, 2008; Spall, 1998). Next,

triangulation was guaranteed by verifying evidence from interviewing eight different

sources. Within these eight sources, two types of information were collected, one from 3

K-12 teachers, and a second from 5 K-12 administrators. Finally, peer debriefing further

established triangulation. A colleague unfamiliar with the math assessment policy

reviewed the method of data collection, interpretations of the researcher, and the

conclusions that stemmed from data analysis. The peer debriefer confirmed that the data

interpretations were valid and reliable. Documentation that was used to establish

evidence of quality can be found in Appendices B, C, D, E, F, and G.

Outcomes

The outcome of and project for this study was a complete evaluation of the math

assessment policy of a local southwestern school district. The stages of the policy process

111

were analyzed, and the resultant efficacy of the creation, implementation, continuation,

and evaluation of the math assessment policy was explained as seen by the participants

interviewed for this study. According to Sabatier (2007), the stages of the policy process

included problem recognition, agenda setting, policy formulation, policy adoption, policy

implementation, and policy evaluation. Problem recognition as an outcome was present

in the initial stages of the math assessment policy, as central office identified a

deterioration in state level math assessment data in 2004. Through the evaluation of

interview data, it was discovered that the agenda that led to adoption and formulation of

the math assessment policy was generated through a top-down administrative model. The

outcome stemming from this part of the evaluation was for the district to consider

investigating a more bottom-up model of policy creation that will help teachers buy into

future policy adoptions. As demonstrated through interpretation of the narrative case

study data, it was shown that a lack of buy-in may have led to some teachers ignoring the

policy or dropping it as soon as it appeared to no longer be rigorously enforced. A third

outcome of the policy evaluation process was the policy’s strength in implementation in

which teacher and administrative teams through intense collaboration created the

assessments that were used to carry on the policy. The final outcome of this project was

the evaluation report that explains the findings, implications, and recommendations

regarding the math assessment policy.

The manner in which individual stakeholders understood the policy would not

have been uncovered were it not for this policy evaluation. Individual participants’

understanding of the policy was a subset of the outcomes of this project. During data

112

collection and analysis, it became evident that each building involved in the policy had a

significantly different understanding of how the policy was to be used and its history. For

example, middle school members thought the policy consisted of a series of common

assessments including an exit assessment. High school members only recalled use of the

exit assessments as the main part of the policy. Elementary school members primarily

recalled use of the common assessments.

This discovery brings to mind the need of the local district to continuously

evaluate and not take for granted how individuals understand the use and function of the

policy. Gunter’s (2009, 2010) power economy regimes state that policies are understood

differently depending on the level of power individuals possess in an organization.

Therefore, it may be possible to determine whether an individual’s interpretation of the

policy stemmed from their own power economy. Making these determinations could help

the local school district understand paradigms that users of the math assessment policy

fall into and keep track of whether or not district personnel have common understandings

of the policy.

The Project Deliverable

The project that has resulted as the outcome of the preceding findings was an

evaluation of the process that led to the math assessment policy in the local district (see

Appendix A). The frameworks that guided the project deliverable were discussed in the

preceding section under findings. The evaluation showed that there were positive aspects

of the process that led to the math assessment policy at the local district. The evaluation

also demonstrated that there were some complicated aspects to the policy as perceived by

113

local teachers and administrators. The project deliverable is discussed in more detail in

the next section.

114

Section 3: The Project

The project that was a result of this study was an evaluation of the process that led

to the math assessment policy in a local district. The deliverable that has been created

through the process of collecting and analyzing qualitative data for this study is present

and complete (see Appendix A). The goal of this project was to inform the local district

stakeholders about the process that led to the creation, implementation, and continuation

of the math assessment policy.

Project Description

The project in this study was the evaluation of the math assessment policy and its

processes from the beginning of the policy until the spring of 2014. This policy

evaluation was initiated in response to needs and responses within a local school district

in the southwest, in which it was discovered by the superintendent that the policy had

never before been evaluated (personal communication, 2012). For the project, a well-

developed policy evaluation was created. To carry out this policy evaluation, a qualitative

case study narrative was used to identify strengths and opportunities for improvement

within the math assessment policy. The conceptual framework for the policy evaluation

was based on Sabatier’s (1991) five elements of better policy process and Bardach’s

eightfold path for better problem solving. The five elements of better policy process

provide policy evaluators, which may include local districts, with a construct to which to

hold current policy accountable. The eightfold path parallels and expands upon the

elements of policy process, providing yet another lens that evaluators can use to assess

the development and outcomes of policy. Both the five elements of better policy process

115

and the eightfold path to better problem solving are discussed at length in the Case Study

Narrative of the Policy Process and the Possible Social Change Implications sections that

can be found in succeeding pages of this study.

Project Goals

The goal of the project was to evaluate the processes used to create the math

assessment policy and to determine whether the policy had accomplished its intended

outcomes. In this section, the project, which was an evaluation of a local math assessment

policy, is described. The study is framed by the theoretical frameworks of Sabatier’s

(2007) policy process theories and Bardach’s (2008, 2011) eightfold path, which was the

framework that assisted in evaluation of the policy.

Rationale for Selecting the Policy Evaluation Genre

The rationale for choosing this policy genre was that the recommendations from

this policy evaluation would provide opportunities to inform perceptions of the math

assessment policy and future policies, enhance teacher instruction, promote student

achievement, and engage the local community. Teachers and administrators may go

beyond traditional concepts of math assessment to a broader, more social framework of

assessment if these policy recommendations are adhered to (Balthasar, 2011; Ceniviva &

Farah, 2007; Patton, 2002).

The rationale for the research design of a policy process evaluation, using

narrative data, was that such a design relies on the participants’ experiences, the order in

which parts of the policy process were instilled, and perspectives of teachers and

administrators regarding the math assessment policy. Additionally, interview data, once

116

analyzed, allowed the categorization of the outcomes of the policy into themes. Personal

narratives addressed the problem directly; participants were asked questions that helped

unwrap the process involved in creating the math assessment policy. Additionally,

participants responded in detail to the research questions about the math assessment and

how it affected instruction and collaboration (Gubrium & Holstein, 2009; Wertz,

Charmaz, & McMullen, 2011).

Literature Related to the Project

The Effect of Communication in the Policy Process

Van Leeuwen et al. (2014) indicated that effective communication is essential to

the success of any policy. As the policy moves forward and new members of the

organization are trained, those members must be communicated with about the policy on

a regular basis. Derthick and Rotherham (2012) discussed the importance of federal

policy in guiding effective communication about education reforms from the state to local

levels. Conversely, Rush and Scherff (2012) demonstrated that assessment, when

communicated poorly, may be interpreted at the local level as a means of humiliating

teachers who fail to improve student achievement.

Karelitz, Fields, Levy, Martinez-Gudapakkam, and Jablonski (2011) asserted that

policy can be interpreted broadly by both teachers and administrators, precipitating

unintended consequences. Through NCLB policy, high mobility in the teaching

profession was exacerbated dependent upon how districts interpreted the requirements of

the law. NCLB marks an important chronological milestone in the development of

education policy. Between 1972 and 2011, math curriculum became more accessible to

117

lower achieving students with a cost to higher achieving students, who actually saw their

scores wane during that period. The current U.S. assessment policy trends in K-12

education make it difficult to motivate students to commit to STEM careers. Students’

views of themselves and stereotyping also determine their choices to take math and

science courses (Graham, 2011; Grey & Wichman, 2012; Vigdor, 2013).

Next, media interpretations, ratings from individuals without children in school,

ratings from individuals with children in school, and hierarchical government structures

affect how local policy makers understand and interpret how policies come into being

(Favero & Kenneth, 2013; Heber, 2011; Roche & Wilsker, 2010; Wildavsky, 2011). State

and local school boards and administrators may benefit from being mindful of this when

creating school policy. Moreover, policy cannot meet its intended outcomes without an

organized effort that includes effective and detailed communication measures and

effective assessment and revision, as well as competent policy-making knowledge at the

local level and remaining mindful of how policies affect students and communities

(Gaston, 2012; Karsten & Visscher, 2010; Miruc, 2010).

Kirp (2013) and Dalton and Brand (2012a, 2012b) stressed the need to minimize

the number of assessments, make them inform instruction rather than punish teachers and

students, and support depth of learning and higher order thinking. Others recommend

using community involvement and feedback from potential employers as a means of

assessing students and supporting continuous school improvement, as well as use of

multiple measures rather than high-stakes testing (Gao & Grisham-Brown, 2011; Longo,

2013; McTier & McGregor, 2011).

118

Successful policy making is confounded by its inherently political nature. Some

have asserted that the standards movement cannot meet its intended outcomes in

education for this reason and because of the skewed nature of who is considered

knowledgeable about education policies (Gunter, 2010; Gunter & Forrester, 2009).

Gunter broke policy making into four regimes. In the first of these, negative power,

positive economy entities use their money and power to affect policy creation despite

their lack of direct access to government institutions. In the second regime, positive

power, positive economy individuals have direct access to public institutions and

influence them by inserting private sector practices. In the third regime, positive power,

negative economy stakeholders, who have no true power but have access to public

institutions, engage with leadership to develop public sector services. Finally, in the

fourth regime, negative power, negative economy individuals seek to develop alternative

approaches to leadership within the public sector and are often critical of that leadership.

Many have recommended that the policy-making process be depoliticized and involve

bottom-up rather than top-down thinking so that policies truly serve those whom they

were intended to serve, and so that what is worth knowing is emphasized (Bardach, 2008,

2011; Howard, Wrobel, & Nitta, 2010; McGuinn, 2012, 2014).

Looking at local policy influences, Race to the Top (RTTT) resulted in new state

and local education policies that were less than systematic due to the subjective nature of

its requirements. Institutional capacity and coalition building determined the recipients

and results of RTTT. Grantees promised to raise student achievement and close gaps at

statistically unattainable levels. All recipients have needed to postpone teacher and

119

administrator evaluation systems and have had inadequate time to conduct and analyze

pilot studies. Data support the idea that student achievement gaps are driven by

socioeconomic gaps in opportunity, and those gaps cannot be filled by any of the

requirements stipulated in RTTT but must instead be met by fulfilling student social

needs (Bowen, 2011; Loepp, 2013; Manna & Ryan, 2011; Noguera & Weingarten, 2010;

Weiss, 2013). These opportunity gaps can be seen clearly when tracing college dropouts

back to their elementary school summer education experiences (Alexander, Entwisle, &

Olson, 2007). From a political standpoint, amendments at the federal level to end RTTP

have been proposed because of many unfunded mandates that are currently in play

(Lanham: Federal Information & News Dispatch, Inc., 2013).

State and local policies that stem from RTTT have revolved primarily around the

Common Core State Standards (CCSS). There are mixed views of the CCSS. One

positive view is that they emphasize rational decision making, allow for multiple entry

points into standards and depth of knowledge, and emphasize personal finance

(Caltabiano, 2013; Cavanagh & Klein, 2012). Conversely, early childhood educators did

not participate in developing the CCSS, and an influential group of these educators has

been outspoken about the developmentally inappropriate nature of the CCSS (Miller &

Calsson-Paige, 2013).

All human interpretive emotions create organizational culture, behavior, and

understanding. Human subconscious drives guide behavior within an organization,

particularly when conflicts arise despite the conscious awareness that emotion is not a

logical human trait. These human characteristics cannot be considered apart from how an

120

organization such as a school district perceives local policies (Shane & Glinow, 2010;

Sichel, 2010). Often, the result is that the manner in which individuals understand

something such as the math assessment policy varies widely among the individuals

affected by it, as previously reported data suggested. One might take for granted that

local organizations set agendas when making policy, but in fact, it has been shown that

this is rarely the case (Lin, Lindquist, & Vedlitz, 2011; Lin, Linquist, Vedlitz, & Vincent,

2010).

Literature Saturation

Literature saturation pertaining to evaluation of the math assessment policy was

reached through a Boolean search. The following search terms were used as a means to

reach saturation: education policy, policy evaluation, assessment policies, math

assessment, educational reform, and educational policy. Literature between the years

2000 and 2015 was reviewed. Relevant data were used to assist in the evaluation of the

math assessment policy. ERIC, Education Research Complete, Political Science

Complete, ProQuest Central, and SAGE Premier Databases were used to gather literature

from peer-reviewed journals and other sources. Additional resources, including State

Department of Education websites, educational websites, and textbooks, were used as

appropriate.

The following recommendations were made using both Bardach’s (2008, 2011)

eightfold path and Sabatier’s (2007) five stages to effective policy process. The most

likely potential barrier to these recommendations will be the length of longitudinal

analysis. A period of 10 years or more for the life span of a local education policy is

121

highly unusual, as illustrated in the previous literature review. NCLB started in 2001, and

waivers for schools not meeting AYP began to be issued in late 2011 and early 2012

(U.S. Department of Education, 2012). Confounding variables that change the dynamics

of a policy being followed through also resulted in similar dynamics with local policy.

Furthermore, the policy had been in use for just under 10 years and so will not meet the

requirement for longitudinal analysis until 2015. This barrier can be overcome by

administrative leadership staying the course on current policies over a period of a decade,

as recommended by Administrator A5.

The first recommendation for future use of the math assessment policy is that

teachers and administrators gain a greater awareness of the assessment tools in use and be

allowed to choose a smaller number of those tools in order to fulfill the goals of

informing instruction and raising student achievement. This would further assist in

determining which two assessment tools are most useful and eliminating the lesser useful

assessments in order to optimize instructional time. Fewer assessment tools were

identified as needed by both teachers and administrators.

The second recommendation is that research be conducted on using assessment

data to evaluate teachers. Effective assessment policy should be grounded in realistic

goals. Therefore, the local district needs to determine whether or not disaggregated data

on student math achievement and growth provided adequate information on which to

base state and local school performance measures and teacher evaluation policies.

Through recently passed legislation, teachers are required to be evaluated based on

student assessment data and principal observation rubrics. However, much of the

122

literature reviewed in the previous section demonstrates a lack of validity and reliability

in assessment tools and student socioeconomic factors having a greater impact on student

achievement results. Federal, state, and local policy makers should pursue further

research to make an informed conclusion about whether or not evaluating teachers based

on assessment data is fair and will lead to achievement gains.

The next recommendation is a district partnership with families and local business

leaders as a means of authentic assessment for the district (Gao & Grisham-Brown, 2011;

Longo, 2013; McTier & McGregor, 2011). This process might take the shape of a

program rather than a policy, with training provided for personnel rather than just a

description of procedures and steps to implementing those procedures (Anderson, 2005).

The local district may want to work to nurture district-family relationships that, far from

being a quick fix, offer further achievement gains over the long term (Bowen, 2011;

Loepp, 2013; Manna & Ryan, 2011; Noguera & Weingarten, 2010; Weiss, 2013). Access

to family and community partnerships is a potential barrier that might be overcome by

direct initiative and leadership from a central office, which could involve home visits by

school staff or community outreach programs.

Another potential barrier to effective policy process involves how individual

stakeholders understand the policy. Consequently, to overcome this barrier, the fourth

recommendation is that the local district evaluate users’ understanding of the math

assessment policy. During data collection and analysis, it became apparent that each case

involved in the policy had a significantly different understanding of how the policy was

to be used. For example, administrators thought the policy was still actively in use,

123

whereas teachers expressed a belief that most teachers, particularly at the elementary

level, were not using it. Using Gunter’s (2009, 2010) theories on power and economy

regimes, it may be possible to determine which individuals have positive power/positive

economy, negative power/negative economy, positive power/negative economy, or

negative power/positive economy. Making these determinations would help the local

district frame each individual’s and school’s understanding of the math assessment policy

and keep track of whether or not district personnel have similar uses of the policy.

Additional research or informal investigation of a student’s math progression

from primary grades to secondary grades may offer a holistic view of math assessment

and development within the district, especially when considering family dynamics among

students that create greater growth and achievement and informing instruction beyond

assessment alone. This longitudinal analysis of students could give the district a “big

picture” look at the efficacy of math assessment policy over time. Longitudinal

evaluation that stretches beyond a decade fits well with Sabatier’s (1991)

recommendations for effective policy process.

Needed Resources

The resources needed to continue the math assessment policy and improve its

future outcomes include already existing assessment resources, teacher collaboration,

continued refinement of assessments by teachers, community connections, and finally,

personnel and time for those persons to devote to district policy evaluations on a

continuous basis so that policies are improved and revised regularly. The implications of

these resources are discussed in greater detail below.

124

First, the district’s administration should consider all assessment resources

currently used and determine whether it would be beneficial to eliminate certain tests in

order to make more room for instructional time (Goldstein, 2009, 2011). The local district

should consider editing the assessments that were created for the policy and ensuring that

they meet the new CCSS. One of the strong points found through this evaluation was the

policy’s ability to increase teacher collaboration. This type of collaboration should

continue through administration support and encouragement (Marzano, 2007, 2010;

Teague & Anfara 2012).

The next resource that will be needed is a partnership with families and local

business leaders as a means of authentic assessment for the district (Gao & Grisham-

Brown, 2011; Longo, 2013; McTier & McGregor, 2011). This process might take the

shape of a program rather than a policy, with training provided for personnel rather than

just a description of procedures and steps to implementing those procedures (Anderson,

2005). The local district may want to work to nurture district-family relationships that far

from a quick fix, offer over the long term further achievement gains (Bowen, 2011;

Loepp, 2013; Manna & Ryan, 2011; Noguera & Weingarten, 2010; Weiss, 2013).

Finally, the district may want to further consider how district personnel

understand the math assessment policy, as data collected in this study uncovered a

disparity in interpretations of the policy among staff and administration. This could be

further explored through the use of personnel already employed by the district who have

experience with policy evaluation or are given the time to educate themselves about

policy evaluation. Costs to the district could remain minimal to none through reallocating

125

an employee members’ time. For example, employees that have partial administration

positions rather than full time teaching loads could be utilized (Picus, 1999). Teacher T2

indicated that these resources currently are not being tapped into and should be

addressed:

I think, you know, when [the senior administrator] came in, she was overwhelmed

with it and the job that she was doing kind of changed, and I’m not saying that’s a

bad thing. She did some great things for our district, but as far as curriculum and

asking teachers to really look at data, that wasn’t her strength, and it definitely

when it wasn’t an emphasis coming from the admin office. I think people were

like, “well, we don’t have to do it. It’s not policy.” Then they didn’t see it as a

policy, so people have definitely quit doing it. Or it’s not consistently being done

say across K-12.

Existing Support

There are several resources already available to district teachers and

administrators that are used to inform instruction. There were a myriad of assessments

being used by the district. This was confirmed during data collection and included

common assessments, exit assessments, Northwest Evaluation Association (NWEA)

assessments, state transitional assessments, and the upcoming Partnership for Assessment

of Readiness for College and Careers (PARCC) assessments. As uncovered in data

collection, both teachers and administrators view the current assessment schedule within

the school as excessive. As teacher T1 asserted, “The problem I see is that there are all of

these overlapping assessments, so I think maybe there is too many, and it would be great

126

if we could have one thing that was integrated all together.” Furthermore, Administrator

A3 stated,

My biggest recommendation, and this was, and I think we have tools like this now

NWEA gives us information right away. We need to be smarter; we need to

assess less and be smarter with the assessments that we give. And that’s the big

picture. And everybody is saying that. Our state assessment, NWEA, classroom

CFAs, they’re we’ve just got too much assessing going on, and we need to be

smarter about, you know, using data and I think the student clickers helped but to

add that all up is kind of arduous you know a system that’s a little bit.

Administrator A4 went on to say,

as far as making a recommendation to change it, we have a lot of assessments a

lot of different ways we are assessing kids, and a lot of them are out of our

control. You know, I wish we could use our own and you know those exit

assessments obviously can become a good method to use on our MSLs [measures

of student learning].

Recommendations for utilizing the already available resources involve teachers

and administrators identifying the most useful two of these tools and eliminating the

others in order to optimize instructional time and inform instruction in a timely fashion

that minimizes the amount of data teachers have to sift through. Furthermore, accessing

resources that combine both community members as assessment tools and formative

assessment tools that are project based and involve optimizing instruction may increase

127

instructional time and higher order thinking skills (Gao & Grisham-Brown, 2011; Longo,

2013; McTier & McGregor, 2011).

Potential Barriers

The potential barriers to refining and improving the math assessment policy

include the degree to which teachers are willing to actively use the current and revised

policy, the degree to which teachers and administrators understand the policy in a similar

sense, and the degree to which administration is willing to spend time and energy

focusing on revising the policy. Teachers believed that the policy is not being adhered to

by staff members, while administrators were under the impression that the policy is being

followed by the district’s math teachers. This discrepancy will need to be addressed in

order for the policy to successfully continue and in order to make positive changes to the

policy. Moreover, this demonstrated an inherent difference in the way teachers and

administrators have framed their understanding of the policy. Ignoring these differences

could have serious consequences for the life of the policy (Ceneviva & Farah, 2007;

Chaisse & Matsushita, 2012).

Community relationships were often challenging. Therefore, gaining access to

family and community partnerships might be a potential barrier that may be overcome by

leadership from a central office. This could include home visits by school staff or

community outreach programs. Community business leaders could also be a potential

barrier to school community relationships. Business leaders have a limited amount of

time in each day and often feel that time is the financial success of their business. School

leaders will have to work to overcome this and convince local businesses that their

128

participation may result in a more skilled and prepared future pool of employees (Garrett,

2012; Hudson, 2013; McTier & McGregor, 2011).

Finally, both continuing and reforming education policy requires a considerable

amount of time on the part of administration and teachers. In order for successful

continuation and refinement of the policy, time will need to be allocated by both

administrators and teachers (Petersen, 2009; Rizvi & Lingard, 2010; Roberge, 2012;

Roche & Wilsher, 2010). In support of this Administrator A4 proclaimed, “If you really

take the time and dig in and look at it which I don’t know that we’re that was ever used in

that way like it could have or should have.” This indicated that prior to this study, time

had not been devoted to evaluation of the policy, which should be considered by the local

district for future uses of programs and policies.

Roles and Responsibilities

Teachers’ roles regarding the math assessment policy were to carry out the policy

by administering common assessments in their classrooms. Teachers were further

expected to use the results of the math assessments to inform and adjust their instruction

and to collaborate with other teachers regarding the assessments. During evaluation of the

policy, teachers explained how they experienced these roles and what they believed the

outcomes of these roles were. They also explained their fidelity to the policy and what

they believed other teachers adherence to the policy consisted of.

School administrators’ roles regarding the math assessment policy were to

oversee math teachers’ implementation, continuation, and collaborative participation

within the district. During the policy evaluation, administrators explained what they

129

believed the original intent of the policy was, what the outcomes of the policy continued

to be, allowed access to their teachers for this study, and reported on the fidelity with

which teachers continued to use the policy and maintain ownership of the policy.

District officials within the central office played a vital role in this policy

evaluation by allowing access to both administrators and teachers for the study. During

the policy evaluation process I served as teacher leader, data collector, and evaluator.

Project Implications

From the policy evaluation recommendations, district personnel, students, and

community members, including parents, can collaborate to ensure that students see the

viability of math in the greater context of their citizenry. Authentic assessment and

expanded learning opportunities can be effective in improving student achievement

outcomes, but teachers will not be able realize these outcomes without strong leadership

implementing these recommendations (Gao & Grisham-Brown, 2011; Longo, 2013;

McTier & McGregor, 2011).

The recommendations from this policy evaluation have the potential to impact all

of the district’s participants in holistic and positive fashion. Regarding the variety of

recommendations that are geared toward meeting students’ social emotional needs, the

district has many avenues available to them for positive policy improvement. The

teachers engaged in the policy process may benefit from relationships fostered and

nurtured within the community. Furthermore, the manner in which they utilize

assessment tools to inform instruction may also be useful. From a national perspective

130

and beyond, many of the recommendations made here correspond to and go beyond

NCLB and RTTT policy and program requirements.

Social Change Implications

Social change may be accomplished through teachers, administrators, and local

stakeholders taking part in an improved policy process. This policy evaluation was based

on teacher and administrator understandings and perceptions of the purposes and

outcomes of the math assessment policy. This policy evaluation and recommendations

involved time for teachers to analyze assessment data; to collaborate with local

community members, families, and business leaders; to plan and implement future

assessment policy, and assess understanding through Gunter’s (2010) power regime

matrix.

Ensuring that recommendations originating from this policy analysis are met will

require use of Sabatier’s (1991) five elements of policy process. Sabatier focused on the

manner in which organizational entities formulate and implement policies. Because

policy theory spans all organizational and government entities, the policy process often

overlooks the role of the public, communities, information on local dynamics, long term

studies, and the part an individual plays in carrying out policy. Implications for this

policy evaluation include positive social change that goes well beyond the math

assessment policy and has a potential to greatly impact the local district. This might occur

through continuing the collaborative practices that grew from the math assessment policy

between K-12 math teachers and administrators. Lastly, K-12 students should be offered

opportunities to increase achievement and see the relevance of what they are learning.

131

Based on the findings from this study, teachers and administrators at the local

district saw evidence that the math assessment policy had met its intended outcomes. The

participants discussed their perceptions in nine areas: (1) failure to abide by the policy,

(2) analyzing data is difficult because it is excessive, (3) a top-down approach in

implementing the policy, (4) consistent use of math standards needed throughout the

district, (5) evolution of the policy, (6) effectiveness of the policy, (7) the policy informs

instruction, (8) new standards impacting the use of the policy, and (9) collaboration as a

central utility of the policy.

Most of the participants who were interviewed had utilized the math assessment

policy. Participant teachers had between 5 and 10 years of teaching experience. Most

administrator participants were teachers when the policy began. Most of the teachers and

administrators were flexible and saw positive aspects of using the policy in district

classrooms. A study participant who admitted to not using the policy still saw potential

positive aspects of the policy. Administrators were the most resolute regarding the

successes of the policy and also had certainty that the policy was still being used. One

teacher was adamant about the successes of the policy, but urged that district leaders

enforce the policy and make sure all teachers are truly using it.

Soliciting input from teachers may lead teachers to consistently follow the policy.

Uniform use of the policy among math teachers may also positively impact student

achievement. A district wide review of the math assessment policy may allow teachers to

advance their knowledge of student learning gaps as they relate to the standards through

the math assessment policy. It is imperative for the local district to have both

132

administrators and teachers look at the usefulness of the math assessment policy on an

ongoing basis. Continuous evaluation of the policy may ensure that assessment data is

useful and manageable to teachers and administrators (Bardach, 2008, 2011).

Administrators may find that smaller amounts of data are a more effective means of

evaluating teachers. An effective math assessment policy allows math teachers the

opportunity to analyze their own practice in order to address student learning gaps and in

turn assists administrators when they evaluate instruction.

Math teachers must meet the needs of all students. The math assessment policy

was a vehicle with which teachers could analyze data and see those needs. At the local

level, it is important that students’ academic needs are met, when these needs are met the

district can move forward demonstrating academic success. Social change can be

accomplished as teachers and administrators seek to ensure the efficacy of the math

assessment policy on a continuous basis.

This policy evaluation and recommendation has the potential to impact the local

setting. Through the project study, social change may result that benefits teachers,

administrators, and students attending schools in the United States and globally.

Understanding how policies are formed and optimized through effective policy process

may potentially help school systems state wide, nationally, and around the world address

effective policy formation.

Implications for Stakeholders

The local district may apply Sabatier’s (1991) five factors for effective policy

process with the goal of improving both future policy and program development

133

centering on assessment and informed instruction. Teachers, administrators, and students

may experience greater instructional time, more efficient use of assessment resources,

and greater community outreach by following the policy evaluation’s recommendations.

The goal of this policy evaluation was to ensure that the goals the policy had already met

reach far beyond the classroom and school walls. In short, future goals may continue to

be met through continuous longitudinal evaluation of the policy in factors of 10 year

increments as recommended by Sabatier.

Sabatier’s (1991) five factors were used to develop recommendations that focus

on community outreach, understanding how policy is interpreted among teachers and

administrators, math assessment policy that takes NCLB and RTTP into consideration,

and more efficient analysis of assessments data. The recommendations reflected the five

factors to better policy process. In these recommendations, opportunities for math

teachers and administrators to collaborate about student assessment data on a regular

basis may be achieved. This policy evaluation might also benefit parents, students, and

teachers as they expand student learning opportunities and better focus assessment

resources through district and community options. Following through with these

recommendations may also lead to a stronger more collaborative community outside the

school walls. Additionally, these recommendations may provide local community

members with greater ownership of local youth assessment outcomes. As recommended,

authentic assessment that requires students to apply their math knowledge to tasks

facilitated through local business leaders will also provide opportunities for students and

families to see relevance behind assessment as students take the ideas from school and

134

apply them to their larger community. These applications may include engineering

design, medicine, economic viability of creating new local businesses, and community

service projects.

Summary

From the local problem, a research-based policy evaluation was conducted. I

addressed the local problem with input from administrators and teachers. The local policy

evaluation was related to a larger problem that was found nation-wide regarding local

implementation of NCLB and RTTT policies. Data were collected through one-on-one

interviews analyzed in narrative form. After analyzing teacher and administrator

perceptions of the math assessment policy, I conducted the evaluation. Finally, I made

recommendations for future policy implementation and continuing the math assessment

policy.

135

Section 4: Reflections and Conclusions

Discussion

The focus of this policy evaluation was on math assessment within a local district

in the southwest. Administrators and teachers with proximity to the math assessment

policy were interviewed, and these data were analyzed for themes. The aim of this

evaluative study was to collect teacher and administrator perceptions of student and

instructional outcomes related to the policy to determine the policy’s effectiveness. The

policy was seen as a success in meeting its original outcomes overall, and from one

degree to another by all participants interviewed. Recommendations for making future

math assessment policies even more effective and successful within the local district were

explored.

As demonstrated in Section 3, there is research in the area of policy evaluation

from a government, business, and educational standpoint. Researchers have identified

practices within the policy process that lead to effective policy, although these are often

overlooked by organizations (Sabatier, 1991). These practices align with a variety of

policy and program evaluations found in numerous studies discussed in the above

literature review. In this qualitative narrative case study, I inquired about administrators’

and teachers’ views on the local math assessment policy. I created the policy evaluation

to address the efficacy of the math assessment policy in terms of its original outcomes.

This policy analysis and accompanying recommendations are included (see Appendix A).

In this section, the policy evaluation’s strengths and limitations; recommendations for

addressing the evaluation differently; policy evaluation development; leadership and

136

change; self-analysis; reflections; and implications, applications, and directions for future

research are discussed.

Policy Evaluation

Policy evaluation is the means by which a person or group checks the effects of

policies in respect to particular organizations’ needs. Efficacy and validity are looked for

during evaluation, and if they are not present, a means of achieving these goals is

recommended. Efficacy and validity will ultimately improve an organization’s planning

and implementation process for the policy under investigation and future policy. The

organization in this study that was in need of policy evaluation was the local district.

Policies are evaluated to improve them or determine if they should be eliminated (Patton

& Sawicki, 1993; Schuster, 2012; Yarrow, 2000).

Policy evaluation is conducted by first determining what the problem is. In the

case of this study, the problem was that a math assessment policy had never been

evaluated, representing a gap in practice. Once the policy that needs to be evaluated has

been determined, the policy evaluator determines what data need to be collected in order

to determine the efficiency and validity of the current policy, including accessing or

creating instruments to collect data. Next, the evaluator collects the data and analyzes

these data. The evaluator uncovers findings of the evaluation through data analysis.

Finally, the evaluator makes recommendations to the organization regarding

improvement of the policy, effectiveness of the policy, and, in some cases, whether or not

eliminating the policy altogether should be considered (Balthasar, 2011; Ceniviva &

Farah, 2007; Mears, 2010; Sabatier, 2007).

137

Political scientists often evaluate policy, but many people from a variety of

backgrounds can evaluate policy. In education policy, it is not uncommon for teachers,

principals, superintendents, and educational contractors to evaluate policy (Naidu, 2011;

Vesely, 2012). It is not uncommon in many policy evaluation situations for government

and international organizations to evaluate policy. Members or groups of members of

think tanks, universities, and colleges often participate in policy evaluation as well

(Mears, 2010; Murchan, Loxley, & Johnston, 2009; U.S. Department of Education,

2014).

This policy evaluation stemmed from the data collected within a local district in

the southwest regarding its math assessment policy. A well-developed policy evaluation

is essential to creating future effective policies and revising past policies so that they can

attain effectiveness. A qualitative case study using one-on-one interviews to identify

issues within the local policy was used with teacher and administrator volunteers. The

theoretical framework on which the policy evaluation was based was Sabatier’s (2007)

theories of policy process and Bardach’s (2008, 2011) eightfold path for policy analysis.

The five components of the policy process provide organizations with metrics that can be

used to optimize policy making. These five components are the actors, substantive policy

information, the role of policy elites, incorporating longitudinal studies, and differences

in political behavior across policy types.

Sabatier (2007) indicated that successful policy process is difficult, is complex,

and involves many entities that compound its complexity. A key factor in the

effectiveness of the policy process is considering the institution affected by the policy.

138

Sabatier recommended that before beginning the policy process, the following items

should be considered:

1. The importance of policy communities/networks/subsystems involving actors

from numerous public and private institutions and from multiple levels of

government.

The importance of substantive policy information.

The critical role of policy elites vis-a-vis the general public.

The desirability of longitudinal studies of a decade or more.

Differences in political behavior across policy types.

Addressing the aforementioned steps, I have created five critical

recommendations for consideration in evaluating and moving forward with the local math

assessment policy. The evaluation reflects the five items recommended by Sabatier while

telescoping in on the themes uncovered during data collection and analysis. Bardach

warned that careful attention must be paid to semantics to avoid confusion about both

alternatives and criteria for a policy. Throughout these recommendations, careful

attention has been given to avoiding the confusion that can result from different

understandings of criteria.

The policy evaluation consists of five considerations that could be put into play to

improve the math assessment policy or future iterations thereof. These considerations

consist of assessment, how to gain information from meaningful data, community

partnerships, frequency of testing, and further avenues of research. Resources and the

139

roles and responsibilities of those involved in the recommendations that stem from the

policy evaluation will be discussed.

Strengths in Addressing the Problem

This study had several strong points in addressing the local problem. First, this

policy evaluation directly addressed the gap in practice of the math assessment policy in

the local district never having been assessed prior to this study. It also addressed a gap in

the literature, as policy evaluations for local public school districts are few and far

between in academic journals. Most participants, despite the policy’s shortcomings and

potential future roadblocks, agreed that the policy had met its intended outcomes, teacher

collaboration and raising student achievement (senior administrator, personal

communication, 2010). A positive finding like this is a great strength of this study, as it is

very unusual in the literature to find education reform policies that have met their

intended targets (Ackley, 2011; Amrein-Beardsley & Collins, 2012; Ansary, 2010;

Crews, Crews, & Burton, 2013).

A second strength is that recommendations that were made as part of the policy

evaluation came from both evidence in the literature and data collected from teachers and

administrators. The policy evaluation was targeted to address the concerns discussed by

the district personnel interviewed for the study. Carrying out the recommendations from

this study will take considerable time, particularly if the district wishes to implement

longitudinal studies of the policy. These recommendations are rooted in the theoretical

frameworks of this study and are research based, lending credibility and transferability to

140

the results (Bardach, 2008, 2011; Sabatier, 1991, 2007; Weible, Heikkila, & Sabatier,

2012).

A second strength inherent in addressing the problem of evaluating local policy is

that the study required no funding whatsoever to conduct. Future policy evaluations of

the math assessment policy or any district policy could be carried out by other staff

members within the local district, alleviating future difficulties in evaluating policy. The

local district’s Gifted and Talented Coordinator or Integration Liaison, who have

extended planning time of 3 to 4 hours per day, could be used for this process in future

years.

A final strength is that this policy evaluation has the potential to be included as a

feasible part of the district’s unified improvement plan. The Math Assessment Policy

Evaluation and Recommendations (see Appendix A) will be used to document the

themes, findings, and recommendations that stem from evaluation of the math assessment

policy. Policy evaluation involves learning, trial and error, and testing theories (Bardach,

2008, 2011; Sabatier, 2007). Stakeholders, administrators, teachers, and policy makers

must contribute energy and time into addressing a policy’s weaknesses and strengths.

This policy evaluation supports the policy process frameworks and outlines means from

the literature the local district can use to improve the policy and future policy

implementations (Bastedo, 2005; Baumgartner, 2013; Dahill-Brown & Lavery, 2012;

Hajer, 2003).

141

Limitations

All policy evaluations pose individual limitations (Flick, 2014; McMillan &

Schumacher, 2014; Yin, 2014). The project that resulted from this study was not only a

policy evaluation, but also an evaluation of the process by which the math assessment

policy came about. The limitations that existed within the evaluation of the policy process

came from the fact that the efficacy of the policy process was discovered through

interview questions and that there was a small number of math teachers within the

district. Qualitative interview research has inherent limitations (Anderson, 2010; Haas &

Springer, 2014; Merriam, 2014). These include the fact that the researcher is present with

the research participants during data collection. The presence of the researcher may

influence biases during data collection and lead to participants self-censoring.

Considerable effort was made to avoid bias and encourage transparent, confidential

disclosure by participants during data collection, analysis, and reporting of results during

the evaluation. Limitations may exist within the project that resulted from this study.

Recommendations for Addressing the Problem Differently

Understanding the perceptions and experiences of teachers and administrators and

how, if at all, the math assessment policy had met its original outcomes was the focus of

this policy evaluation. It is important to consider other studies that could have been

conducted to solve the problem outside the scope of this evaluation of the policy process.

Notably, all participants indicated that the policy was generated from a top-down

leadership model. Although I sought the origin and outcomes of the math assessment

policy using a narrative case study design, a means of addressing the problem differently

142

would be a survey using a sample larger than the eight-participant sample in this study.

Creswell (2010) defined a survey study as “procedures in quantitative research in which

investigators administer a survey to a sample or to the entire population to describe

attitudes, opinions, behaviors, or characteristics of the population” (p. 388). In this case,

if I had chosen to conduct a quantitative study, I could have used multiple data collection

techniques such as cross-sectional surveys, longitudinal surveys, trend surveys, cohort

surveys, and panel studies. Such research presents opportunities for examining changes in

a population, changes in a subpopulation, changes in the same group of people over time,

data at only one point in time, and trends in the same population over time.

The discovery of participants’ perceptions of the math assessment policy was not

limited to academia. For example, three of the eight members associated the math

assessment policy with noncompliance. Likewise, five of the eight participants cited a

direct correlation between the policy and increased teacher collaboration. The

significance of this finding is important to the educational field and represents an

opportunity for administrators, faculty, and staff to clarify the current math assessment

policy and ensure that they are purposeful in conveyance, adopting modifications that

continue to improve the math assessment policy and make it more meaningful.

Alternative ways of addressing the problem, such as a survey study or a mixed method

approach with a longitudinal study, where a researcher had an opportunity to follow and

survey the same math teachers’ adherence to the math assessment policy over time, could

assist stakeholders and policy makers in making more meaningful policy in the future. A

143

survey study might be the approach necessary to discover a more complete picture of the

number of teachers using or not using the math assessment policy.

Scholarship

In this study, the opportunity to evaluate a policy that I had used for a number of

years was realized. As a result of addressing the local problem, this policy evaluation

allowed me to formulate recommendations to solve problems related to the math

assessment policy. In this policy evaluation, implications for positive social change have

been discovered and methods for future policy evaluations have been explained.

From the local problem, I conducted a research-based policy evaluation. I

addressed the problem with input from administrators and teachers. The evaluation of the

policy process was related to a larger problem that is found nationwide regarding local

implementation of NCLB and RTTT policies. I collected data by conducting one-on-one

interviews, which I analyzed in narrative form. I conducted an evaluation of the local

math assessment policy, having analyzed teacher and administrator perceptions. Finally, I

composed recommendations for future policy implementation and continuation.

Leadership and Change

According to Merideth (2000), teacher leadership consists of risk taking,

effectiveness, autonomy, collegiality, and honor. This project has affected my perspective

regarding teacher leaders. Teachers have the strength to inspire and encourage others to

foster change through actions such as pursuing a doctoral degree despite being the first in

their family to do so. This policy evaluation may lead to improvements within the local

district that benefit administrators, teachers, students, parents, and the community.

144

Communicating regularly with the superintendent, teachers, and administrators improved

my skills as a scholar-practitioner. I was mentored and coached from the onset, and

through its duration, mentors and coaches were an active part of this policy evaluation.

The policy evaluation was a model for change in my district, and as a result of

collaboration with local stakeholders, I discovered that minor weaknesses can often

become assets to the scholar-practitioner.

Cortez-Ford (2006) pointed out that a shared vision of teacher leadership evolves

from ongoing conversations with stakeholders as well as trial and error. When teacher

leadership is understood in this light, the emphasis shifts away from always knowing the

answer, to a problem, to responsibility to seek out and re-evaluate old assumptions and

beliefs. This project study has cultivated a professional mindset in me and has allowed

me to build rapport among educators within the local setting. This has invigorated

integrity and added scrupulousness to the policy evaluation process. Trustworthy leaders

are at the heart of successful schools. Adaptive, open-minded, and creative teachers

evolve into resourceful educational explorers (Jackson, Burus, Bassett, & Roberts, 2010).

Completing this policy evaluation has given me confidence as an educational leader and

practitioner.

Reflection on the Importance of the Work

Evaluation of the math assessment policy was vitally important. A primary reason

that the math assessment was important was that prior to this study, no formal evaluation

of the policy had been conducted. This study made me keenly aware of the importance of

evaluating local policies on a regular basis and avoiding policies that are strictly top-

145

down in nature. Had it not been for this study, decision makers in the local district would

also not have known that employees of the local district using the policy saw value in the

math assessment policy and that the district had achieved many of the outcomes the

policy set out to fulfill.

I have completed doctoral work that will benefit not only the local district, but

potentially schools globally when looking to implement successful local policies for

assessing math. I have learned that each aspect of assessment policy affects the people

that carry out the procedures required of the policy by reviewing literature related to

assessment, policy evaluation, and perceptions about policy. I have also learned that grit

and resolve are essential to conducting a policy evaluation and collecting research data.

Conducting narrative case study interviews helped me to gain insight about teacher and

administrator perspectives of policy and the math assessment policy in general. I

evaluated a policy and made recommendations for future policy process at a local district

in the southwest. Consequently, I have grown into a more effective teacher leader,

scholar, and practitioner.

I believe that one of my successes as a policy evaluator was the fact that this

evaluation served to fill a gap in practice at the local setting. Through this lens I was

successful at balancing the positive aspects and shortcomings of the policy while

avoiding bias. I paid careful attention to avoid inserting my own beliefs and feelings

about the policy and was faithful to the information disclosed to me through data

collection.

146

Self Analysis

Self as Scholar

I have developed professional knowledge that will help me to grow as a scholar of

teacher leadership. I have analyzed and made changes based on my weaknesses and

limitations, which have allowed me to progress in my chosen career field. Learning to

conduct research and evaluate local policies helped me understand how significant

research can be as part of a professional growth plan. The information I have acquired

over the course of this project study as it relates to policy evaluation, along with the

alternative assessment options gleaned from the literature, will broaden my horizons as a

scholar. I will continue to look for ways to improve administrators’ and teachers’ means

of assessing students in math and help to encourage effective policies that facilitate this.

As a doctoral candidate, I discovered that I can become skilled in the area of

policy evaluation research. I have made advancements to my education through this

study. I hope to continue my work as a researcher and scholar by helping the local district

and districts beyond my region evaluate their districts’ policies and processes. I have

achieved an understanding of assessment types and the importance of communication in

the policy process as a scholar committed to continuous improvement.

Self as Practitioner

As a high school math department chairperson and a former science teacher, I

developed knowledge from this doctoral study that will help me as a career professional.

I am certain that my capacity as an educator has improved. I have gained knowledge

about district communication as a result of working with administrators and teachers in

147

this study. Through the literature review, I have acquired expertise that will be beneficial

as I pursue professional opportunities in teacher methodologies and mathematics at

postsecondary institutions. I have gained knowledge of concepts that illustrate not only

an educational institution’s need for strategic planning, but also the need to question why

policies and programs are put into place. As a department chair and high school teacher,

these experiences strengthened my confidence in research based decision making. I hope

my ability to assist schools in effective communication will prove to be a resource for the

educational community.

My doctoral studies have resulted in identifying positive outcomes that stemmed

from the math assessment policy. Through educating local district personnel regarding

effective assessment and policy process, I am better able to make recommendations for

improving the policy beyond these positive outcomes. I am now prepared to help K-12

administrators and teachers create future policies. Community members and stakeholders

may benefit from this new knowledge.

As an evolving practitioner, I grew as a lead teacher and department chair. I was

involved in the decision-making process in the district as a member of the strategic

planning committee. I learned about the processes used within the district to initiate

programs and policies. I have had the opportunity to learn about procedures and policies

in the local district. As a member of the high school’s advisory committee, I also worked

with administrators in coordinating resources that may help meet the social emotional

needs of students. On a regular basis, I also collaborated with high school math teachers

to ensure student and curricular needs were being met through facilitation of the CCSS.

148

Through my roles as a teacher practitioner, I have gained an understanding of the role of

administrators and teachers at the elementary, middle school, and high school levels

during the course of this doctoral study.

Self as Policy Evaluator/Project Developer

To analyze myself as a project developer, I must consider my role as one of policy

evaluator. I applied myself to the evaluation in the manner of researcher, practitioner,

scholar, reviewer of literature, and writer. As a teacher leader, I became a policy

evaluator and made recommendations for better use of current and future policy. I based

this policy evaluation on data gathered from five administrators and three teachers after

transcribing and analyzing their interview data. In this process, I worked to review and

compile relevant recommendations from peer-reviewed literature to enhance the policy

evaluation. These roles guided my development as a policy evaluator and project

developer.

Reflection on the Importance of the Work

This project study filled the gap in practice that was the math assessment policy

lacking an evaluative process from 2005 to present day. The study was an evaluation of

the math assessment policy that is the project deliverable. I learned that more than ten

years is an extremely long time to go without evaluating district policies. Therefore, from

what I have learned, the local district may want to consider further studies on its

procedures and protocols for evaluating district policies. As mentioned in section 2 of this

study, many searches of data on evaluating the math assessment policy were conducted,

but none were found. When probed further within the local district, district senior

149

administration confirmed that the policy had not previously been evaluated (personal

communication, 2012). One of many aspects that are important about this work is that the

local district must strive to study this problem and improve the process for all policies

and procedures including the math assessment policy. The following highlight all of the

components I learned during this evaluation of policy process and components that make

this work important.

First, collaboration among K-12 math teachers within the district is of vital

importance. Math teachers of all grade levels not only have to ensure they are teaching

the standards, but also require a measuring tool that helps them analyze students’

instructional needs. Therefore, I learned it is important to continue with the collaboration

that the math assessment policy formed and promoted. Part of enhancing collaboration

may include pinpointing areas that collaboration has waned and improve the cooperation

among those grade levels and content areas.

Secondly, there are only a small number of math teachers within the district. It

could be beneficial if teachers of similar grade levels trained and collaborated with their

peers on the math assessment policy. However, this approach would require multiple

math teachers at every grade level. A small school district rarely has the resources to

employ more than one math coach or specialist per grade level. Therefore, vertically

articulating to the teachers immediately above or below a teacher’s grade level

particularly in small districts is of great importance. In this way, teachers could verify or

correct whether or not they were meeting the needs of students at subsequent grade

levels. This type of collaboration could contribute to a teacher’s knowledge of gaps in

150

standards and inform their instruction prior to and during each school year. These

teachers could also pass on their assessment data to the teachers in the grade levels above

them on a consistent basis.

Finally, I learned that the district needs to consider how district personnel

understand the math assessment policy. Data collected in this study uncovered a wide

range of interpretations among staff and administration. It would be beneficial to the local

district to develop a partnership with families and local business leaders as a means of

authentic assessment for the district as their interpretations may also vary widely (Gao &

Grisham-Brown, 2011; Longo, 2013; McTier & McGregor, 2011). This process could

take the shape of a program rather than a policy with training provided for personnel

instead of a list of procedures and steps (Anderson, 2005; Johnson, 2011). The district

could also include a component that would help educate community stakeholders about

how to better understand the policy (Bowen, 2011; Loepp, 2013; Manna & Ryan, 2011;

Noguera & Weingarten, 2010; Weiss, 2013).

Implications, Applications, and Directions for Future Research

The outcome of this policy evaluation was positive and showed that policies

created at the local level can meet their intended goals. This increased my understanding

of the policy process. In order to optimize this policy evaluation in the local district, I

recommend seeking authentic assessment, minimizing the number of assessments used

and the data analyzed for informing instruction, as well as addressing students’ social and

emotional needs. Further development of the math assessment policy could also include

community and local business leaders’ assessment of students’ math needs. The extent of

151

this project may be transferable to teachers and administrators in other districts, states,

content areas, and grade levels that want to improve their policy process. I recommend

further study to determine what assessment modalities best serve students in local

districts apart from high-stakes testing.

Future researchers could conduct a mixed-methods or quantitative study

researching student results and outcomes as a means to more directly assess the outcomes

of the math assessment policy. Future researchers could apply a mixed-methods approach

in the form of analyzing assessment and student survey data. This data could include

students’ perceptions of the math assessments they are required to take and the usefulness

of each assessment. Data analysis could be used to support the ideas and themes

identified in this policy evaluation. Qualitative studies could be used to identify when

historically the policy was getting the best results and what students’ perceive as the most

useful instrument. Quantitative studies could include statistical methods to analyze and

assess student results on all current and past assessment resources. This assessment data

could be compared to state archival data. This comparison might help to pinpoint the

most effective assessment in order to minimize testing and optimize student learning.

Conclusion

The results of this policy evaluation showed that the local math assessment

policy’s outcomes were achieved to a degree as perceived by district teachers and

administrators. The policy may be further improved by carrying through the

recommendations made in this project study. I conducted a qualitative study to

investigate how teachers and administrators perceived the outcomes of the math

152

assessment policy in light of its expected student achievement outcomes and informed

instruction in the local district. The final product, a policy evaluation with

recommendations, provides an opportunity for the local district to see further success and

improve outcomes of the policy.

Through research questions, a self-designed interview instrument, and participant

interviews, this study was framed using themes within the transcribed data and the

literature review. In Section 2 of the literature review, I focused on the history of federal

education policies beginning with A Nation at Risk and ending with the NCLB Act of

2001. In Section 3 of the literature review, five factors for successful policy process as

designated by Sabatier (1991) and Bardach’s (2008, 2011) eightfold path were used as

the framework for improvement of the current math assessment policy and better creation

of future policies. I used findings about how individuals interpret and understand policy

and how schools assess mathematics. Guiding questions were addressed through the

theoretical framework. This policy evaluation has the potential to bring about social

change through improved future policy implementation. Furthermore, the policy

evaluation recommends alternative means of assessment, optimizing classroom

instructional time, and targeting focused data to help drive instruction.

Students also viewed