Mock Dissertation

profileLionking
ExceedsExpectations-ChapterThreeRevised1.docx

Chapter Three: Procedures and Methodology

Introduction

The goal of education is to increase student achievement and knowledge of the material being taught. It is imperative that, if this is the real goal of education, search for best practices that assist in increasing student achievement. While many different aspects impact student achievement, expanding the practice efforts of educators to help in the classroom is beneficial (Tucker & Strange, 2020). The idea for the study focused around the theory, while it may be considered old by many in education today, from Benjamin Bloom and mastery learning and the utilization of formative assessments and individualized learning to drive instruction (Guskey, 2010). The purpose of this quantitative study was to determine the strength and nature of the relationship between the level of implementation of the diagnostic assessment software PowerSchool and student achievement on the eighth-grade mathematics TCAP test in a semi-rural system in northeast Tennessee.

The target system for this study served a total enrollment of 5,290 students, grades pre-k through grade 12, and consists of 15 schools and one alternative placement setting. While many of the schools from the target district are considered to perform at proficient levels for student achievement, others in the system are or are in danger of becoming target schools by the Tennessee Department of Education based on student achievement. As with all public schools in the state of Tennessee, all third through eighth-grade students in the target district partake in yearly TCAP testing in ELA, mathematics, science, and social studies. As discussed in Chapter 2, TCAP is a criterion-referenced assessment that, when coupled with TVAAS and value-added, is a reliable and valuable source of data for educators statewide. Chapter 3 discusses the methodology of the research as well as the utilization of the TCAP and TVAAS as sources of data. The chapter begins with an introduction and research paradigm and design before moving through the sampling procedures, data collection sources, statistical tests being utilized, and a summary of the chapter.

Research Paradigm

The review of the literature discussed in Chapter two explained how the use of formative assessments and mastery learning could be used to increase student achievement. Furthermore, as mentioned in Chapter One, the state of Tennessee, as well as the nation, is facing a crisis with a large percentage of today’s students performing below grade-level expectations. For this reason, systems nationwide have implemented programs specifically for assisting in increasing student achievement in mathematics such as Response to Intervention (RTI), and continual search for programs that can further help in this goal of improving student achievement and understanding in mathematics.

The goal of this study was to investigate the relationship between the level of implementation of the diagnostic assessment software PowerSchool and student achievement in eighth-grade mathematics in a semi-rural northeast Tennessee school system. A quantitative study was chosen for the study since quantitative research “entails the collection of numerical data and exhibiting the view of the relationship between theory and research as deductive, a predilection for natural science approach, and as having an objectivist conception of social reality” (Bryman & Bell, 2015, p. 160). The dependent variables for this quantitative study focused on the student different data on achievement results based on the eighth grade Tennessee Comprehensive Assessment Program (TCAP) performed in the spring semester of the 2020-2021 school year. The independent variable was based on the different levels of implementation of the diagnostic assessment software PowerSchool during the 2020-2021 school year in a school district in semi-rural northeast Tennessee.

Research Design

The research design for this study includes a correlation design utilizing the independent samples t-test and Chi-square to measure the strength and nature of the relationship between student achievement and the level of implementation of PowerSchool. A correlation design was chosen due to the desire to realize if and how strong of a relationship exists between the level of implementation and student achievement. One group of classes uses PowerSchool as merely a benchmark testing, making up less than five percent of the time spent utilizing PowerSchool for instructional purposes. In contrast, the second group not only uses PowerSchool for the system-wide benchmark testing but also weekly as formative assessments to drive the daily instruction, making up 50% or more of instructional time spent utilizing PowerSchool for instructional purposes. The research used the PowerSchool program, including criterion-referenced benchmark exams based on Tennessee state standards provided through the program, as well as a second criterion-based test in the Tennessee Comprehensive Assessment Program (TCAP) during the 2020-2021 school year. PowerSchool was implemented through an online platform inside the individual classrooms as well as home access provided at home. The TCAP test was presented in the paper-pencil format during Spring 2021 by exiting eighth-grade students in Northeastern Tennessee.

The researcher chose three different sources to serve as dependent variables, one for each research question, and one for the independent variable for each. The dependent variables were based on the different degrees of measure of student achievement on the 8th grade TCAP mathematics test: individual composite scores, TVAAS value-added for each teacher participating in the study based on TCAP scores, and the level of achievement of each student. The basis for the independent variable was the two levels of implementation of the diagnostic assessment software PowerSchool: a full implementation that was used to drive the curriculum and implementation for benchmark testing purposes only.

Sampling Procedures

Prior to conducting this study, approval was asked for and obtained from the University of the Cumberlands Institutional Review Board (IRB). The target school system chosen for this study has acknowledged that a problem exists with student achievement in TCAP testing, especially in middle school mathematics. For this reason, the system implemented the mandatory use of benchmark testing (three total tests throughout the school year) utilizing the PowerSchool software system-wide during the 2019-2020 school year. Permission was granted to conduct research through the district in question by the curriculum supervisor (see Appendix A).

The targeted semi-rural district located in northeast Tennessee was relatively large for a single district. According to data obtained from the personnel department of the target school district, during the 2020-2021 school year, the district employed 473 professional employees: 10 supervisors, 16 principals, eight assistant principals, 18 system-wide support supervisors (curriculum coaches, testing coordinators, etc.), and 421 classroom teacher. Furthermore, the targeted district consists of 15 schools serving students in grades pre-kindergarten – twelfth grade and one alternative placement school. The 15 schools served 5,290 total students containing 756 students that qualify for special education services. The system is considered “direct serve,” which indicates all students kindergarten – eighth grade receive free breakfast and lunch. Each school in the system qualifies as Title 1 schools. The percentage of the ethnic diversity of the 5,290 total students served during the 2020-2021 school year consisted of 95.7% Caucasian, 2.28% Hispanic, and 2.02% identifying as other.

Due to the nature of the study, a non-random, convenience sampling method was chosen for participants. Convenience samples are defined as the “non-probability sampling method that relies on data collection from population members who are conveniently available to participate in the study” (Convenience, 2019). Because convenience sampling was utilized for this study, the study lacks the desired trait of randomness in sampling. However, the purpose of this study was to identify if a relationship between the level of implementation of the diagnostic assessment software PowerSchool in a local northeast Tennessee school system, thus the research and results may not produce data that can be generalized to an overall population. Furthermore, including all eighth-grade students in the targeted district helps to strengthen the validity of the study.

The targeted system consists of seven middle schools, three of which implementing full PowerSchool (50% of instruction) classified as Group X and five only utilizing the program for benchmark tests only ( less than 5% of instruction time) classified as Group Y. The convenience of using all seven middle schools was appropriate. Of the 5,290 total students served by the district, 404 students were served in eighth grade, represented 8.37 % of the population. For this study, the eighth grades were separated into two groups: Group X consisted of 188 individual students ( n = 188) and four teachers, and Group Y consisted of 216 individual students (n = 216) and four teachers.

Data Collection Sources

This study based the collection of data primarily from the results of the eighth grade Tennessee Comprehensive Assessment Program (TCAP) as well as the value-added results formulated from TVAAS. As previously discussed, the TCAP assessment is assumed to be valid and reliable criterion-based. The TCAP assessments will be completed during April 2021, and results will be finalized and reported back to the system during the summer of 2021. TCAP testing is implemented for all students grades three through eight throughout the state of Tennessee. Once the results are reported back to the system, the system will contact the researcher and provide access to the student’s results in coded form for each individual that are part of the study.

The TCAP test provided the overall composite scores in the subject of mathematics for each student as well as their individual level of achievement. Figure 1 shows an example of the reporting data provided by the TCAP for each student.

Figure 1 Individual student TCAP report

The data received from the TCAP results, as well as the TVAAS value-added reporting for each student and teacher involved in the study were then collected, coded, and organized.

The data was collected from the testing department of the targeted school system. In order to prevent bias testing, the data was organized into two groups based on the level of implementation in the study: Group X (full implementation) and Group Y (benchmark utilization only). The testing department also provided the data in each group without the individual names of teachers, students, schools, or any other personal data that could be used as identifying markers. The student’s data were numbered using three-digit codes beginning with 001 for the analysis of the composite scores as well as the level of achievement and value-added data. Table 1, 2, and 3 represents the manner in which the researcher organized the data.

Table 1. Group X student TCAP data for the 2020-2021 school year.

Student

Composite

Score

Equivalent Level of Achievement

Amount of

Value-added

001

002

003

Table 2. Group Y student TCAP data for 2020-2021 school year.

Student

Composite

Score

Equivalent Level of Achievement

Amount of

Value-added

001

002

003

Table 3. The number of students in each level of achievement for both Group X and Group Y on the TCAP test for the 2020-2021 school year.

Level 1:

Below

Level 2:

Approaching

Level 3:

On Track

Level 4:

Mastery

Group X

Group Y

This method of data collection was chosen in hopes of maintaining confidentiality as well as preventing any bias results from the study.

Statistical Tests

The researcher utilized descriptive and inferential data to analyze the data for this quantitative study to determine if a significant difference exists. The researcher performed independent sample t-tests to analyze the individual student composite scores provided by performance on the TCAP test as well as the amount of value that was provided by the TVAAS value-added report. According to SPSS, independent samples t-test is utilized to compare “the means of two independent groups to determine whether there is statistical evidence that the associated population means are significantly different” (2020). The researcher chose to perform Chi-square to determine if a significant distance exists between the in the number of students in each level of achievement on the 8th Grade TCAP test (Below, Approaching, On Track, Mastered) between classes with different levels of PowerSchool implementation (full implementation as opposed to benchmark usage only). For all three tests, the data was analyzed with a confidence level set at p = 0.05 to determine if a significant difference exists. Table 4 represents the data collection and statistical test matrix the researcher utilized for this study.

Table 4. Data collection and statistical test matrix.

Research Question

Data Collection Sources

Statistical Test

Is there a significant difference in the Tennessee Comprehensive Assessment Program (TCAP) 8th Grade composite math scores between classes with different levels of PowerSchool implementation (full implementation as opposed to benchmark usage only) in a northeastern Tennessee school district?

TCAP (Composite mathematics score)

Independent samples t-test

(correlation)

Is there a significant difference in the number of students at each level of achievement on the 8th Grade TCAP test (Below, Approaching, On Track, Mastered) between classes with different levels of PowerSchool implementation (full implementation as opposed to benchmark usage only) in a northeastern Tennessee school district?

TCAP (level of

achievement)

Chi-square

(correlation)

Is there a significant difference in the amount of value-added on the 8th Grade TCAP test among classes with different levels of PowerSchool implementation (full implementation as opposed to benchmark usage only) in a northeastern Tennessee school district?

TVAAS (amount of value-added)

Independent samples t-test

(correlation)

Summary

As previously discussed in Chapter 2, an increase in accountability for student learning as well as teacher effect on student learning has caused schools to search for systematic solutions to assist in increasing student achievement. The system-wide implementation of the use of PowerSchool for benchmark testing in mathematics (one given at the end of each of the first three, nine-week grading periods) took place during the 2019-2020 school year. As per the state of Tennessee policy, the targeted district partakes in the yearly end of the school year, criterion-referenced TCAP testing. This assessment, along with TVAAS, provides individualized student composite scores, level of achievement (one-four), and value-added data. The entire number of students in eighth-grade mathematics in the targeted district was utilized for this study. The students were separated into two groups: Group X (full implementation of PowerSchool) and Group Y (benchmark testing only).

This chapter presented the research design, sample procedures, data collection sources, and type of statistical testing used to analyze the data. Furthermore, the research paradigm was presented and discussed. This study’s results were obtained through quantitative data produced from TCAP scores of eighth-grade students in a northeast Tennessee school district. The study consisted of the three following research question:

1. Is there a significant difference in the Tennessee Comprehensive Assessment Program (TCAP) 8th Grade composite math scores between classes with different levels of PowerSchool implementation (full implementation as opposed to benchmark usage only) in a northeastern Tennessee school district?

2. Is there a significant difference in the number of students at each level of achievement on the 8th Grade TCAP test (Below, Approaching, On Track, Mastered) between classes with different levels of PowerSchool implementation (full implementation as opposed to benchmark usage only) in a northeastern Tennessee school district?

3. Is there a significant difference in the amount of value-added on the 8th Grade TCAP test among classes with different levels of PowerSchool implementation (full implementation as opposed to benchmark usage only) in a northeastern Tennessee school district?

The analysis of the data collected from the study is contained in Chapter 4 through explanation of how the independent sample t-tests as well as the Chi-square were utilized as well as representation of the process and data provided during the study. The results will be compared used a standard of p = 0.05 to determine if a significant difference exists.

References

Bryman, A., & Bell, E. (2015). Business Research Methods (4th ed., p. 160). Oxford, England:

Oxford University Press.

Convenience sampling . (2019). In Research Methodology. Retrieved from https://research-

methodology.net/sampling-in-primary-data-collection/convenience-sampling/

Guide to test interpretation 2019-2020 TCAP assessment. (2019). In Tennessee Department of

Education. Retrieved from https://www.tn.gov/content/dam/tn/education/testing/

TN1124053_TCAP_EOC_GTI_WEBTAG.pdf

Guskey, T. R. (2010, October). Lessons of mastery learning. Educational Leadership68(2), 52-

57.

SPSS tutorials: independent samples t-test. (2020, March 24). In Kent State University:

University Libraries. Retrieved fromhttps://libguides.library.kent.edu/SPSS/Independent

TTest

Tucker, P. D., & Strange, J. H. (2020). Linking teacher evaluations and student learning.

In ASDC. Retrieved from http://www.ascd.org/publications/books/104136/chapters/The-

Power-of-an-Effective-Teacher-and-Why-We-Should-Assess-It.aspx