Millstone 3: Chapter 4

The problem that led to this quantitative causal-comparative study is, it was not known if and to what extent there are differences in chemistry EOC exam scores and ACT science exam scores for high school chemistry students taught using POGIL pedagogy and high school chemistry students taught using non-POGIL pedagogy. Through an exhaustive review of the literature, the researcher identified a gap. This study is driven by the fact that there exists a gap in the research literature to determine whether POGIL is an effective instructional strategy for improving high school chemistry student EOC exam scores and actual student ACT science scores. The gap in the literature was identified using a detailed history of POGIL pedagogy and how it may be a remedy for improving high school student EOC chemistry scores and performance on the ACT science exam.

The theoretical foundations of the Cognitive Development Theory (CDT) and the Information Processing Model (IPM) were addressed for the current causal-comparative study. The purpose of this quantitative, causal-comparative research study was to determine if and to what extent there are differences in chemistry EOC exam scores and ACT science exam scores for high school chemistry students taught using POGIL pedagogy, and high school chemistry students taught using non-POGIL pedagogy in the state of New MexicoUtah. To best address the problem statement and identified gap in the literature, two research questions and four hypotheses were developed to guide this study: Comment by Robert Widner: citation Comment by Robert Widner: citation

RQ1: To what extent, if any, does POGIL pedagogy produce a statistically significant difference in high school EOC chemistry scores?

H01: POGIL pedagogy does not produce a statistically significant difference in high school EOC chemistry scores.

H1a: POGIL pedagogy does produce a statistically significant difference in high school EOC chemistry scores.

RQ2: To what extent, if any, does POGIL pedagogy produce a statistically significant difference in high school ACT science exam scores?

H02: POGIL pedagogy does not produce a statistically significant difference in high school ACT science exam scores.

H2a: POGIL pedagogy does produce a statistically significant difference in high school ACT science exam scores.

To test the research questions and hypotheses for this study, the researcher conducted a one-way MANOVA to determine if there was a statistically significant difference in high school student SAGE chemistry scores and high school student ACT science scores by comparing the effectiveness of POGIL pedagogy between the two comparison groups. As a result, the one-way MANOVA served as an analysis of high school students taught chemistry using POGIL instructional strategies and high school students taught chemistry using non-POGIL instructional strategies.

Chapter 4 continues with a description of the data collection procedure, the data cleaning process, demographics of the target population, and descriptive findings of the study sample total population and independent sample groups. The chapter will also present details of the data analysis procedures that were used to examine the data, and the statistical analysis results as related to the research questions and hypotheses. The chapter will conclude with a summary and all data analyzed and summarized will have been presented to maintain the anonymity of participants and prevent any individual subject from being identified.

This specific section of Chapter 4 provides a narrative summary of sample characteristics and demographics of the population for the participants in the study. The location selected for this study was a large, public school district, in the state of Utah, that includes multiple high schools. Two high schools were selected within this school district and will be referred to throughout this study with the acronyms POGIL high school and non-POGIL high school.

Both high schools shared similar demographics that included student populations of approximately 2500 students and similar ethnic and special populations. The ethnic populations included approximately 78% Caucasian, 15% Hispanic, 1% African-American, and 1% Asian students. The special populations included approximately 19% socially-economically disadvantaged (SED), 3% English language learners (ELL), and 10% students with disabilities (SWD) (USBE, 2018). Both high schools had similar SAGE chemistry exam proficiencies of 44% for the 2015-2016 school year and 51% for the 2016-2017 school year. Both high schools had similar ACT science exam scores of 20 for the 2015-2016 school year and 21 for the 2016-2017 school year (USBE, 2018). Comment by Robert Widner: A table might prove useful for your reader in which all this information is presented.

Both high schools employed eight to ten science teachers in their science departments with two to three of these teachers teaching chemistry during the school year. Chemistry teachers taught a common chemistry curriculum to ensure student preparation for the required state SAGE chemistry exam administered in the month of March. Chemistry teachers at POGIL high school utilized POGIL instructional strategies in their classroom while chemistry teachers at non-POGIL high school utilized traditional instructional strategies in their classrooms. This use of POGIL pedagogy was assessed using the results from a POGIL teacher demographic survey.

The general population for this study was 11th grade high school students that took chemistry in the state of Utah. The target population of the study was 11th grade high school students that took chemistry during the 2015-2016, 2016-2017, 2017-2018, or 2018-2019 school years and took the SAGE chemistry exam and ACT exam during the same year. The study sample used in this study were derived through de-identified student archival data obtained from a large, public school district in the state of Utah that met the target population criteria. The data contained a total sample size of 316 student participants (158 POGIL and 158 non-POGIL) and excluded any identifiers to ensure the anonymity of participants. Comment by Robert Widner: Verb mismatch

Preparation of the data file. After GCU IRB approval was granted, de-deidentified student archival data was collected and shared with the researcher by the Director of Assessment from a large, public school district in the state of Utah. The de-identified student archival data was provided on a Microsoft Excel spreadsheet and included de-identified student ID number, high school student attended, student grade level, year chemistry taken, chemistry teacher, student GPA (before taking chemistry), SAGE chemistry exam raw score, and ACT science exam raw score. The next step was to clean the data by removing participants with missing or incomplete information. Comment by Robert Widner: “data” is plural Comment by Robert Widner: Consider a table

Missing data in quantitative studies may cause a loss of information, produce an increase in standard errors, reduce statistical power, and weaken generalization of findings (Dong & Peng, 2013; Peng & Chen, 2018). Missing data is quite common in quantitative research and is expected at a rate of 15% to 20% in educational studies (Dong & Peng, 2013). To clean the data, the researcher visually analyzed the Microsoft Excel document and screened the hard copy printout to identify missing values.

The initial target population from this de-identified student archival data consisted of 3,878 potential participants; however, this number was quickly reduced during the data cleaning process. The Microsoft Excel spreadsheet did not provide any SAGE chemistry scores for participants during the 2018-2019 school year, and there was no POGIL High School data provided for the 2017-2018 school year. Consequently, both of these years were eliminated from this study and the 158 POGIL high school and 158 non-POGIL high school students were taken from the 2015-2016 and 2016-2017 school years. Comment by Robert Widner: mismatch

After cleaning the data, the final target sample consisted of 158 high school chemistry students that attended POGIL High School and 158 high school chemistry students that attended non-POGIL High School, from the same school district, for a total sample size of 316 participants. De-identified archival data was used to conduct this study, and all participants were 11th grade students when they were enrolled in chemistry during their 2015-2016, 2016-2017, 2017-2018, or 2018-2019 respective school year. The researcher cleaned the data and removed any participants with a missing SAGE chemistry exam score, a missing ACT science exam score, or a missing GPA (before taking chemistry). Comment by Robert Widner: Note how many in each group.

In addition, the remaining POGIL students and non-POGIL students were paired by matching their GPA’s (before taking chemistry) to ensure the extraneous variable of GPA would not falsely favor either comparison group. Student GPA’s were either matched exactly, to the hundredths place, or within +/- .10 grade points. Finally, any participant who entered chemistry more than four weeks after their school year began of left more than four weeks before the end of the school year was eliminated from the study. The step was necessary to ensure that all participants were enrolled in chemistry 70% or more prior to taking the SAGE chemistry exam in the month of March. After the data was cleaned, the sample size reduced to 316 participants (158 POGIL students and 158 non-POGIL students), which still met the minimum 206 sample size to address the research questions and complete the study. The researcher manually inputted these 316 participants into SPSS version 26 software for inferential statistical analysis. Missing data is usually considered a study limitation; however, it was not considered a limitation in the current study since the sample size was still large enough to conduct the data analysis (Warner, 2008). Comment by Robert Widner: Your reader might be a bit confused here. Either provide a table or go into a bit more detail especially with regards to the “matching” process. Comment by Robert Widner: spelling Comment by Robert Widner: please be sure to fix throughout. Comment by Robert Widner: You might refer to the G*Power analyses and reference the appendix here.

Validity and reliability. The two dependent variables for this study consisted of SAGE chemistry exam scores and ACT science exam scores. The SAGE chemistry exam was created by the New MexicoUtah State Board of Education (USBE)Public Education Department, and all New MexicoUtah school districts are required to utilize this state developed high-stakes assessment to measure student mastery and higher-order thinking skills , teacher effectiveness, create common core course measures, and determine whether chemistry students meet the graduation requirement using the EOC chemistry exam as an alternate demonstration of competency (ADC)(USBE (New Mexico PED, 2018). The SAGE chemistry exam is aligned with the New MexicoUtah Core Science Science Standards (UCNMSS), and and the Next Generation Science Standards (NGSS), in addition to utilizing the Theory of Action that allows New Mexico educators to make curricular decisions to improve student achievement (New Mexico PED, 2018). aAll chemistry teachers are expected to administer the EOC (SAGE) chemistry exam, during the testing window in the month of April (USBE, 2018). last three weeks of each semester, and teachers may use the EOC chemistry exam to replace final exams or to count for not more than 15% of the overall semester grade (New Mexico PED, 2018). The SAGE chemistry exam questions are based on the chemistry assessment blueprint (New MexicoUSBE PED, 2018) and is found in Appendix F.

The SAGE chemistry assessment is designed to measure student progress toward achievement of the Utah Core Standards (UCS), therefore, the validity of the SAGE chemistry score interpretations is dependent on how well the test content is aligned with both student learning expectations as listed in the UCS (USBE, 2018). The SAGE test blueprints specify the depth and range with which each of the standards and content strands will be covered in the test administrations, while linking the SAGE chemistry content-based test score interpretations with the UCS (USBE, 2018). The state of Utah began administering SAGE tests beginning in the 2013 – 2014 school year, and operational field-test administration began in the winter and spring of 2014 (USBE, 2018). The Utah State Board of Education (USBE) contracted with the American Institutes for Research (AIR) to convene panels of Utah educators to recommend proficiency standards for the SAGE assessments in English language arts, math, and science (USBE, 2018). The panel decided to utilize standard setting as the means to identify cut scores on SAGE exams to indicate whether a student had achieved a specific level of proficiency (USBE, 2018).

The panel established that the use of goodness-of-fit statistics would be the best method to determine validity for the SAGE assessments in English language arts, math, and science (USBE, 2018). It was also decided by the panel that the Comparative Fit Index (CFI) and Tucker-Lewis Index (TLI) would be the two indices used to determine the goodness-of-fit for the SAGE chemistry second-order model, the goodness-of-fit between the first-order model and multi-factor depth of knowledge (DOK), and the difference in fit between the first-order model and the multi-factor model (USBE, 2018). The goodness-of-fit was measured at 0.96 for both the CFI and TLI in the goodness-of-fit for the SAGE chemistry second-order model and the goodness-of-fit between the first-order model and multi-factor DOK (USBE, 2018). Both of these models showed a good fit since the threshold value for goodness-of-fit is .95 (USBE, 2018). The difference in fit between the first-order model and the multi-factor model yielded a p = 0.000, which was highly significant and suggests there is a difference in fit between the first-order and multi-factor model (USBE, 2018).

The ACT exam uses construct validity, criterion-referenced validity, and content validity (ACT, 2014a). The validity of the ACT exam reflects its ability to meet professionally established guidelines and standards that users can trust (ACT, 2014a). Validity research is a continuous process at the ACT, and the primary focus is on content validity and predictive validity (ACT, 2016a). Content validity attempts to answer the question of whether a test accurately measures what it was designed to measure (ACT, 2016a). This process involves validation utilizing the ACT College and Career Readiness Standards predicated on years of foundational empirical data (ACT, 2016a). The ACT National Curriculum Survey (NCS) is conducted every three to five years by the ACT to collect data about what entering college students should know and be able to do in order to demonstrate readiness for college-level coursework in math, English, science, and reading (ACT, 2016a). This NCS informs the test blueprint for assessments, and the results from the assessments are utilized to validate the ACT College and Career Readiness Standards and the ACT College Readiness Benchmarks (ACT, 2016a).

Since the ACT exam is a college placement test, one way to determine its validity is to examine its ability to predict first-year college entry-level grades. According to ACT Technical Manual (ACT, 2017), their rationale was to produce a test that measures the level of knowledge and skills, for each student, necessary for success in college. The ACT contains more problem-solving exercises and few measures of fine skills, which allows the exam to serve as a content knowledge assessment instead of an aptitude test (ACT, 2017). Each assessment item undergoes a minimum of sixteen examinations, and all test versions are checked to ensure alignment with current secondary and post-secondary curricula; therefore, the ACT is a valid assessment instrument to determine content-subject area readiness for college (ACT, 2017).

According to ACT (2017), validity evidence for using ACT scores in college course placement decisions can be justified using the median accuracy rates for English, math, social science, and science. The sample population for this study included first-year college students enrolled in two and four-year colleges for the 2013-2015 academic school year (ACT, 2017). The ACT English test had a median accuracy rate of 67% to 68% on two types of college entry-level English courses and the ACT mathematics test had a median accuracy rate between 62% and 67% on seven types of college entry-level math courses (ACT, 2017). The ACT, social science test, had a median accuracy rate between 62% to 66% on six college entry-level social science courses, and the ACT science test had a median accuracy rate between 63% to 65% on two college entry-level science course (ACT, 2017).

The second type of validity research conducted by the ACT is predictive validity which uses research data regarding actual course performance to answer the question whether the ACT exam can reliability predict student performance (ACT, 2016a). Through consistent and constant monitoring, ACT ensures both content validity and predictive validity are being met by the utilization of student performance results and research to modify test blueprints, ACT College and Career Readiness Standards, and the ACT College Readiness Benchmarks (ACT, 2016a).

Instrument reliability for sample. The two instruments used for this study were the SAGE chemistry exam and the ACT science exam. The scale score reliability for the 2017 – 2018 student cohort that took the chemistry SAGE exam produced a median reliability coefficient of 0.93 (USBE, 2018). This value indicates high consistency for the reliability of the 2017 - 2018 chemistry EOC (SAGE) exam (ACT, 2017) and these reliability rates are acceptable according to Wallen and Fraenkel (2000). The standard error of measurement (SEM) is closely related to testing reliability and summarizes the amount of error or inconsistency for test scores (ACT, 2017). The score scales for the chemistry SAGE exam were developed to display constant standard errors for any specific chemistry SAGE score and should be approximately the same for both high-scoring and low-scoring examinees. The 2017 - 2018 chemistry SAGE exam median SEM score was 10.47 (USBE, 2018) which indicates the actual score distribution located in the middle of the score range and not at the score range extremes when conditional SEM might deviate (ACT, 2017). The 2015 – 2016 EOC chemistry raw data was subjected to a univariate procedure to produce summary statistics of student performance (A. Rios, personal email communication).

These summary statistics revealed a normal distribution curve with a mean of 21.9, standard deviation of 7.6, and no skewness or kurtosis for the sample of 4612 students (A. Rios, personal email communication). In addition, the Cronbach’s alpha was calculated at 0.86, which indicates high consistency for the reliability of the 2015-2016 EOC chemistry exam administration (ACT, 2017). The standard error of measurement (SEM) is closely related to testing reliability and summarizes the amount of error or inconsistency for test scores (ACT, 2017). The 2015-2016 EOC chemistry exam SEM score was calculated at 2.86 which indicates the actual score distribution located in the middle of the score range and not at the score range extremes when conditional SEM might deviate (ACT, 2017; Appendix G).

The ACT Technical Manual (2014a) and the ACT Technical Manual Supplement (2016b) were utilized to determine the reliability and SEM for this study, however, a more recent ACT Technical Manual 2017 has been released. The ACT exam construct and blueprint designs remain consistent from year to year (ACT, 2017). While this study will utilize archival data from the 2016 - 2019 ACT exam administration, the ACT Technical Manual (2017) is the most current and updated technical manual available from the ACT.

Reliability coefficients are estimates of test score consistency, and they typically range in value from zero to one, with values near one indicating higher consistency and values near zero indicating low or no consistency (ACT, 2017). The scale score reliability for five of the 2015-2016 National ACT administrations produced a median reliability coefficient for the ACT science subtest of 0.85. This value indicates high consistency for the reliability of the 2015-2016 ACT exam (ACT, 2017) and these reliability rates are acceptable according to Wallen and Fraenkel (2000). The SEM is closely related to testing reliability and summarizes the amount of error or inconsistency for test scores. The score scales for the ACT exam were developed to display constant standard errors for any specific ACT score, or subtest score should be approximately the same for both high-scoring and low-scoring examinees. The 2015-2016 ACT science subtest median SEM score was 2.01 (ACT, 2017). This value indicates the actual scale score distribution located in the middle of the score range and not at the score range extremes when conditional SEM might deviate (ACT, 2017).

Research Question One. The following research question and hypotheses framed this causal-comparative study:

RQ1: To what extent, if any, does POGIL pedagogy produce a statistically significant difference in high school EOC chemistry scores?

H01: POGIL pedagogy does not produce a statistically significant difference in high school EOC chemistry scores.

H1a: POGIL pedagogy does produce a statistically significant difference in high school EOC chemistry scores.

The first research question (RQ1) addressed potential differences between the

effectiveness of the POGIL instructional strategy on SAGE chemistry exam performance

for POGIL taught chemistry students and non-POGIL taught chemistry students.

Research Question Two. The following research question and hypotheses framed this causal-comparative study:

RQ2: To what extent, if any, does POGIL pedagogy produce a statistically significant difference in high school ACT science exam scores?

H02: POGIL pedagogy does not produce a statistically significant difference in high school ACT science exam scores.

H2a: POGIL pedagogy does produce a statistically significant difference in high school ACT science exam scores.

The second research question (RQ2) addressed potential differences between the effectiveness of the POGIL instructional strategy on ACT science exam performance for POGIL taught chemistry students and non-POGIL taught chemistry students. Both research questions attempted to address differences between multiple dependent variables; therefore, a one-way multivariate analysis of variance (MANOVA) was determined to be the most appropriate test (Laerd Statistics, 2015) to analyze if and to what extent there are differences in chemistry EOC exam scores and ACT science exam scores for high school chemistry students taught using POGIL pedagogy and high school chemistry students taught using non-POGIL pedagogy.

The test aligned to the research questions as the one-way MANOVA combines two or more dependent variables to maximize differences between two comparison groups. If the omnibus test for the research questions had statistical significance, the researcher analyzed the dependent variables separately using a univariate one-way ANOVA test on SAGE chemistry exam scores and ACT science exam scores, followed with post-hoc comparisons analyses to determine any potentially statistically significant differences between the groups. If the p-value for either research question did not have statistical significance, the null hypotheses would be accepted, and the researcher would not conduct the one-way ANOVA. The null hypothesis is accepted if the results are not statistically significant. The alternative hypothesis is accepted if the results are statistically significant. Comment by Robert Widner: Citation? Comment by Robert Widner: Spell out first time Comment by Robert Widner: Which post-hoc test was used? Comment by Robert Widner: A bit confusing as each RQ addressed on DV. At this point you would have done the ANOVA already.

Descriptive statistics for the sample data. The study’s total sample size consisted of a total of 316 high school participants. The total sample included 158 eleventh grade high school students taught chemistry using POGIL instructional strategies and 158 eleventh grade high school students taught chemistry using non-POGIL instructional strategies for the 2015-2016 and 2016-2017 school years. The SAGE chemistry score and ACT science score was available for the entire sample ( N = 316) included within the study. Table 2 presents the descriptive statistics for academic performance of the total sample group for SAGE chemistry exam scores and ACT science exam scores. Comment by Robert Widner: Is this a sub-heading? If so refer to APA manual for presentation format.

Table 2.

Descriptive Statistics: Academic Performance for the Total Sample Group on Chemistry SAGE and ACT Science Assessments.

The chemistry SAGE exam mean score for the POGIL high school was 845.76 and had a standard deviation of 30.43, while the non-POGIL high school had a mean score of 822.10 and a standard deviation of 45.06. The minimum and maximum chemistry SAGE exam scores for the POGIL high school was 722 and 905, respectively, while the minimum and maximum exam scores for the non-POGIL high school was 722 and 914, respectively. The ACT science exam mean score for the POGIL high school was 23.22 and had a standard deviation of 4.41, while the non-POGIL high school had a mean score of 21.93 and a standard deviation of 4.28. The minimum and maximum ACT science exam scores for the POGIL high school was 14 and 33, respectively, while the minimum and maximum exam scores for the non-POGIL high school was 11 and 35, respectively. Comment by Robert Widner: I would summarize a few major points from the table and not report what is already clear in the table. This help to reduce redundancy. Please refer to some journal articles to give you an idea.

Sample distribution. It is essential to evaluate sample distribution (skewness and kurtosis data) to determine whether the sample is normal distributed. Nearly all statistical methods that are commonly used in the social sciences or psychology are grounded on the premise that collected data are normally distributed (Cain, Zhang, & Yuan, 2017). This data sample was not normally distributed. Comment by Robert Widner: Citation?

Figures 1 and 2 show histograms, with an imposed normal curve, that were the result of the data based on overall mean scores for each of the dependent variables (SAGE chemistry exam scores and ACT science exam scores), which helped visually illustrate what a normal data distribution resembles. The potential skewness and kurtosis were visually identified using the imposed normal curve. If the skewness is less than -1 or greater than 1 standard deviation, the distribution is classified as highly skewed and lacking symmetry (Cain et al., 2017). Kurtosis is a measure of whether the data are light- tailed lack of outliers) or heavy-tailed (outliers) relative to a normal distribution (Cain et al., 2017). The test score frequencies did not display normality and the histogram was an effective tool to display both skewness and kurtosis. The Shapiro-Wilk test is a more robust test for normality that will be presented in the results section of this chapter. Comment by Robert Widner: Try to keep the figures nearer to the discussion (physically placed closer).

The SAGE chemistry exam scores were not normally distributed. Table 3 contains the descriptive statistics used to calculate the skewness and kurtosis values for the dependent variables. The chemistry SAGE mean score change was 833.93 and had a standard deviation of 40.17. The skewness of the chemistry SAGE mean score was -.80, and the kurtosis was .40. The minimum chemistry SAGE mean score was 722, and the maximum mean score was 914. Figure 1 displays the distribution of the total sample groups’ chemistry SAGE exam scores, which comprises the values of this dependent variable. While the chemistry SAGE score showed a skewness of -.80, which falls within the acceptable limits (Cain et al., 2017), it is recommended that a z score calculation be performed to determine whether a normal distribution exists for the sample (Morgan, Leech, Gloeckner, & Barrett, 2013). This calculation required dividing a variable’s skewness and kurtosis value minus zero by its standard error to provide the z score for skewness and kurtosis (Morgan et al., 2013). For SPSS, a z score value less than – 2.58 or greater than 2.58 at p < .01, then the z score for skewness or kurtosis is significant and the sample is not normally distributed (Ghasemi & Zahediasl, 2012).

When the z score calculation was performed on skewness of the SAGE chemistry score with a standard error of .14 (i.e., Z skewness = -.80 – 0/.14), the result yielded a skewness value of – 5.71. When the z score calculation was performed on kurtosis of the SAGE chemistry score with a standard error of .27 (i.e., Z kurtosis = .40 – 0/.27), the result yielded a kurtosis value of 1.48. These values did not produce a normal distribution, but a negatively skewed sample that was kurtosed (Laerd, 2018) (See Figure 1).

The ACT science exam scores were not normally distributed. Table 3 contains the descriptive statistics used to calculate the skewness and kurtosis values for the dependent variables. The ACT science mean score change was 22.58 and had a standard deviation of 4.39. The skewness of the ACT science mean score was .38, and the kurtosis was .07. The minimum ACT science mean score was 11, and the maximum mean score was 35. Figure 2 displays the distribution of the total sample groups’ ACT science exam scores, which comprises the values of this dependent variable.

When the z score calculation was performed on skewness of the ACT science score with a standard error of .14 (i.e., Z skewness = .38 – 0/.14), the result yielded a skewness value of 2.71. When the z score calculation was performed on kurtosis of the ACT science score with a standard error of .27 (i.e., Z kurtosis = .07 – 0/.27), the result yielded a kurtosis value of 0.26. These values did not produce a normal distribution, but a positively skewed sample that was not kurtosed (Laerd, 2018) (See Figure 2).

Table 3. Comment by Robert Widner: Do you think it might be informative to also provide the same analyses but break it down into the two groups?

Descriptive Statistics: Skewness and Kurtosis for the Total Sample Group on Chemistry SAGE Exam Scores and ACT Science Exam Scores

Figure 1. Histogram of mean chemistry SAGE scores with imposed normal curve.

Figure 2. Histogram of mean ACT science scores with imposed normal curve.

This section provided detailed information about data analysis procedures prior to conducting actual statistical tests. Data analysis procedures included several steps that included identifying the target population, preparing the data file, addressing tests of assumption, conducting specific parametric statistical tests, determining the results of statistical procedures for each research question with justification for alignment, and listing the limitations. The data analysis testing proceeded as planned; therefore,

there were no changes to the data analysis required. Laerd Statistics served as the primary Comment by Robert Widner: Citation with date

statistical source for this study. The purpose of this quantitative, causal-comparative research study was to determine if and to what extent there are differences in chemistry SAGE exam scores and ACT science exam scores for high school chemistry students taught using POGIL pedagogy, and high school chemistry students taught using non-POGIL pedagogy in the state of New MexicoUtah.

The data analysis section presented all information up to the actual data analysis in chronological order. The unit of analysis for academic achievement indicated by chemistry SAGE exam scores and ACT science exam scores was at the individual level. The dependent variables, chemistry SAGE exam scores and ACT science exam scores, and the independent variable, presence of POGIL as an instructional intervention indicated by the categorical variable labeled school (POGIL or non-POGIL high school) was examined. Both the independent and dependent variables excluded any identifiers to subjects through the use of de-identified student archival data provided by the director of assessment from a large, public school district in the state of Utah. The researcher assumed that the data file provided by the director of assessment’s office were valid and accurate for all student participants included within the archival data set. The data file was cleaned prior to the implementation of any data analysis within this chapter. Comment by Robert Widner: plural Comment by Robert Widner: Cut from here and place into the Assumptions section of your paper. Comment by Robert Widner: this should have been done as the first step and prior to obtaining Descriptive information. Once done this step need not be reported again as it is assumed that the “final” data set on which the data analyses were performed has been cleaned.

All tests of assumptions were addressed before statistical analysis was conducted, and all tests of assumption were thoroughly addressed in the results section stated under each research question. For the one-way MANOVA, all assumptions were met except for assumptions of univariate or multivariate outliers (outliers did not change the outcome), multivariate normality, homogeneity of variance-covariance matrices, and homogeneity of variances. Violations of these four assumptions were justified as the one-way MANOVA is robust to violations with a large sample size (Tabachnick & Fidell, 2019). All assumptions were met for the one-way ANOVA except for assumptions of univariate or multivariate outliers (outliers did not change the outcome), multivariate normality, and homogeneity of variances, which were justified as the one-way ANOVA is robust to violations with a large sample size (Ghasemi & Zahediasl, 2012). All assumptions were met for the two-tailed Independent Sample t-test except for assumptions of univariate or multivariate outliers (outliers did not change the outcome), multivariate normality, and homogeneity of variances, which were justified as the Independent Sample t-test is robust to violations with a large sample size (Ghasemi & Zahediasl, 2012). Comment by Robert Widner: plural

Tests of assumptions. The current study utilized SAGE chemistry exam scores and ACT science exam scores, which were quantitative in nature. Research questions one and two were assessed using parametric statistical tests that included a one-way MANOVA and a one-way ANOVA, which required that the data analyzed be measured

at the continuous level (Laerd Statistics, 2015). Assumption one was satisfied for the

study as there are two dependent variables (SAGE chemistry exam scores and ACT science exam scores) measured on a numeric scale. The following presented the tests of assumptions by research question. The two research questions for this study included: Comment by Robert Widner: Present at the beginning of this section.

RQ1: To what extent, if any, does POGIL pedagogy produce a statistically significant difference in high school EOC chemistry scores?

H01: POGIL pedagogy does not produce a statistically significant difference in high school EOC chemistry scores.

H1a: POGIL pedagogy does produce a statistically significant difference in high school EOC chemistry scores.

RQ2: To what extent, if any, does POGIL pedagogy produce a statistically significant difference in high school ACT science exam scores?

H02: POGIL pedagogy does not produce a statistically significant difference in high school ACT science exam scores.

H2a: POGIL pedagogy does produce a statistically significant difference in high school ACT science exam scores.

A discussion of the assumptions is presented in the following section.

Sample distribution . It is important to evaluate sample distribution (skewness and kurtosis data) (see Table 3) in this section to provide a review of the sample. Most commonly used statistical methods in the social sciences or psychology are based on the premise that collected data are normally distributed (Cain, Zhang, & Yuan, 2017). This sample was not normally distributed.

Figures 1 and 2 show histograms with an imposed normal curve that resulted from the data based on overall mean composite scores from the combined dependent variables (SAGE chemistry exam scores and ACT science exam scores), which helped illustrate if the data was normally distributed. The normal curve imposed on the histograms visually identified potential skewness or kurtosis. If skewness is less than -1 or greater than 1 standard deviation, the distribution is characterized as highly skewed and lacks symmetry (Cain et al., 2017). Kurtosis is a measure of whether the data are heavy-tailed (outliers) or light tailed (lack of outliers) relative to the normal distribution (Cain et al., 2017). The frequencies and distribution of test scores across the board did not display normality, and the histograms were effective graphs to display both skewness and kurtosis. The Shapiro-Wilk test is a more robust test for normality that will be present in the results section. Comment by Robert Widner: Verb form

The SAGE chemistry exam score histogram data was not normally distributed. The mean value of the SAGE chemistry exam scores had a negative skewness of - .80 (standard error = .14) and had a positive kurtosis of .40 (standard error = .27). This skewness value and kurtosis value did not produce a normal distribution, but a kurtosed distribution (Laerd, 2018) (see Figure 1).

The ACT science exam score histogram data was not normally distributed. The mean value of the ACT science exam scores had a positive skewness of .38 (standard error = .14) and had a positive kurtosis of .07 (standard error = .27). This skewness value and kurtosis value did not produce a normal distribution, but a kurtosed distribution (Laerd, 2018) (see Figure 2).

One-way MANOVA . Research Question One and Research Question Two were tested with a one-way MANOVA. Before conducting a one-way MANOVA, there are ten assumptions that must be addressed. These assumptions consisted of (1) there are two or more dependent variables measured on a continuous level, (2) one independent variable with two or more categorical, independent groups, (3) independence of observation, (4) no univariate or multivariate outliers, (5) multivariate normality, (6) no multicollinearity, (7) a linear relationship between dependent variables for each independent group, (8) an adequate sample size, (9) homogeneity of variance-covariance matrices, and (10) homogeneity of variances (Laerd Statistics, 2015).

The first assumption is that the study has two or more dependent variables that are measured on a continuous level (Laerd Statistics, 2015). This assumption was satisfied for the study as there are two dependent variables (SAGE chemistry exam scores and ACT science exam scores) measured on a continuous level. The SAGE chemistry exam scores were measured on a raw score scale of 684 to 983, and the ACT science exam scores were measured on a raw score scale of 1 to 36. The first assumption was satisfied. Comment by Robert Widner: Redundancy in this section

The second assumption is that the study has one independent variable with two or

more levels (Laerd Statistics, 2015). The term level is normally reserved for groups that Comment by Robert Widner: This may not be clear to readers unfamiliar with statistics. Please provide more detail by what is meant with the term “order”.

have an order (Laerd Statistics, 2015). The study has one independent variable (presence of POGIL) with two levels (POGIL taught students and non-POGIL taught students).

The second assumption was satisfied.

The third assumption is that there is an independence of observation where there is no relationship between the participants in each level (Laerd Statistics, 2015). Having

different student participants in each level is a way to address this assumption (Laerd Statistics, 2015). Assumption three was met since de-identified, student archival data had different participants in each of the two levels. The third assumption was satisfied.

The fourth assumption states there should be no univariate or multivariate outliers

Present in the sample (Laerd Statistics, 2015). This assumption is normally tested in SPSS by following the Explore procedures and visually analyzing boxplots to detect outliers. Any data points that are more than 1.5 box-lengths from the edge of their box are classified as outliers by SPSS and are noted by circular icons on the boxplot diagrams. Data points that are more than three box-lengths away from the edge of their box are noted by an asterisk symbol on the boxplot diagram. This technique may not be a completely trustworthy method; however, it is a straightforward approach for identifying outliers (Laerd Statistics, 2015). Comment by Robert Widner: Not capitalized Comment by Robert Widner: Consider cutting

For the first portion of the fourth assumption, the presence of univariate outliers for the two dependent variables were examined. For SAGE chemistry exam scores, two univariate outliers were observed with low score values of 722 for both outliers (see Figure 3). For ACT science exam scores, four univariate outliers were observed with high score values of 33, 33, 33, and 35 respectively (see Figure 4). The assumption for no univariate outliers was not met; however, all six outliers were retained due to the large sample size and removing them did not alter the outcome (see Appendix H). Boxplots of SAGE chemistry exam scores and ACT science exam scores are presented in Figures 3 and 4, respectively. Comment by Robert Widner: verb Comment by Robert Widner: Why did you keep them?

Figure 3. Boxplot of Chemistry SAGE scores for POGIL and non-POGIL high school

Figure 4. Boxplot of ACT science scores for POGIL and non-POGIL high school

For the second portion of the fourth assumption, the presence of multivariate outliers for the two dependent variables were examined. Mahalanobis distance was used (Laerd, 2015) to determine the presence of any multivariate outliers among the dependent variables. The result of Mahalanobis distance produced a maximum score of 16.54 for the dependent variables. For the one-way MANOVA with two dependent variables, the maximum critical alpha value for Mahalanobis distance is 13.82 (Laerd, 2015). Since the Mahalanobis distance score produced two outliers that exceeded the critical alpha value (16.54), the dependent variables contained multivariate outliers and the second portion of the sixth assumption was not met. The researcher made the decision to keep the two outliers that exceeded the Mahalanobis distance critical alpha value since the one-way MANOVA is robust to multivariate outliers with a large sample ( N = 316) (Laerd, 2015). Comment by Robert Widner: verb Comment by Robert Widner: It is better to remove them – this will likely make your reader feel more comfortable. Or at least preset the analyses and demonstrate similar results with and without the outliers. Comment by Robert Widner: What is meant by a “large sample”? Same issue whenever you use this term. We need an operational definition.

The fifth assumption states that there needs to be multivariate normality. This means that there should be normally distributed data for each level in the independent

variable (Laerd Statistics, 2015). This assumption is commonly tested by utilizing the

Shapiro-Wilks test for normality in SPSS., and with a sample size less than 50 participants. The significance score is the most important number to evaluate the Shapiro-Wilks test for normality. If all of the Sig. values are greater than 0.05 ( p > 0.05), then the dependent variables are normally distributed for each group of the independent variable. All of the values were less than .05 and therefore, the distributions were not normal. The Shapiro-Wilk Test of Normality results are presented in Table 4.

Table 4.

Shapiro-Wilk Test of Normality

According to Laerd Statistics (2015), if the sample size is greater than 50, it is

recommended that graphical methods such as the Normal Q-Q Plot be used due to the fact that at larger sample sizes the Shapiro-Wilk test flags minor deviations from normality as statistically significant (Laerd Statistics, 2015). Normal Q-Q Plots were

generated to further assess the normality of the distributions. When the distribution is normal for the Normal Q-Q Plots, the plotted values form a straight line. Examination of the Normal Q-Q Plots revealed that the distributions did not form a straight line and therefore, were not normal (see Appendix J). However, with large sample sizes, violating the normality assumption is not problematic (Ghasemi & Zahediasl, 2012). Comment by Robert Widner: Again you need to operationalize this term.

The sixth assumption states that there should be no multicollinearity, which

translates that the dependent variables should be somewhat correlated with each other (Laerd Statistics, 2015). The threshold for determining whether multicollinearity was present was r > .70 (Crossley, Subtirelu, & Salsbury, 2013). Utilizing the Bivariate procedure in SPSS, the researcher conducted Pearson correlations between the dependent variables (SAGE chemistry exam scores and ACT science exam scores) to determine whether or not there is a correlation between the variables (Laerd Statistics, Comment by Robert Widner: “should not be” Comment by Robert Widner: There are many different types of correlation coefficients. Justify for your reader why you opted for the Pearson correlation.

2015). A moderate correlation was observed between the two dependent variables, r (316) = .59, p < .001, two-tailed. Since the two dependent variables (SAGE chemistry exam scores and ACT science exam scores; see Table 5) in the MANOVA correlations were below the threshold of .70 (Crossley et al., 2013), the sixth assumption was satisfied. A correlation matrix is presented in Table 5. Comment by Robert Widner: Define what is a “moderate” correlation and provide a citation.

Table 5.

Correlation Matrix Comment by Robert Widner: Not really a matrix. Maybe combine the two rows into one row?

The seventh assumption states that there should be a linear relationship between the dependent variables for each group of independent variables (Laerd Statistics, 2015). A scatterplot matrix for each group of the independent variables identifies if there is a linear relationship (a straight line) or not (a curved line). If the dependent variables are not linearly related, there is a loss of ability to identify differences (Laerd Statistics, 2015). In SPSS, after splitting the data file to separate out the independent levels, the Legacy Dialogues/Scatter/Dot procedure was utilized to assess linearity through scatterplot (Laerd Statistics, 2015). Since the inter-correlations were statistically significant, linear relationships were observed between the dependent variables. A scatterplot matrix for SAGE chemistry exam scores and ACT science exam scores for POGIL high school and non-POGIL high school is presented in Figure 5 and Figure 6, respectively.

Figure 5. Scatterplot matrix for Chem SAGE scores and ACT science scores for POGIL high school.

Figure 6. Scatterplot matrix for Chem SAGE scores and ACT science scores for non-POGIL high school.

The eighth assumption states that there should be an adequate sample size. Laerd

Statistics (2015) states that the larger the sample size the better, but at a minimum, there

needs to be as many participants in each group of the independent variable as there are

number of dependent variables. Since there are two dependent variables and one independent variable with two groups (levels) in the current study, the minimum sample size would be eight; however, GCU requires a minimum of 50 participants per

independent variable level. Therefore, since the current study has two independent variable levels, the minimum sample size would be 100 participants. The G* Power 3.1 software yielded a total sample size of 158 participants for the current study; however, the actual sample size obtained by the researcher was 158 POGIL students and 158 non-POGIL students, for a total sample size of 316 participants. Assumption eight, demonstrating adequate sample size, was satisfied as observed power to detect a medium effect for the one-way MANOVA (see Appendix E). Comment by Robert Widner: Reference appendix where you have captured the output from the G*Power software. Comment by Robert Widner: Sentence structure

The ninth assumption states that there should be homogeneity of variance-covariance matrices (Laerd Statistics, 2015). After unsplitting the file, the assumption Comment by Robert Widner: ?

was tested by utilizing Box’s M test of equality of covariance (see Table 6). The

important row is the significance level ( p-value) of the Box’s M test. If the test is not

statistically significant (i.e., p > .001), there is homogeneity of variance-covariance

matrices and this one assumption would not be violated (Laerd Statistics, 2015). Box’s M

= 28.23, p < .001. Therefore, the assumption was violated (See Table 6). The one-way

MANOVA was still conducted because it is robust to violations with a large sample size;

however, this may distort the alpha level (Warner, 2008).

Table 6.

Box’s Test of Equality of Covariance Matrices

Finally, the tenth assumption states that in order to run a one-way MANOVA there should be homogeneity of variances. Assuming the assumption of homogeneity of

variance-covariance matrices were not violated, the next step is to utilize the Levene’s

test of equality of variances procedure in SPSS (Laerd Statistics, 2015). The one-way

MANOVA makes the assumption that there are identical variances between the levels of

the independent variable for each dependent variable (Laerd Statistics, 2015). The

important part of the output is the significance level ( p-value) of the test (Laerd Statistics,

2015). If the test is not statistically significant ( p > .05), there is similar variances

and the expectation of homogeneity of variances has not been broken (Laerd Statistics,

2015). For the SAGE chemistry exam scores, the significance was less than .05 ( p = .000) for the variables of interest as indicated in Table 7, which means that this assumption was violated; however, for the ACT science exam scores, the significance was greater than .05 ( p = .863) for the variables of interest as indicated in Table 7, which means that this assumption was satisfied. The majority of the assumptions were met, despite four of the ten assumptions being violated. The one-way MANOVA was still run because it is robust to violations with a large sample size (Warner, 2008).

Table 7.

Levene’s Test of Equality of Error Variance

One-way ANOVA . If the one-way MANOVA produced statistically significant results, a one-way ANOVA would be run for each dependent variable. Similar to the one-way MANOVA, in order to run a one-way ANOVA, six assumptions were considered one at a time to ensure the data could actually be analyzed using this test which consisted of (1) continuous dependent variable, (2) independent variable has two or more categorical, independent groups, (3) independence of observations, (4) no significant outliers in the groups of independent variables, (5) dependent variable should be normally distributed for each group of the independent variable, and (6) homogeneity of variances (Laerd Statistics, 2017). These assumptions were previously tested with the one-way MANOVA test.

Two-tailed Independent Sample t-test . If either of the one-way ANOVA test produced statistically significant results, a two tailed Independent Sample t -test would be run for each dependent variable. Similar to the one-way MANOVA, in order to run a one-way ANOVA, six assumptions were considered one at a time to ensure the data could actually be analyzed using this test which consisted of (1) continuous dependent variable, (2) independent variable has two or more categorical, independent groups, (3) independence of observations, (4) no significant outliers in the groups of independent variables, (5) dependent variable should be normally distributed for each group of the independent variable, and (6) homogeneity of variances (Laerd Statistics, 2017). These assumptions were previously tested with the one-way MANOVA test.

This section presents the data analysis and results for each research question and the hypotheses which include the data screening for assumptions of multivariate analysis of variance, descriptive statistics, and inferential statistics of the one-way MANOVA. Two research questions were addressed for this quantitative, causal-comparative study and addressed whether there are any differences in SAGE chemistry exam scores and ACT science exam scores for high school chemistry students taught using POGIL pedagogy and high school chemistry students taught using non-POGIL pedagogy. For the purposes of this study, SAGE chemistry exam score and ACT science exam scores served as indicators of academic achievement for the sample included in this study.

Results by research questions and hypotheses. Two research questions and related hypotheses were defined to address the impact of POGIL pedagogy on student achievement on SAGE chemistry exam scores and ACT science exam scores. The research questions, related hypotheses, and results according to respective hypothesis follow.

RQ1: To what extent, if any, does POGIL pedagogy produce a statistically significant difference in high school EOC chemistry scores?

H01: POGIL pedagogy does not produce a statistically significant difference in high school EOC chemistry scores.

H1a: POGIL pedagogy does produce a statistically significant difference in high school EOC chemistry scores.

RQ2: To what extent, if any, does POGIL pedagogy produce a statistically significant difference in high school ACT science exam scores?

H02: POGIL pedagogy does not produce a statistically significant difference in high school ACT science exam scores.

H2a: POGIL pedagogy does produce a statistically significant difference in high school ACT science exam scores.

Table 8, Table 9, and Table 10 present respective descriptive and inferential statistics for the sample group in this study. Table 8 shows the descriptive statistics for POGIL high school students’ and non-POGIL high school students’ academic performance on the SAGE chemistry exam and ACT science exam. Table 9 exhibits the results of the multivariate analysis of variance tests for SAGE chemistry exam scores and ACT science exam scores for POGIL high school students and non-POGIL high school students, while Table 10 presents tests of between-subjects effects.

Table 8. Comment by Robert Widner: This information has already or should have been presented in the Descriptive Statistics section previously. So refer to the table presented in that section. Don’t repeat here.

Descriptive Statistics

As shown in Table 8, each independent group had a total of 158 participants with SAGE chemistry exam scores and ACT science exam scores. A closer examination of the descriptive statistics shows a higher mean of ( M = 845.76, SD = 30.43) for POGIL students than non-POGIL students ( M = 822.10, SD = 45.06). Additionally, Table 8 shows a higher mean for POGIL students ( M = 23.22, SD = 4.4) than non-POGIL students ( M = 21.93, SD = 4.28) for ACT science exam scores. While the respective groups reflect higher mean scores per category, the indication of statistical significance is determined by the multivariate test that follow.

Table 9.

Multivariate Tests

As presented in Table 9, the results of the multivariate main effect demonstrated that there was a statistically significant difference between the POGIL student group, exposed to the intervention and non-POGIL student group, unexposed to the intervention when considered jointly on the dependent variables of SAGE chemistry exam scores and ACT science exam scores. While the Wilks' Λ is normally used to interpret the multivariate test results, this study sample violated the homogeneity of variance-covariance matrix assumption (Box M test), and the Pillai’s Trace value will be used to interpret the multivariate test results (Laerd, 2015). Specifically, the Pillai’s Trace = .088, F(15, 313) = 15.11, p = .0005, partial η2 = .088, p < .05. As a result of a significant difference outcome for the one-way MANOVA, a separate one-way ANOVA was conducted for each dependent variable (univariate level) with each ANOVA evaluated at an alpha level of .05.

Table 10. Comment by Robert Widner: Generally output from SPSS would be placed into the appendix and just discussed in the body of your paper. Check other GCU dissertations for examples. I will share another one of my student’s dissertation (using a causal-comparative approach) once we have met for a ZOOM call.

Tests of Between-Subjects Effects

As presented in Table 10, the one-way ANOVA result for chemistry SAGE exam scores dependent variable was F(1,44217.23) = 29.91, p = .0005, partial η2 = .087. The level of statistical significance was adjusted for multiple comparisons, and a Bonferroni correction was made in order to accept statistical significance for the univariate ANOVAs while controlling for Type I error rates (Laerd, 2015). The level of statistical significance was set a p < .025 rather than p < .05 since there were two dependent variables for the sample size (Laerd, 2015). Also presented in Table 10, the one-way ANOVA result for ACT science exam scores dependent variable was F(1, 131.70) = 6.98, p = .009, partial η2 = .022. The level of statistical significance was adjusted for multiple comparisons, and a Bonferroni correction was made in order to accept statistical significance for the univariate ANOVAs while controlling for Type I error rates (Laerd, 2015). The level of statistical significance was set a p < .025 rather than p < .05 since there were two dependent variables for the sample size (Laerd, 2015). Table 11 summarizes the ANOVA results for chemistry SAGE scores and ACT science scores between the two groups, with both dependent variables exhibiting statistically significant p-values ( p < .025) of p = .0005 and p = .009, respectively. In addition, Table 12 displays the Welch’s ANOVA equality of means results for the data sample. The Welch’s ANOVA was used since it is a more reliable test when the two samples have unequal variances (Ruxton, 2006). Table 12 summarizes the Welch’s ANOVA results for chemistry SAGE scores and ACT science scores between the two groups, with both dependent variables exhibiting statistically significant p-values ( p < .025) of p = .0005 and p = .009, respectively. Comment by Robert Widner: Some redundancy Comment by Robert Widner: good

Table 11. Comment by Robert Widner: Appenidx.

ANOVA

Table 12.

Robust Tests of Equality of Means

Since the one-way ANOVA resulted in statistically significant p-values for chemistry SAGE scores and ACT science scores between the two groups, a two-tailed Independent Sample t-test was run for each of the dependent variables. Table 13 shows the output results for the two dependent variables, chemistry SAGE exam scores and ACT science exam scores. A Levene’s test was run to determine whether an equality of variance could be assumed for each of the two dependent variables. The chemistry SAGE exam score produced a statistically significant p-value of p = .0005, ( p < .05) which is statistically significant and violates the assumption of homogeneity and causes equal variances for the dependent variable to not be assumed (see Table 13). The ACT science exam score did not produce a statistically significant p-value of .86, ( p > .05) which meets the assumption of homogeneity and causes equal variances to be assumed for the dependent variable (see Table 13). Comment by Robert Widner: So did you do both a “standard” ANOVA and a Welch’s? I would report the results from both approaches (especially if the outcomes are similar).

Table 13 also displays the t-test for equality of means for both dependent variables. While the chemistry SAGE dependent variable violated the assumption of homogeneity, the equal variances not assumed row was used for the Sig (2-tailed) result. This p-value is stated as p = .000; however, the true p-value for the chemistry SAGE score is p = .00000010143 or p < .0005, which shows a mean difference that is statistically significant ( p < .05) (see Table 13). The ACT science dependent variable satisfied the assumption of homogeneity, and the equal variances assumed row was used for the Sig (2-tailed) result. This p-value is stated as p = .009 which shows a mean difference that is statistically significant ( p < .05) (see Table 13). Comment by Robert Widner: When reporting your values adopt a specific number a specific number of decimal places and use throughout your paper. Generally we don’t go beyond three decimal places.

Table 13.

Independent Samples Test

Hypothesis 1. Research hypothesis 1 stated that that there is a statistically significant difference in the chemistry SAGE exam scores for POGIL high school students and non-POGIL high school students. As discussed, the one-way MANOVA demonstrated a significant effect for the two dependent variables (chemistry SAGE scores and ACT science scores) of p = .0005 ( p < .05) (See Table 9). At the univariate level, ANOVA for chemistry SAGE exam scores, which indicate change in chemistry academic achievement level for the independent groups, demonstrated that there was a significant difference between POGIL high school participants and non-POGIL high school participants on chemistry SAGE exam scores of p = .0005 ( p < .025) (see Tables 10 and 12). The two-tailed Independent Sample t-test for the chemistry SAGE scores, produced as statistical significance of p = .0005 ( p < .05), which showed a mean difference that is statistically significant (see Table 13). The decision was to reject the null hypothesis and accept the alternative hypothesis for research question 1. Comment by Robert Widner: This might confuse your reader. Maybe “RQ1 Hypotheses”

Hypothesis 2. Research hypothesis 2 stated that that there is a statistically significant difference in the ACT science exam scores for POGIL high school students and non-POGIL high school students. As discussed, the one-way MANOVA demonstrated a significant effect for the two dependent variables (chemistry SAGE scores and ACT science scores) of p = .0005 ( p < .05) (see Table 9). At the univariate level, ANOVA for ACT science exam scores, which indicate change in ACT exam academic achievement level for the independent groups, demonstrated that there was a significant difference between POGIL high school participants and non-POGIL high school participants on ACT science exam scores of p = .009 ( p < .05) (see Tables 10 and 12). The two-tailed Independent Sample t-test for the ACT science exam scores, produced a statistical significance of p = .009 ( p < .05), which showed a mean difference that is statistically significant (see Table 13). The decision was to reject the null hypothesis and accept the alternative hypothesis for research question 2. Comment by Robert Widner: “RQ2 Hypotheses” Comment by Robert Widner: Adjust alpha? Comment by Robert Widner: When you first introduce the idea of a two-tailed test (I don’t know where you first introduced it) provide your reader with the rationale for using a two-tailed rather than a one-tailed test. Include with this presentation the relationship between tail and power. Below is an example resource. Do more research on this as it is likely to be a question I might ask during your oral defense. https://stats.idre.ucla.edu/other/mult-pkg/faq/general/faq-what-are-the-differences-between-one-tailed-and-two-tailed-tests/

With an indication of significant difference between the independent groups for chemistry SAGE exam scores and ACT science exam scores, the means exhibited in Table 8 were examined to observe the differences between the mean values for the independent groups. As indicated by the mean values, POGIL high school students demonstrated significantly higher chemistry SAGE exam scores ( M = 845.76, SD = 30.43) than did non-POGIL students ( M = 822.10, SD = 45.06). In addition, the mean values for POGIL high school students demonstrated significantly higher ACT science exam scores ( M = 23.22, SD = 4.41) than did non-POGIL students ( M = 21.93, SD = 4.28).

Chapter 4 presented descriptive statistics of the collected data, reviewed the data analysis procedures, and presented the results of the data analysis for the study. The purpose of this quantitative, causal-comparative research study was to determine if and to what extent there were differences in chemistry SAGE exam scores and ACT science exam scores for high school chemistry students taught using POGIL pedagogy, and high school chemistry students taught using non-POGIL pedagogy in the state of New MexicoUtah. This section presented a summary of the data analysis results for each research question and respective hypotheses. Comment by Robert Widner: Did you mean to use plural form? Fix throughout your paper.

This investigation analyzed 2015-2016 and 2016-2017 de-identified archival student data for two demographically similar public high schools located in the same public, school district in the state of Utah. The data sample contained 158 high school students from the POGIL high school and 158 high school students from the non-POGIL high school. The POGIL high school students were taught chemistry using POGIL instructional strategies throughout the academic school year, and the non-POGIL high school students were taught chemistry using traditional instructional strategies.

All 316 cases included, within the de-identified archival data set, had no missing chemistry SAGE scores, ACT science scores, GPA before taking chemistry, enter and exit date for the chemistry class, and grade level. The student GPA’s were matched identically for each student from the two comparison groups or matched within +/- 0.10 grade point to ensure similar test subjects between the POGIL high school group and the non-POGIL high school group. In addition, participants needed to enter and exit their chemistry class no more than four weeks from the beginning of the school year or towards the end of the school year, or participants were removed since they would not have completed at least 70% of the school year in the chemistry course prior to taking the SAGE chemistry exam and ACT exam in the month of March. Prior to analysis, the data were cleaned and screened, including tests for assumptions of the one-way MANOVA to ensure the data were appropriate for the parametric statistical test. All assumptions were met except for assumptions of univariate or multivariate outliers, multivariate normality, homogeneity of variance-covariance matrices, and homogeneity of variances, which were justified as the one-way MANOVA is robust to violations with a large sample size (Warner, 2008). Comment by Robert Widner: Let’s not use “subjects” but perhaps “participants”. Check throughout paper. Comment by Robert Widner: Let’s talk about this via ZOOM. I would run a Welch’s ANOVA and compare results with your standard ANOVA. If the outcomes are the same then your reader is likely to feel more comfortable with the homogeneity of variances assumption violation. I uploaded into your LDP a student’s dissertation (he now has his Ph.D.) in which this was done to give you an idea – see what you think,

There were two research questions for this study and the corresponding hypotheses that were addressed included:

RQ1: To what extent, if any, does POGIL pedagogy produce a statistically significant difference in high school EOC chemistry scores?

H01: POGIL pedagogy does not produce a statistically significant difference in high school EOC chemistry scores.

H1a: POGIL pedagogy does produce a statistically significant difference in high school EOC chemistry scores.

RQ2: To what extent, if any, does POGIL pedagogy produce a statistically significant difference in high school ACT science exam scores?

H02: POGIL pedagogy does not produce a statistically significant difference in high school ACT science exam scores.

H2a: POGIL pedagogy does produce a statistically significant difference in high school ACT science exam scores.

The multivariate main effect demonstrated that there was a statistically significant difference between the POGIL student group, exposed to the intervention and non-POGIL student group, unexposed to the intervention when considered jointly on the dependent variables of SAGE chemistry exam scores and ACT science exam scores, where p = .0005. Prior to examining the univariate ANOVA, the level of statistical significance was adjusted for multiple comparisons, and a Bonferroni correction was made in order to accept statistical significance for the univariate ANOVAs (Laerd, 2015). The level of statistical significance was set a p < .025 rather than p < .05 since there were two dependent variables for the sample size (Laerd, 2015). When examined at the univariate ANOVA level, findings indicated that there was a statistically significant difference in the chemistry SAGE exam scores for POGIL high school students and non-POGIL high school students, thereby addressing research hypothesis 1 of research question 1. The observed probability value ( p = .0005) was lower than the criterion alpha of .025, and partial eta squared ( η2 = .087) rendered a small effect size (Morgan, et al., 2013). Comment by Robert Widner: This is not the correct rationale for using an adjusted alpha. Please do a little more research on this issue.

The two-tailed Independent Sample t-test for the chemistry SAGE scores, produced as statistical significance of p = .0005 ( p < .05), which showed a mean difference that is statistically significant (see Table 13). The decision was to reject the null hypothesis and accept the alternative hypothesis for research question 1. In addition, when examining the descriptive mean values, the statistics demonstrated that POGIL high school participants, who were taught chemistry using POGIL instructional strategies, demonstrated significantly higher chemistry SAGE exam scores ( M = 845.76, SD = 30.43) than did the non-POGIL high school participants for chemistry SAGE exam scores ( M = 822.10, SD = 45.06). Comment by Robert Widner: RQ1

The findings also demonstrated, when examined at the univariate ANOVA level, that there was a statistically significant difference in the ACT science exam scores for POGIL high school students and non-POGIL high school students, thereby addressing research hypothesis 2 of research question 2. The observed probability value ( p = .009) was lower than the criterion alpha of .05, and partial eta squared ( η2 = .022) rendered a small effect size (Morgan, et al., 2013). The two-tailed Independent Sample t-test for the ACT science exam scores, produced a statistical significance of p = .009 ( p < .05), which showed a mean difference that is statistically significant (see Table 13). The decision was to reject the null hypothesis and accept the alternative hypothesis for research question 2. Comment by Robert Widner: RQ2

In addition, when examining the descriptive mean values, the statistics demonstrated that POGIL high school participants, who were taught chemistry using POGIL instructional strategies, demonstrated significantly higher ACT science exam scores ( M = 23.22, SD = 4.4) than did the non-POGIL high school participants for ACT science exam scores ( M = 21.93, SD = 4.28).

Limitations. During the data analysis process, there were two identified limitations: (1) assumptions not met and (2) outliers. It is essential to discuss the limitation that emerged during the data analysis and how these limitations may affect the interpretation of the results. The first limitation were assumptions that were not met for the one-way MANOVA, one-way ANOVA, and two-tailed Independent Samples t-test. All three tests had assumption violations in multivariate normality, no univariate or multivariate outliers, homogeneity of variance-covariance matrices, and homogeneity of variances, which were justified as all three tests are robust to assumption violations with a large sample size (Warner, 2008). Comment by Robert Widner: This is a simplified view and worries me as justification for not following up with additional analyses. For example, see https://stats.stackexchange.com/questions/52091/normality-assumption-and-sample-size

The second limitation was the presence of univariate and multivariate outliers. There were six univariate outliers that included two outliers for the SAGE chemistry exam scores and four outliers for the ACT science exam scores. When the researcher ran the analysis outcomes with and without the outliers, there was no significant change; therefore, the outliers were left in the study sample. There were two multivariate outliers that exceeded the Mahalanbis distance critical value of 13.82 for two dependent variables. Both multivariate outliers had values of 16.54. The researcher decided to leave the two multivariate outliers in the study sample since the one-way MANOVA is robust to multivariate outliers with a large sample size ( N = 316) (Warner, 2008). Comment by Robert Widner: Generally we don’t leave outliers in our data set. I would take them out and then report the results were not impacted by taking them out versus leaving them in. Comment by Robert Widner: https://www.statisticssolutions.com/univariate-and-multivariate-outliers/ You are placing too much reliance on one source. Please check other sources to see what is recommended. We can chat a bit about this in a ZOOM call. I think your AQR reviewer will take issue with not addressing the outlier issues. But we can chat.

Chapter 4 described the data collected including descriptive statistics and data specific to each research question. After the data was analyzed, both the first and second research questions resulted in statistically significant results, rejecting the null hypotheses. Chapter 5 provided a summary of the study, findings, implications, and recommendations for future research. Comment by Robert Widner: I’d like to see a bit more organization to the results section. See the document I upload into your LDP (a learner’s dissertation). Comment by Robert Widner: Remember to fix throughout: “data” is the plural form so should be “data were”. Comment by Robert Widner: I would add a sentence here about what accepting the alternative hypotheses means.

ACT (2014a). Content Covered by the ACT Science Test. Test Descriptions.

ACT (2014b). ACT Technical Manual. Iowa City, IA.

ACT (2016a). The Ccondition of STEM 2016. Iowa City, IA. Retrieved from

Site Authorization Letters On Sun, Jul 21, 2019 at 7:48 AM Edward Mata < [email protected]> wrote:

Good morning Ms. Gough, 

I am a doctoral student with Grand Canyon University conducting a dissertation study on the effectiveness of Process Oriented Guided Inquiry Learning (POGIL) on high school student chemistry SAGE exam scores and ACT science scores. My study has been approved by the Jordan District Research Review Committee, and you should have received an intradistrict communication from Dr. Godfrey and Mr. Jameson requesting the participation of your school in the study. 

I am seeking permission to conduct the following research activities at your school site: 

·     To email recruitment letters to science teachers through school district email to solicit their voluntary participation in the study. 

·     To email informed consent forms to science teachers that volunteer to participate in the study. 

·     To email the online teacher demographic survey to participating teachers and allow these voluntary teachers to use the 5-10 minutes of work time necessary to fill out the surveys.

Thank you for your consideration in allowing me to conduct part of my study with your teaching staff. Please contact me if you have any questions or concerns regarding this request, and I look forward to your response.     

Best Regards,

Edward Mata 

(951)741-8111 

From: Carolyn Gough < [email protected]> Sent: Monday, July 22, 2019 8:30:22 AM To: Edward Mata < [email protected]> Subject: Re: Permission to conduct study

Thank you for your message. I did receive the information from the district and as with all studies, the school has the option of participation or not and is voluntary for teachers. Therefore, I am ok with this study and will provide you with the district emails for my chemistry teachers. I am requesting that when you copy me on all communication with my chemistry teachers. 

Thank you,

Carolyn Gough

On Mon, Aug 5, 2019 at 8:16 AM Edward Mata < [email protected]> wrote:

Good morning Mr. Veazie, 

I am a doctoral student with Grand Canyon University conducting a dissertation study on the effectiveness of Process Oriented Guided Inquiry Learning (POGIL) on high school student chemistry SAGE exam scores and ACT science scores. My study has been approved by the Jordan District Research Review Committee, and you should have received an intradistrict communication from Dr. Godfrey and Mr. Jameson requesting the participation of your school in the study. 

I am seeking permission to conduct the following research activities at your school site: 

·     To email recruitment letters to science teachers through school district email to solicit their voluntary participation in the study. 

·     To email informed consent forms to science teachers that volunteer to participate in the study. 

·     To email the online teacher demographic survey to participating teachers and allow these voluntary teachers to use the 5-10 minutes of work time necessary to fill out the surveys.

Thank you for your consideration in allowing me to conduct part of my study with your teaching staff. Please contact me if you have any questions or concerns regarding this request, and I look forward to your response.     

Best Regards,

Edward Mata 

(951)741-8111 

Re: Permission to conduct study

Bryan Veazie <[email protected]>

Mon 8/5/2019 10:33 AM

To: Edward Mata <[email protected]>

Cc:  Mark Halliday <[email protected]>

Edward,

Thank you for reaching out to me with this request. I have received the notice of approval from the District Research Approval Committee. I am certainly willing to support your efforts of working with our Science teachers on a voluntary basis.

I have included Mark Halliday, Assistant Principal of the Science Department, on this reply. Either one of us can provide you with the information you need to reach out to our Science teachers to extend the invitation.

Sincerely,

Bryan Veazie

Principal / Lead Learner

Copper Hills High School 

(801) 256-5301

[email protected]

Permission Letter

The POGIL Project Excel Spreadsheet POGIL High School Teacher 2016-2017

Permission Letter

SurveyMonkey (Teacher Demographic Survey)

Appendix G. Teacher Demographic Survey (Survey Monkey)

Appendix H. Data Tables with Outliers Removed

Appendix I. Normal Q-Q Plots

GCU Dissertation Template V8.3 01.18.18