MIXED-METHODS CRITIQUE 1
Research Critique: Mixed-Methods
Whitney Young-Magee
School of Education, Liberty University
Research Critique: Mixed-Methods
MIXED-METHODS CRITIQUE 2
The research article titled “Impact of ChatGPT on ESL students; academic writing skills:
a mixed methods intervention study” by Santosh Mahapatra investigates the effectiveness of
utilizing ChatGPT, an AI-based language model, in enhancing the academic writing skills of
English as a Second Language (ESL/ESOL) students. This study utilizes a mixed methods
approach to comprehensively evaluate the impact of ChatGPT on ESL student’ writing
proficiency. The study employs a mixed methods approach to comprehensively evaluate the
impact of ChatGPT on ESL students' writing proficiency.
Summary
Purpose
The purpose of the study conducted by Mahapatra (2024) was to investigate the impact of
ChatGPT as a formative feedback tool on the writing skills of undergraduate ESL students. With
the intent of addressing the minimal amount of empirical evidence on the effectiveness of
ChatGPT in enhancing writing, the study implemented a mixed methods intervention approach.
Through tests and focus group discussions, the goal of the research was to evaluate the extent of
ChatGPT’s positive influence on students’ academic writing skills and the perception of its
impact. Additionally, the study sought to contribute to theories of feedback and affirm
ChatGPT’s potential as a reliable writing tool with practical implications for writing classes.
Participants/Sample
The participants in the study first-year science and engineering students from an elite
private-run university in India (Mahapatra, 2024). They were randomly selected from an intact
class two sections, adhering to the sampling principle for quasi-experimental studies
(Mahapatra, 2024; Perry Jr., 2011). The participants included had no prior experience using
ChatGPt for improving their writing skills (Mahapatra, 2024). The experimental group (EG)
MIXED-METHODS CRITIQUE 3
had 78 students, while the comparison group (CG) consisted of 56 students that agreed to
participate voluntarily (Mahapatra, 2024). Each participant had attended English medium
schools prior to enrolling in the undergraduate program and had English as their second
language (Mahapatra, 2024). The inclusion criteria required attendance in all intervention
classes, totaling six hours, and completion of pre-, post-, and delayed post-tests. As a result, 35
students from the EG and 37 from the CG were integrated in the reported data (Mahapatra,
2024). In addition, six students volunteered for focus group discussions, with five participating
in all three discussions (Mahapatra, 2024).
Research Design
This mixed-methods research study included an embedded quasi-experimental design,
where both quantitative and qualitative data were collected simultaneously (Mahapatra, 2024),
however the qualitative data was embedded within the quantitative data. In this design, the
primary focus is on the quantitative data, but the qualitative data provides additional insights
and explanations, with the intent to establish a cause-and-effect relationship between an
independent and dependent variable (Thomas, 2020). For example, while the study primarily
assessed the impact of ChatGPT on ESL students’ academic writing skills through quantitative
measures such as tests and evaluations, qualitative data from focus group discussions were used
to delve deeper into students’ perceptions and experiences with using ChatGPT for feedback
purposes. This combination allowed for a comprehensive understanding of the intervention's
effects.
Method of Data Collection
The study collected data primarily through three writing tests, each consisting of tasks
focused on process, comparison, and cause-effect writing, which are commonly used in
MIXED-METHODS CRITIQUE 4
academic contexts (Mahapatra, 2024). Rubrics were developed for each task type by the
researcher and another teacher, then validated by experts (Mahapatra, 2024). These rubrics
evaluated content, organization, grammar, and vocabulary (Mahapatra, 2024). The writing
performance of students was evaluated using these rubrics, with strong reliability between
raters (Mahapatra, 2024). Qualitative data were collected through focus group discussions with
five experimental group students, focusing on their experiences using ChatGPT for feedback
(Mahapatra, 2024). The data collection procedure spanned an academic semester. It began with
a pre-test for both the control and experimental groups (Mahapatra, 2024). Next, the
experimental group received training on using ChatGPT for self-assessment (SA) and peer
assessment (PA) (Mahapatra, 2024). Following the training, a six-hour intervention was
conducted, where the experimental group was taught various writing skills and used ChatGPT
for SA and PA purposes (Mahapatra, 2024).
Method of Analysis
The quantitative and qualitative data were analyzed separately and then combined to
answer the research questions (Mahapatra, 2024). For the quantitative analysis, statistical test
were performed on pre-test, post-test, and delayed post-test scores for both the control group
(CG) and experimental group (EG). This included running a one-way RM ANOVA (multiple
test of the same variable) and post-hoc Bonferroni test (multiple comparison test for statistical
analysis) for the EG's scores across the three tests, as well as t-tests to compare scores between
the CG and EG (Mahapatra, 2024). Tests for equality of variance and normality were conducted
to ensure data validity Mahapatra, 2024). Qualitative analysis involved coding transcripts of
focus group discussions using a phronetic iterative approach, which was applicable for
MIXED-METHODS CRITIQUE 5
inductive coding guided by emerging themes from literature and known patterns (Mahapatra,
2024; Tracy, 2019).
Results
The results regarding the use of ChatGPT as a feedback tool on students’ writing skills
was both positive and significant (Mahapatra, 2024). The improvement of writing skills for the
EG was evident across the pre-, post-, and delayed post-test (Mahapatra, 2024). The results of
the of one-way RM ANOVA confirmed a substantial difference among mean scores across the
tests (Mahapatra, 2024). Specifically, significant improvements were observed between pre-test
and post-test scores (p < 0.001, d = − 3.300) and pre-test and delayed post-test scores (p < 0.001,
d = − 3.898), with a significant difference also noted between post-test and delayed post-test
scores (p < 0.01, d = − 0.598) (Mahapatra, 2024). Moreover, independent sample t-test
displayed significant differences between the post-test and delayed post-test scores of the CG
and the EG, with EG consistently outperforming the CG in both instances (Mahapatra, 2024).
Student perceptions and feedback regarding the use of ChatGPT were categorized into
three main themes: content, organization, and grammar, with two sub-themes being positive and
negative (Mahapatra, 2024). When discussing the impact of ChatGPT as a feedback tool,
students in focus group discussions (FGDs) emphasized its assistance in content generation,
organization of ideas, and improvement in grammar (Mahapatra, 2024). For example, students
expressed surprise at how ChatGPT guided them in obtaining necessary information, organizing
ideas, and writing correctly, describing it as a helpful writing companion (Mahapatra, 2024).
They appreciated the tool's ability to provide focused information on given topics and its support
for independent work, reducing their thinking time invested in generating ideas (Mahapatra,
2024). Moreover, students noted how ChatGPT promoted collaboration among peers, facilitated
MIXED-METHODS CRITIQUE 6
faster task completion, and aided in creating strong topic sentences (Mahapatra, 2024). However,
some students also acknowledged potential drawbacks, such as a possible decrease in motivation
to think independently and increased dependence on the machine (Mahapatra, 2024). Despite
these concerns, students generally found ChatGPT beneficial for maintaining focus on the topic
while writing and improving grammatical accuracy (Mahapatra, 2024). Overall, the students'
positive outlook on ChatGPT's impact aligns with previous research on AI-driven tools and their
effects on writing skills (Mahapatra, 2024; Marzuki et al., 2023; Yan, 2023). Their perceptions
regarding content generation, organization, and grammar support existing literature while also
highlighting areas for further investigation (Mahapatra, 2024; Gayed et al., 2022; Guo et al.,
2022).
Critical Analysis
Further Research
Future research should emphasize an extension on investigations into the efficacy of
using ChatGPT as a feedback tool, especially given the focus of current studies predominantly
on undergraduate students. Understanding how ChatGPT can be efficiently integrated into K-12
writing instruction is imperative to providing younger learners with valuable support in
developing their writing skills. Specific attention should be brough to elements directly related
to K-12 environments, such as students’ developmental stages, diverse linguistic backgrounds,
and varying levels of technological proficiency. Furthermore, exploring the potential impact of
ChatGPT on K-12 students’ writing autonomy and creativity is crucial for informing
educational practices tailored to their needs Additionally, future research should expand on the
current study by examining the nuanced effects of ChatGPT on ESL students’ academic writing
skills within diverse education contexts. This could include examining the impact of ChatGPT
MIXED-METHODS CRITIQUE 7
with varying levels of language proficiency or from different academic backgrounds.
Furthermore, deeper exploration into the long-term impact of integrating the language model as
it relates to writing is necessary to understand its sustained impact on students’ writing skills
beyond immediate intervention periods.
Threats to Validity/Undocumented Bias
Multiple potential threats to the validity of this study are at play. Internal validity may be
compromised. For example, elements such as history, maturation, or testing effects, especially
given the quasi-experimental design utilized, should be considered. There is a potential risk of
selection bias because of the non-random assignment of participants to the control and
experimental groups, while inconsistencies in intervention implementation could lead to
treatment diffusion or implementation bias (Pannuci and Wilkins, 2011). External validity may
be limited due to the specific context of the study, potentially impacting the generalizability of
findings to other populations or settings. To add, reliability on self-reported data introduces the
possibility of response bias or desirability bias. Construct validity could also be threatened by
the operationalization of variables and the reliability of measurement instruments, including
writing tasks and rubrics used for assessment. Addressing these threats is crucial for ensuring
the study’s findings are robust and trustworthy.
Original Insight/Criticism
Intriguingly, the study was able to reveal the positive impact on ESL students and their
writing skills, emphasizing its potential as a valuable tool in regard to writing instruction. Both
researchers’ and students assertions were supported by the data, which was reassuring.
However, concerns arose regarding the non-random selection process which could potential
introduce bias and limit the generalizability of the findings. Moreover, the study’s focus on
MIXED-METHODS CRITIQUE 8
short-term effects left many questions in relation to the long-term impact of using ChatGPT as a
feedback tool. Lastly, the absence of exploration into ChatGPT’s potential benefits for K-12 Els
and struggling writers was notable, suggesting a necessity for broader research in this sector.
Implications of Findings
[The findings in Mahapatra’s study (2024) indicate a positive and significant impact of
ChatGPT as a feedback tool on students’ writing skills. Improvement was prevalent across their
pre-, post-, and delayed post-tests, with notable differences observed in mean scores
(Mahapatra, 2024). More specifically, improvements were noted between pre- and post-tests,
and post and delayed-posts tests scores. Moreover, noteworthy changes were found between the
post-test and delayed post-tests scores of the CG and the EG, with the EG consistently
outperforming the CG (Mahapatra, 2024). The student's perceptions and feedback in relation to
ChatGPT were categorized into themes as follows; content, organization, grammar, with sub-
themes including positive or negative aspects. Students emphasized ChatGPT’s assistance in
content generation, organization of ideas, and improvement in grammar, accentuating its role as
a helpful writing companion and its ability to promote collaboration among peers (Mahapatra,
2024). While acknowledging potential drawbacks such as lower motivation and increased
dependence on the learning language model, students overall, found ChatGPT to be beneficial
for maintain focus on the topic and enhancing grammatical accuracy, aligning with previous
research on AI driven tools’ effects on writing skills (Mahapatra, 2024; Marzuki et al., 2023;
Yan, 2023).
Discussion
The rapid advancement of AI consistently piques my interest, driving my desire to delve
deeper into its evolution, advancement, and application. I am more interested in exploring ways
MIXED-METHODS CRITIQUE 9
to seamlessly integrate AI into my classroom, understanding that students will inevitably
encounter and use the platform. I believe that educating students on its responsible use and
leveraging its potential to improve writing skills if preferable to pretending it doesn’t exist or
even worse, demonizing it. Furthermore, I see great potential in the utilization of AI to
personalize learning experiences, especially for students facing challenges such as struggling
writers, individuals with disabilities, and English Language Learners. Pulling from the scripture
2 Corinthians 5:17, the integration of this biblical scripture into the context of education
highlights the transformative power of embracing new approaches and technology. Just as people
may have a spiritual renewal when they accept Christ, educators can foster a sense of renewal by
embracing innovative technology and teaching methods. Just like the old may be replaced with
the new in the spiritual realm, the same can be said for the adoption of new tools and strategies.
This scripture influences educators to embrace change in order to provide a progressive and
conducive learning environment.
MIXED-METHODS CRITIQUE 10
References
Holy Bible, New International Version. 1973, 1978, 1984, 2011
Mahapatra, S. (2024). Impact of ChatGPT on ESL students’ academic writing skills: a mixed
methods intervention study. Smart Learning Environments, 11(1), 1-18.
Pannucci, C & Wilkins, E. (2011) Identifying and avoiding bias in research. Plast Reconstr Surg.
Aug;126(2):619-625. doi: 10.1097/PRS.0b013e3181de24bc. PMID: 20679844; PMCID:
PMC2917255.
Thomas, L. (2020) An Introduction to Quasi-Experimental Designs.
https://www.scribbr.com/methodology/quasi-experimental-design/
