PROGRAM EVALUATION STUDY 2
Research Critique: Program Evaluation Study
The purpose of this paper is to critique Pullmann et al.’s (2022) program evaluation
study “Reducing racial and ethnic disproportionality in school discipline through an
assessment-to-intervention process: A framework and process.” The study aims to evaluate the
framework in which schools evaluate the disparity of racial discipline in schools, the DDAS or
Discipline Assessment for Schools. It will evaluate the study’s methods, participants, methods,
findings, and implications for reducing ethnic inequities in school discipline.
Summary
Purpose
The DDAS is a tool used to address the racial and ethnic disproportionality in school
discipline (REDD) by forcing schools to look at various factors through five steps using three
tools. Specifically, it will assess specific information about the school, allow for reflection and
prioritization of needs, and, finally, an assessment to action. The purpose of the study conducted
by Pullman et al (2022) is to explore the feasibility and effectiveness of this tool in real schools
vs on paper (Pullmann et al., 2022).
Participants/Sample
The study was completed in two steps. First it utilized 59 educators and staff such as
counselors and administrators. They were recruited from two conferences and virtual webinars:
2019 Northwest Positive Behavioral Intervention and Supports (NWPBIS) conference, the 2019
“MTSS Fest East” (Multi-Tiered Systems and Supports) conference, and a webinar on Cultural
Responsiveness and Equity in education practice. All participants watched presentations about
the DDAS, answered four questions, and provided open ended suggestions and responses. The
second step was to assess the actual application of the DDAS in schools. Employees at four
PROGRAM EVALUATION STUDY 3
K-5 schools in Illinois created or uses current equity teams to complete the DDAS at their
schools. This step allowed researchers to observe how the framework functioned in real-world
settings and gather additional data on its effectiveness and practicality (Pullmann et al., 2022).
Research Design
The research design utilized a mix-methods approach. combining both qualitative and
quantitative research methods. It is completed in two parts with the first designed to gather
feedback on the Disproportionality in Discipline Assessment for Schools (DDAS) framework
from educators and school staff. The second was to gather feedback on the real-world
application of the DDAS in schools (Pullmann et al., 2022).
Method of Data Collection
The data collected for this study consisted of both qualitative and quantitative methods.
Surveys were administered to gather feedback from school personnel regarding their
perceptions of the DDAS framework., focusing on feasibility, usability, and validity.
Qualitative methods included interview and focus groups to gather information regarding the
processes and challenges of implementing the DDAS in a K-12 school (Pullmann et al., 2022).
Method of Analysis
As previously mentioned, the researchers used surveys, interviews, and focus groups for
data collection. The information from surveys was then used to determine the mean scores and
standard deviations of various measures such as feasibility, effectiveness, and motivation.
Subscale scores were determined for mean feasibility, effectiveness, and motivation for all three
tools in the DDAS. For the qualitative analysis, a consensus approach was used to analyze the
open-ended comments and suggestions gathered in the interviews and focus groups. The idea
PROGRAM EVALUATION STUDY 4
was to determine and agree upon the meaning and context of the participant responses
(Pullmann et al., 2022).
Results
The study determined that the DDAS framework and its tools were well-received by
educators who found them to be easily used, useful, and appropriate for addressing racial and
ethnic disproportionality in discipline (REDD). While the framework was seen as a promising
tool for tackling racial disparities in schools, some concerns still surfaced. Specifically, there
was uncertainty regarding the validity of each team’s responses. It was concluded that careful
selection of diverse teams who can build trust was required for success (Pullmann et al., 2022).
Critical Analysis
Further Research
Pullmann et al.’s (2022) study provides the perceptions of the DDAS framework from a
limited sample of K-12 schools in a small-city Illinois setting. While the findings do support
this framework as a positive option for tackling REDD, there are areas for further research not
discussed. One such area is moving the target area to urban versus rural schools. Does the
location influence the beliefs, practices, and outcomes? Another is investigating the
perceptions of those experiencing the REDD to determine the DDAS’ impact on them.
Comparative research of the DDAS to other frameworks would also be another valid
opportunity for further research as this would determine if there are more beneficial ways to
address REDD. A final area for further research would be to determine if adapting DDAS to be
culturally sensitive and relevant to the school’s demographic would improve or inhibit its
proficiency (Pullmann et al., 2022).
Threats to Validity/Undocumented Bias
PROGRAM EVALUATION STUDY 5
All research studies have potential threats to validity. Some of those in Pullmann et al.’s
(2022) study include concerns for both the qualitative and quantitative research. For instance,
part of this study that uses quantitative research does not have a control group. As Kitsantas et
al. (2024) plainly states, this is essential for validity. Another potential area for concern is
selection bias as the participants were self-selected and not randomly chosen. There were also
only 59 participants and four schools used. For the qualitative portion of the study, again, only
four rural schools participated. This is not a large sample and not representative of many urban
schools which Kitsantas et al. (2024) also states is a concern (Pullmann et al., 2022).
Original Insight/Criticism
One of the positives of culturally responsive educational practices is adapting to and
understanding the differences in various cultures within a classroom and school. This includes
adapting to understand how culture changes interactions and affects the potential for discipline.
It is enlightening to know that there are tools such of the DDAS framework to assist in adapting
discipline to ensure cultural sensitivity is being addressed (Pullmann et al., 2022).
Implications of Findings
The findings of this study can help multiple schools nationwide. Educators need a tool to
assist in battling REDD and promoting equity in discipline. Frameworks such as the DDES
have the potential to provide just this. Researchers can also work to determine how well this
framework and others could work in urban districts, both those in low social economic areas as
well we high economic areas where disparities may manifest differently. By examining the
framework’s adaptability across diverse contexts, researchers and practitioners can refine its use
to meet the unique needs of schools nationwide (Pullmann et al., 2022).
Discussion
PROGRAM EVALUATION STUDY 6
God was clear in how He wanted His children treated. Leviticus 19:15 is clear when it
says "you shall not render an unfair decision: do not favor the poor or show deference to the
rich: judge your kinsman fairly.” This law in highlights the importance of impartiality and
fairness in God’s eyes. It is straight forward without any qualifiers or exceptions. For
educators, this demonstrates the need to treat all students equitably, taking into consideration
what cultural differences can mean regarding actions and behaviors. Recognizing and
respecting these differences is essential to fostering an environment where discipline and
leadership reflect God’s justice and love, even in a public-school setting. For Christian educators
and administrators, looking beyond the cultural worldview each of us has is imperative to
ensure those we lead and discipline are done so in a manner that represents the love and grace
of God.
PROGRAM EVALUATION STUDY 7
References
Kitsantas, Anastasia; Cleary, Timothy; DiBenedetto, Maria K; Hiller, Suzanne E.. Essentials of
Research Methods for Educators (p. 398). SAGE Publications. Kindle Edition.
Pullmann, M. D., Gaias, L. M., Duong, M. T., Gill, T., Curry, C., Cicchetti, C., Raviv, T., Kiche,
S., & Cook, C. R. (2022). Reducing racial and ethnic disproportionality in school
discipline through an assessment‐to‐intervention process: A Framework and process.
Psychology in the Schools, 59(12), 2486–2505. https://doi.org/10.1002/pits.22651
The Holy Bible, New International Version. (2011). Zondervan
Powered by TCPDF (www.tcpdf.org)
