DB3
This assignment requires the student to explain on the discussion board what the different types of
evaluations are, explain the method and the strengths and weaknesses of each. What type of questions
about policing do you have that cannot be evaluated by an RCT? What type of evaluation would fit
better? Give examples of police research question that is best suited for each of the 5 levels of the
Maryland Scientific Methods Scale.
Evaluation Strategy with Logic Model – Problem Oriented Policing
Gretchen Weller
School of Helms, Liberty University
Introduction
Vito and Huggins (2015) discuss that the primary significance of the process evaluations
is to deliver a true, continuous, and extensive account of the program as it is implemented. The
secondary aim of evaluations is to show whether the program has reached the target population
and does it serve them effectively. From inception, during process, implementation, and
outcomes, the program design should be constantly reevaluated should the need for corrective
action arise. Throughout the process of the evaluation, the focus should be on the effectiveness
of the program: designs based off empirical evidence equate to better effectiveness of outcome
and impact; the more specific aspects of the program that are defined, the better effectiveness of
the outcomes and impacts (Sager, Mavrot & Hadorn, 2015).
Types of Evaluations
National Research Council (2005) refer to two major forms of program evaluation:
process or implementation and impact. Output of a program represents the product of the
program, or rather, the intended services. This can be measured through efficiency and
effectiveness, which can be measured primarily through cost-service ratio and compliance,
which refers to the manner which services are delivered. Evaluation program effects related to
changes in behavior of target population. The more the implementation of the program falls in
line with the program concept, the more likelihood of a positive outcome (Sager et al., 2015).
Evaluation impact refers to the totality of the effects produced amongst the target population,
whether intended or non-intended. For instance, if the population changes behavior intended but
no effect on the problem situation, the program has likely failed. However, if the population
changes behavior and has an effect, the program has likely succeeded as an effective program
(Sager et al., 2015).
Strengths and Weaknesses of Evaluations
As previously mentioned, evaluations are utilized to provide a true and thorough account
of the program, offering many strengths. However, with each process come weaknesses. One of
the greatest weaknesses of impact evaluation is the fact that the evaluation can only answer
whether the intervention had an effect. The intervention could possibly turn out an array of
effects resulting negatively (Cornwall & Aghajanian, 2017). Levitron (2003) discusses that
evaluations are more inclined to be utilized when they focus on implementation information
rather than solely on outcomes. The author continues to state that it would behoove those who
study evaluations to focus on decision contexts, such as the desire to fit an evaluation into the
user’s own context.
Maryland’s Scientific Method Scale
Maryland’s Scientific Method Scale is projected in sorts as a means to rank evaluation
research studies reference three methodological criteria of: control over variables which might
interfere with the effects between an intervention and its outcome; measurement error; and
statistical power to identify program effects (Hope, 2005). However, unless predetermined in the
design, it is difficult to evaluate unintended outcomes (Cornwall & Aghajanian, 2017). The
Maryland Report offers an opportunity for assessing the impact of study design on outcomes in
criminal justice due to being comprehensive in providing available research (Weisburg, Lum &
Petrosino, 2001).
Reference
Cornwall, A. & Aghajanian, A. (2017). How to find out what’s really going on: Understanding
impact through participatory process evaluation. World Development, 99, 173-185.
Hope, T. (2005). Pretend it doesn’t work: The ‘anti-social’ bias in the Maryland Scientific
Methods Scale. European Journal on Criminal Policy & Research, 11, 275-296.
Leviton, L. C. (2003). Evaluation use: Advances, challenges and applications. American Journal
of Evaluation, 24(4), 525-535.
National Research Council. (2005). Improving evaluation of anticrime programs. Washington,
DC: National Academies Press.
Sager, F., Mavrot, C. & Hadorn, S. (2015). Addressing multilevel program complexity by
evaluation design. European Policy Analysis, 1(2), 90-110.
Vito, G. F. & Higgins, G. E. (2015). Practical program evaluation for criminal justice.
Waltham, MA: Elsevier.
Weisburg, D., Lum., C. M. & Petrosino, A. (2001). Does research design affect study outcomes
in criminal justice? Annals of American Academy of Political & Social Sciences, 578, 50-
70.
Reply #1 Strengths Weaknesses/ RCT
Douglas,
Evaluation research has moved towards evidence-based practice and policy, allowing
policy makers to identify what programs and practices are most effective. The evidence has
shown, in theory, that randomized experimental studies yield higher internal validity than do
nonrandom studies (Weisburg, Lum & Petrosino, 2001). Definitive facts reference the
effectiveness of a program is preferred but where definitive facts are not available, decisions are
still guided by evidence (Levitron, 2003). Therefore, assuming experimental designs are the go-
to and preferred design method for evaluating policies and practices, why then do researchers
face several obstacles during evaluation practices such as lack of workplace incentives, lack of
funding, lack of time, lack of organizational support, lack of leadership, and failure to take any
action with results (Peterson, Schmid & Kososki, 2020). Weisburg et al. (2001) asks several
questions as to what price do researchers pay for utilizing other types of study methods, for
instance: Are researchers more likely to overestimate or underestimate positive outcomes, and
should researchers expect the same conclusions from nonrandomized experimental evaluations
as they might expect from well-designed randomized studies. Other obstacles result from
producers of evaluation in such that political and managerial self-interest frequently close off any
opportunity for evaluations to be given merit and are therefore used symbolically or to legitimize
other decisions. Levitron (2003) states that successful evaluations yield from those whose
questions have been formulated from a structured process, reducing any bias or uncertainty
reference significant issues and test assumptions.
NEED ONE MORE SENTENCE
Reference
Leviton, L. C. (2003). Evaluation use: Advances, challenges and applications. American Journal
of Evaluation, 24(4), 525-535.
Peterson, C. H., Schmid, K. & Kososki, R. (2020). A national survey of counselors’ use of five
types of program evaluation. Counseling Outcome Research & Evaluation, 11(2), 71-87.
Weisburg, D., Lum., C. M. & Petrosino, A. (2001). Does research design affect study outcomes
in criminal justice? Annals of American Academy of Political & Social Sciences, 578, 50-
70.
Reply #2 SMS
James,
Weisburg, Lum and Petrosino (2001) broadly summarize the Maryland Scientific Method
Scale as a 5-point scale through assessing the impact of study design on outcomes in criminal
justice: 1 correlates between crime prevention programs and crime risk factors; 2 chronological
order between program and risk outcome; 3 comparison between several units of measure, with
and without the program; 4 comparison between several units of measure, with and without the
program, to control for other factors; and 5 random selection and study of similar units.
Essentially, the method scale utilizes a set number of standards applied to research methods that
represent the methodological principles of the method to draw conclusions related to threats to
validity. This is done through correlation between process of selection and the intervention, and
social processes that mediate between treatment and intervention (Hope, 2005). The validity of a
study will often reveal the strength to understanding the experimental study. For instance, a
research design that clearly identifies effects of treatment/intervention from other effects is
considered to have high internal validity whereas a design in which treatment effects are
mistaken for other factors is considered to have low internal validity (Weisburg et al., 2001).
However, there is always the possibility of an inappropriate application of the Scientific Method
Scale in which bias may be used when detecting, or failing to detect, negative effects, yielding
potential to negatively affect policymaking by introducing error into the evidentiary process
(Hope, 2005). This bias may cause policymakers to reject or even accept programs through
misrepresentation of the program. Without scientifically credible evaluations, researchers and
policymakers cannot learn which programs are beneficial and which are harmful (McCord,
2003).
References
Hope, T. (2005). Pretend it doesn’t work: The ‘anti-social’ bias in the Maryland Scientific
Methods Scale. European Journal on Criminal Policy & Research, 11, 275-296.
McCord, J. (2003). Cures that harm: Unanticipated outcomes of crime prevention programs. The Annals
of the American Academy of Political & Social Science, 587, 16-30.
Weisburg, D., Lum., C. M. & Petrosino, A. (2001). Does research design affect study outcomes in criminal
justice? Annals of American Academy of Political & Social Sciences, 578, 50-70.
Reply #3 Evaluations
Traditional evaluation methods consist of process evaluations and outcome evaluations
which allows for the examination of challenges, review of other evaluations to see what research
is offered, and finally, to offer current suggestions (Presser & Van Voorhis, 2002). As we have
discussed, program evaluation is the gathering of data in which to educate program planning,
increase program effectiveness, and to study and report the impact of services. It is, therefore,
essential to safeguard effective practices. Although program evaluations are typically utilized in
a single case design, oftentimes researchers lack the time necessary to perform full evaluation
and instead perform one of three other types of evaluations: needs assessment, formative
evaluation, or summative evaluation (Peterson, Schmid & Kososki, 2020). Needs assessment
takes place before the intervention and consists of collecting data in order to discover needs and
assess program planning, design, and delivery of services. The formative evaluation also collects
data but for the improvement of current program through feedback or surveys. Summative
evaluation, or outcome evaluation, is performed once the program has a solid foundation
regarding decision-making and accountability. This evaluation measures the outcomes as an
indication of the program design of its effectiveness or areas in need of improvement (Peterson
et al, 2020). Krohn (2019) expresses that evaluations should highlight whether the scientific
subject matter functions or has the capacity to make a difference. Bragga and Weisburg (2013)
state that some designs are more scientifically valid and evidence-based, favoring ‘what works’
and then there are quasi-experimental randomized designs. These authors reported that
evaluation designs that lacked scientific validity, resulted in false positives in favor of the
program.
References
Braga, A. A & Weisburg, D. L. (2013). Editors’ introduction: Advancing program evaluation
methods in criminology and criminal justice. Evaluation Review, 37(3-4), 163-169.
Krohn, M. D. (2019). Making a difference - or not: An evaluation of the effects of failing to
evaluate. American Journal of Criminal Justice, 44, 520–535.
Peterson, C. H., Schmid, K. & Kososki, R. (2020). A national survey of counselors’ use of five
types of program evaluation. Counseling Outcome Research & Evaluation, 11(2), 71-87.
Presser, L & Van Voorhis, P. (2002). Values and evaluation: Assessing processes and outcomes
of restorative justice programs. Crime & Delinquency, 48(1), 162-188.
Weisburg, D., Lum., C. M. & Petrosino, A. (2001). Does research design affect study outcomes in criminal
justice? Annals of American Academy of Political & Social Sciences, 578, 50-70.
McCord, J. (2003). Cures that harm: Unanticipated outcomes of crime prevention programs. The Annals
of the American Academy of Political & Social Science, 587, 16-30.
Cornwall, A. & Aghajanian, A. (2017). How to find out what’s really going on: Understanding impact
through participatory process evaluation. World Development, 99, 173-185.
Hope, T. (2005). Pretend it doesn’t work: The ‘anti-social’ bias in the Maryland Scientific Methods Scale.
European Journal on Criminal Policy & Research, 11, 275-296.
Leviton, L. C. (2003). Evaluation use: Advances, challenges and applications. American Journal of
Evaluation, 24(4), 525-535.
Peterson, C. H., Schmid, K. & Kososki, R. (2020). A national survey of counselors’ use of five types of
program evaluation. Counseling Outcome Research & Evaluation, 11(2), 71-87.
National Research Council. (2005). Improving evaluation of anticrime programs. Washington,
DC: National Academies Press.
Sager, F., Mavrot, C. & Hadorn, S. (2015). Addressing multilevel program complexity by
evaluation design. European Policy Analysis, 1(2), 90-110.
Vito, G. F. & Higgins, G. E. (2015). Practical program evaluation for criminal justice.
Waltham, MA: Elsevier.
An Evaluation of Successful Program Completions across Types of Problem-Solving Courts Kimberly
Kaiser JUSTICE EVALUATION JOURNAL 2020, VOL. 3, NO. 1, 54–68
