Article/Data Analysis Exercise Assignment

farra004

Chapter13740.pdf

Home >Law homework help >Criminal homework help >Article/Data Analysis Exercise Assignment

Chapter 13:

Evaluation Research and Problem Analysis

Learning Objectives

• Summarize evaluation research and problem analysis as examples of applied research in criminal justice

• Describe how different types of evaluation activities correspond to different stages in the policy process

• Explain the role of an evaluability assessment • Understand why a careful formulation of the problem,

relevant measurements, and criteria of success or failure are essential in evaluation research

• Describe the parallels between evaluation research designs and other designs

• Explain the advantages, requirements, and limits of randomized field experiments

Learning Objectives, cont.

• Summarize the importance of process evaluations conducted independently or in connection with an impact assessment

• Describe the role of problem analysis as a planning technique that draws on the same social science research methods used in program evaluation

• Explain how the scientific realist approach focuses on mechanisms in context, rather than generalizable causal processes

• Present an example of how criminal justice agencies are increasingly using problem analysis tools, crime mapping, and other space-based procedures

• Explain how evaluation research entails special logistical, ethical, and political problems

Introduction • Evaluation Research: Refers to a research

purpose rather than a specific method; seeks to evaluate the impact of interventions; if some result was produced

• Problem Analysis: Designed to help public officials choose from alternative future actions

• Policy Intervention: An action taken for the purpose of producing some intended result

• Evidence-Based Policy: The actions of justice agencies are linked to evidence used for planning and evaluation

The Policy Process • Begins with a demand supporting a new

course of action or opposition to existing policy • Policymakers consider ultimate goals and

actions to achieve those goals • Outputs: The means to achieve desired goals • Impacts: Refer to basic questions about what a

policy seeks to achieve • If some policy is taken, then we expect some

result

Discussion Question 1

What if you studied policy implementation? Do you think you’d be more interested in impacts or outputs? Why?

Linking the Process to Evaluation

• Are policies being implemented as planned?

• Are policies achieving their intended goals?

• Evaluation seeks to link intended actions and goals of policy to empirical evidence that: – Policies are being carried out as planned (process evaluation)

– Policies are having the desired effects (impact assessment)

• Often conducted together

Getting Started

• Learning policy goals is a key first step in doing evaluation research

• Evaluability Assessment: “Pre-evaluation”— researcher determines whether requisite conditions are present – Support from relevant organizations

– What goals and objectives are; how they are translated into program components

– What kinds of records or data are available

– Who has a direct or indirect stake in the program

Discussion Question 2

How might you convince a public official that an evaluation is both a necessary thing and a positive thing?

Problem Formulation and Measurement

• Different stakeholders often have different goals and views as to how a program should actually operate

• Must clearly specify outcomes—program goals and objectives

• Create objectives—operationalized statements

• Definition and measurement: target/beneficiary population; decide between using current measures or creating new ones

• Measure program contexts, outcomes, and delivery

Designs for Program Evaluation

• Randomized Evaluation Designs: Avoids selection bias; allows assumption that groups created by random assignment are statistically equivalent; may not be suitable when agency or staff makes exceptions

• Case Flow: Represents process through which subjects are accumulated into experimental and control groups

• Treatment Integrity: Whether an experimental intervention is delivered as intended; ≈ reliability – Threatened by midstream changes in program

Discussion Question 3

What if local researchers wanted to experiment on the criminal justice system in your area? What would be your concerns as far as ethics and integrity?

Conditions for Randomized Experiments • Staff must accept random assignment and agree to

minimize exceptions to randomization • Case flow must be adequate to produce enough

subjects in each group so that statistical tests will be able to detect significant differences in outcome measures

• Experimental interventions must be consistently applied to treatment groups and withheld from control groups

• Need equivalence prior to intervention, and ability to detect differences in outcome measures after intervention

Home Detention: Two Randomized Studies

• Combining home detention with ELMO • Juvenile program paid less attention to delivering

program elements and using ELMO info than adult – Difficult to maintain desired level of control over experimental

conditions – Also difficult when more than one organization is involved

• Randomization does not control for variation in treatment integrity and program delivery; utilize other methods

Quasi-Experimental Designs

• No random assignment to Experimental and Control group

• Often “nested” in experimental designs as backups

• Lack built-in controls for selection and other Internal Validity threats

• You must construct Experimental and Control groups that are as similar as possible

Quasi-Experimental Designs, cont. • Ex post Evaluation: Conducted after experimental

program has gone into effect • Full Coverage Programs: Sentencing guidelines • Larger Treatment Units: Neighborhood crime

prevention program • Interrupted Time-Series Designs: Require

attention to different issues because researchers cannot normally control how reliably the experimental treatment is actually implemented – Instrumentation, History, Construct Validity

Problem Analysis and Scientific Realism

• Problem analysis, coupled with scientific realism, helps public officials use research to select and assess alternative courses of action

• Realists suggest that similar interventions will have different outcomes in different contexts

• Evaluators should search for mechanisms (IVs) acting in context (assorted intervening variables) to explain outcomes (DVs)

• Appropriate in small-scale evaluations directed toward solving a particular problem in a specific context

Problem Analysis

• Problem-Oriented Policing – Problem solving: A fundamental tool in problem-oriented

policing

• How-to-Do-It Guides: A general guide to crime analysis to support problem-oriented policing

• Problem & Response Guides: Describe how to analyze very specific types of problems and what are known to be effective or ineffective responses

Auto Theft in Chula Vista • Nanci Plouffe and Rana Sampson (2004)

began their analysis of vehicle theft by comparing Chula Vista to other southern California cities – Theft rates tended to be higher for cities closer to the

border – Ten parking lots accounted for 25 percent of thefts and

20 percent of break-ins in the city – Six of the ten lots were among the top ten calls-for-

service locations in Chula Vista – Auto theft hot spots also tended to be hot spots for other

kinds of incidents

Other Applications of Policy Analysis

• Space- and Time-Based Analysis: increased prevalence due to technological advances

• Crime maps usually represent at least four different things: – (1) one or more crime types; (2) space or area; (3)

some time period; and (4) some dimension of land use, usually streets

• Problem-solving tools and processes – Strategic Approaches to Community Safety Initiatives

(SACSI)

Political Context of Applied Research

• Different stakeholder interests can produce conflicting perspectives on evaluations

• Researcher must identify stakeholders and perspectives

• Educate stakeholders on why evaluation should be conducted

• Explain that applied research is used to determine what works and what does not

• Political concerns and ideology may color evaluation; be careful