Research Question and Introduction Development

Methodological quality and the evaluation

of anti-crime programs

DAVID P. FARRINGTON Institute of Criminology, Cambridge University, Sidgwick Avenue, Cambridge, CB3 9DT, UK

E-mail: [email protected]

Abstract. The National Research Council (NRC) report is an excellent contribution to knowledge. The

main issues arising are: (1) Evaluability: will consideration of this topic mean that few programs are

evaluated? (2) Methodological quality: a new scale needs to be developed and used. (3) Effect size: a

realistic, easily understandable measure should be used. (4) Benefit:cost ratio: more calculations of this

are needed. (5) Observational methods: their usefulness in evaluation compared with experimental and

quasi-experimental methods is doubtful in most cases. (6) Attrition: research is needed on how to

minimize this. (7) Evaluating area-based programs: more research is needed on how to do this.

(8) Theories: theories and programs should inform each other. (9) Generalizability: research is needed

on how to achieve this from small-scale demonstration projects to large-scale routine application.

(10) Descriptive validity: a checklist of items to be included in research reports should be specified.

Some other issues include the need to calculate statistical power, the need for research on case flow

problems, the difficulty of identifying the active ingredients of a program, the problem of obtaining

access to official data, the desirability of collecting victim survey and self-report data, conflict of

interest issues, and the need for more systematic reviews.

Key words: crime, evaluation, methodological quality, research design

The National Research Council (2005) report on Improving Evaluation of

Anticrime Programs is an excellent contribution to knowledge. It contains much

that I would applaud and little that I could disagree with. In commenting on it I

will take my own paper on Methodological quality standards for evaluation

research (Farrington 2003) as the starting point and organizing principle. I will

review the need for a simple, understandable, widely accepted scale of

methodological quality and will discuss issues arising from the National Research

Council (NRC) report, under the headings of statistical conclusion validity, internal

validity, construct validity, external validity, and descriptive validity. Before that, I

will highlight what was, to me, the most thought-provoking feature of the NRC

report: the emphasis on evaluability and its measurement.

I should preface my comments by saying that I will not focus on the

recommendations in the NRC report that concern organizational infrastructure.

For example, on pp. 5 Y 6, the report recommends that agencies that sponsor evaluation research in criminal justice (such as the National Institute of Justice)

should maintain a separate evaluation unit that is not subject to Bundue program and political influence^, that the agency personnel should have Brelevant research backgrounds^, that there should be continuity in such personnel, and that they should be assisted by outside experts. I agree with these recommendations, but I

have chosen to focus instead on more substantive issues of impact evaluation.

Journal of Experimental Criminology (2006) 2:329Y337 # Springer 2006

DOI: 10.1007/s11292-006-9012-y

Evaluability

The important idea that anti-crime programs might differ in their evaluability was

introduced on p. 20 and discussed in more detail on pp. 31Y33 of the NRC report. The clear assumption is that not all programs should be evaluated. It is argued that

a program should be selected for evaluation on the basis of (a) the significance of

the program for policy and practice, and (b) the extent to which the program is

amenable to sound evaluation research. Criteria under heading (b) include:

The program must be sufficiently well-defined to be replicable, the program

circumstances and personnel must be amenable to an evaluation study, the

requirements of the research design must be attainable (appropriate samples, data,

comparison groups, and the like), the political environment must be stable enough

for the program to be maintained during the evaluation, and a research team with

adequate expertise must be available to conduct the evaluation (p. 62).

The main problem with this idea of evaluability is the likely consequence that

very few programs will ever be evaluated. On p. 32, the NRC report says:

In the most recent round of evaluability assessments, a pool of approximately 200

earmarked programs was reduced to only eight that were ultimately judged to be

good candidates for an impact evaluation that would have a reasonable probability

of yielding useful information.

If 96% of programs escape an impact evaluation, it seems likely that criminal

justice evaluation research will not (to any significant extent) achieve its aim of

informing policy makers and practitioners about which programs work and which

programs do not work. This is the bottom-line question in which many people are

interested.

The NRC report sets very high standards for both programs and evaluations. Of

course, this is highly desirable. However, I wonder if it is wise to restrict research

funding to Rolls-Royce evaluations of Rolls-Royce programs if this means that the

vast majority of programs will not be evaluated. I wonder if it might be desirable to

develop, as an experiment, an evaluability score for each program out of 100 and

randomly choose a sample of programs in each decile for evaluation, moving down

from 100% to some minimum criterion score below which an evaluation would

clearly be a waste of time and money?

Methodological quality

The NRC report (pp. 8 Y11) documents various criticisms by the US General Accounting Office of evaluations conducted under the auspices of the Department

of Justice. The problem is that every evaluation can be criticized, but it should not

be concluded from this that every evaluation is (equally or fatally) flawed. The

reality is that evaluations vary greatly in their methodological quality. I have

argued (Farrington 2003) that a methodological quality scale is needed to

DAVID P. FARRINGTON330

communicate to scholars, policy makers, practitioners, the mass media, and the

general public that all research is not of the same quality and that more weight

should be given to results obtained in higher quality evaluation studies. The NRC

report (p. 50) recommends the use of a checklist.

I suggested that a new methodological quality scale or scales might be

developed, based on five criteria: internal validity, descriptive validity, statistical

conclusion validity, construct validity, and external validity. These terms will be

clarified shortly and used as organizing principles for my remaining comments on

the NRC report. However, I should mention that the most famous methodological

quality scale in criminal justice research is the five-point Maryland scientific

methods scale developed by Sherman et al. (1997). This has the great merit of

being simple and easy to communicate to non-specialists, but it only measures

internal validity and has a number of problems (e.g., arising from its

Bdowngrading^ system). Hence, there is scope for the development of new scales, possibly building on the Maryland scale as a useful starting point.

Ideally, criminal justice policy makers and practitioners should be trained so

that they can assess the methodological quality of evaluation studies using a scale

or scales. Shepherd (2003) contrasted the training of health practitioners (such as

doctors, dentists, and nurses) with criminal justice practitioners (such as police,

probation and prison officers, judges and lawyers). He pointed out that health

practitioners are trained in scientific methods by persons who are simultaneously

scientists and practitioners. Shepherd continued:

A major barrier in criminology, from a medical perspective, is the rarity of

integration of criminal justice research and practice in the same institution by

the academic practitioners that are the foundation of research, teaching, and

practice.... In contrast to medicine where clinical academics identify an

intervention, develop and mount the trial, and disseminate the findings, in

criminology researchers are often recruited to evaluate effectiveness of an

intervention that has been developed by practitioners with little or no

evaluation expertise (Shepherd 2003, p. 307).

The NRC report (pp. 49 Y 50) also notes that:

Practitioners rarely have the training and experience necessary to provide

sound judgments on research methods and implementation, though their

input may be very helpful for defining agency priorities and identifying

significant programs for evaluation.

Statistical conclusion validity

Statistical conclusion validity focuses on whether the presumed cause (the

intervention) and the presumed effect (the outcome) are related, and how strongly

they are related (the effect size). The most common method of assessing whether

the intervention and outcome are related is to test the statistical significance of the

QUALITY AND THE EVALUATION OF ANTI-CRIME PROGRAMS 331

relationship (null hypothesis significance testing). However, this method has many

flaws; for example, a significant result could indicate a small effect in a large

sample or a large effect in a small sample. Nowadays, one is recommended to

calculate the effect size and its 95% confidence interval. This contains all the

information provided by traditional null hypothesis testing and focuses attention

more appropriately on the magnitude and precision of the effect size estimate

(Shadish et al. 2002, p. 43).

The NRC report (p. 42) notes that:

A Bsmall^ effect size as defined by Cohen (1988) would correspond to the difference between a .40 recidivism rate for the intervention group and .50

for the control group. A reduction of this magnitude for a large criminal

population, however, would produce a very large societal benefit.

For many years I have argued that d values are potentially misleading and could

usefully be converted into percentage differences in recidivism. While d = .4 might

possibly be viewed as a Bmodest^ effect, I do not think that the same term could be applied to the equivalent reduction from 50% to 30% reconvicted, potentially

saving many thousands of crimes.

The benefit:cost ratio is arguably a better and more understandable measure of

effect size than is the d value. This ratio is only briefly touched on in the NRC

report (pp. 15 Y 16), but it would be desirable to calculate it in many evaluations. For example, Painter and Farrington (2001) estimated that the financial benefits of

improved street lighting after 1 year (based on reduced crimes) exceeded the

financial costs by between 2.4 and 10 times. The argument that B$5 are saved for every $1 expended^ is very convincing to policy makers.

The NRC report raises two other important issues relevant to statistical

conclusion validity, namely statistical power (p. 42) and case flow problems

(p. 47). Arguably, anyone who is planning to conduct an evaluation should

carry out a statistical power analysis beforehand to assess the extent to which

the design is able to detect the likely program effect. With small numbers such

as 30 experimentals and 30 controls, even large effects are unlikely to be

detected (NRC report, p. 42). Also, researchers should learn from the long

history of case flow problems in experimental research and either anticipate

them more accurately in their planning or devise better methods of maintaining

the case flow over time.

Internal validity

Internal validity refers to the extent to which the evaluation demonstrates

unambiguously that the intervention caused a change in the outcome measure.

It is generally viewed as the most important type of validity. The NRC report

(pp. 35 Y 40) contains an excellent discussion of internal validity and of three important evaluation designs: randomized experiments, quasi-experiments, and

DAVID P. FARRINGTON332

observational methods with statistical modelling. However, I think that the

usefulness of observational methods in comparison with the other two designs is

somewhat doubtful in most cases. Statistical modelling can only estimate an

effect size accurately if all relevant variables are known, measured, and included

in the equations. It seems to me that experimental control of extraneous variables

is likely to be more effective than statistical control in most cases, although there

may be issues (e.g., evaluation of the effects of national crime policies) where

statistical modelling is the only feasible method. A randomized experiment

controls for all measured and unmeasured extraneous variables.

The NRC report (p. 36) correctly notes that the main threat to internal validity

in randomized experiments is differential attrition from experimental versus

control conditions. This was also highlighted by Farrington and Welsh (2005).

Other problems caused by attrition (in estimating prevalence) are pointed out on

p. 30. The clear implication for me is that minimizing attrition should be a high

priority in all research projects, that significant resources should be devoted to

maximizing response rates, and that more research is needed on minimizing

attrition. I also believe that missing data imputation methods are less satisfactory

than maximizing response rates. We managed to maintain a high response rate

on the Cambridge Study in Delinquent Development, which is a prospective

longitudinal survey of 400 London male subjects; 93% were interviewed 40

years after the start of the project. Our methods of tracing and securing

cooperation are described by Farrington et al. (1990).

I would have liked to see more explicit discussion in the NRC report of the

problems of evaluating area-based programs. The executive summary (p. 1) begins

by discussing (a) interventions directed towards individuals; (b) interventions in

neighbourhoods, schools, prisons, or communities; and (c) interventions at a broad

policy level. However, most of the report seems to focus on interventions directed

toward individuals, where internal validity problems can be most easily overcome.

I would like to see more research addressing the challenging issue of how to

evaluate area-based programs, such as closed-circuit television, target hardening,

neighbourhood watch, community policing, and so on, where randomized experi-

ments based on large numbers of units are rarely possible. Research is needed on

such topics as how to measure effect size in area-based research and the

importance of regression to the mean (see e.g., Farrington and Welsh 2006).

Construct validity

Construct validity refers to the adequacy of the operational definition and

measurement of the theoretical constructs that underlie the intervention and the

outcome. The NRC report (p. 43) recommends a process evaluation Bto provide a full picture of the program. If the evaluation then finds a significant effect, it will

be possible to clearly describe what produced it.^ However, it is often challenging to identify the active ingredients of a multimodal program, and further research

QUALITY AND THE EVALUATION OF ANTI-CRIME PROGRAMS 333

that systematically varies different components is likely to be needed. It is also

important to describe what the control condition received in detail, so that it is

possible to answer the question: Bcompared to what?^ Ideally, evaluations should advance knowledge about theories of crime, just as

theories of crime should inform the design of interventions. Loeber and Farrington

(1998, p. 1) began their book on Serious and Violent Juvenile Offenders: Risk

Factors and Successful Interventions with the statement:

The volume aims to integrate knowledge about risk and protective factors

and about the development of juvenile offending careers with knowledge

about prevention and intervention programs... so that conclusions from one

area can inform the other.

The NRC report (p. 66) recommends:

Development and improvement of new and existing data bases in ways that

would better support impact evaluation of criminal justice programs and

measurement studies that expand the repertoire of relevant outcome variables

and knowledge about their characteristics and relationships for purposes of

impact evaluation (e.g., self-report delinquency and criminality, official

records of arrests, convictions, and the like, measures of critical mediators).

A major problem for many evaluators (as noted by the NRC report on p. 29) is to

obtain access to official data. Ideally, routinely collected official data should be

used in evaluations, but this is not always possible. More efforts are needed by

government agencies to increase the quality and accessibility of official data. Also,

researchers should receive funding to collect victim survey and self-reported

offending data in order to overcome the difficulty that programs might only have

changed the behaviour of official agencies (e.g., in recording or reporting crime, or

in arresting or convicting offenders).

External validity

External validity refers to the generalizability of causal relationships across

different persons, places, times, and operational definitions of interventions and

outcomes. A major problem is that effect sizes are typically much greater in small-

scale demonstration projects (Befficacy^ studies: see p. 19 of the NRC report) than in large-scale routine application (Beffectiveness^ studies). For example, in their review of 40 family-based prevention studies, Farrington and Welsh (2003) found

that effect sizes were significantly negatively correlated with sample sizes. More

research is needed on how to translate small-scale tightly-controlled programs

administered by high quality staff into large-scale use.

Conflict of interest issues may arise where a program is developed, marketed,

and evaluated by the same person, or where an evaluation is funded by a

government agency with a great stake in the results (e.g., because it has already

expended millions of dollars on the program and has already trumpeted its

DAVID P. FARRINGTON334

effectiveness in the mass media). The alternative is to achieve a clear separation

between the program implementers and evaluators, but, as the NRC report notes

(p. 57), there may then be problems of getting cooperation from practitioners (and

case flow problems). Support from funding agencies is crucial in overcoming

these difficulties.

Descriptive validity

Descriptive validity refers to the adequacy of reporting of key features of

evaluations (e.g., design, sample sizes, characteristics of experimental units,

descriptions of experimental and control conditions, outcome measures, effect

sizes). This information is needed to carry out systematic reviews and meta-

analyses as recommended by the NRC report (p. 44). Ideally, any new evaluation

should be preceded by a systematic review of relevant prior evaluations, so that

each new study builds on the experiences of previous researchers. Also ideally, it

would be desirable for professional associations, funding agencies, journal editors,

and the Campbell Collaboration (see Farrington and Petrosino 2001) to get

together to develop a checklist of items that must be included in all research

reports on impact evaluations. I ended my first review of randomized experiments

in criminology (Farrington 1983) with a checklist of items that should be reported,

and, since then, the CONSORT statement has been developed for medical research

(Moher et al. 2001) and adopted for American Psychological Association journals.

Conclusions

Unfortunately, evaluation research has tended to be a Bpoor relation^ in criminology. It has never had high status within the discipline, which has

traditionally valued more academic, theoretical studies of the causes of crime

and rather looked down on applied, policy orientated research. Until the recent

founding of the Journal of Experimental Criminology, there was no journal that

focused on criminological evaluations, although many interesting studies have

been published recently in Criminal Justice and Behavior and in the recently

founded Criminology and Public Policy. There was no organization focusing on

criminological evaluations until the recent founding of the Campbell Collaboration

and the Academy of Experimental Criminology.

In the past decade it has been clear that the tide is turning and that evaluation

research in criminology is becoming more prominent and more valued. In the

interests of reducing crime and its associated social problems this development is

extremely welcome. The NRC report should also be welcomed, since it is likely to

encourage high quality evaluations of the effectiveness of criminological

interventions. I hope that both evaluators and funding agencies such as the

National Institute of Justice (NIJ) will make extensive use of it.

QUALITY AND THE EVALUATION OF ANTI-CRIME PROGRAMS 335

References

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale,

NJ: Lawrence Erlbaum.

Farrington, D. P. (1983). Randomized experiments on crime and justice. In M. Tonry & N.

Morris (Eds.), Crime and justice, vol. 4 (pp. 257Y308). Chicago: University of Chicago Press.

Farrington, D. P. (2003). Methodological quality standards for evaluation research. Annals

of the American Academy of Political and Social Science 587, 49 Y 58. Farrington, D. P. & Petrosino, A. (2001). The Campbell Collaboration Crime and Justice

Group. Annals of the American Academy of Political and Social Science 578, 35 Y 49. Farrington, D. P. & Welsh, B. C. (2003). Family-based prevention of offending: A meta-

analysis. Australian and New Zealand Journal of Criminology 36, 127Y151. Farrington, D. P. & Welsh, B. C. (2005). Randomized experiments in criminology: What

have we learned in the last two decades? Journal of Experimental Criminology 1, 9 Y 38. Farrington, D. P. & Welsh, B. C. (2006). How important is Bregression to the mean^ in area-

based crime prevention research? Crime Prevention and Community Safety 8, 50 Y 60. Farrington, D. P., Gallagher, B., Morley, L., St. Ledger, R. & West, D. J. (1990).

Minimizing attrition in longitudinal research: Methods of tracing and securing cooperation

in a 24-year follow-up study. In D. Magnusson & L. Bergman (Eds.), Data quality in

longitudinal research (pp. 122Y147). Cambridge: Cambridge University Press. Loeber, R. & Farrington, D. P. (Eds.) (1998). Serious and violent juvenile offenders: Risk

factors and successful interventions. Thousand Oaks, CA: Sage.

Moher, D., Schulz, K. F. & Altman, D. (2001). The CONSORT statement: Revised

recommendations for improving the quality of reports of parallel-group randomized trials.

Journal of the American Medical Association 285, 1987Y1991. National Research Council (2005). Improving evaluation of anticrime programs. Wash-

ington, DC: National Academies Press.

Painter, K. A. & Farrington, D. P. (2001). The financial benefits of improved street lighting,

based on crime reduction. Lighting Research and Technology 33, 3 Y12. Shadish, W. R., Cook, T. D. & Campbell, D. T. (2002). Experimental and quasi-

experimental designs for generalized causal inference. Boston: Houghton Mifflin.

Shepherd, J. P. (2003). Explaining feast or famine in randomized field trials: Medical

science and criminology compared. Evaluation Review 27, 290 Y 315. Sherman, L. W., Gottfredson, D. C., MacKenzie, D. L., Eck, J. E., Reuter, P. & Bushway, S.

(1997). Preventing crime: What works, what doesn_t, what_s promising. Washington DC: Office of Justice Programs.

About the author

David P. Farrington, O.B.E., is Professor of Psychological Criminology at the Institute of

Criminology, Cambridge University, UK, and Adjunct Professor of Psychiatry at Western Psychiatric

Institute and Clinic, University of Pittsburgh, USA. He is a Fellow of the British Academy, of the

Academy of Medical Sciences, of the British Psychological Society and of the American Society of

Criminology, and an Honorary Life Member of the British Society of Criminology and of the Division

of Forensic Psychology of the British Psychological Society. He is co-Chair of the Campbell

Collaboration Crime and Justice Group, a member of the Board of Directors of the International Society

of Criminology, a member of the jury for the Stockholm Prize in Criminology, joint editor of Cambridge

DAVID P. FARRINGTON336

Studies in Criminology and of the journal Criminal Behaviour and Mental Health, and a member of the

editorial boards of 15 other journals. He received B.A., M.A. and Ph.D. degrees in psychology from

Cambridge University, the Sellin Y Glueck Award of the American Society of Criminology for international contributions to criminology, the Sutherland Award of the American Society of

Criminology for outstanding contributions to criminology, the Joan McCord Award of the Academy

of Experimental Criminology, the Beccaria Gold Medal of the Criminology Society of German-

Speaking Countries, and the Hermann Mannheim Prize of the International Centre for Comparative

Criminology.

QUALITY AND THE EVALUATION OF ANTI-CRIME PROGRAMS 337

Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.