Policy critique on death penalty?

845© 2009 American Society of Criminology Criminology & Public Policy • Volume 8 • Issue 4

P O L I C Y E S S AY

D e t e r r e n c e a n D e x e c u t I o n s

Can’t tell Comments on “Does the death penalty save lives?”

richard berk U n i v e r s i t y o f P e n n s y l v a n i a

T omislav V. Kovandzic, Lynne M. Vieraitis, and Denise Paquette Boots (2009) offer us another round in the dueling econometrics of the death penalty. They bring to the dispute the most recent data readily available, substantial theoretical sophistication,

and an encyclopedic application of standard econometrics for panel data. They find no evidence for deterrent effects. We have probably not, however, heard the end of this issue. With each new year, another year’s worth of data become available. There will be variation over time in death penalty practices and in homicide patterns. The political process will continue to seek findings to its liking. And researchers with sunk costs in the death penalty debates will have strong incentives to stay the course.

Yet, we are unlikely ever to know what impact on crime the death penalty may have had. Kovandzic et al.’s (2009) study is my sense of where we were 20 years ago, where we are now, and where we will be 20 years in the future. Therefore, as long as we have some researchers who claim to have found deterrent effects for the death penalty, it is good that we have other researchers who claim not to have found deterrent effects. We always will have advocates who are looking for simple and congenial answers to the most difficult empirical questions. Therefore, a message that simple answers do not exist, when in fact none do exist, is healthy.

That said, I am fundamentally skeptical about the research tradition this work represents. My comments, therefore, will focus on why “can’t tell” is where we now stand and where we will be standing in the future. I also will suggest more promising alternatives and briefly conclude with lessons for policy. The discussion is necessarily brief, but extensive references are provided for those readers who want to consider the issues in more depth.

Direct correspondence to richard berk, Department of Criminology, university of Pennsylvania, 483 mcNeil building, 3718 locust Walk, Philadelphia, PA 19104 (e-mail: [email protected]).

09008-CrimJournal-Guts.indd 845 10/27/09 7:31:59 PM

Criminology & Public Policy846

Well-Known Problems with the conventional econometric Paradigm Growing out of economic research undertaken 60 years ago and becoming the dominant sta- tistical formulation within economics, political science, sociology, and criminology during the 1970s, the econometric paradigm—which can found in any elementary econometrics textbook (Cameron and Trivedi, 2005: 3–10; Pindyck and Rubinfeld, 1981: xiiixvii)—has the following features. It depends fundamentally on a model, which is a simplified theory of how the data were generated. The canonical model can be represented as

y i = f(X

i ) + ε

i (1)

where i = 1, 2, … N observations, y i is the quantitative response variable, X

i is a matrix that

includes all explanatory variables (and usually a column of 1s for the constant), and ε i is an

unobservable disturbance term that behaves as if each disturbance is drawn independently of one another and X

i from a single distribution with a mean of 0.0 and a common variance. The

f(X i ) needs to be specified before looking at the data. The canonical model can be extended to

encompass (1) more than one equation, (2) response variable(s) that are categorical or ordinal, (3) data that can be arrayed in time and space, and (4) less-restrictive disturbance terms proper- ties. However, the basic features remain.

In the absence of a randomized experiment, one bedrock assumption exists that researchers cannot avoid no matter how elaborate the econometrics employed. To obtain proper estimates of the key regression parameters, the disturbances must be uncorrelated with the regressors—at least in large samples. At a minimum, this process requires that (1) one has in X

i all regressors

required by the “correct” theory (or that one can persuasively argue that the omitted regressors captured by ε

i are uncorrelated with regressors in the model); (2) that all regressors and the

response variable(s) are measured well;1 and (3) that one knows and can specify the correct f(X

i ). In practice, it is impossible to know whether these requirements are met (Berk, 2003:

Section 9.6; Freedman, 2005: Section 8.6), especially with observational data of the kind used in death-penalty deterrence research (Berk, 2005), and ample reasons abound for enormous skepticism. For example, if you read the fine print in the primary sources, then you will learn that you cannot test your way to the correct model unless you are prepared to make several heroic assumptions about what is not being tested. As the literature on deterrence and the death penalty illustrates well, endless debate of the “my-model-is-better-than-your-model” variety can follow from these problems. Now, we have at least 40 years of sophisticated criticism of the econometric paradigm (Berk, 2003; Box, 1976; Breiman, 2001; de Leeuw, 1994; Holland, 1986; Leamer, 1978; Lieberson, 1985; Rubin, 1986; Freedman, 1987, 2005; Kish, 1987; Pearl, 2000), and some of the most interesting recent criticism comes for the leading econometricians (Heckman, 2000; Imbens, 2009; Manski, 1995). I think a fair response to econometrician Ed

1. The response variable should be measured with no bias and, ideally, with little random measurement error. The regressors must be measured with no bias and with no random measurement whatsoever—at least in theory. It is important to appreciate that even random measurement error (i.e., independent errors with a mean 0.0 uncorrelated with the true score) is not permitted formally for the regressors.

P o l i c y e s s a y D e t e r r e n c e a n d e x e c u t i o n s

09008-CrimJournal-Guts.indd 846 10/27/09 7:32:00 PM

847Volume 8 • Issue 4

Leamer’s early plea (1983) to “take the con out of econometrics” is that the con is built in. We need other approaches.

alternatives for observational Data If traditional econometric methods can be a dead end for most observational data sets, then what are the alternatives? Elsewhere, this matter is addressed in depth (Berk, 2003: Ch. 11; Berk, Sherman, Barnes, Kurtz, and Ahlman, 2009; Breiman, 2001; Gifi, 1990; Imbens, 2004). A few highlights will have to suffice here.

True Experiments and Quasi-Experiments When traditional econometric procedures are applied to observational data, they risk being compromised by weak research designs. Since the mid-1960s, it has been apparent that a wide range of designs can improve on conventional observational studies (Campbell and Stanley, 1963). If causal inference is the top priority, then randomized experiments are the ideal. When they are not practical, several strong quasi-experimental designs often will perform nearly as well (Berk, Barnes, Ahlman, and Kurtz, 2010; Cook, Shadish, and Wong, 2008). Indeed, regres- sion continuity designs are one topic on which econometricians have recently made important contributions (Imbens and Lemieux, 2008). In the absence of either a randomized experiment or a strong quasi-experiment, arriving at credible estimates of causal effects through econometric modeling can be difficult.

Model-Lite Causal Analysis When one has no choice but to work with observational data, as is the case here, approaches to causal inference are available that are much less model dependent, more transparent, and more credible. Morgan and Winship (2007) provide an excellent overview that builds on the work of William Coschran, Donald Rubin, Paul Rosenbaum, Judea Pearl, and many others. The basic strategy is to conceptualize observational data within a randomized experiment framework that is conditional on a set of covariates. This strategy has several benefits (Rubin, 2008). A reliance on the Neyman–Rubin casual model makes a critical distinction between one or more causal

variables and one or more covariates, or nuisance variables. Casual inference is of no concern for the covariates. This point permits a much clearer rendering of appropriate counterfactuals than death-penalty deterrence research has achieved to date. It also fosters analysis strategies that do not have to depend as heavily on untestable assumptions. For example, because one does not care about the causal impact of the covariates, matching or stratifying on those covariates can be a useful approach (Rosenbaum, 2002). Finally, this perspective helps prevent data snoop- ing (Freedman, 2005: Section 4.9). The response variable is ignored while efforts to achieve balance in the data are undertaken. One cannot capitalize on chance that leads to overfitting. It also precludes model selection that is recognized increasingly as a danger to causal inference and estimation far more fundamentally (Leeb and Pötscher, 2005).

b e r k

09008-CrimJournal-Guts.indd 847 10/27/09 7:32:00 PM

Criminology & Public Policy848

“Mere” Description A long tradition in statistics and some social sciences favors quantitative description. Among the preferred tools are multivariate statistics (Anderson, 1958; Gifi, 1990), exploratory data analysis (Diaconis, 1985; Tukey, 1977), dynamic graphics (Cook and Swayne, 2007), and machine/ statistical learning (Hastie, Tibshirani, and Friedman, 2009; Lohr, 2009). The intent is to find systematic patterns in data without worrying about cause and effect. With modern algorithms and powerful computers, the search possibilities virtually are limitless. Some researchers see this technology as a precursor to explicit model building. Others see it as a substitute for model building, especially when credible model building is not practical. These approaches are find- ing their way into criminology in several interesting applications. GIS studies of crime are one common illustration (Chainey and Ratcliffe, 2005; Groff and La Vigne, 2001). Others include inductively developed crime forecasting models (Berk, Barnes, Ahlman, and Kurtz, 2010) and capturing crime dynamics in time and space (Berk and McDonald, 2009).

conclusions Responding several years ago to possibly the same set of studies that motivate the Kovandzic et al.’s (2009) work, I wrote (Berk, 2005: 328): “[My] analyses reported here are hardly exhaustive and are perhaps affected by [my] misunderstandings about the data provided, or by errors in the data themselves. Nevertheless, the results raise serious questions about whether anything useful about the deterrent value of the death penalty can ever be learned from an observational study with the data that are likely to be available.” The panel data have been made more current, and a wider rage of econometric procedures have been brought to bear. But the observational design still is weak and the measurement problems in the data essentially remain, as do all the questions that naturally follow from trying to build a credible casual model.

I stand by my earlier conclusions. Nevertheless, Kovandzic et al. (2009) have done a great service by showing once again that conventional econometric modeling can be a fragile enterprise. They also have helped to debunk claims that the death penalty has demonstrable deterrent value and, possibly more important, they perhaps have motivated some researchers to consider alternative empirical approaches. Three lessons for evidence-based policy follow. First, no credible evidence exists that the death penalty, as implemented in the United States since 1979, has any deterrent value. Second, no credible evidence exists to rule out any deterrent effects. Third, if evidence-based policy is to be more than another catch-phase, then one must be careful about what qualifies as evidence.

When causal effects are primary, evidence-based policy means implementing true experiments or strong quasi-experiments. When causal effects are important, but significant tradeoffs exist with other priorities, observational studies analyzed through well-executed matching methods can provide useful evidence. When estimates of casual effects are secondary, premature, or ir- relevant, description alone can be instructive. Finally, judgment must be used to identify those empirical questions whose answers are beyond the data and the methods by which those data

P o l i c y e s s a y D e t e r r e n c e a n d e x e c u t i o n s

09008-CrimJournal-Guts.indd 848 10/27/09 7:32:00 PM

849Volume 8 • Issue 4

can be analyzed. Trying to estimate the causal impact of the death penalty on crime is one good illustration. There are others.

references Anderson, Theodore. 1958. An introduction to multivariate statistical analysis, 1st Edition.

New York: Wiley.

Berk, Richard A. 2003. Regression analysis: A constructive critique. Newbury Park, CA: Sage.

Berk, Richard A. 2005. New claims about executions and general deterrence: De ja vu all over again. Journal of Empirical Legal Studies, 2: 303–330.

Berk, Richard A. 2009. What now? Some brief reflections on model-free data analysis. International Econometric Review, 1: 14–23.

Berk, Richard A., Geoffrey Barnes, Lindsay Ahlman, and Ellen Kurtz. 2010. When second best is good enough: A comparison between a true experiment and a regression discontinuity quasi-experiment. Journal of Experimental Criminology. In press.

Berk, Richard A. and John McDonald. 2009. The dynamics of crime regimes. Criminology. In press.

Berk, Richard A., Lawrence Sherman, Geoffrey Barnes, Ellen Kurtz, and Lindsay Ahlman. 2009. Forecasting murder within a population of probationers and parolees: A high stakes application of statistical forecasting. Journal of the Royal Statistical Society, 172: 191–211.

Box, George E. P. 1976. Science and statistics. Journal of the American Statistical Association, 71: 791–799.

Breiman, Leo. 2001. Statistical modeling: Two cultures (with discussion). Statistical Science, 16 :199–231.

Cameron, A. Colin and Pravin K. Trivedi. 2005. Microeconomics: Methods and applications. Cambridge, UK: Cambridge University Press.

Campbell, Donald T. and Julian Stanley. 1963. Experimental and quasi-experimental designs for research. New York: Wadsworth.

Chainey, Spencer and Jerry Ratcliffe. 2005. GIS and crime mapping. New York: Wiley.

Cook, Dianne and Deborah F. Swayne. 2007. Interactive dynamic graphics and data analysis. New York: Springer.

Cook, Thomas D., William R. Shadish, and Vivian C. Wong 2008. Three conditions under which experiments and observational studies produce comparable causal estimates: new findings from within-study comparisons. Journal of Policy Analysis and Management, 727: 724–750.

de Leeuw, Jan. 1994. Statistics and the sciences, in (I. Borg and P.P. Mohler, eds.), Trends and perspectives in empirical social science. New York: Walter de Gruyter.

Diaconis, Persi. 1985. Theories of data analysis: From magical thinking through classical statistics. In (D.C. Hoaglin, F. Mosteller, and J. Tukey, eds.), Exploring data tables, trends, and shapes. New York: Wiley.

b e r k

09008-CrimJournal-Guts.indd 849 10/27/09 7:32:00 PM

Criminology & Public Policy850

Freedman, David A. 1987. As others see us: A case study in path analysis (with discussion). Journal of Educational Statistics, 12: 101–223.

Freedman, David A. 2005. Statistical models: Theory and practice. Cambridge, UK: Cambridge University Press.

Gifi, Albert. 1990. Nonlinear multivariate analysis. New York: Wiley.

Groff, Elizabeth and Nancy G. La Vigne. 2002. Forecasting the future of predictive crime mapping. In (N. Tilley, ed.), Analysis for crime prevention. Monsey, NY: Criminal Justice Press.

Hastie, Trevor, Robert Tibshirani, and Jerome Friedman. 2009. The elements of statistical learning, 2nd Edition. New York: Springer.

Heckman, James J. 2000. Causal parameters and policy analysis in economics: A twentieth century retrospective. Quarterly Journal of Economics, 115: 45–97.

Holland, Pail W. 1986. Statistics and causal inference. Journal of the American Statistical Association, 81: 945–960.

Imbens, Guido W. 2004. Nonparametric estimation of average treatment effects under exogeneity: A review. Review of Economics and Statistics, 86: 4–29.

Imbens, Guido W. 2009. Better late than nothing: Some comments on Deaton (2009) and Heckman and Urzua (2009). Working paper, Department of Economics, Harvard University, Cambridge, MA.

Imbens, Guido W. and Thomas Lemieux. 2008. Regression continuity designs: A guide to practice. Journal of Econometrics, 142: 615–635.

Kish, Leslie. 1987. Statistical design for research. New York: Wiley.

Kovandzic, Tomislav V., Lynne M. Vieraitis, and Denise Paquette Boots. 2009. Does the death penalty save lives? New evidence from state panel data, 1977 to 2006. Criminology & Public Policy. This issue.

Leamer, Edward E. 1978. Specification searches: Ad hoc inference with non- experimental data. New York: Wiley.

Leamer, Edward E. 1983. Let’s take the con of econometrics. American Economics Review, 73: 31–43.

Leeb, Hannes and Benedikt M. Pötscher. 2005. Model selection and inference: Facts and fiction. Econometric Theory, 21: 21–59.

Lieberson, Stanley. 1985. Making it count: The improvement of social research and theory. Berkeley: University of California Press.

Lohr, Steven. 2009, August 5. For today’s graduate, just one word: Statistics. New York Times.

Manski, Charles. 1995. Identification problems in the social sciences. Cambridge, UK: Cambridge University Press.

Morgan, Stephen L. and Christopher Winship. 2007. Counterfactuals and causal inference: Methods and principles for social research. Cambridge, UK: Cambridge University Press.

Pearl, Judea. 2000. Causality: Models, reasoning and inference. Cambridge, UK: Cambridge University Press.

P o l i c y e s s a y D e t e r r e n c e a n d e x e c u t i o n s

09008-CrimJournal-Guts.indd 850 10/27/09 7:32:00 PM

851Volume 8 • Issue 4

Pindyck, Robert S. and Daniel L. Rubinfeld. 1981. Econometric models and economic forecasts. New York: McGraw Hill.

Rosenbaum, Paul 2002. Observational studies, 2nd Edition. New York: Springer-Verlag.

Rubin, Donald B. 1986. Which ifs have causal answers? Journal of the American Statistical Association, 81: 961–962.

Rubin, Donald B. 2008. For objective causal inference, design trumps analysis. Annals of Applied Statistics, 2: 808–840.

Tukey, John. 1977. Exploratory data analysis. Reading, MA: Addison-Wesley.

richard berk is a professor in the departments of statistics and criminology at the University of Pennsylvania. He is an elected fellow of the American Association for the Advancement of Science, the American Statistical Association, as well as the Academy of Experimental Crimi- nology and has been a member of the Committee on Applied and Theoretical Statistics of the National Research Council and the Social Science Research Council’s Board of Directors. He has been awarded the Paul F. Lazarsfeld Award by the Methodology Section of the American Sociological Association. Berk works on various topics in applied statistics that include causal inference, statistical learning, and methods for evaluating social programs. Among his criminol- ogy applications are inmate classification and placement systems, forecasts of behavior while on parole or probation, the role of race in capital punishment, detecting violations of environmen- tal regulations, claims that the death penalty serves as a general deterrent, and describing and forecasting short-term changes in urban crime patterns. Berk’s latest book is Statistical learning from a regression perspective for the Springer Series in Statistics (2008).

b e r k

09008-CrimJournal-Guts.indd 851 10/27/09 7:32:01 PM