Police Operations

Read the following transcript in which Dr. James Chip Coldren discusses the need for methodological rigor in studies of policing, highlighting "Smart Policing."

Summary: Dr. James Chip Coldren discusses the need for methodological rigor in studies of policing, highlighting "Smart Policing."

· DR. JAMES CHIP COLDREN, PHD: Hi. My name is Dr. James Chip Coldren. [Dr. James Chip Coldren, PhD, Managing Director for Justice Programs, Center for Naval Analyses (CNA)] I'm the managing director of Justice programs at the CNA Institute for Public Research. For today's case study, we're going to look at research in policing, and look at several different ways that are being attempted to advance the science of police research. So I'll talk a little bit about the state of the art in police research.

· 00:38

DR. JAMES CHIP COLDREN, [continued]: I'll provide some recent examples of attempts to improve the quality of research in policing. Then, we'll spend just a little bit of time talking about the ongoing needs and challenges in policing research. [The Maryland Scientific Scale] Now, before we start to talk about the state-of-the-art in policing research, we need to talk about the Maryland Scientific Scale. This is a scale that was developed by Larry Sherman, [Larry Sherman] and his colleagues at University of Pennsylvania in the late 1990s to assess

· 01:11

DR. JAMES CHIP COLDREN, [continued]: the methodological rigor [methodological rigor] of research designs. The scale goes from one to five, with a score of one indicating a low rigor descriptive study, and a score of five indicating the most rigorous study that you can attempt, which is a randomized controlled experiment. [randomized controlled experiment] Typically, in the world of criminal justice and social science research, a score of a three or higher on the Maryland Scientific Scale

· 01:40

DR. JAMES CHIP COLDREN, [continued]: will indicate a rigorous methodology with a comparative design. So for example, if your study scores at level three, that means that your study involves a comparison between two or more comparable units of analysis, such as groups of individuals or police beats or neighborhoods; one group experiencing the program or intervention, and another group not experiencing. So it's a comparative design. A score of level four includes a comparison between multiple units within and without the program or intervention, controlling for other factors

· 02:17

DR. JAMES CHIP COLDREN, [continued]: or using comparison units that evidence only minor differences. And a level five, as I indicated before, is a randomized controlled experiment, where you have treatment and control units that are and are not receiving the program or intervention. Now, let me go back to level four for a second. Typically, when a study scores for on the Maryland Scientific Scale, that means that there's a comparative design with matched units. They're not randomly assigned, but let's say, for example,

· 02:47

DR. JAMES CHIP COLDREN, [continued]: that you pick a number of neighborhoods to implement a new policing program. To introduce the comparison, you would find another set of neighborhoods that are very similar and characteristics to the neighborhoods that you assigned the treatment or the intervention to. So that's typically called a match design, or a quasi-experimental design. [Need for Methodological Rigor] Several years ago, the perception

· 03:19

DR. JAMES CHIP COLDREN, [continued]: developed that there was a need for more methodological rigor in policing studies. This is partly the result of research that was conducted by David Weisburd and his colleagues [David Weisburd, et al.] at George Mason University, which was published in 2010. David Weisburd and his colleagues examined over 5,000 research reports on the effectiveness of problem-oriented policing. These were reports that were published in journals, or they were reports that were issued by government agencies. But that they identified over 5,000 reports. And they examined each of those reports

· 03:53

DR. JAMES CHIP COLDREN, [continued]: for the type of research methodology and the level of rigor in the research methodology. What they found is that of those over 5,000 reports, there were only 11 studies with a methodological rigor of a level three or higher on the Maryland Scientific Scale. So only a very small percentage of those 5,000 studies actually had a comparative design that allowed you to assess the impact on the problem that was being worked on with a comparative design to a neighborhood or a police beat that

· 04:31

DR. JAMES CHIP COLDREN, [continued]: was comparable to the place that got the intervention in the first place. So as a result of this and some of the work that had been done, there was clearly a perceived need to shore up the research methodology in research on police effectiveness. [Recent Advancements] OK. So let's talk about some recent advancements that have made and some recent attempts at improving the rigor of research in policing.

· 05:01

DR. JAMES CHIP COLDREN, [continued]: I'm going to talk about the Campbell Collaboration systematic reviews of policing research. I'm going to talk about the Center for Evidence-Based Crime Policy at George Mason University. I'll talk a little bit about the National Institute of Justice crimesolutions.gov initiative, and I'll talk about the Bureau of Justice Assistance Smart Policing Initiative. These are all recent attempts in the last five to seven years to improve the methodological rigor of research and policing. [Campbell Collaboration] The Campbell Collaboration is an international research network that produces systematic reviews of the effects

· 05:43

DR. JAMES CHIP COLDREN, [continued]: of social interventions in crime and justice, education, international development, and social welfare. So typically, what the Campbell Collaboration will do is identify a topic something around policing, and identify as many research reports and studies as possible to include in their review, look at the methodological rigor of those studies, and then come up with a meta-analysis, or an overall assessment of the effectiveness of that particular approach to policing.

· 06:12

DR. JAMES CHIP COLDREN, [continued]: Since 2007, the Campbell Collaboration has conducted 10 reviews of law enforcement and policing. And several are underway right now. In 2013, they published a systematic review of research that has been conducted on police legitimacy. For this review, they identified over 900 studies on police legitimacy in the United States, 30 of which had sufficient methodological rigor to include in their analysis. So here we are again, identifying a great number of studies of a police topic-- in this case, police

· 06:48

DR. JAMES CHIP COLDREN, [continued]: legitimacy-- with only 30 out of 900 having strong methodological designs. And in those 30, none of those reached the level five, which is the experimental design. In 2012, they published a report on a systematic review of "Hot Spots" policing. ["Hot Spots" Policing (2012)] This is a very popular approach to policing these days. It's called place-based policing. Policing that identifies the small geographic areas that account for a high percentage of criminal activity.

· 07:18

DR. JAMES CHIP COLDREN, [continued]: And so police agencies and communities focus their resources on those hot spots. So they identified actually 19 rigorous studies of hot spots policing, 10 of which involved randomized experiments. So here, we have an example where there's more randomized experiments than we normally find in policing research. Also in 2012, they published a systematic review of studies on focused deterrence. [focused deterrence (2012)]

· 07:46

DR. JAMES CHIP COLDREN, [continued]: Focused deterrence is an approach that looks at people, rather than places. Typically, in a focused deterrence approach, you will identify a number of individuals who are what you would think of as prolific offenders. The small group of people that are committing the most violence or the most trouble in certain neighborhoods or communities. And you focus your efforts on those people, not necessarily on hot spots or places. And those efforts typically involve two things: a strong message delivered by law enforcement and community representatives that explain to the individuals identified that they are known chronic and prolific offenders, that the police and probation and corrections

· 08:29

DR. JAMES CHIP COLDREN, [continued]: and the prosecutors now have them on a list and are watching them carefully. If they continue to persist in violent criminal activity, they'll feel the full force of the law, maybe federal law, and maybe federal imprisonment. But at the same time, they offer this group of offenders legitimate opportunities to engage in training and services and to pursue pro-social legitimate lifestyles. So the focus is on the offenders, the deterrence is the community message that's given to them to explain to them why they're being watched,

· 09:02

DR. JAMES CHIP COLDREN, [continued]: what will happen if they're caught, and what opportunities they have to change their behavior. It's a very popular approach in criminal justice these days. The Campbell Collaboration found 10 rigorous studies of focused deterrence, but no experiments. So they ended up determining that focused deterrence was a promising approach to reducing crime, because in almost each of those 10 studies, they found a reduction in crime and a reduction

· 09:32

DR. JAMES CHIP COLDREN, [continued]: in criminal behavior as a result of the focused deterrence strategy. But since none of those was a level five, a randomized controlled experiment, they wouldn't go to the level to say that it was a proven practice. They called it a promising practice. So these are examples of what the Campbell Collaboration does to review and assess the methodological rigor of police research. [Center for Evidence-Based Crime Policy] The Center for Evidence-Based Crime Policy at George Mason

· 10:04

DR. JAMES CHIP COLDREN, [continued]: University conducts reviews of policing studies along three dimensions. What's the extent of the proactivity of the policing intervention being applied versus being reactive, what type of focus does a study have. Does it have a focus for example, on certain individuals or certain communities? Or is there a broader focus across entire neighborhoods or police jurisdictions. And what exactly is the scope or target of the intervention? Are they trying to change the behavior of individuals?

· 10:34

DR. JAMES CHIP COLDREN, [continued]: Are they focusing on groups or neighborhoods? Or again, police beats or something like that. So they look at these three dimensions of police intervention and police approaches. And they look at the research findings. And they're coming up with some findings regarding what types of combinations of these dimensions seem to produce the most effect in terms of positive benefits. So what they have found generally across a number of studies that they've looked at, is that when police interventions are highly focused and highly proactive,

· 11:12

DR. JAMES CHIP COLDREN, [continued]: either on groups, microplaces or neighborhoods-- not necessarily on individuals-- they tend to have more success. They also found mixed results when there is a reactive focus on individuals. Sometimes, the reactive focus on individuals produces a positive effect and a reduction in crime. Sometimes, it doesn't produce any effect. And sometimes actually, in a few cases they identified, it does more harm than good. The individuals that are focused on in this reactive manner

· 11:43

DR. JAMES CHIP COLDREN, [continued]: end up committing more criminal behavior, rather than less compared to a comparison group of individuals. So this graphic that I'm going to show you will give you a depiction of how these studies are reviewed. On the upper left hand side of this matrix, you'll see a grouping of figures. Each of these figures represents one study that was reviewed. If it's a dark black circle, that means the study found a positive effect. If it's a white circle, it means there was no effect. And if it's a red triangle, it means that there was a negative effect,

· 12:18

DR. JAMES CHIP COLDREN, [continued]: that the intervention applied produced more crime rather than less crime. And since this cluster of studies is in the upper left hand corner of this matrix, it means that the interventions were focused, they were not general, and that they were focused on individuals, not on groups or neighborhoods or jurisdictions, or even at the state level. Now, if you look to the right of this matrix a little bit, you'll see several columns of clusters of these figures towards the back of the matrix and towards the top. So since they're towards the top,

· 12:49

DR. JAMES CHIP COLDREN, [continued]: that means these are studies that were highly focused on groups. And since they're towards the back of the matrix, it means that they were more proactive than reactive. And what you see here is a preponderance of black circles, which indicates that most of the studies indicated that these policing interventions had a positive effect. They produced reductions in crime and reductions in fear of crime, or positive social benefits. So what the Center for Evidence-Based Crime Policy it's telling us is that if you have a police intervention that's highly proactive, that's more focused than general, and that works with groups or microplaces or neighborhoods,

· 13:33

DR. JAMES CHIP COLDREN, [continued]: you're much more likely to have a positive effect than otherwise. This is very important and very good news. And it's leading the policing field in to certain types of interventions that we might not have otherwise focused on. And again, in order to be included in this matrix, you have to have a study that's got a methodological rigor on the scientific scale of a three or above, or your study will not be included. So the existence of the matrix and the work

· 14:02

DR. JAMES CHIP COLDREN, [continued]: of the Center for Evidence-Based Crime Policy is promoting rigor and policing science. [Crime Solutions] This is a US Department of Justice sponsored program through the National Institute of Justice that again, reviews programs and practices, looks at methodological rigor, looks at findings. And based on certain rigid standards, upon review, a program or a practice might be placed

· 14:37

DR. JAMES CHIP COLDREN, [continued]: on the crimesolutions.gov website as a proven or a promising practice. So there are two types of review that the crimesolutions.gov initiative engages in. There's what's called a program review, which is a review of an individual program. For example, the Shawnee, Kansas police department implemented a program called data-driven approaches to crime and traffic safety. That's one instance of one program that was evaluated by crimesolutions.gov.

· 15:04

DR. JAMES CHIP COLDREN, [continued]: There's also what's called a practice review, which is a review of a group of similar programs. So if there were a number of jurisdictions that had tried the DDACTS approach, the data-driven approach, then crimesolutions.gov would have gone through a practice review, rather than a program review. They're actually very similar, but one looks at individual programs, the other looks at groups of programs as a practice review. Like the other initiatives I mentioned, in order to be included in a review by crimesolutions.gov,

· 15:36

DR. JAMES CHIP COLDREN, [continued]: you have to have a rigorously designed study that's at a level III or above on a scientific scale. There have to be crime-related outcomes measured as part of the study. The report has to be published in a peer-reviewed journal or in a government report of some type. And it must have been done after the year 1980. So as of July 2016, the crimesolutions.gov program

· 16:03

DR. JAMES CHIP COLDREN, [continued]: had reviewed 72 law enforcement programs. 17 of those 72 were found to be effective, or 24%. If a program is found to be effective in crimesolutions.gov, it means that there is very strong evidence to indicate that they achieved their intended outcomes when implemented with fidelity. And by strong evidence, they mean a randomized controlled design, or quasi-experimental design which is a level IV or a level V on that scientific scale. 46 of the programs were found to be promising, which is 64%. When they say promising, they mean

· 16:36

DR. JAMES CHIP COLDREN, [continued]: there is some evidence to indicate that the programs achieved their intended outcomes. But it's not supported by the most rigorous research. So what crimesolutions.gov is saying is that this looks promising, but we're not ready to say it's a proven practice. Interestingly, 9 of the 72 programs were found to have no effect, or 12%. This means that there is strong evidence indicating that they had no effects or had harmful effects when implemented with fidelity. [Smart Policing Initiative] Smart policing was established by the Bureau of Justice Assistance in 2009. [Bureau of Justice Assistance] Under this initiative, the federal government gives grants to police agencies to develop and test

· 17:21

DR. JAMES CHIP COLDREN, [continued]: innovative and cost-efficient approaches to crime control and prevention. Two key factors influence the development of smart policing. The first was the economic downturn in 2008. During this time, the cost of policing were rising dramatically and local government budgets were shrinking. So the police had to find a way to do more with less, and the federal government was interested in finding ways to help them through the testing of new innovative strategies through research. The other influence on smart policing

· 17:54

DR. JAMES CHIP COLDREN, [continued]: goes back to what I mentioned about the work of David Weisburd and his colleagues before. The Bureau of Justice Assistance knew that policing research was not being conducted with strong methodological rigor. So they included in the smart policing initiative a mandate that the police agencies receiving federal funding for their innovative practices, test them through research partnerships and strong methodological designs.

· 18:20

DR. JAMES CHIP COLDREN, [continued]: Since 2009, the federal government has provided funding to over 45 police agencies. [Examples] So the Los Angeles Police Department instituted a program called LASAR, which stands for the Los Angeles Strategic Extraction and Restoration Program. Los Angeles used pretty advanced analytics and the decentralization of field intelligence analysts, meaning they took intelligence analysts out of headquarters and put them in the local district offices

· 18:53

DR. JAMES CHIP COLDREN, [continued]: to work directly with detectives and police officers working on violent crime problems. And they used a problem-oriented policing approach. And using a quasi-experimental research design, they found statistically significant reductions in homicide and violent crime. And these have been sustained over some time. Since the introduction of smart policing in Los Angeles, they have expanded it to five additional police districts from the original one that it was tested out in. And recently, Charlie Beck, the chief of police in Los Angeles,

· 19:25

DR. JAMES CHIP COLDREN, [continued]: publicly announced that he's going department-wide with smart policing and the LASAR program. Shawnee, Kansas, as I mentioned earlier, implemented a test of the data-driven approaches to crime and traffic safety. This is a unique approach to crime analysis, where you identify places where crime and traffic safety problems co-exist. And through a focused presence of police, traffic police, and other police, pretty much the police presence

· 19:58

DR. JAMES CHIP COLDREN, [continued]: and the focus on these particular problems drives crime and traffic safety problems down. So they implemented DDACTS in a test jurisdiction. They compared it to a comparison jurisdiction. So this is again, a quasi-experimental design as a result of the implementation of DDACTS in Shawnee, Kansas. They saw over 80% reductions in targeted street crimes, and a 24% reduction in crashes.

· 20:26

DR. JAMES CHIP COLDREN, [continued]: In Kansas City, Missouri, they institute a focused deterrence approach called the no violence alliance, or NOVA. This was the identification of over 120 chronic gang members who were involved in homicides and violent crime over several years, and addressing them with that focused deterrence approach. The strong surveillance, the strong message from law enforcement that their behavior will no longer be tolerated. And if they continue their criminal behavior, they'll be subject to the full extent of the law. And if they want to, they can take

· 20:57

DR. JAMES CHIP COLDREN, [continued]: advantage of pro-social and social treatment opportunities to change their lifestyle. Kansas City NOVA was evaluated with a quasi-experimental design, and an interesting application of social network analysis to the study of gangs. Social network analysis is actually a mathematical procedure through which you can identify groups and individuals in groups, measure the strength of their associations, the frequency of their contacts, and identify who the central people are in these groups and these networks, so you can focus your efforts on those individuals who

· 21:32

DR. JAMES CHIP COLDREN, [continued]: are the most influential and those groups and those networks. Kansas City NOVA also experienced statistically significant reductions in violent crime as a result of their focused deterrence approach. Boston, Massachusetts undertook a very interesting detailed analysis of violent crime in micro-hot spots in Boston. A micro-hot spot is basically a street segment or a street intersection. It's a very small geographical space. And through their analysis, they identified a small percentage of micro-hot spots in Boston that we're accounting for 40% to 50% of the violent crime in that city,

· 22:13

DR. JAMES CHIP COLDREN, [continued]: and had consistently accounted for that level of crime in those small spots for well over 25 or 30 years. So the violence problem in Boston was entrenched in a number of very small places that were accounting for a great percentage of violent crime in the city. They identified those spots, they developed "Safe Street Teams" that were enforcement-oriented but with community and problem-oriented policing approaches. They worked those areas. And again, through a quasi-experimental research design where they matched hot spots that did not

· 22:46

DR. JAMES CHIP COLDREN, [continued]: get this Safe Street Team's intervention to the ones that did. And they found strong statistical reductions in violent street crimes. Interestingly in the case of Boston, they didn't see reductions in homicide. So this is a good example of what happens in smart policing. The police executives and the police analysts and the researchers follow the data. They were pleased with the reduction in violent street crimes, but they were not pleased with the lack of reduction in homicides. So since then, the Boston Police Department has undertaken a thorough re-examination and reorganization of its homicide unit,

· 23:25

DR. JAMES CHIP COLDREN, [continued]: and is currently working on increasing the clearances of homicide cases. The results are not out from that research yet, but all indications are pointing to again, very positive results from this. [Challenges to Rigorous Research in Policing] So I'm going to step back a bit now and just talk a little bit about challenges to this need for rigorous research and policing. We've seen several examples and several initiatives that have worked very hard to increase

· 23:57

DR. JAMES CHIP COLDREN, [continued]: the level of methodological rigor in policing science and policing research. The truth is, this is a very challenging endeavor for several reasons. It is difficult to integrate rigorous research-- research in which the scientist is primarily in control of the study-- into real life policing in the field, in police organizations and communities. The conduct of rigorous research like randomized experiments or quasi-experimental designs actually

· 24:30

DR. JAMES CHIP COLDREN, [continued]: requires that the police somehow adjust their operations to meet the needs of the researcher. That's difficult to do in real life. As you can imagine, there are ethical concerns about subjecting individuals and communities to randomized conditions. This happens in several ways. Some people, when a police department starts to focus on certain individuals and certain communities, even if the data show that those communities and those individuals are accounting for the greatest percentage of crime, it feels like unfair targeting.

· 25:01

DR. JAMES CHIP COLDREN, [continued]: It feels like a biased approach to policing, and people get concerned about that. On the other hand, people also get concerned when they see some communities receiving the benefits of these new interventions, and other communities not receiving them. And so they get concerned about the fact that some people are getting new police resources and some are not. So these concerns happen on both sides of the coin. Some are concerned about the way policing is being done, some are concerned that the police are not spreading their resources evenly across the community. These concerns come up even though people generally

· 25:37

DR. JAMES CHIP COLDREN, [continued]: understand that randomized experiments are the best way to find out what works. Even with that understanding, there are these ethical concerns. In my experience, I think it's also true that research capacities are not evenly distributed across communities and across police departments. So for the Bostons and the Chicagos, and the New Yorks, and the Miamis and the Los Angeles' of our country, it's not hard to find a local university with a strong cadre of very experienced sociologists and criminologists who are

· 26:10

DR. JAMES CHIP COLDREN, [continued]: interested and capable of doing the type of research that needs to be done. And that have sufficient analytical information systems and data capacities within their own departments. When you get down to mid-size and small-size agencies which account for more than 60% or 70% of the police departments in this country, their data analytics and their information systems and their access to researchers who are experienced in doing this kind of work is not always what it needs to be.

· 26:39

DR. JAMES CHIP COLDREN, [continued]: Even though we want to increase these capacities, we want more rigorous research. There are challenges to doing this across the board in small and medium-sized communities. And it's also true that randomized and rigorously controlled studies are not always what is needed most. So for example, there are general questions such as what's the nature and extent of human trafficking in the United States? Or what are the trends in civilian ambushes of police in the United States?

· 27:10

DR. JAMES CHIP COLDREN, [continued]: These things can be studied without a rigid, randomized, controlled evaluation design. You need good solid research methods to do the survey work and the data collection to learn about the nature and extent of these problems. But these questions don't necessarily call for a randomized experiment or a quasi-experimental design. Similarly, there are frequent needs for what we would think of as opinion studies or perception studies. We want to know what the extent of fear of crime is in a community.

· 27:42

DR. JAMES CHIP COLDREN, [continued]: Or currently, there are a lot of questions about how much residents actually trust the police. One question that's coming up recently has to do with what police actually think about the risk and stress on their job, and how to get them to focus on wellness. So these again, are examples of studies that could be done with good strong scientific research methods, but don't require these rigorous experimental designs. And finally, there's a class of study called an after action analysis that's very important [after action analysis] when

· 28:14

DR. JAMES CHIP COLDREN, [continued]: critical events happen in communities and police agencies. For example, when there's a police shooting of a civilian, or when there's an ambush and a killing of police officers, or when there's a disaster that police agencies respond to, an after action analysis will involve the use of research to collect data about what happened in these particular incidents, data about community behavior, data about videos that were taken, analysis of videos and police reports. And these after action analyses are very important for police agencies to learn what they did right,

· 28:53

DR. JAMES CHIP COLDREN, [continued]: what they did wrong, and how they could respond better in future situations. So while we've talked at length about the need for more rigorous designs and policing research, there are actually instances where these rigorous experimental designs are not necessarily required. [Conclusion] As I indicated, there was a perceived need to improve the rigor of research into policing effectiveness.

· 29:22

DR. JAMES CHIP COLDREN, [continued]: And there have been several initiatives undertaken that will help improve policing science. We talked about the crimesolutions.gov initiative from the National Institute of Justice, the Center for Evidence-Based Crime Policy at George Mason University, and the Smart Policing Initiative of the Bureau of Justice Assistance. Each of these is a vigorous attempt to improve the level of methodological rigor in police science and police research. We also learned that there will always be challenges to conducting rigorous scientific research in the public sector.

· 29:54

DR. JAMES CHIP COLDREN, [continued]: Though on the basis of what we've talked about today, things seem to be moving in the right direction. I'd like you to think of several questions after reflecting on the issues we've talked about today. First, are you aware of any research done on your local police department? Second, are the rigors of that research up to the standards we've discussed? Third, why is the methodological research important for police officers, as well as for public safety?

What is Smart Policing? How did it originate, and what has been learned from the test sites and where has it been tested? Do other countries have Smart Policing strategies?

350-450 words excluding reference, APA format and a minimum of 3 references.