Data Collection Tools and Procedures

ElementsofResearchDesign103-110.pdf

Home >Law homework help >Criminal homework help >Data Collection Tools and Procedures

Elements of Research Design 103

the program (treatment) are unknown, the control group offenders are not being deprived. Once the effectiveness of the program is deter- mined, control group offenders can be admitted.19

Second, the classical experiment has the potential to encroach upon the decision-making authority of criminal justice agents. The clear solution here, as the Provo Experiment indicated, is to use random assignment after a decision is made. Surely, experimenters cannot ex- pect to randomly sentence people to prison or probation. Modifica- tions can be adopted which do not restrict the power of the classical experimental design while protecting the rights of individuals.

Third, there is the major issue of the feasibility of carrying out an experiment in the field.

The random assignment process must be structured to meet both the demands of the classical experiment and the operations of the criminal justice system — in these two examples, sentencing by judges and arrest and other sanctions by police. It requires very close coop- eration between the operational agencies and the researchers. Imple- menting random assignment and maintaining it throughout the pro- cess can be problematic. Also, to repeat, the subjects selected for study must be eligible for treatment and the experimental and control groups must be kept separate so that the integrity of the treatment is maintained.20 After all, exposure to the treatment should be the only difference between the two groups.

The Quasi-Experimental Design

The classical experiment is not the only form of research design. However, it serves as the point of departure for other designs which attempt to approximate its key features. The quasi-experimental de- sign is the mirror image of the classical experiment with one key dif- ference: the absence of random assignment. As previously mentioned, it is not always possible to implement the classical experiment. Yet, it is vital that a group is constructed to compare findings generated by the experimental group. Thus, a comparison group is selected using a method other than random assignment to insure that it is compara- ble to the experimental group.

Our example here is the evaluation of a prison-based treatment pro- gram for drug/alcohol abusing inmates.21 We need to find a group of inmates in the same prison who have the same problem, but who were

Co py ri gh t © 2 00 8. C ha

rl es C T ho ma s. A ll r ig ht s re se rv ed . Ma y no t be r ep ro du ce d in a ny f or m wi th ou t pe rm is si on f ro m th e pu bl is he r, e xc ep t fa ir u se s pe rm it te d un de r U. S. o r ap pl ic ab le

co py ri gh t la w.

EBSCO Publishing : eBook Collection (EBSCOhost) - printed on 1/15/2020 10:52 PM via PURDUE UNIVERSITY GLOBAL AN: 446194 ; Vito, Genn, Tewksbury, Richard, Kunselman, Julie.; Introduction to Criminal Justice Research Methods : An Applied Approach Account: ns019078

104 Introduction to Criminal Justice Research Methods

never involved in the program in any way. After all, if we select- ed inmates who were excluded from the program, we would commit the error of selection bias. Similarly, if we chose inmates who dropped out of the program, the data would be contaminated because the com- parison group would have been exposed to the treatment. What is left? Adams’ suggests screening inmates who were considered for, but for reasons of their own, decided not to take part in the program — a self-drop group.22 Before such a group can be considered however, two important considerations must be checked out. First, the re- searcher must be certain that such inmates were not thrown out of the program by project administrators (Beware of selection bias!). Second, you must be certain that these inmates were never enrolled in or ex- posed to the treatment (contamination of data problem). It would also be possible to check for eligible inmates who were simply unaware of the program. It is also possible to use a variation of the matching tech- nique (again the missing element is random assignment). Here, the experimenter would construct a comparison group which was identi- cal to the experimental group on a number of known variables (i.e., age, race, prior record, present offense, education, marital status, etc.).

In any event, since randomization was not utilized, it is necessary to record relevant personal and socio-demographic information on such inmates and compares them directly to the experimental group. If dif- ferences do exist, it would be necessary to control for them statistical- ly. Remember that the crucial issue here is that the experimental and comparison groups must be similar. The problem is that, even if you determine that the two groups are comparable, they may still differ on some important attribute which was beyond your means to measure. This is not a problem when random assignment is used, hence the power of the classical experimental design.

However, since it is not always possible to use random assignment, the quasi-experiment gives you another possibility to conduct accurate research. Quite simply, it may not be possible for you to do anything else and it is especially valuable when performing evaluation research (see Chapter Eleven).

Other Types of Research Designs

Pre-Experimental Designs. Pre-experimental designs take a vari- ety of forms which emphasize description but typically fail to make a

Co py ri gh t © 2 00 8. C ha

co py ri gh t la w.

Elements of Research Design 105

comparison between the experimental group and another group of subjects. Typically, they are undertaken out of necessity because they, like the quasi-experimental design, offer a feasible alternative to re- search when the classical experiment is impossible to conduct. The chief problem is that the researcher is then unable to protect the integrity of the research results and clearly state that they reflect the effect of the treatment and not some other force. They are not as reli- able as the classical experiment.

The first type of pre-experimental design is the one-shot case study or the one group post test design. Here, measurements are ob- tained for one group after the treatment has been administered. For example, one could determine if a group of police officers became more sensitive to diversity issues after they went through a diversity training program. Their score on an examination would determine if they had learned the principles of diversity covered in the training program.

Probably the most common type of pre-experimental design is the before-after study (also known as the “one group pretest-posttest de- sign”). This design is simply the first half of the experimental design. The performance of the experimental group is recorded before and after the treatment is administered. It is the simplest design but, due to the absence of comparison, it fails to document the effectiveness of the treatment. It is commonly used when it is difficult to construct a com- parison or control group. For example, let’s say that we wish to evalu- ate the effectiveness of a neighborhood watch program in a particular neighborhood. Under a before and after study, we would measure the burglary rate (number of reported burglaries) before and then after the implementation of the neighborhood watch. Even if the burglary rate declined after the establishment of the neighborhood watch program, we would not be able to make a comparison to another neighborhood that did not have a neighborhood watch. Our only comparison is to our selected neighborhood before and then after the neighborhood watch began.

One way to attempt to provide this comparison is to use the static group comparison design. With this design, the number of reported burglaries in a neighborhood that has had a neighborhood watch pro- gram would be compared to the number of reported burglaries in another neighborhood that did not have a neighborhood watch pro- gram. Here, the weakness is that, before the research is conducted,

Co py ri gh t © 2 00 8. C ha

co py ri gh t la w.

106 Introduction to Criminal Justice Research Methods

there is no examination to determine whether the two neighborhoods are comparable. The research conclusion is based upon comparing the impact upon each group to determine the effect of the treatment on one of them.

Cross-Sectional Design. A cross sectional design takes measure- ments of subjects at a single time in their lives. The case study is a form of cross sectional design used in criminal justice research. The case can be an individual (a professional criminal), an event (a police strike), or a place (Alcatraz). This type of design is capable of generat- ing great quantities of descriptive information which can be used by policy makers. It is especially valuable in time of rapid change because it allows you to respond immediately to an historical event or a nat- ural experiment — i.e., the effect of a judicial order on the operations of a prison or a change in sentencing policy.23 Follow-up recidivism research on the former Kentucky death row inmates is an example of a natural experiment because their death sentences were commuted by a U.S. Supreme Court decision and they were later released by the parole board.24

Longitudinal Design. A longitudinal design is similar to the cross sectional design with the key exception that measurements are taken at more than one point in time. One form of longitudinal design is the time series design. The time series design is one in which the treatment is introduced during a series of measurements on the depen- dent variable. All of the measurements are obtained from the same group. For example, a researcher could examine the deterrent effect of an execution by measuring the homicide rate in a state in the months prior to and following an execution. A sharp change in the trend of measurements on the dependent variable (the homicide rate) immedi- ately following the treatment (the execution) is assumed to be attribut- able to the treatment. Again, the comparison in results is within the experimental group only.

Another form of longitudinal design is a cohort study. It examines the behavior of a particular group over time. Cohorts are groups con- structed by the researcher that share some common experience (i.e., involvement in the same program, graduating from the police acade- my in the same year). Their performance over time is then recorded. Wolfgang, Figlio, and Sellin (1972) tracked the delinquency records of a birth cohort of boys born in 1945 who lived in Philadelphia from their tenth to eighteenth birthdays. They discovered that 35 percent of

Co py ri gh t © 2 00 8. C ha

co py ri gh t la w.

Elements of Research Design 107

the boys had some type of contact with the police and that about 50 percent of the juveniles who commit an offense are likely to commit another.25

Regardless of the type of design, there are several threats to validi- ty that can affect the results of the study.

THREATS TO INTERNAL VALIDITY

Internal validity refers to ways in which the process of experi- mentation may affect the research results. In other words, the re- searcher is then uncertain if the outcomes generated by the research are a result of the treatment or the way in which the experiment was conducted. In effect, the experiment becomes a treatment in itself.

Cook and Campbell identified the following sources of internal validity:26

1. History: events, in addition to the treatment, may occur be- tween the pre and post tests which are beyond the control of the experimenter. For example, Vito, Longmire, and Kenney (1984) reported that, during their evaluation of a police burglary sup- pression program, the state of California passed legislation re- quiring a mandatory prison sentence for burglary.26 As a result, the researchers were uncertain if the number of reported bur- glaries recorded during the project were affected by the new methods of police operations or due to the new law. Often, the only thing which researchers can do when an historical event occurs in the middle of their experiment is report that it oc- curred and let the findings be interpreted accordingly.

2. Maturation refers to the processes operating within research subjects as a function of the passage of time, including growing older, growing hungrier or becoming more tired. Boredom could also be a problem which could affect the behavior of the research subjects. For example, Vito and Wilson (1988) con- ducted a long term follow up of former death row inmates in Kentucky whose sentence had been commuted to life in prison as a result of the Furman v. Georgia (1972) decision. Was their on parole behavior due to the commutation of their death sentence or due to aging?28

Co py ri gh t © 2 00 8. C ha

co py ri gh t la w.

108 Introduction to Criminal Justice Research Methods

3. Testing concerns the effect of taking a test upon the scores of a second testing. If the same instrument is used for the pre and post test, the subjects in the control group may be able to tease out what the instrument is designed to measure (i.e., fear of crime) and try to answer “the right way” rather than express their own true feelings.

4. Instrumentation. If, upon repeated use, an instrument yields the same results, it is considered to be reliable. But what would happen if your instrument was altered somehow between the pre and post tests? If there is some alteration in your instrument, the research results would be affected.

5. Statistical Regression is especially problematic when research subjects have been selected on the basis of their extreme scores or attributes. “Regression toward the Mean” is a statistical phe- nomenon which operates in nature. Any extreme attribute tends to be balanced out over time. The problem, therefore, is that extreme subjects tend to improve over time regardless of the treatment. Their behavior or performance goes to the average level for the group of subjects under study. For example, in his book, The Future of Imprisonment, Morris (1974) proposed a new prison model which he would like to test using the “toughest group of inmates.” Morris encouraged the use of a classical experimental design to assign such inmates to his model institu- tion. Clearly, statistical regression could be a threat to his pro- posed experiment. If these inmates are so “bad” to begin with, their behavior may simply regress toward the mean. If they did improve, it would be difficult to say that the benefits were due to the new prison design.29

6. Experimental Mortality has to do with the loss of subjects from your experimental and control groups. If large numbers of subjects “drop out” for whatever reason, the groups may change so much that they are no longer comparable. Thus, the major strength of randomization is violated. Researchers conducting recidivism studies have particular problems with mortality since parolees are often mobile and do not leave forwarding address- es and they often literally die while on supervision.30

7. Selection Biases: Remember, the groups must be comparable to begin with. If techniques other than random assignment are used, selection biases may affect the research results. Put simply,

co py ri gh t la w.

Elements of Research Design 109

you do not wish to compare apples to oranges. Researchers or program officials should not put all the “best risks” in the exper- imental group and then compare them to a group of poor risks.

8. Interactions of the Above Problems. To make matters worse, it is possible that your research can be affected by combinations of the problems just mentioned. The design proposed by Morris (1974) could not only be subject to problems due to statistical re- gression, but also due to maturation. And what would happen to the research results if a riot or escape occurred during the study (History)?

9. Causal Time Order. If somehow the time order between the treatment and the measure of the dependent variable (post-test) is fouled up, it is obvious that the causal relationship between variables is no longer being tested.

10. Diffusion or Imitation of Treatment. If the respondents in the control group can communicate with the members of the exper- imental group, they each may discover information intended for the other group. Put simply, the physical closeness of the two groups may render them equal by exposing them both to the treatment. The Provo Experiment was plagued by this problem since both the experimental and control (probation) groups were supervised by the same probation office.

11. Compensatory Equalization of Treatments. When the exper- imental treatment provides goods or services generally believed to be desirable, the experimenter (or administrators in charge of a project) may be sympathetic toward the control group and pro- vide them with some compensatory benefit, such as special attention. Of course, this special attention would thus become another form of treatment and the original design would suffer.

12. Compensatory Rivalry by Respondents Receiving Less De- sirable Treatments. When the assignment of persons to exper- imental or control groups is made public (as is frequently re- quired by ethical and legal considerations), competition may be generated. In particular, the control group (the natural under- dog) may be motivated to perform at the highest possible level.

13. Resentful Demoralization of Respondents Receiving Less Desirable Treatments. This potential response is very much related to rivalry. The control group may become demoralized

co py ri gh t la w.

110 Introduction to Criminal Justice Research Methods

about the conditions of the research and thus perform more poorly than the experimental group or get angry and revolt.

Other internal validity questions of special interest to criminal jus- tice researchers have been identified by Adams:31

14. Masking. Experimental treatments may have opposite effects upon different kinds of subjects. Vito (1982) has suggested that it is simply illogical to assume that all members of the experimen- tal group were amenable to or served equally by a correctional treatment program. Unless some measure of the effectiveness of the treatment among the experimental subjects is included in the study, masking could cloud the findings by failing to make such differentiations in the experimental group.32

15. Contamination of Data. If the subjects in the control group be- come exposed to the treatment, their post-program performance may be affected. This may have been one of the problems re- garding the controversial Kansas City Preventive Patrol Experi- ment.33 The treatment in this experiment was proactive police patrolling — a test of deterrence theory. The experimental neigh- borhoods received proactive patrolling, the reactive (control) areas underwent traditional patrolling (police responding to in- coming calls for service and patrolling only the perimeter of the beat or an adjacent proactive beat), while officers in the control sections were to patrol as they normally would. The problem was that the 15 neighborhoods in the study were adjacent to one another. Was the treatment clearly isolated or did the neighbor- hoods, in effect, all receive the same type of patrol?

16. “Erosion” of the Treatment Effect. The gradual or abrupt dis- appearance of performance superiority shown by the experi- mental group in the early months after treatment may decrease or simply wear off. This problem could be especially pro- nounced if the researcher is following the performance of the experimental subjects over a long period of time.

These problems are not insurmountable. They have been presented because the researcher must be aware of them in order to combat them. Some can be dealt with through the use of randomization and the classical experimental design (i.e., selection bias and statistical regression). Others can be handled by careful monitoring of the con- duct of the research. It is vital that the experimental and control

co py ri gh t la w.