Assignment #2
farra004
CH2VitoandHiggins.pdf
15
PLANNING A PROGRAM EVALUATION
2 Keywords problem-oriented policing (POP) SARA S.M.A.R.T. logic model
CHAPTER OUTLINE
Introduction 15
Problem-Oriented Policing 16
Planning an Evaluation Strategy 18
Goal Setting: S.M.A.R.T. 18
Goal–Objective Relationship 19
Develop Evaluation Measures 19
Data Collection 20
Determine Analysis Methods 21
Logic Model 22
Politics of Evaluation Research 24
Ethical Issues in Evaluation Research 25
Ethics of Conducting Research 25
Ethics and Social Relationships in Evaluation Research 28
Summary 29
Discussion Questions 29
References 30
Introduction Criminal justice programs are designed to address a particular need
and solve a defi ned problem. Crime problems are often intractable and diffi cult to address. Strategic thinking, planning, and operations are
Planning proceeds step by step and each step must be evaluated before the next step can be taken.
—Edward Suchman (1987)
Vito, G. F., & Higgins, G. E. (2014). Practical program evaluation for criminal justice. Taylor & Francis Group. Created from liberty on 2023-01-26 20:17:10.
C op
yr ig
ht ©
2 01
4. T
ay lo
r &
F ra
nc is
G ro
up . A
ll rig
ht s
re se
rv ed
.
16 Chapter 2 PLANNING A PROGRAM EVALUATION
required to address the sources of crime problems, both individual and systemic. Panaceas and “silver-bullet” solutions to crime problems are often sought but never found. Collection of data and the compilation of research evidence typically are a part of program evaluation.
Th is is where program planning begins and program evaluation fol- lows. Th is process has been adequately summarized under the prob- lem-oriented policing (POP) acronym (SARA) ( Goldstein, 1990 ): ● Scanning: Identify recurring problems and how they aff ect com-
munity safety. ● Analysis: Determine the causes of the problem. ● Response: Seeking out, selecting, and implementing activities to
solve the problem. ● Assessment: Determine if the response was eff ective or identify
new strategies. Th e aim of this process is to collect data that are related to the prob-
lem, determine the validity of these data, trace causal relationships that could lead to problem identifi cation and program development to address the problem, and then determine if the program was eff ective in solving the problem.
Scanning involves looking for and identifying problems. Th e ini- tial analysis is made to determine if a problem exists and whether a detailed analysis is required to uncover it. Problems are also prioritized and personnel allocated.
Analysis is concerned with learning about the causes, scope, and eff ects of the problem. Who are the actors involved in the problem (both off enders and victims)? What are the specifi c incidents of the problem and what is the sequence of events leading to it? What responses have been made by the community and government agencies?
Response involves acting to alleviate the problem. Planned solu- tions can be organized regarding total elimination of the problem, material reduction of the problem, reduction of the harm caused by the problem, using the best possible solution to the problem, and even removing the problem from police consideration.
Assessment is where program evaluation comes in. Th at is, did the response work?
Problem-Oriented Policing POP requires that the police develop a systematic process for exam-
ining and addressing the problems that the public expects them to handle. It requires identifying these problems in more precise terms, researching each problem, documenting the nature of the current police response, assessing its adequacy and the adequacy of existing authority and resources, engaging in a broad exploration of alternatives
Vito, G. F., & Higgins, G. E. (2014). Practical program evaluation for criminal justice. Taylor & Francis Group. Created from liberty on 2023-01-26 20:17:10.
C op
yr ig
ht ©
2 01
4. T
ay lo
r &
F ra
nc is
G ro
up . A
ll rig
ht s
re se
rv ed
.
Chapter 2 PLANNING A PROGRAM EVALUATION 17
to present responses, weighing the merits of these alternatives, and choosing from among them ( Goldstein, 1979 ).
As defi ned by the San Diego, CA, police department ( Capowich & Roehl, 1994, pp. 127–128 ):
POP emphasizes identifying and analyzing problems (criminal, civil, or public nuisance) and implementing solutions to resolve the underlying causes of the problem. It emphasizes proactive intervention rather than reactive responses to calls for service, resolution of root causes rather than symptoms, and use of multiparty, community-based problem solving rather than a unilateral police response. POP focuses on a problem in a long-term, comprehensive manner, rather than handling the problem as a series of separate incidents to be resolved via arrest or other police action.
In San Diego, the police focused on street robberies for one year. Research determined that calls for service did not decrease over the one-year period. Yet, the indicators revealed that the POP approach improved the circumstances for those who used the stations and reduced the police workload at those stations. However, the role played by citizen groups in this project was very limited.
Elsewhere, research on POP reports positive results. In both Newport News, VA, and Baltimore County, MD, offi cers concentrated on under- lying causes of crime. Th ey collected information and enlisted the sup- port of public and private agencies. As a result, both crime and fear of crime were substantially reduced in both areas ( Eck & Spelman, 1987 ). Th e Center for Problem-Oriented Policing provides copies of evaluations ( www.popcenter.org/casestudies/ ) of POP operations that have eff ec- tively addressed such disparate problems as drunken driving, repeat sex off enders, construction site thefts, thefts from cars in parking facilities, burglary of single-family houses, drug dealing in apartment complexes, loud car stereos, street prostitution, and residential speeding.
Basically, there are two ways to defi ne a policing problem. In clear terms, a policing problem exists when something is going wrong for someone somewhere and somebody wants the police to do something about it. Problems can be perceived as a group of events that are simi- lar in one or more ways that are harmful to members of the public and that citizens expect the police to handle.
Th e problem analysis process must ferret out relationships among the variables collected in the data to determine the extent and nature of the problem. It is also important to ascertain if the problem can be adequately measured and thus serve as the basis for an outcome evalu- ation. First, there are concerns regarding the data: Can adequate data be obtained? Second, there are resource questions to be addressed: Does the police have the tools necessary to collect data and analyze it? What amount of time and money are necessary to conduct the study? Hopefully, the problem addressed is signifi cant enough to advance
Vito, G. F., & Higgins, G. E. (2014). Practical program evaluation for criminal justice. Taylor & Francis Group. Created from liberty on 2023-01-26 20:17:10.
C op
yr ig
ht ©
2 01
4. T
ay lo
r &
F ra
nc is
G ro
up . A
ll rig
ht s
re se
rv ed
.
18 Chapter 2 PLANNING A PROGRAM EVALUATION
knowledge and practice in criminal justice and the analysis will con- tribute to future attempts to address the problem in question. In this manner, evaluation research not only examines the eff ectiveness of a particular program or policy, but also adds to crime prevention knowl- edge by revealing those strategies that have proven to be able to solve a crime problem.
Planning an Evaluation Strategy Th e program evaluator must fi rst develop a plan to conduct the
research process. Basically, evaluation planning involves the comple- tion of fi ve basic steps: 1. State the goals of the program in clear and measurable (quantifi able)
terms. 2. Determine the relationship between goals and objectives. 3. Develop evaluation measures. 4. Determine the data to be collected on these measures. 5. Determine analysis methods.
Th ese steps help clarify the aims and abilities of the program in question to address the crime problem.
Goal Setting: S.M.A.R.T. Program goals are not often stated in clear terms, and they often
express the wishes of a particular group rather than a defi nite target. Th ey are not stated in a manner that will make it possible to deter- mine whether the program is eff ective. Here, the evaluator attempts to state the goals of the program in measurable terms that indicate their achievement. Th ey are often stated as a percentage to be attained over a specifi ed time period. Th ey should be attainable, reasonable, and refl ect what can be achieved if the program is eff ective. For example, the goal of a burglary prevention program could be stated in measurable terms as: To reduce the number of reported burglaries in the target area by 25% over a one-year period. Note that this statement indicates what the unit of measurement is (the number of reported burglaries) as well as the percentage target (25%) and the time period (one year after the pro- gram begins). However, what is the basis for such an expectation? Has previous research on burglary prevention programs revealed that a 25% reduction is possible? If the target specifi ed by the goal is unrealistic and unattainable, its use as a performance benchmark is worthless.
One method to avoid these problems in goal setting is to follow the acronym S.M.A.R.T, as shown in Figure 2.1 . (see Locke & Latham, 2013 ). Well-defi ned goals should be: ● Specifi c: Goals must be clear and specifi c. Th ey should represent
what the program is trying to accomplish and accurately measure successful operations.
Vito, G. F., & Higgins, G. E. (2014). Practical program evaluation for criminal justice. Taylor & Francis Group. Created from liberty on 2023-01-26 20:17:10.
C op
yr ig
ht ©
2 01
4. T
ay lo
r &
F ra
nc is
G ro
up . A
ll rig
ht s
re se
rv ed
.
Chapter 2 PLANNING A PROGRAM EVALUATION 19
● Measureable: If the goal is not measurable, there is no way to gauge progress.
● Attainable: Again, goals must be realistic. Some eff ort and stretch- ing is always in order, but goals should neither be too high or too low so they become meaningless.
● Relevant: Th e goals should represent and mirror the vision and mission of the program.
● Time-bound: Goals must have starting and end points and the duration of the measurement should be stated clearly. Formulating goals in this fashion will ensure good performance
by letting stakeholders know what is expected of them. Of course, they will also guide the course of the impact evaluation.
Goal–Objective Relationship In terms of our example, the burglary prevention program is
designed to reduce the incidence of this type of crime in the geographic area served by the program. Th is analysis can be extended to com- pare the incidence of burglary citywide in the areas not served by the program.
Develop Evaluation Measures Th e next step is to identify the evaluation measures for the pro-
gram under consideration – in this instance, a burglary prevention program. Here, the measures can take three basic forms: ● Eff ectiveness: Th ese measures determine the degree of success
of the program in dealing with the problem at hand. Here, we have stated that the quantifi able goal of the burglary prevention program is to reduce the incidence of reported burglaries in the targeted area by 25% over a one-year period.
Sp
ec ific
T im
e -bound
Relevant Atta
in a
bl e
Measurable
Smart goals
Figure 2.1 S.M.A.R.T. goals.
Vito, G. F., & Higgins, G. E. (2014). Practical program evaluation for criminal justice. Taylor & Francis Group. Created from liberty on 2023-01-26 20:17:10.
C op
yr ig
ht ©
2 01
4. T
ay lo
r &
F ra
nc is
G ro
up . A
ll rig
ht s
re se
rv ed
.
20 Chapter 2 PLANNING A PROGRAM EVALUATION
● Effi ciency: Th ese measures should indicate how well the program has been implemented and whether it has been implemented according to the original plan for the program.
● Attitudinal: Th ese measures can indicate whether the program has been successful by assessing the attitudes of the program’s clients. In our example, one way to accomplish this would be to conduct a before-and-after survey of the fear of crime among residents of the targeted area to determine if they felt safer after the burglary pre- vention program was implemented. In addition, the evaluator should consider the impact of factors that
occurred during the implementation of the program that could also aff ect these measures. For example, if the state in which the burglary prevention program was implemented passed a “rob a house, go to prison” (i.e., mandatory incarceration) bill, it may have an impact on the incidence of the number of reported burglaries.
Valid performance measures should have the following character- istics. First, they should be credible—accurate and relevant represen- tations of both the quality and quantity of services provided by the program. Second, they should provide a fair indication of program performance and refl ect the factors and operations that program administrators can truly infl uence and control. Th ird, they should be clear—easy to utilize and comprehend and practical to administer and implement.
Data Collection Th e fourth step in this process is to determine the data necessary to
perform the evaluation, the constraints on their availability, how these data will be collected and managed, and a method to determine the validity of these measures. In terms of understanding the problem that the proposed program addresses, four strategies are recommended for the evaluator ( Bickman, Rog, & Hedrick, 1998, p. 7 ): ● Hold discussions with research clients or sponsors to obtain the
clearest possible picture of their concerns. ● Review the relevant literature on the subject. ● Gather current information from experts and major interested
parties on the issue. ● Conduct information-gathering visits and observations to obtain
a real-world sense of the context, and talk with persons actively involved in the issue. In our example, it will be necessary to determine the baseline level
of reported burglaries in the target area prior to program implementa- tion and then the same measure after the program has been in place for one year. It will be necessary for the evaluator to establish methods to obtain these measures from the police department in question and
Vito, G. F., & Higgins, G. E. (2014). Practical program evaluation for criminal justice. Taylor & Francis Group. Created from liberty on 2023-01-26 20:17:10.
C op
yr ig
ht ©
2 01
4. T
ay lo
r &
F ra
nc is
G ro
up . A
ll rig
ht s
re se
rv ed
.
Chapter 2 PLANNING A PROGRAM EVALUATION 21
to determine how they collect and report information on reported bur- glaries in their jurisdiction. Th e evaluator must determine if these mea- sures are valid indicators of the incidence of burglaries in the area or whether it will be necessary to conduct a victimization survey of homes in the target area. Such a survey could determine whether burglaries occurred that were not reported to the police as well as the previously mentioned assessment of the fear of crime in the area (both before and after the program was implemented).
Th e cost of obtaining these data must also be considered by the evaluator. Obtaining the data from the police is much less costly than the possible victimization survey of homes in the target area. Burglary data would be regularly collected by the police, but the survey must be conducted by the evaluator. In any event, the evaluator must design an information system to collect all forms of data required by the program evaluation in a format that is amenable to statistical analysis and com- puterized. Th e availability of computers (tablet, laptop, or desktop) and statistical analysis programs (Excel, SPSS, SAS) helps to simplify this task.
Data must be carefully collected with an eye toward quality control. Procedures to collect the data must be established and variables must be explicitly defi ned in measurable terms so that the meaning of the data is clear and will be followed carefully. Th e source of the data must be clearly identifi ed, and the process of data storage, maintenance, and processing must be specifi ed.
Determine Analysis Methods Th e fi nal step is the determination of quantitative and/or qualita-
tive methods of analysis that will be utilized in the evaluation. Th ese methods will be determined in part by the evaluation design, the type of evaluation measures used, and their validity and reliability. Th e eval- uator should consider how the evaluation measures will be calculated and if these measures can be combined.
In our burglary prevention program example, a before-and-after comparison of the number and rate of reported burglaries before and one year after program implementation in terms of the expressed target of a 25% reduction is a rather simple comparison. However, other points of comparison could be added by conducting the same method of analysis for the areas of the city not served by the pro- gram. In addition, the evaluator could conduct a regression analysis based on past levels of reported burglaries to determine whether the number of reported burglaries after the one-year period of imple- mentation compared to the expected number of reported burglar- ies calculated by the regression analysis. In addition, the proposed victimization survey could provide another level of comparison by asking the respondents in the area their experiences with burglary
Vito, G. F., & Higgins, G. E. (2014). Practical program evaluation for criminal justice. Taylor & Francis Group. Created from liberty on 2023-01-26 20:17:10.
C op
yr ig
ht ©
2 01
4. T
ay lo
r &
F ra
nc is
G ro
up . A
ll rig
ht s
re se
rv ed
.
22 Chapter 2 PLANNING A PROGRAM EVALUATION
victimization over the time period in question, as well as their fear of crime over this period.
One of the mechanisms used to display and describe the under- lying theory and purpose of a program prior to conducting program evaluation is the use of a logic model.
Logic Model Th e process of establishing goals, objectives, and performance
indicators requires that the theory that provides the basis for the program must be specifi ed. In turn, the theory informs program operations by: ● Driving the selection of treatments. ● Clarifying the description of the services provided to clients with
defi ned needs. ● Helping to determine what variables need to be measured. ● Driving how one interprets a simple comparison of the outcomes
of two programs to deeper analyses in terms of research on the topical area in general. Th ese are also the elements of the logic model of a program—the
framework that shows how the program might theoretically produce the desired outcomes and impacts ( Boruch, 1998, p. 172 ).
Th e logic model specifi es the conceptual framework of an evalua- tion by establishing the variables to be measured and the expected rela- tionships among them. Th e logic model provides an explanation of how the program is expected to work and how the program goals, processes, resources, and outcomes will provide the direction for the program. Th us, they provide a clear “roadmap” of what is planned and expected results—a review of the strength of the connection between activi- ties and outcomes ( Knowlton & Phillips, 2013, p. 5 ). As a diagram, the logic model demonstrates the cause-and-eff ect mechanism between how the program will meet certain needs to achieve desired outcomes ( Davidson, 2005, p. 38 ). It focuses the evaluation of the program’s eff ec- tiveness and can communicate changes in its implementation over the life of the program ( Bickman et al., 1998, p. 8 ).
Table 2.1 presents a logic model for adult drug court programs as proposed by the National Institute of Justice. First is the program inputs, the resources required to operate the program. Th en, the pro- gram activities are the components of the program. Next, program outputs indicate the work done and the results of activities. Finally, related to outcomes are the short-term and long-term outcomes that represent the objectives of the program. Note that both the short-term and long-term program outcomes are measures of program perfor- mance for which data must be collected and compiled on program operations and activities. Th us, they will serve as the basis for the assessment of program performance and must be collected in a timely
Vito, G. F., & Higgins, G. E. (2014). Practical program evaluation for criminal justice. Taylor & Francis Group. Created from liberty on 2023-01-26 20:17:10.
C op
yr ig
ht ©
2 01
4. T
ay lo
r &
F ra
nc is
G ro
up . A
ll rig
ht s
re se
rv ed
.
Chapter 2 PLANNING A PROGRAM EVALUATION 23
fashion—hopefully in a computer-based information system operated and maintained by the adult drug court program. Th e external factors listed in Table 2 refer to outside forces that can aff ect the drug court program. For example, both the community and the courthouse work group could oppose or support the creation of a drug court. In addition, the legal/penal code of the state must be consulted to determine if the operations of the drug court are permissable.
By specifying the relationships among these key factors, the logic model clarifi es why the program was adopted and how it is expected to work. Th us, it serves as a blueprint for the evaluation of the eff ec- tiveness of the program.
Table 2.1 Adult Drug Court Program Logic Model
Inputs Activities Outputs
Short-Term
Outcomes
Long-Term
Outcomes
External
Factors
Probation Risk/needs
assessments
Program intake
screen
Recidivism
in-program
Recidivism
post-program
Community
Community Judicial
interaction
Program
admission
Alcohol and
other drug use
in-program
Alcohol and
other drug use
post-program
Legal/penal
code
Public
resources
Alcohol and other
drug monitoring
(including
testing)
Court
appearances
Supervision
violation
Program/
graduation
termination
Courthouse
Courthouse Community
supervision
Treatment
admission
Program violation Probation
revocation/
successful
termination
Treatment Graduated
sanctions or
incentives
(including jail)
Alcohol and other
drug tests
Treatment
retention
Jail/prison
imposed
Jail Alcohol and other
drug treatment
services
Probation
contracts
Skills
development
Employment,
education,
housing, and
health
Grant funds Ancillary services Classes attended Service needs met
Technical
assistance
Services accessed Criminal thinking
Jail stays
Source: National Institute of Justice, http://www.nij.gov/topics/courts/drug-courts/full-logic-model.htm .
Vito, G. F., & Higgins, G. E. (2014). Practical program evaluation for criminal justice. Taylor & Francis Group. Created from liberty on 2023-01-26 20:17:10.
C op
yr ig
ht ©
2 01
4. T
ay lo
r &
F ra
nc is
G ro
up . A
ll rig
ht s
re se
rv ed
.
24 Chapter 2 PLANNING A PROGRAM EVALUATION
Politics of Evaluation Research Up to this point, we have presented a rational model of program
evaluation. But it must be stressed that programs are political ani- mals. Th e reputations and egos of program sponsors and adminis- trators are tied to the success of the proposed program. It is unlikely that these groups will approach the evaluation in an experimental fashion. Th us, the evaluator must navigate through some treacher- ous territory, trying to maintain research accuracy and validity while measuring program performance in a fair fashion.
One of the founders of evaluation research, Dr. Donald T. Campbell, examined the diff erences between “trapped” and “experi- mental” administrators. A trapped administrator is committed to the relevance and signifi cance of the program. Th erefore, if the evalua- tion research fi ndings are critical or negative, a trapped administra- tor will feel threatened and will be inclined to question the validity of the research, regardless of previous involvement in the development of the evaluation process. An experimental administrator has a decid- edly broader view and is committed to the improvement of public pol- icy rather than the promotion of a particular program. If the program under review is found to be ineff ective, an experimental administrator will be disappointed but ready to plan a new initiative to address the problem at hand. Th is administrator is thus pragmatic, thinking stra- tegically about improving public policy rather than defending a failed program ( Campbell, 1969 ).
Th e answers that the program administrator seeks from program evaluation can be specifi ed as follows ( Behn, 2003, p. 588 ): ● Evaluate: How well is my agency performing? ● Control: How can I ensure that my subordinates are doing the
right thing? ● Manage the budget: On what programs, people, or projects should
my agency spend the public’s money? ● Motivate: How can I motivate line staff , middle managers, non-
profi t, and for-profi t collaborators, stakeholders, and citizens to do the things necessary to improve performance?
● Promote: How can I convince political superiors, legislators, stake- holders, journalists, and citizens that my agency is doing a good job?
● Celebrate: What accomplishments are worthy of the important organizational ritual of celebrating success?
● Learn: How will I know why a program is working or not? ● Improve: What exactly should be done diff erently to improve
performance? But, once again, these purposes refl ect rational motives. A trapped
administrator can seek to use the program evaluation for illegitimate purposes ( Weiss, 1998 ). He or she may seek to use the evaluation to
Vito, G. F., & Higgins, G. E. (2014). Practical program evaluation for criminal justice. Taylor & Francis Group. Created from liberty on 2023-01-26 20:17:10.
C op
yr ig
ht ©
2 01
4. T
ay lo
r &
F ra
nc is
G ro
up . A
ll rig
ht s
re se
rv ed
.
Chapter 2 PLANNING A PROGRAM EVALUATION 25
postpone and delay a diffi cult decision by conducting the evaluation before making one. He or she may use the evaluation results to duck responsibility for performance. Finally, evaluations are often conducted because they are required by the funding agency’s grant requirements and program administrators may view this as an obligation that serves no true purpose for them.
Regardless of these perceptions, the evaluator must attempt to cre- ate a climate for the research that maintains the priorities necessary for reliable and relevant fi ndings that can credibly inform public policy. Th ere must be continuous interaction between the evaluator and pro- gram administrators to promote an exchange of ideas that will lead to the best possible evaluation process. Th e evaluator must be especially careful not to overreact to both positive and negative fi ndings and to instead interpret them in a constructive, neutral, and realistic manner. Program administrators should have input in the research process to ensure accurate and realistic measurement of program performance while prohibiting their strict control over the research results. Th e evalu- ator must address these issues with program administrators through- out the entire process of program evaluation to keep everyone aware of their rightful roles in the enterprise. Similarly, Carol Weiss (1998) advises evaluators to determine who initiated the idea of conducting a program evaluation and for what purposes. She recommends an exami- nation of the commitment among program administrators to use the results of the evaluation to improve decision making in the future.
Ethical Issues in Evaluation Research Ethical issues in program evaluation can be divided into two related
areas: those relating to the conduct of the research and the other to how the evaluator responds to the social nature of program evaluation.
Ethics of Conducting Research Ethics refers to how the proposed evaluation research conforms
to professional standards of what is “right” and “wrong.” Th e ultimate aim is to prevent harm to research subjects while promoting a research design that will generate valid and relevant results that will help inform public policy. It is often a diffi cult balance to maintain, but protection of human subjects must remain paramount over the need for scien- tifi c knowledge. Th e basic issue is whether there is potential harm to research participants and whether or not this outweighs the poten- tial benefi ts of the study. Additionally, the evaluation researcher must make ethical decisions in analyzing the data and reporting the research fi ndings. Th e evaluation research may face political pressure to report fi ndings in a certain “acceptable” way—that is, minimizing negative
Vito, G. F., & Higgins, G. E. (2014). Practical program evaluation for criminal justice. Taylor & Francis Group. Created from liberty on 2023-01-26 20:17:10.
C op
yr ig
ht ©
2 01
4. T
ay lo
r &
F ra
nc is
G ro
up . A
ll rig
ht s
re se
rv ed
.
26 Chapter 2 PLANNING A PROGRAM EVALUATION
results (a “whitewash”) and emphasizing positive ones. Either way, sig- nifi cant fi ndings should not be minimized or compromised.
Table 2.2 lists the fi ve ethical principles adopted by the American Evaluation Association ( AEA, 2004 ). Th ese principles are intended to guide the professional practice of evaluators and to inform evaluation clients and the public about what ethical principles evaluators should follow and uphold. Th ese fi ve principles provide an ethical framework within which criminal justice program evaluation should be conducted.
Principle A has to do with systematic inquiry and the methodology of evaluation. Evaluators should utilize methods that have the highest technical standards that are appropriate to the questions and subjects of the evaluations. Th ey should answer questions clearly and present their methodology and analysis techniques in suffi cient detail to permit understanding to allow others to interpret and critique their work. As noted in Chapter 1 , such detail allowed researchers to make the assess- ment of what works that was detailed in the “Maryland Report.”
Principle B discusses the competence of evaluators. Th ey should possess the technical skills, background, and education to perform the research tasks required by the evaluation. In addition, evaluators should refl ect “cultural competence”—awareness of their own culturally based assumptions, their understanding of the diverse worldviews of the par- ticipants and stakeholders in the evaluation, and the use of appropri- ate evaluation methods in working with culturally diverse groups with regard to race, ethnicity, gender, religion, socioeconomics, or other fac- tors relevant to the program evaluation.
Principle C calls for the integrity and honesty of evaluators. Evaluators should honestly negotiate with clients and stakehold- ers regarding the costs of the research, the tasks to be undertaken (especially data collection and maintenance), the limitations of the
Table 2.2 Guiding Principles for Evaluators
A. Systematic inquiry: Evaluators conduct systematic, data-based inquiries.
B. Competence: Evaluators provide competent performance to stakeholders.
C. Integrity/honesty: Evaluators display honesty and integrity in their own behavior, and attempt to ensure the
honesty and integrity of the entire evaluation process.
D. Respect for people: Evaluators respect the security, dignity, and self-worth of respondents, program participants,
and other evaluation stakeholders.
E. Responsibilities for general and public welfare: Evaluators articulate and take into account the diversity of
general and public interests and values that may be related to the evaluation.
Source: American Evaluation Association, http://www.eval.org/publications/GuidingPrinciplesPrintable.asp .
Vito, G. F., & Higgins, G. E. (2014). Practical program evaluation for criminal justice. Taylor & Francis Group. Created from liberty on 2023-01-26 20:17:10.
C op
yr ig
ht ©
2 01
4. T
ay lo
r &
F ra
nc is
G ro
up . A
ll rig
ht s
re se
rv ed
.
Chapter 2 PLANNING A PROGRAM EVALUATION 27
methodology, the scope of the results obtained, and the uses of the data from the evaluation. Before the research is undertaken, evaluators should reveal any and all potential confl icts of interest they may have with their role as evaluator. If any changes in originally negotiated plans are made, evaluators should record why they were made and how they may signifi cantly aff ect the scope and results of the research. Clients and stakeholders should be informed of these changes in a timely fashion. Evaluators should explicitly divulge their own, the client’s, and the stake- holders’ interests and values in conducting the evaluation. Th ey should not misrepresent the methodology or data fi ndings of the research, as well as attempt to prevent the misuse of their research by others. Finally, evaluators should fully disclose the limitations of the research fi ndings and the source of funding and of the request for the evaluation.
Principle D calls for the respect for people. Evaluators should be aware of the contextual factors that may infl uence the results of a study, including geographic location, timing, political and social climate, economic conditions, and other relevant activities occurring simulta- neously. Evaluators must adhere to traditional social science ethical principles regarding the protection of human subjects, including: ● Anonymity: Th e data obtained for the evaluation must not to
be matched to an individual participant. No one, including the researcher, should able to determine who in the sample or popula- tion participated in the study and, if an individual did participate, what his or her responses were.
● Confi dentiality: Th is guarantees that the relationship will not be identifi ed in any written or verbal communication. Data may be analyzed individually or in aggregate, but when the data are reported, no identifying characteristics should allow for individual- level data/responses to be matched with a participant.
● Disclosure: Th e full disclosure of potential harm to subjects in a proposed evaluation should be provided upfront. While the harm or potential harm to subjects that may follow from research is not intentional, the researcher must be aware of the potential conse- quences and make decisions as a means of resolving the confl ict associated with the potential for harm. Th is is especially pertinent to criminal behavior where the potential harm to subjects is increased. Evaluators should seek to maximize the benefi ts and minimize any
unnecessary harm that might occur provided that these procedures do not compromise the integrity of the evaluation fi ndings. Evaluators must carefully judge the costs and benefi ts of performing certain evalu- ation procedures and anticipate them during initial negotiations for the evaluation. Finally, the results of the evaluation should be presented in a way that respects the dignity, self-worth, and diversity of the stake- holders, and to account for such diff erences in the planning, conduct, analysis, and reporting of program evaluation fi ndings.
Vito, G. F., & Higgins, G. E. (2014). Practical program evaluation for criminal justice. Taylor & Francis Group. Created from liberty on 2023-01-26 20:17:10.
C op
yr ig
ht ©
2 01
4. T
ay lo
r &
F ra
nc is
G ro
up . A
ll rig
ht s
re se
rv ed
.
28 Chapter 2 PLANNING A PROGRAM EVALUATION
Principle E upholds the evaluator’s responsibilities for the general and public welfare. In the planning and reporting of evaluations, evalu- ators should include the relevant perspectives and interests of the full range of stakeholders, including not only the immediate operations and outcomes for the program but also their implications and potential side eff ects. Access should be given to the results of the evaluation with the dissemination of results provided in a clear fashion that promotes understanding of the policy implications of the research fi ndings to not only the stakeholders but to society as a whole.
In sum, these principles were promoted and adopted to stimulate discussion of the proper practice of evaluation, to resolve potential problems that arise during the planning and conduct of the research, and to guide the practice of evaluation.
Ethics and Social Relationships in Evaluation Research
Th e evaluation researcher must confront the nature of his or her relationship not only with the subjects of the research but fi rst with the program administrators and staff . Th ere is a fi ne line between operat- ing as an evaluator and a consultant. Th e evaluator must maintain an independent and objective stance but may be called on to off er advice about program operations as it is designed, implemented, and becomes operational. Feedback on program construction and operations is an important role for the evaluation researcher. Th e question here is: Can the evaluator play these related roles while maintaining objectivity about program performance? Is it ethical not to off er advice that could improve program operations and performance to maintain research objectivity?
As a program develops and progresses, the evaluation researcher will develop personal ties with program administrators and staff that may make it diffi cult to deliver “bad news” about program perfor- mance. Such relationships are important for professionalism but, again, independence and objectivity must be maintained if the researcher is to maintain credibility. Th e relationship between the evaluator and the program staff and administration is a symbiotic one. After all, the evalu- ator has been hired because of his or her scientifi c expertise in research methodology and data analysis. Th is knowledge will ensure the valid- ity of the research fi ndings. But the staff and administration are the “experts” in terms of the service provided by the program. Th ey have the knowledge and experience in program substance and delivery that is needed to properly inform the program evaluation, which can lead to the accurate and valid measurement of program goals and the interpre- tation of research results in a grounded and realistic way. Together, the evaluator and program staff and administration can off er their specifi c
Vito, G. F., & Higgins, G. E. (2014). Practical program evaluation for criminal justice. Taylor & Francis Group. Created from liberty on 2023-01-26 20:17:10.
C op
yr ig
ht ©
2 01
4. T
ay lo
r &
F ra
nc is
G ro
up . A
ll rig
ht s
re se
rv ed
.
Chapter 2 PLANNING A PROGRAM EVALUATION 29
expertise to each other in a way that maximizes the integrity of the research while not compromising independence and objectivity. After all, both groups must be able to deal with and accept the consequences of the research fi ndings and outcomes. As Weiss (1998) has dutifully indicated, the evaluator must be able to live with the study, its uses, and his or her conscience at the same time.
Another aspect that is unique about program evaluation is that the evaluator is typically a contract researcher who has been hired to perform the task of program evaluation. Most often, the evaluation is funded by public fi nancing through grant funding. Typically, the pro- gram evaluator is part of the team that writes the initial grant proposal because program evaluation is a component of the grant requirements. One of the guidelines used by the funding agency in the decision to fund a proposal is the quality of the proposed evaluation design. Once funded, the research contract can lead to several misinterpretations and ethical diffi culties. First, the existence of a contract often clouds the issue of who owns the research work and results—the funding agency, the program administration, or the evaluator? Th ese issues must be resolved as soon as possible—ideally, before the research is undertaken. Also, as previously noted, the uncertainty of the fi nd- ings can aff ect the program administrators in negative ways. Th ey are often unsure of what they are looking for from the evaluation research other than the natural desire for a positive result and endorsement of the program ( Inciardi & Siegal, 1981, pp. 170–171 ). Finally, it must be stressed that the evaluator must maintain both independence and objectivity. In the end, the service provided is “paid for” but the research results are not “bought.”
Summary Careful planning for a program evaluation includes attention to
methodology, analysis, and implementation, as well as social and political issues. Evaluators must be attentive to how the proposed research will aff ect all stakeholders while maintaining professional integrity and independence and meeting the requirements of social science research. Th is means that both methodological and social issues must be addressed continuously throughout the process of eval- uation research.
Discussion Questions 1. What is problem-oriented policing and how does the SARA model
demonstrate the need for planning in evaluation research?
Vito, G. F., & Higgins, G. E. (2014). Practical program evaluation for criminal justice. Taylor & Francis Group. Created from liberty on 2023-01-26 20:17:10.
C op
yr ig
ht ©
2 01
4. T
ay lo
r &
F ra
nc is
G ro
up . A
ll rig
ht s
re se
rv ed
.
30 Chapter 2 PLANNING A PROGRAM EVALUATION
2. Why is it important to state evaluation goals in measurable terms? What are the obstacles to this task and how can evaluators over- come them?
3. Defi ne and discuss the elements of a logic model for program evaluation.
4. Discuss the problems raised by the politics of evaluation research. 5. Discuss all of the ethical principles established by the AEA and
why they are important.
References American Evaluation Association (2004, July). Publications . Retrieved January 27,
2013, from American Evaluation Association. < http://www.eval.org/publications/ GuidingPrinciplesPrintable.asp > .
Behn , R. D. ( 2003 ). Eight purposes that public managers have for measuring performance . Public Administration Review , 63 ( 5 ) , 586 – 606 .
Bickman , L. , Rog , D. J. , & Hedrick , T. E. ( 1998 ). Applied research design: A practical approach . In L. Bickman & D. J. Rog (Eds.), Handbook of applied social research methods (pp. 5 – 38 ) . Th ousand Oaks, CA : Sage .
Boruch , R. F. ( 1998 ). Randomized controlled experiments for evaluation and planning . In L. Bickman & D. J. Rog (Eds.), Handbook of applied social research methods (pp. 161 – 192 ) . Th ousand Oaks, CA : Sage .
Campbell , D. T. ( 1969 ). Reforms as experiments . American Psychologist , 24 ( 4 ) , 409 – 429 . Capowich , G. E. , & Roehl , J. A. ( 1994 ). Problem-oriented policing: Actions and
eff ectiveness in san diego . In D. Rosenbaum (Ed.), Th e challenge of community policing: Testing the promises (pp. 127 – 128 ) . Th ousand Oaks, CA : Sage .
Davidson , E. J. ( 2005 ). Evaluation methodology basics: Th e nuts and bolts of sound evaluation . Th ousand Oaks, CA : Sage .
Eck , J. C. , & Spelman , W. ( 1987 ). Who ya gonna call? the police as problem-busters . Crime and Delinquency , 33 , 31 – 52 .
Goldstein , H. ( 1979 ). Improving policing: A problem-oriented approach . Crime and Delinquency , 25 , 236 – 258 .
Goldstein , H. ( 1990 ). Problem-oriented policing . New York : McGraw Hill . Inciardi , J. A. , & Siegal , H. A. ( 1981 ). Whoring around: Some comments on deviance
research in the private sector . Criminology , 19 ( 2 ) , 165 – 183 . Knowlton , L. W. , & Phillips , C. ( 2013 ). Th e logic model guidebook: Better strategies for
great results (2nd ed.) . Th ousand Oaks, CA : Sage . Locke , E. A. , & Latham , G. P. ( 2013 ). New developments in goal setting and task
performance . East Sussex, UK : Routledge . McDavid , J. C. , & Hawthorn , L. R. ( 2006 ). Program evaluation and performance
management: An introduction to practice . Th ousand Oaks, CA : Sage . Sieber , J. E. ( 1998 ). Planning ethically responsible research . In L. Bickman & D. J. Rog
(Eds.), Handbook of applied social research methods (pp. 127 – 156 ) . Th ousand Oaks, CA : Sage .
Suchman , E. A. ( 1974 ). Evaluative research: Principles and practice in public service and social action programs . New York : Russell Sage Foundation .
Weiss , C. ( 1998 ). Evaluation research: Methods for assessing program eff ectiveness . Engelwood Cliff s, NJ : Prentice-Hall .
Vito, G. F., & Higgins, G. E. (2014). Practical program evaluation for criminal justice. Taylor & Francis Group. Created from liberty on 2023-01-26 20:17:10.
C op
yr ig
ht ©
2 01
4. T
ay lo
r &
F ra
nc is
G ro
up . A
ll rig
ht s
re se
rv ed
.
