4 pg report - 4 question 1 pg each

Shoestring Evaluations 2

Summary of major arguments

The journal article, “Shoestring Evaluations: Designing Impact Evaluations under

Budget, Time and Data Constraints,” Bamberger, Rugh, Church, and Fort (2004), outlines a six

step methodology for responding to some commonly reported issues in conducting evaluations.

The authors suggest these issues arise in two typical scenarios. In one instance, the evaluator is

brought in early, even at the very start of the project. Then, for a variety of reasons, he or she is

left out as the project progresses. In the second situation, the evaluator does not join the project

until late in the process. While noting the variance in the timing, Bamberger et al. (2004) observe

there are three common reasons, or constraints, that typically drive an evaluator timing decision.

These are: budget, time and data. In response to these commonalities and current evaluation

practice, they have developed a “shoestring evaluation” model.

In the shoestring model, Bamberger et al. (2004) provide an approach that responds to the

constraints of time, budget and data. At the same time, they attempt to overcome the threats to

validity and potential diminishment of quality inherent in a limited deployment of established

evaluation principles. Further, the model addresses some of the specific compromises fostered by

client demands for smaller budgets, quick results and limited data collection.

Bamberger et al. (2004) outline, in narrative and table form, common constraint scenarios

and define groups which will benefit. The six steps in the model are the designing the overall

plan (step one) acknowledging and responding to the three constraints (steps two through four),

and (steps five and six) assessing and addressing inherent weaknesses and strengths of the

evaluation design. The final two steps focus on threats to validity, objectivity, and replication.

Bamberger et al. provide comprehensive instructions and suggestions for each step. They suggest

ways to simplify the design of an evaluation and to collect and use data effectively all while

Shoestring Evaluations 3

maintaining the integrity of the criteria and judgments. The model also addresses pretest-posttest

design, control groups, lack of baseline data, researcher bias, limited documentation, minimal

data collection, and maintenance of quality. Specific cost saving suggestions from Bamberger et

al. include: design simplification, reduced sample size, and inexpensive data collection methods.

The content and my experience

I found this journal article compelling because it so closely matches my professional

experience. During the time I worked in marketing, I conducted many evaluations. Many fit the

“evaluator brought in after the project started model” identified by Bamberger et al. (2004).

Often a client suggested some measurement should be made regarding the success of a project

we were conducting. Since we were both the program conductors and evaluators, the issue of

bias was noted. Generally, however, there were very specific, precise, measurable criteria to use

in the evaluation. Yet, the judgment – what level of achievement is success for those criteria—

was determined after start up. Or the value judgment was subconsciously understood, albeit

unspoken from the start. In addition, budgets were limited and time was critical, again matching

the observations of Bamberger et al.

As a higher education administrator, I now regularly participate in program evaluations. I

am nearly always involved from the beginning. On occasion, I am the outside evaluator brought

in the planning stage. In some cases as the project progresses, I become distanced from the work,

usually due to time constraints. More commonly, as noted by Bamberger et al. (2004), due to

budget constraints, administrators become both the creators and evaluators. For example, I am

currently part of a team charged with developing a faculty evaluation model. This team

previously evaluated an existing process and found that it was not measuring the appropriate

Shoestring Evaluations 4

factors. For budget, time and data efficiency, it made sense for this same group to work on

creating and evaluating the new process. We recently agreed, however, to seek an independent

outside evaluator at some point in the process. Of course, this fits the evaluator brought in mid-

process model identified by Bamberger et al. The article matches my professional experience and

many of the concerns related to bias, validity and compromise raised by the authors. Since I do

not expect to move away from “the shoestring” given the current state of Michigan’s higher

education funding, I expect to use the Bamberger et al. six-step process soon.

Comments on the content

One self-disclosed limitation of the study is that all the demonstration cases are from

developing countries where budget, time and data constraints are common. Yet, the authors are

persuasive in their argument that the process has potential in broad application. At least, the

current evidence of many constraints being placed on evaluation endeavors suggests that the

methodology is worth testing. Additionally, the comprehensive discussion of the shoestring

model and its connection to established evaluation practices is evident. For example, the model

supports: addressing client needs, defining the program theory model for the project, integrating

qualitative and quantitative approaches, employing data strategies to reduce time and lessen

duplication, considering the views of key stakeholders and using participatory / collaborative

approaches. The current evaluation literature lists many of these elements as essential

requirements.

A key aspect of the article is specificity, precision and detail. For example, one table

outlines seven possible data designs in descending order of robustness. It clearly delineates the

resulting compromises and suggests ways to adjust other portions of the evaluation to temper the

Shoestring Evaluations 5

losses. Another table charts roles for practitioners and managers for three starting times: at the

beginning, during implementation and at the end.

In addition to outlining the shoestring model with precision and specificity, Bamberger et

al. (2004) provide four narrative descriptions of specific applications of the model. These case

studies add a level of honesty to the arguments for using the model. The examples, with source

citation, acknowledge the limitations of the process and demonstrate the reality of a deployment.

Another useful, and credibility enhancing feature of the article is an “Integrated Checklist

for Assessing the Validity and Adequacy of Multi-method Shoestring Evaluation Design” and a

“Summary: Shoestring Evaluation in a Nutshell” (Bamberger et al., 2004, p. 31-32). The

checklist provides precise advice regarding: objectivity/confirmability, reliability/dependability;

internal validity/credibility/authenticity; external validity/transferability/fittingness; and

utilization/application/action orientation. The summary provides a concise compendium of the

key principles for easy reference making it accessible to beginning and experienced evaluators.

The tables, charts, narrative examples, checklist and summary make it easy for a

practitioner to access and apply the shoestring process. Extensive use of peer-reviewed,

contemporaneous supporting references adds to the credibility of the article. In a broad sense,

shoestring evaluation is a methodology that directly responds to, and acknowledges, the current

reality without resorting to absolutes. Bamberger et al. state, “The Shoestring Evaluation

approach is being developed to respond to the demand for ways to work within budget, time and

data constraints while at the same time ensuring maximum possible methodological rigor within

the given evaluation context” (p. 33).

Shoestring Evaluations 6

References

Bamberger, M., Rugh, J., Church, M., & Fort, L. (2004). Shoestring evaluations: Designing

impact evaluations under budget, time and data constraints. American Journal of

Evaluation, 25(1), 5-37.