MANAGEMENT INTERVENTIONS

EVALUATION: PURPOSE AND DEFINITION

While some rightly say that the fundamental purpose of evaluation is the determination of the worth or merit of a program or solution (Scriven, 1967), the ultimate purpose, and value, of determining this worth is in providing the information for making data-driven decisions that lead to improved performance of programs and organizations (Guerra-López, 2007a). The notion that evaluation's most important purpose is not to prove but to improve was originally put forward by Egon Guba when he served on the Phi Delta Kappa National Study Committee on Evaluation around 1971 (Stufflebeam, 2003). This should be the foundation for all evaluation efforts, now and in the future. Every component of an evaluation must be aligned with the organization's objectives and expectations and the decisions that will have to be made as a result of the evaluation findings. These decisions are essentially concerned with how to improve performance at all levels of the organization: internal deliverables, organizational gains, and public impact. At its core, evaluation is a simple concept:

· It compares results with expectations.

· It finds drivers and barriers to expected performance.

· It produces action plans for improving the programs and solutions being evaluated so that expected performance is achieved or maintained and organizational objectives and contributions can be realized (Guerra-López, 2007a).

Some approaches to evaluation do not focus on predetermined results or objectives, but the approach taken in this book is based on the premise of performance improvement. The underlying assumption is that organizations, whether they fully articulate this or not, expect specific results and contributions from programs and other solutions. As discussed in later chapters, this does not prevent the evaluator or performance improvement professional from employing means to help identify unanticipated results and consequences. The worth or merit of programs and solutions is then determined by whether they delivered the desired results, whether these results are worth having in the first place, and whether the benefits of these results outweigh their costs and unintended consequences.

An evaluation that asks and answers the right questions can be used not only to determine results but also to understand those results and to modify the evaluation so that it can better meet the intended objectives within the required criteria. This is useful not only to identify what went wrong or what could be better but also to identify what should be maintained. Through appreciative inquiry (Cooperrider & Srivastva, 1987), evaluation can help organizations identify what is going right. Appreciative inquiry is a process that searches for the best in organizations in order to find opportunities for performance improvement. Here too the efforts are but a means to the end of improving performance. Although the intentions of most evaluators are just that, the language and approach used are charged with assumptions that things are going wrong. For instance, the term problem solving implies from the start that something is wrong. Even if this assumption is not explicit in the general evaluation questions, it makes its way into data collection efforts. Naturally the parameters of what is asked will shape the information evaluators get back and, in turn, their findings and conclusions. If we ask what is wrong, the respondents will tell us. If we ask what went right, again they will tell us. The key point is that evaluation should be as unbiased as possible. Evaluators should ask and answer the right questions, so that the data they get are indeed representative of reality.

In specific terms, before evaluators start to plan, and certainly before they collect data, they must determine why they are conducting an evaluation. Is this their initiative, or were they directed to do this work? What is the motivation for the study? What are they looking to accomplish and contribute as a result of this evaluation? Here are some general reasons for conducting an evaluation:

· To see if a solution to a problem is working, that is, delivering valued ends

· To provide feedback as part of a continual monitoring, revision, and improvement process

· To provide feedback for future funding of initiatives

· To confirm compliance with a mandate

· To satisfy legal requirements

· To determine if value was added for all stakeholders

· To hold power over resources

· To justify decisions that have already been made

Although the last two in this list are particularly driven by political agendas, in reality most reasons can be politicized; thus, it takes an insightful evaluator to recognize the feasibility of conducting an honest evaluation. An experienced evaluator will recognize, most of the time, whether evaluation stakeholders are truly interested in using evaluation findings to improve performance or are more concerned with advancing their political interests. With careful attention to detailed planning, either goal can be made to fit a data-driven and results-oriented action approach to evaluation. But if taken too narrowly—in isolation and without proper context—each has its own narrow set of problems, blind spots, and special data generation and collection issues. Perception of the purpose of the evaluation can shape and limit the data that are observed (or not observed), collected (or not collected), and interpreted (or ignored). Thus, evaluators and stakeholders must begin the planning process with a clear articulation of what decisions must be made with the results of their findings, decisions that are linked to the overall purpose for conducting the evaluation.

PERFORMANCE IMPROVEMENT: A CONCEPTUAL FRAMEWORK

The field of performance improvement is one of continuous transition and development. It has evolved through the experience, reflection, and conceptualization of professional practitioners seeking to improve human performance in the workplace. Its immediate roots stem from instructional design and programmed instruction. Most fundamentally, it stems from B. F. Skinner and his colleagues, whose work centered on the behavior of individuals and their environment (Pershing, 2006).

The outgrowth of performance improvement (also called human performance technology) from programmed instruction and instructional systems design was illustrated in part by Thomas Gilbert's behavioral engineering model, which presented various categories of factors that bear on human performance: clear performance expectations, feedback, incentives, instruments, knowledge, capabilities, and internal motives, for example. This landmark model was published in Gilbert's 1978 book, Human Competence: Engineering Worthy Performance, and was based in large part on the work Gilbert conducted with Geary Rummler and Dale Brethower at the time. Pershing (2006) declares that Joe Harless's 1970 book, An Ounce of Analysis Is Worth a Pound of Objectives, also had a significant impact on the field and was well complemented by Gilbert's work. Together these works served as the basis for many researchers who have contributed to and continue to help develop the performance improvement field.

Currently the International Society for Performance Improvement, the leading professional association in the field, defines performance improvement as a systematic approach to improving productivity and competence, using a set of methods and procedures—and a strategy for solving problems—for realizing opportunities related to the performance of people. More specifically, it is a process of selection, analysis, design, development, implementation, and evaluation of programs to most cost-effectively influence human behavior and accomplishment. This series of steps, commonly known as the ADDIE model, is the basic model from which many proposed performance improvement evaluation models stem. Pershing (2006) summarized performance improvement as a systematic combination of three fundamental processes: performance analysis (or needs assessment), cause analysis (the process that identifies the root causes of gaps in performance), and intervention selection (selecting appropriate solutions based on the root causes of the performance gaps). These three processes can be applied to individuals, small groups, and large organizations. The proposition that evaluation of such interventions should also be at the core of these fundamental processes is presented in the final chapter of this book.

This is the context in which evaluation is seen and described in this book—not as an isolated process but rather as one of a series of processes and procedures that, when well aligned, can ensure that programs and organizations efficiently and effectively deliver valuable results.

BENEFITS OF EVALUATION

Conducting an evaluation requires resources, but the benefits outweigh those costs in most situations. Here are some of the many benefits to include in an evaluation proposal or business case:

· Evaluation can provide relevant, reliable, and valid data to help make justifiable decisions about

how to improve programs and other solutions,

what programs and solutions to continue or discontinue,

how to get closer to organizational goals, and

whether current goals are worth pursuing.

· Evaluation plans and frameworks provide the basis for design, development, and implementation project management plans.

· Evaluation can identify any adjustments that have to be made during and after development and implementation, so that resources are maximized.

· Evaluation provides the means to document successes so that

the merit of decisions, department, staff, and solutions is recognized by all;

budget requirements and jobs are justified;

the quality of this work is respected by organizational partners;

the value of opinions and data is taken into account throughout the organization; and

evaluators gain credibility and competence, are granted autonomy and power along with accountability, and are seen as true strategic partners in the organization.

· Evaluation reports can be used to disseminate and market the organization's successes to internal and external partners, such as current and prospective customers.

https://scholar.flatworldknowledge.com/books/27617/portolesedias_1.0-ch11_s00/read

https://ebookcentral-proquest-com.ezproxy.snhu.edu/lib/snhu-ebooks/reader.action?docID=876641&ppg=16

https://scholar.flatworldknowledge.com/books/27617/portolesedias_1.0-ch11_s01/read

https://eds-a-ebscohost-com.ezproxy.snhu.edu/eds/detail/detail?vid=0&sid=9d6eccd0-74ad-46a2-96f3-20b8d047c73e%40sdc-v-sessmgr01&bdata=JnNpdGU9ZWRzLWxpdmUmc2NvcGU9c2l0ZQ%3d%3d#AN=130981413&db=bsu

GENERAL EVALUATION ORIENTATIONS

Two common distinctions in evaluation are formative and summative. Formative evaluation typically occurs during the developmental stage of a program and can be used to improve the program before it is formally launched. The formative approach can also be used to improve all stages of performance improvement, from assessment to implementation, and the evaluation itself.

Summative evaluation occurs after the implementation of a program or solution and usually requires some appropriate amount of time to have transpired so that the object of evaluation has the opportunity to have the full impact required on performance at various levels of the organization. It is worth noting that summative evaluation can also be used to improve programs and solutions. Stufflebeam and Webster (1980) hold that an objectives-based view of program evaluation is the most common type of evaluation. Once the results that have been accomplished have been determined, the evaluator is well advised to identify causal factors contributing to those results. These data should provide insights as to what the drivers and barriers to the success of the program are, thereby providing the basis for recommendations for improving performance.

Another distinction often made among evaluation orientations is that of process evaluation versus results evaluation. These terms are used to describe the same processes that formative and summative approaches, respectively, take. Depending on how these are interpreted and implemented, they can also differ somewhat from their counterparts described above. For instance, the Canadian Evaluation Society uses the term process evaluation (also referred to as efficiency evaluation) to describe the monitoring of the implementation of programs. Obviously, there should be a well-planned logic model with specified results and processes, but modifications are made if a discrepancy between the program design and the actual implementation is found. For example, one might want to determine if the program is being delivered as intended, if it is being delivered to the targeted clients or participants, or if it is being delivered with the intended effort or in the intended quantity.

Process evaluation is critical in helping evaluators address the variations in program delivery. The greater the variation in program delivery, the greater the requirement is for useful data gathered through a process evaluation approach. For instance, there may be differences in staff, clients, environments, or time, to name a few variables.

Stufflebeam and Webster (1980) have argued that objectives-based program evaluation is the most prevalent type used in the name of educational evaluation. Scriven (1972) proposed goal-free evaluation to urge evaluators to also examine the process and context of the program in order to find unintended outcomes.

Results evaluation, also referred to as effectiveness evaluation, is used to determine whether the immediate outcomes of a program meet predetermined objectives specified by program planners; impact evaluation tends to refer to an evaluation that looks at not only immediate outcomes but also the long-term outcomes of a program and their interdependency. A results evaluation approach is important because it allows us to ensure and document that we are on track by gathering data that show quality accomplishments. It also helps us stay accountable and our programs to stay cost-effective by making program benefits and costs tangible.

Other evaluation approaches are associated with effectiveness evaluation. Cost-benefit evaluation is the translation of costs and benefits into monetary terms, which is used to compare the relative net benefits of doing one thing versus another. However, monetary terms are not always applicable, and they are seldom sufficient to appreciate costs and benefits. Cost-effectiveness evaluation considers alternative forms of program delivery according to both their costs and their effects with regard to producing some result or set of results. Of course, a stable measure of result should be defined. The least costly program is not necessarily the best one. And in the context of technology solutions, an additional orientation to evaluation is usability testing, which focuses on whether people are using the product and how well they are using it to meet required objectives.

https://ebookcentral-proquest-com.ezproxy.snhu.edu/lib/snhu-ebooks/reader.action?ppg=36&docID=1204651&tm=1535618228962

CHAPTER 2

PRINCIPLES OF PERFORMANCE-BASED EVALUATION

This chapter establishes and examines six principles that guide every high-quality evaluation. Evaluators must (1) ask the right questions. To do this, they must (2) understand the distinction between means and ends and (3) examine the goals of the organization as a function of its valid needs. (4) Needs should be validated using a top-down external approach. (5) Evaluation standards should be based on a useful and real strategic vision. (6) The set of questions derived from the first five principles drives the evaluation study.

Evaluation seeks the answer to the general question, What worked and what didn't? It should also ask at the same time, Of what value is this, and what does it contribute? Successful evaluation hinges on how the questions are framed in the context of results-oriented programs that serve an organizational and societal purpose. It also seeks to determine whether the results obtained are of value to the client organization and the society it serves.

The results and consequences of evaluation are not arbitrary. It is important not only to see whether we get the results we hope for but also to ensure that what we hope for is reflected in the means selected and that the ultimate value was added by a program, project, solution, or organizational system to our shared society. All of these are important.

Societal and organizational needs are met through organizational efforts focused on value-added outcomes. The primary focus of performance evaluation is to determine that the goals and objectives of programs are valid, as well as effective and efficient, in producing desired results internal and external to the program and organization.

Planning for legitimate programs starts by identifying desired organizational impacts on the organization's clients and market as well as on society—needs, not wants—and moves into the organization to derive strategic, tactical, and operational objectives. Thus, evaluators should look at the alignment of programs, organizational objectives, and desired external impact. All must be clearly aligned and linked. These ideas are essential threads that must be woven into the fabric of every evaluation.

Words and concepts are rigorously defined here, for evaluation should be used to measurably improve what organizations use, do, produce, and deliver. Undefined ideas and hasty selection of methods will not serve anyone well. Thus, we must be thoughtful and reflective as the pragmatic evaluation landscape is created. The journey will be worth the planning time and attention.

The following set of principles, based on Kaufman, Guerra, and Platt (2006), will be helpful to define and determine what worked and what did not.

https://scholar.flatworldknowledge.com/books/27617/portolesedias_1.0-ch11_s02/read

https://ebookcentral-proquest-com.ezproxy.snhu.edu/lib/snhu-ebooks/reader.action?ppg=30&docID=876641&tm=1535061883493 (PAGES 17-30)

https://www-proquest-com.ezproxy.snhu.edu/docview/1037692128?accountid=3783

OVERVIEW OF CLASSIC EVALUATION MODELS

Many worthwhile models that will not be covered here can certainly be applied (and in fact already may have been) in any number of contexts, including performance improvement. Before examining the evaluation models covered in the upcoming chapters, this chapter mentions some worthy evaluation approaches without studying them in detail.

Objective-Based Evaluation: Tyler (1949)

Over the course of Ralph Tyler's more than sixty-year career, he influenced education at all levels, including curriculum, testing, evaluation, and educational policy. Directly and indirectly, he influenced many noteworthy developments such as objective-referenced testing, objective-based program evaluation, mastery learning, achievement test construction, item banking, the taxonomic classification of educational outcomes, and cooperative test development (Madaus & Stuffleman, 1989). Objective-based evaluation describes whether students have met their goals, with the results informing how to handle a new instructional strategy (revise, adopt, or reject). One noted weakness of this approach is that the evaluator may overlook unexpected outcomes or benefits of instruction beyond the original goals.

Consumer-Oriented Evaluation: Scriven (1967)

This approach is focused on meeting consumer needs and societal ideals more than achieving the developer's objectives for a given program. Scriven made a major contribution to this approach with his distinction between formative and summative evaluation. He proposed that summative evaluation enables administrators to decide whether the entire finished curriculum, refined by the use of formative evaluation, represents a sufficiently significant advance on the available alternatives to justify the expense of adoption by a school system (Fitzpatrick et al., 2004). Scriven proposed a set of seminal criteria for evaluating any education product, titling it a Key Evaluation Checklist (Scriven, 1991, 2002). He continues to revise this checklist and uses it as part of a data-reduction process, where large amounts of data are obtained and assessed and then synthesized in an overall judgment of value (Stufflebeam & Shinkfield, 2007). Scriven's checklist also addresses meta-evaluation.

Discrepancy Model of Evaluation: Provus (1971)

For Provus, preestablished objectives were the basis for evaluation; however, he also emphasized the importance of providing data about the consistency of (or discrepancy between) what was planned and what was actually executed. The model has four basic phases: establishing objectives, collecting evidence of compliance with the standards, identifying any discrepancies between preestablished objectives and what was accomplished, and identifying and starting corrective actions. This model lends itself to a self-evaluation framework and a systematic approach to improvement. This model is the theoretical foundation for the Impact Evaluation Process (Guerra-López, 2007a).

Goal-Free Evaluation: Scriven (1974)

This approach compensates for inherent weaknesses in a goals-oriented approach by providing an unbiased perspective of ongoing events. Here, the evaluator purposely remains uninformed of the program's predetermined goals and looks for all effects of a program regardless of its developer's intended objectives. If a program is meeting its intended purpose, the evaluation should confirm that. In addition, the evaluator will be more likely to find unanticipated effects that goal-based evaluators could miss because of the specificity of their search. Stufflebeam and Shinkfield (2007) indicate that goal-free evaluation provides important supplementary information, expands the sources of evaluative information, is effective for finding unexpected information, is cost-efficient, and is welcomed by clients.

Responsive/Client-Centered Evaluation: Stake (1975)

Based on Ralph Tyler's conception that evaluators should compare observed outcomes to intended outcomes, Stake's approach expanded evaluation focus to include background, process, standards, judgments, and outcomes. Although his philosophy of evaluation first appeared in the literature in 1967, his formal proposal of this approach, named responsive evaluation, was not published until 1975. In this work, he broke away from Tyler's perspective of examining whether intentions had been realized, instead assuming that intentions would change and calling for continuing communication between evaluator and stakeholders for the purposes of discovering, investigating, and addressing important issues (Stufflebeam & Shinkfield, 2007).

Utilization-Focused Evaluation: Patton (1997)

Closely linked to decision-oriented theory, this approach to evaluation is concerned with designing evaluations that inform decision making. Although many authors have contributed to this approach, Stufflebeam and Shinkfield (2007) credit Michael Patton as the most prominent figure (1980, 1984, 1997, 2003). In his 1997 book, Utilization-Focused Evaluation, Patton fully articulated his approach to evaluation. Here he defined a utilization-focused program evaluation as one done “for and with specified intended primary users for specific, intended uses” (p. 23).

SELECTING A MODEL

There is no one best model. The utility of evaluation models, as with any other type of model or tool, depends entirely on the situation at hand. What works in one organization does not necessarily work in another, and what worked yesterday will not necessarily work in the same way today. Stufflebeam and Webster (1980) analyzed thirteen alternative evaluation approaches in terms of their adherence to the definition of an educational evaluation: one that is designed and conducted to assist some audience to judge and improve the worth of an educational endeavor. Their analysis resulted in three categories of evaluation studies: politically oriented, or pseudo-evaluations; question oriented, or quasi-evaluations; and values oriented, or true evaluations. They provided strengths and weaknesses for each in order to offer evaluators a variety of frameworks and perspectives.

The most important part of choosing a responsive model is that we clearly identify the requirements of the situation and use that as the criterion for selecting an evaluation model. Popham (1975) has said that comparing evaluation approaches in order to select the best model is usually a waste of time. He argues that instead of focusing on similarities and differences, evaluators should become sufficiently proficient with the evaluation models to decide which is the most appropriate for the situation. Moreover, he has argued for an eclectic approach: selectively drawing from various models the concepts that are most helpful. Following Popham's advice, we might find it useful to borrow the most relevant bits and pieces of various models to form an evaluation plan. This approach can be taken for every evaluation. Cronbach (1982) also supports this view and states that evaluation design must start from a blank slate at each new undertaking, addressing the countless decisions to be made in each situation.

CONCEPTUALIZING A USEFUL EVALUATION THAT FITS THE SITUATION

Nevo (1981) revised a list of questions initially proposed by Stufflebeam (1974) in an effort to conceptualize evaluation and its key dimensions. The revised list remains applicable today:

· How is evaluation defined?

· What are the functions of evaluation?

· What are the objects of evaluation?

· What kinds of information should be collected regarding each object?

· What criteria should be used to judge the merit or worth of an evaluated object?

· Who should be served by the evaluation?

· What is the process of doing an evaluation?

· What methods of inquiry should be used in evaluation?

· Who should do evaluation?

· By what standards should evaluation be judged?

Some reflective questions will help formulate a model for a specific evaluation. Note that this list is not exhaustive but rather is a starting point:

· What are the characteristics of the evaluation task?

· What is the object of evaluation (the evaluand)?

· At what stage of conception (planning, design, development, implementation, maintenance, evaluation) is the evaluand?

· What are the limitations and constraints of the evaluation effort?

· Which evaluation models best lend themselves to address these characteristics?

· What are the pros and cons of each model with regard to the evaluation task?

· Is blending these approaches to create a unique model for this situation more feasible and responsive to the evaluation task than using one of them as it stands?

You may find it useful to keep these questions in mind as you read through the following chapters.

https://scholar.flatworldknowledge.com/books/27617/portolesedias_1.0-ch11_s03/read

https://ebookcentral-proquest-com.ezproxy.snhu.edu/lib/snhu-ebooks/reader.action?ppg=142&docID=876641&tm=1536688820672

https://ebookcentral-proquest-com.ezproxy.snhu.edu/lib/snhu-ebooks/reader.action?ppg=64&docID=876641&tm=1535062846299

https://scholar.flatworldknowledge.com/books/27617/portolesedias_1.0-ch10_s01/read

https://www-proquest-com.ezproxy.snhu.edu/docview/216364942?accountid=3783

https://ebookcentral-proquest-com.ezproxy.snhu.edu/lib/snhu-ebooks/reader.action?ppg=157&docID=876641&tm=1535620278020

https://learn.snhu.edu/d2l/le/content/854335/viewContent/14578818/View

Performance

Evaluation Sheet.docx

Performance Evaluation Sheet

This document contains performance evaluation forms for three employees. Replace the bracketed text with specific information.

Performance Evaluation One

Employee’s Name: [Insert employee name]

Employee’s Title: Customer Service Representative

Employee’s Department: Customer Service

Please put an X in the area that best describes this employee’s performance.

Provide specific comments to describe the ratings for each category:

[Using the provided employee sample cases, write comments addressing the employee’s performance, using best practices for providing employee feedback.]

Outline any remediation steps: