id5000-wk8

DeterminingtheSuccessoftheInstructionalDesignProductandProcessChapter10fromBrownGreen.pdf

Home >Education homework help >id5000-wk8

Chapter 10

Determining the Success of the Instructional Design Product and Process

Tobias is the lead instructional designer for a company that is developing training materials for web-based software. Although his team is making progress toward completing the project, he feels there are some issues that need to be resolved within the design and production team if the beta version of the training is going to be released on time. He wonders whether he should conduct a formative evaluation to determine issues that might be resolved to help improve the group’s work- ing relationship.

Elizabeth works for a faculty development center at a university. Her university has spent signifi- cant amounts of money over the past two years on two web-based synchronous communication and learning tools. Elizabeth was asked to create and deliver a three-part workshop on integrating the tools in order to increase faculty use of them. After conducting the workshop over two semesters, she was asked to evaluate the workshop’s effectiveness.

Guiding Questions

• What information can formative and summative evaluations provide to improve the instruc- tional design process?

• Why is it important to include various types of evaluation in the instructional design process? • How can the success of formative and summative evaluations be judged? • How can an instructional designer use evaluation to improve an instructional design team’s

effectiveness?

Source: Shutterstock 195817664.

01/27/2020 - RS0000000000000000000002052606 (May Katitar-Thompson) - The Essentials of Instructional Design

Determining the Success of the ID Product 163

Key Terms

formative evaluation (page 163) group processing (page 176) program evaluators (page 171) rapid prototyping (page 168) summative evaluation (page 170) usability testing (page 168)

Chapter Overview

Evaluation is a crucial part of instructional design. As we discussed in Chapter 9, evalua- tion is a process that allows for data to be gathered that help determine the level of success of someone or the worth of something. In instructional design, a major purpose of evalu- ation is to help determine how successful a learner has been as a result of participating in instruction. This type of evaluation is called “learner evaluation.” Although learner evalu- ation is extremely important, there are two other important evaluation types: formative and summative. These two types concentrate on the instructional design process and the instruction developed as a result of the process. This chapter’s focus is on defining forma- tive and summative evaluations, describing specific approaches of instructional design experts, and how formative and summative evaluations can be designed and implemented.

Formative and Summative Evaluation

It is a common misconception of beginning instructional designers that evaluation only takes place at the conclusion of instruction to evaluate learners. Evaluating the learners is only one of several types of evaluation that instructional designers must understand how to design and implement. In reality, evaluation should also take place at various stages of the instructional design process through the use of formative and summative evaluations. Knowing when and how to use various evaluation types helps instructional designers develop instructional interventions that are efficient and effective.

Formative Evaluation

Formative evaluation is used throughout the instructional design process to gather data that can be used to provide feedback on how the process is going. It is especially useful during the early stages of the instructional design process. The feedback allows an instructional designer to make improvements to the instruction before it is completely developed. This helps ensure that high-quality instruction is developed. In addition to helping improve the instruction, data gathered through a formative evaluation can be shared with a client to indicate how the project is progressing. Periodic communication with your client about how the project is progressing helps make certain that project goals and expectations will be met.

Approaches to Formative Evaluation

Instructional design experts take a variety of approaches to formative evaluation. Despite their differences, they all have in common the goal of helping improve the instruction that is being developed. Let us begin by looking at three different approaches to formative evaluation.

Smith and Ragan (2004) write that formative evaluation is conducted to “determine the weakness in the instruction so that revisions can be made to make them more effective

01/27/2020 - RS0000000000000000000002052606 (May Katitar-Thompson) - The Essentials of Instructional Design

164 Determining the Effect of the Intervention

and efficient.” This helps an instructional designer know “whether the instructional mate- rials are ‘there’ yet, or whether she needs to continue the design process” (p. 388). The stages that Smith and Ragan advocate for formative evaluation are design reviews, expert reviews, learner validation, and ongoing evaluation.

The first stage—the design review—is completed prior to the development of instruc- tion. Stage two—the expert review—typically occurs after the instruction is completed but before it is used with learners. The final two phases include the use of the actual instruc- tion with learners who represent the intended learners.

Design reviews are conducted after various phases of the instructional design process, such as the needs analysis, task analysis, goals and objective analysis, and learner analysis. Design reviews help to verify the accuracy of information at each stage of the instructional design process before instruction is developed. Smith and Ragan (2004) advocate revisiting the data gathered during these instructional design phases to determine how accurate they are.

Expert reviews are conducted to gather information about the instruction to determine if it is accurate and current. Various experts—such as content experts, instructional design experts, pedagogical experts, and experts on the learners—can be used to provide various perspectives on the instruction. The instruction provided for expert reviews is typically at the draft stage. Smith and Ragan (2004) suggest that expert reviewer comments should be divided into three categories: (1) revisions that need to be made immediately, (2) suggested revisions that require additional data that can be gathered during the final two stages of the formative evaluation, and (3) suggested revisions that should be ignored.

Learner validation includes three levels: one-on-one evaluation, small-group evalua- tion, and field trials. The three levels are identical to those promoted by Dick, Carey, and Carey (2011), discussed later in this chapter.

The final stage—ongoing evaluation—includes gathering data on the long-term effec- tiveness of instruction. Smith and Ragan (2004) state that if instruction is meant to be used multiple times, then provisions need to be made in a formative evaluation plan to collect effectiveness data. These data, which are similar to data collected during a field trial, provide feedback on what revisions needed to be made to instruction based on its actual use and whether the revisions made have been effective.

Morrison, Ross, and Kemp (2007) advocate a basic model for formative evaluation based on the work of Gooler (1980). Gooler’s approach follows these eight steps:

1 purpose 2 audience 3 issues 4 resources 5 evidence 6 data gathering techniques 7 analysis 8 reporting.

There are three main phases to this approach: planning, conducting, and reporting. Phase one includes steps one through five, while phase two includes steps six and seven. Phase three is the eighth and final step, reporting the results.

Phase one includes determining the evaluation purpose, the primary audience the results will be disseminated to, the issues that need to be addressed, the resources that are needed to address the issues, and the types of evidence that will be acceptable to address these issues. It is extremely important to work closely with the client during phase one in order to clearly articulate and manage expectations.

01/27/2020 - RS0000000000000000000002052606 (May Katitar-Thompson) - The Essentials of Instructional Design

Determining the Success of the ID Product 165

The first step—determining the purpose or purposes of the evaluation—is done in con- sultation with the client. The two most common purposes are to improve the instruction that is being developed and to satisfy administration requirements of the client you are working for. Many corporations and institutions require evaluations to be conducted. Most state agencies are required to include an evaluation component to help ensure that project goals are being met and to report how monies have been spent.

The audience of the evaluation is important to determine because it will establish the types of information that need to be collected and reported. The client will be able to help determine who the intended audience will be. Conducting an evaluation for multiple audi- ences should be avoided because it will be difficult to satisfy varying needs within a single report. It is best to try to narrow the audience down as much as possible.

After determining the purposes of the evaluation and the intended audience, it is time to determine the issues that need to be addressed. The issues are generally stated as questions that need to be answered. For example, in the case of an interactive, instructional kiosk:

• Are learners able to successfully utilize the kiosk after completing the instruction? • What elements of the kiosk do learners find most understandable and least

understandable?

Or in the case of web-based instruction:

• What is the average time spent on the summary exercises? • Do the learners find the web-based self-instructional materials helpful in learning the

content?

Once questions have been developed based on the evaluation issues, resources should be identified that are needed to answer the questions. People, artifacts, and equipment are com- mon types of resources that are often needed to address evaluation issues. It may be difficult to actually secure all the resources needed to address the evaluation issues. If this situation occurs, the instructional designer needs to communicate this to his or her client. Issues may need to be adjusted if the necessary resources cannot be secured. The final step in phase one is to identify the types of evidence that are needed to satisfy the evaluation issues.

Phase two—conducting the evaluation—includes determining the data collection tech- niques that will be used, gathering the data, and analyzing the data. When determining the data collection techniques to be used, Morrison et al. (2007) state: “Two key, and often opposing, factors need to be weighed: precise measurement versus feasible or practical measurement” (p. 310). For example, during the planning stage of a formative evaluation of a nurse-training program, an instructional designer determined that all participants would have specific skills evaluated by using hospital patients. However, due to logis- tical impracticalities (e.g., amount of time required), the nurse’s skills were tested on other nurses in the training program. An instructional designer must often make a choice between what is ideal and what is practical when gathering data.

The various data collection techniques that can be used are the same as those presented in Chapter 9. They include observations, questionnaires, interviews, paper-and-pencil tests, and performance tests—to name a few. It is highly recommended to use a variety of techniques to gather multiple sources of data. This can help to triangulate the findings.

Step seven—analyzing the data—should reflect the purpose of a formative evaluation: to provide usable and useful information that helps the instructional designer improve instruction. Complex statistical analyses are typically not required. Morrison et al. (2007) recommend the following typical analysis procedures:

01/27/2020 - RS0000000000000000000002052606 (May Katitar-Thompson) - The Essentials of Instructional Design

166 Determining the Effect of the Intervention

• frequency distributions • frequency graphs or histograms • descriptive statistics, such as percentages, means, and medians • listing of actual comments made by respondents.

(p. 311)

The final phase—step eight—is reporting the results of the evaluation to the primary audi- ence. This is typically done as an evaluation report, with the format of the report tailored to the audience the report will be disseminated to. A typical evaluation report is formatted in the following way:

1 Executive summary: an abstract that outlines the major findings, conclusions, and recommendations.

2 Evaluation purpose:

• evaluation issues—stated as questions • description of the instruction being evaluated.

3 Methodology used:

• participants • instruments used to gather data.

4 Results:

• analysis • findings.

5 Conclusions and recommendations.

It is not uncommon that, in addition to a written report, an oral report of the project will be requested. The oral report can either be conducted one-on-one with the client or more formally with a larger group. It is important to remember that the purpose of the evalu- ation is to provide recommendations based on the data collected. When disseminating a final report, be certain to highlight the conclusions and recommendations.

Dick et al. (2011) write: “The emphasis in formative evaluation is on the collection and analysis of data and the revision of instruction” (p. 258). They provide three basic phases of formative evaluation: (1) one-on-one or clinical, (2) small group, and (3) field trial. It is important to note that the type of formative evaluation Dick et al. advocate is designed to work specifically with self-instructional materials. Despite this, elements of their approach can be used with other types of instruction and materials.

During phase one, the instructional designer works with individual learners to gather data that are used to revise the instruction. The purpose of the phase is to remove obvious errors and omissions in the instruction, gauge initial learner performance, and determine learner reactions to the content. Dick et al. (2011) suggest that the instructional designer work with at least three learners (with varying abilities, attitudes, and/or experiences) who represent the target population the instruction is intended for. However, only one learner at a time should be worked with during a one-on-one evaluation. The instructional designer should be actively involved during this phase.

After selecting the learners who will participate in the one-on-one evaluation, the steps to this phase are:

1 Explain to the learner that you would like his or her reactions to new instructional materials that have been developed.

2 Give a pretest on the content.

01/27/2020 - RS0000000000000000000002052606 (May Katitar-Thompson) - The Essentials of Instructional Design

Determining the Success of the ID Product 167

3 Encourage the learner to talk about the materials as he or she goes through the instruc- tion and to be candid about what he or she likes or does not like, what makes sense, and what does not make sense.

4 Give a post-test; discuss the results with the learner by asking him or her to explain the responses made.

5 Note the time it takes the learner to go through the instruction.

Phase two has two primary purposes: (1) to determine how effective the changes are that were made as a result of phase one and (2) to determine if learners can go through the instruction without the assistance of the instructor. For this phase, eight or more learners should be selected who are a representative sample of the entire intended group of learners. After selecting the learners who will participate in the small-group evaluation, the steps to this phase are:

1 Explain to the group of learners that you would like reactions to new instructional materials that have been developed; you want them to work through the material on their own as best they can.

2 Conduct the instruction as it is intended to be used in its final form; give a pretest if that is part of the instruction.

3 Observe what is taking place without involvement unless the learners are unable to continue the instruction; make note of the difficulties learners have in completing the instruction.

4 Conduct a post-test once the instruction is finished. 5 Administer an attitude questionnaire. 6 Debrief the group of learners.

The final phase—the field-trial evaluation—is intended to help determine how effective the changes made during the small-group evaluation are and whether the instruction is capable of being used in its intended environment. A representative group of 30 learners should be used for this phase. The environment chosen for the field-trial evaluation should be similar to an environment for which the instruction is being developed.

The steps to this phase are similar to those of the small-group evaluation. The major difference is that an instructor, not the instructional designer, should administer the instruction. The role of the instructional designer is to observe and take notes of what takes place. The outcome of a field-trial evaluation is to determine the attitudes of the learners toward the instruction as well as the learners’ achievement.

In addition to these three phases, Dick et al. (2011) advise that the instruction should be given to subject matter experts and/or instructional experts to note whether it is accurate and current. This information—along with the data gathered during the three phases—should provide an instructional designer with enough information to make neces- sary revisions to the instruction being developed.

Professionals in Practice

Design and development occur in parallel—an existing course does not require the preliminary design steps that a new course typically requires. Loosely, the format is lecture-discussion-lab with an emphasis on avoiding PPT overkill. All courseware is first implemented in a beta environment so that the course flow can be observed, and new labs

(continued)

01/27/2020 - RS0000000000000000000002052606 (May Katitar-Thompson) - The Essentials of Instructional Design

168 Determining the Effect of the Intervention

(continued)

can be tested on live systems. It can be a stressful exercise. In some cases, “secret beta” classes are conducted (typically by the instructional designer) in which students don’t real- ize that what they are experiencing is new. Formative evaluation is conducted every step of the way until a class is released—warts and all. After a beta class, student feedback is collected via questionnaire, and post-mortems are conducted with any instructors who were in attendance. Everything related to instruction is fair game. Here are some recent examples:

• Are fonts in the PPT too small to be seen from the last row in the room? • Does a lab work with one web browser but not another? • Are numbered steps in a lab out of order? • Should number ordering restart after each section in a lab? • Should a PPT slide which introduces a lab include starting *and* ending pages? • Is the lab so simple that a monkey could do it by just blindly following instructions

(remember the whole “why” issue)?

If courseware issues are too complex to fix within a short timeframe, we try to fix them the next time around. It’s very much an iterative process.

Erik Novak Technical Training Developer

F5 Networks

Rapid Prototyping and Usability Testing

In addition to the three approaches we described in the previous section, two additional— and related—approaches to formative evaluation are important for instructional designers to understand and have as part of the instructional design skill repertoire: rapid prototyp- ing and usability testing. Both are often overlooked as formative evaluation approaches because they are traditionally thought of as “product processes” rather than formative evaluation processes. However, built in to each approach is formative evaluation, which allows the instructional designer to determine how well the instructional intervention is being designed.

As we discuss in Chapter 1, the essential idea behind rapid prototyping is to arrive at a final product through the creation of a series of prototypes. Each prototype is evaluated by some combination of experts and end users; each successive prototype is more like the final product—that is, “fidelity” of the prototypes increases with each new one until a working product is achieved. A rapid prototyping approach requires that the design environment allow for the relatively quick and easy creation of instructional materials (Tripp & Bichelmeyer, 1990).

Rapid prototyping suggests that each member of the design/development team is a “co-inquirer” (Rathburn, Saito, & Goodrum, 1997) and that there is a social process of design and development in which everyone offers feedback and criticism to each other. With rapid prototyping, each time a new version of the product is tested, it provides a way to critically reflect on what the final product should actually look like (Rathburn et al., 1997). The analysis of learners’ needs and the content of the final product depend in part

01/27/2020 - RS0000000000000000000002052606 (May Katitar-Thompson) - The Essentials of Instructional Design

Determining the Success of the ID Product 169

on the knowledge that is gained by actually building and trying out a prototype (Tripp & Bichelmeyer, 1990).

Usability testing is a concept originally borrowed from engineering that has gained popularity among software developers, multimedia producers, and instructional designers. Usability testing is a type of formative evaluation that consists of evaluating the product by observing it in action with potential end users. Typically, a usability test consists of evaluators observing a user or users interacting with the product under controlled condi- tions. The user(s) interact with the product for a specified amount of time, having been asked to complete a set of tasks determined in advance by the evaluation team. The tasks are based on the goals and objectives the product is designed to meet. Results of a usability test inform the production team of the problems inherent in the design (e.g., the graphic for a button is not well understood or the instructions are worded in a manner that is confusing) and suggest—through observation of end users in action—ways to effectively eliminate those problems.

Professionals in Practice

Usability testing is key. There’s no way to know whether your user interface is actually usable, unless you test it with real users. This may sound pedantic, but you’d be amazed how frequently this step is overlooked in user experience design. Validating design with fellow designers, developers, managers, or any other internal people, is simply not enough to ensure that end users will be able to interact with your UI efficiently.

Usability test early and often, WITH USERS. Test your idea, your paper prototypes, your interactive prototype, and developed product. Early testing gives you time to fix things before development is underway (a much less costly timeframe to make major changes to a workflow or page layout). Later testing helps to confirm that your design survived its translation into completed code.

Usability testing resolves arguments. Design sessions can get heated, and rival opinions emerge, on the “right” or “best” or “most usable” solution. Users cut right through that argument. They don’t care who designed something, or how much time you spent on a particular design. Either they can use it or they can’t. If you have more time, you can test multiple ideas. If you have less time, you can test and refine one design (and hope you had a good starting point). Either way, the user is your objective third party to shut down disagreements.

Usability testing doesn’t have to be complicated. A one-on-one qualitative usability test can be completed with as few as five users (one at a time, that is) and will give you a high level of confidence that you have identified key issues. Large-scale tests (100+ users) can be done online and there are many third-party providers of systems to support this. And don’t hesitate to invite your team to observe a test! There’s a huge difference between you reporting the results of a user test, versus a developer or a designer, with his own eyes, watching a user struggle with a “perfect” design.

There’s an art and a science to usability testing. Identifying and creating tasks for users to perform is a critical part of it. “Use the website” or “Give us your opinions” are not tasks! A good task includes a goal, such as “Send an email to the instructor that includes

(continued)

01/27/2020 - RS0000000000000000000002052606 (May Katitar-Thompson) - The Essentials of Instructional Design

170 Determining the Effect of the Intervention

(continued)

an attachment” or “Where can you find information about office hours?” that involves observable interaction with the UI.

At Ipswitch, we build complex applications for network administrators. Our software is data-heavy and designed to be used by people with deep technical skills. Even so, we can’t assume that just because our users are technically savvy, they’re able to interpret our user interfaces. What might seem blindingly simple to us (since we’ve been staring at our designs for months on end) will be baffling to these technical adepts. So, we test. And change our designs. And test again. And again, until we get to a point of confidence that our user interface is acceptable to the people we are asking to use it.

Lisa Hansen Senior Manager, User Experience

Ipswitch, Inc.

Tobias realizes, after talking separately to various members of his instructional design team, that they need to formally engage in some group processing. There seem to be some misconceptions regarding team member responsibilities. Although the group processing will most likely be initially uncomfortable for everyone and it will take time away from working on their product, Tobias believes it will help refocus the team.

Elizabeth believes that she needs to conduct a formative evaluation to determine if the workshops had a positive impact on participants. She is specifically interested in determining if and how faculty are using the web-based synchronous communication and learning tools in their courses.

Designing and Conducting a Formative Evaluation

Seasoned instructional designers understand the importance of formative evaluation in helping create efficient and effective instruction. Because of this understanding, they con- sistently include plans for formative evaluation when beginning ID projects. The approach to formative evaluation an instructional designer will take is dictated by the scope of the ID project. It may not be feasible for every ID project to conduct a complete formative evaluation by using one of the three approaches described in this chapter. A less robust formative evaluation may need to be developed and carried out for smaller ID projects. No matter what the scale of an ID project is, it is extremely important that some form of formative evaluation is carried out.

Summative Evaluation

As part of the instructional design process, summative evaluation takes place after an instructional intervention has been implemented. The major goal of a summative evalua- tion is to gather data that allow for its effectiveness to be determined. Did the instruction bring about the desired changes? Were the goals of the client met? These are two major questions that summative evaluations help to answer.

The results of a summative evaluation are described in a formal report outlining the impact of the instructional intervention. The evaluation report will typically outline who

01/27/2020 - RS0000000000000000000002052606 (May Katitar-Thompson) - The Essentials of Instructional Design

Determining the Success of the ID Product 171

participated in the instructional intervention, what activities impacted them, and what changes occurred from their participation. The evaluation report will often include the costs and benefits of implementation, descriptions of the essential conditions necessary to continue the program or reproduce it, and recommendations on whether the instructional intervention should be modified, discontinued, or continued as is.

Approaches to Summative Evaluation

For the beginning instructional designer, summative evaluation can be a daunting task. It is a complex process that takes a great deal of skill and experience to successfully carry out. There are evaluation experts whose sole professional activity is to conduct summative evaluations; they are typically referred to as program evaluators. A program evaluator uses various approaches to conduct a summative evaluation. In Program evalu- ation: Alternative approaches and practical guidelines (4th edition), Fitzpatrick, Sanders, and Worthen (2004) describe six such approaches (see Figure 10.1). It is beyond the scope of this book to describe the intricacies of these approaches and how to become a program evaluator. If you wish to learn more about summative evaluation than what is presented in this chapter, we suggest that you start by reading their book and by taking a course on program evaluation. The knowledge and skills gained as a result will certainly help you as an instructional designer.

Criterion Formative Evaluation Summative Evaluation

Purpose To determine value or quality To determine value or quality

Use To improve a program or instruction

To make decisions about the instruction’s future or adoption

Audience Program administrators and staff Program administrators and/or potential consumer or funding agency

By Whom Primarily internal evaluators, supported by external evaluators

External evaluators, supported by internal evaluators in unique cases

Major Characteristics Provides feedback so program personnel can improve it

Provides information to enable program personnel to decide whether to continue it, or consumers to adopt it

Design Constraints What information is needed? When?

What evidence is needed for major decisions?

Purpose of Data Collection Diagnostic Judgmental

Measures Sometimes Informal Valid and Reliable

Frequency of Data Collection Frequent Infrequent

Sample Size Often Small Usually Large

Questions Asked What is working? What needs to be improved? How can it be improved

What results occur? With whom? Under what conditions? With what cost?

Figure 10.1 Differences between formative and summative evaluation.

Source: Fitzpatrick, J., Sanders, J., & Worthen, B. Program evaluation: Alternative approaches and practical guidelines, 4th edition. 2004. Printed and electronically adapted by permission of Pearson Education, Inc., Upper Saddle River, New Jersey.

01/27/2020 - RS0000000000000000000002052606 (May Katitar-Thompson) - The Essentials of Instructional Design

172 Determining the Effect of the Intervention

We focus our discussion of summative evaluation on four different approaches that instructional design experts advocate. The purpose is to introduce how summative evaluation is approached in instructional design; it is not to make you an expert.

One of the most cited approaches to summative evaluation in ID is Kirkpatrick’s Four Levels of Evaluation. Kirkpatrick developed this model to evaluate the effective- ness of training programs—specifically, training programs in industry. The four levels of his model are: (1) reactions, (2) learning, (3) transfer, and (4) results. According to Kirkpatrick (1994), evaluation should always begin with level one and then progress through the remaining levels as time and the budget allows. However, to truly evaluate the effectiveness of a training program, all four levels should be used.

Each of Kirkpatrick’s levels requires more time and rigor than previous levels. Data col- lected from each level serve as a base for the next level and the evaluation that takes place at that level. As each level is conducted, more precise measures of the effectiveness of the training program are gathered.

Level 1—reactions—attempts to provide data on how participants reacted to the train- ing. Did participants enjoy the training? Was the training relevant to the participants? Kirkpatrick (1994) indicates that all training programs should at least include this mini- mum level of evaluation in order to provide data that will help improve the training. A typical method used to gather data at this level is an attitude survey, in which participants indicate how satisfied they were with the training. If students indicate negative feelings toward the training, this typically shows that little to no learning took place (however, positive reactions do not necessarily indicate that learning took place).

Level 2—learning—is conducted to determine whether participants’ skills, knowledge, or attitudes changed as a result of the training. Determining this is much more laborious than Level 1 because it requires gathering data at multiple times. Typically, pretests and post-tests are used to measure these changes.

Level 3—transfer—attempts to answer the question of whether the newly acquired skills, knowledge, or attitudes are being used by participants in their real-world environ- ments. In other words, have participants transferred what they learned in the training into their everyday environments? This is often considered to be the truest measure of a train- ing program’s effectiveness. Evaluating at this level is complicated because it is difficult to determine when a participant will actually display what he or she learned. Therefore, decisions will need to be made on when, how often, and how long the evaluation will take place in order to determine if transfer has taken place.

Level 4—results—attempts to evaluate a training program’s effectiveness in business measures, such as increased sales, improved product quality, fewer on-the-job accidents, and so forth. Evaluation at this level must be long term in nature in order to determine trends that have taken place. Data that help support the effectiveness of training at this level are often gathered through methods such as interviews with managers, focus group meetings with customers, and post-training surveys. Smith and Ragan (1999) write:

Within the context of instructional design, the purpose of summative evaluation is to collect, analyze, and summarize data to present to decision makers in the client orga- nization so that they can make a judgment regarding the effectiveness, and perhaps appeal and efficiency, of the instruction.

(p. 352)

The judgment that is being made is whether the use of the instruction should be continued. The specific question being asked is, “Does the instruction adequately solve the ‘problem’ that was identified in the needs assessment and that resulted in the development of the

01/27/2020 - RS0000000000000000000002052606 (May Katitar-Thompson) - The Essentials of Instructional Design

Determining the Success of the ID Product 173

instructional materials?” (p. 353). They provide an eight-step process for conducting a summative evaluation:

1 Determine the goals of evaluation: Identify the questions that should be answered. The questions should be developed in consultation with the client. Typically, more questions will be developed than can be answered during the course of the evaluation. It is important to come to an agreement on the exact questions that are feasible to answer given the available resources.

2 Select indicators of success: Where will data be gathered? What needs to be looked at in order to answer the evaluation questions? Again, in consultation with the client, these issues need to be addressed and agreed on.

3 Select the orientation of evaluation: Will the evaluation be objective or subjective? An objective orientation focuses on answering the evaluation questions based on data collected through quantitative methods (e.g., questionnaires, paper-and-pencil tests, per- formance tests). The advantage of this orientation is that results are generally replicable. A subjective orientation is based on the perspective of the individual conducting the evaluation. Qualitative methods (e.g., interviews and observations) are used to gather data that describe the impact of the instructional intervention. The advantage of this ori- entation is that a rich description is provided that describes the impact the instructional intervention has had. The downside to this orientation is that the results can be influ- enced by the biases of the evaluator. Most summative evaluations do not strictly follow one or the other orientation; typically, a combination of the two orientations is used.

4 Select design of evaluation: Determine how the data will be collected, when they will be collected, and under what conditions.

5 Design or select evaluation measures: Decide what measures will be looked at to determine effectiveness of the instructional intervention. Learning transfer, learning outcomes, attitudes, level of implementation, and costs are all measures that can be used to determine effectiveness.

6 Collect data. 7 Analyze data. 8 Report results: A summative evaluation report should include the following sections:

summary, background information, description of the evaluation study, results, and conclusion and recommendations. The background information will come from the analyses conducted early on in the ID process (e.g., needs, learner). The description of the evaluation study includes the purpose of the evaluation, the evaluation design, and the outcome measures. The results will include a discussion on how well the outcomes were met.

The approach taken by Morrison et al. is similar in many ways to the approach described by Smith and Ragan. Morrison et al. (2007) write: “A summative evaluation permits a designer or instructor to reach unbiased objective answers to evaluation questions con- cerning expected program outcomes and to then decide whether the program is achieving those outcomes” (p. 318). A summative evaluation can help examine the following issues:

• Effectiveness of learner or trainee learning. • Efficiency of learner or trainee learning. • Cost of program development and continuing expenses in relation to effectiveness

and efficiency. • Attitudes and reactions to the program by learners, instructors, and staff. • Long-term benefits of the instructional program.

(p. 320)

01/27/2020 - RS0000000000000000000002052606 (May Katitar-Thompson) - The Essentials of Instructional Design

174 Determining the Effect of the Intervention

These issues fall within three major areas: program effectiveness, program efficiency, and program costs. A program is referring to the instructional intervention that has been designed and implemented. Evaluating program effectiveness is the major issue that is dealt with by a summative evaluation. The program effectiveness helps answer this ques- tion: “How successful have learners been in meeting the learning objectives?” Specifically, what is being determined is whether change has taken place with the learners. For exam- ple, are employees able to do their jobs more efficiently and effectively after participating in an instructional intervention?

Morrison et al. (2007) list the major steps of a summative evaluation process as:

1 Specifying program objectives: Revisit the instructional goals and objectives of the instructional intervention that was developed.

2 Determining the evaluation design for each objective: How will data be collected that will help determine if the learning goals and objectives have been met? Determine what types of data are needed.

3 Developing data collection instruments and procedures for each objective: Appropriate data collection instruments and procedures were discussed earlier in this chapter. Pretests, post-tests, questionnaires, and observations are all examples of data collec- tion instruments or procedures.

4 Carrying out the evaluation: It is advised that data are collected from the beginning stages of the project. This will ensure that the necessary data are collected, especially data regarding costs and time involvement. Data collection may need to be scheduled.

5 Analyzing the results from each instrument. 6 Interpreting the results. 7 Disseminating the results and conclusions: Develop a summative evaluation report

(refer to the previous section on Smith and Ragan (1999) to see how an evaluation report can be formatted). Individual discussions and group presentations are often useful (and required by the client) to disseminate evaluation findings.

Dick et al. (2011) advocate a very different approach from the two previous approaches we described. Despite the different look and how it is conducted, the fundamental goal is the same as the other two approaches: to determine the effectiveness of the instructional intervention. Dick et al. (2011) define summative evaluation as “the design of evaluation studies and the collection of data to verify the effectiveness of instructional materials with target learners” (p. 320). Its major purpose is to decide whether currently used instruc- tional materials should continue to be used or whether new instructional materials need to be adopted that have the potential to meet the instructional needs of an organization.

They state that a summative evaluation has two main phases: an expert judgment and a field trial. The purpose of the expert judgment phase is to determine whether instruction that is currently being used or instruction that is being considered for use has the potential to meet the instructional needs of an organization. The field-trial phase has the purpose of documenting the effectiveness of “promising instruction” given to learners from the intended target group given in the actual setting. There is a series of steps that takes place during each phase that helps reach these purposes.

Professionals in Practice

Once I have built all of my instructional media and developed the course, the implementa- tion piece is simply to run it. According to the ADDIE model, evaluation comes at the end.

01/27/2020 - RS0000000000000000000002052606 (May Katitar-Thompson) - The Essentials of Instructional Design

Determining the Success of the ID Product 175

I disagree. For me the evaluation piece is constant and necessary. Your users are always very quick at letting you know what is working and what is not. You may find that you need to make changes to both content and design long before any assessment is given or the course is completed. This is a good thing! By making your evaluation constant, you are making your course instructionally sound and solid now and not later, giving current users the best experience.

Kara Andrew Manager, Instructional Design University Extended Education

California State University, Fullerton

Designing and Conducting a Summative Evaluation

In an ideal situation, the instructional designer would be directly involved in designing the summative evaluation but would not be responsible for conducting it. The evaluation would be conducted by an evaluator outside of the organization that contracted the ID project. Having the instructional designer responsible for designing and conducting the summative evaluation is often seen as a conflict of interest because the instructional designer is biased and has a tremendous stake in the project’s success. However, this is not to imply that an instructional designer cannot be unbiased; it only complicates the situation.

As we mentioned, conducting a successful summative evaluation is a complex process that takes a great deal of skill and experience. If at all possible, we advise that begin- ning instructional designers do not take on the role of primary evaluator for a summative evaluation. However, the reality is that, as an instructional designer, you will be required to conduct a summative evaluation. If this is the case, it is important to plan for a summative evaluation at the beginning stages of an ID project. Discuss with the client what the expec- tations are for the summative evaluation. Especially important to discuss are the summative evaluation goals. Although these may change once the summative evaluation is formally developed, the goals will help an instructional designer know what the client expectations are for the ID project. In addition, planning in advance will allow for data to be gathered through the formative evaluation that can be used in the summative evaluation.

The evaluation goals will dictate the remaining steps that are to be taken in the summative evaluation. A process, such as the one advocated by Smith and Ragan, can be used as a guiding model for a beginning instructional designer. As with successful ID projects, continual and consistent communication with the client is a key element in determining the level of success an instructional designer will have in conducting a summative evaluation.

Group Processing: Evaluating the Instructional Design Team

It is rare for an instructional designer to work alone. Most projects require that instruc- tional designers work in teams that often consist of individuals with various roles, such as subject matter expert, programmer, graphic artist, content editor, web developer, or media developer—to name a few. In addition to working with these individuals on a team, the instructional designer is often called on to be the project manager and manage the team. Being the project manager requires that the instructional designer is able to manage all aspects of the ID process, including how the team works together.

01/27/2020 - RS0000000000000000000002052606 (May Katitar-Thompson) - The Essentials of Instructional Design

176 Determining the Effect of the Intervention

Although project management is formally dealt with in Chapter 11, one aspect we will focus on now deals with evaluating how well a team is working together. A technique that can be used to accomplish this is group processing. Group processing can be considered a type of formative evaluation. Group processing refers to the reflections that group mem- bers have on their work and interactions with team members. The reflections help the team focus on improving the working relationship of the group to ensure that the team effectively meets project goals. It allows team members to voice concerns, share successes, and provide feedback that will help the team to complete projects successfully.

Johnson, Johnson, and Holubec (1998) have written much about group processing. Although their work focuses on teams in a classroom environment, what they have writ- ten is appropriate for instructional design teams. We have often used group processing in the instructional design projects we have worked on. It has allowed us to form positive working relationships within our teams. According to Johnson et al. (1998), group processing helps:

• improve the quality of how the team approaches tasks; • increase individual accountability of team members by focusing attention on the tasks

a team member must complete; • the team to learn from each other by spending time discussing the tasks individuals are

completing; • eliminate problems that may be occurring.

Group processing is a time when team members can share information. Four different parts of processing should be included: feedback, reflection, goal improvement, and celebration. Feedback should be given to team members about how well they are com- pleting their tasks and how well the team is working together. Team members should then have time to reflect on the feedback they have received. If improvement needs to be made on how the team is working together, then the team should discuss ways to improve and come up with a plan of action. Additionally, the team members should help each other and set goals for improving the quality of their work. Finally, the team should cel- ebrate the hard work of the team members and the team’s success (Johnson et al., 1998).

Tobias decided to approach group processing with his team by using a multipronged approach. The initial phase was carried out during a weekly team meeting, where Tobias described the situation as he currently saw it. He allowed team members to voice concerns about how the project was progressing and how the team was work- ing together. The second phase consisted of individual meetings with team members, where Tobias provided direct feedback to each member on his or her work and the contributions each was making. During these meetings, Tobias clarified team member duties and expectations. The third phase took place at a team meeting. He clarified to the group the various team member roles and expectations. Tobias also provided a time for praises and wishes. The praises included successes of the team and team members. The wishes consisted of constructive criticism of how the team needed to progress and what was expected for the team during the upcoming week.

Elizabeth settled on Kirkpatrick’s Four Levels of Evaluation as a framework to guide her evaluation. This framework would allow Elizabeth to gather data on multiple levels to provide a holistic picture of how effective her workshops were in influencing participants to use the synchronous tools in their courses. Elizabeth hoped to answer the following questions as part of the evaluation:

01/27/2020 - RS0000000000000000000002052606 (May Katitar-Thompson) - The Essentials of Instructional Design

Determining the Success of the ID Product 177

• Did the workshops Elizabeth conducted provide faculty with the incentives, knowledge, and skills faculty needed?

• Did faculty members take what they learned and apply it to their own courses? • If so, did the use of these tools positively influence the teaching and learning pro-

cesses in the courses?

The data gathered from the evaluation will allow Elizabeth to determine the success of her workshops. She will be able to use the data to improve her workshops.

Product and Process Evaluation and the Instructional Design Process

Determining the success of the instructional design product and process is a significant task that can help lead to an improved ID product and process. Although there are various approaches to conducting an evaluation, the questions an instructional designer will need to answer before beginning an evaluation are the same. These are:

• What is the goal of the evaluation? • Who is asking for the evaluation? • What type of evaluation is needed? • When does the evaluation need to take place? • What are the questions that need to be answered? • What types of data need to be gathered? • Do the data currently exist? • What instruments are needed to gather the data? • How will the data be gathered? • How will the data be analyzed? • How will the data be reported? • How will the results be used to improve the instructional design product or process

or both?

Summary

Sustained and varied evaluation is a crucial element of the instructional design process. Evaluation provides information that allows decisions to be made about learners, instruc- tion, and the ID process. Formative, summative, and learner evaluations are three types of evaluation that instructional designers commonly employ.

Formative evaluation is used throughout the ID process to gather data that can be used to provide feedback on how the process is going. Typically, formative evaluation is used to provide feedback on the instruction being developed. Feedback obtained during a formative evaluation can help instructional designers make changes to the instruction that makes it more efficient and effective. Formative evaluation should be built into an ID project from the beginning stages.

Although ID experts conduct formative evaluation for a common reason, the approaches they take are different. We described three approaches to formative evaluation. Dick et al. (2011) advocate three basic phases of formative evaluation: (1) one-on-one or clinical, (2) small group, and (3) field trial. Smith and Ragan (1999) provide a four-step approach to formative evaluation: design reviews, expert reviews, learner validation, and ongoing eval- uation. Morrison et al. (2007) recommend Gooler’s (1980) eight-step process: purpose, audience, issues, resources, evidence, data gathering techniques, analysis, and reporting.

01/27/2020 - RS0000000000000000000002052606 (May Katitar-Thompson) - The Essentials of Instructional Design

178 Determining the Effect of the Intervention

Summative evaluation takes place after an instructional intervention has been imple- mented. The major goal of a summative evaluation is to gather data that allow for its effectiveness to be determined. Two main questions are to be answered: “Did the instruc- tion bring about the desired changes?” and “Were the client’s goals met?”

It is often considered a conflict of interest to have the instructional designer conduct a summative evaluation on an ID project he or she has worked on. If at all possible, an instructional designer should not be the primary evaluator. He or she should act in a sup- porting role by helping design the evaluation and provide necessary data. However, reality dictates that instructional designers are asked to conduct summative evaluations on proj- ects they have worked on. To ensure success of a formative evaluation, the instructional designer should meet with the client early on in the ID project to discuss expectations of the evaluation, especially the evaluation goals. A process such as the one advocated by Smith and Ragan is a good model to be used by a beginning instructional designer.

An additional type of formative evaluation called “group processing” should be included as part of the instructional design process. Group processing allows for instructional design team members to regularly reflect on and communicate how well the team is working to meet project goals. It consists of four areas: feedback, reflection, goal improvement, and celebra- tion. Group processing helps to build healthy and productive instructional design teams.

Connecting Process to Practice Activities

1 How would you describe to your client the differences between learner, formative, and summative evaluations?

2 What decisions are being made with each type of evaluation? 3 What are the major issues associated with conducting an evaluation on an instruc-

tional design product? 4 What are the major issues associated with conducting an evaluation on the instruc-

tional design process that an instructional design team has gone through (or is currently going through)?

5 Consider Elizabeth’s scenario. What issues do you see with the approach she advo- cated? Explain.

6 Could Tobias have taken a different approach to examining the situation with his instructional design team? Explain the approach.

7 Your client asks you to justify why you have suggested multiple evaluations through- out the instructional design process. What do you tell your client?

8 You have been contracted by a large law firm (approximately 50 employees) to design its intranet. The firm would like the following initial functionalities built into the intranet: client billing, document uploading and retrieval, and a calendar of firm events (e.g., court and deposition dates and times, scheduled vacations). As part of the contract, you have been asked to provide training to the paralegals and legal sec- retaries working at the firm. What type of evaluation(s) do you include in your plan? What do you anticipate using, and how do you communicate this to your client?

9 You are the lead instructional designer on a project team that has six members. Why is it important to include group processing as part of this instructional design project? Describe how you would incorporate group processing.