Evaluation

CHAPTER GOAL The purpose of this chapter is to present approaches for evaluation of distance education courses, programs, and systems.

CHAPTER OBJECTIVES After reading and reviewing this chapter, you should be able to

1. Differentiate between research and evaluation.

2. Define evaluation. 3. Explain the six categories of evaluation

information: in measures of activity, efficiency, outcomes, program aims, policy, and organizations.

4. Describe the AEIOU approach to evaluation and its five levels—accountability, effectiveness, impact, organizational context, and unanticipated consequences.

Evaluating Teaching and Learning at a Distance

347

CHAPTER12

RESEARCH AND EVALUATION

The best way to find things out is not to ask questions at all. If you fire off a question, it is like firing off a gun—bang it goes, and everything takes flight and runs for shelter. But if you sit quite still and pretend not to be looking, all the little facts will come and peck around your feet, situations will venture forth from thickets, and intentions will creep out and sun themselves on a stone; and if you are very patient, you will see and understand a great deal more than a person with a gun does. (Huxley, 1982, p. 20)

This marvelous quote from Huxley’s The Flame Trees of Thika illustrates a metaphorical rationale for a major refocusing of procedures for evaluation of distance education systems. Traditional evaluation models have concentrated on the empirical and

C ap

ti on

T o

C om

e

M12_SIMO7313_05_SE_C12.QXD 11/16/10 3:36 PM Page 347

348 PART 3 ■ MANAGING AND EVALUATING DISTANCE EDUCATION

quantitative procedures that have been practiced for decades (Fitzpatrick, Sanders, & Worthen, 2004; Stufflebeam & Shinkfield, 2007). More recently, evaluators of distance education programs have begun to propose more qualitative models that include the collection of many non-numerical types of information (Rovai, 2003; Sherry, 2003).

Because it is easy to think of them as being the same thing, it is important to differentiate between theory-based research and evaluation. Hanson, et al. (1996) and Simonson (2002) have provided reviews of distance education literature including research on and about distance education. Hanson et al. summarized distance educa- tion research as follows:

■ Distance education is just as effective as traditional education in regard to learner outcomes.

■ Distance education learners generally have more favorable attitudes toward distance education than traditional learners, and distance learners feel they learn as well as nondistant students.

■ The research clearly shows that distance education is an effective method for teaching and learning (p.1).

Evaluation, as contrasted to research, is the systematic investigation of the worth or merit of an object. Program evaluation is the systematic investigation of the worth of an ongoing or continuing distance education activity (Joint Committee on Standards for Educational Evaluation, 1994). Martinez, Liu, Watson, and Bichelmeyer (2006) discuss the importance of evaluating distance education programs. Evaluation of pro- grams is used to identify strengths and weaknesses as well as the benefits and draw- backs of teaching and learning online. They asked students, administrators, and instructors to evaluate course management categories, such as registration, support services, advising, and sense of community. One important finding of this study was the equivalence of the distance education program to the traditional program (Martinez et al., 2006).

This chapter focuses on approaches to evaluation for the purpose of improving distance education and determining the worth of distance education activities. Rose (2000) identified a number of databases related to evaluation of distance education courses that are available on the Internet. These online databases provide a repository of up-to-date information about online courses. Additional information related to evaluation and distance education is available in Cyrs and Smith (1990), Fitz-Gibbon and Morris (1987), Fitzpatrick et al. (2004), Rossi, Lipsey, and Freeman (2003), Ruhe and Zumbo (2009), Thompson and Irele (2007), and Willis (1994).

EVALUATION AND DISTANCE EDUCATION—FIVE STEPS

Evaluation procedures are becoming of critical interest to trainers and teachers who are adopting e-learning or distance education (Peak & Berge, 2006). As new distance education systems are being planned and implemented there is considerable concern that the time and effort required to move to distance delivery of instruction produced a valuable educational experience, thus, evaluation is regularly a part of plans to move from traditional face-to-face instruction to distance education. Kirkpatrick’s (2006)

M12_SIMO7313_05_SE_C12.QXD 11/16/10 3:36 PM Page 348

CHAPTER 12 ■ EVALUATING TEACHING AND LEARNING AT A DISTANCE 349

evaluation approach with its four levels of evaluation as shown below, supplemented by Phillips’s (2003) fifth evaluation level—return on investment (ROI)—seems to be the preferred approach of many trainers, and some educators.

Kirkpatrick’s evaluation approach has been traditionally used to evaluate classroom training and teaching, especially in the private, government, and military sectors. It is a straightforward approach that produces usable information for the trainer. The four levels of the approach are designed to obtain answers to commonly asked questions about training: Did they like it? Did they learn it? Will they use it? Will it matter? (Simonson, 2007).

Level 1—Reactions (Did they like it?)

As the word reactions implies, evaluation at this level measures how participants in the training program feel about the educational activity. Students are asked what they liked and did not like about training, sometimes several times during a course or program. Students are required to use checklists, Likert responses (selecting from one of several options, such as highly agree, agree, disagree and strongly disagree) to statements, and open-ended comments, all to determine if the training was perceived positively by participants.

Level 2—Learning (Did they learn it?)

At this level, evaluation strategies attempt to determine more than learner satisfaction. Rather, evaluators assess the extent to which learners have advanced in skills, knowledge, or attitude. What and how much did participants learn? What new skills do they possess? And, what new and appropriate attitudinal positions have been produced?

Methods include objective testing, team assessment, and self-assessment. Often pretest– posttest change is used as a measure at level 2.

Level 3—Transfer (Will they use it?)

At this level, evaluators attempt to determine if the skills, knowledge, and attitudes learned as a result of training are being transferred to the workplace or to actual learner activities. Evaluation questions deal with the use of new skills or the application of new knowledge to events. Timing of the evaluation at this level is critical and problematic since it is difficult to know when transfer actually occurs.

Level 4—Results (Will it matter?)

Evaluation activities at this level attempt to measure the success of the training or teach- ing program in terms of increased productivity, improved quality, lower costs, and, for businesses, even higher profits. Trainers are increasingly being asked to demonstrate the direct and indirect impact of training on the success of the organization and to relate training to mission accomplishment. In schools, level 4 evaluations often look at enrollments in additional courses, learning motivation, and educational achievement.

M12_SIMO7313_05_SE_C12.QXD 11/16/10 3:36 PM Page 349

350 PART 3 ■ MANAGING AND EVALUATING DISTANCE EDUCATION

Level 5—Return on Investment

Increasingly, many training and educational organizations that are adopting e-learning and distance education are interested in the concept of return on invest- ment (ROI)—converting training results from e-learning activities into monetary values and comparing these costs to the cost of the training program to determine a return on investment. Phillips (1996) describes a five-step process to determine return on investment.

1. First, it is necessary to collect level-4 data to determine if there is a change in job or educational performance that is positive and also measurable. This assumes that there were evaluation data collected concerning the first four levels of the Kirkpatrick model.

2. Second, evaluators need to identify the training that contributed to the change in performance. Testing can be used, as can control groups that receive different training or no training at all.

3. Third, it is necessary to convert the results of training or education into monetary values. This often means a relatively subjective process must be undertaken to quantify outcomes related to the training.

4. Next, the evaluation process requires the determination of the total cost of training. This includes trainer costs, facilities expenses, materials purchased, and other expenses.

5. Fifth, return on investment is determined by comparing the monetary benefits to the costs. In this manner, it is possible to quantify the impact of training, the effectiveness of education and the value of the instruction.

The ROI process is time consuming, requires a skilled evaluation team, and is sometimes criticized because it produces evaluation results that look at what has happened rather than what will happen. Peak and Berge (2006) also recommend that not everything needs to be measured. Rather, leaders should determine what they think is important and then trainers can evaluate those areas.

Although evaluation has always been somewhat important in corporate and military training and of interest to a lesser extent in education, the recent phenomenal growth of e-learning and distance education has made many leaders want to know what the implications are of moving to training and teaching that is not face-to-face. Thus, Kirkpatrick’s and Phillips’s evaluation approaches have received increased attention, especially since most evidence clearly demonstrates distance education works academically to produce required achievement gains. The evidence is clear that students learn just as effectively when they are taught at a distance as compared to when they learn in a traditional classroom (Simonson, 2002). Thus, it can be general- ized that traditional training and e-learning work equally well. The question for evaluators then becomes the determination of the advantages, if any, of moving to an e-learning environment. Evaluators are looking at cost savings, time savings, increased motivation and satisfaction, economies of scale, and other non-achievement outcome metrics. Evaluation of e-learning should provide leaders evidence they need to support or to refute training decisions.

M12_SIMO7313_05_SE_C12.QXD 11/16/10 3:36 PM Page 350

CHAPTER 12 ■ EVALUATING TEACHING AND LEARNING AT A DISTANCE 351

EVALUATION AND THE OPEN UNIVERSITY

Program evaluation at the Open University of Great Britain is the systematic investigation of the merit of a particular distance education program, curriculum, or teaching method, and how it might be improved compared with alternatives. As part of the evaluation pro- cedures for distance education by the Open University (Woodley & Kirkwood, 1986), two alternative strategies have been merged. The first is the traditional, positivist–empiricist approach to evaluation. This represents an attempt to apply the rules and procedures of the physical sciences to evaluation. The second is a more eclectic view of evaluation that incor- porates qualitative and naturalistic techniques for the evaluation of distance education.

The traditional strategy normally includes an experiment that determines the effectiveness of a distance education strategy. The distance education project is structured from its beginning with the requirements of the evaluator in mind. Carefully matched samples are picked, controls are established, and variables are selected for which comparison data will be collected. Next, objective tests of variables are selected or constructed. Data are collected before, during, and always after the instructional event or procedures. Then the evaluator takes the data and prepares the evaluation report, which is submitted weeks or months later.

The primary outcome of this type of evaluation is the comparison of the data collected from the two or more categories of learners. For example, the distant learners are compared with those taught locally, and conclusions about the effectiveness of the distance education activity are made.

This approach represents the traditional process for the evaluation of distance education. Recently at the Open University and elsewhere, a countermovement has emerged (House, 2010). Advocates of this counterapproach are united in one primary way: They are opposed to the traditional, quantitative procedures for evaluation. Increasingly, evaluation activities are incorporating more naturalistic methodologies with holistic perspectives. This second perspective for evaluation uses focus groups, interviews, observations, and journals to collect evaluation information in order to obtain a rich and colorful understanding of events related to the distance education activity.

From a practical standpoint, most evaluators now use a combination of quantitative and qualitative measures. Certainly, there is a need to quantify and count. Just as certainly, there is a need to understand opinions and hear perspectives.

According to Woodley and Kirkwood (1986), six categories of evaluation information can be collected about distance education activities:

1. Measures of activity. These measures are counts of the numbers of events, people, and objects. Administrative records often provide data for activity questions. Activity questions are ones such as: ■ How many courses were produced? ■ How many students were served? ■ How many potential students were turned away?

2. Measures of efficiency. Measures of efficiency are closely related to measures of activity, and often administrative records can be the source of efficiency information. Efficiency questions often asked are ones such as: ■ How many students successfully completed the course? ■ What was the average student’s workload?

M12_SIMO7313_05_SE_C12.QXD 11/16/10 3:36 PM Page 351

352 PART 3 ■ MANAGING AND EVALUATING DISTANCE EDUCATION

■ How many students enrolled in additional courses? ■ How much did the course cost? ■ How much tuition was generated?

3. Measures of outcomes. Measures of adequate learning are usually considered the most important measures of outcomes of distance education activities. Often, interviews with learners are used to supplement course grades in order to find students’ perceptions about a distance education activity. Mail surveys are also efficient ways to collect outcome information from distant learners. Other outcome measures include documenting the borrowing and use of courses and course materials by other institutions as an indicator of effectiveness, and the enrollment by students in additional, similar courses as indicators of a preliminary course’s success.

4. Measures of program aims. Some distance teaching programs specify their aims in terms of what and whom they intend to teach, and evaluation information is collected to establish the extent to which these aims were met. One common aim of distance education programs is to reach learners who otherwise would not be students. Surveys of learners can be used to collect this type of information.

5. Measures of policy. Evaluation in the policy area often takes the form of market research. Surveys of prospective students and employers can be used to determine the demand for distance education activities.

Policy evaluation can also include monitoring. Students can be surveyed to determine if tuition is too high, if appropriate courses are being offered, and if there are impediments to course success, such as the lack of access to computers or the library.

Sometimes policy evaluation can be used to determine the success of experimental programs, such as those for low achievers or for students who normally are not qualified for a program. The purpose of policy evaluation is to identify proce- dures that are needed or that need to be changed, and to develop new policies.

6. Measures of organizations. Sometimes it is important to evaluate a distance education institution in terms of its internal organization and procedures. Evaluators sometimes are asked to monitor the process of course development or program delivery to help an organization be more efficient. This category of evaluation requires on-site visits, interviews, and sometimes the use of journals by key organization leaders.

These six categories of evaluation are not used for every distance education activity. Certainly, some modest evaluation activity is almost always necessary. It is important that the activities of evaluators be matched to programmatic needs. Woodley and Kirkwood (1986) have summarized evaluation in distance education as being a fairly eclectic process that utilizes procedures that should match program needs to evaluation activities.

THE AEIOU APPROACH

Fortune and Keith (1992) proposed the AEIOU approach for program evaluation, especially the evaluation of distance education projects. The effectiveness of this approach has been demonstrated through evaluating the activities of the Iowa Distance Education Alliance Star Schools Project (Simonson & Schlosser, 1995a;

M12_SIMO7313_05_SE_C12.QXD 11/16/10 3:36 PM Page 352

CHAPTER 12 ■ EVALUATING TEACHING AND LEARNING AT A DISTANCE 353

Sorensen, 1996, Sorensen & Sweeney, 1995, 1996, 1997; Sweeney, 1995), a multiyear, statewide distance education activity. Additionally, the model has been used to evaluate a number of other innovative projects, such as the Iowa Chemistry Education Alliance in 1995, the Iowa General Chemistry Network in 1994, and the DaVinci Project: Interactive Multimedia for Art and Chemistry (Simonson & Schlosser, 1995b). More recently, a major distance education initiative in South Dakota used a modified version of the AEIOU approach (Simonson, 2005).

The AEIOU approach is similar to Woodley and Kirkwood’s in that it is an eclectic one that uses quantitative and qualitative methodologies. It has two primary purposes as an evaluation strategy. First, the model provides formative information to the staff about the implementation of their project. Second, it provides summative information about the value of the project and its activities. The AEIOU evaluation process provides a framework for identifying key questions necessary for effective evaluation. Some evaluation plans use only parts of the framework, whereas other, more compre- hensive plans use all components. Some examples of evaluation questions asked in comprehensive distance education projects are presented next.

Component 1—Accountability (A) Did the project planners do what they said they were going to do? This is the first step in determining the effectiveness of the project or course and is targeted at deter- mining if the project’s objectives and activities were completed. Evaluation questions typically center on the completion of a specific activity and often are answered “yes” or “no.”Additionally, counts of numbers of people, things, and activities are often collected.

Questions such as the following are often asked to determine project accountability:

■ Were the appropriate number of class sessions held? ■ How many students were enrolled? ■ How many copies of program materials were produced, and how many were

distributed?

Methods Used: Accountability information is often collected from project administra- tive records. Project leaders are often asked to provide documentation of the level of completion of each of the project’s goals, objectives, and activities. Sometimes evaluators interview project staff to collect accountability data.

Component 2—Effectiveness (E) How well done was the project? This component of the evaluation process attempts to place some value on the project’s activities. Effectiveness questions often focus on participant attitudes and knowledge. Obviously, grades, achievement tests, and attitude inventories are measures of effectiveness. Less obvious are other ways to determine quality. Often, raters are asked to review course materials and course presentations to determine their effectiveness, and student course evaluations can be used to collect reactions from distance education participants.

M12_SIMO7313_05_SE_C12.QXD 11/16/10 3:36 PM Page 353

354 PART 3 ■ MANAGING AND EVALUATING DISTANCE EDUCATION

Examples of questions to determine effectiveness include:

■ Were the in-service participants satisfied with their distance education instruction? ■ Did the students learn what they were supposed to learn? ■ Did the teachers feel adequately prepared to teach distant learners?

Methods Used: Standardized measures of achievement and attitude are traditionally used to determine program effectiveness. Surveys of students and faculty can be used to ask questions related to perceptions about the appropriateness of a project or program. Focus groups (Morgan, 1996) also provide valuable information. Participants are systematically asked to respond to questions about the program. Finally, journals are sometimes kept by project participants and then analyzed to determine the day-to-day effectiveness of an ongoing program.

Component 3—Impact (I) Did the project, course, or program make a difference? During this phase of the evaluation, questions focus on identifying the changes that resulted from the project’s activities, and are tied to the stated outcomes of the project or course. In other words, if the project had not happened, what incident of importance would not have occurred? A key element of measurement of impact is the collection of longitudinal data. The impact of distance education courses is often determined by following learn- ers’ progress in subsequent courses or in the workplace to determine if what was learned in the distance education course was useful.

Determinants of impact are difficult to identify. Often, evaluators use follow-up studies to determine the impressions made on project participants; and sometimes in distance education programs, learners are followed and questioned by evaluators in subsequent courses and activities. Questions might include:

■ Did students register for additional distance education courses? ■ Has the use of the distance education system increased? ■ Have policies and procedures related to the use of the distance education system

been developed or changed?

Methods Used: Qualitative measures provide the most information to the evaluator interested in program impact. Standardized tests, record data, and surveys are sometimes used. Also, interviews, focus groups, and direct observations are used to identify a program’s impact.

Component 4—Organizational Context (O) What structures, policies, or events in the organization or environment helped

or hindered the project in accomplishing its goals? This component of evaluation has traditionally not been important even though evaluators have often hinted in their reports about organizational policies that either hindered or helped a program. Recently,

M12_SIMO7313_05_SE_C12.QXD 11/16/10 3:36 PM Page 354

CHAPTER 12 ■ EVALUATING TEACHING AND LEARNING AT A DISTANCE 355

however, distance educators have become very interested in organizational policy analysis in order to determine barriers to the successful implementation of distance education systems, especially when those systems are new activities of traditional educational organi- zations such as large public universities.

The focus of this component of the evaluation is on identifying those contex- tual or environmental factors that contributed to, or detracted from, the project or course’s ability to conduct activities. Usually these factors are beyond the control of the project’s participants. Effective evaluation of organizational context requires the evaluator to be intimately involved with the project or course in order to have a good understanding of the environment in which the project or course operates. Questions typically addressed in evaluating organizational context include:

■ What factors made it difficult to implement the project or to successfully complete the course?

■ What contributed most to the success or failure of the project or the students in the course?

■ What should be done differently to improve things and make the course more effective?

Methods Used: Organizational context evaluation uses interviews of key personnel such as faculty or students, focus groups made up of those impacted by a program, and document analysis that identifies policies and procedures that influence a program or course. Direct participation in program activities by the evaluator is also important. Sometimes evaluators enroll in distance education courses. More often, a student is asked to complete a journal while enrolled in a course. By participating, the evaluator is confronted directly with the organizational context in which a program exists, and can comment on this context firsthand.

Component 5—Unanticipated Consequences (U) What changes or consequences of importance happened as a result of the

project that were not expected? This component of the AEIOU approach is to identify unexpected changes of either a positive or negative nature that occurred as a direct or indirect result of the project or course. Effective evaluators have long been interested in reporting anecdotal information about the project or program that they were evaluating. It is only recently that this category of information has become recognized as important, largely because of the positive influence on evalu- ation of qualitative procedures. Often, evaluators, especially internal evaluators who are actively involved in the project or course’s implementation, have many opportunities to observe successes and failures during the trial-and-error process of beginning a new program. Unanticipated consequences of developing new or modified programs, especially in the dynamic field of distance education, are a rich source of information about why some projects are successful and others are not. Central to the measurement of unanticipated outcomes is the collection of ex post facto data.

M12_SIMO7313_05_SE_C12.QXD 11/16/10 3:36 PM Page 355

356 PART 3 ■ MANAGING AND EVALUATING DISTANCE EDUCATION

Examples of questions asked include:

■ Have relationships between collaborators or students changed in ways not expected?

■ Have related, complementary projects been developed? ■ Were unexpected linkages developed between groups or participants? ■ Was the distance education system used in unanticipated ways? ■ Did the distance education system have an impact on student learning other than

that expected?

Methods Used: Interviews, focus groups, journals, and surveys that ask for narrative information can be used to identify interesting and potentially important consequences of implementing a new program. Often, evaluators must interact with project partici- pants or course students on a regular basis to learn about the little successes and failures that less sensitive procedures overlook. Active and continuous involvement by evaluators permits them to learn about the project as it occurs.

Sweeney (1995) advocates an eclectic approach to evaluation, an approach also supported by Fitzpatrick et al. (2004). The AEIOU model is a dynamic one that permits the evaluator to tailor the process of program evaluation to the specific situation being studied.

PROGRAM EVALUATION: EXAMPLES

South Dakota

South Dakota has a network for distance education that connects every school in the state. Currently, hundreds of classrooms are connected to the Digital Dakota Network (DDN). The DDN was funded using state monies and grants from telecommunications providers, such as QWEST Communications.

Implementation of the DDN was called the Connecting the Schools project. As the network came online and began to be used, it was decided that a comprehensive evalu- ation effort was needed. Evaluators used the AEIOU approach and collected both quantitative and qualitative information (Simonson, 2005; Simonson & Bauck, 2001).

Quantitative information was collected using a locally developed survey called the Connecting the Schools Questionnaire (CSQ). The CSQ asked respondents to provide four categories of information: demographics, information about personal innovativeness, questions about organizational innovativeness, and questions about distance education.

Demographic information was collected in order to obtain a profile of the teachers in the state, and included questions about age, years of experience, gender, academic background, and professional involvement. The second part of the CSQ was a modified version of Hurt, Joseph, and Cook’s (1977) innovativeness scale (IS). The IS is a standardized measure of how innovative a person thinks he or she is. Part three of the CSQ was a modified version of Hurt and Tiegen’s (1977) Perceived Organizational Innovativeness Scale (PORGI). The PORGI is a standardized measure of a person’s perception of his or her employer’s organizational innovativeness. The final section of the CSQ asked questions about distance education. These questions were to find out

M12_SIMO7313_05_SE_C12.QXD 11/16/10 3:36 PM Page 356

CHAPTER 12 ■ EVALUATING TEACHING AND LEARNING AT A DISTANCE 357

how much South Dakota teachers knew about distance education and to determine their general feelings about the impact of distance education on teaching and learning.

The qualitative portion of the CSQ evaluation in South Dakota used focus groups, participant observations, interviews, and site visits. Three questions were at the heart of the quantitative evaluation. First, evaluators tried to determine what educators thought would be the greatest benefits provided by implementing distance education. Second, attempts were made to determine what was preventing individuals from becoming involved in distance education. Next, school superintendents were selected randomly and interviewed to determine their perceptions of the impact of distance education and the Digital Dakota Network on education in their school districts (Calderone, 2003). Finally, questions were asked about the impediments to distance education.

When quantitative data were combined with qualitative information, a rich understanding was provided to education leaders about South Dakota’s ability to adopt distance education (Learning at a Distance: South Dakota, www.tresystems.com/projects/). Complete results of the evaluation were reported in Simonson (2005). In general, the evalu- ation of the South Dakota project verified that Rogers’s (2003) theory concerning the diffu- sion of innovations was directly applicable to distance education efforts in South Dakota and that this theory could effectively serve as a model for promoting the adoption of innova- tions, such as the DDN specifically, and distance education in public schools, more generally.

Iowa

Several years ago, it was decided that a three-phase plan should be implemented to establish distance education classrooms throughout the state of Iowa. For the first phase, 15 area community colleges, 3 public universities, and Iowa Public Television had classrooms built and connected with fiber-optic cables capable of carrying 48 full- motion video signals in addition to virtually unlimited voice and data information. The second phase of the plan connected a classroom site in any of Iowa’s 99 counties that was not already served by a community college site. These classrooms were connected with 12 fiber-optic cables. A total of 103 sites were built and connected as part of phases 1 and 2. During phase 3, additional sites were connected throughout the state. Recently, over 750 sites were connected to this distance education infrastructure, which was named the Iowa Communications Network (ICN).

As part of the implementation plan for the ICN, a comprehensive evaluation program was put into action. This program utilized the AEIOU approach and collected data from thousands of sources and individuals. The evaluation approach went through several stages during the five years it was used. First, evaluators concen- trated on evaluating the construction, connection, and implementation of the ICN’s physical infrastructure. Records related to classroom design, construction schedules, and dollars spent were collected and reviewed, and summary results were reported. This related to the accountability component of the AEIOU approach.

Next, those involved in the decision-making process for establishing the network were interviewed and completed surveys. Evaluators used the results to develop reports on the effectiveness of the processes used to construct the ICN. To determine impact, evaluators conducted follow-up investigations of classroom utilization and examined records of how the system was used.

M12_SIMO7313_05_SE_C12.QXD 11/16/10 3:36 PM Page 357

358 PART 3 ■ MANAGING AND EVALUATING DISTANCE EDUCATION

The program evaluators examined many interesting organizational issues, such as who made decisions about where classrooms were located, how funds were obtained and spent, and who controlled access to the system. Finally, program evaluators identified unantici- pated outcomes. One of the most significant was the infusion of several millions of dollars from federal, state, and local sources to support the development of the network. How these funds were obtained and used added to the importance of the evaluation report.

Once the network was built and a plan for its continued growth was put into place, evaluators shifted their primary focus to the human side of the growth of distance education in the state. Staff development, technical training, curriculum revisions, and school restructuring became the focus of network planners and funding agencies, so program evaluators used the AEIOU model to obtain information about these activities. The approach was used to provide formative information about the develop- ment of programs and their impact on teachers and learners, and also to provide information on outcomes, or summative information, to document the successes and failures of various program activities.

A true understanding of activities of evaluators of this statewide, multiyear project can only be gained by reviewing the yearly reports they submitted. However, it is impor- tant to note that the evaluation plan provided the following information:

Accountability. Evaluators checked records, interviewed staff, and visited classrooms to determine the status of the development of the ICN, both as a physical system and as a tool used by teachers to deliver courses to distant learners. The accountability focus shifted during the project as its activities shifted from construction to implementation and finally to maintenance.

Effectiveness. Evaluators conducted interviews and questioned focus groups to determine what impact the availability of the ICN had on classroom education. Surveys were sent and reports were generated that helped education leaders to better understand what role distance education was playing.

Impact. As the network became widely available and the number of courses and activities increased, it became possible to determine the impact of the ICN and distance education events on education in the state. Students were tested and grades reported. Most of the achievement data showed that learning occurred and good grades were obtained. More importantly, the availability of new learning experiences grew considerably.

Organizational Context. From the beginning of the ICN project, the role of the state as compared with local educational organizations was a focus of evaluation activities. One outcome was to identify where cooperation between agencies was necessary (e.g., as in scheduling) and where local control (e.g., as in course selection) should be maintained. Project evaluators identified and reported on what the data seemed to indicate were the barriers and the contributors to the effective growth and utilization of the ICN.

Unanticipated Outcomes. During the project, scores of unanticipated outcomes were identified and reported. Among the most interesting were:

■ The movement of the ICN into the role of Internet service provider ■ The role of the ICN in attracting external grants

M12_SIMO7313_05_SE_C12.QXD 11/16/10 3:36 PM Page 358

CHAPTER 12 ■ EVALUATING TEACHING AND LEARNING AT A DISTANCE 359

■ The role of distance education and the ICN in the movement to restructure schools

■ The impact of the ICN on positive attitudes toward technology in education ■ The emerging role of the public television station in Iowa education

There were also many other unanticipated outcomes. The AEIOU approach was useful in helping the state’s educators evaluate the role of distance education as an approach and the ICN as an infrastructure. Evaluation played a significant part in the positive implementation and use of this new technology in the state of Iowa.

STUDENT EVALUATION OF DISTANCE

EDUCATION COURSES

The purpose of a course evaluation is to fulfill accreditation requirements and to provide a means for reporting course and instructor effectiveness. Standardized course evaluation forms are available that have already been developed and have gone through rigorous psychometric analyses. The literature suggests course and instructor evalua- tion models that focus on six constructs:

■ Teaching and learning ■ Developing a community of learners ■ The instructor ■ The student ■ Implementation of the course ■ Technology use

Evaluation instruments should possess the psychometric characteristics of standardized measures, meaning they should be valid, reliable, administered in a consistent manner, and have normative tables so scores can be compared.

Valid instruments measure what they are supposed to measure, in this case the effectiveness of online courses and online teaching. Reliable measures are consistent— in other words, if the measure were administered a second time the scores should be very similar. Consistent administration of course evaluations ensures that more or less favorable conditions of testing do not influence the results. Finally, scores for any course evaluation are difficult to decipher if there is no comparison data. Often, scores from evaluations for many courses are collected so that the scores for any individual course and instructor can be compared with others. Usually, any identifiers for com- parison courses are removed. It is important to remember that course and instructor evaluations are to be used for continuous improvement, and to provide input for course revisions.

A sample evaluation instrument to collect students’ perceptions about the six constructs, the Online Course Evaluation Instrument (OCEI, pronounced ooh-see), is shown in Figure 12–1.

M12_SIMO7313_05_SE_C12.QXD 11/16/10 3:36 PM Page 359

360 PART 3 ■ MANAGING AND EVALUATING DISTANCE EDUCATION

ONLINE COURSE EVALUATION INSTRUMENT (OCIE)

Course Name:

Gender: _____ Male

_____ Female

Age: _____ Years

Class Level: _____ Undergraduate

_____ Master

_____ Doctorate

Class Term: _____ Summer

_____ Fall

_____ Winter

Class Size: _____ Class size 1 to 10

_____ Class size 11 to 20

_____ Class size 21 to 30

_____ Class size 31 to 40

_____ Class size 41 and above

First Experience in an Online Course: _____ Yes _____ No

Please rate each item using the following scale:

5 – Strongly agree 4 – Agree 3 – Neither agree nor disagree 2 – Disagree 1 – Strongly disagree

Teaching and Learning

1. The course has clearly stated objectives __________

2. The course activities are consistent with course objectives __________

3. The course syllabus is an accurate guide to course requirements __________

4. The course materials are a helpful guide to key concepts covered in the class ___________

5. The course projects and assignments build understanding of concepts and principles ___________

6. The course presents appropriate skills and techniques ___________

7. The course is current with developments in the field ____________

Developing a Community of Learners

1. Collaborative work is a valuable part of this course ____________

2. There is opportunity to learn from other students _____________

3. Differing viewpoints and discussions are encouraged in this class ____________

4. Mutual respect is a concept practiced in this course ______________

5. Each student has an opportunity to contribute to class learning _____________

The Instructor

1. The instructor clearly states the methods of evaluation that will be used to assess student work ___________

2. The instructor uses a variety of methods to evaluate student progress on class objectives ___________

FIGURE 12–1 An evaluation instrument.

M12_SIMO7313_05_SE_C12.QXD 11/16/10 3:36 PM Page 360

CHAPTER 12 ■ EVALUATING TEACHING AND LEARNING AT A DISTANCE 361

FIGURE 12–1 (continued)

SUMMARY

As distance education in the United States increases in importance, evaluation will continue to be a critical component of the process of improvement. Certainly, the literature is clear. Eclectic models of evaluation such as the ones advocated by Woodley and Kirkwood (1986) and Sweeney (1995) are most applicable to distance education program evaluation. Evaluators should use quantitative and qualitative procedures. Distance education programs and even single courses should be accountable to their goals, should be at least as effective as alternative approaches, and should have a positive impact. Evaluators should attempt, when possible, to identify what organizational contexts support effective distance education systems, and unanticipated events should be shared with interested readers and should be used to improve courses.

If you are very patient, you will see and understand. (Huxley, 1982, p. 20)

REFERENCES

Calderone, T. (2003). Superintendents’ perception of their role in the diffusion of distance education. Unpublished doctoral dissertation. Nova Southeastern University, Fort Lauderdale, FL.

Cyrs, T., & Smith, F. (1990). Teleclass teaching: A resource guide (2nd ed.). Las Cruces, NM: Center for Educational Development.

Fitz-Gibbon, C., & Morris, L. (1987). How to design a program evaluation. Newbury Park, CA: Sage. Fitzpatrick, J., Sanders, J., & Worthen, B. (2004). Program evaluation: Alternative approaches and

practical guidelines (3rd ed.). Upper Saddle River, NJ: Pearson/Allyn and Bacon. Fortune, J., & Keith, P. (1992). Program evaluation for Buchanan County Even Start. Blacksburg, VA:

College of Education, Virginia Polytechnic Institute and State University. Graber, J., Post, G., & Erwin R. (n.d.) Using ROI forecasting to develop a high-impact, high-volume

training curriculum. Retrieved August 20, 2007 from www.businessdecisions.com/docs/ ASTDROIChapter.doc.

Hanson, D., Maushak, N, Schlosser, C., Anderson, M, & Sorensen, M (1997). Distance education: Review of the Literature, 2nd ed. Washington, DC: Assn for Educational Communities and Technology.

House, E. (Ed) (2010), New directions in educational evaluation, Lewes, England: Falmer, Press. Hurt, H., Joseph, K., & Cook, C. (1977). Scales for the measurement of innovativeness. Human

Communications Research, 4(1), 58–65.

3. The instructor shows respect for the various points of view represented in this class __________

4. The instructor makes learning interesting and motivates students to learn ____________

5. The instructor uses technology in ways that help learning of concepts and principles __________

6. The instructor responds to questions with consideration _____________

7. The instructor displays a clear understanding of course topics ____________

M12_SIMO7313_05_SE_C12.QXD 11/16/10 3:36 PM Page 361

362 PART 3 ■ MANAGING AND EVALUATING DISTANCE EDUCATION

Hurt, H., & Teigen, C. (1977). The development of a measure of perceived organizational innovativeness. Communication Yearbook I (pp. 377–385). New Brunswick, NJ: International Communications Association.

Huxley, E. (1982). The flame trees of Thika: Memories of an African childhood. London: Chatto and Windus.

Joint Committee on Standards for Educational Evaluation. (1994). The program evaluation standards (2nd ed.). Thousand Oaks, CA: Sage.

Kirkpatrick, D., & Kirkpatrick, J. (2006). Evaluating training programs: The four levels (3rd ed.). San Francisco, CA: Berrett-Koehler.

Martinez, R., Liu, S., Watson, W., & Bichelmeyer, B. (2006). Evaluation of a web-based master’s degree program: Lessons learned from an online instructional design and technology program. Quarterly Review of Distance Education, 7(3), 267–283.

Morgan, D. (1996). Focus groups as qualitative research. Newbury Park, CA: Sage. Phillips, J. (2003). Return on investment, 2nd ed. Burlington, MA: Butterworth-Heinmann. Peak, D. & Berge, Z (2006). Evaluation and eLearning, Turkish Online Journal of Distance Education,

7(1), Article 11. Phillips, J., Phillips, P., Duresky, L. & Gaudet, C. (2002). Evaluating the return on investment of

elearning. In Alison Rossett (Ed.). The ASTD Elearning Handbook (pp. 387–397). Madison, WI: McGraw-Hill.

Rogers, E. (2003). Diffusion of innovations (5th ed.). New York: Free Press. Rose, E. (2000). An evaluation of online distance education course databases. DEOSNEWS, 10(11),

1–6. Available online at www.ed.psu.edu/acsde/deos/deosnews.deosarchives.asp. Rossi, P., Lipsey, M., & Freeman, H. (2003). Evaluation: A systematic approach (7th ed.). Newbury

Park, CA: Sage. Rovai, A. (2003). A practical framework for evaluting online distance education programs. Internet

and Higher Education, 6(2), 109–124. Ruhe, V., & Zumbo, B. (2009) Evaluation in distance education and e’learning. New York: Guilford. Sherry, A. (2003). Quality and its measurement in distance educaton. In M. Moore & W. Anderson

(Eds.), Handbook of distance education (pp. 435–459). Mahwah, NJ: Erlbaum. Simonson, M. (2002). In case you are asked: The effectiveness of distance education. Quarterly

Review of Distance Education, 3(4), vii. Simonson, M. (2005). South Dakota’s statewide distance education project. In Z. Berge & T. Clark

(Eds.), Virtual schools: Planning for success (pp. 183–197). New York: Teachers College Press. Simonson, M. (2007). Evaluation and distance Education. Quarterly Review of Distance Education,

8(3), vii–ix. Simonson, M., & Bauck, T. (2001). Learning at a distance in South Dakota: Description and evaluation

of the diffusion of an innovation. Proceedings of Research and Development Papers presented at the Annual Convention of the Association for Educational Communications and Technology, Atlanta, GA. (ERIC Document Reproduction Service No. ED47103).

Simonson, M., & Schlosser, C. (1995a). More than fiber: Distance education in Iowa. Tech Trends, 40(3), 13–15.

Simonson, M., & Schlosser, C. (1995b). The DaVinci Project. Iowa Computer-Using Educators Conference, Des Moines.

Sorensen, C. (1996). Final evaluation report: Iowa distance education alliance. Ames, IA: Research Institute for Studies in Education.

Sorensen, C., & Sweeney, J. (1995). ICN Technology Demonstration Evaluation. USDLA Education at a Distance, 9(5), 11, 21.

Sorensen, C., & Sweeney, J. (1996, November). AEIOU: An approach to evaluating the statewide integration of distance education. Paper presented at the annual meeting of the American Evaluation Association, Atlanta, GA.

M12_SIMO7313_05_SE_C12.QXD 11/16/10 3:36 PM Page 362

CHAPTER 12 ■ EVALUATING TEACHING AND LEARNING AT A DISTANCE 363

Sorensen, C., & Sweeney, J. (1997, November). A-E-I-O-U: An inclusive framework for evaluation. Paper presented at the annual meeting of the American Evaluation Association, San Diego, CA.

Strother, J. (2002). An assessment of the effectiveness of e-learning in corporate training programs. International Review of Research in Open and Distance Learning, 3(1), 1–17.

Stufflebeam, D., & Shinkfield, A. (2007). Evaluation theory, models, and applications. Hoboken, NJ: Jossey-Bass.

Sweeney, J. (1995). Vision 2020: Evaluation report. Ames, IA: Research Institute for Studies in Education. Thompson, M., & Irele, M. (2007). Evaluating distance education programs. In M. Moore (Ed.),

Handbook of distance education (2nd ed.). Mahwah, NJ: Erlbaum. Willis, B. (Ed.). (1994). Distance education: Strategies and tools. Englewood Cliffs, NJ: Educational

Technology Publications. Woodley, A., & Kirkwood, A. (1986). Evaluation in distance learning. Paper 10. Resources in

Education. (ERIC Document Reproduction Service No. ED304122).

SUGGESTED READING

Cronbach, L. (1982). Designing evaluations of educational and social programs. San Francisco: Jossey-Bass.

M12_SIMO7313_05_SE_C12.QXD 11/16/10 3:36 PM Page 363