HRDE instructional learning
Evaluation
Chapter 11
Evaluation
Purpose- to determine if instructional objectives/goals were met
Three types of evaluation;
Formative Evaluation – Provides an understanding of how well the instructional program is achieving the objective, therefore it is done during development and tryouts. Feedback from learners can indicate deficiencies in the learning sequence, procedures or materials. Asks “How are we doing?”
Summative Evaluation- Measures the degree to which the major outcomes were attained at the end of the course. Were the instructional objectives taught and retained by the audience? Asks “How have we done?”
Confirmative Evaluation- Once summative evaluation is conducted confirmative evaluation assures the learners continue to perform well over time and meet the objectives.
Validity and Reliability of the Tests
Validity- a test is valid when it measures what was learned, as specified by the instructional objectives for the unit. Use two types of measures used most often by Instructional designers is “Face validity” supported by expert judgment that the measure appears to assess what it is supposed to. The second is content validity – scrutiny that each content domain ( as described on page 263 of text for cognitive learning objectives) is appropriately addressed.
Reliability- refers to a test’s ability to produce consistent results whenever it is used. Rule is the more questions related to an instructional objective the better and be sure test are administered in a similar fashion when it is given to individuals.
Standards of Achievement
Criterion referenced testing- measures how well the individual student did in achieving the required level of competence specified in each objective. The specific standard is the criterion specified in each instructional objective. In criterion referenced testing the student is not compared to others but to themselves and their level of achievement.
Mastery learning is when the learner attains the minimum essential knowledge/skills at a reasonable competency level. Learners are given more than one opportunity to achieve this level.
Standardized testing is comparison of groups of learners usually with the outcome of all achieving a minimum level of performance.
Other methods of testing for achievement
Student self-evaluation- grade themselves
Pretesting- assess learners level of competence to study the course or topic and/or gives a benchmark to measure the degree of improvement after instruction is done.
Benefits of pretesting include alters learners what they know and don’t know about the topic, Provides learner with overview of what is included in course.
Provides baseline to compare pre-test to post-test results
Provides formative evaluation to the designer.
Development of Evaluation instruments
Chapter 12
Testing Knowledge
There is a direct relationship between instructional objectives and test items. The verb used in the objectives specifies how the attainment of the knowledge should be tested for. Examples- to list, label, state, describe, solve, calculate are all verbs that specify how the attainment of knowledge specified in the objective should be tested for.
If the objective specifies calculate then you test by asking the learner to calculate the answer to a problem.
If the objective specifies list then the learner is asked to list items for their response.
Types of tests
Multiple choice including writing higher level questions
True/False
Matching items
Constructed-response tests include;
Short-answer
Essay questions
Problem solving questions
Testing for behavior requires direct observation to see if the learner demonstrates the action or behavior.
Testing for attitude requires judgment of what the learner demonstrates in words/behaviors. The assessor infers the attitude is present.
Conducting Formative and Summative Evaluations
Chapter 13
Formative Evaluation
Purpose- to provide feedback to assist designer to improve materials and/or determine time requirements.
Audience-Helps determine primary audience
Issues-Uncovers issues with sequencing, interest, conflicts and so on with instruction.
Resources- Identifies resources needed or how to sequence and use resources
Evidence-Helps determine the amount and of what to ensure validity and provide information.
Data Gathering
Analysis of the data
Reporting
Progression of Formative Evaluation
One-to-one- Designer tries out the instruction with individual learners to gain feedback on methods and clarity
Small-Group trials- Group of 8+ to allow for observation, performance data and then correct for weakness
Field Trials Use of instruction with full sized learner group under realistic conditions.
Summative Evaluation
Purpose is to allow designer to reach unbiased, objective answers to evaluation questions about expected program outcomes and then decide if program is achieving those outcomes.
Assess program effectiveness – to what degree did students accomplish the learning outcomes for each unit of instruction?
Program efficiency looks at time required by learners to achieve unit objectives, number of faculty/staff needed to do instruction, use of facilities assigned to the program.
Program costs- what does it cost to develop and operate a specific program for the number of learners served?
Confirmative Evaluation – Determining outcomes over time
Learner orientated- the degree to which, as time passes, consumers of the instruction retain the skills and knowledge needed to perform at desired levels.
Context Oriented- the degree to which the instructional product remains effective as conditions ( policies, resources, technology) change over time
Evaluations of Educational Programs
Educational programs- usually measure broad objectives. Difficult to measure success as often attainment of objectives is truly seen in courses taken subsequently or found in on-the-job proficiency. Most often test for competence by;
Completing questionnaires
Conducting interviews
Making observations
Examining performance records of learners.
Asking employers/ alumni about job skills
Evaluations of Training Programs
Measure appropriateness of the training- have circumstances changed in the job site that necessitate updating the training? Use confirmative evaluation to determine.
Competencies of employees- Can employees transfer knowledge/skills to the work environment?
Benefits to the organization- Need to measure return on investment to justify training expense- study objectives and determine what problem training was addressing. Then measure success in resolving the problem with person who were trained.