Chapter 6 HOMEWORK

Chapter6.pdf

Home >Applied Sciences homework help >Chapter 6 HOMEWORK

Susan J. Beck, Ph.D., MLS(ASCP) CM

Evaluating Instructional and Program Effectiveness OBJECTIVES 1. List the characteristics of effective instructors. 2. Discuss the reliability and validity of student evaluations of instruction. 3. Explain the rationale for a policy of anonymous and voluntary evaluations of instruction. 4. Develop a plan for the evaluation of instruction based on learner, peer and self‐assessments. 5. Design an evaluation form for a course. 6. Discuss the advantages and disadvantages of using rating scales in evaluation forms. 7. Discuss the problems associated with using learner performance as a measure of instructional effectiveness. 8. Describe methods to improve the objectivity of self‐evaluation of instruction. 9. List guidelines for peer observers of instruction. 10. State the NAACLS requirements for program evaluation. 11. Discuss the value and limitations of outcomes measures used in clinical laboratory educational programs.

EVALUATING INSTRUCTION Some instructors appear to be born with all the qualities and skills needed for effective teaching; most, however, develop their skills as they go along. Teaching is partly an art which "cannot be taught just as painting cannot be taught, but given the requisite interest and a good teacher, art lessons are valuable, as evidenced by successful students of great artists. So, too, are instructions in teaching valuable. Not every teacher will be a great teacher, but most are subject to improvement’’.

l Poor instruction is rarely

due to a deliberate or even conscious effort on the part of the instructor. Research has shown that most poor instruction occurs when the instructor has been inadequately prepared, when the instructor has developed bad habits that are reinforced over time, or when the instructor does not receive adequate feedback.

What constitutes good teaching? A look at the characteristics of effective instructors can help clinical laboratory professionals identify their personal strengths and the areas in which they need further development as instructors. Many studies have been undertaken to identify characteristics that are needed for effective teaching.

3‐6 Most

of these studies have been performed by asking learners in a variety of educational settings to rate the characteristics that they find most helpful in instructors. Some of the important characteristics of effective instructors include knowledge and enthusiasm for the subject matter, organization, the ability to clearly present and explain material, fairness in evaluation and grading, showing respect for the learners, and modeling professional skills and attitudes.

Susan J. Beck, Ph.D., MLS(ASCP) CM

Knowledge Effective instructors are those who can demonstrate their knowledge and understanding of the material in lecture, laboratory and discussion settings. Whether the learners are students in an educational program or participants in a continuing education session, they depend on the instructor to present current and accurate information. The knowledgeable instructor is able to give a perspective on current developments in the field, discuss different viewpoints on important issues, and direct learners to additional sources of information. The instructor must be constantly learning and must be conscientious in preparing for an instructional activity. The need to remain knowledgeable in the profession of clinical laboratory science is one of the most challenging but also rewarding aspects of teaching. Instructors often find that they have benefited more than learners in preparing for instructional experiences because the review of the subject material has deepened their understanding of the material or has given them a new perspective on a subject. Organization Effective instructors are described as organized in their presentations and in the material that they distribute to the learners. The organization of an instructional activity should be reflected in the instructor’s goals, objectives, handouts, assignments and evaluation tools. Organization is especially important in clinical laboratory science because of the amount and complexity of information that comprises most courses or continuing education programs. When the learners understand the organization of the material that is presented, they are better able to recall previously learned information and retain new information.

7 The instructor can communicate the organization of the

learning activity by clearly stating the goals and objectives, presenting material in a logical sequence, emphasizing key points throughout the presentation and summarizing major concepts. Enthusiasm for the subject and for teaching Effective instructors are able to command the learners’ attention and stimulate interest through their energy and their enthusiasm for the subject and for teaching. In student evaluations, enthusiasm is often the highest rated characteristic of effective instructors. The importance of an enthusiastic presentation was demonstrated in the “Dr. Fox” experiment. An actor, who presented a lecture offering no substantive information, was given high ratings by students because his presentation was very enjoyable.

8 Although presenting an entertaining lecture is not the sole criteria for effective teaching, the

instructor’s presentation skills should not be under‐emphasized. The instructor can convey enthusiasm for the subject matter by using movement and gestures during a lecture, by using interesting examples and by using humor. Instructional Skills Effective instructors focus on facilitating learning. This involves choosing the learning activities and instructional aids that will best meet the goals and objectives for the activity. An effective instructor then guides the learning process by explaining principles and procedures, clarifying points, pointing out relationships, giving positive reinforcement, and making suggestions for improvement. Other important instructional skills include the ability to speak clearly and audibly, to listen, to answer questions precisely, and to encourage learners’ questions.

Susan J. Beck, Ph.D., MLS(ASCP) CM

Evaluation Skills The instructor’s skill in evaluating performance is often mentioned in course evaluations as an important trait of effective teaching. Fairness in evaluation and grading is facilitated when the evaluation tools are directly derived from the course goals and objectives. Learners do not necessarily consider the most lenient instructors to be the best teachers, rather, they value instructors who set standards and are consistent in the application of those standards.

Personal Qualities and Attitudes Another important aspect of effective instruction involves the personal qualities and attitudes that an instructor demonstrates. Learners respond well to instructors who are friendly, patient and show respect for learners. Effective instructors are described as approachable, both in the instructional setting and outside of the classroom or learning activity. They monitor the learners’ responses to the instructional activity and are willing to provide additional help when it is needed. Instructors who listen and respond to concerns with sincere interest create a climate that facilitates learning. Effective Clinical Teaching The clinical component of an educational program is essential for the development of competent laboratory practitioners. In this setting, students apply the knowledge they have gained in the classroom to the real world of a clinical laboratory. Clinical instructors guide the students as they learn to prioritize work, organize time, handle ambiguous situations, and grow in independence. Effective clinical teachers are enthusiastic about teaching and welcome students into their work setting. They have a passion for the profession and are eager to pass that on to students. To be effective, clinical instructors must be competent and confident in their own skills. They should be willing to respond to student questions as well as pose questions that challenge students to apply their knowledge. One of the most important characteristics of effective clinical teachers is their ability to give feedback to students on their performance.

9 To give useful feedback, the clinical instructor must understand the criteria for

acceptable performance and be able to assess how well the student meets those criteria. Feedback that is specific, timely, and focused on changeable behaviors can motivate students to improve or reinforce good work. EVALUATING INSTRUCTIONAL SKILLS Instruction is both a skill and an art that can always be improved. Evaluation of instruction serves to reinforce the instructor’s strong points and to identify areas in which the instructor could be more effective. In general, evaluation works best if it is continuous because changes in instruction take place gradually and could be missed by a single measurement. Evaluation should also be based on a broad sample of the instructor’s efforts, rather than on one instructional situation. Evaluation should describe specific behaviors of the instructor and suggest concrete steps that the instructor could take to improve. Since there is no one way to teach, instructors must consider their own personality, their subject matter, and the environment in which they teach when evaluating their skills as an instructor. Multiple sources of information, including the learners, peers and the instructor’s self evaluation should be used to evaluate instruction.

Susan J. Beck, Ph.D., MLS(ASCP) CM

Learner or Participant Evaluation of Instruction Learners provide instructors with important information on their effectiveness. The instructor is constantly receiving feedback from the learners by the looks on their faces, the questions that they ask, their attendance in class and so on. To be of value, that feedback needs to be gathered in a systematic and formal manner. The approach to collecting information may vary depending on whether the learners are students in a formal educational program or participants in a continuing education session. Student evaluation of instruction in formal education programs. In academic settings, student evaluations are the most common source of information for assessing instructional effectiveness. Research indicates that student evaluations are generally reliable and valid. Reliability studies have shown that the results of student evaluations are stable over time and that different sets of students consistently agree on the ratings of an instructor. However, the reliability of student ratings is affected by the number of students in a class. If the class contains less than 8 to 10 students or if less than 75 percent of the class completes the evaluation forms, the results should be interpreted with caution.

10 In programs with small class sizes, instructors should review evaluations

over several years for more reliable results. Validity studies have addressed whether or not student evaluations correlate with other measures of instructional effectiveness. These studies have shown that student evaluations correlate positively with student achievement and with the instructor’s own assessment of effectiveness.

11 In general, students are in a good position to provide feedback on the

instructor’s classroom skills including preparation for class, organization, communication skills, enthusiasm and fairness. Students may be less able to provide valid feedback on the instructor’s knowledge of the subject matter. The reliability and validity of student evaluations in clinical laboratory programs is improved when the following guidelines are followed: 1. All students should be given the opportunity to evaluate an instructor, but the evaluation should not

be mandatory. The students who only fill out evaluation forms because they are required to do so usually do not give helpful feedback.

2. Students should be asked for comments on those areas of instruction that are under the instructor’s control. For example, students may dislike the fact that a class meets at 8:00 A.M., but the instructor may not be in a position to change the schedule.

3. Students should also be asked for comments only in those areas in which they are qualified to give feedback. Students would be qualified to comment on whether or not a textbook was helpful to them, for example, but not on whether or not it was the best textbook to choose for the course.

4. Instructors should let the students know that they are interested in improving and that they value the students’ comments When students perceive that the instructor is interested in making changes, the students often feel that evaluations are worth the time and effort required to thoughtfully complete them.

5. A standardized procedure should be followed for the distribution, collection and reporting of the evaluation forms. If paper evaluations forms are used, written comments should be typed before they are given to the instructor.

6. Students should be able to give evaluations anonymously and they should be assured that their comments will not affect their grades.

7. It is helpful to give evaluation forms to the students one to two weeks before the end of the semester. Students who are concerned with final examinations have little energy left to devote to extra tasks such as filling out evaluation forms.

Susan J. Beck, Ph.D., MLS(ASCP) CM

8. If given during class time, the instructor should allow sufficient time to complete the evaluation forms.

9. If online forms are used outside of class time, students should be sent reminders to complete the forms. This system makes the evaluation process more efficient and flexible, but the response rate may be lower than the rate for evaluations given during class time.

10. The forms should not require more than 15 to 20 minutes to complete because the students will tire of the exercise and provide less thoughtful responses.

Because student evaluations are typically given at the end of a course, students in the course do not benefit from the results. To address this problem, instructors may wish to conduct mid‐course evaluations or may establish a suggestion box for use during the course. Participant evaluation of instruction in continuing education programs Evaluations from participants in continuing education presentations provide useful information for improving instruction and may be required for continuing education credit. In continuing education settings, the evaluation forms should be brief and focused on the questions of greatest interest to the instructor. At the end of the presentation, the instructor or the moderator should remind the participants that the evaluation forms are very important and will provide information for the speaker and for future program planning. The speaker should allow sufficient time at the end of the presentation for completion of the form. If participants are rushed to get back to work or to another continuing education presentation, they will give very little time, if any, to the evaluation forms. The organizers of the presentation should collect the forms and provide the speaker with a summary of the ratings or comments. Evaluation Forms Most evaluation forms use a combination of rating scale items and open‐ended questions. Examples of statements that could be rated on a course evaluation form include:

 The goals and objectives of this course  The time given to each topic  The instructor’s explanation of the concepts in the course  The organization of the course  The instructor’s use of handouts and audiovisual aids  The instructor’s enthusiasm about the course material  The instructor’s availability for help  The instructor’s ability to answer questions  The assignments in the course  The usefulness of the textbook  The appropriateness of the examinations  The fairness of grading  The intellectual challenge of the course  The overall rating of the course  The overall rating of the instructor

Rating scales are popular because they can be standardized and because they can address a wide range of instruction behaviors in a short amount of time. However, rating scales have a tendency to provide general information and may not give enough specific feedback to the instructor who is interested in

Susan J. Beck, Ph.D., MLS(ASCP) CM

improving. One way to get more information from rating scales is to ask the respondents to look at the items they rated the highest and the lowest and to list one or two reasons or specific behaviors that led them to give that rating.

11 This not only gives the instructor more feedback, but also helps the

respondents be more objective in their evaluations. Adding open‐ended questions to evaluation forms such as “How can this course be improved?” or “What are the strengths of this course?” can provide more specific information. Often the written comments provide the most revealing and rewarding feedback to the instructor. Using information from student and participant evaluations Student or participant evaluations can improve instruction if the instructor believes the feedback is valid, if the instructor is motivated to improve, and if the feedback provides specific information about instructional behaviors. Often reading evaluations can be rewarding for instructors who see that their efforts are appreciated. Conversely, a bad evaluation can be discouraging. Sharing the information collected from evaluations with colleagues can help put them in perspective. If most of the evaluations are good and only one is bad, the instructor should avoid focusing on the bad evaluation. One bad evaluation may be due to personality differences between the instructor and the student or practitioner. Even the best instructors are not able to satisfy all of the learners all of the time. If, however, all evaluations are negative, the instructor needs to carefully consider the comments and seek feedback on instruction from additional sources. Learner Performance Teaching is effective if it contributes to learning. Assessing the performance of learners following an instructional activity or an entire course can help instructors evaluate the effectiveness of their instructional skills. In order to use the learners’ progress as a measurement of instructional effectiveness, the instructor must first establish where the learners are at the beginning of the instructional situation and what the learners’ should achieve as a result of the instruction. To determine the learners’ capabilities at the beginning of an instructional activity, the instructor can review the objectives from a prior instructional activity or give a pre‐test. The instructor’s objectives for the learning activity define what the learner should be able to do at the end of the instruction. After the instructional activity, an analysis of the learners’ performance on each objective can identify areas in which the instructor needs to improve the approach he or she is using to facilitate learning. There are some limitations associated with using learner performance as an indicator of effectiveness. There are many factors that contribute to the learners’ success including the learner’s capabilities and motivation at the beginning of the instructional activity. The best instructor may not be able to help some learners, and other learners may succeed in spite of ineffective teaching. Also, in most educational programs, many different instructors and instructional settings are involved in a student’s education, so it is often difficult to make a direct correlation between student performance and a specific instructor’s effectiveness. Finally, some goals and objectives for learners stress the retention and application of material to new situations and may not be measured immediately after the instructional activity. It may be difficult to correlate learning with a particular instructor’s effectiveness until several months or years have elapsed. Self‐Evaluation

Susan J. Beck, Ph.D., MLS(ASCP) CM

Self‐evaluation is a basic requirement for the evaluation and improvement of teaching; however, it is always subjective and should never stand alone as the sole source of information about effectiveness. Instructors may be too critical of their own skills or they may be unwilling to recognize any areas needing improvement. The objectivity of self‐evaluation can be improved by asking the instructors to evaluate their effectiveness using the same form as the learners or by providing the instructor with a checklist of effective instructional behaviors. Objectivity is also improved if the teacher views a video of their teaching. In viewing the video, the instructor can identify approaches that are working well and discover any areas that need improvement. Nervous habits may be revealed to the instructor for the first time on video. If the audience is included in the video, the instructor can also assess the learners’ reactions during the instruction. Often, viewing the video provides the incentive that an instructor needs to change. Peer Evaluation Peer evaluation of instruction is often used in academic settings for improving instruction and for personnel decisions. Colleagues are in an excellent position to give feedback on an instructors’ knowledge of the subject and ability to convey that information to the learners at the appropriate level. In classroom observations, colleagues can also judge the instructor’s ability to explain the material, the relationship of the material covered to the goals and objectives of the course, and the instructor’s use of appropriate instructional aids. Outside of the classroom, colleagues can be asked to evaluate course materials in order to assess the instructor’s organization and knowledge of the subject. Problems with peer observations include objectivity and reliability. To improve objectivity, peer observers should be given evaluation forms to guide their critique of instruction or course materials. Reliability is improved by using multiple observers on multiple occasions and by using trained peer observers. When peer observation is used for personnel decisions, the institution should develop standard procedures and guidelines. Peer observation is most effective in facilitating improvement if the instructors choose an observer with whom they feel comfortable and whom they trust to help them identify areas needing improvement. Before the observation session, the instructor and peer observer should meet to discuss the instructional activity and the specific behaviors on which the instructor desires feedback. After the session, the instructor should meet with the observer and ask for feedback and discuss ways to improve. The observer should concentrate on reporting what was seen, avoid passing judgment, and give positive feedback along with suggestions for improvement. IMPROVING INSTRUCTION The results of the evaluation should provide the instructor with a sense of satisfaction about the approaches that are working well and the impetus to change the things that aren’t working well. When addressing areas needing improvement, the instructor should identify a few specific instructional behaviors and begin to work on them. For example, an instructor may experiment with asking more questions during a lecture to increase the learners’ participation. By working with a few specific behaviors at a time, the instructor can more easily adopt and evaluate the behavior. Often in implementing a new method, the instructor is not immediately comfortable with the new approach and it may take several attempts to accurately evaluate the method. Sometimes trying out the method on the most receptive group of learners or letting the learners know that a new method is being evaluated may contribute to the success of the endeavor.

EVALUATION OF PROGRAMS

Susan J. Beck, Ph.D., MLS(ASCP) CM

A continuous quality improvement plan is important for clinical laboratory educational programs in order to respond to changes in the profession and the needs of the health care community. Programs accredited by NAACLS are required to “document a plan for continuous and systematic assessment of the effectiveness of the program.”

12 To meet NAACLS Standards, programs must review and evaluate

outcome measures from the past three years as part of their continuous improvement plan. Outcome measures may include: Certification or licensure exam results

This information provides an assessment of how graduates of a program compare to a national or state standard. While important, it may also be affected by the size of the class. In a small educational program, one student who does not pass the exam lowers the program’s overall pass rate significantly. This outcome also only addresses entry level competencies and does not indicate how well the program prepared graduates for future professional growth.

Graduation rates and attrition rates

There are a number of factors that affect attrition from clinical laboratory educational programs. Most of the academic reasons for attrition are identified early in the program while behavioral, emotional, and financial problems could lead to attrition at any point in the program. The NAACLS definition of graduation rate specifies that it is the number of students who complete the final year of the program so that students who are dismissed from the program in the first year do not affect this outcome measure.

Job placement

Another indication of program success is the ability of students to find jobs after graduation. This indicates that employers are satisfied with the graduates of a program, but it can also be affected by the job market in the area the graduate seeks employment.

Further education

The ability of graduates to progress in their education indicates that the educational program provided a solid foundation and that the program is respected by other educational institutions.

Student Capstone projects

Major projects completed by students provide documentation that the program prepared the student to integrate information from a variety of sources, analyze the information, and draw conclusions.

End of program evaluations

At the end of the educational program, students may be asked to look back over the entire curriculum and provide feedback. This could be conducted as a survey or an exit interview. When students see how the entire curriculum worked together to prepare them for entry level competency, they have a new perspective and may provide useful suggestions.

Graduate evaluations

Susan J. Beck, Ph.D., MLS(ASCP) CM

After graduates have been working in the field for a year or more, they are in an excellent position to provide feedback to the program. Focus groups or online surveys can be used to collect information from graduates.

Employer surveys

Employers who have hired graduates of a program can be asked to assess the employee’s preparation. In some cases, a deficiency may be attributed to the individual employee, but it may also indicate an area of the curriculum that needs strengthening.

Graduates’ professional success

By looking at the success of the graduates in their professional careers, a program can document the long term effectiveness of the program.

Multiple outcome measures should be considered in the evaluation of an educational program because each of these measures has limitations. The outcome measures should be discussed by the program’s faculty and shared with the program’s Advisory Board. The end result of the continuous quality improvement plan should be a documentation of the program’s successes as well as the identification of areas of improvement. SUMMARY Teaching and educational programs can always be improved. Resources of time and funding are always stretched thin and often a teacher or the faculty in an education program must prioritize those resources. Collecting feedback on teaching effectiveness and overall program effectiveness will help guide those priorities. REFERENCES

1. Ford CW: Instructional skills‐microteaching. In Ford CW, Morgan MK (eds):Teaching in the Health Professions. St. Louis, Mosby, 1976, p. 158.

2. Knope HJ, Diekelmann NL: Approaches to Teaching in the Health Sciences. Reading,Mass, Addison‐Wesley, 1978, p. 232.

3. Irby DM: Clinical faculty development. In Ford CW (ed): Clinical Education for the Allied Health Professions. St. Louis, Mosby, 1978, pp. 95‐105.

4. Irby DM. Clinical teacher effectiveness in medicine. J Med Educ 53:808‐815 1978. 5. Williams, N. What is Good College Teaching? The Teaching Professor. 4(1):1‐2,1990. 6. Centra JA. Reflective Faculty Evaluation. Jossey‐Bass Publishers, San Francisco.1993. pp.47‐93. 7. Bell‐Gredler ME, Learning and Instruction. Macmillan Publishing Co. New York.1986. pp 151‐187. 8. Naftulin DH, Ware JE, Donnelly FA. The Doctor Fox Lecture: A Paradigm of Educational

Seduction. J Med Educ 48:630‐635, 1973. 9. Steves A.M. Improving the Clinical Instruction of Student Technologists. J Nuclear Medicine

Technology. 2005; 33(4):205‐209 10. Feldman KA. Effective college teaching from the student’s and faculty’s view. Matched or

mismatched priorities? Research in Higher Education. 28:2291‐344. 1988. 11. Fuhrman BS, Grasha AF: A Practical Handbook for College Teachers. Boston, Little,Brown, 1983,

p. 200.

Susan J. Beck, Ph.D., MLS(ASCP) CM

12. NAACLS Standards for Accredited and Approved Programs. National Accrediting Agency for Clinical Laboratory Sciences. Adopted 2012, Revised 1/2014. Available at www.naacls.org. Accessed June 24,