CHAPTER 5 HOMEWORK

Chapter5.pdf

Home >Science homework help >CHAPTER 5 HOMEWORK

Susan J. Beck, Ph.D., MLS(ASCP) CM

Evaluating Performance in the Clinical Laboratory OBJECTIVES The material presented in this unit should enable the learner to:

1. Describe the sequence of steps involved in planning an instructional activity. 2. Discuss the limitations of objective tests in evaluating performance in the clinical laboratory. 3. Describe the common errors associated with rating scales. 4. Design a rating scale for the evaluation of affective skills. 5. Given an instructional situation, suggest an appropriate evaluation method. 6. Given an example of a written comment on an evaluation form, suggest ways to make the

comment more useful. 7. Discuss the use of peer and self‐ratings in assessing performance. 8. List the evaluation methods that can be used to assess performance in the cognitive, affective,

or psychomotor domains. 9. Identify the required procedures in a competency testing program. 10. Design a competency assessment program.

THE INSTRUCTIONAL SEQUENCE

Goals: The first step in planning an instructional activity is to identify goals. Goals are statements that describe the general knowledge, skills, or attitudes that the learner will possess after the instructional activity.

Objectives: The next step in instructional planning is writing objectives. Objectives are statements that describe the specific learning outcomes of an instructional activity and they are written in greater detail than goals. Objectives specify the observable knowledge, skills or attitudes that the learner is expected to exhibit after completing an instructional activity. Objectives serve to communicate the instructor's intent for a particular learning experience.

Learning Activities: Instructional activities or learning activities are chosen to help the learner master the objectives. They include such things as lectures, laboratory exercises, demonstrations, and online instruction.

Evaluation: An evaluation process should be planned to tell the instructor whether or not the learners were able to master the objectives and whether or not the instructor's approach was effective in assisting learners.

Susan J. Beck, Ph.D., MLS(ASCP) CM

Evaluation of students or practitioners in the clinical laboratory is a complex task that can be very satisfying and rewarding, or it can be frustrating and ineffective. The time and effort devoted to developing good evaluation methods and procedures can prevent frustrations and promote improved performance. EVALUATION METHODS Performance in the clinical laboratory can be measured by a variety of methods including objective tests, direct observations, judging the work produced by students or practitioners, peer‐evaluations, and self‐evaluation. Because performance in the clinical laboratory involves a combination of cognitive, psychomotor, and affective skills, a variety of methods should be used to assess overall performance. 1. Tests and Assignments

Objective Tests:

 Objective tests are often used in the clinical laboratory to assess cognitive abilities. Most objective tests given in the clinical laboratory are written or online. If well‐constructed, they can measure higher level cognitive skills such as analysis, synthesis, and evaluation.

 Objective tests should not be the sole source of evaluative information for clinical laboratory performance, however, because they usually do not measure technical skills or attitudinal behaviors and they may tend to oversimplify the problems encountered in the clinical laboratory.

Written Assignments:

 Assignments such as written papers and case studies can be given to develop and assess higher‐ level cognitive skills. A research paper or case study can also assess an individual’s ability to correlate information from all laboratory sections and from other health care disciplines. If the paper or case study is presented to the clinical laboratory staff, these assignments can also assess communication skills. Before considering these types of assignments, the criteria for evaluating the work should be defined.

2. Direct Observation Performance in the clinical laboratory can be readily observed but it must be documented in order to be useful for evaluation purposes. Tools such as anecdotal records, rating scales, and checklists have been developed to help instructors or supervisors record observations and make assessments of performance. Anecdotal Records. An anecdotal record is a written description of an individual’s behavior over a brief period of time. It provides documentation of an incident that may be included in the evaluation of an individual’s performance. Certain objectives, particularly those in the affective domain, may be effectively evaluated using anecdotal records.

Susan J. Beck, Ph.D., MLS(ASCP) CM

Tips for using anecdotal records:

 The instructor or supervisor should keep the description of the behavior separate from any interpretation or comments on the behavior.

 The instructor or supervisor should observe the individual long enough to make the report meaningful and should record the observations as soon as possible after the incident.

 After the anecdotal record is complete, it should be discussed with the student or the practitioner. A place for the student’s or practitioner’s comments and signature can be provided on the form.

Problems with anecdotal records include:

 It takes a considerable amount of time to write descriptions of observations.  To be valid, more than one evaluator should observe an individual’s performance and this is

difficult to schedule.

 Evaluators tend to report only the unusually positive or negative behaviors while omitting the ordinary acceptable behaviors.

Because of the problems associated with anecdotal records, they are rarely used as the sole method of documentation for evaluation. An example of anecdotal record is shown in Figure 1: Figure 1

ANECDOTAL RECORD FORM Student: Marian Jones Instructor: Alfred Smith Date: 08/23/14 Description of Behavior: Marian was in the second day of her phlebotomy rotation. She was asked to collect blood from a patient who turned out to be very fearful and difficult to draw. The patient refused to have her blood drawn at first and Marian did not pressure her. Marian talked with her about the procedure and was able to calm the woman’s fears. She was able to obtain all the blood samples needed for laboratory testing. Marian used excellent interpersonal skills and she exceeded our expectations for a student with her level of experience. Corrective Action needed (if any): None Student’s Comments:

This was a learning experience for me! I am now more confident both in my ability to draw blood and interact with patients. __ Alfred Smith 08/23/14 Marian Jones 08/23/14 Instructor’s Signature Date Student’s Signature Date

Susan J. Beck, Ph.D., MLS(ASCP) CM

Rating Scales. A rating scale is a list of traits along with a scale that allows the evaluator to indicate the degree to which the student or practitioner has demonstrated that trait. The scales used may be numerical or graphic and may include descriptions of the trait at each defined point. Rating scales are often used to assess traits such as efficiency, judgment, leadership, ability to work with others, dependability, ability to work under pressure, and willingness to admit mistakes. Tips for designing and using rating scales include:

 Rating scales should have between three and seven defined points for each trait.  There should be a space for the rater to mark "not observed" or "unable to evaluate."  Several evaluators should observe the student or practitioner and the ratings should be

combined for an overall evaluation of the performance.

 The respondents can be asked to look at the items they rated the highest and the lowest and to list one or two reasons or specific behaviors that led them to give that rating. This not only gives the learner more feedback, but also helps the instructors be more objective in their evaluations.

 Open‐ended questions can be added such as “How can this student show improvement?” or “What are this employee’s strengths?” to provide more specific information.

Rating scales generally require less time to use than anecdotal records and they can guide the evaluator’s judgments about an individual’s performance. The results of the ratings can yield specific, concrete feedback for the student or the employee. Rating scales can be prone to errors, however, because subjective judgments are made by a variety of people. Some of the common errors associated with rating scales include:

Error of leniency.

This occurs when an evaluator’s ratings of an individual tends to be higher than they should be. This usually occurs because the evaluator knows and likes the student or the practitioner.

Error of central tendency. Some evaluators hesitate to give high or low ratings and tend toward an average rating for all students or employees. This is especially true if the evaluator has not had much time to observe the individual’s behavior.

Halo effect. This occurs when the evaluator has a general impression of an individual and this impression influences the judgment of all of that individual’s traits.

Contrast error. This error occurs when evaluators contrast the trait in themselves and in the student or practitioner. For example, instructors who perceive themselves as extremely efficient may give students low ratings on efficiency.

Proximity error. The evaluator may tend to rate traits that are placed next to one another on the rating scale form in a similar fashion.

Most of these errors can be overcome by providing adequate training for the evaluators using a rating scale.

Susan J. Beck, Ph.D., MLS(ASCP) CM

Examples of Rating Scales: Rating scales can vary a great deal in length and complexity. The type of rating scale that is used will depend on the objectives that are being evaluated and the feasibility of using the form with the staff in a clinical laboratory setting. In the example of a rating scale shown in Figure 2 below, each trait is defined and descriptions are provided to help the instructor rate the student’s performance or ability. This type of rating scale requires the most time to complete; however, it is also the most objective type of rating scale because each point is clearly defined. Figure 2

EVALUATION FORM Student:______________________ Date:________________ Instructor:____________________ Efficiency and Organization: Rate the student's ability to work in an organized fashion and make good use of time. ____ Not observed ____ Slow with assignments, disorganized at work station ____ Occasionally slow with routine work, disorganized with new or complex tasks ____ Completes routine work on time and in organized fashion, sometimes slow or disorganized with

new or complex tasks ____ Completes routine and complex tasks on time and is usually organized ____ Always completes work in allotted time, is very organized Dependability: Rate the student's consistency in carrying out laboratory policies and procedures. ____ Not observed ____ Cannot be relied on to complete tasks ____ Frequently late, leaves tasks unfinished ____ Occasionally late or not in lab when expected, occasionally has difficulty following through with

assigned tasks ____ Usually present on time, usually completes all assigned tasks ____ Always present on time, can be relied on to carry out any task assignment Ability to Work under Pressure:Rate the student's ability to accurately perform laboratory tests and remain calm in stressful situations. ____ Not observed ____ Cannot perform under pressure, completely stops work ____ Continues to work but gets nervous, slows down and makes errors ____ Occasionally gets flustered but continues to work, slows down but does accurate work ____ May get nervous but continues to do accurate work at a normal pace ____ Demonstrates self‐control, performs all tests accurately

Susan J. Beck, Ph.D., MLS(ASCP) CM

A rating scale with less detail is shown in Figure 3 below. This form is easier to use and may be more practical for the clinical laboratory because it requires less time to complete. It is more open to interpretation than the rating scale with descriptions of each trait because evaluators’ perceptions of general terms like "average" or "above average" will vary. A space for comments on the ratings or the addition of open‐ended questions can add to the specificity of the information provided by the evaluator. Figure 3

EVALUATION FORM

Student: _________________________ Instructor: __________________________ Date: __________________________ Use the following scale to rate the student’s performance on each characteristic: 1 = Poor 2 = Below average 3 = Average 4 = Above average 5 = Outstanding N = Not observed 1. Preparation for daily assignments....................................... 1 2 3 4 5 N 2. Organization and efficiency of work.................................... 1 2 3 4 5 N 3. Ability to concentrate on work at hand................................ 1 2 3 4 5 N 4. Willingness to work on problems until they are resolved..... 1 2 3 4 5 N 5. Observance of safety rules and laboratory policies............. 1 2 3 4 5 N 6. Willingness to help others with work.................................... 1 2 3 4 5 N 7. Ability to work independently................................................ 1 2 3 4 5 N 8. Willingness to accept constructive criticism ......................... 1 2 3 4 5 N 9. Ability to work cooperatively with others............................. 1 2 3 4 5 N 10. Adherence to dress code....................................................... 1 2 3 4 5 N 11. Quality of work performed................................................... 1 2 3 4 5 N 12. Ability to handle stressful situations.................................... 1 2 3 4 5 N Please comment on any items rated poor, below average or outstanding. Comments:

Susan J. Beck, Ph.D., MLS(ASCP) CM

Checklists. A checklist is a list of statements describing the behaviors that comprise a particular task or procedure. The behaviors are checked to indicate whether or not they occurred. A checklist is differentiated from a rating scale by its "all or none" format. With a checklist there is no judgment on how well the individual performed on the behavior as there is with the rating scale. Checklists are most useful in evaluating psychomotor skills that can be broken down into clearly defined steps. The evaluator can identify problems while watching the individual perform a procedure and can then give specific suggestions for improvement. Tips for developing checklists:

 List the steps or important behaviors in a procedure.  Use only the critical items to keep the form short enough to be usable.  The items on the checklist should be written in specific behavioral terms and arranged in the

natural sequence of performance to aid the observer. An example of a checklist is shown in Figure 4: Figure 4

Student name: _____________________ Instructor: _________________________ Date:___________

Antibody Screening using the hemagglutination method ___ Label test tubes for each cell suspension + patient’s ID ___ Add 2 drops of patient serum to each tube ___ Add 1 drop of the appropriate cells to each tube ___ Mix tubes ___ Centrifuge at appropriate speed ___ Examine tubes for hemolysis ___ Resuspend tubes and examine for agglutination ___ Record results ___ Add PEG ___ Incubate at 37

o C for 20 minutes

___ Wash cells 3 times with normal saline ___ Add Anti‐IgG to cell suspensions ___ Centrifuge at appropriate speed ___ Resuspend tubes and examine for agglutination ___ Record results ___ Record correct interpretation

Susan J. Beck, Ph.D., MLS(ASCP) CM

Consider the four evaluation tools for documenting direct observation shown in Figures 1 – 4.

A. Anecdotal records (Figure 1) B. Rating scale with detailed descriptions of traits (Figure 2) C. Rating scale with general descriptions of traits (Figure 3) D. Checklist (Figure 4)

Which of the methods is:

1. Most subjective 2. Best for evaluating a student or employee’s ability to follow a procedure 3. Most objective way to evaluate a number of affective skills 4. Most efficient way to evaluate a number of affective skills

3. Evaluation of Work Students or practitioners can also be evaluated by judging the quality of the work they produce. This can be accomplished by giving practical examinations or by reviewing laboratory records and results. Practical Examinations. Practical exams allow the evaluator to control the environment and provide a similar situation for all individuals. Practical exams can be developed to test the ability to perform the routine tasks as well as tasks involving problem solving and troubleshooting. Because the practical examination is a controlled situation, however, the evaluator cannot be completely assured that the exam will assess how an individual would respond to an actual problem. Students or practitioners taking a practical examination may be very nervous and may make mistakes that they would ordinarily avoid. Alternatively, they may be extremely cautious and work very slowly, repeating tests that they would not normally repeat. Students will be more relaxed in taking a practical exam if they are reassured that the procedures or problems on the exam are ones that they have been taught and have been able to practice. Similarly, practitioners who are given practical examinations will relax if they are well‐informed about the content of the practical exam and the expectations for satisfactory performance. Reviewing Results and Records of Work. Performance can also be evaluated by reviewing the results of the laboratory tests that an individual has completed. In clinical laboratory educational programs, students’ work is supervised and the results are checked by the instructor before they are reported. This provides a continuous mechanism for the evaluation of the student’s work. Looking at the results that a particular student has produced over an extended period of time may be very helpful in detecting behaviors that have improved or those that are not improving. For example, a review of a student’s work may show frequent result reporting errors on laboratory reports early in a rotation and improvement in result reporting at the end of the rotation. For competency testing, a supervisor may wish to monitor a practitioner’s test result reporting to assess compliance with laboratory procedures and to evaluate the individual’s ability to detect and resolve problems.

Answers: 1. A 2. D 3. B 4. C

Susan J. Beck, Ph.D., MLS(ASCP) CM

4. Peer Evaluation Peers may provide helpful information for evaluation in some settings. Students or practitioners can be asked to observe an individual performing a laboratory procedure and use a checklist to indicate whether or not each step occurred. The experience of observing and documenting performance can be a learning experience for the peer observer as well as for the individual performing the test. Peers can also be asked to complete rating scales or other evaluation forms. When using peer‐evaluation, the instructor or supervisor should clearly explain how the information will be used. Often peers are reluctant to give honest feedback if they feel it will adversely affect a colleague and they may not report problems or errors. Peer evaluation is most helpful when it is used in the context of formative evaluation rather than evaluation that affects a grade or job performance rating.

5. Self Evaluation Self evaluation can help students or practitioners assess their progress toward goals and objectives, identify their strengths, and recognize the areas needing improvement. Students or practitioners can be asked to assess their own performance by completing the same checklists or rating scales used by the instructor or supervisor. Other means of developing self assessment skills include asking individuals to grade their own objective tests or practical examinations. Students or practitioners can compare their performance to the expected standards and determine their own grade. This approach can provide immediate feedback and can make the exams a more effective learning experience. Whenever individuals are asked to assess their own work, the instructor or supervisor plays an important role as a consultant and advisor. Students and practitioners should receive feedback on their performance as well as on their ability to evaluate that performance. In subjective evaluations, such as rating scales, the most honest self assessments will be obtained when the student or practitioner is not concerned about the adverse effects of reporting problems or errors. To encourage accuracy and honesty in self‐evaluations, the instructor or the supervisor may wish to limit the use of information from self‐evaluations to formative evaluations. COMPETENCY TESTING The Clinical Laboratory Improvement Amendments of 1988 (CLIA’88) define standards for technical supervisors and they include the requirements for evaluating the competency of laboratory staff. The technical supervisor is responsible for “Evaluating the competency of all testing personnel and assuring that the staff maintain their competency to perform test procedures and report test results promptly, accurately and proficiently.”

3 For non‐waived testing, competency testing must occur semiannually

during the first year of employment and then annually unless the test methodology or instrumentation changes. If the instrumentation or test methods change, the employee’s competency must be re‐ evaluated to include the new test methods or instrumentation before the employee can report patient results. The six procedures listed in Table 1 must be used to assess competency. Each of the evaluation methods described in this chapter can be used to document each of these procedures:

Susan J. Beck, Ph.D., MLS(ASCP) CM

Table 1

Six Competency Testing Procedures Evaluation Method

1 Direct observations of routine patient test performance, including patient preparation, if applicable, specimen handling, processing and testing

Checklists

2 Monitoring the recording and reporting of test results Review of records

3 Review of intermediate test results or worksheets, quality control records, proficiency testing results, and preventive maintenance records

Review of records

4 Direct observation of performance of instrument maintenance and function checks

Checklists

5 Assessment of test performance through testing previously analyzed specimens, internal blind testing samples or external proficiency testing samples

Practical exams, Review of Proficiency Testing results

6 Assessment of problem solving skills Practical exams, written exams

EFFECTIVE USE OF EVALUATION METHODS Evaluating performance in the cognitive, affective and psychomotor domains Using a variety of types of evaluation methods facilitates the measurement of skills in the cognitive, affective, and psychomotor domains. Summary of Classification of Objectives

The Cognitive Domain includes intellectual learning outcomes such as the recall of information, the comprehension of that information, and the processes of analysis, synthesis, and evaluation.

Affective Domain. The affective domain includes those objectives that emphasize values, attitudes, and interests. In the affective domain, a learner passes from a level of simply being aware of a phenomenon to the development of a value system that guides behavior.

Psychomotor Domain. Psychomotor behaviors are those that require neuromuscular coordination. When procedures and techniques are taught in clinical laboratory science, the learning outcomes are often in this domain. The psychomotor skills in this domain range from simple ones such as labeling to complex ones such as phlebotomy.

Susan J. Beck, Ph.D., MLS(ASCP) CM

Given the following evaluation methods, suggest the major domain or domains that the method measures:

Evaluation Method Major Domain (s)

1 Written test given at the end a 3 week rotation in the Virology laboratory

2 Practical exam in which a practitioner must identify antibodies

3 Observation using a checklist of an employee performing phlebotomy

4 Evaluation of an employee’s professionalism using a rating scale

5 Review of a student’s results on a previously tested patient in HLA

6 Evaluation of a student’s adherence to the dress code and attendance policy using a rating scale

Improving Written Comments Written comments on evaluation forms can be a very powerful tool for providing feedback to students or practitioners on their performance. Unfortunately, written comments are often too vague to be useful. For example, the comment “Great work” on an evaluation form, does not give a student or a practitioner enough information about the skills or attitudes that should be continued. A vague comment about a problem is equally unhelpful. To be most useful, written comments should be as specific as possible in describing observed behaviors. It is helpful to separate the description of the individual’s behavior from any interpretations. Evaluative information should also be given to students or employees in a timely manner. The student or practitioner should be given enough time to correct the problem. Examples:

Poor: This is the worst student I have ever taught. She is unsafe in the clinical laboratory. Better:

Description: After four days of instruction on the urinalysis bench, this student was unable to perform the microscopic examination of urine samples. The student could not identify white blood cells, red blood cells, or bacteria. She was not able to discuss the correlation of the chemical results with the microscopic findings in urine samples.

Interpretation: This individual is performing well below the expectations for students who have had four days of instruction in urinalysis.

Often evaluators are unwilling to write negative comments because they are concerned about the consequences of their comments. Evaluators may be reluctant to write a comment that could affect an individual’s job or a student’s progress in an academic program. Evaluators may also fear that they will be sued for libel or defamation of character. Comments on evaluation forms are not considered

1. Cognitive, 2. Cognitive and Psychomotor, 3. Psychomotor, 4. Affective, 5. Cognitive and Psychomotor, 6. Affective

Susan J. Beck, Ph.D., MLS(ASCP) CM

defamatory if they are not made with malicious intent, if they are only shared with the people who have a legitimate need to know the information and if they are a “good faith exercise of professional judgment”.

SUMMARY Evaluating performance is one of the most difficult and most important aspects of instruction. In formal educational programs, the evaluation process often challenges the instructor to make difficult decisions about a student’s grade or progress in a program. These decisions are a part of a clinical laboratory educator’s professional responsibilities because educators must assure the public that their graduates are competent to practice in the working environment. Outside of formal educational programs, laboratory professionals who are involved in new employee orientation, in‐service programs, and competency testing also need to evaluate practitioners’ ability to meet objectives or standards. In all settings, effective evaluation methods are critical to the identification of problems, the improvement of performance, and the recognition of excellence. REFERENCES 1. Irby DM, Dohner CW: Student clinical performance. In Ford CW, Morgan MK (eds): Teaching in

the Health Professions. St. Louis, Mosby, 1976, pp. 213‐ 220. 2. Milan SD, Marshall, RD. Impact of Regents of the University of Michigan v. Ewing on academic

dismissals from graduate and professional schools. J Coll Univ Law. 1987. 13(4): 335‐52. 3. Code of Federal Regulations. Part 493. Laboratory Requirements. 42CFR493.1451. Available at:

http://www.access.gpo.gov/nara/cfr/. Accessed June 24, 2014. 4. Gronlund, NE. Measurement and Evaluation in Teaching. New York: MacMillan Publishing Co.

5th Ed. 1985. p. 408