u1d2b

Assessment of Learning

Following instructional delivery, Schrunk (2012) identifies five broad techniques for assessing whether students have learned: (a) direct observation, (b) written responses, (c) oral responses, (d) ratings by others, and (e) self-reports.  Each technique has advantages and disadvantages.

Direct Observation

For obvious reasons, attempting to assess learning by direct observation would only be effective if the learning outcome is an action or behavior.  For example, to determine if a nursing student has learned to properly conduct a physical examination, one would watch the student perform using an actual patient.  This type of learning assessment is more reliable if the observer is using an itemized checklist that deconstructs the educational goal (complete patient assessment) into individual tasks (e.g., checks pupil reaction; listens to the patient’s heart sounds; checks patient’s reflexes, etc.).  The assessment will be even more reliable if the tasks are clear and the observer is not forced to determine the quality of the tasks.

One serious limitation is that one cannot reliably conclude that absence of the learning in the event the behavior does not occur.  Using the example above, it is possible the student actually learned how to perform the examination, but did not perform appropriately during that particular assessment.  This type of assessment is also prone to observer bias introduced because of familiarity with the student or observer fatigue.

Written Responses

Written responses provide an effective and often efficient method of testing cognitive learning.  They are the most common method of learning assessment in higher education (Goubeaud & Yan, 2004).  Although written response assessments can take many forms, the most common is often called a paper and pencil test (Frank & Barzilai, 2004).

Written response assessments are advantageous because teachers can administer the exams with relative ease to a large group of students.  In many cases, the exams can be mechanically graded relatively quickly.  Computerized grading can also provide item analysis statistics to help the instructor assess the quality of the assessment tool.

Depending on the type of written response assessment used, grading the assessment can require a considerable time investment and introduce subjectivity.  Additionally, the use of some forms of written response instruments may not allow assessment of certain types of learning.  For example, a multiple-choice exam will not permit an assessment of writing ability.  Finally, Roediger and Marsh (2005) demonstrated that use of multiple-choice examinations can actually create false knowledge.  This occurs when the student reads the distractors for each item and comes to believe some are actually the correct answer.

Oral Responses

Another form of learning assessment involves soliciting oral responses.  Commonly, this formative evaluation technique begins with a teacher simply asking a student a question related to the educational goals of the lesson.  Based on the student’s response, the teacher can make a judgment on whether the student is learning the material.  This form of assessment is quick, requires little planning, and is easy to conduct.

A more formalized type of oral response can be part of a summative examination.  In this situation, the evaluator should use well-organized questions to maximize the benefits of the technique (Joughin, 1998).  This type of assessment is common in the medical profession where physicians, nurses, and paramedics sit before a panel that presents clinical information and ask questions specific to that case.  The oral response format allows the evaluators to probe the depth and extent of the student’s learning.

One drawback to this method is that it often provokes anxiety in the student, which could affect student performance.  Soliciting oral responses can also be time consuming especially for summative evaluations involving a large group of students.  Finally, as with observational assessment, oral assessment is prone to evaluator bias.

Ratings by Others and Self-Reports

In the context of learning assessment and similar to direct observation, these two assessment techniques requires a subjective rating of learning by either the student, or another person or group of individuals.  Raters in the ‘other’ group could be a mentor, preceptor, or even a group of student’s peers.  The strongest advantage of these types of assessments is the ability to provide useful data about the affective domain of learning, which cannot be easily measured with other assessment techniques.

A serious disadvantage is the subjective nature of these assessments, which has the potential to introduce bias and significant challenges to reliability.  Generally, ratings by others are considered more predictive of performance than self-reports (Atkins & Wood, 2002).  In fact, in a study of 63 members of an Australian service industry organization, Atkins and Wood (2002) found a nonlinear relationship between self-report assessments and performance as measured by a standardized and impartial assessment center.  Self-ratings and ratings by others as a method of learning assessment are rarely used in higher education (Alquraan, 2012).

Measuring my Variable of Interest

My research interest is in the construct of empathy, specifically, empathy among health care providers.  Obviously, empathy itself is not observable.  This will require an operational definition, such as a score on a validated measurement instrument or the observance of a certain behavior.

Fortunately, such a measurement instrument exists.  The Jefferson Scale of Physician Empathy is the most commonly administered instrument for measuring empathy and has been validated in both practicing physicians and medical students (Hojat et al., 2001).  Researchers modified the original instrument and validated the tool in a variety of other health professions including nurses (Fields et al., 2011; Ward et al., 2009) and paramedic students (Williams, Boyle, & Earl, 2013).

In addition, I hope to be able to link empathy scores with a behavior, specifically the administration of analgesic medication.  My thoughts are that low empathy scores are partially responsible for paramedics withholding these medications from people who need them, although I have not fully researched this angle.

References

Alquraan, M. F. (2012). Education, business and society. Contemporary Middle Eastern Issues, 5(2), 124-133. doi:10.1108/17537981211251160

Atkins, P. W. B., & Wood, R. E. (2002). Self- versus others’ ratings as predictors of assessment center ratings: Validation evidence for 360-degree feedback programs. Personnel Psychology, 55(4), 871–904. doi:10.1111/j.1744-6570.2002.tb00133.x

Fields, S. K., Mahan, P., Tillman, P., Harris, J., Maxwell, K., & Hojat, M. (2011). Measuring empathy in healthcare profession students using the Jefferson Scale of Physician Empathy: Health provider – student version. Journal of Interprofessional Care, 25(4), 287-293. doi:10.3109/13561820.2011.566648

Frank, M., & Barzilai, A. (2004). Integration alternative assessment in a project based learning course and technology teachers. Assessment and Evaluation in Higher Education, 29(1), 41-61.

Goubeaud, K., & Yan, W. (2004). Teacher educators' teaching methods, assessments, and grading: A comparison of higher education faculty's instructional practices. The Teacher Educator, 40(1), 1-16.

Hojat, M., Mangione, S., Nasca, T. J., Cohen, M. M., Gonnella, J. S., Eedmann, J. B., & Veloski, J. (2001).  The Jefferson Scale of Physician Empathy: Development and preliminary psychometric data. Educational and Psychological Measurement, 61(2), 349-365. doi:10.1177/00131640121971158

Joughin, G. (1998). Dimensions of oral assessment. Assessment and Evaluation in Higher Education, 23(4), 367-178.

Roediger, H. L. III., & Marsh, E. J. (2005). The positive and negative consequences of multiple-choice testing. Journal of Experimental Psychology: Learning, Memory, and Cognition, 31(5), 1155–1159. doi:10.1037/0278-7393.31.5.1155

Schrunk, D. H. (2012).  Learning theories: An educational perspective (6th ed.). Boston, MA: Pearson.

Ward, J., Schaal, M., Sullivan, J., Bowen, M. E., Erdmann, J. B., & Hojat, M. (2009). Reliability and validity of the Jefferson Scale of Empathy in undergraduate nursing students. Journal of Nursing Measurement, 17(1), 73-88. doi:10.1891/1061-3749.17.1.73

Williams, B., Boyle, M., & Earl, T. (2013). Measurement of empathy levels in undergraduate paramedic students. Prehospital Disaster Medicine, 28(2), 145-149. doi:10.1017/S1049023X1300006X