week 4

profiledianaaranda10
Itemanalysispresentation.ppt
Home>Nursing homework help>week 4

SUMMER 2016

FACULTY DEVELOPMENT

Learning Objectives

  • Understanding and Interpreting Item Analysis
  • Discuss Test Item Construction
  • Review Test Blueprinting
  • Utilize the ParScore Application
  • Open Discussion Forum

Understanding Item Analysis

Using Data to Drive Decisions

Bobbi Martin MSN, RN, CNE

Learning Objectives

  • Identify key data from a standard item analysis report
  • Describe the meaning of a Kudel-Richardson (KR-20) reliability coefficient
  • Apply p-value data to determine item difficulty and make data driven decisions regarding test questions.
  • Use point biserial scores to make data driven decisions regarding test item discrimination
  • Evaluate whether or not a test item needs revision

Purpose of Item Analysis Reports

  • Find flaws in the test so you can adjust the mark before returning to students
  • Identify questions that were too difficult, too easy, had two correct answers etc. that you may want to consider eliminating from the exam
  • Scientifically make decisions about items on a test instead of getting defensive, making decisions on a random basis, or arguing with students

Item Analysis

  • Test Reliability
  • Item difficulty (too easy, too hard?)
  • Also known as p-value
  • Item discrimination
  • Does the test question discriminate between those students who really knew the material and those who did not?
  • Effectiveness of alternatives in multiple-choice tests

Standard Item Analysis Report

Looking at the Item Analysis Report… What does it all mean?

  • Test Reliability
  • Kuder-Richardson Formula
  • ParSCORE calculates this for us
  • Reliability refers to the consistency of test scores
  • If a student took an exam today, would the same or similar score be obtained if he/she took the same exam again one week later?
  • Example: KR20=0.82 indicates the exam is highly reliable in that high scoring students scored high consistently and low scoring student scored low consistently

KR20…What’s acceptable?

  • Most authors describe an acceptable reliability coefficient for teacher-made tests as 0.70 or higher
  • Schools with very discriminate admission criteria can use a coefficient that is 0.60

Item Analysis

P-Value

  • Item difficulty
  • Percentage of correct responses to a question
  • If an item is too difficult or too easy it will distract from the test’s reliability level because it will not discriminate effectively between those who know the material and those who do not.

P-Value

  • What’s a good or bad P-Value?
  • For tests that are multiple choice and have 4 answer alternatives, 0.63 is uses as the optimum P-Value by most nursing schools
  • This means that 63% of those taking the exam should answer the question correctly

P-Value

  • What about those easy questions?
  • Items answered correctly by 90% provide little discrimination and are thought to be too easy (for the exception of the first one or two questions to give students confidence)
  • Too many easy items decreases the test’s reliability = lower motivation for students to study!

P-Values

  • Often items with difficulty levels between 0 - 0.2 and 0.8 - 1.0 are discarded because they are either too difficult or too easy, respectively. They are not differentiating the population

41. Oral antiviral therapy is prescribed for a client with herpes zoster. The nurse would explain to the

client that the medication is used to decrease:

  • Itching
  • Pain
  • Recurrence
  • Spreading

Try it yourself

  • Based on what you just learned, try categorizing the item difficulty indexes below by placing them into the appropriate boxes below.

.53

.90

.35

.25

.87

.62

Try it yourself

  • If you categorized your p-values like this, congratulations!

.53

.90

.35

.25

.87

.62

Point Biserials—Item discrimination

  • The “worth” of an item is judged by it’s ability to discriminate. Such discrimination determines a test’s reliability
  • This tells us the item’s ability to discriminate between those who know the content and those who don’t!

Point Biserial

  • Where p-values assess item difficulty, point biserials assess item quality

Point Biserial

  • As a rule of thumb, in terms of discrimination index
  • .40 and greater are very good items
  • .30 to .39 are reasonably good but possibly subject to improvement
  • .20 to .29 are marginal items and need some revision,
  • Below .19 are considered poor items and need major revision or should be eliminated

Distractor Discriminations

  • Measure the proportion of students selecting each wrong answer and the ability of each wrong answer to distinguish between strong and weak students.
  • In detailed item analysis, discrimination indices are provided for each incorrect answer.
  • A positive point biserial is desired for the correct answer, and a negative point biserial is desired for each incorrect response.

What should you look for?

Difficulty Discrimination
Item (Key) Between 0.30 and 0.80 (Items that are too easy or too hard will not discriminate well. Higher is better. At least 0.20 for classroom exams. 0.30 or higher for standardized measures.
Distractors At least 0.20 (many detailed reports still do not report the p-value for each distractor, only the item total p-value) All should be negative. The more negative, the better. The larger the distractor difficulty, the stronger the distractor discrimination should be.

Example of biserials for each answer option

Answer choices, “A”, “B” and “D” are all incorrect answers

and yield a negative point biserial. The correct answer

is “C” and yields a positive point biserial.

 
Option A B C D Total Students
N 5 22 64 22 113
 
Biserial -0.15 -0.16 0.43 -0.28  
P-Value=0.57        
 

Basic Item Analysis

 
No. Total Upper 27% Lower 27% Point Biserial
5 0.57 75% 10% 0.43
 

P-Value

  • So what to do??
  • Review the P-Values
  • Look at items 30% or lower and 80% or higher
  • Ask yourself, “Is this a good test item?”
  • Review the Point Biserials
  • Decide whether to nullify the question or not nullify
  • Consult with other faculty members if you’re unsure what to do
  • Talk with the director if you have questions
  • Item analysis data enables almost ANY item to be edited so that it can become a worthwhile test item!
  • Edit the question for the next test and see if your P-Value changes

Nullify or Throw Out the Question?

  • That is the question!
  • Two ways to eliminate questions
  • Nullify which reduces the total number of questions on the exam
  • Give everyone credit for the item.

Time to go…