need someone who know statistics
byorv20n7
lecture_9.pptx1
EDRE 9400 Measurement Theory and Test Construction
The Measurement Model (chapter 5)
Sections pertaining to Item Fit
The Three Uses of IRT
2
Today we will…..
Examining the fit of item difficulty estimates to the Rasch model
Examining the fit of person ability estimates to the Rasch model
As you know, inferential statistics is about relating your sample parameter estimate to the true (but unobserved) population parameter
This is done by estimating, say, a mean and then it’s standard error.
The standard error is the estimate of how much we would expect estimated means to vary if we were to conduct the identical study many times over and over again.
The standard error therefore allows us to estimate how accurate our estimated mean is based on our sample.
IRT estimated person ability and item difficulty values are also estimated!
We want to know the true ability and the true difficulty of the items.
We are estimating true population values based on estimates from our data.
So, items with smaller SE’s relative to size of the item difficulty (i.e., the t-ratio) are more precisely estimated that items that have larger (item difficulty/SE) ratios.
We can obtain a standard error of estimate for our item difficulty (delta) values:
Item Fit
10
Here are the item mean difficulties along with standard errors:
Item Fit
10
We can compute confidence intervals for each of the item difficulty levels:
Item difficulty +/- (1.96 X SE)
For example, for the first item, the 95% confidence interval is 3.225 +/- (1.96 X .038) = 3.15 to 3.299
Item Fit
10
The size of the item difficulty relative to its standard error is not the only way to measure the precision of estimation.
Fit indexes = based on differences between the observed score on each item by each person and the expected value of each item based on the Rasch model
Outfit mean square (aka unweighted mean square) =
Has an expectation of 1 if item is perfectly fitted by the Rasch Model.
Has a range from 0 to infinity.
Any value greater than 1 is noise (i.e., misfit).
Values are all relative to the perfect fit of 1. So a value of 1.2 = (1.2-1)*100 = 20% misfit of the item to the data (20% noise).
Values less than 1.0 suggest that the data are “overfit” (i.e,. Items too predictable). That is, item way too easy or difficult. So, a value of .8 = (.8-1)*100 = 20% or item is too well fit to the data by 20%.
Wilson-Hilferty transformation used to create a t-statistic with infinite DF. Remember, that makes t into a z! So, 1.96 or greater value indicates significant standardized OUTFIT value.
Null hypothesis = item fits the Rasch model exactly
Alternative hypothesis = item does not fit the Rasch model exactly (i.e., it is either over or under fitted).
We want the outfit value to be not significant!
Item Fit
10
Infit mean square (aka weighted mean square) =
Infit is weighted in such a way that the effects of respondents with item performance further away from the expected score are minimized.
The weighting reduces the influence of items with low variance (i.e., too much overfit) and too much underfit (i.e., off-target between item performance and expected item performance).
Therefore, the items with median fit are given more weight with the infit versus the outfit mean square).
A Rule of Thumb (see page 129)
Infit mean square values between .75 and 1.33 are acceptable regardless of t value.
Item Fit
10
A Rule of Thumb (see page 129)
Infit mean square values between .75 and 1.33 are acceptable regardless of t value.
How many items in this analysis are acceptable?:
Item Fit
10
A Rule of Thumb (see page 129)
Infit mean square values between .75 and 1.33 are acceptable regardless of t value.
Here’s a more graphical display of items with Infit MSerrors
Item Fit
Notice all values within the lower and upper bounds of rule of thumb.
10
All participants should tend to get easier items right and harder items wrong.
But that is not perfectly so. Indeed, some students may actually get harder items right and easier items wrong. The latter would be an example of respondent misfit.
Each person’s ability estimate (with its SE), and infit outfit indices reported:
Respondent Fit
10
Each person’s ability estimate (with its SE), and infit outfit indices reported:
Respondent Fit
We can scroll down the list to find people whose estimated ability estimate poorly fits data.
This can be imported into SPSS.
10
12
Respondent Fit
Excel file can be created for each person’s latent ability and SE only.
In conclusion,
IRT separates test scores into both item difficulty and student ability based on a statistical model (the Rasch model being the most popular). Consequently, it should not be surprising to you that the fit of both the items and the persons to the stat model are rarely ever perfect, and indeed could be problematic.
Fortunately, it is rather easy to identify problematic items and students based on indices of misfit.
In Conclusion
10
14
EDRE 9400 Measurement Theory and Test Construction
3 uses of IRT video
15
EDRE 9400 Measurement Theory and Test Construction
http://www.youtube.com/watch?v=K6qkwuWSZMw&list=PLJNUIJnElUzDmrIPunMyF3tTvIHb65wNb
The Computer Adaptive Test
Test is more valid because SE
is lower thanks to IRT
Results are scored “on the fly”
Person stops taking test when
test information is sufficient
Usually less items need to be
administered than the usual test
16
The NREMT is transitioning to a computer based test delivery format on January 1, 2007 for the following reasons.
The test “adapts” in the sense that items are presented that are most informative for measuring ability
Therefore, different people are given different items and the test tailors which items to present based on the person’s performance on prior items.
Computer Adaptive Testing
Item Response Theory (IRT) Makes CAT More Precise
Once an item is answered the algorithm allows for estimation of a person’s theta. Then another item is presented which has a delta that is a little higher than the person’s estimated theta.
This continues until CAT determines items with the highest delta that can be answered correctly.
So, items determined based on
high ability person has high likelihood of answering a more easy item correct.
Low ability person has low likelihood of answering a hard question corect.
Accuracy of the instrument for estimating ability goes up because items that are presented in tailored to person’s ability.
And a byproduct of this is that only a small number of items need to given to person.
18
In terms of the testing modalities considered by the NREMT we looked at these three:
Staying the course with a paper-pencil exam. [Elaborate on what the NREMT does today with scheduling, exam materials shipping and management and test delivery.]
Changing to utilize a Computer-based form of the examination that basically computerizes the paper exam and delivers it in a linear, question by question, fashion to the test taker. High-stakes computer-based exams, like the NREMT exam, are delivered in formal computer testing centers by computer testing delivery providers.
The third modality examined is the Computer-adaptive form of the exam which basically ‘adapts’ the way in which questions are delivered to the candidate based upon his or her ability. The same formal computer testing centers are used in this type of testing modality.
We will talk in more detail about each modality later on in the presentation.
Why Combine CAT and IRT
Test results are based on giving items with the most information and obtaining the estimate of theta with sufficient information
Every question that is selected is estimated to have a high amount of information given the person’s estimated theta.
19
In terms of the testing modalities considered by the NREMT we looked at these three:
Staying the course with a paper-pencil exam. [Elaborate on what the NREMT does today with scheduling, exam materials shipping and management and test delivery.]
Changing to utilize a Computer-based form of the examination that basically computerizes the paper exam and delivers it in a linear, question by question, fashion to the test taker. High-stakes computer-based exams, like the NREMT exam, are delivered in formal computer testing centers by computer testing delivery providers.
The third modality examined is the Computer-adaptive form of the exam which basically ‘adapts’ the way in which questions are delivered to the candidate based upon his or her ability. The same formal computer testing centers are used in this type of testing modality.
We will talk in more detail about each modality later on in the presentation.
Why Combine CAT and IRT
CAT estimates person’s theta over and over again until the theta is estimated with sufficient information:
20
In terms of the testing modalities considered by the NREMT we looked at these three:
Staying the course with a paper-pencil exam. [Elaborate on what the NREMT does today with scheduling, exam materials shipping and management and test delivery.]
Changing to utilize a Computer-based form of the examination that basically computerizes the paper exam and delivers it in a linear, question by question, fashion to the test taker. High-stakes computer-based exams, like the NREMT exam, are delivered in formal computer testing centers by computer testing delivery providers.
The third modality examined is the Computer-adaptive form of the exam which basically ‘adapts’ the way in which questions are delivered to the candidate based upon his or her ability. The same formal computer testing centers are used in this type of testing modality.
We will talk in more detail about each modality later on in the presentation.
Why Combine CAT and IRT
Often CAT selects items that have 50% change of being answered correctly (remember this is where test information is usually high:
21
In terms of the testing modalities considered by the NREMT we looked at these three:
Staying the course with a paper-pencil exam. [Elaborate on what the NREMT does today with scheduling, exam materials shipping and management and test delivery.]
Changing to utilize a Computer-based form of the examination that basically computerizes the paper exam and delivers it in a linear, question by question, fashion to the test taker. High-stakes computer-based exams, like the NREMT exam, are delivered in formal computer testing centers by computer testing delivery providers.
The third modality examined is the Computer-adaptive form of the exam which basically ‘adapts’ the way in which questions are delivered to the candidate based upon his or her ability. The same formal computer testing centers are used in this type of testing modality.
We will talk in more detail about each modality later on in the presentation.
That depends on algorithm. Typically, items are presented until student starts getting items incorrect.
When person gets items with a certain delta wrong 3 or 4 times than the test stops.
The hardest item delta is used as the estimate of the person’s theta.
Recall that the theta and delta are on the same scale.
So, the hardest delta that a person gets correct is the person’s theta.
When is a CAT Test Over?
*=infrequent
22
Different people will get different number of items.
The test is calibrated so that it is challenging up to the limit of the examinee's ability, theta.
So, with CAT
23
It is very common that candidates finish the exam confident that they were unsuccessful. This is because we are not used to taking exams were every question is targeted directly at our ability level. The best strategy for success is to answer every question to the best of your ability and not to jump to conclusions regarding whether you passed or failed.
