psyc 545 db replies

Peer 1 Maximilian

The topic being discussed is surrounding test validity. There are many factors that impact test validity. Validity measures whether or not a test is actually measuring what it is intended to measure. There are many assessments in the world that are intending to measure certain factors but are failing to do so. We must better understand what impacts validity so as test takers, research users, and test creators, we can be sure that the correct information is being captured.

For example, one of the key issues with validity is test speededness (Ying, 2007). There are certain tests that set a specific period of time the test should be completed in. But for what circumstance? Are we measuring the correct amount of time to truly capture what is most important in a direct way or are we setting a time limit for practical administrative purposes (Ying, 2007)? The purpose of speededness of a test can effect a students’ test performance and validity of the scores. Too much or too little time both can have negative impacts on test performance and ultimately validity of the scores. These speededness issues occur within computer-based and computerized adaptive testing (Ying, 2007). There are several ways to overcome this: mode of administration, test speededness, the implementation of experimental designs, and available test item information (Ying, 2007).

The validity of these tests has become such an integral focus that there are now court defined job analysis requirements for validation of teacher certificate tests (Kuehn, 1990). This is critical for the future of the field of assessment because prior tests were discriminatory such as the TECAT towards black teachers over the age of 40 (Kuehn, 1990). An additional concern to the emphasis of valid tests is the requirement for many job holders to attain and retain certification or licensure such as doctors, lawyers, dentists, and the like. Even the timing of tests needs to be supported by empirical research (Ying, 2007). With all of the considerations in place, the future of testing with proper requirements and procedures in place will be better for our society as a whole. We shall move forward in the modern day with learning our lessons from the past that we can now carry with us into our future.

The Bible states, “All Scripture is breathed out by God and profitable for teaching, for reproof, for correction, and for training in righteousness,” (English Standard Version, 2001, 2 Timothy 3:16). We must aim to assess and train with sound methodology as this is what will continue to push our world in the right direction.

  

References:

English Standard Version Bible. (2001). ESV Online. https://www.openbible.info/topics/truth

Kuehn, P. A., Stallings, W. M., Holland, C. L. (1990). Court-defined job analysis requirements for validation of teacher certification tests.  Educational Measurement: Issues and Practice, 9 (4), 21-24.

Nelson, D. S. (1994). Job analysis for licensure and certification exams: Science or politics?  Educational Measurement: Issues and Practice, 13(3), 29-35.

Ying, L., & Sireci, S. G. (2007). Validity issues in test speededness.  Educational Measurement: Issues and Practice, 26(4), 29-37.

 

 Peer 2 Katelyn

     The topic I chose was standard setting. Standard setting can be defined as the procedure for developing what is considered as an acceptable level of performance (Cizek, et. al. 2004). However, performance standard setting has many aspects, such as why it is defined as a process. Setting performance standards do not have a consensus on what the best process for development is (Linn, 2003). Standard setting involves individuals from nationals requiring standards, individuals creating performance-level descriptors (PLD), individuals who set the standards, people who have to accommodate for the standards, people who are meant to meet these standards, individuals who make decisions based on the standard outcomes met or not met (nation, states, colleges, etc.). Performance-level descriptors (PLD) explain the amount of understanding and techniques that is needed for each performance degree (Perie. 2008).

     Multiple-choice assessments are not necessarily in the same category as performance standards setting. However, multiple-choice assessment can be utilized when setting such degrees. The performance standards used to be based on either a pass or fall (dichotomy) item (Cizek, et. al. 2004). However, in recent years, educational performance standard settings have switched mainly to an assessed set of performance degrees (extremely well qualified, well qualified, qualified, possibly qualified, no recommendation) (Linn, 2003). To say multiple-choice assessment is more beneficial than establishing standard settings, would not make sense. But, to say the utilization of multiple-choice assessment enhances the richness of performance standard settings (Perie, 2008) does make sense.

     Performance standards settings has furthered the field of assessment by simply adding more published research. The setting of performance standards is very contentious (Cizek, 2004; Linn, 2003). Therefore, many studies and literature reviews have been conducted to further understand the topic. Researchers have focused on areas where performance standards should be required and places where it is not necessary (Linn, 2003). Others have studied the common processes used to create the performance levels (Cizek, 2004). Researchers have also focused on PDLs and how they are created and how they can affect the outcome of standard settings if used properly (Perie, 2008).

     Performance standards settings have had many researchers dive into the validity of how they are developed and how they measure. It has been studied where it is necessary to have standards (licensure, certifications, etc.) (Linn, 2003). A multitude of researchers have argued against performance standards. One such individual acknowledged that one question gives minimal information alone, however “one item may well make the difference between passing and failing” (Green, 1981, p. 1005; Linn, 2003). PLDs have been researched and methods have been suggested to ensure less variability when it comes to developing standards (Perie, 2008). Rather than having a pass or fail, standards have relatively switched to different levels of performance (Cizek, et. al. 2004). However, there will always be variability due to the inconsistent definitions of proficient and varying performance standards throughout the states (Linn, 2003). As Christians, we have a standard to set. Our standards are not inconsistent, we follow the Bible. And the Bible says that we are supposed to be “the salt of the earth” and “the light of the world” ( New King James, 1982, Matthew 5:13-14). Be a light that no one can ignore!

 

References

Cizek, G. J., Bunch, M. B., & Koons, H. (2004). Setting performance standards: Contemporary methods.  Educational Measurement: Issues and Practice,  23(4), 31–50. https://doi.org/10.1111/j.1745-3992.2004.tb00166.x 

Linn, R. L. (2003). Performance standards: Utility for different uses of assessments.  Education Policy Analysis Archives,  11(31), 1–20.  https://doi.org/10.14507/epaa.v11n31.2003Links to an external site.  

New King James. (1982). Thomas Nelson.

Perie, M. (2008). A guide to understanding and developing performance-level descriptors.  Educational Measurement: Issues and Practice,  27(4), 15–29. https://doi.org/10.1111/j.1745-3992.2008.00135.x