Nursing Research 6
glovesme
chapter_11.pptxReliability, Validity, and Trustworthiness
Chapter Eleven
Introduction
Foundation of “Good” research is the trustworthiness of the data
Reliability and validity – most important concepts in decision making process
Reliability – consistency or repeatability of the test results; descriptors – consistency, repeatability, objectivity, dependability
Validity – degree to which the results are truthful; dependent on reliability of the test
So what do they really mean?
Reliability – the instrument consistently measure the same thing
Validity – the instrument measures what it is supposed to measure
The Topics of Reliability and Validity in the Evidence Based Nursing Practice
Reliability
Consistency
Repeatability
Stability
Accuracy
Relevance
Validity
Truthfulness
Reliability and validity are the most important concepts in the decision making process. If either of these concepts is missing from the data, then the practitioner is unable to make an informed decision, and therefore more likely to make an incorrect decision.
Reliability
Consistency: the degree to which you can expect to get the same results
Stability: the results remain constant
Accurate: the results are correct
The Relationship Between Reliability, Relevance and Validity
If a test is not valid it cannot be reliable, regardless of the relevance.
Click to edit Master text styles
Second level
Third level
Fourth level
Fifth level
Validity
Reliability
Relevance
The Relationship Between Reliability, Relevance and Validity
RELEVANCE
RELIABILITY
VALIDITY
The Instrument that is the most reliable and most relevant will also be the most valid
Relevance
Degree of the relation between the test and its objective
Does the test reflect what it was reported to be testing?
Example – Measurement of the height of a patient
Validity can not exist without reliability AND relevance
Reliability and relevance can exist independently of validity
A test can be
Reliable without being valid
Relevant without reliability
But a valid test must have some degree of reliability and relevance
Reliability
Reliability can be described as the proportion of the observed score that is the true score.
TRUE
SCORE
ERROR
SCORE
OBSERVED
SCORE
Reliability
Repeatability or consistency of the data
Need to understand about observed score
Actual score of the results
The error score can add or subtract from the correctness of any score
Error score increases the variability of the observed score from the true score
Reliability – degree that the observed score of a test reflects the true score of the test
Reliability
Theoretically we can statistically define reliability as the proportion of the observed score variance that is the true score variance.
Reliability
Calculation will fall from 0-1.0
Perfect reliability or 100% would be 1.0
No reliability 0
0.80 and up is considered high reliability
If you get something like 1.15, do the math again, you did something wrong.
Forms of Reliability
Correlation Coefficient
The outcome of the calculation of the relationship between the observed score and the true score
Reliability is high when results are close to 1.0
Ex: Nurse was wanting to test the reliability of a BP machine. The nurse take the BP with the machine and then manually. The nurse takes the two scores and calculates the correlation coefficient. Her answer is 0.959. Is the blood pressure machine reliable?
Interclass vs. intraclass
Interclass reliability
Reliability between two and only two variables or trials
Intraclass reliability
Reliability between more than two variables or trials
Interclass Reliability
Consistency
Expect to get the same results when measuring a variable more than once on a single day
Called Test-Retest reliability
Stability of the test
Ex: Test blood. Retest blood to see if similar result
Equivalency Reliability
Two forms of a test report same results
Comparison of types of tests or tools
Ex: Is the systolic BP taken by the BP machine equivalent to the systolic BP taken manually
Internal Consistency
Split halves reliability
Divide tool (test) in half and compare results from each half to see if consistent results obtained
Ex: Split the questionnaire in half based on odd/even questions. Compare. Calculate. If the number is close to 1.0 then there is some internal consistency.
Intraclass Reliability
Difference between interclass and intraclass reliability is the number of variables
Intraclass allows the comparison of more than two variables
Uses ANOVA as test(shows errors between the test and also within the test)
Allows a better estimate of overall reliability of scales and the errors
Intraclass Reliability - continued
Objectivity
Reliability of scores assigned by judges or reviewers
Ex: Judges at the Olympics watch a figure skater: scores are: 8.0, 8.1, and 6.8. What about nursing?
Accuracy
Measurement is known as standard error of measure (SEM)
Reflects fluctuation of observed score due to error score
Factors affecting objectivity and accuracy – fatigue, practice, timing, same types of the testing conditions, level of difficulty, precision, and environment
Validity
A valid test is a test that truthfully measures what it reports to measure
Classified as logical or statistical in nature
Logical: requires an understanding of the subject
Statistical: use stats to compare questions against a valid measure
Types
Content related validity
Criterion related validity
Construct validity
Content related validity
Based on logical thought process and interpretation of measure
As known as face or logical validity
Ex: if it looks like a duck and quacks like a duck, then it must be a duck.
It takes the face value or the logical explanation
In nursing, the bp cuff is a valid method for taking bp. Content related validity looks back at the actual process. If the person taking the bp took it with too small of cuff then it would be an invalid measurement
Criterion related validity
Based on comparison between tests being used to some known criterion
Ex: O2 sat taken by ABG. Too invasive. Use O2 sat machine on finger. Not invasive, still measure. If we wanted to test validity we would test it against an ABG.
Test being used: Finger pulse ox/O2 sat
Compared to known criterion: ABG O2 result
Construct validity
Focuses on tests associated with a psychological characteristic
Attempts to develop validity for measures that exist on theory
Ex: Pain scale. Several psychological characteristics can influence pain. In order for the pain scale 0-10 to be truly valid, it must undergo a lot of work.
Summary Points
Trustworthiness relates to being only as good as the tool/test used to collect data
Reliability and validity – most important concepts in decision making process
Reliability – determines a tool/test consistency in measuring the same thing
Validity – determines that an tool/test measures what it is suppose to measure
Validity cannot exist without reliability and relevance
Reliability and relevance can exist independently of validity
Summary Points - continued
Correlational coefficient is the degree (positive or negative) of the relationship of variables
Intraclass reliability – consistency between two measures that are present in the data
Types of interclass reliability – consistency of measures, equivalency reliability, and internal consistency
Intraclass reliability allows comparisons of more than two variables.
Consider objectivity and accuracy within intraclass reliability
Summary Points - continued
Types of validity – content related validity, criterion related validity, and construct related validity
Content related validity – level at which a sample of items, tasks, or questions represent the defined content
Criterion related validity – demonstrates that test scores systematically relate to one or more identified measures
Construct related validity – concentrates on the test scores associated with psychological characteristics
