What is the difference between reliability and validity?
shopittvr
the_mini_project_task.docxThe Mini Project Task
Instructions: Read about validity and reliability starting on page 324 of the textbook. Your assignment is to create a 5-page paper addressing the following questions:
a. What is the difference between reliability and validity? Which is more important? Why?
b. What are the different ways of assessing reliability?
c. What are the different ways of assessing validity?
d. What are the different ways of obtaining validity evidence?
The analysis requires the additional components:
· APA formatted paper including:
· Font: Times New Roman, 12 point, and double spaced.
· Margins: One inch margins, all around.
· Indents: One-half inch indent as to begin a paragraph.
· Proper APA citations and references.
· Proper use of Level 1 headings as to label the introduction, main body, and conclusions
segments.
· Proper use of Level 2 headings as to label the sections within the main body and
Conclusions.
_______________________________________________________________________________
Research information to use.
What is the difference between reliability and validity? Which is more important?
Reliability refers to the consistency or stability of the test scores; validity refers to the accuracy of the inferences or interpretations you make from the test scores. Both of these characteristics are important. Note also that reliability is a necessary but not sufficient condition for validity (i.e., you can have reliability without validity, but in order to obtain validity you must have reliability).
5.5. What are the definitions of reliability and reliability coefficient?
Reliability refers to the consistency or stability of a set of test scores. The reliability coefficient is a correlation coefficient that is used as an index of reliability.
Note that there are several different forms of reliability. First is test-retest reliability (the consistency of a group of individuals’ scores over time). The second type is equivalent-forms reliability (consistency of a group of individuals’ scores on two equivalent forms of a test). The third type or reliability is internal consistency reliability (consistency of items in measuring a single construct). The two subtypes of internal consistency aresplit-half reliability and coefficient alpha. The fourth major type of reliability is inter-scorer reliability (consistency or degree of agreement between two or more scorers, judges, or raters).
5.6. What are the different ways of assessing reliability?
Most of the types of reliability are assessed with simple correlation coefficients (called reliability coefficients). Test-retest reliability is the correlation between a group’s scores on the same test given at two different times (i.e., give a set of people a test twice and see if the two sets of scores are correlated). Equivalent-forms reliability is the correlation between a group’s scores on two forms of the same test (i.e., give everyone in a group two forms of the same test and correlate those two sets of scores). Split-half reliability is the correlation between a group’s scores on two halves of the same test (everyone in the group takes the test once and you give everyone a score on both of the two halves of the test; then you correlate those two sets of scores). Coefficient alpha can be viewed as the average of the correlations of all of the items on a test with each other (e.g., if a test only had 3 items it would be the average of the correlation between items 1 and 2, 1 and 3, and 2 and 3). It tells you if the items tend to be related. The basic inter-scorer reliability is the correlation between two raters’ ratings of a set of objects (e.g., a set of essay questions).
5.7. Under what conditions should each of the different ways of assessing reliability be used?
Test-retest is used to determine consistency of the scores on a test over time.
Equivalent forms reliability is used to see if different forms of a test give consistent results.
Internal consistency reliability is used to see if the different items on a test give consistent results. Inter-scorer reliability is used to see if two raters of a set of items give consistent results.
5.8. What are the definitions of validity and validation?
Validity is the accuracy of the inferences, interpretations, or actions made on the basis of test scores. Validation is the process of gathering evidence that supports the inferences made on the basis of test scores.
5.9. What is meant by the unified view of validity?
It means that all validity can be viewed as part of construct validity. That’s because to be discussing measurement validity, there has to be something that we intend to measure. The term “construct” simply refers to what we want to measure whether it be age, gender, IQ, knowledge.
5.10. What are the characteristics of the different ways of obtaining validity evidence?
The three major types of evidence include:
(1) Evidence based on content.
(2) Evidence based on internal structure of the test.
(3) Evidence based on relations to other variables.
5.11. What are the purposes and key characteristics of the major types of tests discussed in your textbook?
The major types of tests discussed are:
Intelligence tests (goal is to measure one or more types of intelligence)
Personality tests (goal is to measure one or more dimensions of personality)
Educational assessment tests (including preschool assessment tests for identifying “at risk” children, achievement tests for measuring learning from formal learning experiences, aptitude tests for measuring informal learning that goes on in life, and diagnostic tests for identifying academic difficulties in students).
5.12. What is a good example of each of the major types of tests that are discussed in this chapter?
Some examples intelligence tests are the Stanford-Binet Intelligence Test, the Wechsler Adult Intelligence Scale, the Slosson Intelligence Test.
Some examples of personality tests are the Minnesota Multiphasic Personality Inventory, the California Psychological Inventory, Work Values Inventory, Minnesota School Attitude Survey, and the Thematic Apperception Test.
Some examples of educational assessment tests are Peabody Individual Achievement Test, Nelson Reading Skills Tests, and the Basic English Skills Test.
Reliability is the degree to which an assessment tool produces stable and consistent results.
Types of Reliability
1. Test-retest reliability is a measure of reliability obtained by administering the same test twice over a period of time to a group of individuals. The scores from Time 1 and Time 2 can then be correlated in order to evaluate the test for stability over time.
Example: A test designed to assess student learning in psychology could be given to a group of students twice, with the second administration perhaps coming a week after the first. The obtained correlation coefficient would indicate the stability of the scores.
2. Parallel forms reliability is a measure of reliability obtained by administering different versions of an assessment tool (both versions must contain items that probe the same construct, skill, knowledge base, etc.) to the same group of individuals. The scores from the two versions can then be correlated in order to evaluate the consistency of results across alternate versions.
Example: If you wanted to evaluate the reliability of a critical thinking assessment, you might create a large set of items that all pertain to critical thinking and then randomly split the questions up into two sets, which would represent the parallel forms.
3. Inter-rater reliability is a measure of reliability used to assess the degree to which different judges or raters agree in their assessment decisions. Inter-rater reliability is useful because human observers will not necessarily interpret answers the same way; raters may disagree as to how well certain responses or material demonstrate knowledge of the construct or skill being assessed.
Example: Inter-rater reliability might be employed when different judges are evaluating the degree to which art portfolios meet certain standards. Inter-rater reliability is especially useful when judgments can be considered relatively subjective. Thus, the use of this type of reliability would probably be more likely when evaluating artwork as opposed to math problems.
4. Internal consistency reliability is a measure of reliability used to evaluate the degree to which different test items that probe the same construct produce similar results.
A. Average inter-item correlation is a subtype of internal consistency reliability. It is obtained by taking all of the items on a test that probe the same construct (e.g., reading comprehension), determining the correlation coefficient for each pair of items, and finally taking the average of all of these correlation coefficients. This final step yields the average inter-item correlation.
B. Split-half reliability is another subtype of internal consistency reliability. The process of obtaining split-half reliability is begun by “splitting in half” all items of a test that are intended to probe the same area of knowledge (e.g., World War II) in order to form two “sets” of items. The entire test is administered to a group of individuals, the total score for each “set” is computed, and finally the split-half reliability is obtained by determining the correlation between the two total “set” scores.
Validity refers to how well a test measures what it is purported to measure.
Why is it necessary?
While reliability is necessary, it alone is not sufficient. For a test to be reliable, it also needs to be valid. For example, if your scale is off by 5 lbs, it reads your weight every day with an excess of 5lbs. The scale is reliable because it consistently reports the same weight every day, but it is not valid because it adds 5lbs to your true weight. It is not a valid measure of your weight.
Types of Validity
1. Face Validity ascertains that the measure appears to be assessing the intended construct under study. The stakeholders can easily assess face validity. Although this is not a very “scientific” type of validity, it may be an essential component in enlisting motivation of stakeholders. If the stakeholders do not believe the measure is an accurate assessment of the ability, they may become disengaged with the task.
Example: If a measure of art appreciation is created all of the items should be related to the different components and types of art. If the questions are regarding historical time periods, with no reference to any artistic movement, stakeholders may not be motivated to give their best effort or invest in this measure because they do not believe it is a true assessment of art appreciation.
2. Construct Validity is used to ensure that the measure is actually measure what it is intended to measure (i.e. the construct), and not other variables. Using a panel of “experts” familiar with the construct is a way in which this type of validity can be assessed. The experts can examine the items and decide what that specific item is intended to measure. Students can be involved in this process to obtain their feedback.
Example: A women’s studies program may design a cumulative assessment of learning throughout the major. The questions are written with complicated wording and phrasing. This can cause the test inadvertently becoming a test of reading comprehension, rather than a test of women’s studies. It is important that the measure is actually assessing the intended construct, rather than an extraneous factor.
3. Criterion-Related Validity is used to predict future or current performance - it correlates test results with another criterion of interest.
Example: If a physics program designed a measure to assess cumulative student learning throughout the major. The new measure could be correlated with a standardized measure of ability in this discipline, such as an ETS field test or the GRE subject test. The higher the correlation between the established measure and new measure, the more faith stakeholders can have in the new assessment tool.
4. Formative Validity when applied to outcomes assessment it is used to assess how well a measure is able to provide information to help improve the program under study.
Example: When designing a rubric for history one could assess student’s knowledge across the discipline. If the measure can provide information that students are lacking knowledge in a certain area, for instance the Civil Rights Movement, then that assessment tool is providing meaningful information that can be used to improve the course or program requirements.
5. Sampling Validity (similar to content validity) ensures that the measure covers the broad range of areas within the concept under study. Not everything can be covered, so items need to be sampled from all of the domains. This may need to be completed using a panel of “experts” to ensure that the content area is adequately sampled. Additionally, a panel can help limit “expert” bias (i.e. a test reflecting what an individual personally feels are the most important or relevant areas).
Example: When designing an assessment of learning in the theatre department, it would not be sufficient to only cover issues related to acting. Other areas of theatre such as lighting, sound, functions of stage managers should all be included. The assessment should reflect the content area in its entirety.
What are some ways to improve validity?
1. Make sure your goals and objectives are clearly defined and operationalized. Expectations of students should be written down.
2. Match your assessment measure to your goals and objectives. Additionally, have the test reviewed by faculty at other schools to obtain feedback from an outside party who is less invested in the instrument.
3. Get students involved; have the students look over the assessment for troublesome wording, or other difficulties.
4. If possible, compare your measure with other measures, or data that may be available.
Reliability
As discussed in Chapter 7 , the reliability of a measure is established by testing for both consistency and stability. Consistency indicates how well the items measuring a concept hang together as a set. Cronbach’s alpha is a reliability coefficient that indicates how well the items in a set are positively correlated to one another. Cronbach’s alpha is computed in terms of the average intercorrelations among the items measuring the concept. The closer Cronbach’s alpha is to 1, the higher the internal consistency reliability.
Another measure of consistency reliability used in specific situations is the split-half reliability coefficient. Since this reflects the correlations between two halves of a set of items, the coefficients obtained will vary depending on how the scale is split. Sometimes split-half reliability is obtained to test for consistency when more than one scale, dimension, or factor, is assessed. The items across each of the dimensions or factors are split, based on some predetermined logic (Campbell, 1976 ). In almost every case, Cronbach’s alpha is an adequate test of internal consistency reliability. You will see later in this chapter how Cronbach’s alpha is obtained through computer analysis.
As discussed in Chapter 7 , the stability of a measure can be assessed through parallel form reliability and test–retest reliability. When a high correlation between two similar forms of a measure (see Chapter 7 ) is obtained, parallel form reliability is established. Test–retest reliability can be established by computing the correlation between the same tests administered at two different time periods.
Excelsior Enterprises – checking the reliability of the multi-item measures
Because distributive justice, burnout, job enrichment, and intention to leave were measured with multi-item scales, the consistency of the respondents’ answers to the scale items has to be tested for each measure. In Chapter 7 , we explained that Cronbach’s alpha is a popular test of interitem consistency. Table 11.3 provides an overview of Cronbach’s alpha for the four variables. This table shows that the alphas were all well above 0.60.
In general, reliabilities less than 0.60 are considered to be poor, those in the 0.70 range, acceptable, and those over 0.80 good. Thus, the internal consistency reliability of the measures used in this study can be considered to be acceptable for the job enrichment measure and good for the other measures.
It is important to note that all the negatively worded items in the questionnaire should first be reversed before the items are submitted for reliability tests. Unless all the items measuring a variable are in the same direction, the reliabilities obtained will be incorrect.
A sample of the result obtained for the Cronbach’s alpha test for job enrichment, together with instructions on how it is obtained, is shown in Output 11.3.
The reliability of the job enrichment measure is presented in the first table in Output 11.3. The second table provides an overview of the alphas if we take one of the items out of the measure. For instance, it is shown that if the first item (Jobchar1) is taken out, Cronbach’s alpha of the new three-item measure will be 0.577. This means that the alpha will go down if we take item 1 out of our measure. On the other hand, if we take out item 3, our alpha will go up and become 0.851. Note that, in this case, we would not take out item 3 for two reasons. First, our alpha is above 0.7 so we do not have to take any remedial actions. Second, if we took item 3 out, the validity of our measure would probably decrease. We did not include item 3 for nothing in the original measure!
If, however, our Cronbach’s alpha was too low (under 0.60) then we could use this table to find out which of the items would have to be removed from our measure to increase the interitem consistency. Note that, usually, taking out an item, although improving the reliability of our measure, affects the validity of our measure in a negative way.
TABLE 11.3 Reliability of the Excelsior Enterprises measures.
|
Variable |
Number of items |
Cronbach’s alpha |
|
Distributive justice |
5 |
0.862 |
|
Job enrichment |
4 |
0.715 |
|
Burnout |
10 |
0.806 |
|
Intention to leave |
2 |
0.866 |
Now that we have established that the interitem consistency is satisfactory for perceived equity, job enrichment, burnout, and intention to leave, the scores on the original questions can be combined into a single score. For instance, a new “perceived equity” score can be calculated from the scores on the five individual “perceived equity” items (but only after items 1, 2, and 4 have been reverse coded). Likewise, a new “job enrichment” score can be calculated from the scores on the four individual “job enrichment” items, and so on. We have already explained that this involves calculating the summed score (per case/participant) and then dividing it by the number of items.
Output 11.3 Reliability analysis
From the menus, choose:
Analyze
Scale
Reliability Analysis …
Select the variables constituting the scale.
Choose Model Alpha (this is the default option).
Click on Statistics.
Select Scale if item deleted under Descriptives
Abstract
The rejection of reliability and validity in qualitative inquiry in the 1980s has resulted in an interesting shift for "ensuring rigor" from the investigator’s actions during the course of the research, to the reader or consumer of qualitative inquiry. The emphasis on strategies that are implemented during the research process has been replaced by strategies for evaluating trustworthiness and utility that are implemented once a study is completed. In this article, we argue that reliability and validity remain appropriate concepts for attaining rigor in qualitative research. We argue that qualitative researchers should reclaim responsibility for reliability and validity by implementing verification strategies integral and self-correcting during the conduct of inquiry itself. This ensures the attainment of rigor using strategies inherent within each qualitative design, and moves the responsibility for incorporating and maintaining reliability and validity from external reviewers’ judgements to the investigators themselves. Finally, we make a plea for a return to terminology for ensuring rigor that is used by mainstream science.
Without rigor, research is worthless, becomes fiction, and loses its utility. Hence, a great deal of attention is applied to reliability and validity in all research methods. Challenges to rigor in qualitative inquiry interestingly paralleled the blossoming of statistical packages and the development of computing systems in quantitative research. Simultaneously, lacking the certainty of hard numbers and p values, qualitative inquiry expressed a crisis of confidence from both inside and outside the field. Rather than explicating how rigor was attained in qualitative inquiry, a number of leading qualitative researchers argued that reliability and validity were terms pertaining to the quantitative paradigm and were not pertinent to qualitative inquiry (Altheide & Johnson, 1998; Leininger, 1994). Some suggested adopting new criteria for determining reliability and validity, and hence ensuring rigor, in qualitative inquiry (Lincoln & Guba, 1985; Leininger, 1994; Rubin & Rubin, 1995).
In seminal work in the 1980s, Guba and Lincoln substituted reliability and validity with the parallel concept of "trustworthiness," containing four aspects: credibility, transferability, dependability, and confirmability. Within these were specific methodological strategies for demonstrating qualitative rigor, such as the audit trail, member checks when coding, categorizing, or confirming results with participants, peer debriefing, negative case analysis, structural corroboration, and referential material adequacy (Guba & Lincoln, 1981; Lincoln & Guba, 1985; Guba & Lincoln, 1982). Later, Guba and Lincoln developed authenticity criteria that were unique to the constructivist assumptions and that could be used to evaluate the quality of the research beyond the methodological dimensions (Guba & Lincoln, 1989). While Guba warned that their criteria were "primitive" (Guba, 1981, p. 90), and should be used as a set of guidelines rather than another orthodoxy (Guba & Lincoln, 1982), aspects of their criteria have, in fact, been fundamental to development of standards used to evaluate the quality of qualitative inquiry.
Thus, over the past two decades, reliability and validity have been subtly replaced by criteria and standards for evaluation of the overall significance, relevance, impact, and utility of completed research. Strategies to ensure rigor inherent in the research process itself were backstaged to these new criteria to the extent that, while they continue to be used, they are less likely to be valued or recognized as indices of rigor.
While researchers have continued to use the terminology of reliability and validity in qualitative inquiry in Great Britain and Europe, those who do so in North America are a minority voice. These few authors argue that the broad and abstract concepts of reliability and validity can be applied to all research because the goal of finding plausible and credible outcome explanations is central to all research (Hammersley, 1992; Kuzel & Engel, 2001; Yin, 1994). We are concerned, nonetheless, that the focus on evaluation strategies that lie outside core research procedures results in a deemphasis on strategies built into each phase of the research strategies that can act as a self-correcting mechanism to ensure the quality of the project.
This is an important issue and must be seen as more than just a paradigm debate. We suggest that by focusing on strategies to establish trustworthiness (Guba and Lincoln’s 1981 term for rigor1) at the end of the study, rather than focusing on processes of verification during the study, the investigator runs the risk of missing serious threats to the reliability and validity until it is too late to correct them.
This shift from constructive (during the process) to evaluative (post hoc) procedures occurred subtly and incrementally. Now, there is often no distinction between procedures that determine validity in the course of inquiry and those that provide research outcomes with such credentials. The literature on validity has become muddled to the point of making it unrecognizable, as Wolcott notes: "Whatever validity is, I apparently ‘have’ or ‘get’ or ‘satisfy’ or ‘demonstrate’ or ‘establish’ it. . ." (Wolcott, 1990, p. 121). We are also concerned that by refusing to acknowledge the centrality of reliability and validity in qualitative methods, qualitative methodologists have inadvertently fostered the default notion that qualitative research must therefore be unreliable and invalid, lacking in rigor, and unscientific (Morse, 1999). For the past two decades, qualitative researchers have complained of difficulty in getting funding and difficulty in getting published, and of being ignored by policy makers and practitioners. We suggest qualitative findings are still not regarded as solid empirical research. The purpose of this article is to reestablish reliability and validity as appropriate to qualitative inquiry; to identify the problems created by post hoc assessments of qualitative research; to review general verification strategies in relation to qualitative research, and to discuss the implications of returning the responsibility for the attainment of reliability and validity to the investigator.
Reliability and Validity
Guba and Lincoln (1981) stated that while all research must have "truth value", "applicability", "consistency", and "neutrality" in order to be considered worthwhile, the nature of knowledge within the rationalistic (or quantitative ) paradigm is different from the knowledge in naturalistic (qualitative) paradigm. Consequently, each paradigm requires paradigm-specific criteria for addressing "rigor" (the term most often used in the rationalistic paradigm) or "trustworthiness", their parallel term for qualitative "rigor". They noted that, within the rationalistic paradigm, the criteria to reach the goal of rigor are internal validity, external validity, reliability, and objectivity. On the other hand, they proposed that the criteria in the qualitative paradigm to ensure "trustworthiness" are credibility, fittingness, auditability, and confirmability (Guba & Lincoln, 1981). These criteria were quickly refined to credibility, transferability, dependability, and confirmability (Lincoln & Guba, 1985). They recommended specific strategies be used to attain trustworthiness such as negative cases, peer debriefing, prolonged engagement and persistent observation, audit trails and member checks. Also important were characteristics of the investigator, who must be responsive and adaptable to changing circumstances, holistic, having processional immediacy, sensitivity, and ability for clarification and summarization (Guba & Lincoln, 1981).
These authors were rapidly followed by others either using Guba and Lincolns’ criteria (e.g., Sandelowski, 1986) or suggesting different labels to meet similar goals or criteria (see Whittemore, Chase, & Mandle, 2001). This resulted in a plethora of terms and criteria introduced for minute variations and situations in which rigor could be applied. Presently, this situation is confusing and has resulted in a deteriorating ability to actually discern rigor. Perhaps as a result of this lack of clarity, standards were introduced in the 1980’s for the post hoc evaluation of qualitative inquiry (see Creswell, 1997; Frankel, 1999; Hammersley, 1992; Howe & Eisenhardt, 1990; Lincoln, 1995; Popay, Rogers & Williams, 1998;
Thorne, 1997).
Standards
While standards are a comprehensive approach to evaluating the research as a whole, they remain primarily reliant on procedures or checks by reviewers to be used following completion of the research. They represent either a minimally accepted level or an unobtainable gold standard for the researcher in the field. Subsequent clashes between the "ideal" and the "real" in the attainment of each standard are sometimes unavoidable. Those who evaluate completed research often forget that decisions that greatly influence the quality of the finished product may have, of necessity, been made quickly in the field
without the privilege of knowing the overall research outcome or without being able to see the ramifications of such a decision. Using standards, therefore, is a judgement of the relative worth of the research applied on completion of the project at a time when it is too late to correct problems that result in a poor rating.
Problems with post-hoc evaluation
Using standards for the purpose of post-hoc evaluation is to determine the extent to which the reviewers have confidence in the researcher’s competence in conducting research following established norms.
Rigor is supported by tangible evidence using audit trails, member checks, memos, and so forth. If the evaluation is positive, one assumes that the study was rigorous. We challenge this assumption and suggest that these processes have little to do with the actual attainment of reliability and validity. Contrary to current practices, rigor does not rely on special procedures external to the research process itself. For example, audit trails may be kept as proof of the decisions made throughout the project, but they do little to identify the quality of those decisions, the rationale behind those decisions, or the responsiveness and sensitivity of the investigator to data. Of importance, an audit trail is of little use for identifying or justifying actual shortcomings that have impaired reliability and validity. Thus, they can neither be used to guide the research process nor to ensure an excellent product, but only to document the course of development of the completed analysis.
Further, although Guba and Lincoln (1981) described member checks as a continuous process during data analysis (for example, by asking participants about hypothetical situations) this has largely been interpreted and used by researchers as verification of the overall results with participants. While it is an attractive idea to return the results to the original participants for verification, it is actually not a verification strategy. In fact, several methodologists (Hammersley, 1992; Morse, 1998), including Guba and Lincoln (1981), have warned against the tendency to define verification in terms of whether readers, participants, or potential users of the research judge the analysis to be correct, stating that it is actually more often a threat to validity.
The problem of member checks is that, with the exception of case study research and some narrative inquiry, study results have been synthesized, decontextualized, and abstracted from (and across) individual participants, so there is no reason for individuals to be able to recognize themselves or their particular experiences (Morse, 1998; Sandelowski, 1993). Investigators who want to be responsive to the particular concerns of their participants may be forced to restrain their results to a more descriptive level in order to address participants’ individual concerns. Therefore, member checks may actually invalidate the work of the researcher and keep the level of analysis inappropriately close to the data. The result is that there is presently no distinction between procedures that determine validity during the course of inquiry, and those that provide the research with such credentials on completion of the project (Wolcott, 1994).
Moreover, we suggest that the terms reliability and validity remain pertinent in qualitative inquiry and should be maintained. We are concerned that introducing parallel terminology and criteria marginalizes qualitative inquiry from mainstream science and scientific legitimacy. Morse (1999) argues that, rather than clarifing, the development of alternative criteria actually undermines the issue of rigor.
Compounding the problem of duplicate terminology is the trend to treat standards, goals, and criteria synonymously, and the criterion adopted by one qualitative researcher may be stated as a goal by another scholar. For example, Yin (1994) describes trustworthiness as a criterion to test the quality of research design, while Guba and Lincoln (1989) refer to it as a goal of the research. Later, researchers followed Guba and Lincoln’s 1989 shift toward post hoc evaluation, developing criteria as standards for evaluating
the worth of a project or as evidence that rigor had been attended to in the research process (see, for example, Popay, Rogers & Williams, 1998).
We are concerned that, in the time since Guba and Lincoln developed their criteria for trustworthiness, there has been a tendency for qualitative researchers to focus on the tangible outcomes of the research (which can be cited at the end of a study) rather than demonstrating how verification strategies were used to shape and direct the research during its development. While strategies of trustworthiness may be useful in attempting to evaluate rigor, they do not in themselves ensure rigor. While standards are useful
for evaluating relevance and utility, they do not in themselves ensure that the research will be relevant and useful.
It is time to reconsider the importance of verification strategies used by the researcher in the process of inquiry so that reliability and validity are actively attained, rather than proclaimed by external reviewers on the completion of the project. We argue that strategies for ensuring rigor must be built into the qualitative research process per se. These strategies include investigator responsiveness, methodological coherence, theoretical sampling and sampling adequacy, an active analytic stance, and saturation. These strategies, when used appropriately, force the researcher to correct both the direction of the analysis and the development of the study as necessary, thus ensuring reliability and validity of the completed project.
The Nature of Verification in Qualitative Research
Verification is the process of checking, confirming, making sure, and being certain. In qualitative research, verification refers to the mechanisms used during the process of research to incrementally contribute to ensuring reliability and validity and, thus, the rigor of a study. These mechanisms are woven into every step of the inquiry to construct a solid product (Creswell, 1997; Kvale, 1989) by identifying and correcting errors before they are built in to the developing model and before they subvert the analysis. If the principles of qualitative inquiry are followed, the analysis is self-correcting. In other words, qualitative research is iterative rather than linear, so that a good qualitative researcher moves back and forth between design and implementation to ensure congruence among question formulation, literature, recruitment, data collection strategies, and analysis. Data are systematically checked, focus is maintained, and the fit of data and the conceptual work of analysis and interpretation are monitored and confirmed constantly. Verification strategies help the researcher identify when to continue, stop or modify the research process in order to achieve reliability and validity and ensure rigor.
While much has been written about the use of these strategies in various methods, the literature has focused on "how to do" rather than the contribution that these strategies make in optimizing the research outcome. In actual fact, it is the analytical work of the investigator that underlies these strategies that ensures their effectiveness. For example, many research decisions may underlie the sampling selection, which requires responsiveness to the needs of developing variation, verification, and the developing theory.
Investigator Responsiveness
Research is only as good as the investigator. It is the researcher’s creativity, sensitivity, flexibility and skill in using the verification strategies that determines the reliability and validity of the evolving study. For example, ongoing analysis results in the dynamic formulation of conjectures and questions that force purposive sampling. The researcher analyses the data, which would then determine future participant recruitment. Within the notions of categorization and saturation lie sampling strategies to ensure replication and confirmation.
Responsiveness of the investigator to whether or not the categorization scheme actually holds (and is kept), or appears thin and muddled (and the scheme is changed), influences the outcome. In this way, it is essential that the investigator remain open, use sensitivity, creativity and insight, and be willing to relinquish any ideas that are poorly supported regardless of the excitement and the potential that they first appear to provide. It is these investigator qualities or actions that produce social inquiry and are crucial to the attainment of optimal reliability and validity.
The lack of responsiveness of the investigator at all stages of the research process is the greatest hidden threat to validity and one that is poorly detected using post hoc criteria of "trustworthiness." Lack of responsiveness of the investigator may be due to lack of knowledge, overly adhering to instructions rather than listening to data, the inability to abstract, synthesize or move beyond the technicalities of data coding, working deductively (implicitly or explicitly) from previously held assumptions or a theoretical framework, or following instructions in a rote fashion rather than using them strategically in decision making.
Verification Strategies
Within the conduct of inquiry itself, verification strategies that ensure both reliability and validity of data are activities such as ensuring methodological coherence, sampling sufficiency, developing a dynamic relationship between sampling, data collection and analysis, thinking theoretically, and theory development2. Each of these will be discussed briefly.
First, the aim of methodological coherence is to ensure congruence between the research question and the components of the method. The interdependence of qualitative research demands that the question match the method, which matches the data and the analytic procedures. As the research unfolds, the process may not be linear. Data may demand to be treated differently so that the question may have to be changed or methods modified. Sampling plans may be expanded or change course altogether. The fit of these components with data to meet the analytic goals must be coherent, with each verifying the previous component and the methodological assumptions as a whole.3
Second, the sample must be appropriate, consisting of participants who best represent or have knowledge of the research topic. This ensures efficient and effective saturation of categories, with optimal quality data and minimum dross. Sampling adequacy, evidenced by saturation and replication (Morse, 1991), means that sufficient data to account for all aspects of the phenomenon have been obtained. Seeking negative cases is essential, ensuring validity by indicating aspects of the developing analysis that are initially less than obvious. By definition, saturating data ensures replication in categories; replication verifies, and ensures comprehension and completeness.4
Third, collecting and analyzing data concurrently forms a mutual interaction between what is known and what one needs to know. This pacing and the iterative interaction between data and analysis (as discussed earlier) is the essence of attaining reliability and validity.
The fourth aspect is thinking theoretically. Ideas emerging from data are reconfirmed in new data; this gives rise to new ideas that, in turn, must be verified in data already collected. Thinking theoretically requires macro-micro perspectives, inching forward without making cognitive leaps, constantly checking and rechecking, and building a solid foundation.
Lastly, the aspect of theory development is to move with deliberation between a micro perspective of the data and a macro conceptual/theoretical understanding. In this way, theory is developed through two mechanisms: (1) as an outcome of the research process, rather than being adopted as a framework to move the analysis along; and (2) as a template for comparison and further development of the theory. Valid theories are well developed and informed, they are comprehensive, logical, parsimonious, and consistent (see Glaser 1978; Morse 1997).5
Together, all of these verification strategies incrementally and interactively contribute to and build reliability and validity, thus ensuring rigor. Thus, the rigor of qualitative inquiry should thus be beyond question, beyond challenge, and provide pragmatic scientific evidence that must be integrated into our developing knowledge base.
Discussion
We challenge the prevailing notion that the danger of using the generic term "validity" is that a particular method, for example ethnography, will be derailed from its philosophical underpinnings (Hammersley, 1992). Our argument is based on the premise that the concepts of reliability and validity as overarching constructs can be appropriately used in all scientific paradigms because, as Kvale (1989) states, to validate is to investigate, to check, to question, and to theorize. All of these activities are integral components of qualitative inquiry that insure rigor. Whether quantitative or qualitative methods are used, rigor is a desired goal that is met through specific verification strategies. While different strategies are used for each paradigm, the term validity is the most pertinent term for these processes. We advocate a return to Guba’s (1981) early writings before Guba and Lincoln (1981) substituted trustworthiness for the qualitative paradigm. While this term bridges both reliability and validity concepts, the criteria they suggest still do not apply to all qualitative methods. For instance, Guba and Lincoln’s confirmability is not pertinent to phenomenology, nor for postmodern philosophies such as feminism and critical theory in which the investigator’s experience becomes part of data, and which perceive reality as dynamic and changing.
We argue for a return to validity as a means for obtaining rigor through using techniques of verification. Verification takes into account the varying philosophical perspectives inherent in qualitative inquiry, thus, the strategies used will be specific to, and inherent in, each methodological approach. At the same time, the terminology remains consistent with science.
Refocusing the qualitative research process to verification strategies is not without profound implications. It will, for example, enhance researcher’s responsiveness to data and constantly remind researchers to be proactive, and take responsibility for rigor.6 Student projects, although necessarily smaller in scope, must also be responsive to rigor. We are concerned that in order for projects to be manageable within the constraints of student time-frames, abilities and budgets, rigor is seriously undermined by the narrow delimiting of the topics. We recommend that major concepts be verified and others left "hypothetical", rather than the student working with incomplete, thin data sets. 7 Such strategies will enable students to assume projects small in scope but with the depth required by qualitative inquiry and thereby gain the grounding experience necessary to become an excellent researcher. Attending to rigor throughout the research process will have important ramifications for qualitative inquiry. Rather than relegating rigor to one section of a post hoc reflection on the finished work (such as stating that an audit trail was maintained, that member checks were done, or that the researcher was "reflective"), verification and attention to rigor will be evident in the quality of the text. Excellent inquiry is stunning: the arguments are sophisticated in that they are complex yet elegant, focused yet profound, surprising yet obvious.
In summary, we need to refocus our agenda for ensuring rigor and place responsibility with the investigator rather than external judges of the completed product. We need to return to recognizing and trusting the strategies within qualitative inquiry that ensure rigor. For too long, we have used the wrong tactics to defend qualitative inquiry. It is time to attend to our own research and work toward finding
consensus in broader criteria, appreciating how it is attained within the qualitative project itself, using criteria and terminology that is used in mainstream science.
Regardless of the standard or criteria used to evaluate the goal of rigor, our problem remains the same: they are applied after the research is completed, and therefore are used to judge of quality. Standards and criteria applied at the end of the study cannot direct the research as it is conducted, and thus cannot be used pro-actively to manage threats to reliability and validity.
References
Altheide, D., & Johnson, J. M. C. (1998). Criteria for assessing interpretive validity in qualitative research. In N. K. Denzin & Y. S. Lincoln (Eds.), Collecting and interpreting qualitative materials.(pp. 283-312). Thousand Oaks, CA: Sage.
Creswell, J. W. (1997). Qualitative inquiry and research design: Choosing among five traditions. Thousand Oaks, CA: Sage.
Frankel, R. (1999). Standards of qualitative research. In B. F. Crabtree & W. L. Miller (Eds.) Doing qualitative research (2nd ed). (pp. 333-346). Thousand Oaks, CA: Sage.
Glaser, B. G. (1978). Theoretical sensitivity. Mill Valley, CA: Sociology Press.
Guba, E. G. (1981). Criteria for assessing the trustworthiness of naturalistic inquiries, Educational Communication and Technology Journal, 29 (2), 75-91.
Guba, E. G., & Lincoln, Y. S. (1981). Effective evaluation: Improving the usefulness of evaluation results through responsive and naturalistic approaches. San Francisco, CA: Jossey-Bass.
Guba, E. G., & Lincoln, Y. S. (1982). Epistemological and methodological bases of naturalistic inquiry. Educational Communication and Technology Journal 30 (4), 233-252.
Guba, E. G., & Lincoln, Y. S. (1989). Fourth generation evaluation. Newbury Park, CA: Sage. Hammersley, M. (1992). What’s wrong with ethnography? London: Routledge.
Howe, K., & Eisenhardt, M. (1990). Standards for qualitative (and quantitative) research: A prolegomenon. Educational Researcher, 19 (4), 2-9.
Kuzel, A. & Engel, J. (2001). Some pragmatic thought on evaluating qualitative health research.. In J. Morse, J. Swanson, & A. Kuzel (Eds.), The Nature of Qualitative Evidence (pp. 114- 138).Thousand Oaks, CA: Sage.
Kvale, S. (1989). Issues of validity in qualitative research. Lund, Sweden: Chartwell Bratt.
Leininger, M. (1994). Evaluation criteria and critique of qualitative research studies. In J. M. Morse (Ed.), Critical Issues in Qualitative Research Methods. Newbury Park, CA: Sage.
Lincoln, Y. S. (1995). Emerging criteria for quality in qualitative and interpretive research. Qualitative Inquiry, 1, 275-289.
Lincoln, Y. S. & Guba, E. G. (1985). Naturalistic inquiry. Beverly Hills, CA: Sage.
Meadows, L. & Morse, J. M. (2001). Constructing evidence within the qualitative project. In J. M. Morse,
J. Swanson & A. Kuzel, (Eds.). The nature of evidence in qualitative inquiry. (pp. 187 — 202),Thousand oaks, CA: Sage.
Morse, J. M. (1991). Strategies for sampling. In J. Morse (Ed.), Qualitative nursing research: A contemporary dialogue (Rev. Ed.). (pp. 117-131). Newbury Park, CA: Sage.
Morse, J. M. (1997). "Perfectly healthy, but dead": The myth of inter-rater reliability. [Editorial] Qualitative Health Research, 7, 445-447.
Morse, J. M. (1998). Validity by committee. [Editorial] Qualitative Health Research, 8, 443-445.
Morse, J. M. (1999). Myth #93: Reliability and validity are not relevant to qualitative inquiry. Qualitative Health Research, 9, 717.
Popay, J., Rogers, A. & Williams, G. (1998). Rationale and standards for the systematic review of qualitative literature in health services research. Qualitative Health Research, 8, 341-351.
Rubin, H. J. & Rubin, I. S. (1995). Qualitative interviewing: The art of hearing data. Thousand Oaks, CA: Sage.
Sandelowski, M. (1986). The problem of rigor in qualitative research. Advances in Nursing Science, 8 (3), 27 — 37.
Sandelowski, M. (1993). Rigor or rigor mortis: The problem of rigor in qualitative research revisited. Advances in Nursing Science, 16 (2), 1— 8.
Seale, C. (1999). The quality of qualitative research. London: Sage.
Thorne, S. (1997). The art (and science) of critiquing qualitative research. In J. Morse (Ed.), Completing a qualitative project: Details and dialogue, (p. 117 — 132). Thousand Oaks, CA: Sage.
Wolcott, H. (1990). On seeking–and rejecting–validity in qualitative research. In E. W. Eisner & A. Peshkin, (Eds.), Qualitative Inquiry in Education: The Continuing Debate.(pp. 121-152. New York: Teachers College Press.
Wolcott, H. (1994). Transforming qualitative data: Description analysis and interpretation. Thousand Oaks, CA: Sage.
Whittemore, R., Chase, S. K., & Mandle, C. L. (2001). Validity in qualitative research. Qualitative Health Research, 11,.117-132.
Yin, R. K. (1994). Discovering the future of the case study method in evaluation research. Evaluation Practice, 15, 283-290.
·
References
American Educational Research Association, American Psychological Association, &
National Council on Measurement in Education. (1985). Standards for educational and psychological testing . Washington, DC: Authors.
Cozby, P.C. (2001). Measurement Concepts. Methods in Behavioral Research (7th ed.).
California: Mayfield Publishing Company.
Cronbach, L. J. (1971). Test validation. In R. L. Thorndike (Ed.). Educational
Measurement (2nd ed.). Washington, D. C.: American Council on Education.
Moskal, B.M., & Leydens, J.A. (2000). Scoring rubric development: Validity and
reliability. Practical Assessment, Research & Evaluation, 7(10). [Available online: http://pareonline.net/getvn.asp?v=7&n=10].
The Center for the Enhancement of Teaching. How to improve test reliability and
validity: Implications for grading. [Available online: http://oct.sfsu.edu/assessment/evaluating/htmls/improve_rel_val.html].
