N494-M2
JS79
Melnyk_PPT_Chapter_05.pptCopyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
Chapter 5
Critically Appraising Quantitative Evidence for Clinical Decision Making
*
Copyright © 2015 Wolters Kluwer • All Rights Reserved
Preliminary Questions to Ask Prior to Critical Appraisal of Quantitative Studies
1. Why was the study done? Is there a succinctly stated explanation of why the study was carried out? This may be elaborated on in the aims of the study.
2. What is the sample size? Is a power analysis reported in the methods section of the report?
*
Copyright © 2015 Wolters Kluwer • All Rights Reserved
Preliminary Questions to Ask Prior to Critical Appraisal of Quantitative Studies—(cont.)
3. Are the measurements of major variables valid and reliable? Are there descriptions of how it is assured that the instruments used are measuring what is purported to be measured (validity)? Can it be determined from the report that instruments used will perform the same way each time they are used under the same conditions (reliability)?
4. How were the data analyzed? Is there a clear description of how data were analyzed and rationale given for the methods used?
*
Copyright © 2015 Wolters Kluwer • All Rights Reserved
Preliminary Questions to Ask Prior to Critical Appraisal of Quantitative Studies—(cont.)
5. Were there any untoward events during the conduct of the study? Were explanations provided for all adverse events and withdrawals from the study and how those events might affect the final results?
6. How do the results fit with previous research in the area? Does the discussion section discuss the study findings in the light of what is already known and how those findings complement or contradict previous work?
*
Copyright © 2015 Wolters Kluwer • All Rights Reserved
Preliminary Questions to Ask Prior to Critical Appraisal of Quantitative Studies—(cont.)
7. What does this research mean for clinical practice? Are the results meaningful for clinical practice? Can the results be extrapolated to patients in other clinical settings?
*
Copyright © 2015 Wolters Kluwer • All Rights Reserved
Study Versus Instrument Validity
- Study validity refers to whether the study results can be “believed” (internal validity) and whether the results can be generalized to other settings/patient populations (external validity)
- Instrument validity refers to whether the study measurements are measuring what they are supposed to be measuring
*
Copyright © 2015 Wolters Kluwer • All Rights Reserved
Study Versus Instrument Reliability
- Study reliability refers to the “repeatability” of the study—that is, if the study were repeated under the same circumstances, would the results be the same?
- Instrument reliability refers to the consistency of the instruments—will they yield the same results under the same conditions?
*
Copyright © 2015 Wolters Kluwer • All Rights Reserved
Common Statistical Errors to Watch for During Preliminary Study Appraisal
- The statistical test was chosen because it gave the answer which the investigator predicted
- Several rounds of data analyses were conducted on the same data
- Statistical significance is confused with clinical significance (or vice versa)
- Missing data with no explanation
- Selective reporting of only significant findings
*
Copyright © 2015 Wolters Kluwer • All Rights Reserved
Critical Appraisal of Quantitative Studies
- The critical appraisal process focuses on three broad questions:
- Are the results of the study valid? (Validity)
- What are the results? (Reliability)
- Will the results help me in caring for my patients? (Applicability)
*
Copyright © 2015 Wolters Kluwer • All Rights Reserved
Are the Results of the Study Valid?
Study bias
- Anything that distorts study findings in a systematic way arising from the methodology of the study
- Can compromise the validity of the findings
*
Copyright © 2015 Wolters Kluwer • All Rights Reserved
Are the Results of the Study Valid?— (cont.)
Types of study bias
1. Selection bias: Can occur if participants are selectively assigned to groups (e.g., the “sicker” patients are assigned the control group). This is reduced when researchers randomly assign participants to groups.
2. Knowledge of who receives the intervention: Can occur if subjects or those measuring outcomes know subject group assignment. This is reduced by blinding, in which subjects and those measuring outcomes do not know the group assignments.
*
Copyright © 2015 Wolters Kluwer • All Rights Reserved
Are the Results of the Study Valid?— (cont.)
Types of study bias
3. Gatekeeping: Can occur in convenience sampling when patients may be chosen because they are more likely to volunteer, resulting in a sample not representative of the target population
4. Measurement bias: Can occur if instruments are incorrectly calibrated (thus consistently producing higher or lower measurements) or if data collectors deviate from established data collection protocols (especially if observation is being used or psychological variables are being measured)
*
Copyright © 2015 Wolters Kluwer • All Rights Reserved
Are the Results of the Study Valid?— (cont.)
Types of study bias
5. Recall bias: Can occur when subjects are asked to recall past actions or events (such as in case–control studies). Subjects may give answers that are “socially acceptable” or that they “think” is what happened.
6. Information bias: Can occur in longitudinal cohort studies if participants know the subject of the study and “act differently” (e.g., seek out information and act on it) if they know that they are in the group that has been “exposed” to the condition being studied
*
Copyright © 2015 Wolters Kluwer • All Rights Reserved
Are the Results of the Study Valid?— (cont.)
Types of study bias
7. Loss to follow up: Attrition could occur due to unforeseen side effects of the intervention or burdensome data collection procedures, leading to noncomparable groups
8. Contamination: Can occur if intervention and control groups have interaction and information is shared, especially if the intervention is educational in nature
*
Copyright © 2015 Wolters Kluwer • All Rights Reserved
Are the Results of the Study Valid?— (cont.)
Confounding variables
- Study results may be confounded when a relationship between two variables is actually due to a third unknown or unconsidered variable
- Most often encountered in studies about lifestyle and health
- In longitudinal studies, may be a result of history or maturation
*
Copyright © 2015 Wolters Kluwer • All Rights Reserved
Are the Results of the Study Reliable?
Main concerns
- Size of the intervention’s effect (the effect size)
- Precision in effect measurement
*
Copyright © 2015 Wolters Kluwer • All Rights Reserved
Are the Results of the Study Reliable?—(cont.)
Questions to be asked
1. Do the numbers add up? Does the sum of all n values equal the original N reported?
*
Copyright © 2015 Wolters Kluwer • All Rights Reserved
Are the Results of the Study Reliable?—(cont.)
*
Copyright © 2015 Wolters Kluwer • All Rights Reserved
Are the Results of the Study Reliable?—(cont.)
Questions to be asked
2. What is the magnitude of the effect?
- The magnitude of effect is the degree of the difference in effect (or its lack) between study groups
- Effect is the rate of occurrence in each group for the outcome of interest
- Statistical tests are usually conducted to determine whether the effects differ significantly between/among groups
*
Copyright © 2015 Wolters Kluwer • All Rights Reserved
Are the Results of the Study Reliable?—(cont.)
Questions to be asked
3. What is the strength of the association?
- Absolute risk reduction (ARR): When risk of an undesirable outcome is less for treatment than for control group
- Absolute risk increase (ARI): When risk of an undesirable outcome is more for treatment than for control group
- Relative risk (RR): Likelihood that outcome will occur in one group versus the other
- Odds ratio (OR): Odds of treatment group having the outcome
*
Copyright © 2015 Wolters Kluwer • All Rights Reserved
Are the Results of the Study Reliable?—(cont.)
Questions to be asked
4. What is the clinical significance? Are the results clinically meaningful?
5. What is the precision of the measurement of effect?
- The extent to which random error may influence a measurement may be reported using statistical significance (p values) or by confidence intervals (CIs)
*
Copyright © 2015 Wolters Kluwer • All Rights Reserved
Question
The question that best addresses study reliability is:
- How big is the magnitude of the effect?
- Did the investigator(s) applying the intervention know to which groups subjects were assigned?
- Were the instruments calibrated according to manufacturer specifications?
- How many subjects were lost to follow-up and what groups were they in?
*
Copyright © 2015 Wolters Kluwer • All Rights Reserved
Answer
a. How big is the magnitude of the effect?
Rationale: Magnitude of effect allows for determination of the certitude of results in similar patient populations (generalizability). Answers “b” and “d” are questions related to the validity of the study. Answer “c” is related to instrument reliability, not study reliability.
*
Copyright © 2015 Wolters Kluwer • All Rights Reserved
Appraising Case Studies
Disadvantages
- Ranked lower in the hierarchy of quantitative evidence
- Based on one or few clinical cases, therefore, not reliable as a sole source of evidence
- Unable to make generalizations to general population
Advantages
- May alert clinicians to new issues and rare and/or adverse events in practice
- Assists in hypothesis generation
*
Copyright © 2015 Wolters Kluwer • All Rights Reserved
Appraising Case–Control Studies
Studies in which a sample is categorized as having a condition (case) or not (control). A retrospective investigation is performed to ascertain the presence of presumed causes.
- Validity questions: How were the cases obtained; were appropriate controls selected; and were data collection methods the same for the cases and controls?
- Reliability questions: What were the results and how were they reported?
- Applicability questions: Is the sample similar to my patient(s)? Is more evidence needed before making a decision?
*
Copyright © 2015 Wolters Kluwer • All Rights Reserved
Appraising Cohort Studies
Studies in which a sample exposed to a treatment or condition is followed over time for a presumed outcome. There may not be a comparison group.
- Validity questions: If a comparison group was used, was this group similar to the exposed group and was follow-up begun at a similar time point? Was follow-up sufficiently long and complete? Were objective and unbiased outcome criteria used? Did the analysis adjust for important prognostic risk factors and confounding variables?
*
Copyright © 2015 Wolters Kluwer • All Rights Reserved
Appraising Cohort Studies—(cont.)
- Reliability questions: What was the magnitude of the relationship between treatment/exposure and targeted outcome? How likely was the outcome to be seen within the time of study follow-up? How precise were the study estimates?
- Applicability questions: Were the study patients similar to my own? Will the results lead directly to selecting or avoiding therapy? Could the results be used to counsel patients?
*
Copyright © 2015 Wolters Kluwer • All Rights Reserved
Appraising Randomized Controlled Trials (RCTs)
The most appropriate research design to establish cause and effect through the use of controlled methodology:
- Validity questions: Were the subjects randomly assigned to the experimental and control groups? Were subjects and providers kept blind to group assignment? Were there plausible reasons as to why subjects did not complete the study? Were follow-up assessments performed long enough after the intervention to fully study the effects? Were subjects analyzed in the group to which they were assigned? Was the control appropriate? Were instruments valid and reliable? Were subjects in each group similar on demographic and baseline variables?
*
Copyright © 2015 Wolters Kluwer • All Rights Reserved
Appraising Randomized Controlled Trials (RCTs)—(cont.)
- Reliability questions: How large was the intervention or treatment effect? How precise was the intervention or treatment effect?
- Applicability questions: Were all clinically important outcomes measured? What were the risk/benefits of the treatment? Was the treatment feasible in my clinical setting? Will the treatment be in alignment with my patient(s)’ values and expectations?
*
Copyright © 2015 Wolters Kluwer • All Rights Reserved
Question
A team of researchers has received a grant to investigate the potential links between diet and the development of stomach cancer. What research design is most likely to answer the researchers’ clinical question?
- Case–control
- Case study
- Randomized controlled trial (RCT)
- Qualitative study
*
Copyright © 2015 Wolters Kluwer • All Rights Reserved
Answer
a. Case–control
Rationale: A case–control study selects individuals with a particular disease (e.g., stomach cancer) and looks back to identify factors that may underlie that disease (e.g., diet). Neither a case study nor a qualitative study would inform this relationship and an RCT would be unethical and impractical.
*
Copyright © 2015 Wolters Kluwer • All Rights Reserved
Appraising Systematic Reviews (SRs)
Systematic reviews of RCTs are produced by a rigorous search strategy, including determination of study inclusion and exclusion criteria, resulting in answering a clinical question:
- Validity questions: Are the studies contained in the SR all RCTs? Is a detailed description of the search strategy given? Does the SR describe how study validity was assessed? Were the results consistent across studies? Were individual patient data or aggregate data used in the analysis?
*
Copyright © 2015 Wolters Kluwer • All Rights Reserved
Appraising Systematic Reviews (SRs)—(cont.)
- Reliability questions: How large was the treatment effect? How precise was the intervention or treatment effect (CI)?
*
Copyright © 2015 Wolters Kluwer • All Rights Reserved
Appraising Systematic Reviews (SRs)—(cont.)
- Applicability questions: Are my patients similar to the ones included in the review? Is it feasible to implement the findings in my practice setting? Were all clinically important outcomes considered, including risks and benefits of the treatment? Are there any contraindications or circumstances that would inhibit me from implementing the treatment? What are my patient’s and his or her family’s preferences and values about the treatment under consideration?
*
Copyright © 2015 Wolters Kluwer • All Rights Reserved
Evaluation and Synthesis of Quantitative Studies
- Once studies are selected, each study should be evaluated for validity, reliability, and applicability for use in the synthesis
- Synthesis occurs as the study data are entered into the evaluation table and the findings are compared and contrasted
*
Copyright © 2015 Wolters Kluwer • All Rights Reserved
Question
Is the following statement true or false?
The main difference between an evaluation table and a synthesis table is that evaluation tables are used to compare and contrast studies that are included in a systematic review.
*
Copyright © 2015 Wolters Kluwer • All Rights Reserved
Answer
False
Rationale: Evaluation tables are methods by which to “sort” the evidence found in a literature search and to determine whether the study should be included in the systematic review. Synthesis tables are used in the “next step” of comparing and contrasting the studies that have been determined to be valid and reliable.
*
