PICOT appraisal

Hgl120112

Appendix_B_Rapid_Critical_Appraisal_Checklists.pdf

Home >Nursing homework help >PICOT appraisal

Appendix B Rapid Critical Appraisal Checklists

These rapid critical appraisal (RCA) checklists have been developed over several years and are

designed to help you strive for mastery of the RCA phase of critical appraisal of evidence. The

questions in the first section of the checklists are about what makes good research—and, there-

fore, contribute to determining the validity of a study. The second section of the checklists asks

about information that helps establish study reliability within clinical practice. The third section

of the RCA checklists contains questions about applicability of the study findings to patients.

Becoming proficient at understanding the different aspects within the RCA checklists and con-

sistently using them when appraising evidence will improve clinicians’ understanding of various

types of research methods, designs, statistical analyses, and translation of evidence into practice.

Caveat: Though these resources have been developed over several years, they don’t get old.

Thinking that requires everything to have the latest year on it is flawed thinking. These resources

have been updated as the science of rigorous research methodology grows; however, their es-

sence has not changed from the original intent of helping clinicians find the pearls within a

study that demonstrate the validity, reliability, and applicability of research.

Note: These RCA checklists are also available in MS Word format on for ease of use.

708

Appendix B / Rapid Critical Appraisal Checklists

GENERAL APPRAISAL OVERVIEW FOR ALL STUDIES

Date:

709

Reviewer:

Article Citation (APA):

PICOT Question:

Overview/General Description of Study

• Purpose of Study:

• Study Design:

• General Description of Study:

Research Question(s) or Hypotheses:

Study Aims:

Sampling Technique, Sample Size, and Characteristics:

Major Variables Studies:

• Independent Variable(s):

• Dependent (outcome) Variable(s):

Variable Analysis Used (include whether appropriate to answer research questions/hypothesis or discover themes):

© Fineout-Overholt, 2010. This form was designed to help learners engage research as a fundamental tool in evidence-based decision-making. If you use this form for another use, please contact the author at [email protected].

mailto:[email protected]

710 Appendix B / Rapid Critical Appraisal Checklists

RAPID CRITICAL APPRAISAL QUESTIONS FOR SYSTEMATIC REVIEWS AND META-ANALYSES OF CLINICAL INTERVENTIONS QUESTION

VALIDITY

1. Are the results of the review valid?

a. Are the studies contained in the review randomized controlled trials (RCTs)?

b. If not, were all relevant studies included in the review?

c. Does the review include a detailed description of the search strategy to find all relevant studies?

d. Does the review describe how validity of the individual studies was assessed (e.g., methodological quality, including the use of random assignment to study groups and complete follow-up of the participants)?

e. Were the results consistent across studies?

f. Were individual patient data or aggregate data used in the analysis?

g. Does the review include a description of how studies were compared using statistical analysis?

RELIABILITY

2. What were the results?

a. How large is the intervention or treatment effect (OR, RR, effect size)?

b. How precise is the intervention or treatment (CI)?

APPLICABILITY

Yes No Unknown

Individual Aggregate

Yes No Unknown

3. Will the results assist me in caring for my patients?

a. Are my patients similar to the ones included in the review?

b. Is it feasible to implement the findings in my practice setting?

c. Do the pooled or combined results of the studies support the hospital’s values and goals of service delivery? (i.e., Is it feasible to implement the findings in my practice setting?)

d. Were all clinically important outcomes considered, including risks and benefits of the treatment?

e. What is my clinical assessment of the patient and are there any contraindications or circumstances that would inhibit me from implementing the treatment?

f. What are my patient’s and his or her family’s preferences and values about the treatment that is under consideration?

Yes No Unknown

Would you use the study results in your practice to make a difference in patient outcomes?

• If yes, how? • If yes, why? • If no, why not?

Additional Comments/Reflections:

Recommendation for article use within a body of evidence:

____________________

Appendix B / Rapid Critical Appraisal Checklists 711

RAPID CRITICAL APPRAISAL QUESTIONS FOR RANDOMIZED CLINICAL TRIALS (RCTS)

VALIDITY

1. Are the results of the study valid?

a. Were the participants randomly assigned to the experimental and control groups?

b. Was random assignment concealed from the individuals who were first enrolling participants into the study?

c. Were the participants and providers blind to the study group?

d. Were reasons given to explain why participants did not com- plete the study?

e. Were the follow-up assessments conducted long enough to fully study the effects of the intervention?

f. Were the participants analyzed in the group to which they were randomly assigned?

g. Was the control group appropriate?

h. Were the instruments used to measure the outcomes valid and reliable?

i. Were the participants in each of the groups similar on demo- graphic and baseline clinical variables?

RELIABILITY

2. What are the results?

Yes

No Unknown

a. How large is the intervention or treatment effect (NNT, NNH, effect size?

b. How precise is the intervention or treatment (CI)?

APPLICABILITY

3. Will the results help me in caring for my patients?

a. Were all clinically important outcomes measured? Yes No Unknown

b. What are the risks and benefits of the treatment?

c. Is the treatment feasible in my clinical setting? Yes No Unknown

d. What are my patient’s/family’s values and expectations for the outcome that is trying to be prevented and the treatment itself?

Would you use the study results in your practice to make a difference in patient outcomes?

• If yes, how? • If yes, why? • If no, why not?

Additional Comments/Reflections:

Recommendation for article use within a body of evidence:

712 Appendix B / Rapid Critical Appraisal Checklists

RAPID CRITICAL APPRAISAL QUESTIONS FOR QUASI-EXPERIMENTAL STUDIES Explain your answers and recommendation for use of this study in the body of evidence to

answer your PICOT question.

VALIDITY

1. Are the results of the study valid?

• Study participants in intervention and comparison groups are similar.

• The intervention is clearly identified.

• There is a control group.

• Participants in the comparison group(s) received a reasonable treatment/care to the exposure or inter- vention of interest given to the intervention group.

• Follow-up between groups is adequately de- scribed and analyzed.

• Appropriate statistical analysis was used for the data gathered.

• Measurement of the outcome was obtained pre- and postintervention.

• The outcomes are the same across all groups and were measured with the same instrument.

• Outcomes were measured with valid and reliable instruments.

RELIABILITY

2. What are the results?

• What were the magnitude of the results?

• What was the precision of the results?

APPLICABILITY

Yes No

1 2

Unknown

Rationale/ Comment

3. Will the results help me in caring for my patients?

• Were the study patients similar to my own? 1

• Will the results lead directly to selecting or avoid- 1 ing therapy?

• Are the results useful for reassuring or counseling 1 patients?

Would you use the study results in your practice to make a difference in patient outcomes?

• If yes, how? • If yes, why? • If no, why not?

Additional Comments/Reflections:

Recommendation for article use within a body of evidence:

____________________

713Appendix B / Rapid Critical Appraisal Checklists

RAPID CRITICAL APPRAISAL QUESTIONS FOR COHORT STUDIES

VALIDITY

1. Are the results of the study valid?

a. Was there a representative and well-defined sample of patients at a similar point in the course of the disease?

b. Was follow-up sufficiently long and complete?

c. Were objective and unbiased outcome criteria used?

d. Did the analysis adjust for important prognostic risk factors and confounding variables?

Comments

Yes No Unknown

RELIABILITY

2. What are the results?

a. What is the magnitude of the relationship between predictors (i.e., prognostic indicators) and targeted outcome?

b. How likely is the outcome event(s) in a specified period of time?

c. How precise are the study estimates?

APPLICABILITY

3. Will the results help me in caring for my patients?

a. Were the study patients similar to my own? Yes No Unknown

b. Will the results lead directly to selecting or avoiding therapy? Yes No Unknown

c. Are the results useful for reassuring or counseling patients? Yes No Unknown

Comments

Would you use the study results in your practice to make a difference in patient outcomes?

• If yes, how? • If yes, why? • If no, why not?

Additional Comments/Reflections:

Recommendation for article use within a body of evidence:

714 Appendix B / Rapid Critical Appraisal Checklists

RAPID CRITICAL APPRAISAL QUESTIONS FOR DESCRIPTIVE STUDIES

VALIDITY

1. Are the results of the study valid?

• Were study/survey methods appropriate for the question? Yes No Unknown

• Was sampling methods appropriate for the research question? Yes No Unknown

• Were sample size implications on study results discussed? Yes No Unknown

• Were variables studied appropriate for the question? Yes No Unknown

• Dependent variables are: • Independent (outcome) variables are:

• Were outcomes appropriate for the question?

• Were valid and reliable instruments used to measure outcomes?

• Were the chosen measures appropriate for study outcomes?

• Were outcomes clearly described?

• Did investigators and/or funding agencies declare freedom from conflict of interest?

RELIABILITY

2. What are the results?

Yes No Unknown

• What were the main results of the study?

• Was there statistical significance? Explain.

• Was there clinical significance? Explain.

• Were safety concerns, including adverse events and risk/benefit Yes No Unknown described?

APPLICABILITY

3. Will the results help me in caring for my patients?

• Are the results applicable to my patient population? Yes No Unknown

• Will my patients’ and families’ values and beliefs be supported Yes No Unknown by the knowledge gained from the study?

Would you use the study results in your practice to make a difference in patient outcomes?

• If yes, how? • If yes, why? • If no, why not?

Additional Comments/Reflections:

Recommendation for article use within a body of evidence:

715Appendix B / Rapid Critical Appraisal Checklists

RAPID CRITICAL APPRAISAL QUESTIONS FOR QUALITATIVE EVIDENCE

VALIDITY

1. Are the results of the study valid (i.e., trustworthy and credible)?

a. How were study participants chosen?

b. How were accuracy and completeness of data assured?

c. How plausible/believable are the results?

i. Are implications of the research stated? Yes No Unknown

1. May new insights increase sensitivity to others’ needs? Yes No Unknown

2. May understandings enhance situational competence? Yes No Unknown

d. What is the effect on the reader?

1. Are results plausible and believable? Yes No Unknown

2. Is the reader imaginatively drawn into the experience? Yes No Unknown

RELIABILITY

2. What were the results?

a. Does the research approach fit the purpose of the study? Yes No Unknown

i. How does the researcher identify the study approach? Yes No Unknown

1. Are language and concepts consistent with the approach? Yes No Unknown

2. Are data collection and analysis techniques appropriate? Yes No Unknown

ii. Is the significance/importance of the study explicit? Yes No Unknown

1. Does review of the literature support a need for the study? Yes No Unknown

2. What is the study’s potential contribution?

iii. Is the sampling strategy clear and guided by study needs? Yes No Unknown

1. Does the researcher control selection of the sample? Yes No Unknown

2. Do sample composition and size reflect study needs? Yes No Unknown

b. Is the phenomenon (human experience) clearly identified?

i. Are data collection procedures clear? Yes No Unknown

1. Are sources and means of verifying data explicit? Yes No Unknown

2. Are researcher roles and activities explained? Yes No Unknown

ii. Are data analysis procedures described? Yes No Unknown

1. Does analysis guide direction of sampling and when it ends? Yes No Unknown

2. Are data management processes described? Yes No Unknown

c. What are the reported results (description or interpretation)?

i. How are specific findings presented?

1. Is presentation logical, consistent, and easy to follow? Yes No Unknown

2. Do quotes fit the findings they are intended to illustrate? Yes No Unknown

ii. How are overall results presented?

1. Are meanings derived from data described in context? Yes No Unknown

2. Does the writing effectively promote understanding? Yes No Unknown

(continued )

716 Appendix B / Rapid Critical Appraisal Checklists

APPLICABILITY

3. Will the results help me in caring for my patients?

a. Are the results relevant to persons in similar situations? Yes No Unknown

b. Are the results relevant to patient values and/or circumstances? Yes No Unknown

c. How may the results be applied in clinical practice?

Would you use the study results in your practice to make a difference in patient outcomes?

• If yes, how? • If yes, why? • If no, why not?

Additional Comments/Reflections:

Recommendation for article use within a body of evidence:

Appendix B / Rapid Critical Appraisal Checklists 717

RAPID CRITICAL APPRAISAL QUESTIONS FOR EVIDENCE-BASED PRACTICE (EBP) IMPLEMENTATION OR QUALITY IMPROVEMENT (QI) PROJECTS

Indicate the extent to which the item is met in the published report of the EBP or QI project.

4 5 Validity of Evidence Synthesis (i.e., good methodology)

1 No

2 A Little

3 Somewhat

Quite a Bit

Very Much

1. The title of the publication identifies the re- port/project as an EBP implementation or QI project.

2. The project report provides a structured summary that includes, as applicable, data to establish the existent and background of the clinical issue; inclusion and exclusion criteria and source(s) of evidence; evidence synthesis, objective(s), and setting of the EBP or QI project; project limitations; re- sults/outcomes; and recommendation and implications for policy.

3. Report includes existing internal evidence to adequately describe the clinical issue.

4. Provides an explicit statement of the ques- tion being addressed with reference to participants or population/intervention/ comparison/outcome (PICO).

5. Explicitly describes the search method (i.e., was it systematic), inclusion and exclusion criteria, and rationale for search strategy limits.

6. Describes multiple information sources (e.g., databases, contact with study authors to identify additional studies, or any other additional search strategies) included in the search strategy, and date.

7. States the process for title, abstract, and article screening for selecting studies.

8. Describes the method of data extraction (e.g., independently or process for validat- ing data from multiple reviewers).

9. Includes conceptual and operational defi- nitions for all variables for which data were abstracted (e.g., define blood pressure as systolic blood pressure, diastolic blood pressure, ambulatory blood pressure, auto- matic cuff blood pressure, or arterial blood pressure).

10. Describes methods used for assessing risk of bias of individual studies (including spec- ification of whether this was done at the study or outcome level).

(continued )

718 Appendix B / Rapid Critical Appraisal Checklists

Validity of Evidence Synthesis (i.e., good methodology)

11. States the principal summary measures (e.g., risk ratio, difference in means).

12. Describes the method of combining results of studies, including quality, quantity, and consistency of evidence.

13. Specifies assessment of risk of bias that may affect the cumulative evidence (e.g., publication bias, selective reporting within studies).

14. Describes appraisal procedure and conflict resolution.

15. Provides number of studies screened, as- sessed for eligibility, and included in the review, with reasons for exclusion at each stage, ideally with a flow diagram.

16. For each study, presents characteristics for which data were extracted (e.g., study size, design, method, follow-up period) and pro- vides citations.

17. Presents data on risk of bias of each study and, if available, any outcome-level assessment.

18. For all outcomes considered (benefit or harms), includes a table with summary data for each intervention group, effect esti- mates, and confidence intervals, ideally with a forest plot.

19. Summarizes the main findings, including the strength of evidence for each main out- come; considering their relevance to key groups (i.e., healthcare providers, users, and policy makers).

20. Discusses limitations at study and outcome levels (e.g., risk of bias) and at review level (e.g., incomplete retrieval of identifed re- search, reporting bias).

21. Provides a general interpretation of the re- sults in the context of other evidence, and implications for further research, practice, or policy changes.

Validity of Implementation (i.e., well-done project)

1. Purpose of project flows from evidence synthesis

2. Stakeholders (active and passive) are iden- tified and communication with them is described

4 5 1 2 3 Quite Very

No A Little Somewhat a Bit Much

Appendix B / Rapid Critical Appraisal Checklists 719

4 1 2 3 Quite Very

No A Little Somewhat a Bit Much

3. Implementation protocol is congruent with evidence synthesis (fidelity of the intervention)

4. Implementation protocol is sufficiently de- tailed to provide for replication among proj- ect participants

5. Education of project participants and other stakeholders is clearly described

6. Outcomes are measured with measures sup- ported in the evidence synthesis

Reliability of Implementation Project (i.e., I can learn from or implement project results)

1. Data are collected with sufficient rigor to be reliable for like groups to those participants of the project

2. Results are evidence implementation and are clinically meaningful (statistics are inter- preted as such)

Application of Implementation (i.e., this proj- ect is useful for my patients)

1. How feasible is the project protocol?

2. Have the project managers considered/ included all outcomes that are important to my work?

3. Is implementing the project safe (i.e., low risk of harm)?

Summary Score

Recommendations with consideration of this type of level IV intervention evidence:

• 32–64: consider evidence with extreme caution • 65–128: consider evidence with caution • 128–160: consider evidence with confidence

720 Appendix B / Rapid Critical Appraisal Checklists

RAPID CRITICAL APPRAISAL QUESTIONS FOR CASE STUDIES

VALIDITY

1. Are the results of the study valid?

• Is the study question/issue clearly articulated? Yes No Unknown

• Is the researcher’s perspective clearly described and taken Yes No Unknown into account?

• Are the methods for collecting data clearly described? Yes No Unknown

• Are the methods for analyzing the data likely to be valid and Yes No Unknown reliable?

• Are quality control measures used? Yes No Unknown

Comments

RELIABILITY

2. What are the results?

• Are the results credible, and if so, are they relevant for Yes No Unknown practice?

Comments

APPLICABILITY

3. Will the results help me in caring for my patients?

• Are the conclusions drawn justified by the results? Yes No Unknown

• Are the findings of the study transferable to other settings? Yes No Unknown

Comments

Would you use the study results in your practice to make a difference in patient outcomes?

• If yes, how? • If yes, why? • If no, why not?

Additional Comments/Reflections:

Recommendation for article use within a body of evidence:

721Appendix B / Rapid Critical Appraisal Checklists

RAPID CRITICAL APPRAISAL QUESTIONS FOR LITERATURE REVIEW (LEVEL VII)

VALIDITY

1. Are the results of the review valid?

A. Are the designs of the articles in the review identified? Yes No Unknown

B. Does the review include a detailed description of the Yes Yes Unknown search strategy to find all relevant studies and was it systematic?

C. Do the reviewers use standard criteria to describe the va- Yes No Unknown lidity of the individual studies (e.g., criteria about method- ological quality)?

D. Were the results consistent across studies? Yes No Unknown

RELIABILITY

2. What were the results?

A. Were the results described across the studies or were the Yes No Unknown findings described study by study? (Hint: Were there syn- thesis tables?)

B. What are sources of bias within the report that make the literature review unreliable? (See validity questions above.)

C. Does the bias within the literature review methodology Yes No Unknown make the results as described unusable? (if so, stop here.)

APPLICABILITY

3. Will the results assist me in caring for my patients?

A. Are my patients similar to the ones included in the review?

B. Is it feasible to implement the findings in my practice setting?

C. Were all clinically important outcomes considered, includ- ing risks and benefits of the treatment?

D. What is my clinical assessment of the patient and are there any contraindications or circumstances that would inhibit me from implementing the treatment?

E. What are my patient’s and his or her family’s prefer- ences and values about the treatment that is under consideration?

Yes

No Unknown

Would you use the study results in your practice to make a difference in patient outcomes?

• If yes, how? • If yes, why? • If no, why not?

Additional Comments/Reflections:

Recommendation for article use within a body of evidence:

722 Appendix B / Rapid Critical Appraisal Checklists

RAPID CRITICAL APPRAISAL QUESTIONS FOR EVIDENCE-BASED GUIDELINES

CREDIBILITY

1. Who were the guideline developers?

2. Were the developers representative of key stakeholders in this specialty (interdisciplinary)?

3. Who funded the guideline development?

4. Were any of the guidelines developers funded researchers of the reviewed studies?

5. Did the team have a valid development strategy?

6. Was an explicit (how decisions were made), sensible, and impar- tial process used to identify, select, and combine evidence?

7. Did the developers carry out a comprehensive, reproducible literature review within the past 12 months of its publication/ revision?

8. Were all important options and outcomes considered?

9. Is each recommendation in the guideline tagged by the level/ strength of evidence upon which it is based and linked with the scientific evidence?

10. Do the guidelines make explicit recommendations (reflecting value judgments about outcomes)?

11. Has the guideline been subjected to peer review and testing?

APPLICABILITY/GENERALIZABILITY

12. Is the intent of use provided (e.g., national, regional, local)?

13. Are the recommendations clinically relevant?

14. Will the recommendations help me in caring for my patients?

15. Are the recommendations practical/feasible (e.g., resources— people and equipment— available)?

16. Are the recommendations a major variation from current practice?

17. Can the outcomes be measured through standard care?

Yes No Unknown

Modified from Slutsky, J. (2005). Using evidence-based guidelines: Tools for improving practice. In B. M. Melnyk & E. Fineout-Overholt (Eds.), Evidence-based practice in nursing & healthcare. A guide to best practice (pp. 221–236). Philadelphia, PA: Lippincott, Williams & Wilkins. This form may be used for educational, practice change, and research purposes without permission.