Evidence base #3

Evidence Levels

Quality Ratings

Level I

Experimental study, randomized controlled trial (RCT)

Explanatory mixed method design that includes only a level I quaNtitative study

Systematic review of RCTs, with or without meta- analysis

QuaNtitative Studies

A High quality: Consistent, generalizable results; sufficient sample size for the study design; adequate control; definitive conclusions; consistent recommendations based on comprehensive literature review that includes thorough reference to scientific evidence.

B Good quality: Reasonably consistent results; sufficient sample size for the study design; some control, fairly definitive conclusions; reasonably consistent recommendations based on fairly comprehensive literature review that includes some reference to scientific evidence.

C Low quality or major flaws: Little evidence with inconsistent results; insufficient sample size for the study design; conclusions cannot be drawn.

Level II

Quasi-experimental study

Explanatory mixed method design that includes only a level II quaNtitative study

Systematic review of a combination of RCTs and quasi-experimental studies, or quasi-experimental studies only, with or without meta-analysis

Level III

Nonexperimental study

Systematic review of a combination of RCTs, quasi-experimental and nonexperimental studies, or nonexperimental studies only, with or without meta-analysis

Exploratory, convergent, or multiphasic mixed methods studies

Explanatory mixed method design that includes only a level III quaNtitative study

QuaLitative study Meta-synthesis

QuaLitative Studies

No commonly agreed-on principles exist for judging the quality of quaLitative studies. It is a subjective process based on the extent to which study data contributes to synthesis and how much information is known about the researchers’ efforts to meet the appraisal criteria.

For meta-synthesis, there is preliminary agreement that quality assessments of individual studies should be made before synthesis to screen out poor-quality studies1.

A/B High/Good quality is used for single studies and meta-syntheses2.

The report discusses efforts to enhance or evaluate the quality of the data and the overall inquiry in sufficient detail; and it describes the specific techniques used to enhance the quality of the inquiry. Evidence of some or all of the following is found in the report:

· Transparency: Describes how information was documented to justify decisions, how data were reviewed by others, and how themes and categories were formulated.

· Diligence: Reads and rereads data to check interpretations; seeks opportunity to find multiple sources to corroborate evidence.

· Verification: The process of checking, confirming, and ensuring methodologic coherence.

· Self-reflection and scrutiny: Being continuously aware of how a researcher’s experiences, background, or prejudices might shape and bias analysis and interpretations.

· Participant-driven inquiry: Participants shape the scope and breadth of questions; analysis and interpretation give voice to those who participated.

· Insightful interpretation: Data and knowledge are linked in meaningful ways to relevant literature.

C Low quality studies contribute little to the overall review of findings and have few, if any, of the features listed for high/good quality.

Evidence Levels

Quality Ratings

Level IV

Opinion of respected authorities and/or nationally recognized expert committees or consensus panels based on scientific evidence

Includes:

· Clinical practice guidelines

· Consensus panels/position statements

A High quality: Material officially sponsored by a professional, public, or private organization or a government agency; documentation of a systematic literature search strategy; consistent results with sufficient numbers of well-designed studies; criteria-based evaluation of overall scientific strength and quality of included studies and definitive conclusions; national expertise clearly evident; developed or revised within the past five years

B Good quality: Material officially sponsored by a professional, public, or private organization or a government agency; reasonably thorough and appropriate systematic literature search strategy; reasonably consistent results, sufficient numbers of well-designed studies; evaluation of strengths and limitations of included studies with fairly definitive conclusions; national expertise clearly evident; developed or revised within the past five years

C Low quality or major flaws: Material not sponsored by an official organization or agency; undefined, poorly defined, or limited literature search strategy; no evaluation of strengths and limitations of included studies, insufficient evidence with inconsistent results, conclusions cannot be drawn; not revised within the past five years

Level V

Based on experiential and nonresearch evidence

Includes:

· Integrative reviews

· Literature reviews

· Quality improvement, program, or financial evaluation

· Case reports

· Opinion of nationally recognized expert(s) based on experiential evidence

Organizational Experience (quality improvement, program or financial evaluation)

A High quality: Clear aims and objectives; consistent results across multiple settings; formal quality improvement, financial, or program evaluation methods used; definitive conclusions; consistent recommendations with thorough reference to scientific evidence

B Good quality: Clear aims and objectives; consistent results in a single setting; formal quality improvement, financial, or program evaluation methods used; reasonably consistent recommendations with some reference to scientific evidence

C Low quality or major flaws: Unclear or missing aims and objectives; inconsistent results; poorly defined quality improvement, financial, or program evaluation methods; recommendations cannot be made

Integrative Review, Literature Review, Expert Opinion, Case Report, Community Standard, Clinician Experience, Consumer Preference

A High quality: Expertise is clearly evident; draws definitive conclusions; provides scientific rationale; thought leader(s) in the field

B Good quality: Expertise appears to be credible; draws fairly definitive conclusions; provides logical argument for opinions

C Low quality or major flaws: Expertise is not discernable or is dubious; conclusions cannot be drawn