PICOT Statement Paper

Evidence-BasedPracticeProcessQuality.pdf

Home >Nursing homework help > PICOT Statement Paper

Original Article

Evidence-Based Practice Process Quality Assessment: EPQA Guidelines Mei Ching Lee, PhD, RN • Karen L. Johnson, PhD, RN • Robin P. Newhouse, PhD, RN, NEA-BC, FAAN • Joan I. Warren, PhD, RN-BC, NEA-BC

Keywords

evidence-based practice,

evidence-based practice process

quality assessment, evidence-based practice process

guidelines

ABSTRACT Background: Nurses are increasingly engaged in evidence-based practice (EBP) processes to answer significant questions and guide nursing practice. However, there are no criteria to eval- uate the rigor and quality of EBP projects, making the decision about whether to implement a recommended practice change questionable.

Aim: The purpose of this study was to achieve consensus among nationally recognized EBP nurse experts on criteria that could be used to appraise the methodological quality of an EBP project as well as to serve as a guideline to plan for an EBP project.

Methods: A modified two-round Delphi method was used. Twenty-three nationally known EBP experts were invited by e-mail to participate in completing a web-based questionnaire.

Results: Items converged after two rounds (response rate [52% (n = 12/23) for Round 1 and 35% (n = 8/23) for Round 2]) and resulted in the development of the EBP Process Quality Assessment (EPQA) guidelines that include 34 items.

Implications: The EPQA guidelines can be used to guide and evaluate the methodological quality of EBP projects. They can be used in practice settings to critically appraise an EBP project prior to translating recommendations into practice. Educators can use the EPQA guidelines as a rubric to evaluate student EBP projects. EPQA guidelines can be utilized in research to assess interventions and to build or improve EBP capacity.

BACKGROUND Evidence-based practice (EBP) is an essential element in deliv- ering quality patient care (The Joint Commission, 2008). The use of EBP guidelines improves patient outcomes (Cochrane Collaboration, 2005; Horbar et al., 2004; Thomas et al., 1999) as well as outcomes for family members, staff, the organiza- tion, and community (Worral, Levin, & Arsenault, 2010).

Many accrediting bodies in the United States have in- creased their attention on the use of evidence to make decisions and provide care that will improve patient outcomes (Worral et al., 2010). The Joint Commission recognizes the use of EBP as an effective way to improve healthcare delivery (The Joint Commission, 2008). Organizations achieving Magnet recog- nition must possess established and evolving programs related to EBP programs that include infrastructure and resources in place to support the advancement of EBP (American Nurses Credentialing Center, 2008). Professional organizations, such as the American Association of Critical Care Nurses and the Oncology Nursing Society, have extensive tools and resources on their websites to help members conduct EBP projects. Many models have been developed to educate and guide nurses on EBP methods. Scholarship for EBP is an essential curricular element for baccalaureate education (American Association of Colleges of Nursing, 2008). Schools of nursing have integrated

EBP methods into their curricula to meet accreditation require- ments. Similarly, accreditation standards for postbaccalaureate nurse residency programs require inclusion of an EBP project in the curricula (Commission on Collegiate Nursing Educa- tion, 2008). Many organizations sponsor EBP workshops and related continuing education programs. Education programs emphasize EBP as an essential component of nursing care and provide knowledge on guidance on how to conduct an EBP project (Worral et al., 2010). As a result of all these initiatives, the number of EBP projects produced and published has grown exponentially.

It is imperative that EBP projects are conducted using rig- orous methods to ensure valid, unbiased recommendations because EBP projects are conducted to solve problems and make clinical decisions (Newhouse, Dearholt, Poe, Pugh, & White, 2007). The final goal in conducting an EBP project is to provide valid practice recommendations based on a thor- ough review and critical appraisal of evidence. The evidence is used to determine if a change in current practice is needed (Cvach & Lee, 2010). The decision to translate EBP recom- mendations into clinical practice requires confidence in the validity of the practice recommendations. Practice recommen- dations from a poorly conducted EBP project maybe biased and therefore should not be translated into practice. Prior to

Original Article translating recommendations into practice, a critical appraisal of the methodological quality used to generate the recommen- dations must be made.

“Quality” as a concept is difficult to define. Verhagen et al. (1998) suggest that quality is “a set of parameters in the design and conduct of a (project) that reflects the validity of the outcome, and is related to external and internal validity and the statistical model used” (Verhagen et al., 1998, p. 1239). Methodological quality should be assessed in all steps in the EBP process: formulation of a focused clinical question, the search for and critical appraisal of the evidence, translation of the evidence into practice recommendations, and evaluation of the outcomes as a result of the implementation of the recom- mendations.

Published criteria guidelines exist for quality assessment of randomized controlled trials (Verhagen et al., 1998) and sys- tematic reviews and meta-analyses (Moher, Liberati, Tetzlaff, Altman, & the PRISMA Group, 2009). These guidelines help authors improve reporting of the results and also are useful for the critical appraisal of published reports. To our knowledge, there are no such similar published guidelines to aid in the quality assessment of EBP projects.

Significance EBP in nursing is recognized as an essential element in quality care. Accrediting bodies stress the importance of using evi- dence to make decisions and impact patient outcomes. As a result, the number of EBP projects produced and published in recent years has grown exponentially. However, assessment tools are lacking to help clinicians evaluate the rigor and quality of these EBP projects prior to adopting the recommendations into practice. This is a significant problem since wide variances in the EBP process can result in flawed recommendations and therefore may adversely affect patient outcomes, be inefficient and not cost effective. Criteria are needed to aid in the evalua- tion of the rigor and quality of EBP projects to ensure nursing care is delivered with the best evidence available.

Purpose The purpose of this study was to achieve consensus among na- tionally recognized EBP nurse experts on criteria that could be used to appraise the methodological quality of an EBP project as well as to serve as a guideline to plan for an EBP project.

METHODS Design A modified Delphi method was used to generate criteria that could be used in an instrument designed to evaluate the methodological quality of an EBP project. The Delphi method is widely used and accepted for consensus building on a specific topic (Hsu & Sandford, 2007). Nurses have used the Delphi method to ascertain priorities or determine developments for research, education, and clinical practice (Kirkwood, Wales, & Wilson, 2003). The Delphi method was selected for this study because it maintains subject anonymity and minimizes bias

or coercion due to influence of individual members in group discussion (Hsu & Sandford, 2007).

The Delphi method uses a series of rounds in which each participant is given a list of items to review and evaluate. The research team summarizes individuals’ evaluation, revises the list, and summarizes the opinions of the participants as a whole. The items are then sent back to individual participants to review and evaluate again. This iteration and feedback process continues until a consensus is reached among participants. In most cases a series of three rounds are recommended to reach consensus.

IRB Approval This study was reviewed and approved as an exempt study by the Institutional Review Board at the University of Maryland Baltimore.

Procedures A team (the manuscript authors) was formed to conduct this research. All authors are doctorally prepared nurses who have roles in leading organizational EBP projects and teach EBP and research methods to graduate nursing students. The team identified and selected items on the draft checklist, constructed surveys, analyzed and interpreted the qualitative and quantita- tive data. All authors contributed individually to the develop- ment, revision, and approved the final version of the submitted manuscript.

Instrument Development Items in the initial instrument were developed and modi- fied with permission from an established guideline that eval- uates the quality of published systematic reviews and meta- analyses, the PRISMA statement (Moher et al., 2009). Since the PRISMA guideline was developed for evaluation of system- atic reviews, which uses a different methodology than EBP, adjustment of criteria was needed. Domains were created to represent various aspects of the EBP process (question gener- ation, methods for evidence synthesis, etc.).

Selection of experts. To establish content validity, known EBP nurse experts were identified and invited to participate via e- mail. All invited experts were doctorally prepared clinicians or academicians. All had publications in peer-reviewed journals or books or both related to EBP. Many were recognized public speakers on the topic of EBP. Content validity is a crucial factor in instrument development as it establishes whether items on an instrument adequately measure a desired domain of content (Grant & Davis, 1997). Criteria used in the selection of the content experts included a history of publications and national presentations related to EBP methods. While there is no consensus on the optimal number of participants in a Delphi study, others have suggested 10–15 participants are adequate if the background of the participants is homogenous (Delbecq, Van de Ven, & Gustafson, 1975). Anticipating a 60% response rate, we invited 25 experts to participate.

EPQA Guidelines

Participation of experts. For each round, experts received in- structions, definitions of terms and a questionnaire. They were provided with an Internet link to access the questionnaire on- line. All responses were anonymous, IP addresses were not tracked, and no identifiers were collected. This allowed each participant to express their opinions with no pressure for con- formity in their communication or exchange of information, and produce nonbiased focused opinions (Hsu & Standford, 2007). Participants were asked to rate the relevancy of each item in the questionnaire using a 4-point modified Likert scale (1 = not relevant, 2 = somewhat relevant, 3 = quite relevant, 4 = highly relevant). Additionally for each item, they were asked to provide comments on the clarity of the item (“Was the item well-written, distinct?”). If they felt the item was not clear, they were asked to provide suggestions on how to add clarity to the item. At the conclusion of the questionnaire, experts were asked to comment on the overall comprehensiveness of the items in the questionnaire, if items should be added and whether there was some component of an EBP project that was not captured.

Delphi rounds. As previously recommended (Delbecq et al., 1975), for each round the experts were given 2 weeks to com- plete the questionnaire. After each round the research team met to review the results, revise or delete each item based on participant comments and evaluate and select the items to in- clude in the next round’s questionnaire.

Analyses The analysis of the responses from the Delphi rounds was both qualitative and quantitative. Quantitatively, the mean, median and standard deviation of the 4-point modified Likert scale scores were examined for each item in the survey. An a priori decision was made by the team to retain items that met the following two criteria: (1) median score of 3.25 or higher and (2) at least 70% of the Delphi subjects indicated a rating of 3 or higher (Hsu & Sandford, 2007). Qualitatively, the suggestions and comments from the participants were summarized in a narrative form to provide an audit trail of group decisions.

RESULTS Round 1 Response rate for Delphi Round 1 (conducted in May 2011) was 52% (12/23). Table 1 includes the category heading, items, means, standard deviation, and median scores for each item included in the Round 1 questionnaire. Means for the 22 items ranged from 3.40 (relevant) to 4.0 (highly relevant) and there- fore indicated agreement that all items were relevant. All items had greater than 70% of Delphi subjects rate 3 or higher on the scale. All median scores were 4.0 with the exception of one item: item 10. “Describes methods used for assessing risk of bias of individual studies (including specification of whether this was done at the study or outcome level)” (median = 3.5). The three items with the greatest dispersion of scores were: item 7, “States the process for title, abstract and article screen- ing for selecting studies” (M = 3.47, SD = 1.08, median = 4);

item 18, “For all outcomes considered (benefit or harms), in- clude a table with summary data for each intervention group, effect estimates and confidence intervals, ideally with a forest plot” (M = 3.50, SD = 0.85, median = 4); and item 22, “De- scribes sources of funding for systematic review and other support (e.g., supply of data), and the role of funders for the evidence-based practice project” (M = 3.40, SD = 0.84, median = 4).

All qualitative responses were reviewed and discussed among the investigative team. Several respondents indicated the need to define “EBP project” to distinguish it from a system- atic review. The following definition was added to the Round 2 questionnaire: A project that is generated in response to a clinical or administrative problem. The project delineates a clear, precise, and answerable question. A strategic and com- prehensive procedure is used to search for evidence (research and nonresearch) to answer the question. Evidence is reviewed and critically appraised for quality and consistency using an established rating scale. The evidence is synthesized and rec- ommendations for further research, practice or policy changes are made.

There were multiple comments stating that some items were more appropriate for systematic review than EBP. Twelve items were added reflecting new content or item revision so that each item reflected unique concepts. The resulting ques- tionnaire for Round 2 included 34 items.

Round 2 Response rate for Round 2 (conducted in July 2011) was 35% (8/23). Table 2 includes the category heading, items, means, standard deviation and median scores for each item included in the Round 2 questionnaire. Item means ranged from 3.63 to 4.0 and all median scores were 4.0, which indicated that all items were evaluated to be highly relevant. All items had scores 3 or higher as rated by more than 70% of Delphi subjects. All 34 items met the a priori criteria and all items in the Round 2 questionnaire were retained. All qualitative responses were re- viewed by the investigative team. No further item revisions were required and consensus on items was attained. Although Round 3 was planned it was determined to be unnecessary and was not conducted.

DISCUSSION AND IMPLICATIONS A two-round modified Delphi method was used to generate 34 items to serve as criteria to evaluate the methodological quality of an EBP project. The EBP Process Quality Assessment (EPQA) Guidelines can be used by clinicians to evaluate the quality of EBP projects in practice, by researchers to test the fidelity of the EBP process, and a grading rubric for educators who require EBP projects in coursework.

Practice EBP activities are increasing exponentially in the healthcare setting. Regulatory agencies, accrediting bodies, consumer

Original Article Table 1. Results of Delphi Round 1

Resultsa

Item ItemContent Mean SDb Median

Title

1 Identifies the report/project as anevidence-basedpractice project 3.55 0.52 4.00

Abstract

2 Provides a structured summarywhich includes, as applicable: data toprovide the backgroundof theproblem, statement of theproblem, objective of theEBPproject, setting, inclusion andexclusion criteria, source(s) of evidence, appraisalmethod, limitations, conclusion, recommendation and implications.

4.00 0.00 4.00

Introduction 3 Describes the rationale for the evidence-basedpractice project includingdata to support the problemandwhat is already known.

4.00 0.00 4.00

4 Provides anexplicit statement of thequestionbeing addressedwith reference toparticipants or population/intervention/comparison/outcome (PICO).

3.70 0.48 4.00

Method 5 Explicitly describes the searchmethod, inclusion andexclusion criteria and rationale for search strategy limits.

3.90 0.32 4.00

6 Describesmultiple information sources (e.g., databases, contactwith studyauthors to identify additional studies, or anyotheradditional searchstrategies) included in thesearch strategy, anddate.

3.80 0.42 4.00

7 States theprocess for title, abstract andarticle screening for selecting studies 3.47 1.08 4.00

8 Describes themethodof data extraction (e.g., independently or process for validatingdata frommultiple reviewers).

3.70 0.68 4.00

9 Includes conceptual andoperational definitions for all variables forwhichdatawere abstracted (e.g., definebloodpressureassystolicbloodpressure,diastolicbloodpressure, ambulatory bloodpressure, automatic cuff bloodpressure or arterial bloodpressure).

3.90 0.32 4.00

10 Describesmethodsused for assessing risk of bias of individual studies (including specification ofwhether thiswasdoneat the studyor outcome level).

3.50 0.53 3.50

11 States theprincipal summarymeasures (e.g., risk ratio, difference inmeans). 3.60 0.52 4.00

12 Describe themethodof combining results of studies includingquality, quantity, and consistencyof evidence.

3.80 0.42 4.00

13 Specifiesassessmentof riskof bias thatmayaffect thecumulativeevidence (e.g., publication bias, selective reportingwithin studies).

3.80 0.42 4.00

14 Describes appraisal procedure andconflict resolution. 3.70 0.48 4.00

Results 15 Provides number of studies screened, assessed for eligibility, and included in the review,with reasons for exclusion at each stage, ideallywith aflowdiagram.

3.80 0.63 4.00

16 For each study, presents characteristics forwhichdatawere extracted (e.g., study size, design,method, follow-upperiod) andprovides citations.

3.90 0.32 4.00

17 Present data on risk of bias of each studyand, if available, anyoutcome-level assessment. 3.50 0.71 4.00

18 For all outcomesconsidered (benefit or harms), includea tablewith summarydata for each intervention group, effect estimates, and confidence intervals, ideallywith a forest plot.

3.50 0.85 4.00

groups, and professional organizations call for care that is based on evidence. The conduct of EBP and research is re- quired to achieve Magnet designation. However, many prac- tices are being implemented unchecked under the guise of

evidence base. Leaders and staff alike in healthcare settings are often older and did not have the opportunity to for- mally learn about the rigor required for a practice to be deemed an EBP. Frequently leaders misguidedly refer to quality

EPQA Guidelines

Table 1. (Continued)

Resultsa

Item ItemContent Mean SDb Median

Discussion 19 Summarizes themainfindings including the strength of evidence for eachmain outcome; considering their relevance to key groups (i.e., healthcareproviders, users, andpolicy makers).

4.00 0.00 4.00

20 Discusses limitations at studyandoutcome level (e.g., risk of bias), andat review level (e.g., incomplete retrieval of identified research, reportingbias).

3.90 0.32 4.00

21 Provides a general interpretation of the results in the context of other evidence, and implications for further research, practice, or policy changes.

4.00 0.00 4.00

Funding 22 Describes sources of funding for systematic reviewandother support (e.g., supply of data), and the role of funders for the evidencebasedpractice project.

3.40 0.84 4.00

aResults basedon4-point Likert scale (1 = not relevant; 4 = highly relevant). bStandarddeviation.

improvement activities or institutional best practices as EBP. Even more confusion abounds as clinicians read about practice- based evidence and comparative effectiveness research to im- prove clinical practices. The multitude of methods makes it difficult to determine what and when to translate recommen- dations into practice.

Clinically, the EPQA guidelines can serve as a guide for leaders who are responsible for developing policies and proce- dures to support EBP processes, as well as for formative and summative evaluations. Professional development specialists may use the guidelines to educate staff and to guide nurse res- idents in completing an EBP project as part of an accredited residency program. Moreover, the guidelines will assist clini- cians in the assessment of the EBP process to build confidence in the recommendations.

Research The EPQA guidelines have utility for researchers to: (1) mea- sure the quality of EBP process; (2) evaluate the development of EBP capacity over time; and (3) build investigator confidence in choosing interventions based on recommendations after an EBP project. The first application to research is in terms of the quality of the EBP process. Higher scores would be associated with higher quality. Investigators who study interventions to improve nurse EBP competencies have many choices for in- struments to measure attitudes of healthcare providers (Kitson et al., 2008; Estabrooks, Squires, Cummings, Birdsell, & Nor- ton, 2009; McCormack, Henderson, Wilson, & Wright, 2009; Newhouse, 2010a). The EPQA guidelines can be used to com- plement the assessment of the quality of specific projects, as a measure of group knowledge and skill. Second, the EPQA guidelines can be used to evaluate the EBP program from an organizational perspective over time. Building organizational capacity for EBP is the cornerstone of successful EBP programs (Newhouse, 2007; Newhouse, 2010b). Having an assessment

tool will go a long way to build individual competencies for nurses, as well as organizational capacity for rigorous decision- making.

Third, the EPQA guidelines can be used to evaluate the con- clusions of an evidence review when choosing interventions. As organizations (both professional and health care) embrace the EBP process, they are often faced with conducting research when clear recommendations cannot be made. The inability to make recommendations usually results because research ev- idence does not exist or the quality, quantity or consistency of the evidence is suspect. The issue becomes, what interven- tion should be implemented? These interventions are new and untested within the organization, and usually not standard of care. The benefits and harms of the intervention need to be assessed. It is important to be confident that the intervention was selected based on the evidence, and that any uncertainly in terms of bias was considered.

Education Faculty members that teach in the academic setting can use the EPQA as a tool to engage students’ discussion about attributes of a high quality EBP process. Students could self-evaluate their group process and make recommendations for how the process could be strengthened.

It is typical to include grading rubrics for student EBP as- signments (White et al., 2008). The EPQA can be used as a grading rubric for undergraduate students (that learn how to engage in the EBP process), graduate students (that learn to lead EBP projects) and doctoral students (that learn to mentor leaders and staff engaged in EBP projects). The EPQA will be particularly helpful to Doctor of Nursing Practice (DNP) stu- dents and graduates who will be prepared to take the lead in mentoring the translation of evidence to practice (Newhouse, Buckley, Idzik, & Grant, unpublished manuscript).

Original Article Table 2. Delphi Round 2

Resultsa

Item ItemContent Mb SDc Median

Title

1 1 Identifies the report/project as anevidence-basedpractice project 3.63 0.74 4.00

Abstract

2 2 Provides a structured summarywhich includes, as applicable: data toprovide thebackgroundof theproblem, statement of theproblem, objective of the EBPproject, setting, inclusion andexclusion criteria, source(s) of evidence, appraisalmethod, limitations, conclusions, recommendations and implications.

3.75 0.46 4.00

Introduction 3 3 Describes the rationale for the evidence-basedpractice project. 4.00 0.00 4.00

New item 4 Includes internal data, suchasquality improvement data, to provide background in support of theproblem.

3.88 0.35 4.00

New item 5 Includes external data to frame theproblem in a large context of thepatient population in question.

3.88 0.35 4.00

4 6 Provides anexplicit statement of thequestionbeing addressedusing established formats, suchaspopulation, intervention, comparison, outcome (PICOPIO, PICOT,COPES).

3.88 0.35 4.00

New item 7 Realistic goals for theproject are establishedanddemonstrate theopportunity for improvement.

3.75 0.46 4.00

Methods 5 8 Explicitly describes the searchmethod, inclusion andexclusion criteria and rationale for search strategy limits.

3.88 0.35 4.00

6 9 Describesmultiple information sources (e.g., databases, contactwith study authors to identify additional studies, or anyother additional search strategies) included in the search strategy, anddate ranges.

3.88 0.35 4.00

New item 10 List the keywords andphrasesused for eachdatabase. 4.00 0.00 4.00

7 11 States theprocess for title, abstract, andarticle screening for selecting studies.

3.75 0.46 4.00

8 12 Describes themethodof data abstraction (e.g., independently or process for validatingdata frommultiple reviewers).

3.75 0.46 4.00

9 13 Includes conceptual andoperational definitions for all variables forwhichdata were extracted (e.g., definebloodpressure as systolic bloodpressure, diastolic bloodpressure, ambulatory bloodpressure, noninvasive blood pressure or arterial bloodpressure).

4.00 0.00 4.00

New item 14 Aquality assessment rating tool, appropriate for the level of evidence, is used (AGREE,CONSORT, etc.)

3.63 0.52 4.00

10 15 Includes assessment of potential risk of bias of individual studies. 3.88 0.35 4.00

11 16 States theprincipal summarymeasures (e.g., risk ratio, difference inmeans). 3.75 0.46 4.00

12 17 Describes themethodof combining results of studies includingquality, quantity and consistencyof evidence.

4.00 0.00 4.00

13 18 Specifies assessment of risk of bias thatmayaffect the cumulative evidence (e.g., publicationbias, selective reportingwithin studies).

3.63 0.52 4.00

14 19 Describesprocedures used to resolve conflict andachieve consensus. 3.86 0.38 4.00

EPQA Guidelines

Table 2. (Continued)

Resultsa

Item ItemContent Mb SDc Median

Results 15 20 Provides number of studies screened, assessed for eligibility, and included in the review,with reasons for exclusion at each stage, ideallywith aflow diagram.

3.88 0.35 4.00

16 21 For each study, presents characteristics forwhichdatawere extracted (e.g., sample characteristics, design, intervention, findings, and conclusions, level andquality rating, strengths andweaknesses) andprovides citations. Ideally, this is presented in table format.

4.00 0.00 4.00

17 22 Present data on risk of bias of each studyand, if available, anyoutcome-level assessment.

3.75 0.46 4.00

18 23 For all outcomes considered (benefit or harms), includea tablewith summary data for each intervention group, effect estimates, and confidence intervals, ideallywith a forest plot.

3.75 0.71 4.00

Discussion 19 24 Summarizes themainfindings including the strength of evidence for each main outcome; considering their relevance to key groups (i.e., healthcare providers, users, andpolicymakers).

4.00 0.00 4.00

20 25 Discusses limitations at studyandoutcome level (e.g., risk of bias), andat review level (e.g., incomplete retrieval of identified research, reportingbias).

4.00 0.00 4.00

21 26 Providesageneral interpretationof the results in thecontextof other evidence, and recommendations for further research, practice, or policy changes.

4.00 0.00 4.00

Implementation (NEW)

New item 27 Describes reasonable plan and translation strategieswhenapplicable. 3.88 0.35 4.00

New item 28 Identifies stakeholders andmethods for gaining stakeholder buy-in. 3.75 0.46 4.00

New item 29 Describes the interactionswith an institutional reviewboard, if applicable. 3.63 0.52 4.00

New item 30 If theplanwas implemented, an evaluation of the effectivenessof the practice/policy change ismade, including the extent towhich the practice/policy changewas implemented (processoutcome) and the extent towhich thedesiredoutcomeswere achieved (outcomeevaluation).

4.00 0.00 4.00

Funding 22 31 Describes sources of funding for systematic reviewandother support (e.g., supply of data), and the role of funders for the evidencebasedpractice project.

3.63 0.74 4.00

Teamcapacity (NEW)

New item 32 The teamhas the researchandclinical capacity to produce reliable and valid recommendationsbasedon the available evidence.

3.38 0.74 3.50

References (NEW) New item 33 References cited are relevant to theproject. 4.00 0.00 0.71

New item 34 Themost current evidence is included. 3.75 0.71 4.00

aResults basedon4-point Likert scale (1 = not relevant; 4 = highly relevant). bMean. cStandarddeviation.

STUDY LIMITATIONS Limitations of this study included sampling considerations (response rate and representativeness). The response rate for the first Delphi round was 52% (n = 12), however the second round response rate was 35% (n = 8). This low response rate

can threaten the validity of results (Goodman, 1987; Sumsion, 1998) and affect the generalizability of the results. We also do not know if the respondents for Round 1 and 2 were the same, since no identifiers were required. However, there are two rea- sons to believe that the items are highly relevant. The first is

Original Article Table 3. The Evidence-Based Practice Process Quality Assessment (EPQA) Guidelines

Section/Topic Item# Checklist Item

Title

Title 1 Identifies the report/project as anevidence-basedpractice project.

Abstract

Structured summary 2 Provides a structured summarywhich includes, as applicable: data toprovide thebackgroundof the problem, statement of theproblem, objective of theEBPproject, setting, inclusion andexclusion criteria, source(s) of evidence, appraisalmethod, limitations, conclusions, recommendations, and implications.

Introduction

Rationale 3 Describes the rationale for the evidence-basedpractice project.

Internal data 4 Includes internal data, suchasquality improvement data, to providebackground in support of the problem.

External data 5 Includes external data to frame theproblem in a large context of thepatient population in question.

Problemstatement 6 Provides anexplicit statement of thequestionbeing addressedusing established formats, suchas population, intervention, comparison, outcome (PICO, PIO, PICOT,COPES).

Goals 7 Realistic goals for theproject are establishedanddemonstrate the opportunity for improvement.

Methods

Searchmethod 8 Explicitly describes the searchmethod, inclusion andexclusion criteria and rationale for search strategy limits.

Information sources 9 Describesmultiple information sources (e.g., databases, contactwith studyauthors to identify additional studies, or anyother additional search strategies) included in the search strategy, anddate ranges.

Keywords 10 List the keywords andphrasesused for eachdatabase.

Study selection 11 States theprocess for title, abstract, andarticle screening for selecting studies.

Data collectionprocess 12 Describes themethodof data abstraction (e.g., independently or process for validatingdata from multiple reviewers).

Data items 13 Includes conceptual andoperational definitions for all variables forwhichdatawere extracted (e.g., definebloodpressureas systolic bloodpressure, diastolic bloodpressure, ambulatorybloodpressure, noninvasive bloodpressure or arterial bloodpressure).

Rating tools 14 Aquality assessment rating tool, appropriate for the level of evidence, is used (AGREE,CONSORT, etc.)

Risk of bias of individual studies

15 Includes assessment of potential risk of bias of individual studies.

Summarymeasures 16 States theprincipal summarymeasures (e.g., risk ratio, difference inmeans).

Synthesis of results 17 Describes themethodof combining results of studies includingquality, quantity, and consistencyof evidence.

Riskofbiasacrossstudies 18 Specifies assessment of risk of bias thatmayaffect the cumulative evidence (e.g., publicationbias, selective reportingwithin studies).

Consensusprocedures 19 Describesprocedures used to resolve conflict andachieve consensus.

Results

Study selection 20 Providesnumberof studies screened, assessed for eligibility, and included in the review,with reasons for exclusion at each stage, ideallywith aflowdiagram.

EPQA Guidelines

Table 3. (Continued)

Section/Topic Item# Checklist Item

Study characteristics 21 For each study, presents characteristics forwhichdatawere extracted (e.g., sample characteristics, design, intervention, findingsandconclusions, level andquality rating, strengthsandweaknesses) and provides citations. Ideally, this is presented in table format.

Risk of biaswithin studies 22 Present data on risk of bias of each studyand, if available, anyoutcome-level assessment.

Results of individual studies

23 For all outcomesconsidered (benefit or harms), includea tablewith summarydata for each intervention group, effect estimates andconfidence intervals, ideallywith a forest plot.

Discussion

Summaryof evidence 24 Summarizes themainfindings including the strength of evidence for eachmain outcome; considering their relevance to key groups (i.e., healthcareproviders, users, andpolicymakers).

Limitations 25 Discusses limitations at studyandoutcome level (e.g., risk of bias), andat review level (e.g., incomplete retrieval of identified research, reportingbias).

Conclusions 26 Provides a general interpretation of the results in the context of other evidence, and recommendations for further research, practice or policy changes.

Implementation

Translation strategies 27 Describes reasonable plan and translation strategieswhenapplicable.

Stakeholders 28 Identifies stakeholders andmethods for gaining stakeholder buy-in.

Ethical review 29 Describes the interactionswith an institutional reviewboard, if applicable.

Outcomes 30 If theplanwas implemented, an evaluation of the effectivenessof thepractice/policy change ismade, including the extent towhich thepractice/policy changewas implemented (processoutcome)and the extent towhich thedesiredoutcomeswere achieved (outcomeevaluation).

Other

Funding 31 Describes sources of funding for theproject andother support (e.g., supply of data), and the role of funders for the evidence-basedpractice project.

Team 32 The teamhas the researchandclinical capacity to produce reliable and valid recommendationsbased on the available evidence.

References 33 References cited are relevant to theproject.

34 Themost current evidence is included.

Modifiedwith permission fromAnnals of InternalMedicine. TheAmericanCollege of Physicians is not responsible for the accuracyof the translation. Note: An evidence-basedpractice project is definedas aproject that is generated in response to a clinical or administrative problem. Theproject delineates a clear, precise, and answerable question. A strategic and comprehensive procedure is used to search for evidence (research and nonresearch) to answer the question. Evidence is reviewed and critically appraised for quality and consistency using an established rating scale. The evidence is synthesized and recommendations for further research, practice, or policy changesaremade.

that items converged so quickly in two rounds. The second is that the first round questionnaire was developed and modified from the PRISMA guidelines (Moher et al., 2009), which are endorsed by major high impact journals as guidelines to assess the quality and rigor of systematic reviews and meta-analyses.

CONCLUSIONS As summarized in Table 3, this two-round Delphi study re- sulted in 34 items. The EPQA can be used to appraise the quality of a completed EBP project or as a guideline to plan for an EBP project. The EPQA guidelines have important im-

plications for use in practice, education, and research. Use of the EPQA guidelines to evaluate the quality of EBP projects provides a higher level of confidence that the recommenda- tions made and implemented are credible and valid. The next step is to evaluate the utility of the EPQA instrument in field settings and modify accordingly, perhaps with another Delphi round. WVN

Author information

Mei Ching Lee, Postdoctoral Fellow, Johns Hopkins University School of Nursing, Baltimore, MD, USA; Karen L. Johnson,

Original Article Research Director, Nursing, Banner Healthcare, Baltimore, MD, USA; Robin P. Newhouse, Chair & Professor, Univer- sity of Maryland School of Nursing, Baltimore, MD, USA; Joan I. Warren, Director, Nursing Research, MedStar Franklin Square Hospital Center, Baltimore, MD, USA. Address correspondence to Mei Ching Lee, Johns Hopkins University School of Nursing, 511 N. Washington Street, Balti- more, MD 21213 USA; [email protected]

References American Association of Colleges of Nursing. (2008). The essentials

of baccalaureate for professional nursing practice. Washington, DC: Author.

American Nurses Credentialing Center. (2008). Magnet recognition program? Application manual. Silver Spring, MD: Author.

The Cochrane Collaboration. (2005). Informational leaflets. Leaflet 1: The Cochrane Collaboration: An introduction. Retrieved from www.cochrane.org/resources/brochure.htm.

Commission on Collegiate Nursing Education. (2008). Standards for accreditation of post-baccalaureate nurse residency programs. Washington, DC: Author.

Cvach, M. & Lee, M. (2010). Selecting the pathway to translation. In S. Poe & K. White (Eds.), Johns Hopkins Nursing evidence- based practice implementation and translation (1st Ed; pp. 181– 199). Indianapolis, IN: Sigma Theta Tau International, Honor Society of Nursing.

Delbecq, A., Van de Ven, A. & Gustafson, D. (1975). Group tech- niques for program planning (1st Ed). Glenview, IL: Scott, Fores- man, and Co.

Estabrooks, C.A., Squires, J.E., Cummings, G.G., Birdsell, J.M. & Norton, P.G. (2009). Development and assessment of the Alberta context tool. Biomedical Center Health Services Research, 9(234), doi: 10.1186/1472-6963/9/234.

Goodman, C. (1987). The Delphi technique: A critique. Journal of Advanced Nursing, 12(6), 729–734.

Grant, J.S. & Davis, L.L. (1997). Selection and use of content experts for instrument development. Research in Nursing and Health, 20, 269–274.

Horbar, J., Carpenter, J., Buzas, J., Soll, R., Suresh, G., Bracken, M., Leviton, L., Plsek, P. & Sinclair, J. (2004). Collaborative quality improvement to promote evidence-based surfactant for preterm infants: A cluster randomized trial. British Medical Jour- nal, 329(7473), 1593–1602.

Hsu, C. & Sandford, B. (2007). The Delphi technique: Making sense of consensus. Research & Evaluation, 12(10), 1–8.

The Joint Commission on Accreditation of Healthcare Orga- nizations. (2008). Performance measurement. Retrieved from http://www.jointcommission.org/Performance Measurement.

Kirkwood, M., Wales, A. & Wilson, A. (2003). A Dephi study to de- termine nursing research priorities in the North Glascow Uni- versity Hospitals NHS Trust and the corresponding evidence

base. Health Information and Libraries Journal, 20(Suppl. 1), 53– 58.

Kitson, A.L., Rycroft-Malone, J., Harvey, G., McCormack, B., Seers, K. & Titchen, A. (2008). Evaluating the successful implemen- tation of evidence into practice using the PARiHS framework: Theoretical and practical challenges. Implementation Science, 3, doi: 10.1186/1748-5908-3-1

McCormack, B., Henderson, E., Wilson, V. & Wright, J. (2009). Making practice visible: The workplace culture critical analysis tool (WCCAT). Practice Development in Health Care, 8(1), 28–43.

Moher, D., Liberati, A., Tetzlaff, J., Altman, D.G. & the PRISMA Group. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. Annals of Internal Medicine, 151(4), 264–270.

Newhouse, R.P. (2007). Creating infrastructure supportive of evidence-based nursing practice: Leadership strategies. World- views on Evidence-Based Nursing, 4(1), 21–29.

Newhouse, R.P. (2010a). Instruments to assess organizational readiness for evidence-based practice. Journal of Nursing Admin- istration, 40(10), 404–407.

Newhouse, R.P. (2010b). Establishing organizational infrastruc- ture. In S. Poe & K. White (Eds.), Johns Hopkins nursing evidence- based practice implementation and translation (1st Ed; pp. 55–72). Indianapolis, IN: Sigma Theta Tau International, Honor Society of Nursing.

Newhouse, R.P., Buckley, K., Idzik, S. & Grant, M. (2013). Recon- ceptualization of a doctoral EBP course from in-class to blended format: Lessons learned from a successful transition. Journal of Professional Nursing, in press.

Newhouse, R.P., Dearholt, S., Poe, S., Pugh, L. & White, K. (2007). Johns Hopkins nursing evidence-based practice: Model and guide- lines. Indianapolis, IN: Sigma Theta Tau International, Honor Society of Nursing.

Sumsion, T. (1998). The Delphi technique: An adaptive research tool. British Journal of Occupation Therapy, 61(4), 153–156.

Thomas, L., Cullum, N., McColl, E., Rousseau, N., Soutter, J. & Steer, N. (1999). Guidelines in professions allied to medicine. Cochrane Database of Systematic Reviews, (1), doi: 10.1002/14651858. CD00349.

Verhagen, A.P., De Vet, H.C.W., De Bie, R.A., Kessels, A.G.H., Boers, M., Bouter, L.M. & Knipschild, P.G. (1998). The Delphi list: A criteria list for quality assessment of randomized clinical trials for conducting systematic reviews developed by Delphi consensus. Journal of Clinical Epidemiology, 51(12), 1235–1241.

White, K.W., Newhouse, R.P., Dearholt, S.L. & Pugh, L.C. (2008). Instructors’ guide for Johns Hopkins nursing evidence-based practice model and guidelines. Indianapolis, IN: Sigma Theta Tau Inter- national, Honor Society of Nursing.

Worral, P.S., Levin, R. F. & Arsenault, D. C. (2010). Doc- umenting an EBP project: Guidelines for what to include and why. Journal of the New York State Nurses Association, 40(2), 12–19.

doi 10.1111/j.1741-6787.2012.00264.x WVN 2013;10:140–149

Copyright of Worldviews on Evidence-Based Nursing is the property of Wiley-Blackwell and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.