Foundations and Essentials for the Doctor of Nursing Practice
Tinacherry100$
Developingguideline.pdf
IMPLEMENTATION PROJECT
©2019 Un
Developing guideline-based quality indicators: assessing gaps in pain management practice
Valerie J. Fiset MScN, RN,1 Barbara L. Davies PhD, RN,1 Ian D. Graham PhD,2 Wendy Gifford PhD, RN1
and Kirsten Woodend MSc, PhD, RN3
1School of Nursing, 2School of Epidemiology, Public Health and Preventive Medicine, University of Ottawa, Ottawa, and 3Trent/Fleming School of
Nursing, Peterborough, Ontario, Canada
AB S T R AC T
Aim: In this article, the authors discuss a multiphase approach for developing quality indicators based on pain practice guidelines, and the challenges associated with the process. The presentation is based on previously published reporting standards for guideline-based quality indicators.
Methods: The following steps of the indicator development process were undertaken: topic selection; guideline selection; extraction of recommendations; quality indicator selection and practice test.
Results: Eleven practice guidelines were reviewed for quality, and three high-quality guidelines were compared for pertinent recommendations. From these three guidelines, 12 recommendations were extracted and judged appropriate to examine the practice gap for nursing students and clinicians on an oncology and palliative care unit. Quality indicators were then identified by a consensus process, resulting in 24 discrete indicators that were included in the practice test.
Conclusion: Quality indicators can be used to examine gaps in pain management practice, and to evaluate change after guideline implementation. However, their development can be challenging, and guideline developers could facilitate uptake of guidelines by including clear, relevant quality indicators as part of guideline creation and presentation.
Key words: Evidence-based practice, pain, performance measures, practice guidelines, quality indicators
Int J Evid Based Healthc 2019; 17:92–105.
Correspondence: Valerie J. Fiset, MScN, RN, School of Nursing,
University of Ottawa, 451 Smyth Road, Room RGN 1118, Ottawa,
ON, Canada K1H 8M5. E-mail: [email protected]
DOI: 10.1097/XEB.0000000000000160
What is known about the topic?
� Evidence-based practice guidelines are helpful tools for individuals and teams looking to improve clinical practice.
� It is important to examine evidence-practice gaps prior to implementing guideline recommendations and to evaluate any change in practice following implementation.
� Quality indicators based on clinical practice guideline recommendations can be used to examine evidence-practice gaps and to evaluate practice change; however, many practice guidelines do not include them.
What does this article add to the topic? � An example of a process for the development of quality indicators from
practice guidelines. � Lessons learned from the process for research teams wanting to create
quality indicators from practice guidelines, especially in relation to the evaluation of evidence-based recommendations for patient care processes and outcomes.
� Suggestions for guideline developers related to quality, wording and grading of recommendations, and for including indicator development as part of the guideline development process.
92 International Journal of Evidence-Based
iversity of Adelaide, Joanna Briggs Institute. U
Background
M any health care providers (HCPs) implement best
practice guidelines (BPGs) to improve practice
and patient outcomes. According to the authors of the
Registered Nurses’ Association of Ontario (RNAO) ‘Toolkit
for the Implementation of BPGs’ BPGs: ‘are developed and
implemented to deliver effective care based on current
evidence’.1(p.7) They also indicate that BPG recommenda-
tions are statements that reflect the best evidence to
support a given practice. The suggested first phase of
the BPG implementation process includes conducting a
needs assessment to determine the ‘practice gap’
between recommended practices and current prac-
tise.1–4 Authors of the RNAO toolkit for the implementa-
tion of BPGs propose a ‘gap analysis tool’ to determine the
extent to which practices are evidence-based, whether
they be fully, partially or not at all rooted in BPG
recommendations.1(p.26) The Joanna Briggs Institute (JBI)
Healthcare � 2019 University of Adelaide, Joanna Briggs Institute
nauthorized reproduction of this article is prohibited.
IMPLEMENTATION PROJECT
©2019 Un
Practical Application of Clinical Evidence System (PACES)5
supports preimplementation audits to document current
practice, prior to implementation of best practice recom-
mendations, and was also used in a number of recent
studies.6–9
Kitson and Straus10 indicate that quality indicators
can also be used to examine practice gaps. They define
quality indicators as ‘measures that monitor, assess and
improve quality of care and organizational functions
that affect patient outcomes’.(p.60)Approaches to devel-
oping quality indicators include: examination of best
evidence from high-quality systematic reviews or prac-
tice guidelines, formal consensus methods such as the
Delphi or nominal group techniques and consensus
development conferences.10,11 In their scoping review
of development methods for nursing process health
care indicators, Xiao et al.11 also identified focus groups,
questionnaires, interviews, workshops and other meet-
ings as means of elucidating indicators. Once quality
indicators are determined, the gap can be measured
through needs assessments and examination via chart
audits or observation.10
The purpose of this article is to describe the process
used to develop guideline-based quality indicators.
This was part of a larger project describing nursing
students’ use of guidelines for pain management in
their oncology and palliative care clinical placements,
where the authors set out to describe the gap between
the recommended practices from practice guidelines
related to pain assessment and management and the
actual practice of nursing students, practicing nurses
and other clinicians. The research team and advisory
committees on each research site anticipated that this
gap analysis would in turn provide insights into learn-
ing needs and practice processes needed to support
improved efficacy of pain management. The starting
point for this gap analysis was to develop guideline-
based indicators.
Methods Kotter et al.,12(p.3) for the purposes of data extraction for
a systematic review of the literature focusing on meth-
odological approaches to guideline-based quality indi-
cator development, identified six steps for guideline-
based quality indicator development: first, topic selec-
tion; second, guideline selection; third, extraction of
recommendations; fourth, quality indicator selection;
fifth, practice test; and sixth, implementation. The latter
stepwas not undertaken as the purpose of this aspect of
the larger project was gap analysis. Advisory commit-
tees that provided input throughout the process are
described below.
International Journal of Evidence-Based Healthcare � 2019 University
iversity of Adelaide, Joanna Briggs Institute. Un
Composition of advisory committees The larger study was conducted at two sites, with
advisory committees for each site. The advisory commit-
tees at each site consisted of nursing and allied health
clinical staff from the units (one medical oncology unit
and one palliative care unit), nurse educators from both
clinical and academic settings (who also had experience
with quality improvement initiatives) and nursing
student representatives.
Results of process Topic selection Topic selection was undertaken by the primary investi-
gator (a PhD in nursing candidate and nurse educator
with a clinical background in oncology and palliative
care), and other members of the research team who
specialize in knowledge translation and guideline imple-
mentation. Project advisory committees were also con-
sulted during the topic selection phase. Despite the fact
that pain BPGs exist, and that pain management is a part
of professional entry to practice competencies,13 and
prelicensure competencies,14,15 authors of a recent
review of the literature showed that medical and nursing
students still lack satisfactory knowledge of pain man-
agement.16 Conceptualizing this evidence-practice gap
is essential, to best prepare graduates for the realities of
practice, and to improve patient outcomes related to
pain management. Effective pain management is also an
issue of high clinical importance – authors of a system-
atic review of under-treatment of cancer pain reported
that the pain of one in three cancer patients still goes
under-treated.17
Guideline identification, appraisal and selection Following topic selection, the next step of the process
was to identify practice guidelines for the management
of pain. A systematic search of websites and electronic
databases was conducted in 2014. The PIPOH framework
(population, intervention, professionals/patients, out-
comes, health care setting)18 was used to define the
inclusion and exclusion criteria for the search (Table 1),
and a research librarian was consulted regarding the
guideline search process. The authors looked to two key
publications to guide them: the process for guideline
and systematic review search strategy in the ‘Best prac-
tice guideline: Assessment and management of Pain’19
and the search methods for the ‘Development and
evaluation of evidence-informed clinical nursing proto-
cols for remote assessment, triage and support of cancer
treatment-induced symptoms’.20 All of the websites
searched in the RNAO BPG search strategy were
of Adelaide, Joanna Briggs Institute 93
authorized reproduction of this article is prohibited.
Table 1. Inclusion and exclusion criteria for guideline search strategy
Criteria Inclusion criteria Exclusion criteria
Population Adults with pain Children Pain related to a specific condition other than
cancer i.e. chronic pain, lower back pain or specific type of pain
Intervention Any symptom intervention to assess or manage pain Limited to pharmacologic interventions only
Professionals targeted Nurses, physicians and other allied health care provi- ders
Outcomes Effective symptom management (decreased incidence of moderate-to-severe pain)
Health care setting Inpatient, outpatient, long-term care or palliative care settings
Methodology Clinical practice guidelines/consensus statements Systematic reviews
Language English or French Languages other than English or French
Publication dates 2009–2014
VJ Fiset et al.
©2019 Un
searched for pain guidelines. Additional sources for
guidelines searched include: the National Guideline
Clearinghouse (https://www.guideline.gov/); the SAGE
directory of Cancer Guidelines (http://www.cancer-
view.ca/treatmentandsupport/grcmain/grcsage/); Fraser
Health Hospice Palliative Care Symptom Guidelines
(http://www.fraserhealth.ca/health-professionals/profes-
sional-resources/hospice-palliative-care/); Cancer Care
Ontario symptom management tools (https://www.can-
cercare.on.ca/pcs/treatment/sympmgmt/); BC Guideli-
nes.ca (http://www2.gov.bc.ca/gov/content/health/
practitioner-professional-resources/bc-guidelines) and
the National Consensus Project for Quality Palliative Care
(http://www.nationalcoalitionhpc.org/guidelines-2013/).
The project was started in 2014, and the previous 5 years
(2009–2014) were chosen for the search time frame to
ensure that the guidelines considered were up to date. In
addition, Medline, Embase, CINAHL and PsychINFO were
searched for guidelines reflecting the following search
terms: pain assessment and management; guideline or
practice guideline.
Eleven pain guidelines were identified (Table 2).
Quality assessment using the AGREE II tool was subse-
quently completed.21 This valid and reliable tool was
developed to assess the methodological quality of prac-
tice guidelines across six domains: first, scope and pur-
pose; second, stakeholder involvement; third, rigour of
development; fourth, clarity of presentation; fifth, appli-
cability; and sixth, editorial independence. To narrow the
number of guidelines for full quality assessment, two
reviewers used the rigour domain to assess all 11 of the
guidelines. This domain ‘relates to the process used to
gather and synthesize the evidence, the methods to
formulate and update recommendations’,21(p.10) and
was deemed useful and important by the authors in
determining the quality of the guidelines. Guidelines
94 International Journal of Evidence-Based
iversity of Adelaide, Joanna Briggs Institute. U
with rigour scores of less than 50% were not included.
The process of determining the quality of the guidelines
by two reviewers was facilitated by using the online
AGREE PLUS appraisal platform. This web application
allows reviewers to complete an individual score on
each domain, and automatically calculates the domain
score based on the assessor’s input. The review coordi-
nator (primary investigator) compared scores. In situa-
tions of disagreement on the scores on the domains, the
reviewers met to discuss to ensure agreement. A third
review was not required for any scores. Of the 11 pain
guidelines screened, eight were excluded because of low
rigour domain scores. Reviewers noted extreme variabil-
ity in the quality of the guidelines. Of the 11 guidelines
reviewed for rigour using the AGREE II criteria, scores
ranged from 2 to 75%, mean¼ 39%. Only three guide-
lines qualified for full quality assessment (rigour scores
>50%). Table 3 provides a comparison of the levels and
grades of evidence used in each guideline to reflect the
quality of their recommendations.
Extraction of guideline recommendations A summary matrix of the recommendations, based on a
template in an article by Graham et al.,22 was created,
including the level of evidence for each recommenda-
tion (where available), as well as the full assessment on
all quality domains using the AGREE II tool for each of the
three included guidelines. Recommendations were
grouped together in the following areas: first, pain
assessment and diagnosis; second, patient issues and
care processes; third, nonpharmacologic interventions;
and fourth, pharmacologic/medical interventions. This
matrix was presented to the advisory committee at the
study sites. As part of a focus group meeting facilitated
by the primary investigator, members reviewed the
recommendations from the three higher quality
Healthcare � 2019 University of Adelaide, Joanna Briggs Institute
nauthorized reproduction of this article is prohibited.
Table 2. Summary of guidelines identified
Title Author, year Rigor
score (%) Further screening/
comments
Included
Assessment & Management of Pain, 3rd ed. http:// rnao.ca/sites/rnao-ca/files/AssessAndManagemen- tOfPain2014.pdf
RNAO, December 201421 75 Broad focus, pharmacolog- ical management is broad
Symptom Management Guides-to-Practice: Pain, https://www.cancercare.on.ca/toolbox/symptools/
Cancer Care Ontario (CCO), August 201026
66 Based on SIGN guideline for cancer pain (Novem- ber, 2008)23
Guidelines on Pain Management & Palliative Care European Association of Urology (EAU), 201327
Discontinued
52 Broad, holistic focus
Excluded low rigor score
Initial Assessment And Management of Pain: A Pathway for Care
British Pain Society, October 201346
24 Focus of guideline very generic. Pathway format vs. guideline format
Adult Cancer Pain National Comprehensive Cancer Network (NCCN), 2014 Updated 201747
23 Rigor score low – likely related to minimal amount of information available regarding guideline development process. Some general information available. Need to be member to access guideline online
General Palliative Care Guidelines for the Manage- ment of Pain at the End of Life in Adult Patients https://www.rqia.org.uk/RQIA/files/e0/e0a81c25- acb8-4982-9970-1ed62e9f2015.pdf
Guidelines and audit implementation net- work (GAIN), February 201148
20 Informed by SIGN guide- line and Cochrane reviews. Holistic guide- line
Care Management Guidelines: Pain Management http://www.dhhs.tas.gov.au/__data/assets/pdf_file/ 0006/36951/Care_Management_Guidelines_- _Pain_management_in_palliative_care_- _20160622.pdf
Tasmania Department of Health and Human Services, December 200949
16 Project lead sent develop- ment information, though minimal infor- mation provided
Management of Cancer Pain: ESMO Clinical Practice Guidelines http://annonc.oxfordjournals.org/con- tent/23/suppl_7/vii139.full.pdf+html
European Society of Medical Oncology, October 201250
16 Good intro with stats for background lit review. Holistic focus
Nursing Standard of Practice Protocol: Pain Manage- ment in Older Adults https://consultgeri.org/geri- atric-topics/pain
Ann L. Horgas, RN PhD FGSA FAAN, Saunjoo L. Yoon, PhD, RN, Mindy Grall, PhD APRN BC, July 201251
10 Holistic focus
Pain Assessment & Management Clinical Practice Guidelines http://www.wrha.mb.ca/extranet/eipt/ files/EIPT-017-001.pdf
Winnipeg Regional Health Authority, April 201252
5 In introduction, says it is based on RNAO guide- line (2007), later says ‘based on a compilation of published CPGs as well as review and feed- back’
Pain Management Guideline http://www.hcanj.org/ files/2013/09/hcanjbp_painmgmt2_3.pdf
Health care association of NJ Confirmed for accuracy in 201153
2 Holistic focus, assessment tools included
CPG, Clinical Practice Guideline; RNAO, Registered Nurses’ Association of Ontario.
IMPLEMENTATION PROJECT
©2019 Un
guidelines, and highlighted clinically important recom-
mendations for their specific clinical context, as well as
those that were common across two or three of the
guidelines. For example, advisory committee members
recommended that data gathered included whether
International Journal of Evidence-Based Healthcare � 2019 University
iversity of Adelaide, Joanna Briggs Institute. Un
patients were being followed by the palliative care
consultation team, as part of the indicator relating to
the involvement of the interprofessional team in pain
management. In addition, members emphasized that
data be collected related to the reassessment of pain
of Adelaide, Joanna Briggs Institute 95
authorized reproduction of this article is prohibited.
Table 3. Levels of evidence and grades of recommendations used in selected guidelines
Levels of evidence and grade of recommendations in CCO guideline
24 Levels of evidence in RNAO guideline19 (did not grade recommendations)
Levels of evidence and grade of recommendations in EAU guideline25
Levels of evidence 1þþ High-quality meta-analyses, systematic reviews or RCTs, or RCTs with a very low risk of bias
1þ Well conducted meta-analyses, systematic reviews or RCTs with a low risk of bias
1� Meta-analyses, systematic reviews or RCTs with a high risk of bias
2þþ High-quality systematic reviews of case– controls or cohort studies. High-quality case– control or cohort studies with a very low risk of confounding or bias and a high probability that the relationship is causal
2þ Well conducted case–control or cohort studies with a low risk of confounding or bias and a moderate probability that the relationship is causal
2� Case–control or cohort studies with a high risk of confounding or bias and a significant risk that the relationship is not causal
3 Nonanalytic studies, e.g. case reports, case series 4 Expert opinion
Grades of recommendations A – At least one meta-analysis, systematic review, or RCT that is rated as 1þþ, and is directly applicable to the target population: or a body of evidence that consists principally of studies rated as 1þ, directly applicable to the target population and demonstrating overall consistency of results
B – A body of evidence including studies rated as 2þþ, directly applicable to the target population, and demonstrating overall consistency of results; or extrapolated evidence from studies rated as 1þþ or 1þ
C – A body of evidence including studies rated as 2þ, directly applicable to the target population and demonstrating overall consistency of results; or extrapolated evidence from studies rated as 2þþ
D – Evidence level 3 or 4; or extrapolated evidence from studies rated as 2þ
Good practice points U Recommended best practice based on the clinical
experience of the guideline development group
Levels of evidence Ia – Evidence obtained from meta-analysis or systematic reviews of RCTs
Ib – Evidence obtained from at least one RCT
IIa – Evidence obtained from at least one well designed controlled study without ran- domization
IIb – Evidence obtained from at least one other type of well designed quasiexperimental study, without randomization
III – Evidence obtained from well designed nonexperimen- tal descriptive studies, such as comparative studies, correla- tion studies and case studies
IV – Evidence obtained from expert committee reports or opinions and/or clinical experiences of respected authorities
Levels of evidence Ia – Evidence obtained from meta-analysis or systematic reviews of randomized con- trolled trials
Ib – Evidence obtained from at least one RCT
IIa – Evidence obtained from at least one well designed con- trolled study without random- ization
IIb – Evidence obtained from at least one other type of well designed quasiexperimental study, without randomization
III – Evidence obtained from well designed nonexperimen- tal descriptive studies, such as comparative studies, correla- tion studies and case studies
IV – Evidence obtained from expert committee reports or opinions and/or clinical experiences of respected authorities
Grades of recommendations A – Based on clinical studies of good quality and consistency addressing the specific recom- mendations and including at least one randomized control trial
B – Based on well conducted clinical studies, but without randomized control trials
C – Made despite the absence of directly applicable clinical studies of good quality
CCO, Cancer Care Ontario; EAU, European Association of Urology; RCT, randomized controlled trial; RNAO, Registered Nurses’ Association of Ontario.
VJ Fiset et al.
©2019 Un
following the intervention(s), as related to the recom-
mendation ‘reassess the person’s response to the pain
management interventions by using the same re-evalu-
ation tool’.19(p.38) Agreement was sought informally, in
line with focus group processes described by Xiao et al.11
Table 4 includes a brief description of the included
guidelines and the recommendations that were selected
by the advisory committee members.
Quality indicator selection After selecting relevant recommendations from the
included practice guidelines, the advisory committees
96 International Journal of Evidence-Based
iversity of Adelaide, Joanna Briggs Institute. U
and research team developed quality indicators.
Nothacker et al.26 recommend consideration of rele-
vance, scientific soundness and feasibility when deciding
on the most appropriate quality indicators. In this case,
relevance and feasibility played a central role, more so
than scientific soundness, given the low levels of evi-
dence supporting many of the included guideline rec-
ommendations. Committee members looked to those
outcomes that would contribute to improvements in
patient care, and also considered documentation tools
and practices to determine how ‘measureable’ the indi-
cators actually were. In several cases, recommendations
Healthcare � 2019 University of Adelaide, Joanna Briggs Institute
nauthorized reproduction of this article is prohibited.
Table 4. Pain guidelines comparison and indicators
Cancer Care Ontario (CCO) Registered Nurses’ Association of Ontario (RNAO) European Association of Urology (EAU)
Publication year 2010 2014 2013
Quality assessment (all subscales)
Scope and purpose 76% 95% 62%
Stakeholder involvement 48% 52% 33%
Rigor of development 66% 75% 52%
Clarity 86% 100% 76%
Applicability 36% 64% 18%
Editorial independence 100% 71% 79%
Strengths/limitations Nicely laid out, excellent support tools for implementation (pocket guide & algorithm, app). Limited discussion of non-medically focused interventions
Broad focus, recently updated. Good emphasis on nursing interventions within context of IP team. Long
Broad focus on many types of pain and management strategies. Focused on urological cancer populations for the most part. Recommendations may not be a fit with Canadian standards. Heavy emphasis on medical intervention
Cancer Care Ontario (CCO)
Registered Nurses’ Association of Ontario (RNAO)
European Association of Urology (EAU) Initial indicators Final indicators
Recommendations – assessment and diagnosis
Screen for the pres- ence or risk of any type of pain on admission or visit with a HCP; after a change in medical status, prior to, during and after a procedure
Not present Present U
Level of evidence: Ib Not present Pain screening: Proportion of
patients that had a pain assessment documented on admission
(1) Proportion of patients that had a pain assess- ment documented on admission
Prior to treatment, an accurate assess- ment should be done to determine the cause(s), type(s) and sever- ity of pain and its impact on the patient
Present U
Grade: based on clin- ical experience
Present U
Level of evidence: Ib Not present Cause: Proportion of patients with
pain on admission or episodes during hospital stay that have a documented diagnosis of the physical cause of pain
Type: Proportion of patients with pain on admission or episodes during hospital stay that have a documented description of their pain
Severity: Proportion of patients that had pain with a documen- ted pain score
Impact: Proportion of patients that had pain on admission that had a further pain assess- ment completed (ESAS, BPI)
(2) Proportion of patients with pain on admission that have a documented diagnosis of the physical cause of pain
(3) Proportion of patients that had pain on admis- sion that had a pain score documented
(4) Proportion of patients that had pain on admis- sion that had a further pain assessment com- pleted (ESAS, BPI)
(5) Proportion of patients that had a pain episode that had a pain score documented
Patients with cancer pain should have treatment out- comes monitored regularly using visual analogue scales, numerical rating scales or verbal rating scales and multidimen- sional instruments as necessary
Present U
Grade: based on clin- ical experience
Present U
Level of evidence: Ib Present U
Level of evi- dence: not stated
Treatment outcomes monitored: Proportion of patients whose pain was reassessed after an intervention
Rating scale: Proportion of patients that had pain with a documented pain score
Multidimensional instrument: Pro- portion of patients that had pain that had a further pain assessment completed (ESAS, BPI)
(3) Proportion of patients that had pain on admis- sion with a documented pain score
(6) Mean and range of pain scores on admission
(4) Proportion of patients that had pain on admis- sion that had a further, more detailed, pain assessment completed (ESAS, BPI)
(5) Proportion of patients that had a pain episode that had a pain score documented
(7) Mean and range of pain scores at the time of a pain episode
The patient should be the prime assessor of his or her pain
Present U
Grade: A Not present Present U
Level of evi- dence: not stated
Patient as prime assessor: Propor- tion of patients that had pain episodes who were the prime assessor of their pain
(8) Proportion of patients that had pain episodes who were the prime assessor of their pain
(Continued )
IMPLEMENTATION PROJECT
International Journal of Evidence-Based Healthcare � 2019 University of Adelaide, Joanna Briggs Institute 97
©2019 University of Adelaide, Joanna Briggs Institute. Unauthorized reproduction of this article is prohibited.
Table 4. (Continued)
Cancer Care Ontario (CCO)
Registered Nurses’ Association of Ontario (RNAO)
European Association of Urology (EAU) Initial indicators Final indicators
Reassess the person’s response to the pain management interventions by using the same re-evaluation tool
Not present Present U
Level of evidence: IIb Present U
Level of evi- dence: not stated
Reassessment: Proportion of patients that received an inter- vention for pain that had their pain reassessed using pain score
(9) Proportion of patients that received an interven- tion for pain that had their pain reassessed
Recommendations – patient issues and care processes
Establish a compre- hensive plan of care that incorpo- rates the goals of the person and IP team
Present U
Grade: based on clin- ical experience
Present U
Level of evidence: III Not present Comprehensive plan of care: Pro-
portion of patients with pain that have a plan of care docu- mented
Goals of person: Proportion of patients with a comprehensive plan of care that explicitly mentions person’s goals
Goals of IP team: Proportion of patients with a comprehensive plan of care that explicitly mentions IP team member’s role in meeting goals
Involvement of IP team: Propor- tion of patients with pain on admission or pain episodes that had interprofessional team members involved in their care (specify IP team members)
Involvement of other services: Proportion of patients with pain on admission or pain episodes that had other ser- vices involved in their care (specify other services)
(10) Proportion of patients with pain on admission or pain episodes that had interprofessional team members involved in their care (specify IP team members)
(11) Proportion of patients with pain on admission or pain episodes that had other services involved in their care (specify other services)
Recommendations – nonpharmacologic interventions
Evaluate any non- pharmacological interventions (physical and psy- chological) for effectiveness and the potential for interactions with pharmacological interventions
Not present Present U
Level of evidence: Ib Not present Evaluation: Proportion of patients
that received an intervention for pain that had their pain reassessed
Type of interventions: Proportion of patients with pain that had different interventions to ease the pain [Specify pharmacolog- ical, nonpharmacological, phar- macological and nonpharmacological, no addi- tional interventions than the current pain management plan (patient is on a pain manage- ment regime and no additional medication (including regular dose or as needed medications (PRNs))) are given at that time], or no intervention (the patient is not on a pain management regime and no pain medica- tion is provided)
Nonpharmacologic interventions: Of patients that had their pain treated with nonpharmacologi- cal interventions, what propor- tion of different interventions were used (specify interven- tions)
(12) Proportion of patients with pain that had non- pharmacological, pharma- cological and nonpharmacological, no additional interventions than the current pain management plan [patient is on a pain man- agement regime and no additional medication (including regular dose or as needed medications (PRNs)) are given at that time], or no intervention (the patient is not on a pain management regime and no pain medication is provided)
(13) Of patients that had their pain treated with nonpharmacological interventions, what pro- portion of different inter- ventions were used (specify interventions)
Teach the person, their family and caregivers about the pain manage- ment strategies in their plan of care and address known concerns and misbeliefs
Not present Present U
Level of evidence: Ib Not present Patient and family education: Pro-
portion of patients receiving an intervention for pain had documentation of teaching about the pain management
(14) Proportion of patients receiving an intervention for pain had documenta- tion of teaching about the pain management
VJ Fiset et al.
98 International Journal of Evidence-Based Healthcare � 2019 University of Adelaide, Joanna Briggs Institute
©2019 University of Adelaide, Joanna Briggs Institute. Unauthorized reproduction of this article is prohibited.
Table 4. (Continued)
Cancer Care Ontario (CCO)
Registered Nurses’ Association of Ontario (RNAO)
European Association of Urology (EAU) Initial indicators Final indicators
Always offer psycho- logical support to cancer patients and their loved ones
Not present Not present Present U
Level of evi- dence: Ia
Psychological support: Proportion of patients with a pain episode that had psychological support documented as a nonpharma- cological intervention
(15) Proportion of patients with a pain episode that had psychological sup- port mentioned as a non- pharmacological interven- tion
Recommendations – pharmacological/medical interventions
Analgesia for continu- ous pain should be prescribed on a regular basis not as needed (PRN)
Present U
Grade: D Not present Not present Regular analgesia: Proportion of
patients with a pain episode for whom analgesic prescribed regularly (not PRN)
(16) Proportion of patients receiving pharmacologic interventions for whom analgesic prescribed reg- ularly (not PRN)
Appropriate analgesia for breakthrough pain (PRNs) must be prescribed
Present U
Grade: D Not present Not present PRN prescribed: Proportion of
patients with pain that had a PRN dose prescribed
Proportion of patients with a PRN dose prescribed, for whom the analgesia, dose, route and fre- quency were appropriate (see below)
Appropriate analgesia: for PRN (if there is a regular opioid to control pain the breakthrough medication is the same)
Appropriate dose: for break- through pain is available (the usual dose for breakthrough pain is 10% of the total daily dose of the opioid)
Appropriate route: for PRN anal- gesia is being used (oral for mild–mod pain; parenteral for severe pain)
Appropriate frequency: PRN anal- gesia is prescribed at an appropriate frequency [ensure that the breakthrough medica- tion is available at appropriate times (frequency changes depending on the medication and route)]
Proportion of patients with a PRN dose prescribed, for whom the analgesia, dose, route and frequency were appropriate (see below) (17) Appropriate analgesia for PRN (if there is a regular opioid to control pain the breakthrough medication is the same) (18) Appropriate dose for breakthrough pain is available (the usual dose for breakthrough pain is 10% of the total daily dose of the opioid) (19) Appropriate route for PRN analgesia is being used (oral for mild–mod pain; parenteral for severe pain) (20) PRN analgesia is pre- scribed at an appropriate frequency (ensure that the breakthrough medica- tion is available at appro- priate times – frequency varies depending on the medication and route)
Implement the pain management plan using principles that maximize effi- cacy and minimize the adverse effects of pharmacological interventions including multi- modal analgesic approach; chang- ing opioids, dose or route PRN; pre- vention of adverse effects; and pre- vention, assess- ment and management of opioid risk
Not present Present U
Level of evidence: Ib Present U
Level of evi- dence: IIb
Multimodal approach: Proportion of patients that had pain epi- sodes for whom a multimodal analgesic approach was used (see below)
A multimodal approach includes nonopioid analgesics (e.g. anti- inflammatory drugs), opioids (e.g. morphine) and adjuvant medications (e.g. anticonvul- sants)
Treatment changed: Proportion of patients that had pain that had the opioid changed/ titrated or route changed
Bowel regime: Proportion of patients receiving opioids that had a bowel regime prescribed
Antiemetic: Proportion of patients that had an antiemetic pre- scribed
Opioid risk: Proportion of patients receiving opioids that had an opioid risk assessment com- pleted
(21) Proportion of patients that had a pain episode that were treated with a pharmacologic approach
(22) Proportion of patients that had pain episodes for whom a multimodal analgesic approach was used (see below) A multimodal approach includes nonopioid analgesics (e.g. anti- inflammatory drugs), opioids (e.g. morphine) and adjuvant medications (e.g. anticonvulsants)
(23) Proportion of patients receiving opioids that had a bowel regime pre- scribed
(24) Proportion of patients that had an antiemetic prescribed
BPI, Brief Pain Inventory; ESAS, Edmonton Symptom Assessment Scale; HCP, health care provider; IP, interprofessional; PRN, as needed.
IMPLEMENTATION PROJECT
International Journal of Evidence-Based Healthcare � 2019 University of Adelaide, Joanna Briggs Institute 99
©2019 University of Adelaide, Joanna Briggs Institute. Unauthorized reproduction of this article is prohibited.
VJ Fiset et al.
©2019 Un
were reduced into feasible and measureable indicators,
as many of the guideline recommendations reflect more
than one clinical ‘action’, or the recommendation was too
vague to create a concrete indicator, making it difficult to
have one indicator to reflect one recommendation, and
to operationalize the indicator.
For example, one recommendation was ‘Screen for
the presence or risk of any type of pain on admission or
visit with a HCP; after a change in medical status, prior to,
during and after a procedure’. For the purposes of gap
analysis on an inpatient unit, knowing how pain assess-
ment and management were documented, the indicator
was worded as ‘Proportion of patients that had a pain
assessment documented on admission’. In the case of
the recommendation ‘Reassess the person’s response to
the pain management interventions by using the same
re-evaluation tool’,9(p.7) it was unclear if the recommen-
dation is focused on the what – reassessment of the
person’s response to the pain management interven-
tions, or if it is focused on how – through the use of the
same re-evaluation tool. A list of initial indicators was
created and specified as proportions, and are included in
Table 4. A final list of indicators was achieved through
informal consensus during focus groups with the advi-
sory committees on each site, and research team meet-
ings. The final list of indicators is included in Table 4.
Practice test For the practice test, a retrospective chart audit was
conducted, reflecting 4 days of practice on the two units.
The chart audits reflected the full 96 h of patient care,
and included the students’, staff nurses’ and other clini-
cians’ pain management practices, as well as the
patients’ pain scores. All the charts were reviewed, as
opposed to a just a representative sample of charts, as
the goal of the audit was to describe the care of the
entire population of patients on the units for the
audit days.
Neither site had an electronic health record – at one
site, scanned electronic versions of charts were used for
data collection, and at the other, article versions of charts
were used. A codebook was developed with definitions
of the indicators, as well as a description of where data
could be found in the charts. Chart audits were com-
pleted by research assistants (a clinical nurse employed
at each hospital, familiar with the patient population and
documentation system) and the primary investigator (a
nurse educator). Both had clinical expertise in oncology
and palliative care. The primary investigator trained the
research assistants by reviewing the overall purpose of
the study; the purpose of the audits; the codebook and
how to enter data using the Excel spreadsheet.
100 International Journal of Evidence-Based
iversity of Adelaide, Joanna Briggs Institute. U
Chart abstractions were completed jointly (the same
charts reviewed at the same time by the research assis-
tant and primary investigator) and then separately, com-
paring results side-by-side to ensure consistency. At one
site, after reviewing five charts, extraction sheets were
developed to promote accurate chart data extraction as
each document (i.e. medication administration records,
flow sheets, progress notes) had to be opened separately
making it difficult to track documentation across a single
patient and pain episode. Data were then entered on an
Excel spreadsheet. At each site, 10 charts were reviewed
by both the principal investigator and the research
assistant, and results (by variable and cases) were com-
pared using SPSS version 24 (IBM Corp., Armonk, NY,
USA) to ensure agreement. When there was a discrep-
ancy between the two auditors, the charts were checked
again to ensure correct information was collected. Reg-
ular discussions were held between the audit team
members to ensure ongoing agreement and opportu-
nities for clarification. Notes were kept by the auditors,
providing clear explanation and rationale for some of the
data collected. For example, notes were made if patients
had a continuous infusion of opioids or if it was uncertain
if an opioid analgesic was given for pain, sedation or
shortness of breath. The practice test demonstrated that
the quality indicators identified were feasible and meas-
ureable, and would provide data helpful for identifying
practice gaps for pain management.
Discussion The process of identifying practice guidelines and rec-
ommendations, through to development and testing of
indicators was described. Overall, the process of deter-
mining and practice-testing guideline-based quality
indicators is complex and labour-intensive. Throughout
this process several observations were made that can
help inform the development of guidelines in the future,
and serve as guidance for teams seeking to identify the
gap between recommended and actual practice.
Guideline selection: retrieval and content At the guideline selection stage, it was important to
ensure that the practice guideline selection process was
replicable. It was also important to ensure that the most
recent version of the guideline was retrieved as there
were, in some cases, a number of versions of guidelines.
Access to some guidelines requires membership in asso-
ciations or networks. The assistance of a research librar-
ian and library technician was pivotal in achieving this
component of the project.
One issue related to guideline selection is that there
are many types of guidelines for pain management, and
Healthcare � 2019 University of Adelaide, Joanna Briggs Institute
nauthorized reproduction of this article is prohibited.
IMPLEMENTATION PROJECT
©2019 Un
they vary in content focus: by types of pain (i.e. neuro-
pathic, chronic, others); by patient populations (i.e.
homeless); and by phase of the patient care process
(i.e. assessment, procedure-related). Some guidelines are
very specific in terms of the types of interventions
included. For example, some guidelines focus solely
on pharmacologic management. Others have a more
holistic and broad focus, providing guidance in terms of
approaches to care from assessment to evaluation; and
addressing interventions that are both pharmacologic
and nonpharmacologic. This latter approach is most
helpful to guide practice, especially in light of the
subjective and multifactorial nature of pain.
The content of the three guidelines that eventually
informed the indicator development were very similar:
one developed in Ontario, Canada, based on a Scottish
guideline,23,24 by an interprofessional working group;
another developed by nurses in Ontario, Canada with
national representation on the development panel19; and
the third developed by a multidisciplinary team of Euro-
pean physicians.25 Given the similarities across the three
guidelines, and the commitment of the WHO27 to the
adequate treatment of pain, it would be beneficial for
international and interprofessional collaborations to
develop holistic pain guidelines. The International Associ-
ation for the Study of Pain offers assistance such as
administrative support and possibly funding to members
and organizations28 to develop guidelines focused on
clinical practicemanagement of a specific clinical situation
or condition or new technologies or treatments.
Guideline selection: quality The variability in guideline quality could be readily
addressed by guideline developers using the AGREE II
criteria,21 or the Institute of Medicine’s Standards for
Developing Trustworthy Clinical Practice Guidelines29
to guide the creation of the guidelines. The National
Guideline Clearinghouse (NGC) has required, in their
most recent inclusion criteria that guideline summaries
for consideration be consistent with the latter set of
standards.30 Unfortunately, the NGC website will no
longer be available after 16 July 2018, as federal funding
for the programme is no longer available.31
Another issue related to the quality of the guidelines
relates to the way in which the levels of evidence were
presented and how recommendations were graded
(Table 3). At the time that the three included guidelines
were developed, there was inconsistency in how the
strength of the evidence that supported the recommen-
dations was rated and how the grading systems were
used. For example, in the case of the grading of recom-
mendations, Cancer Care Ontario (CCO)26 graded the
International Journal of Evidence-Based Healthcare � 2019 University
iversity of Adelaide, Joanna Briggs Institute. Un
recommendations using the ABCD system; the European
Association of Urology (EAU)25 used a modified version
of the ABCD grading system; and the RNAO19 did not
grade the recommendations in their guideline.
Approaches to grading of recommendations are
evolving. As of 2013, SIGN is not using the ABCD grading
of recommendations that was evident in the CCO guide-
line,32 and are instead implementing the GRADE system,
rating recommendations as either ‘strong’ (for interven-
tions that should or should not be used) or ‘conditional’
(for interventions that should be considered).33 The
authors of the GRADE system indicate that quality of a
body of evidence (vs. individual studies) should be
considered along a continuum of high, moderate, low
and very low.34 The National Institute for Health and Care
Excellence also support the use of the GRADE system35;
however, the NGC is not requiring the use of GRADE for
guidelines included in their database.32 The JBI, as of
2014, is using the GRADE systemwith similar two levels –
Grade A (strong recommendations) and Grade B (weak
recommendations).36 There is a global trend toward
using the GRADE system for rating the quality of evi-
dence.37 Advantages to the use of the GRADE system
include ‘guideline development by international experts;
differentiation between the quality and strength of the
evidence; and the provision of clear interpretations of
recommendations for stakeholders’.38(p.855)
Wording of recommendations In terms of selection of the recommendations, it was very
helpful to have a sample template for use in comparing
the presence and strength of recommendations across
included guidelines.22 This comparison facilitated the
review of the recommendations by the advisory com-
mittees as part of the determination of quality indicators
selected. Many of the guideline recommendations
reflect more than one clinical ‘action’, making it difficult
to have one indicator to reflect one recommendation,
and to operationalize the indicator. Examples have been
provided in the results section, but another includes
‘Establish a comprehensive plan of care that incorporates
the goals of the person and IP team’.19(p.8) Given the
holistic nature of pain, an interprofessional team
approach is essential. The wording of this recommenda-
tion includes both what – establishing a comprehensive
plan of care, as well as who – the interprofessional team.
The latter indicator was feasible to gather through the
practice chart audit, not the former, as there was no
documentation tool outlining the plan of care.
In addition, it was challenging to create measureable
indicators for some recommendations. For example, in
the recommendation above related to interprofessional
of Adelaide, Joanna Briggs Institute 101
authorized reproduction of this article is prohibited.
VJ Fiset et al.
©2019 Un
team, what constitutes involvement of different team
members? Is it one team member beyond the physician
and nurse? Should consideration be given to the most
appropriate member of the interprofessional team given
the patient and family circumstance? For the recommen-
dation related to reassessment of pain following an
intervention, a timeframe should be included to guide
the reassessment, based on the expected time to effec-
tiveness of the intervention delivered.
Selection of indicators Ideally, guidelines would be presented in a way that
facilitates their use for measuring evidence-practice gaps
and outcomes. One of the AGREE II items under the
applicability domain, is that ‘the guideline presents
monitoring and/or auditing criteria’.2(p.3) None of the
three guidelines included in this process provided a
complete set of indicators directly related to the
included recommendations. Including a range of possi-
ble indicators to assess adherence to guideline recom-
mendations would facilitate guideline implementation
teams in assessing gaps as well as in evaluating the
implementation of the guidelines. An example of a
guideline that includes key quality indicators and targets
is the Canadian Stroke Best Practice Recommendations.39
Evidence summaries from the JBI have indicators that are
presented as evidence-based audit criteria that are read-
ily available to members through their PACES.40
The RNAO has developed and made available, to Best
Practice Spotlight Organizations41 (BPSOs), Nursing
Quality Indicators for Reporting and Evaluation (NQuIRE).
NQuIRE is ‘a database of quality indicators derived from
recommendations within RNAO’s clinical Best Practice
Guidelines’.24 This rich source of indicators based on
RNAO’s guidelines are only available to BPSOs at
this time.
Practice test The practice test was greatly facilitated by the develop-
ment of the coding book; however, one site-specific
challenge was extracting data from scanned documents
from patient charts. Development of a data tracking form
to document each pain episode from a variety of sources
(progress notes, medication administration records and
assessment flow sheets) expedited the data extraction
process. The practice test demonstrated that the quality
indicators identified were feasible and measureable, and
would provide helpful data for identifying practice gaps
for pain management. However, the practice test was
time consuming, which could limit feasibility for imple-
mentation teams, because the volume of data necessi-
tated having one person dedicated to the data collection
102 International Journal of Evidence-Based
iversity of Adelaide, Joanna Briggs Institute. U
(approximately 1.5 h per chart), and another person
available to verify the collected. However, most clinical
settings now have electronic health records that would
greatly facilitate this process.
Project limitations One limitation of this project is that the search strategy
may not have identified all relevant guidelines. However,
we replicated the search strategy used to identify pain
guidelines from two credible guideline developer groups
and we therefore think it is unlikely wemissed important
guidelines that may have been available at the time of
the search. Only two reviewers conducted the guideline
quality assessments. The authors of the AGREE II user
manual recommend that ‘each guideline be assessed by
at least two appraisers, and preferably four, as this will
increase the reliability of the assessment’.42 This limita-
tion likely did not result in the erroneous rejection of any
of the guidelines as the rigour scores of the excluded
guidelines were extremely low; however, the use of more
than two assessors would have been preferable.
Not using a more structured approach to seek con-
sensus on included recommendations and indicators is
another potential weakness of the project. In their recent
scoping review of indicator development methods, Xiao
et al.11 found that focus groups were used as a develop-
ment method in seven of 23 included studies. One of
their key conclusions was that formal decision making
approach such as the Delphi technique be used after the
review of the literature and key stakeholder consultation.
In terms of the clinical implications of our findings, the
guideline review is dated. This would limit the ability of
using the included indicators as there may be more
recent guidelines that would emerge if the search was
repeated. Since the time of the search, the RNAO guide-
line remains themost recent versions produced, the CCO
guideline was updated in September 2018, and the EAU
guideline has been discontinued. As the intent of the
article was to describe the process of developing guide-
line-based quality indicators, the fact that the actual
indicators that were developed may be dated does
not take away from the findings about the usefulness
and feasibility of the proposed process for developing
quality indicators from practice guidelines, or the diffi-
culties in doing so.
Conclusion Nothacker et al., on behalf of the Guidelines International
Network Performance Measures Working Group devel-
oped reporting standards for guideline-based perfor-
mance measures.26 The term performance measures
hasbeenused interchangeablywithquality indicators.43,44
Healthcare � 2019 University of Adelaide, Joanna Briggs Institute
nauthorized reproduction of this article is prohibited.
IMPLEMENTATION PROJECT
©2019 Un
All aspects of these standards are addressed in this article,
with the exception of ‘review and re-evaluation of perfor-
mance measures’.26(Table 2) Challenges in the process of
indicator development include the complexities inherent
in each step of the process, influenced by the nature of the
guidelines and their presentation. In situations where
quality indicators are not provided as part of relevant
practice guidelines, organizations implementing BPGs
could use this process to determine what guidelines
and guideline recommendations are most salient for their
clinical setting, and then proceed to develop relevant
indicators to be assessed. Once guideline recommenda-
tions have been put in to practice, the extent to which
practice has changed can be evaluated by using the same
indicators. In 2003, Dykes45 concluded that ‘. . .the ability
to measure adherence with guideline recommendations
and the impact of those guidelines on practice patterns
and patient outcomes is vital’.(p.67) Fifteen years later, this
imperative remains. Experience with developing guide-
line-based quality indicators has shown that this area of
health carequality improvement andevaluationcontinues
to evolve and to be an essential part of guideline use.
Acknowledgements Conflicts of interest The work contained in this review was supported by a
Doctoral Research Award from the Canadian Institutes of
Health Research (CIHR) to Valerie Fiset, and by a CIHR
Foundation Grant no. 143237 to I.D.G. For the remaining
authors, none were declared.
References 1. Registered Nurses’ Association of Ontario. Toolkit: imple-
mentation of best practice guidelines. 2nd ed. Toronto
(ON): Registered Nurses’ Association of Ontario; 2012;
Available from: http://rnao.ca/sites/rnao-ca/files/RNAO_-
ToolKit_2012_rev4_FA.pdf. [Cited 23 July 2018].
2. Innis J, Dryden-Palmer K, Perreira T, Berta W. How do health
care organizations take on best practices? A scoping
review. Int J Evid Based Healthc 2015; 13: 254–72.
3. Harrison MB, van den Hoek J. Canadian Guideline Adapta-
tion Study Group. CAN-IMPLEMENT�: a guideline adapta-
tion and implementation planning resource Kingston (ON):
Queen’s University and Canadian Partnership against Can-
cer; 2011.
4. Graham ID, Logan J, Harrison MB, et al. Lost in knowledge
translation: time for a map? J Contin Educ Health Prof 2006;
26: 13–24.
5. Joanna Briggs Institute. JBI PACES user guide. [Internet].
Adelaide, AU: Joanna Briggs Institute; 2009; Available from:
http://paces.jbiconnectplus.org/GlobalResources/Paces_
UserGuide.pdf. [Cited 23 July 2018; updated 12 March
2009].
International Journal of Evidence-Based Healthcare � 2019 University
iversity of Adelaide, Joanna Briggs Institute. Un
6. Choi M, Kim HS, Ahn MJ, et al. Evidence-based practice for
pain management for cancer patients in an acute care
setting. Int J Nurs Pract 2014; 20: 60–9.
7. Ong AY, Tan J, Yeo HL, Goh ML. Patient-centred hand
hygiene information in orthopaedics units: an evidence-
based implementation project. Int J Evid Based Healthc
2017; 15: 22–9.
8. Shum A, Wong KS, Sankara K, Goh ML. Securement of the
indwelling urinary catheter for adult patients: a best prac-
tice implementation. Int J Evid Based Healthc 2017; 5: 3–12.
9. Sykes PK, Walsh K, Darcey CM, et al. Prevention of venous
thromboembolism amongst patients in an acute tertiary
referral teaching public hospital: a best practice implemen-
tation project. Int J Evid Based Healthc 2016; 14: 64–73.
10. Kitson AL, Straus SE. Identifying knowledge to action gaps.
In: Straus SE, Tetroe J, Graham ID, editors. Knowledge
translation in health care: moving from evidence to practice.
Chichester, UK: John Wiley & Sons, 2013; pp. 97–109.
11. Xiao S, Widger K, Tourangeau A, Perta W. Nursing process
health care indicators: a scoping review of development
methods. J Nurs Care Qual 2017; 3: 32–9.
12. Kotter T, Blozik E, Sherer M. Methods for the guideline
based development of quality indicators – a systematic
review. Implement Sci 2012; 7: 21.
13. College of Nurses of Ontario. Competencies for entry-
level registered nurse practice. Toronto (ON): College of
Nurses of Ontario; 2014; Available from: http://www.cno.
org/globalassets/docs/reg/41037_entrytopracitic_final.pdf.
[Cited 23 July 2018].
14. Canadian Association of Schools of Nursing. Palliative and
end-of-life care: entry-to-practice competencies and indi-
cators for registered nurses Ottawa (ON): Canadian Associ-
ation of Schools of Nursing; 2011; Available from: http://
casn.ca/wp-content/uploads/2014/12/PEOLCCompeten-
ciesandIndicatorsEn1.pdf. [Cited 23 July 2018].
15. Herr K, Marie St, Gordon B, et al. An interprofessional
consensus of core competencies for pre-licensure educa-
tion in pain management: curriculum application for nurs-
ing. J Nurs Educ 2015; 54: 317–27.
16. Ung A, Salamonson Y, Hu W, Gallego G. Assessing knowl-
edge, perceptions and attitudes to pain management
among medical and nursing students: a review of the
literature. Br J Pain 2016; 10: 8–21.
17. Greco MT, Roberto A, Corli O, et al. Quality of cancer pain
management: an update of a systematic review of under-
treatment of patients with cancer. J Clin Oncol 2014; 32:
4149–54.
18. Fervers B, Burgers JS, Haugh MC, et al. Adaptation of clinical
guidelines: literature reviewand proposition for a framework
and procedure. Int J Qual Health Care 2006; 18: 167–76.
19. Registered Nurses’ Association of Ontario. Clinical best
practice guidelines: assessment and management of
pain /sb:maintitle> Toronto (ON): Registered Nurses’ Associ-
ation of Ontario; 2013; Available from: http://rnao.ca/sites/
rnao-ca/files/AssessAndManagementOfPain2014.pdf. [Cited
23 July 2018].
of Adelaide, Joanna Briggs Institute 103
authorized reproduction of this article is prohibited.
VJ Fiset et al.
©2019 Un
20. Stacey D, Macartney G, Carley M, Harrison MB. The Pan-
Canadian Oncology Symptom Triage and Remote Support
Group (COSTaRS). Development and evaluation of evi-
dence-informed clinical nursing protocols for remote
assessment, triage and support of cancer treatment-
induced symptoms. Nurs Res Pract 2013; 2013: 171872.
21. Brouwers M, Kho ME, Browman GP, et al. AGREE II: advanc-
ing guideline development, reporting and evaluation in
healthcare. Can Med Assoc J 2010; 182: E839–42.
22. Graham ID, Harrison MB, Lorimer K, et al. Adapting national
and international leg ulcer practice guidelines for local use:
the Ontario Leg Ulcer Community Care Protocol. Adv Skin
Wound Care 2005; 18: 307–18.
23. Scottish Intercollegiate Guidelines Network. SIGN guide-
lines 106 control of pain in adults with cancer: a National
Clinical Guideline Edinburgh (UK): Scottish Intercollegiate
Guidelines Network; 2008;Available from: http://www.sign.
ac.uk/assets/sign106.pdf. [Cited 23 July 2018].
24. Cancer Care Ontario. Cancer Care Ontario’s symptom man-
agement guides-to-practice: pain Toronto (ON): Cancer
Care Ontario; 2010; Available from: https://www.cancercar-
e.on.ca/toolbox/symptools/. [Cited 23 July 2018].
25. Paez Borda A, Charnay-Sonnek F, Fonteyne V, Papaioannou
EG. Guidelines on pain management & palliative care.
Arnhem (NL): European Association of Urology; 2013 ;
Available from: https://uroweb.org/wp-content/uploads/
25-Pain-Management_LR.pdf. [Cited 23 July 2018].
26. Nothacker M, Stokes T, Shaw B, et al. Reporting standards
for guideline-based performance measures. Implement Sci
2016; 11: 6.
27. World Health Organization. WHO normative guidelines on
pain management: report of a Delphi study to determine
the need for guidelines and to identify the number and
topics of guidelines that should be developed by WHO
Geneva (CH): World Health Organization; 2007 ; Available
from: http://www.who.int/medicines/areas/quality_safety/
delphi_study_pain_guidelines.pdf. [Accessed 23 June
2017; cited 23 July 2018].
28. International Association for the Study of Pain. Develop-
ment of clinical practice guidelines in the field of pain.
Washington (DC): International Association for the Study
of Pain; 2017; Available from: https://www.iasp-pain.org/
Education/Content.aspx?ItemNumber=1218. [Cited 23 July
2018].
29. Institute of Medicine of the National Academies. Clinical
practice guidelines we can trust standards for developing
trustworthy clinical practice guidelines (CPGs). Washington
(DC): National Academies Press; 2011;Available from: http://
www.nationalacademies.org/hmd/�/media/Files/
Report%20Files/2011/Clinical-Practice-Guidelines-We-Can-
Trust/Clinical%20Practice%20Guidelines%202011%20In-
sert.pdf. [Cited 23 July 2018].
30. National Guideline Clearinghouse. Guideline summaries
FAQs. Rockville (MD): Agency for Healthcare Research
and Quality; 2017; Available from: https://guideline.gov/
help-and-about/summaries/faq. [Cited 4 July 2018].
104 International Journal of Evidence-Based
iversity of Adelaide, Joanna Briggs Institute. U
31. National Guideline Clearinghouse. Homepage. Rockville (MD):
Agency for Healthcare Research and Quality; Date unknown;
Available from:https://www.guideline.gov/. [Cited4 July 2018]
32. Scottish Intercollegiate Guidelines Network. Methodologi-
cal principles. Edinburgh (UK): Scottish Intercollegiate
Guidelines Network; Date unknown; Available from:
http://www.sign.ac.uk/methodological-principles.html.
[Cited 23 July 2018]
33. Guyatt GH, Oxman AD, Vist GE, et al. GRADE: an emerging
consensus on rating quality of evidence and strength of
recommendations. Br Med J 2008; 33: 924–6.
34. Balsham H, Helfand M, Schunemann HJ, et al. GRADE
guidelines: 3. Rating the quality of evidence. J Clin Epidemiol
2011; 64: 401–6.
35. National Institute for Health and Care Excellence. Reviewing
research evidence. London (UK): National Institute for
Health and Care Excellence; 2017; Available from: https://
www.nice.org.uk/process/pmg20/chapter/reviewing-
research-evidence#assessing-the-quality-of-the-evidence.
[Cited 23 July 2018].
36. Joanna Briggs Institute. Grades of recommendations. Ade-
laide (AU): Joanna Briggs Institute; 2014; Available from: http://
joannabriggs.org/jbi-approach.html#tabbed-nav=Grades-of-
Recommendation. [Cited 23 July 2018; updated March 2014].
37. Dijkers M. Introducing GRADE: a systematic approach to
rating evidence in systematic reviews and to guideline
development. KT Update 2013; 5: 9.
38. Cruz JE, Fahim G, Moore K. Practice guideline development,
grading, and assessment. J Clin Pharm Ther 2015; 40: 854–7.
39. Heart and Stroke Foundation. Quality of stroke care in
Canada: key quality indicators and stroke case definitions
Ottawa (ON): Heart and Stroke Foundation; 2016; Available
from: http://www.strokebestpractices.ca/wp-content/
uploads/2016/09/2016-StrokeCaseDefn-KQI-Update-
FINAL-Sept2016.pdf. [Cited 23 July 2018].
40. Joanna Briggs Institute. Welcome to JBI PACES. Adelaide
(AU): Joanna Briggs Institute; Date unknown; Available
from: http://paces.jbiconnectplus.org/. [Cited 23 July 2018]
41. Registered Nurses’ Association of Ontario. Nursing quality
indicators for reporting and evaluation (NQuIRE). Toronto
(ON): Registered Nurses’ Association of Ontario; Date
unknown; Available from: http://rnao.ca/bpg/initiatives/
nquire. [Cited 23 July 2018]
42. AGREE Next Steps Consortium. The AGREE II instrument.
Hamilton (ON): AGREE Next Steps Consortium; 2017 ;
Available from: https://www.agreetrust.org/wp-content/
uploads/2017/12/AGREE-II-Users-Manual-and-23-item-
Instrument-2009-Update-2017.pdf. [Cited 23 July 2018].
43. Gagliardi AR, Brouwers MC, Palda VA, et al. How can we
improve guideline use? A conceptual framework of imple-
mentability. Implement Sci 2011; 6: 26.
44. Health Quality Ontario. Measurement for quality improve-
ment. Toronto (ON): Queen’s Printer for Ontario; 2013; 18p,
Available from: http://www.hqontario.ca/Portals/0/Docu-
ments/qi/qi-measurement-primer-en.pdf. [Cited 23 July
2018].
Healthcare � 2019 University of Adelaide, Joanna Briggs Institute
nauthorized reproduction of this article is prohibited.
IMPLEMENTATION PROJECT
©2019 Un
45. Dykes PC. Practice guidelines and measurement: state-of-
the-science. Nurs Outlook 2003; 51: 65–9.
46. Price C, Lee J, Taylor AM, Baranowski AP. British Pain Society.
Initial assessment and management of pain: a pathway for
care developed by the British Pain Society. Br J Anaesth
2014; 112: 816–23.
47. National Comprehensive Cancer Network. NCCN clinical
practice guidelines for the management of pain at the
end of life in adult patients. Fort Washington (PA): National
Comprehensive Cancer Network; 2014; Available from:
https://www.nccn.org/professionals/physician_gls/defaul-
t.aspx#supportive. [Cited 23 July 2018].
48. Guidelines and Audit Implementation Network. General
palliative care guidelines for the management of pain at
the end of life in adult patients. Belfast (UK): Guidelines and
Audit Implementation Network; 2011; Available from:
https://www.rqia.org.uk/RQIA/files/e0/e0a81c25-acb8-
4982-9970-1ed62e9f2015.pdf. [Cited 23 July 2018].
49. Tasmanian Department of Health and Human Services.
Care management guidelines: pain management. Hobart
(TAS): Tasmanian Department of Health and Human
International Journal of Evidence-Based Healthcare � 2019 University
iversity of Adelaide, Joanna Briggs Institute. Un
Services; Date unknown; Available from: http://
www.dhhs.tas.gov.au/__data/assets/pdf_file/0006/36951/
Care_Management_Guidelines_-_Pain_management_in_-
palliative_care_-_20160622.pdf. [Cited 23 July 2018]
50. Ripamonti CI, Santini D, Maranzano E, et al. ESMO Guide-
lines Working Group. Management of cancer pain: ESMO
clinical practice guidelines. Ann Oncol 2012; 23 (Suppl 7):
vii39–154.
51. Horgas AL, Yoon SL, Grall M. Pain management. In: Boltz M,
Capezuti E, Fulmer T, Zwicker D, editors. Evidence-based
geriatric nursing protocols for best practice. New York, NY:
Springer Publishing Company, 2012; pp. 246–267.
52. Winnipeg Regional Health Authority. Pain assessment and
management: clinical practice guidelines Winnipeg (MB):
Winnipeg Regional Health Authority; 2012; Available from:
http://www.wrha.mb.ca/extranet/eipt/files/EIPT-017-
001.pdf. [Cited 23 July 2018].
53. Health Care Association of New Jersey. Pain management
guideline. Hamilton (NJ): Health Care Association of New
Jersey; 2006; Available from: http://www.hcanj.org/files/
2013/09/hcanjbp_painmgmt2_3.pdf. [Cited 23 July 2018].
of Adelaide, Joanna Briggs Institute 105
authorized reproduction of this article is prohibited.
