Customer Experience Scenario

Editorial

Using quality improvement methods for evaluating health care A Niroshan Siriwardena MMedSci PhD FRCGP Foundation Professor of Primary Care, School of Health and Social Care, University of Lincoln, UK

Quality improvement initiatives are a ubiquitous feature

of modern healthcare systems because of actual and

perceived gaps in the quality of healthcare delivery.1,2

However, such initiatives are often not subject to

evaluation, or when evaluation is conducted this is

done poorly.3

Quality improvement methods are increasingly being used to aid diffusion of innovations in health and can

be used as a research tool to model and design complex

healthcare interventions.4 However, as well as being

components of quality improvement programmes they

can sometimes be a useful adjunct to other more trad-

itional evaluation methods, thus serving a dual role.

Evaluation is often undertaken to determine the

quality of care being provided by an individual, team or service where quality is taken to mean the effec-

tiveness, efficiency, safety or patient experience of that

care.1 Evaluation is also undertaken to ensure that the

aims of care are being met, to provide information for

service users, commissioners, healthcare providers or

other stakeholders about the quality of services being

provided, and finally to establish the basis for future

improvements. Quality improvement research is ap- plied research involving evaluation of quality improve-

ment initiatives which is aimed at informing policy

and practice.5 Current guidelines for reporting quality

improvement include ‘descriptions of the instruments

and procedures (qualitative, quantitative or mixed)

used to assess the effectiveness of implementation, the

contributions of intervention components and context

to effectiveness of the intervention and the impact on

primary and secondary outcomes’.6

A useful starting point for an evaluation is a logic

model where the clinical population and problem that

the healthcare intervention is aimed at, inputs (in

terms of resources provided for planning, implemen-

tation and evaluation), outputs (in terms of healthcare processes implemented and the population that is actu-

ally reached) and longer-term outcomes are measured

in terms of health and wider benefits or harms, whether

intended or incidental and in the short, medium or

long term (see Figure 1).7

A logic model can be expanded, either as a whole or

in specific areas to form a ‘cause and effect’ (sometimes

call a fishbone or Ishikawa) diagram (see Figure 2). The central line representing the patient pathway, is

affected by patients themselves, but also by the other

inputs and outputs (processes) as patients are travel-

ling through the healthcare system being evaluated.8

Traditional evaluation methods look at the struc-

ture, processes (outputs) or outcomes of care using

various qualitative or quantitative methods (see Box 1).9

However, a number of quality improvement methods can also be used for evaluation and these overlap

considerably with traditional evaluative techniques

(Box 2). These methods have potential to enable better

understanding of the processes of care and, import-

antly, to shed light on how to improve upon these.

Clinical audit, which is the ‘systematic, critical analysis

of the quality of medical care, including the procedures

Figure 1 A logic model for evaluating health care

Quality in Primary Care 2009;17:155–9 # 2009 Radcliffe Publishing

AN Siriwardena156

used for diagnosis and treatment, the use of resources and the resulting outcome for the patient’10 builds

evaluation into the process. It involves measurement

of care (‘how are we doing?’) against established criteria

and standards (‘what should we be doing?’) through

which performance and changes in performance can

be measured (‘have the changes we have made led to

improvement?’). Audit can and has been used as an

evaluation method, even in randomised studies. Significant event audit is another technique that is

frequently used to evaluate care, particularly care that

is considered to fall below standards or that is out-

standingly good.11 It is a powerful tool for evaluating

healthcare processes by attempting to understand the

detailed factors that led to care being outside the

norm, but it can also help improve communication,

team building and quality.12

Plan, do, study, act (PDSA) cycles are another

means of investigating care processes while rapidly

implementing evidence-based or common sense

changes to processes of care, enabling changes to be

spread more easily and effectively.13 The third stage of

the PDSA cycle involves studying the effect of a change

using numerical or qualitative data – even with small-

scale changes, the effect over time on processes of care

can be measured and analysed using statistical process

control techniques. The PDSA model is a useful means

of evaluating while introducing rapid change to health-

care processes.14

Focus groups and individual interviews are import-

ant traditional techniques for gathering data about the experiences of patients and staff about services. An

important quality improvement tool, which is a de-

velopment from this, is the ‘discovery interview’.15

This narrative technique involves listening to the

Box 2 Examples of quality improvement evaluation methods

Audit and improvement cycles 1 Clinical audit

2 Significant event analysis

3 Plan–do–study–act cycles

Analysis of barriers and facilitators to improve- ment

4 Discovery (narrative) interviews, focus groups

5 Participant and non-participant observation,

naturalistic story gathering (ethnography)

6 Organisational case study

7 Critical to quality (CTQ) trees

Change management 8 WIFM (‘what’s in it for me’) charts

9 Strengths, weaknesses, opportunities, threats

(SWOT) or strengths, challenges, opportun-

ities, threats (SCOT) analysis 10 Force field analysis

Transformation methods 11 Process redesign 12 Collective sense making (action research)

Measurement for change 13 Benchmarking

14 Confidence charts or funnel plots

Box 1 Examples of traditional healthcare evaluation methods

Structure or processes of care (outputs) 1 Equipment, staff, guidelines, protocols

2 Process and pathway mapping

3 Process performance measurement against in-

dicators

Outcomes/impact of care 4 Cost analysis

5 Intermediate (proxy) or true health outcome

measures

6 Adverse event analysis

Both 7 Patient or staff questionnaires

Figure 2 Cause and effect (‘fishbone’) diagram

Quality improvement methods for evaluating health care 157

stories of patients and carers of the care that they have

received in order to understand experiences from a

user perspective. Other narrative techniques for qual-

ity improvement research and evaluation include

naturalistic story gathering during a project or collec-

tive sense-making of a complete project by a partici- pant observer and the organisational case study.5

Root cause analysis is a specific type of significant

event analysis which aims to find explanations for

adverse or untoward events through the systematic

review of written and oral evidence to establish under-

lying causes.16 The analysis involves defining the

problem, gathering evidence, identifying possible

root causes and the underlying reasons for these and

then deciding which causes are amenable to change.

This leads to recommendations, the effect of which

can be further evaluated.17

The Pareto (or 80/20) principle (see Figure 3),

describes how a relatively small number of key causes

will lead to most of the important outcomes, for example, 80% of outputs, outcomes or harms are due

to 20% of inputs or causes. This can help to distin-

guish the most important causes.18

Process mapping can describe the patient journey

through the system of care and even complex path-

ways can be visualised using spaghetti diagrams or

‘swim lane’ diagrams (see Figure 4) to separate pro-

cesses into different job roles or team activities.

Figure 3 Pareto diagram for prescribing errors

Figure 4 Swim lane diagram for asthma care

AN Siriwardena158

Components of a process which are critical to quality

(CTQ) can be represented as a CTQ tree (see Figure 5).

Such evaluations can determine whether the right

treatment is given by the right person at the right

time and place.19

Another important aspect of evaluation is the human factors involved in change.20 Ownership of

change is particularly important for healthcare pro-

fessionals, such as doctors and nurses, who at the front

line of care have the power to promote or subvert

change. This, the inverted pyramid of control,21 has

been applied to health care to emphasise the import-

ance of clinical leadership.22 An understanding of

internal strengths and challenges (weaknesses) as well as external opportunities and threats, together with

individual and group drivers and barriers to change is

critical to successful health services, an approach

which has its basis in Lewin’s ‘forcefield theory’.23

Comparing and benchmarking individual or

organisational performance using statistical process

control can help identify differences or gaps in per-

formance,24 which enable ‘special causes’ to be high- lighted and explanations to be sought to look at ways

of changing practice to improve performance (Figure 6).

Statistical process control charts plotted against

time can also show where improvements have occurred

in response to planned interventions,25 and feedback

using this technique as part of ongoing evaluation can

contribute to improvement.26,27

Larger-scale evaluation or more robust evalu- ations may require more complex techniques such

as quasi-experimental methods including time series or

Figure 6 Funnel plot showing institutional performance for aspirin administration to patients with ST-elevation myocardial infarction

Figure 5 Critical to quality (CTQ) tree

Quality improvement methods for evaluating health care 159

non-randomised control group designs as well as

cost analysis.28,29

Quality improvement methods, despite their in-

creasing application to health services,30 have not been

widely considered or used as part of healthcare evalu-

ation but could provide a useful addition to the evaluative techniques that are currently in use.

REFERENCES

1 Darzi AD. High Quality Care for All: NHS Next Stage

Review final report. London: Stationery Office, 2008.

2 Institute of Medicine. Crossing the Quality Chasm: a new

health system for the 21st century. Washington DC:

National Academy Press, 2001.

3 Øvretveit J. Producing useful research about quality

improvement. International Journal of Health Care Quality

Assurance Incorporating Leadership in Health Services 2002;

15:294–302.

4 Siriwardena AN. The exceptional potential for quality

improvement methods in the design and modelling of

complex interventions. Quality in Primary Care 2008;

16:387–9.

5 Greenhalgh T, Russell J and Swinglehurst D. Narrative

methods in quality improvement research. Quality and

Safety in Health Care 2005:14: 443–449.

6 Davidoff F, Batalden P, Stevens D, Ogrinc G and

Mooney S. Publication guidelines for quality improve-

ment in health care: evolution of the SQUIRE project.

Quality and Safety in Health Care 2008;17(Suppl 1):i3–

i9.

7 Medeiros LC, Butkus SN, Chipman H et al. A logic

model framework for community nutrition education.

Journal of Nutrition Education and Behaviour 2005;37:

197–202.

8 Volden CM and Monnig R. Collaborative problem

solving with a total quality model. American Journal of

Medical Quality 1993;8:181–6.

9 Marsh P and Glendenning R. The Primary Care Service

Evaluation Toolkit. Leeds: National Coordinating Centre

for Research Capacity Development, 2005.

10 Secretaries of State for Health, Wales, Northern Ireland

and Scotland. Working for Patients. The health service:

working for the 1990s. Cm 555. London: HMSO, 1989.

11 Pringle M. Significant event auditing. Scandinavian

Journal of Primary Health Care 2000;18:200–202.

12 Westcott R, Sweeney G and Stead J. Significant event

audit in practice: a preliminary study. Family Practice

2000;17:173–9.

13 Langley GJ. The Improvement Guide: a practical approach

to enhancing organizational performance. San Francisco:

Jossey-Bass, 1996.

14 Plsek P. Innovative thinking for the improvement of

medical systems. Annals of Internal Medicine 1999;131:

438–44.

15 NHS Modernisation Agency. A Guide to Using Discovery

Interviews to Improve Care. Leicester: Department of

Health, 2003.

16 Burroughs TE, Cira JC, Chartock P, Davies AR and

Dunagan WC. Using root cause analysis to address

patient satisfaction and other improvement opportunities.

The Joint Commission Journal on Quality Improvement

2000;26:439–49.

17 Woloshynowych M, Rogers S, Taylor-Adams S and

Vincent C. The investigation and analysis of critical

incidents and adverse events in healthcare. Health Tech-

nology Assessment 2005;9:1–143, iii.

18 Ziegenfuss JT Jr and McKenna CK. Ten tools of con-

tinuous quality improvement: a review and case example

of hospital discharge. American Journal of Medical Qual-

ity 1995;10:213–20.

19 NHS Modernisation Agency. Improvement Leaders’

Guide: process mapping, analysis and redesign. London:

Department of Health, 2005.

20 NHS Modernisation Agency. Improvement Leaders’

Guide: managing the human dimensions of change.

London: Department of Health, 2005.

21 Quinn JB. Intelligent Enterprise: a knowledge and service

based paradigm for industry. New York: Free Press, 1992.

22 Ham C. Improving the performance of health services:

the role of clinical leadership. Lancet 2003;361:1978–80.

23 Lewin K. Frontiers in group dynamics. Human Relations

1947;1:4–41.

24 Mohammed MA, Worthington P and Woodall WH.

Plotting basic control charts: tutorial notes for health-

care practitioners. Quality and Safety in Health Care

2008;17:137–45.

25 Mohammed MA. Using statistical process control to

improve the quality of health care. Quality and Safety in

Health Care 2004;13:243–5.

26 Thomson O’Brien MA, Oxman AD, Davis DA et al.

Audit and feedback: effects on professional practice and

health care outcomes. Cochrane Database of Systematic

Reviews 2000:CD000259.

27 Thor J, Lundberg J, Ask J et al. Application of statistical

process control in healthcare improvement: systematic

review. Quality and Safety in Health Care 2007;16:387–

99.

28 Ukoumunne OC, Gulliford MC, Chinn S, Sterne JAC

and Burney PGJ. Methods for evaluating area-wide and

organisation-based interventions in health and health-

care: a systematic review. Health Technology Assessment

1999:3.

29 Siriwardena AN. Experimental methods in health re-

search. In: Saks M and Allsop J (eds). Researching Health:

qualitative, quantitative, and mixed methods. Los Angeles:

Sage, 2007.

30 Plsek PE. Quality improvement methods in clinical

medicine. Pediatrics 1999;103:203–14.

CONFLICTS OF INTEREST

None.

ADDRESS FOR CORRESPONDENCE

A Niroshan Siriwardena, School of Health and Social

Care, University of Lincoln, Lincoln LN6 7TS, UK.

Tel: +44 (0)1522 886939; fax: +44 (0)1522 837058;

email: [email protected]