Annotated Bibliograpgy

Information in practice

Systematic review of cost effectiveness studies of telemedicine interventions Pamela S Whitten, Frances S Mair, Alan Haycox, Carl R May, Tracy L Williams, Seth Hellmich

Abstract Objectives To systematically review cost benefit studies of telemedicine. Design Systematic review of English language, peer reviewed journal articles. Data sources Searches of Medline, Embase, ISI citation indexes, and database of Telemedicine Information Exchange. Studies selected 55 of 612 identified articles that presented actual cost benefit data. Main outcome measures Scientific quality of reports assessed by use of an established instrument for adjudicating on the quality of economic analyses. Results 557 articles without cost data categorised by topic. 55 articles with data initially categorised by cost variables employed in the study and conclusions. Only 24/55 (44%) studies met quality criteria justifying inclusion in a quality review. 20/24 (83%) restricted to simple cost comparisons. No study used cost utility analysis, the conventional means of establishing the “value for money” that a therapeutic intervention represents. Only 7/24 (29%) studies attempted to explore the level of utilisation that would be needed for telemedicine services to compare favourably with traditionally organised health care. None addressed this question in sufficient detail to adequately answer it. 15/24 (62.5%) of articles reviewed here provided no details of sensitivity analysis, a method all economic analyses should incorporate. Conclusion There is no good evidence that telemedicine is a cost effective means of delivering health care.

Introduction Proponents of telemedicine have championed its potential economic benefits in recent years. In Britain telemedicine systems have been proposed as a cost effective means of responding to structural problems in the organisation of the NHS.1 2 In the United States telemedicine systems have been used, especially in rural areas, as a means of easing the problem of obtaining specialist advice and making referrals over wide distances.3 Even so, major questions about the cost effectiveness of telemedicine systems remain because the data used to support such claims are dispersed across studies of widely differing systems and diverse organisational contexts. Reported studies are

often small in scale, methodologically flawed, and reflect pragmatic evaluations rather than controlled trials, making them unsuitable for formal meta- analysis.4 Given these problems, our objective was to address the question of whether the existing literature allows any conclusions to be drawn about the cost effectiveness of telemedicine interventions.

Telemedicine is defined here as clinical practice for diagnosis, review, or management undertaken syn- chronously or asynchronously through the medium of information and telecommunications technologies (excluding telephone and fax).

Methods Two independent systematic reviews were started, one in the United States (Whitten and colleagues) and the other in Britain (Mair and colleagues). When each group became aware of the other’s work it was decided to adopt a common strategy and share results, while still reviewing articles independently. As a result, both groups performed analyses on articles obtained through a common search strategy. Both groups categorised articles into those with or without data. The first group analysed articles without data by themes and articles with data by results. The second group determined which articles with data met quality criteria justifying inclusion in a quality review and performed that review.

Search strategy We identified empirical studies of the costs associated with telemedicine interventions by searching the following electronic bibliographic databases: Medline (1966-June 2000), Embase (1988-June 2000), ISI Science Citation Index (1981-June 2000), ISI Social Science Citation Index (1981-June 2000), ISI Arts and Humanities Citation Index (1981-June 2000), and the database of the Telemedicine Information Exchange (TIE). Searching was restricted to articles in English language journals, with the keywords “telemedicine or telehealth or telemonitoring or telecommunications and cost or cost-effectiveness or economic or cost analy- sis or budget or financial or health care costs.” We used multiple keyword sets to maximise results from the searches.

Data analysis We divided the articles for analysis into two sets: those with and those without data. We categorised articles

Details of the studies reviewed are given in a table and reference list on bmj.com

Department of Telecommunications, Michigan State University, East Lansing, MI 48824-1212, USA Pamela S Whitten associate professor Seth Hellmich research assistant

Department of Primary Care, University of Liverpool Frances S Mair senior lecturer

Department of Pharmacology, University of Liverpool Alan Haycox senior lecturer

Centre for Health Services Research, University of Newcastle-upon-Tyne Carl R May professor of medical sociology

School of Primary Care, University of Manchester Tracy L Williams research fellow

Correspondence to: P S Whitten [email protected]

BMJ 2002;324:1434–7

1434 BMJ VOLUME 324 15 JUNE 2002 bmj.com

without data by theme, and achieved intercoder reliability of 90% for this thematic coding. For articles with data, we devised a data extraction instrument for coding aspects of each study. These included: location; type of service (such as specialty); delivery (synchro- nous or asynchronous); type of facility; duration and dates of study; number of patients, consultations, and images transmitted; cost revenue variables; and study conclusions.

The next stage required us to select articles with data for inclusion in a full review to assess their scien- tific quality. Variations in study design, patient groups, study context, and types of analysis meant that formal meta-analysis was inappropriate. For this reason, we present a qualitative analysis of the studies obtained. Although randomised controlled trials are the preferred source of evidence for inclusion in systematic reviews,5 hardly any such studies are to be found in the field of telemedicine. In view of this, we decided to include all published studies of telemedi- cine systems. When one project led to multiple reports (redundant publications), we reviewed the principal paper (with the greatest number of subjects) focusing on cost effectiveness. Our inclusion criterion for formal review was x Original research on telemedicine examining cost effectiveness of healthcare delivery.

Our exclusion criteria were x Papers reporting cost benefits of telemedicine used mainly for educational or administrative purposes x Papers reporting hypothetical cost analyses or modelling exercises without any associated formal clinical trial x Papers reporting economic analyses without any means of substantiating claimed resource use.

Two groups of reviewers examined the papers. Reviewers had expertise in telecommunications (PSW), family practice (FSM), health economics (AH) and sociology (CRM); all had experience with telemedicine services and research. The almost complete absence of randomised trials in the dataset meant that it was not possible to use traditional meas- ures for assessing trial quality. However, two reviewers (FSM and AH) independently rated reports using an established checklist of criteria for assessing the quality of economic evaluations in health care (see box).6

Results Our literature search identified 612 articles, which we categorised as without cost data (n=557) or with cost data (n=55). We then thematically categorised the reports without cost data to determine the main premise for each article, and table 1 shows the primary categories.

Quantitative analysis of reports with cost data The 55 articles with data on costs and benefits reflected the diversity of the clinical and institutional contexts in which telemedicine is undertaken.w1–w55 Most of the reports (33, 60%) were from the United States; seven (13%) were from Norway, and the rest were from Australia, Canada, the rest of Europe, and Japan. Table 2 shows the specific variables used in the 55 studies, and table 3 lists the conclusions that were drawn, almost universally positive.

Qualitative analysis of reports with cost data Of the 55 studies, we subjected only 24 to full review with our checklist.w4–w6 w8 w13–w16 w19 w24 w26 w31 w33–w35 w39 w42 w48 w50–w55 (See extra table on bmj.com.) We excluded the other 31 studies because of duplication in publication

Criteria used for assessing cost effectiveness studies6

• Presence of a clear hypothesis • Presence or absence of a clear statement regarding the perspective from which cost effectiveness assessed • Methods • Presence or absence of a comparator and whether appropriate • Is the quality of medical evidence adequate? • Appropriate costs considered • Appropriate benefits considered • Assessment of whether a marginal analysis has been undertaken • Has a sensitivity analyses been undertaken? • Is the analysis appropriate to the local environment?

Table 1 Primary categories of 557 articles examining cost effectiveness of telemedicine for healthcare delivery that lacked cost data

Category No (%) of articles

Project overview, or telemedicine history 194 (34.8)

Technology 95 (17.1)

Positive economic outcomes 80 (14.4)

Legal, regulatory, or policy 52 (9.3)

Miscellaneous 49 (8.8)

Editorial or opinion 27 (4.8)

Positive medical outcomes 20 (3.6)

Negative economic outcomes 13 (2.3)

Review analysis 12 (2.2)

Continuing medical education or continuing education 11 (2.0)

Negative medical outcomes 4 (0.7)

Table 2 Variables used in the cost analysis of the 55 articles examining cost effectiveness of telemedicine for healthcare delivery that had cost data

Variable No (%) of studies

Line charges 36 (65)

Equipment cost 33 (60)

Other project specific cost 25 (45)

Consultation fees or wages 25 (45)

Support staff fees or wages 23 (42)

Patient non-emergent transportation expenses 21 (38)

Standard ancillary cost (such as labs, ambulance) 19 (35)

Consultation transportation cost 13 (24)

Standard hospital cost (inpatient hospital costs) 12 (22)

Table 3 Conclusions drawn in the 55 articles examining cost effectiveness of telemedicine for healthcare delivery that had cost data

Conclusion No (%) of studies

Telemedicine saves money 20 (36)

Telemedicine saves time and money 11 (20)

Telemedicine cost effective only if a certain threshold is achieved 9 (16)

More work needed to accurately determine cost effectiveness 7 (13)

Other (such as enthusiasm, staff involvement) 4 (7)

Telemedicine does not save money 4 (7)

Information in practice

1435BMJ VOLUME 324 15 JUNE 2002 bmj.com

of final results or their being limited to hypothetical or modelling exercises without formal trial data, being published only in abstract form, or providing insufficient information to assess the quality and reliability of the underlying study.

Although most of the 24 reports showed clear aims, only three constructed a formal hypothesis. Twenty were restricted to simple cost comparisons, rather than exploring wider implications for patient care. Only 11 studies specified the perspectives from which economic analysis was undertaken and so delin- eated the boundaries of analysis in a way that made their proper interpretation possible. Only four studies used a societal perspective, whereas the other 20 examined costs and benefits only from the narrow per- spective of the healthcare provider (such as the hospi- tal or prison). This partial approach impedes generalisability and neglects the comprehensive analy- sis of overall long term implications of a system— especially the analysis of opportunity costs, where health resources are valued in terms of the best alternative use for such resources.

Failure to make clear the perspective and bounda- ries of analysis was paralleled by use of inappropriate economic analytical techniques. Five of the studies used cost analysis, 15 used cost minimisation analysis, and four used cost effectiveness analysis. None of the studies used cost utility analysis, the conventional means of establishing the “value for money” that an intervention represents. Failure to calculate costs and benefits using an appropriate technique did not prevent considerable attention being paid to the potential economies of scale that would come about once telemedicine systems were more widely used. However, only seven studies attempted to explore the level of use that would be needed for telemedicine services to compare favourably with traditionally organised health care, and none addressed this question in sufficient detail to answer it adequately. Comparisons of costs with traditional forms of health care were absent in two studies and hypothetical in nine. (See extra table on bmj.com for details of the comparators and methods used in each study.)

Most of the studies entirely equated benefits with cost savings, with no analysis of changes in benefit to patients. Seventeen avoided the problem of benefits measurement by assuming equivalence with conven- tional medical practice on the basis of little or no effi- cacy data. This fundamental assumption of equivalence with conventional medical practice remains unproved.7

The short duration of projects (only 20% of interventions lasted longer than 12 months) intro- duced additional problems in interpreting the results. In any service evaluation, early point estimates of costs and benefits may not reflect costs and benefits that arise once the service has become established and is operating at a steady state. Sensitivity analysis is there- fore essential to incorporate anticipated future changes in costs and benefits arising either through technological development, enhanced scale, or simply as a consequence of increasing familiarity with telemedicine. However, 15 of the studies simply relied on single point estimates obtained in a highly variable and dynamic environment.

Discussion Our database search of English language articles iden- tified only 55 that provided actual cost data on telemedicine interventions, and of these only 24 stood up to a full review using an established instrument for assessing the quality of economic evaluations. Most of the studies analysed were small scale, short term, prag- matic evaluations that added little to our knowledge of the costs and benefits that would be expected to result from the introduction of telemedicine services into routine clinical practice.

With such services it is difficult to generalise the results of individual cost effectiveness studies. In large part, comparative cost effectiveness of telemedicine systems depends on the unique local aspects of the individual service being evaluated. A telemedicine service that is cost effective in the remote highlands of Scotland is unlikely to generate the same cost effective- ness in the middle of Manchester. The difficulty of gen- erating generalisable messages from evaluations undertaken in specific contexts emphasises the importance of assessing the local applicability of individual examples of the use of telemedicine. It is important to recognise that a service may be highly clinically and cost effective in one context but highly ineffective when transferred to another context in which accessibility and quality of local services are far higher.

Use of an established instrument to assess quality in economic evaluations serves to highlight the serious deficiencies in this body of peer reviewed and published literature. There was no uniformity of analysis, and the studies we reviewed were marked by poor design and inadequate technical quality. None of the studies used cost utility analysis to compare simul- taneously variations in cost and outcomes. Cost utility analyses use outcome measures that are comparable

What is already known on this topic

The use of telemedicine has garnered much attention in the past decade

Hundreds of articles have been published claiming that telemedicine is cost effective

However, missing from the literature is a synthesis or meta-analysis of these publications

What this study adds

A comprehensive literature search of cost related articles on telemedicine identified more than 600 articles, but only 9% contained any cost benefit data

Only 4% of these articles met quality criteria justifying inclusion in a formalised quality review, and most of these were small scale, short term, pragmatic evaluations with few generalisable conclusions

There is little published evidence to confirm whether or not telemedicine is a cost effective alternative to standard healthcare delivery

Information in practice

1436 BMJ VOLUME 324 15 JUNE 2002 bmj.com

and generalisable across different treatments as a means of comparing their “value for money.” The lack of such studies means it is impossible to assess the extent to which telemedicine represents a sensible priority for healthcare investment. Until research in this area is conducted in accordance with generally accepted standards for economic analysis of health services, claims made on the basis of this body of literature should be regarded with considerable caution.

Conclusion Reviews leading to negative conclusions are rarely welcomed. In the case of telemedicine, although claims about the utility and efficacy of new telecommunications systems in practice have been widely made,4 these are not founded on strong evidence. Given the paucity of methodologically sound studies producing robust and generalisable conclusions, there is presently no persuasive evidence about whether telemedicine represents a cost effective means of delivering health care.

The conventional response to such news is to emphasise the need for better designed and conducted trials, and these are certainly necessary. However, focusing on deficiencies in the design and conduct of research should not divert our attention from the other deficiencies revealed by this review. Peer review also seems to have failed to address adequately the range of basic errors in design and

analysis that littered the studies we reviewed. Interested clinicians, policy makers, and healthcare providers should not assume that peer reviewed publication is necessarily an adequate guarantee of quality for economic evaluations of telemedicine.

Contributors: PSW is guarantor for the study and contributed to its conception and design, data analysis and interpretation, drafting the article, and final approval of the submitted paper. FSM contributed to the study conception and design, data analysis and interpretation, drafting the article, and final approval of the submitted paper. AH contributed to data analy- sis and interpretation and to drafting the article. CRM contrib- uted to the study conception and design, data interpretation, and drafting the article. TLW contributed to data analysis and interpretation. SH contributed to the data analysis.

Funding: None. Competing interests: None declared.

1 Wootton R. Telemedicine in the National Health Service. J R Soc Med 1998;91:289-92.

2 NHS Executive. Information for health: an information strategy for the modern NHS 1998-2001. London: NHS Executive, 1998.

3 Peredinia DA, Allen AA. Telemedicine technology and clinical applications. JAMA 1995;273:483-8.

4 May C, Mort M, Mair FS, Ellis NT, Gask L. Evaluation of new technologies in health care: what’s the context? Health Inform J 2000;6:64-8.

5 NHS Centre for Reviews and Dissemination. Report number 4. York: Uni- versity of York, 1996.

6 Haycox A, Walley T. Pharmacoeconomics: evaluating the evaluators. Br J Clin Pharmacol 1997;43:451-6.

7 Currell R, Urquhart C, Wainwright P, Lewis R. Telemedicine versus face to face patient care: effects on professional practice and health care out- comes. Cochrane Database Syst Rev 2000;(2):CD002098.

A memorable patient Long ago, at another hospital

She was a vicar’s wife in her 50s who had developed a sudden agonising headache. Her general practitioner diagnosed a subarachnoid haemorrhage, which was confirmed by computed tomography. Angiography revealed a right posterior communicating artery aneurysm. She remained well, and because of the risk of re-bleeding we decided to operate as soon as possible. My usual consultant anaesthetist was away on holiday, and a relatively junior (but experienced) registrar standing in. (The level of cover provided when seniors were away had long been a source of contention.)

From the outset conditions were difficult: she was tachycardic; the skull, bone edges, and extradural space bled profusely; the brain bulged. I complained that conditions were suboptimal. Response: a grunt. At last, the bleeding came under control, and the dura slackened sufficiently for us to continue. I feared she had re-bled from her aneurysm. Eventually I was able to expose the aneurysm—it looked angry and red, but had not bled during the procedure. My own registrar commented, somewhat unhelpfully, that it looked about to burst. Fortunately, the dissection of the neck proved easy, and I clipped it uneventfully.

Then, a most remarkable change occurred. The brain, hitherto bulging underneath the frontal lobe retractor, became slack, and the tachycardia disappeared. I mentioned this to the anaesthetist, who muttered something about changing the ventilator. Closure was routine.

After 30 minutes in recovery, the patient had recovered her poise and sense of humour. She said, “Mr Choksey, you seem awfully concerned about me.” I explained that I had had a struggle. She said, “I know.” I asked her how. She said, “I heard all of it—I was awake throughout the operation.” She then went on to describe, in graphic detail, the conversations I had had with the anaesthetist and my registrar’s remark that the aneurysm looked

about to burst (she prayed it wouldn’t). Her account of the operation was lucid, and it was obvious that she had been awake until just after the aneurysm was clipped. She remembered that she could just wriggle her right big toe and hoped that someone would notice. Nobody did.

I contacted her husband, the vicar, and explained what had happened. Although clearly furious, his response was humane. He interviewed both the chief executive and the chairman of the Division of Anaesthesia. The main reason for his wife’s awareness during anaesthesia had been that the anaesthetic monitoring equipment was inadequate and that she had been given 100% oxygen, with no anaesthetic gas. Her awareness had been manifest by the difficult operating conditions. She only went off to sleep when the ventilator was changed, accompanied by a sudden improvement in the operating conditions.

The vicar insisted that the hospital install all the anaesthetic equipment required in all the operating theatres to ensure that this could never happen again

Munchi Choksey consultant neurosurgeon, Walsgrave Hospital, Coventry

We welcome articles of up to 600 words on topics such as A memorable patient, A paper that changed my practice, My most unfortunate mistake, or any other piece conveying instruction, pathos, or humour. If possible the article should be supplied on a disk. Permission is needed from the patient or a relative if an identifiable patient is referred to. We also welcome contributions for “Endpieces,” consisting of quotations of up to 80 words (but most are considerably shorter) from any source, ancient or modern, which have appealed to the reader.

Information in practice

1437BMJ VOLUME 324 15 JUNE 2002 bmj.com