informatics

i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 9 ( 2 0 1 0 ) 736–771

j o u r n a l h o m e p a g e : w w w . i n t l . e l s e v i e r h e a l t h . c o m / j o u r n a l s / i j m i

Effectiveness of telemedicine: A systematic review of reviews

Anne G. Ekeland a,∗, Alison Bowes b, Signe Flottorp c,d

a Norwegian Centre for Integrated Care and Telemedicine, University Hospital of North Norway, P.O. Box 6060, N-9038 Tromsø, Norway b Department of Applied Social Science, University of Stirling, Scotland, UK c Norwegian Knowledge Centre for the Health Services, Oslo, Norway d Department of Public Health and Primary Health Care, University of Bergen, Norway

a r t i c l e i n f o

Article history:

Received 23 April 2010

Received in revised form

11 July 2010

Accepted 29 August 2010

Keywords:

Telemedicine

Telecare

Systematic review

Effectiveness

Outcome

a b s t r a c t

Objectives: To conduct a review of reviews on the impacts and costs of telemedicine services.

Methods: A review of systematic reviews of telemedicine interventions was conducted. Inter-

ventions included all e-health interventions, information and communication technologies

for communication in health care, Internet based interventions for diagnosis and treat-

ments, and social care if important part of health care and in collaboration with health care

for patients with chronic conditions were considered relevant. Each potentially relevant sys-

tematic review was assessed in full text by one member of an external expert team, using

a revised check list from EPOC (Cochrane Effective Practice and Organisation of Care Group)

to assess quality. Qualitative analysis of the included reviews was informed by principles of

realist review.

Results: In total 1593 titles/abstracts were identified. Following quality assessment, the

review included 80 heterogeneous systematic reviews. Twenty-one reviews concluded that

telemedicine is effective, 18 found that evidence is promising but incomplete and others

that evidence is limited and inconsistent. Emerging themes are the particularly problem-

atic nature of economic analyses of telemedicine, the benefits of telemedicine for patients,

and telemedicine as complex and ongoing collaborative achievements in unpredictable

processes.

Conclusions: The emergence of new topic areas in this dynamic field is notable and review-

ers are starting to explore new questions beyond those of clinical and cost-effectiveness.

Reviewers point to a continuing need for larger studies of telemedicine as controlled inter-

ventions, and more focus on patients’ perspectives, economic analyses and on telemedicine

innovations as complex processes and ongoing collaborative achievements. Formative

assessments are emerging as an area of interest.

© 2010 Elsevier Ireland Ltd. All rights reserved.

Contents

1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 737 2. Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 738

∗ Corresponding author. Tel.: +47 952 66791. E-mail address: [email protected] (A.G. Ekeland). URL: http://www.telemed.no (A.G. Ekeland).

1386-5056/$ – see front matter © 2010 Elsevier Ireland Ltd. All rights reserved. doi:10.1016/j.ijmedinf.2010.08.006

Appendix 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 742 Appendix B. Supplementary data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 769

. . . . .

1

P i t r t t a p t c l s c u t b t s h c i e

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. Introduction

revious reviews of telemedicine have concluded that rrefutable evidence regarding the positive impact of elemedicine on clinical outcomes still eludes us. One eview [1] of more than 150 articles concluded that poten- ial effectiveness could only be attributed to teleradiology, elepsychiatry, transmission of echocardiographic images nd consultations between primary and secondary health roviders. Another systematic review [2] that assessed more han 1300 papers making claims about telemedicine out- omes found only 46 publications that actually studied at east some clinical outcomes. A review that analyzed the uitability of telemedicine as an alternative to face-to-face are [3] concluded that establishing systems for patient care sing telecommunications technologies is feasible; however, he studies provided inconclusive results regarding clinical enefits and outcomes. A report on peer-reviewed litera- ure for telemedicine services that substituted face-to-face ervices with ICT-based services at home and in offices or

ospitals [4] identified 97 articles that met the inclusion riteria for analysis. The authors concluded that telemedicine s being used even if the use is not supported by high quality vidence. Reviews on cost outcomes have fared similarly. A

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 769

study focused on cost-effectiveness interventions concluded that there is no good evidence that telemedicine is or is not a cost-effective means for delivering healthcare [5].

The quality of studies is a recurrent concern in these reviews [1,2,4–6]. There is also a debate about appropriate research methodologies. For example, economic analysis of telemedicine has not yet met accepted standards [5]; there is a relative lack of exploration of the socio-economic impact of telemedicine [7]; evidence on factors promoting uptake of telemedicine is lacking [8]; there is relatively undeveloped use, at the time, of qualitative methods [9]; many studies have not been well-designed [4,10]; and, considering perceived difficul- ties of building a robust evidence base for recent innovations, researchers have argued that simulation modelling needs fur- ther development [11].

The lack of consensus raises questions about the quality of research evidence in terms not only of the data collected and analysed, but also in terms of the approaches to evalua- tion, that is, the underlying methodologies used, which may not be capable of addressing the questions to which different stakeholders seek answers. Others have noted that evaluation

i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 9 ( 2 0 1 0 ) 736–771 737

3. Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 738 3.1. Inclusion criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 738

3.1.1. Population/participants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 738 3.1.2. Interventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 738 3.1.3. Comparisons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 738 3.1.4. Outcomes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 738 3.1.5. Languages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 738

3.2. Exclusion criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 738 3.2.1. Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 738 3.2.2. Participants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 738 3.2.3. Interventions considered not relevant for the review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 738 3.2.4. Outcomes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 738

3.3. Information sources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 739 3.4. Search. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 739 3.5. Study selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 739 3.6. Data collection process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 739 3.7. Data items . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 739 3.8. Quality of systematic reviews and risk of bias in individual studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 739 3.9. Summary measures and synthesis of results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 739

4. Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 739 5. Telemedicine is effective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 739 6. Telemedicine is promising . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 740 7. Evidence is limited and inconsistent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 740 8. Economic analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 741 9. Is telemedicine good for patients? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 741 10. Asking new questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 741 11. Reflections on the methodology of our study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 742 12. Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 742

Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 742

traditions do not sufficiently collaborate to cross borders and that a common language for evaluation is missing [12].

This paper reports on research funded under EU SMART 2008/0064, which sought to review the evidence on the

i c a l

738 i n t e r n a t i o n a l j o u r n a l o f m e d

effectiveness of telemedicine with particular reference to both outcomes and methodologies for evaluation. This paper focuses mainly on the evidence about effectiveness, and assesses the range of conclusions drawn by reviewers about the effectiveness of telemedicine and the gaps in the evidence base. A companion paper focuses on the methodological issues and recommendations [13].

2. Objectives

The objective of the work was to conduct a review of reviews on the impacts and costs of telemedicine services and con- sider qualitative and quantitative results, with the purpose of synthesizing evidence to date on the effectiveness of telemedicine. The key questions addressed were firstly, how are telemedicine services defined and described in terms of participants, interventions, comparisons and outcome mea- sures; secondly, what are the reported effects of telemedicine: thirdly which methodologies were used to produce knowl- edge about telemedicine in studies included; fourthly, what are the strengths and weaknesses of these methodologies, including HTA methodologies; and finally what are the knowl- edge gaps and what methodologies can be recommended for future research? The present paper addresses the first two of these questions, and identifies assessments of the evidence base provided within the reviews and knowledge gaps in terms of outcomes.

3. Methods

An initial search identified systematic reviews of telemedicine published from 1998. A systematic review was defined as an overview with an explicit question and a method section with a clear description of the search strategy and the methods used to produce the systematic review. The review should also report and analyse empirical data. In addition, reviews which described or summarised methods used in assessing telemedicine were included. Because of the large number of reviews retrieved, a decision was taken to include only reviews published from 2005 and onwards in the final review.

3.1. Inclusion criteria

3.1.1. Population/participants Systematic reviews on patients and consumers, health pro- fessionals and family caregivers, regardless of diagnoses or conditions, were included in the searches for systematic reviews.

3.1.2. Interventions All e-health interventions, information and communication technologies (ICT) for communication in health care, Internet

based interventions for diagnosis and treatments, and social care if an important part of health care and in collaboration with health care for patients with chronic conditions were considered relevant.

i n f o r m a t i c s 7 9 ( 2 0 1 0 ) 736–771

3.1.3. Comparisons Reviews of studies comparing telemedicine to standard care or to another type of care, as well as reviews of studies comparing different e-health solutions were included.

3.1.4. Outcomes Only reviews reporting relevant outcomes were included, specified as health related outcomes (morbidity, mortality, quality of life, patient’ satisfaction), process outcomes (qual- ity of care, professional practice, adherence to recommended practice, professional satisfaction) and costs or resource use. Systematic reviews reporting emerging issues, such as an unexpected finding or important new insights were also included.

3.1.5. Languages No articles were excluded based on language, although the main focus of the project was telemedicine in Europe.

3.2. Exclusion criteria

3.2.1. Design Reviews considered not systematic, including commentaries and editorials, were excluded. Systematic reviews with major limitations (low quality reviews) according to a revised check- list for systematic reviews from EPOC (Cochrane Effective Practice and Organisation of Care Group) were excluded.

If the same authors had produced several publications of the same review, the most updated and/or the full report of the review was selected, and other versions excluded. Disser- tations, symposium proceedings, and irretrievable documents were excluded.

3.2.2. Participants Studies with participants considered not relevant for the review, for instance studies on use of ICT on people outside health care were excluded. Animal studies were excluded.

3.2.3. Interventions considered not relevant for the review Other exclusions were studies on interventions considered not relevant for the review, such as studies on Internet and other ICT media used for information seeking; quality of informa- tion on the Internet; Internet based education of students and health professionals, including use of games; medical tech- nology in clinical practice in general, i.e. medical and surgical examinations and treatments based on computer technolo- gies, except when used as remote diagnosis and treatment (telehealth); ordinary use of electronic patient records; use of telephone (including cell phones) only; e-health as only a very limited part of an intervention; use of Internet for surveys and research; online prescriptions; mass media interventions and veterinary medicine.

3.2.4. Outcomes Articles without relevant outcomes, i.e. not on the list of out- comes specified above under inclusion criteria, were excluded.

a l i n

3

L L e C o m S t m E

u

3

T i

3

B i t w A i

3

D t a r P t a i r a A r

3

D i g i i c a t

3 i

T s

i n t e r n a t i o n a l j o u r n a l o f m e d i c

.3. Information sources

iterature searches of the following databases: ACM Digital ibrary (ACM – The Association for Computing Machin- ry), British Nursing Index, Cochrane library (including ochrane database of systematic reviews (CDSR), Database f reviews of effects (DARE), Health Technology Assess- ent Database (HTA), CSA, Ovid Medline, Embase, Health

ervices/Technology Assessment Text (HSTAT), Interna- ional Network of Agencies for Health Technology Assess-

ent (INAHTA), PsycInfo, Pubmed, Telemedicine Information xchange (TIE), Web of Science.

The main search was performed in February 2009, and an pdated search was performed in July 2009.

.4. Search

he search strategies are available on the website: (to be nserted).

.5. Study selection

ased on the criteria for inclusion and exclusion, AGE and SF ndependently screened the lists of titles/abstracts identified hrough searches for systematic reviews. Any discrepancies ere solved by discussion with the third member of the team, B. The potentially relevant systematic reviews were retrieved

n full text.

.6. Data collection process

ata collection was carried out online using a data extrac- ion form. Each potentially relevant systematic review was ssessed in full text by one member of an expert panel of eviewers. A revised check list from EPOC (Cochrane Effective ractice and Organisation of Care Group) was used to assess he quality of the systematic reviews. The quality domains ssessed according to this checklist were methods used to dentify, include and critically appraise the studies in the eview, methods used to analyse the findings and an overall ssessment of the quality of the review. The review team (AGE, B and SF) subsequently checked review reports for agreement

egarding the inclusion and exclusion criteria.

.7. Data items

ata on type of participants, interventions and outcomes ncluded in the reviews were collected. Other data items were: eographical coverage of review, time frame of included stud- es, range of data collection methods used in studies included n the reviews, disciplines/areas covered and methodologi- al traditions included in the review. The reviewers were also sked to indicate emerging issues identified by the authors of he reviews.

.8. Quality of systematic reviews and risk of bias in

ndividual studies

he members of the expert team assessed the quality of the ystematic reviews, including questions regarding the degree

f o r m a t i c s 7 9 ( 2 0 1 0 ) 736–771 739

to which the systematic reviewers had assessed risk of bias in individual studies.

Systematic reviews with major limitations were excluded. We assessed the methodological quality of studies in the field of telemedicine based on the review authors’ assess- ments of risk of bias in the primary studies they had included.

3.9. Summary measures and synthesis of results

The authors analysed the data collected by the members of the expert team. Due to the expected heterogeneity of stud- ies, regarding participants, interventions, outcomes and study designs, a quantitative summary measure of the results was not planned. We did a qualitative and narrative summary of the results of the systematic reviews. The results of the literature review were presented and discussed in two work- shops intending to validate results. In the first workshop different user groups took part and in the second workshop methodology experts participated. The analysis was inspired by principles of realist review [14], considered appropriate for complex interventions.

4. Results

We identified 1593 records through the searches and excluded 1419 following screening. We retrieved 174 potentially rel- evant articles in full text. We excluded 94 of these based on the pre-specified inclusion and exclusion criteria. The qualitative synthesis below relate to 73 of the 80 included articles.

The results of the 80 systematic reviews included are sum- marised in seven tables in Appendix 1. Tables one through six list populations, interventions, outcomes, results and con- clusions for the reviews cited in this paper, according to the headlines presented in the discussion below. Table 7 list the seven included reviews not cited in this paper.

5. Telemedicine is effective

Twenty reviews (Table 1) concluded that telemedicine works and has positive effects. These include therapeutic effects, increased efficiencies in the health services, and technical usability.

Types of interventions that were found to be therapeuti- cally effective include online psychological interventions [15]; programmes for chronic heart failure that include remote monitoring [16]; home telemonitoring of respiratory con- ditions [17]; web and computer-based smoking cessation programmes [18]; telehealth approaches to secondary preven- tion of coronary heart disease [19]; telepsychiatry [20]; virtual reality exposure therapy (VRET) for anxiety disorders [21]; robot-aided therapy of the proximal upper limb [22]; inter- net and computer-based cognitive behavioural therapy for the treatment of anxiety [23,24]; home telehealth for diabetes,

heart disease and chronic obstructive pulmonary disease [25]; and internet based physical activity interventions [26]. A review comparing telepsychiatry and face-to-face work [27] found no differences between the two, and suggested that

i c a l

740 i n t e r n a t i o n a l j o u r n a l o f m e d

telepsychiatry will increase in use, particularly where it is more practical.

Interventions that are effective in reducing health service use include vital signs monitoring at home with telephone follow-up by nurses [28]; computerised asthma patient educa- tion programs [29]; and home monitoring of diabetes patients [30].

Technical effectiveness and reliability are reported in respect of remote interpretation of patient data [31]; smart home technologies [32]; and home monitoring of heart failure patients [33].

One review concluded that home based ICT interventions in general give comprehensive positive outcomes for chronic disease management, despite only identifying a small number of heterogeneous studies [34].

6. Telemedicine is promising

Nineteen reviews (Table 2) were less confident about the effec- tiveness of telemedicine, suggesting that it is promising, or has potential, but that more research is required before it is pos- sible to draw firm conclusions. In some cases, in which the same conditions and interventions are discussed, these more tentative conclusions must temper those of authors who find conclusive evidence.

One review [35] for example found internet-delivered CBT to be a ‘promising’ and ‘complementary’ development, but did not provide the endorsements that others [23,24] did for CBT for the more specific conditions of anxiety and depression. Similarly psychotherapy using remote communication tech- nologies was seen as promising [36], but still requiring more evidence.

Areas in which review authors agreed that telemedicine shows therapeutic promise, but still requires further research, include virtual reality in stroke rehabilitation [37,38]; improv- ing symptoms and behaviour associated with and knowledge about specific mental disorders and related conditions [39]; diabetes [40,83]; weight loss intervention and possibly weight loss maintenance [41]; and alcohol abuse [88].

Other authors found promise in terms of health service utilisation. One review [42] for example suggested that asyn- chronous telehealth developments could result in shorter waiting times, fewer unnecessary referrals, high levels of patient and provider satisfaction, and equivalent (or better) diagnostic accuracy. Another [43] found that home telehealth has a positive impact on the use of many health services as well as glycaemic control of patients with diabetes.

Positive patient experiences were highlighted as promising in relation to home telemonitoring for respiratory conditions [17]. There is potential for using Internet/web-based services for cancer patients in rural areas [44], and telemonitoring can empower patients with chronic conditions [45].

Promising impacts on service delivery were identified [46,47] in use of electronic decision support systems and telemedicine consultations promise to support improved

delivery of tPA in patients with stroke (a treatment which requires to be administered within 3 h) [48]. Computer reminders to professionals at the point of care show ‘small to modest improvements’ in professional behaviour, but studies

i n f o r m a t i c s 7 9 ( 2 0 1 0 ) 736–771

are heterogeneous and interventions complex, making these difficult to understand [49].

7. Evidence is limited and inconsistent

Twenty-two reviews (Table 3) however concluded that the evi- dence for the effectiveness of telemedicine is still limited and inconsistent, across a wide range of fields.

In terms of therapeutic effectiveness, there is some lim- ited evidence regarding telemonitoring for heart failure [50]; despite reviewers suggesting that electronic transfer of self- monitored results has been found to be feasible and acceptable in diabetes care, they find only weak evidence for improve- ments in HbA1c or other aspects of diabetes management [51]; others found only weak evidence of benefit relating to infor- matics applications in asthma care [52]; and no evidence of improvement in clinical outcomes following teleconsultation and video-conferences in diabetes care [53].

Frequently, these reviewers call for further research, notably in the form of RCTs. Examples include calls relating to web-based alcohol cessation interventions [54]; and vir- tual reality in stroke therapy, despite this being found [37] to be ‘potentially exciting and safe’. More work on telemonitor- ing in heart failure is called for [55]; on e-therapy for mental health problems [56]; on smart home technologies [57]; and on technological support for carers of people with demen- tia [58]. Others [28] underlined that lack of evidence does not imply lack of effectiveness, and that in many cases interven- tions are simply ‘unproven’. Caution is also urged by reviewers [59] who identified small numbers of heterogeneous studies in relation to chronic disease management. One review [60] found it impossible to draw any significant conclusions about the impact of interventions to promote ICT use by health care personnel.

Several reviewers found that research has been somewhat narrowly focused and suggested further research which takes a broader perspective or a different one. They suggested that telemedicine researchers have not yet asked all the impor- tant questions, or conducted research in appropriate ways. For example, in the cases of dermatology, wound care and ophthalmology, it was argued that evaluation has explored ICT-based asynchronous services for efficacy, but outcomes or access issues have not been considered [61]. In a simi- lar vein, although most of the studies of smart homes found technical feasibility, there remain certain topics that require further research, notably, ‘technical, ethical, legal, clinical, economical and organisational implications and challenges’ [32]. Others [44], whilst seeing significant potential for teleon- cology, especially in rural areas, suggested that local studies may be needed to confirm this. A further contribution to the debates about CBT (see above), found that whilst it appears to be effective for panic disorders, social phobia and depression, its effects on obsessive–compulsive disorder and anxiety and depression combined remain insufficiently clear [62]. Causal pathways in HbA1c decline in diabetes care remain unclear,

and this conclusion can be linked with the variations in pro- gramme designs [63]. Whilst smoking cessation programmes appear to be effective across a range of studies, nevertheless the mechanisms of action are not well understood [64].

a l i n

n t i S r [ t s [ n

8

A k (

b f e c c t m a i t s l d g p a

e o t o t m i s w

t r b l o n

9

A t t (

i n t e r n a t i o n a l j o u r n a l o f m e d i c

Telemedicine is a dynamic field, and new studies and ew systematic reviews are rapidly being published. As elemedicine extends into new clinical areas, it is unsurpris- ng that reviewers give renewed accounts of limited evidence. ome examples of new areas from our review include little esearch on health promotion provided through the Internet 65]; a Cochrane review that found no studies of smart homes hat met their inclusion criteria [57]; a review of studies on piritual care that found little systematic research in this area 66]; and a review concluding that formative evaluation is eeded for remote monitoring in hypertension [90].

. Economic analysis

n important emerging issue from our review is the lack of nowledge and understanding of the costs of telemedicine

Table 4). Several reviewers suggested that telemedicine seemed to

e cost-effective, but few draw firm conclusions. One review ound that 91% of the studies showed telehomecare to be cost- ffective, in that it reduced use of hospitals, improved patient ompliance, satisfaction and quality of life [67]. This was the learest conclusion, with others being much more cautious: elemedicine was found to be cost-effective for chronic disease

anagement, but the authors cautioned that studies were few nd heterogeneous [34]. A comparison of the costs of telemon- toring and usual care for heart failure patients found that elemonitoring could reduce travel time and hospital admis- ions, whilst noting that benefits are likely to be realised in the ong term [68]. Others found home telehealth for chronic con- itions to be cost saving, though underlining that studies were enerally of low quality [25]. One review found remote inter- retation in medical encounters to be more expensive than its lternatives [31].

Other reviewers did not find good evidence about cost- ffectiveness; the cost-effectiveness of home telecare for lder people and people with chronic conditions is uncer- ain [28]; there is a lack of consistent results regarding costs f synchronous telehealth in primary care [69]; there is lit- le evidence for the economic viability of home respiratory

onitoring [17]; the cost-effectiveness of IT in diabetes care s undetermined [40]; one review was able to identify only one tudy of the costs of CBT, with significant weaknesses [70], ith another finding little evidence in the same area [62].

A particular limitation identified in terms of costs concerns he wider social and organisational costs of telemedicine. One eview found that a societal perspective on costs has not yet een developed for home telehealth [71] and another high-

ighted the need to consider not only costs to health services f interventions, but also costs to service users and their social etworks [72].

. Is telemedicine good for patients?

second emerging issue concerns patient satisfaction with elemedicine, and indications that telemedicine may alter he relationships between patients and health professionals Table 5).

f o r m a t i c s 7 9 ( 2 0 1 0 ) 736–771 741

One review found that health service users with ICT used in support, education and virtual consultation feel more confident and empowered, with better knowledge and improved health outcomes, as well as experiencing better nurse-patient relationships [73]. The reviewers call for more research on the mechanisms for these changes. Generally there is evidence of high patient satisfaction ratings for telere- habilitation, but reviewers argue that more process research, case studies and qualitative studies are needed to improve our understanding of these outcomes [74]. Interactive health communication applications (IHCAs) for people with chronic disease appear to give benefit in terms of improved sup- port, better knowledge and improved health outcomes, but the authors asked for more larger studies to be conducted [75].

Others found no consistent results regarding user expe- riences, though suggested that access can be improved [69]. Alongside development of technologies which aim to ben- efit patients and citizens as well as professionals, we need research on the impacts of technologies for these groups [76]. An example is that information websites relating to dementia are geared more to carers than to people with dementia them- selves, and that the websites do not usually offer personalised information [77].

10. Asking new questions

We have already noted the emergence of new topic areas in this dynamic and complex field. The focus on patient bene- fits however indicates a more basic development, namely that reviewers are starting to explore new questions beyond those of clinical and cost-effectiveness. Our review produced two key examples (Table 6). Firstly, a review that identified gender differences in computer-mediated communications relating to online support groups for people with cancer cautioned that studies are limited and heterogeneous [78]. Nevertheless, the authors suggested that this issue needs to be considered by those designing interventions of this kind. This implies a con- sideration that telemedicine is an ongoing intervention where users influence its development and hence that effectiveness of outcome is a complex collaborative achievement. Secondly, a review focusing on stroke thrombolysis service configura- tions, their potential impact and ways of recording data to inform which configuration could be most suitable for a partic- ular situation, highlighted the need to consider a wider range of service delivery issues [79]. Similarly, it was argued that in post-stroke patients, the consideration of caregivers’ mental health and high levels of patient satisfaction should be an integral element of studies [80].

Furthermore, some of the papers included in the review explored issues which can inform the future development of telemedicine, that is, they provide formative assessments. Examples include a review of 104 definitions of telemedicine [81] which, in identifying four broad types of definitions, suggested how stakeholder interests can alter perceptions

of priorities in telemedicine interventions, such that some may focus on delivering healthcare over a distance and others on the potential of technology per se; and work argu- ing that clinical and technical guidelines can inform the

i c a l i n f o r m a t i c s 7 9 ( 2 0 1 0 ) 736–771

Summary points “What was already known on this topic”

• Evidence regarding the effectiveness of telemedicine is patchy and incomplete.

• The quality of much of the research conducted is poor.

“What this study added to our knowledge”

• The evidence base is accumulating robust knowledge about the effectiveness of telemedicine.

• As the field is rapidly evolving however, new knowl- edge is constantly needed.

• Continuing areas of weakness but also of great interest include economic analyses, understandings of patient perspectives, of effectiveness and outcomes as com- plex and ongoing collaborative achievements, and

742 i n t e r n a t i o n a l j o u r n a l o f m e d

future development of telemedicine and facilitate evaluation [20,82].

11. Reflections on the methodology of our study

Our study is a review of systematic reviews. There are some inherent weaknesses in this approach. In general we have to rely on the information in the included reviews. The quality of the reviews may vary; the reviews may have done a poor job in specifying their inclusion and exclusion criteria, the searches may not be comprehensive, the review authors may not have assessed or extracted data from the primary studies adequately, nor analysed and synthesised the findings across the studies properly. But even using high quality reviews, we necessarily lose information and details that we can only find if we go back to the primary studies.

Although we did a thorough job in developing the search strategy and identified a vast amount of reviews on the effects of telemedicine, we might have missed relevant systematic reviews.

Some of the included reviews are probably outdated. Stud- ies that are published after the search date in the reviews are not included. Ideally we could have supplemented the review with more recent primary studies not included in the reviews, but we did not have the resources to do this.

We did not check whether reviews included the same ref- erences. Several reviews have studied similar or overlapping topics, and have at least partially included the same studies. It may therefore be that evidence is counted twice, or that different interpretations of effectiveness are given by review authors. We have not analysed the degree to which there are discrepancies in the analyses of similar studies, nor the rea- sons for different interpretations of the same findings, for instance did we not analyse the heterogeneity of the results among the reviews based on the quality of the reviews.

The data collection and assessment of each included review was accomplished by one external expert, while two is considered to be optimal in order to reduce risk of bias. We did not train the data extractors, and we did not pilot the data extraction form. The experts were not completely consistent in their judgments. This limitation was partly due to the resources and organisation of the project, in that two workshops were held, intending to validate results. In addi- tion, the review team made a quality check of the reviews by comparing the reported data with information in the full text papers. Any unclear themes were discussed in the team to reach consensus.

We have limited information regarding effect sizes and the strength of evidence for the outcomes that we have studied.

We have however demonstrated that it is possible to make such a large overview in quite a short time, involving both methodology and content experts. We have used systematic methods in the literature searches and the assessment of the reviews, and we have excluded reviews of low methodological

quality.

In combining rigorous and systematic methods with a pragmatic approach we have produced a relevant and rich overview of the field.

formative assessments.

12. Conclusions

Despite large number of studies and systematic reviews on the effects of telemedicine, high quality evidence to inform policy decisions on how best to use telemedicine in health care is still lacking. Large studies with rigorous designs are needed to get better evidence on the effects of telemedicine interventions on health, satisfaction with care and costs. As the field is rapidly evolving, different kinds of knowledge are also in demand, e.g. a stronger focus on economic analyses of telemedicine, on patients’ perspec- tives and on the understanding of telemedicine as complex development processes, and effectiveness and outcome as ongoing collaborative achievements. Hence formative assess- ments are also pointed out as an area of weakness and interest.

Acknowledgements

The study was funded by the EU under SMART 2008/0064 and was conducted as part of the MethoTelemed project. We acknowledge the support of our MethoTelemed colleagues, the group of external review experts, the workshop participants, the project officers at the Norwegian centre for integrated care and telemedicine, and Ingrid Harboe at the Norwegian Knowl- edge Centre for the Health Services, who did the literature searches.

Appendix 1.

In Tables 1–7, columns listing results and conclusions quote from the authors’ work. Where a review appears in more than one table, this reflects the range of evidence produced. Full

access to a searchable database of abstracts of items included in the review will be available on the MethoTelemed web- site, which also includes guidance for evaluating telemedicine. www.telemed.no/MethoTelemed.

in t

e r

n a

t io

n a

l jo

u r

n a

l o

f m

e d

ic a

l in

f o

r m

a t

ic s

7 9

( 2

0 1

0 )

7 3

6 –7

7 1

7 4 3

Table 1 – Systematic reviews reporting that telemedicine is effective.

Reference Conditions included

Geographic area

Service/ intervention

Outcome Authors’ summary of results Authors’ conclusions

Barak et al. [15] Mental health Not stated Internet based psychotherapy

Behavioural, Health, Percep- tion/satisfaction, Social

Sixty-four studies included covering 94 services. The overall mean weighted effect size was 0.53, similar to the average effect size of traditional, face-to-face therapy. Comparison between face-to-face and Internet intervention across 14 studies showed no differences in effectiveness.

Internet based intervention is as effective as face-to-face intervention.

Clark et al. [16] Cardio- vascular (CHF)

All countries Remote monitoring, telephone support

Behavioural, Cost/economic, Health

Fourteen studies (RCTs) included. Four evaluated telemonitoring, nine structured telephone support, and one both. Remote monitoring programmes reduced the rates of admission to hospital for chronic heart failure by 21% and all cause mortality by 20%. Three studies reported quality of life improvements and four, reduced cost, one found no gain in cost-effectiveness.

Programmes for chronic heart failure that include remote monitoring have a positive effect on clinical outcomes in community dwelling patients with chronic heart failure.

Jaana et al. [17] Respiratory conditions

USA, Europe, Israel, Taiwan

Remote monitoring

Behavioural, Cost/economic, Feasibility/pilot, Health, Percep- tion/satisfaction

Twenty-three studies included. Good levels of data validity and reliability were reported. However, little quantitative evidence exists about the effect of remote monitoring on patient medical condition and utilization of health services. Positive effects on patient behaviour were consistently reported. Only two studies performed a detailed cost analysis.

Home telemonitoring of respiratory conditions results in early identification of deteriorations in patient condition and symptom control. Positive patient attitude and receptiveness of this approach are promising. However, evidence on the magnitude of clinical and structural effects remains preliminary, with variations in study approaches and an absence of robust study designs and formal evaluations.

Myung et al. [18]

Smoking cessation

Worldwide Web and computer-based programmes

Behavioural Twenty-two studies included (RCTs). In a random-effects meta-analysis of all 22 trials, the intervention had a significant effect on smoking cessation. Similar findings were observed in nine trials using a Web-based intervention,(and in 13 trials using a computer-based intervention Subgroup analyses revealed similar findings for different levels of methodological rigor, stand-alone versus supplemental interventions, type of abstinence rates employed, and duration of follow-up period, but not for adolescent populations.

The meta-analysis of RCTs indicates that there is sufficient clinical evidence to support the use of Web- and computer-based smoking cessation programs for adult smokers.

7 4 4

in t

e r

n a

t io

n a

l jo

u r

n a

l o

f m

e d

ic a

l in

f o

r m

a t

ic s

7 9

( 2

0 1

0 )

7 3

6 –7

7 1

Table 1 (Continued)

Reference Conditions included

Geographic area

Service/ intervention

Outcome Authors’ summary of results Authors’ conclusions

Neubeck et al. [19]

Cardio- vascular (CHD)

USA (3 studies), Norway (1), Canada (3), Australia (3), Germany (1)

Communication using ICT, patient- professional

Behavioural, Health, psychosocial state, quality of life

Eleven studies included (RCTs). Telehealth interventions were associated with non-significant lower all-cause mortality than controls. These interventions showed a significantly lower weighted mean difference at medium long-term follow-up than controls for total cholesterol, systolic blood pressure, and fewer smokers. Significant favourable changes at follow-up were also found in high-density lipoprotein and low-density lipoprotein.

Telehealth interventions provide effective risk factor reduction and secondary prevention. Provision of telehealth models could help increase uptake of a formal secondary prevention by those who do not access cardiac rehabilitation and narrow the current evidence-practice gap.

Pineau et al. [20]

Psychiatric conditions (adult and paediatric)

Focus on Canada and USA

Telepsychiatry Cost/economic, Ethical issues, Legal, Organizational, Technology related, clinical guidelines and technical standards

‘About 60’ studies included. The authors argue that definition of clinical guidelines and technological standards aimed at standardising telepsychiatric practice will promote its large scale implementation.

The review concludes that telepsychiatry should be implemented in Québec and provides detailed clinical and technical guidelines for implementation. They add that taking into account human and organizational aspects plays a part in ensuring the success of this type of activity; that legal and ethical aspects must also be considered; and that a detailed economic analysis should be carried out prior to any large investment in telepsychiatry. Finally, implementation of psychiatry should be subjected to rigorous downstream assessment in order to improve management and performance.

Powers and Emmelkamp [21]

Anxiety (especially phobias)

Not stated Virtual reality exposure therapy

Behavioural, Percep- tion/satisfaction, Psychophysiology, perceived control over phobias

Thirteen studies included. VRET (Virtual reality exposure therapy) is highly effective in treating phobias and more so than inactive control conditions. VRET is slightly, but significantly more effective than exposure in vivo, the gold standard in the field. Advantages of VRET: can be conducted in the therapist’s office, rather than in vivo situations, the possibility of generating more gradual assignments and of creating idiosyncratic exposure. VRET is cost-effective.

Given the advantages and the efficacy of VRET supported by this meta-analysis a broader application in clinical practice seems justified.

in t

e r

n a

t io

n a

l jo

u r

n a

l o

f m

e d

ic a

l in

f o

r m

a t

ic s

7 9

( 2

0 1

0 )

7 3

6 –7

7 1

7 4 5

Prange et al. [22]

Stroke USA Rehabilitation (robots)

Health Eleven studies included. Robot-aided therapy of the proximal upper limb improves short and long-term motor control of the paretic shoulder and elbow: however, there is no consistent influence on functional abilities.

This systematic review indicates that robot-aided therapy of the proximal upper limb can improve short and long-term motor control of the paretic shoulder and elbow. Robot-aided therapy appears to improve motor control more than conventional therapy.

Reger and Gahm [23]

Mental health problems (anxiety)

Not stated Internet/computer- based treatment

Behavioural, Health Ninteen studies included (RCTs). Meta-analysis showed that ICT was superior to waitlist and placebo assignment across outcome measures The effects of ICT were equal to therapist-delivered treatment across anxiety disorders. Conclusions were limited by small sample sizes, the rare use of placebo controls, and other methodological problems. The number of available studies limited the opportunity to conduct analyses by diagnostic group.

The results of this meta-analysis provide preliminary support for the use of Internet and computer-based CBT for the treatment of anxiety.

Spek et al. [24] Mental health (depression and anxiety)

Global CBT via internet Health Twelve studies included (RCTs). Authors concluded that eCBT was effective, but noted that there was only a small number of studies and significant heterogeneity.

Despite study limitations, eCBT seemed to be effective.

Tran et al. [25] Diabetes, heart failure, COPD and other chronic diseases

Canada focused, but international publications included

Home telehealth Cost/economic, Health, Percep- tion/satisfaction

Seventy-nine studies included. Of the included studies, 26 pertained to diabetes, 35 to CHF, nine to COPD, and eight to mixed chronic diseases. The comparator “no care” was not identified in any of the included studies, so usual care was used as the comparator throughout the clinical review. Home telehealth appeared generally clinically effective and no patient adverse effects were reported. Evidence on health service utilization was more limited, but promising The economic review suggested cost-effectiveness, but the quality of studies was low.

Conclusions relate to the potential for home telehealth in Canada which is seen as positive. However, more research, such as multicentre RCTs, is warranted to accurately measure the clinical and economic impact of home telehealth for chronic disease management to support Canadian policy makers in making informed decisions.

van den Berg et al. [26]

Internet based physical activity interventions

Not mentioned other than language limitations

Physical activity Behavioural, Health Ten studies included. The analysis focused on the methodological quality of the studies, which showed variation in study populations and interventions making generalization difficult.

There is indicative evidence that internet based physical activity interventions are more effective than a waiting list strategy.

7 4 6

in t

e r

n a

t io

n a

l jo

u r

n a

l o

f m

e d

ic a

l in

f o

r m

a t

ic s

7 9

( 2

0 1

0 )

7 3

6 –7

7 1

Table 1 (Continued)

Reference Conditions included

Geographic area

Service/ intervention

Outcome Authors’ summary of results Authors’ conclusions

Hyler et al. [27] Mental health France, Australia, Canada, Japan, UK and US

Telepsychiatry Feasibility/pilot, Per- ception/satisfaction, Quality of different instruments used for consultations

Fourteen studies included. Telepsychiatry was found to be similar to In person for studies using objective assessments. Bandwidth was a moderator. Heterogeneous effect sizes for different moderators (bandwidth) High bandwidth was slightly superior for assessments requiring detailed observation of patients.

Only a handful studies have attempted to compare telepsychiatry with in-person psychiatry (IP) directly, using standardised assessment instruments to permit meaningful comparison. According to the meta-analysis, there was no difference in accuracy or satisfaction between the two modalities. Telepsychiatry is expected to replace IP in certain research and clinical situations.

Barlow et al. [28]

Elderly people, chronic diseases

Worldwide Home telecare Behavioural, Health, Organisational, Safety

Sixty-eight RCTs and 30 observational studies with 80 or more participants included. Results show that the most effective telecare interventions appear to be automated vital signs monitoring (for reducing health service use) and telephone follow-up by nurses (for improving clinical indicators and reducing health service use). Evidence on cost-effectiveness is less clear, and on safety and security alert systems insufficient.

Having identified where there is evidence of effectiveness, and where it is lacking, the authors conclude that insufficient evidence does not amount to lack of effectiveness: more research is needed.

Bussey-Smith and Rossen [29]

Asthma USA, Hawaii, Sweden

Computer-based patient education programmes (CAPEPs)

Behavioural, Cost/economic, Health, Percep- tion/satisfaction, Social

Nine studies included. One study each showed reduced hospitalizations, acute care visits, or rescue inhaler use. Two reported lung function improvements. Four showed improved asthma knowledge, and five showed improvements in symptoms.

Although interactive CAPEPs may improve patient asthma knowledge and symptoms, their effect on objective clinical outcomes is less consistent

Jaana et al.[30] Diabetes North America, Europe and Asia

Remote monitoring

Behavioural, Health, Technology related, Structural

Seventeen studies included. Most studies reported overall positive results in Diabetes mellitus type 2, and found that IT based interventions improved health care utilisation, behaviour attitudes and skills.

Positive effects are reported, but there is variation in patient characteristics (background, ability, medical condition) sample selection and approach for treatment of control groups.

Azarmina and Wallace [31]

All All countries Remote interpretation in medical encounters

Cost/economic, Feasibility/pilot, Health, Organisational, Per- ception/satisfaction, Safety

Nine studies included. Results showed that time between encounters was reduced, but evidence on consultation length was not consistent. Good client and doctor satisfaction was shown, but those interpreting data preferred to do so face to face. Costs of these interventions are high, but efficiency gains are possible.

The review suggests that remote interpretation is an acceptable and accurate alternative to traditional methods, despite the higher associated costs.

in t

e r

n a

t io

n a

l jo

u r

n a

l o

f m

e d

ic a

l in

f o

r m

a t

ic s

7 9

( 2

0 1

0 )

7 3

6 –7

7 1

7 4 7

Demiris and Hensel [32]

Older people, people with disabilities

Europe, USA, Asia

Smart home Behavioural, Health, Safety, Social, Physiological and functional

Twenty-one projects included (drawing on 114 publications). A table is presented with their technologies, target audience, technologies and different outcome. A lack of evidence on clinical outcomes is identified.

Most of the studies demonstrated the feasibility of the technological solution. Technical, ethical, legal, clinical, economical and organisational implications and challenges need to be studied in-depth for the field to grow further.

Martinez et al. [33]

Heart failure All countries Remote monitoring (home)

Behavioural, Cost/economic, Feasibility/pilot, Health, Legal, Organizational, Per- ception/satisfaction, Safety, Social, Technology related

Forty-two studies included. (1) Remote monitoring for cardiac heart failure appears to be technically effective for following the patient remotely; (2) it appears to be easy to use, and it is widely accepted by patients and health professionals; and (3) it appears to be economically viable.

Evaluating the articles showed that home monitoring in patients with heart failure is viable.

Gaikwad and Warren [34]

Chronic disease

Not stated Home based ICT interventions

Behavioural, Cost/economic, Health, Percep- tion/satisfaction

Twenty-seven studies included. These systems can improve functional and cognitive patient outcomes in chronic disease and reduce costs. However, the research is not yet sufficiently robust.

Telecare, telehealth etc. have positive clinical and cost outcomes – although studies are few in number and heterogeneous. Better evidence-based outcome measures are needed, especially regarding costs and physician perspectives.

7 4 8

in t

e r

n a

t io

n a

l jo

u r

n a

l o

f m

e d

ic a

l in

f o

r m

a t

ic s

7 9

( 2

0 1

0 )

7 3

6 –7

7 1

Table 2 – Systematic reviews reporting that telemedicine is promising.

Reference Conditions included

Geographic area

Service/ intervention

Outcome Authors’ summary of results Authors’ conclusions

Cuijpers et al. [35] Pain and other health problems

Global CBT via internet

Health Twelve studies included (RCTs). Three studies focused on pain, three on headache, and six on other health problems. Effects found for Internet interventions targeting pain and headache were comparable to effects found for face-to-face treatments, and the same was true for interventions aimed at headache. Other interventions also showed some effects, which differed across target conditions.

Internet-delivered cognitive behavioural interventions are a promising addition and complement to existing treatments. The Internet will most likely assume a major role in the future delivery of cognitive behavioural interventions to patients with health problems. More research on eCBT is needed.

Bee et al. [36] Anxiety and depression

Not stated Psychotherapy mediated by remote com- munication technology

Behavioural outcomes

Thirteen studies included. Pooled effect sizes for remote vs. conventional services were 0.44 for depression and 1.15 for anxiety related disorders. Few studies compare remote and face-to-face psychotherapy. Data suggest that good effects may not be dependent on patient and therapist being co-located, but the evidence is limited.

Remote therapy has the potential to overcome some of the barriers to conventional psychological therapy services, but large scale trials are needed.

Crosbie et al. [37] Stroke Not stated Virtual reality (VR) in stroke rehabilitation

Behavioural, Health, Safety

Eleven studies included. Five cover upper limb rehabilitation, three, gait and balance, two cognitive interventions, and one both upper and lower limb rehabilitation. Three were AACPDM Level I/Weak, two Level III/Weak, three Level IV/Weak and three Level V quality of evidence. Three RCTs obtained statistical significance, and eight studies found VR based therapy to be beneficial. None reported any significant adverse effects.

VR is a potentially exciting and safe tool for stroke rehabilitation but its evidence base is too limited by design and power issues to permit a definitive assessment of its value. Thus, while the findings of this review are generally positive, the level of evidence is still weak to moderate, in terms of research quality. Further study in the form of rigorous controlled studies is warranted.

Henderson et al. [38]

Stroke Global Virtual reality (VR) in stroke rehabilitation

Health Six studies included. The results of the reviewed studies suggest that immersive VR may have an advantage over no therapy in the rehabilitation of the upper limb in patients with stroke, but the results are still questionable.

Current evidence on the effectiveness of VR in the rehabilitation of upper limb in patients with stroke is limited but sufficiently encouraging to justify further research in this area.

in t

e r

n a

t io

n a

l jo

u r

n a

l o

f m

e d

ic a

l in

f o

r m

a t

ic s

7 9

( 2

0 1

0 )

7 3

6 –7

7 1

7 4 9

Griffiths and Christensen [39]

Mental health Global Internet interventions

Behavioural, Health, Percep- tion/satisfaction

Sixteen papers included (reporting15 RCTs). The review demonstrates that Internet interventions show promise as a means of improving symptoms and behaviour associated with and knowledge about specific mental disorders and related conditions.

Most interventions were reported to be effective in reducing risk factors or improving symptoms, although many of the studies had methodological limitations. Three of the interventions that reported positive outcomes are available without charges to the public.

Jackson et al. [40] Diabetes Not stated Computer assisted interactive IT

Behavioural, Cost/economic, Health

Twenty-seven papers included (reporting 26 studies) Significant impacts on behavioural, clinical and structural levels.

There is growing evidence that emerging IT may improve diabetes care. Future research should characterize benefits in the long term, establish methods to evaluate clinical outcomes, and determine the cost-effectiveness.

Azar and Gabbay [83]

Diabetes Not stated Web-based management of glucose uploads

Clinical, Health, Behavioural

Eight studies on type 1 diabetes seven papers on type 2 (or mixed populations). The type 1 studies tended to show equivalent HbA1c improvements in intervention and control groups. For type B patients, there were statistically significant differences for HbA1between intervention and control groups.

Patients benefitted from web-based diabetes management, through savings in time and cost. Major obstacles to wider implementation are patient computer skills, adherence to the technology, architectural and technical design and the need to reimburse providers for their care.

Weinstein [41] Obesity USA Internet based programmes

Health Eight studies included. All studies examining weight loss via Internet programs reported positive results, except one investigating a commercial program. Results from the three weight loss maintenance programs conducted on the Internet were equivocal. Because the subjects of all these studies were predominately white, educated women, generalisability of findings is limited.

Preliminary studies suggest that the internet may be an adequate vehicle for weight loss intervention and possibly for programs directed at weight loss maintenance.

7 5 0

in t

e r

n a

t io

n a

l jo

u r

n a

l o

f m

e d

ic a

l in

f o

r m

a t

ic s

7 9

( 2

0 1

0 )

7 3

6 –7

7 1

Table 2 (Continued)

Reference Conditions included

Geographic area

Service/ intervention

Outcome Authors’ summary of results Authors’ conclusions

Elliott et al. [88] Alcohol abuse Mostly US studies

Computer- based treatment

Behavioural, Health, Percep- tion/satisfaction, Social

Seventeen studies included (RCTs). The e-interventions emerge as more effective than no treatment, and approximately equivalent to alternative intervention approaches. E-interventions appear to be superior to assessment-only (AO) control conditions. In addition, possible moderators (e.g. baseline drinking patterns) and mediators (e.g. corrected drinking norms) have emerged.

These studies compared the effectiveness of e-interventions with other commonly used techniques, reading materials, and assessment-only control conditions. Overall, findings provide some support for such programs, especially in comparison with assessment-only control conditions.

Deshpande et al. (asynchronous telehealth) [42]

All, but especially dermatologi- cal conditions

World Store and forward of clinically important digital samples for assessment at a convenient time

Cost/economic, Health, Organisational, Safety, Social

Fifty-two studies included (16 of good quality). An environmental scan found 39 organisations using both real time and asynchronous services. Overall study quality is poor, evidence suggests shorter waiting times, fewer unnecessary referrals, high levels of patient and provider satisfaction, and equivalent (or better) diagnostic accuracy when compared with face-to-face consultations. It is unknown whether the benefits that have been shown in small local studies could be realized after wide-scale implementation.

Pragmatic objectives for implementation and evaluation are needed, involving policy makers, specifically in Canada. Standardisation procedures are already in progress and are likely to improve collaboration between providers.

Polisena et al. (diabetes) [43]

Diabetes Worldwide Home telehealth

Health Twenty-six studies included. Study results indicated that home telehealth helps to reduce the number of patients hospitalised, hospitalizations and bed days of care. Home telehealth was similar or favourable to UC across studies for quality-of-life and patient satisfaction outcomes.

In general, home telehealth had a positive impact on the use of numerous health services and glycaemic control. More studies of higher methodological quality are required to give more precise insights into the potential clinical effectiveness of home telehealth interventions.

Holland et al. [89] Heart failure Europe, USA, Australia, New Zealand, Argentina

Remote monitoring, telephone support

Health Thirty studies included (RCTs). Sensitivity analysis of combined data from 30 interventions showed reduction in hospital admission and reduction in all-cause mortality. Comparison of interventions in different settings indicated that effective interventions were delivered at least partly at home.

Results suggest that multidisciplinary interventions for patients with heart failure reduce hospital admissions and mortality. Interventions include telemedicine, but there is limited evidence in this area.

in t

e r

n a

t io

n a

l jo

u r

n a

l o

f m

e d

ic a

l in

f o

r m

a t

ic s

7 9

( 2

0 1

0 )

7 3

6 –7

7 1

7 5 1

Jaana et al. [17] Respiratory conditions

USA, Europe, Israel, Taiwan English- language publications in peer- reviewed journals

Remote monitoring

Behavioural, Cost/economic, Feasibility/pilot, Health, Percep- tion/satisfaction

Twenty-three studies included. Good levels of data validity and reliability were reported. However, little quantitative evidence exists about the effect of remote monitoring on patient medical condition and utilization of health services. Positive effects on patient behaviour were consistently reported. Only two studies performed a detailed cost analysis.

Home telemonitoring of respiratory conditions results in early identification of deteriorations in patient condition and symptom control. Positive patient attitude and receptiveness of this approach are promising. However, evidence on the magnitude of clinical and structural effects remains preliminary, with variations in study approaches and an absence of robust study designs and formal evaluations.

Hailey et al. [44] Cancer United States, Canada, Denmark, Finland, Spain, UK

Comparing a wide range of interventions

Cost/economic, Health, Percep- tion/satisfaction, Social

Fifty-four studies included. Positive findings from higher-quality studies suggested that telephone-based technology was effective for promoting mammography and colposcopy in specific populations, for increasing fruit and vegetable consumption, and as an alternative to in-person support groups for women with breast cancer. Eight economic assessments provided some indications of cost advantages in diagnosis and treatment, and in palliative care. 20 papers reported patient and family satisfaction. However, the significance and generalisability of these findings appears limited.

From the perspective of the Alberta Cancer Board, the literature suggests some useful possibilities for developing new services using internet or web-based, telephone-based, and video-based technologies for cancer patients in rural areas. However, it seems likely that these applications would need validation with suitable local studies.

Pare et al. [45] Chronic disease

US, Canada, Europe, Asia

Remote monitoring

Behavioural, Health, Organisational, Policy, Technology related

Sixty-five studies included. The magnitude and significance of the effect of (automated) telemonitoring of patients with four chronic conditions (hypertension, diabetes, COPD and cardiac failure) remains inconclusive. Patients’ compliance is high, The clinical effectiveness (e.g. less frequent hospitalisation) is more consistent in cardiac and pulmonary studies than in diabetes and hypertension. Economic viability was poorly evidenced.

Telemonitoring seems to be a promising approach to chronic patient management. Technology is reliable. It empowers patients, and potentially improves clinical conditions.

7 5 2

in t

e r

n a

t io

n a

l jo

u r

n a

l o

f m

e d

ic a

l in

f o

r m

a t

ic s

7 9

( 2

0 1

0 )

7 3

6 –7

7 1

Table 2 (Continued)

Reference Conditions included

Geographic area

Service/ intervention

Outcome Authors’ summary of results Authors’ conclusions

Nannings and Abu-Hanna [46]

All (especially chronic)

Not mentioned

Decision support models for professional use

Technology related, Conceptual models

Sixty-five studies included. The objectives were to define the term DSTS, to propose a general DSTS model, and to propose model-based templates to aid DSTS development for three medical tasks. The definition, general model and model-based templates are based on a systematic literature search. The applicability of the templates to new DSTSs found in a separate limited literature search was tested. A definition of DSTSs is provided, a conceptual model for understanding DSTSs is proposed and a set of reusable templates, and examples for using them are provided.

The conceptual model and the reusable modelling templates are demonstrated to be useful in understanding and modelling DSTSs during the early stages of their development.

Sintchenko et al. [47]

Diabetes, heart failure, and chronic obstructive pulmonary disease (COPD). Other chronic diseases that could be managed by using home telehealth

Not stated Home telehealth

Clinical outcomes; health service use, QoL, Patient satisfaction

Twenty-four studies included. The authors feel they have shown that EDSS improves prescribing practices and treatment outcomes of patients with acute illnesses, but are less effective in primary care.

The authors state that further research is needed to quantify the range of benefits of EDSS, and to explore new measurement metrics and enhance the appropriate clinicaluse of electronic decision support.

Wu and Langhorne [48]

Stroke Found studies only from Germany, US, France

Acute medicine

Cost/economic, Feasibility/pilot, Health, Percep- tion/satisfaction, Safety

Seventeen studies included. Studies reported that telemedicine systems were feasible and acceptable. Inter-rater reliability was excellent for global clinical assessments and decisions on radiological exclusion criteria although agreement for individual assessment items was more variable. Telemedicine systems were associated with increased use of tPA.

Although there is limited reliable evidence, observational studies have indicated that telemedicine systems can be feasible, acceptable, and reliable in acute-stroke management. In addition, telemedicine consultations were associated with improved delivery of tPA.

Shojania et al. [49] Medication, vaccination and tests (various conditions unspecified)

Worldwide Decision support for clinicians

Health Twenty-eight studies included. The studies tested reminders to prescribe specific medications, to warn about drug interactions, to provide vaccinations, or to order tests. The review found small to moderate benefits. The reminders improved physician practices by a median of 4%. In eight of the studies, patients’ health improved by a median of 3%.

Point of care computer reminders generally achieve small to modest improvements in provider behaviour. A minority of interventions showed larger effects, but no specific reminder or contextual features were significantly associated with effect magnitude.

in t

e r

n a

t io

n a

l jo

u r

n a

l o

f m

e d

ic a

l in

f o

r m

a t

ic s

7 9

( 2

0 1

0 )

7 3

6 –7

7 1

7 5 3

Table 3 – Systematic reviews reporting that evidence on telemedicine is limited and inconsistent.

Reference Conditions included

Geographic area

Service/ intervention

Outcome Authors’ summary of results Authors’ conclusions

Chaudhry et al. [50] Heart failure Telephone-based monitoring: Argentina (1 study), USA 4 studies. Automated monitoring USA 1 study. Automated physiologic monitoring USA 1 study. Comparisons of two or more methods of telemonitoring: Germany/the Netherlands and UK: one study, USA 1 study

Remote monitoring

Cost/economic, Health

Nine studies included. Six suggested reduction in all-cause and heart failure hospitalisations with telemonitoring. Of the three negative studies, two enrolled low-risk patients and patients with access to high quality care, and one enrolled a very high-risk Hispanic population. Studies comparing forms of telemonitoring demonstrated similar effectiveness. Intervention costs were higher with more complex programs.

The evidence base for telemonitoring in heart failure is currently quite limited. Based on the available data, telemonitoring may be an effective strategy for disease management in high-risk heart failure patients.

Farmer et al. [51] Diabetes Not stated Self- monitoring, Data transfer

Cost/economic, Feasibility/pilot, Health, Organizational, Percep- tion/satisfaction

Twenty-six studies included. Electronic transfer of glucose results appears feasible in a clinical setting. Only two of the RCTs included more than 100 patients, and only three extended to one year. Only one study was designed to show that telemedicine interventions might replace clinic interventions without deterioration in HbA Results pooled from the nine RCTs with reported data did not provide evidence that the interventions were effective in reducing HbA to 0.04%) (p. 1372).

Telemedicine solutions for diabetes care are feasible and acceptable, but evidence for their effectiveness in improving HbA or reducing costs while maintaining HbA levels, or improving other aspects of diabetes management is not strong. Further research should seek to understand how telemedicine might enhance educational and self-management interventions and RCTs are required to examine cost-effectiveness.

Sanders and Aronsky [52]

Asthma USA Diagnostics, prevention and monitoring, decision support tools, patient- centred education tools

Behavioural, Health, Social

Sixty-four studies included, but only 21 prospective trials. The mean quality score was 6.6 (range: 3–10). None of the studies reported on allocation concealment. Of the 13 studies that reported a clinical outcome, seven reported a positive effect of the computerised intervention and six reported no significant change. Of the eight studies reporting a non-clinical outcome, seven reported a statistically significant positive effect of the computerised intervention.

Most studies took place in the outpatient clinic environment, with minimal study of the emergency department or inpatient settings. Few studies demonstrated evidence of computerised applications improving clinical outcomes.

7 5 4

in t

e r

n a

t io

n a

l jo

u r

n a

l o

f m

e d

ic a

l in

f o

r m

a t

ic s

7 9

( 2

0 1

0 )

7 3

6 –7

7 1

Table 3 (Continued)

Reference Conditions included

Geographic area

Service/ intervention

Outcome Authors’ summary of results Authors’ conclusions

Verhoeven et al. [53]

Diabetes Worldwide Teleconsultation and video- conferencing

Cost/economic, Health, Percep- tion/satisfaction, Technology related

Thirty-nine studies included. They found no significant statistical heterogeneity among the pooled randomised controlled trials that they identified within this group. Most of the improvements found concerned (a) satisfaction with technology, (b) improved metabolic control or (c) cost savings. No significant benefits were found in relation to (a) quality of life, (b) transparency, or (c) better access to care.

The study did not support any conclusion that these interventions improved clinical values (e.g. blood pressure). The authors argued that diversity in design of the studies meant that strong conclusions were premature.

Bewick et al. [54] Alcohol abuse

Worldwide Web-based interventions

Behavioural, Cost/economic

Ten studies included. Five gave process evaluations and five, some pre- and post-intervention results. Only one study was an RCT, and the studies scored low on quality criteria.

There is inconsistent evidence on the effectiveness of electronic screening and brief intervention (eSBI) for alcohol use. Process research suggests that web-based interventions are generally well received. However further controlled trials are needed to fully investigate their efficacy, to determine which elements are keys to outcome and to understand if different elements are required in order to engage low and high-risk drinkers.

Crosbie et al. [37] Stroke Not stated Virtual reality in stroke rehabilitation

Behavioural, Health, Safety

Eleven studies included. Five cover upper limb rehabilitation, three, gait and balance, two cognitive interventions, and one both upper and lower limb rehabilitation. Three were AACPDM Level I/Weak, two Level III/Weak, three Level IV/Weak and three Level V quality of evidence. Three RCTs obtained statistical significance, and eight studies found VR based therapy to be beneficial. None reported any significant adverse effects.

VR is a potentially exciting and safe tool for stroke rehabilitation but its evidence base is too limited by design and power issues to permit a definitive assessment of its value. Thus, while the findings of this review are generally positive, the level of evidence is still weak to moderate, in terms of research quality. Further study in the form of rigorous controlled studies is warranted.

Maric et al. [55] Heart failure

Not stated Remote monitoring

Cost/economic, Feasibility/pilot, Health, Organisational

Fifty-six papers included. Many of the identified papers on telemonitoring of heartfFailure (HF) patients demonstrate at least some benefit. There is a general trend towards improvement associated with the use of most modalities. However inconsistent evidence between trials for the same modality and difference between modalities make a definitive conclusion difficult. Evidence of benefit from video consultation alone is lacking. The majority of papers reported trials that are non-randomised and many papers had small sample sizes.

This study reviewed studies conducted in HF telemonitoring, to describe the nature of the modality, the methods and the results. Telemonitoring appeared to be an acceptable method for monitoring HF patients. Controlled, randomized studies directly comparing different modalities and evaluating their success and feasibility when used as routine clinical care are now required.

in t

e r

n a

t io

n a

l jo

u r

n a

l o

f m

e d

ic a

l in

f o

r m

a t

ic s

7 9

( 2

0 1

0 )

7 3

6 –7

7 1

7 5 5

Postel et al. [56]. Mental health problems

North American and European

E-therapy Health Fourteen studies included. E-therapy seems to be promising but study results are still based only on small groups.

More research is needed, and should be better reported.

Martin et al. [57] Physical disability, cognitive impairment, learning disability

Not explicitly mentioned

Smart home Cost/economic, health, organizational, social and technology issues

No study identified met the inclusion criteria.

The review does not provide sufficient evidence to support or refute the integration of smart home technologies into health and social care. RCTs should be adopted. International descriptive terminology is needed. The rate of abandonment by individuals of the technologies needs investigation.

Powell et al. [58] Dementia (carers’ stress and depression)

North America Five specific technological interventions

Behavioural, Cost/economic, knowledge

Fifteen studies included. They describe five interventions: ComputerLink, AlzOnline, Caring for Others and two studies from the REACH project (TLC and CTIS). The interventions reviewed were multifaceted with elements of networked peer support. Outcomes were inconsistent but suggested that the interventions had moderate effects on improving carer stress and depression. Treatment effects were found to vary with caregiver characteristics such as ethnic groups, formal support and baseline burden.

Further evaluation is needed in robust trials with good follow-up.

Barlow et al. [28] Elderly people, chronic diseases

Worldwide Home telecare

Behavioural, Health, Organisational, Safety

Sixty-eight RCTs and 30 observational studies with 80 or more participants included. Results show that the most effective telecare interventions appear to be automated vital signs monitoring (for reducing health service use) and telephone follow-up by nurses (for improving clinical indicators and reducing health service use). Evidence on cost-effectiveness is less clear, and on safety and security alert systems insufficient.

Having identified where there is evidence of effectiveness, and where it is lacking, the authors conclude that insufficient evidence does not amount to lack of effectiveness: more research is needed.

7 5 6

in t

e r

n a

t io

n a

l jo

u r

n a

l o

f m

e d

ic a

l in

f o

r m

a t

ic s

7 9

( 2

0 1

0 )

7 3

6 –7

7 1

Table 3 (Continued)

Reference Conditions included

Geographic area

Service/ intervention

Outcome Authors’ summary of results Authors’ conclusions

Garcia-Lizana and Sarria- Santamera [59]

Chronic disease

Nations around the world

Management, control and prevention

Health, Percep- tion/satisfaction, Safety

Twenty-four studies included. Improvements in clinical outcomes were not shown, though no adverse effects were identified. In the detection and follow-up of cardio-vascular diseases however, ICTs provided better clinical outcomes, mortality reduction and lower health services utilization. Systems used for improving education and social support were also shown to be effective.

Although some positive results were identified, at present the evidence about the clinical benefits of ICTs for managing chronic disease is limited.

Gagnon et al. [60] None (imple- mentation issues)

Not stated Educational Behavioural, Cost/economic

Twenty-seven references listed. The authors find very little data on interventions to promote ICT use, and argue that more studies are needed.

Too few studies, which are too heterogeneous, make it impossible to draw any significant conclusions about the impact of interventions to promote ICT use by HCPs.

Hersh et al. [4] Various conditions (Medicare population)

USA, UK, Australia, Europe

Comparing store and forward, home based, office- hospital- based services

Health, Organisational, Concordance studies, diagnosis and management, access

Ninety-seven studies included. Store and forward services have been studied in many specialties, especially dermatology, wound care, and ophthalmology. The evidence for their efficacy is mixed, and in most areas, there are not corresponding studies on outcomes or improved access to care. Several limited studies showed benefits of home based telemedicine interventions in chronic diseases. These interventions appear to enhance communication with providers and provide closer monitoring of general health, but the studies were conducted in settings requiring additional resources and dedicated staff. Studies of office/hospital-based telemedicine suggest that telemedicine is most effective for verbal interactions, e.g. videoconferencing for diagnosis and treatment neurology and psychiatry.

Outside of a small number of clinical specialties, the evidence base for the efficacy of telemedicine is weak. Further well-designed and targeted research that provides high quality data will provide a strong contribution to understanding how best to deploy technological resources in health care.

in t

e r

n a

t io

n a

l jo

u r

n a

l o

f m

e d

ic a

l in

f o

r m

a t

ic s

7 9

( 2

0 1

0 )

7 3

6 –7

7 1

7 5 7

Demiris and Hensel [32]

Older people, people with disabilities

Europe, USA, Asia Smart home Behavioural, Health, Safety, Social, Physiological and functional

Twenty-one projects included (drawing on 114 publications). A table is presented with their technologies, target audience, technologies and different outcome. A lack of evidence on clinical outcomes is identified.

Most of the studies demonstrate the feasibility of the technological solution. Technical, ethical, legal, clinical, economical and organisational implications and challenges need to be studied in-depth for the field to grow further.

Hailey et al. [44] Cancer United States, Canada, Denmark, Finland, Spain, UK

Comparing a wide range of interventions

Cost/economic, Health, Percep- tion/satisfaction, Social

Fifty-four studies included. Positive findings from higher-quality studies suggested that telephone-based technology was effective for promoting mammography and colposcopy in specific populations, for increasing fruit and vegetable consumption, and as an alternative to in-person support groups for women with breast cancer. Eight economic assessments provided some indications of cost advantages in diagnosis and treatment, and in palliative care. 20 papers reported patient and family satisfaction. However, the significance and generalisability of these findings appears limited.

From the perspective of the Alberta Cancer Board, the literature suggests some useful possibilities for developing new services using internet or web-based, telephone-based, and video-based technologies for cancer patients in rural areas. However, it seems likely that these applications would need validation with suitable local studies.

Linton [62] Anxiety and depression

Not specified Computer- based cognitive behavioural therapy (CBT)

Behavioural, Cost/economic, Ethical issues, Health, Percep- tion/satisfaction, self-assessment

Twelve studies included. The paper investigates what effects and costs are associated with computer-based CBT in treating adult patients with anxiety disorders or depression.

There is limited scientific evidence indicating that computer-based CBT has favourable, short-term effects on symptoms in the treatment of panic disorder, social phobia, and depression. The scientific evidence is insufficient to assess the effects of treatment on obsessive–compulsive disorder and mixed anxiety/depression. The scientific evidence is insufficient to assess the cost-effectiveness of the method.

7 5 8

in t

e r

n a

t io

n a

l jo

u r

n a

l o

f m

e d

ic a

l in

f o

r m

a t

ic s

7 9

( 2

0 1

0 )

7 3

6 –7

7 1

Table 3 (Continued)

Reference Conditions included

Geographic area

Service/ intervention

Outcome Authors’ summary of results Authors’ conclusions

Mathur et al. [63] Diabetes Not stated Evaluation Feasibility/pilot, Health

Fourteen studies included. Marked variability in operational design of programs and poor rationalization of choice of outcome metrics with program components found.

Causal pathways in decline of HbA1c remain unclear, even though 11 of 14 studies showed decline. The authors recommend a standardized methodology for analyzing communications technology use in diabetes care.

Walters et al. [64] Smoking cessation

Worldwide Web and computer- based programmes

Behavioural Ninteen studies included. 47% of included studies reported statistically significant or improved outcomes at the longest follow-up, relative to a comparison group.

Few patterns emerged in terms of subject, design or intervention characteristics that led to positive outcomes. The ‘first generation’ format, where participants were mailed computer-generated feedback reports, was the modal intervention format and the one most consistently associated with improved outcomes.

Lintonen et al. [65] Promotion of health

Global Emerging information and commu- nications technology, especially the internet

Classified uses of IT generally

Fifty-six papers included. Using the papers studied, the authors classified IT in health promotion: IT as an intervention medium; IT as a research focus; IT as a research instrument and IT for professional development.

IT in health promotion is only just emerging. While a lot of health promotion information is provided over internet, little published research exists on its use and usefulness.

Van Nooten et al. [66]

All Not stated Spiritual care Information, assessments, theories of role of spiritual care and interventions

Eighteen studies included (17 articles and one book). There are very few systematic studies on use of Internet for spiritual care as e-health. Four different categories of spiritual and religious care are identified: information search, assessments, theories and interventions.

No systematic studies were found on how ICTs are being used to integrate religious and spiritual care into healthcare.

Jaana et al. [90] Cardio- vascular (hyperten- sion)

USA, Europe, Japan, Israel, Malaysia

Remote monitoring

Behavioural, Cost/economic, Feasibility/pilot, Blood pressure control

Fourteen studies included. The studies present evidence on the positive impacts of telemonitoring on patients and their condition: significant BP control, better medication adherence, changes in patient lifestyle and attitudes. The studies were considered limited because of study designs. Little is known on the effects of telemonitoring on services utilization. Only one study demonstrated a detailed cost-effectiveness analysis.

Preliminary evidence exists on benefits. However, limited information exists on effects of utilization and economic viability. Future studies should include both RCT’s and research to understand processual conditions and mechanisms through which BP control is achieved. (Formative evaluations).

in t

e r

n a

t io

n a

l jo

u r

n a

l o

f m

e d

ic a

l in

f o

r m

a t

ic s

7 9

( 2

0 1

0 )

7 3

6 –7

7 1

7 5 9

Table 4 – Emerging issue 1: economic analysis.

Reference Conditions included

Geographic area

Service/intervention Outcome Authors’ summary of results Authors’ conclusions

Rojas and Gagnon [67]

All Worldwide Telehomecare (THC)

Cost/economic, Health, Percep- tion/satisfaction

Twenty-three studies included. THC was found to be a cost-effective alternative to traditional approaches in 91% of the studies. Main benefits included decreased hospital utilisation; improved patient compliance with treatment plans; improved patient satisfaction with health services and improved quality of life.

The authors argue that one of the major disadvantage is of THC studies has been the failure to adopt a set of common indicators to calibrate their cost-effectiveness.

Gaikwad and Warren [34]

Chronic disease

Not stated Home based ICT interventions

Behavioural, Cost/economic, Health, Percep- tion/satisfaction

Twenty-seven studies included. These systems can improve functional and cognitive patient outcomes in chronic disease and reduce costs. However, the research is not yet sufficiently robust.

Telecare, telehealth etc. have positive clinical and cost outcomes – although studies are few in number and heterogeneous. Better evidence-based outcome measures are needed, especially regarding costs and physician perspectives.

Seto [68] Heart failure Worldwide Remote monitoring

Cost/economic Eleven studies included. The author considered a variety of direct and indirect cost categories including costs to the health system and costs to the patient. These are described.

Telemonitoring has a positive role to play in reducing costs by reducing re-hospitalisation and travel costs.

Tran et al. [25] Diabetes, heart failure, COPD and other chronic diseases

Canada focused, but international publications included

Home telehealth

Cost/economic, Health, Percep- tion/satisfaction

Seventy-nine studies included. Of the included studies, 26 pertained to diabetes, 35 to chronic heart failure, nine to COPD, and eight to mixed chronic diseases. The comparator “no care” was not identified in any of the included studies, so usual care was used as the comparator throughout the clinical review. Home telehealth appeared generally clinically effective and no patient adverse effects were reported. Evidence on health service utilization was more limited, but promising The economic review suggested cost-effectiveness, but the quality of studies was low.

Conclusions relate to the potential for home telehealth in Canada which is seen as positive. However, more research, such as multicentre RCTs, is warranted to accurately measure the clinical and economic impact of home telehealth for chronic disease management to support Canadian policy makers in making informed decisions.

7 6 0

in t

e r

n a

t io

n a

l jo

u r

n a

l o

f m

e d

ic a

l in

f o

r m

a t

ic s

7 9

( 2

0 1

0 )

7 3

6 –7

7 1

Table 4 (Continued)

Reference Conditions included

Geographic area

Service/intervention Outcome Authors’ summary of results Authors’ conclusions

Azarmina and Wallace [31]

All All countries Remote inter- pretation in medical encounters

Cost/economic Feasibility/pilot, Health, Organizational, Percep- tion/satisfaction, Safety

Nine studies included. Results showed that time between encounters was reduced, but evidence on consultation length was not consistent. Good client and doctor satisfaction was shown, but those interpreting data preferred to do so face to face. Costs of these interventions are high, but efficiency gains are possible.

The review suggests that remote interpretation is an acceptable and accurate alternative to traditional methods, despite the higher associated costs.

Barlow et al. [28] Elderly people, chronic diseases

Worldwide Home telecare

Behavioural, Health, Organizational, Safety

Sixty-eight RCTs and 30 observational studies with 80 or more participants included. Results show that the most effective telecare interventions appear to be automated vital signs monitoring (for reducing health service use) and telephone follow-up by nurses (for improving clinical indicators and reducing health service use). Evidence on cost-effectiveness is less clear, and on safety and security alert systems insufficient.

Having identified where there is evidence of effectiveness, and where it is lacking, the authors conclude that insufficient evidence does not amount to lack of effectiveness: more research is needed.

Deshpande et al. (synchronous telehealth) [69]

Chronic disease, CHF, psychological and neurological problems

All countries Home telehealth, in real time

Behavioural, Cost/economic, Feasibility/pilot, Health, Percep- tion/satisfaction, Safety, Social, Technology related

Thirty-one systematic reviews included (11 of high quality). Most reviews and most of the studies they review are low quality. The authors suggest that this illustrates the resource constraints for researchers and policy makers. Nevertheless, weak evidence that real-time telehealth can improve service access, user satisfaction and resource utilization is found. For patients with psychiatric and neurological conditions in remote areas, evidence of benefit is stronger.

Evidence is generally weak and studies of poor quality, but telehealth is nevertheless promising, especially in a Canadian context over a large geographical area.

in t

e r

n a

t io

n a

l jo

u r

n a

l o

f m

e d

ic a

l in

f o

r m

a t

ic s

7 9

( 2

0 1

0 )

7 3

6 –7

7 1

7 6 1

Barak et al. [15] Mental health Not stated Internet based psy- chotherapy

Behavioural, Health, Percep- tion/satisfaction, Social

Sixty-four studies included services. The overall mean weighted effect size was 0.53 (medium effect), similar to the average effect size of traditional, face-to-face therapy. Comparison between face-to-face and Internet intervention across 14 studies showed no differences in effectiveness.

Internet based intervention as effective as face-to-face intervention.

Jackson et al. [40] Diabetes Not stated Computer assisted interactive IT

Behavioural, Cost/economic, Health

Twenty-seven papers included (reporting 26 studies) Significant impacts on behavioural, clinical and structural levels.

There is growing evidence that emerging IT may improve diabetes care. Future research should characterize benefits in the long term, establish methods to evaluate clinical outcomes, and determine the cost-effectiveness.

Kaltenthaler et al. [70]

Mental health (depression and anxiety)

Worldwide Computerised cognitive behavioural therapy (CCBT)

Cost/economic, Health, Percep- tion/satisfaction, Safety

Twenty-four CCBT studies and one economic analysis included. There is some evidence for positive effects of CCBT, and of cost-effectiveness compared with usual care. The authors identify ‘significant uncertainty’ regarding whether the treatments should be adopted.

The authors identify a long list of needs for further research, including access to CBT, comparing CCBT with other interventions, CCBT via the internet, more RCT type studies, patient preferences, studies in GP settings and studies of co-morbidities.

7 6 2

in t

e r

n a

t io

n a

l jo

u r

n a

l o

f m

e d

ic a

l in

f o

r m

a t

ic s

7 9

( 2

0 1

0 )

7 3

6 –7

7 1

Table 4 (Continued)

Reference Conditions included

Geographic area

Service/intervention Outcome Authors’ summary of results Authors’ conclusions

Linton [62] Anxiety and depression

Not specified Computer- based CBT

Behavioural, Cost/economic, Ethical issues, Health, Percep- tion/satisfaction, Self-assessment

Twelve studies included. The paper investigates what effects and costs are associated with computer-based CBT in treating adult patients with anxiety disorders or depression.

There is limited scientific evidence (indicating that computer-based CBT has favourable, short-term effects on symptoms in the treatment of panic disorder, social phobia, and depression. The scientific evidence is insufficient to assess the effects of treatment on obsessive–compulsive disorder and mixed anxiety/depression. The scientific evidence is insufficient to assess the cost-effectiveness of the method.

Polisena et al. (chronic disease) [71]

Chronic disease

Worldwide Home telehealth

Cost/economic Twenty-two studies included. Home telehealth was found to be cost saving from the healthcare system and insurance provider perspectives in all but two studies, but the quality of the studies was generally low. An evaluative framework was developed which provides a basis to improve the quality of future studies to facilitate improved healthcare decision-making and an application of the framework is illustrated using data from and existing programme evaluation of a home telehealth program.

Current evidence suggests that home telehealth has the potential to reduce costs, but its impact from a societal perspective remains uncertain until higher-quality studies become available.

Griffiths et al. [72] Many conditions – all involving the intervention

Sweden, United States, Australia, Canada, UK, Spain, Denmark, Italy, Netherlands, Japan

Internet delivery of healthcare interventions

Reasons for use, reflections

Thirty-seven studies included. These covered a range of health conditions. Reasons for Internet delivery included low cost and resource implications due to the nature of the technology; reducing cost and increasing convenience for users; reduction of health service costs; overcoming isolation of users; the need for timely information; stigma reduction; and increased user and supplier control of the intervention.

Internet delivery may have unintended consequences. It may overcome isolation by time, mobility, and geography, but it may not be a substitute for face-to-face contact. Costs to service users and their social networks as well as providers need to be considered. Reasons for choosing internet delivery must be made clear. Internet delivery needs to be compared with other modes of delivery.

in t

e r

n a

t io

n a

l jo

u r

n a

l o

f m

e d

ic a

l in

f o

r m

a t

ic s

7 9

( 2

0 1

0 )

7 3

6 –7

7 1

7 6 3

Table 5 – Emerging issue 2: is telemedicine good for patients?

Reference Conditions included

Geographic area

Service/ Intervention

Outcome Authors’ summary of results Authors’ conclusions

Akesson et al. [73]

All Worldwide Support, education/ information/ virtual consultation

Behavioural, Feasibility/pilot, Health, Percep- tion/satisfaction, Quality of life

Twelve studies included. Three themes identified: support and help, education and information, and telecommunication instead of on-site visiting. Findings show consumers feeling more confident and empowered, with increased knowledge and improved health status due to the ICT resources. Lack of face-to-face meetings or privacy did not appear to be a problem.

ICT can improve the nurse-patient relationship and augment well-being for consumers. More research is needed to measure consumers’ experiences and factors that influence it.

Kairy et al. [74]

Physical disabilities

Not stated Telerehabilitation Behavioural, Cost/economic, Health, Percep- tion/satisfaction, Social

Twenty-eight studies included. Clinical outcomes were generally improved; clinical process outcomes were high; consultation time was longer; patient satisfaction ratings were high. Healthcare utilization evidence was unclear. There is some evidence of potential cost savings.

While evidence is mounting concerning the efficacy and effectiveness of telerehabilitation, high quality evidence regarding impact on resource allocation and costs is still needed to support clinical and policy decision-making.

Murray et al. [75]

Chronic disease

Not stated Interactive health com- munication Applications (IHCA)

Behavioural, Health

Twenty-four studies included (RCTs). Computer-based programmes (’Interactive Health Communication Applications’) for people with chronic disease had a significant positive effect on knowledge, social support, and clinical outcomes. Results suggest it is more likely than not that IHCAs have a positive effect on self-efficacy. IHCAs had a significant positive effect on continuous behavioural outcomes. Binary behavioural outcomes also showed a positive effect for IHCAs, although this result was not statistically significant. It was not possible to determine the effects of IHCAs on emotional or economic outcomes.

IHCAs appear to have largely positive effects on users, in that users tend to become more knowledgeable, feel better socially supported, and may have improved behavioural and clinical outcomes compared to non-users. There is a need for more high quality studies with large sample sizes to confirm these preliminary findings, to determine the best type and best way to deliver IHCAs, and to establish how IHCAs have their effects for different groups of people with chronic illness.

7 6 4

in t

e r

n a

t io

n a

l jo

u r

n a

l o

f m

e d

ic a

l in

f o

r m

a t

ic s

7 9

( 2

0 1

0 )

7 3

6 –7

7 1

Table 5 (Continued )

Reference Conditions included

Geographic area

Service/ Intervention

Outcome Authors’ summary of results Authors’ conclusions

Deshpande et al. (syn- chronous telehealth) [69]

Chronic disease

All countries Home telehealth, in real time

Behavioural, Cost/economic, Feasibility/pilot, Health, Percep- tion/satisfaction, Safety, Social, Technology related

Thirty-one systematic reviews included (11 of high quality). Most reviews and most of the studies they review are low quality. The authors suggest that this illustrates the resource constraints for researchers and policy makers. Nevertheless, weak evidence that real-time telehealth can improve service access, user satisfaction and resource utilization is found. For patients with psychiatric and neurological conditions in remote areas, evidence of benefit is stronger.

Evidence is generally weak and studies of poor quality, but telehealth is nevertheless promising, especially in a Canadian context over a large geographical area.

Koch [76] Chronic diseases, elderly population, paediatrics

Country and no of publications: USA 238; UK 52; Japan 39; Germany 31; Greece 22; Australia 20; Canada 18; France 17; Spain 17; China 14; Italy 14; Sweden 13; Finland 6; other 77

Home telehealth

Cost/economic, Percep- tion/satisfaction, Feasibility/pilot, Health, Organizational, Technology related

Five hundred and seventy-eight studies included. 44% of publications come from the United States, followed by UK and Japan. Most cover vital sign parameter (VSP) measurement and audio/video consultations. Publications about IT tools for improved information access and communication as well as decision support for staff, patients and relatives are relatively sparse. Clinical application domains are mainly chronic diseases, the elderly population and paediatrics.

Internationally, we observe a trend towards tools and services not only for professionals but also for patients and citizens. However, their impact on the patient—provider relationship and their design for special user groups, such as elderly and/or disabled needs to be further explored. In general, evaluation studies are rare and further research is critical to determine the impacts and benefits, and limitations, of potential solutions and to overcome a number of hinders and restrictions, such as the lack of standards to combine incompatible information systems; the lack of an evaluation framework considering legal, ethical, organisational, clinical, usability and technical aspects; the lack of proper guidelines for practical implementation of home telehealth solutions.

Lauriks et al. [77]

Dementia No limits ICT-based interventions

Health, Percep- tion/satisfaction, Social, Technology related

Forty-six publications and 22 websites included. Needs identified are generalized and personalized information; support with regard to symptoms of dementia; social contact and company; health monitoring and perceived safety.

Information is generally not personalised and is not attuned to the person with dementia; ICT solutions aimed at compensating for disabilities demonstrate that people with mild to moderate dementia can handle simple electronic equipment and can benefit from it. Instrumental ICT-support for coping with behavioural and psychological changes in dementia is relatively disregarded as yet, while support for social contact can be effectively realised through, for example, simplified (mobile) phones. GPS technology and monitoring systems are proven to result in enhanced feelings of safety and less fear and anxiety.

in t

e r

n a

t io

n a

l jo

u r

n a

l o

f m

e d

ic a

l in

f o

r m

a t

ic s

7 9

( 2

0 1

0 )

7 3

6 –7

7 1

7 6 5

Table 6 – Asking new questions.

Reference Conditions included

Geographic area

Service/ intervention

Outcome Authors’ summary of results Authors’ conclusions

Mo et al. [78] All Not stated Online support groups

Behavioural, Health, Outcome of posted messages used to categorise gender differences

Twelve studies included. Half of the studies examined gender differences by comparing male and female cancer discussion boards. Some gender differences were observed in these studies. However, for studies that analysed mixed-gender communities, gender differences were less evident.

Results seemed to reveal gender differences in communications in single-sex online health support groups, and similarities in communication patterns in mixed-sex online health support groups. However, findings should be treated with caution due to the diversity in studies and methodological issues highlighted in the present review.

Price et al. [79]

Stroke Oceania, Asia, Europe, North America, Europe

Service configurations

Safety, Activity level and response times for thrombolysis treatment

Fifty-four studies included. Local service configuration provides less thrombolysis activity than wider collaborations and report as many protocol violations despite their simpler design. Local variations in population density, geography, experts and activity and safety data reporting formats makes it difficult to compare service configurations. Reporting of activity and safety in a standardised format is urged.

Stroke services should continue to publish thrombolysis activity and safety data in a recommended format in order to determine the most suitable configuration for different settings.

Deshpande et al. (stroke manage- ment) [80]

Stroke US, Canada, Germany, Italy, Netherlands, China

Acute medicine Cost/economic, Feasibility/pilot, Health, Percep- tion/satisfaction, Technology related

Twenty-two studies included (eight of high quality). Results showed improved access to thrombolysis, acceptable times between hospital arrival and thrombolysis, and decreased need to transfer patients across institutions. Mortality rates, and three and six month functional outcomes were comparable with those of face-to-face stroke care. Mortality rates were also similar. Patients and provider satisfaction was high, though not assessed in detail.

Although it is difficult to draw conclusions from this small sample of studies, the trend suggests that in post-stroke patients, telehealth led to improvements in caregivers’ mental health and high levels of patient satisfaction. There was minimal evidence regarding the impact on resource utilization.

7 6 6

in t

e r

n a

t io

n a

l jo

u r

n a

l o

f m

e d

ic a

l in

f o

r m

a t

ic s

7 9

( 2

0 1

0 )

7 3

6 –7

7 1

Table 6 (Continued )

Reference Conditions included

Geographic area

Service/ intervention

Outcome Authors’ summary of results Authors’ conclusions

Sood et al. [81]

All Not stated All Definitions of telemedicine

Hundred and four studies included, which generate 104 definitions. The authors identify four types of definition: (1) Medical: mention of “providing healthcare services”. (2) Technological: indication of technology’s role. (3) Spatial: “geographical separation of patient and doctor” pertains to or involves nature of space/distance. (4) Benefits: medical care is brought to people when it is not feasible to get people to medical care.

The article provides formative evaluation by classifying the different approaches to defining telemedicine and their underlying theoretical assumptions.

Pineau et al. (telepsychi- atry) [20]

Psychiatric conditions (adult and pediatric)

Not stated Focus on Canada and USA

Telepsychiatry Cost/economic, Ethical issues, Legal, Organizational, Technology related, clinical guidelines and technical standards

‘About 60′ studies included. The authors argue that definition of clinical guidelines and technological standards aimed at standardizing telepsychiatric practice will promote its large scale implementation.

The review concludes that telepsychiatry should be implemented in Québec and provides detailed clinical and technical guidelines for implementation. They add that taking into account human and organizational aspects plays a part in ensuring the success of this type of activity; that legal and ethical aspects must also be considered; and that a detailed economic analysis should be carried out prior to any large investment in telepsychiatry. Finally, implementation of psychiatry should be subjected to rigorous downstream assessment in order to improve management and performance.

Pineau et al. [82]

Multiple conditions

References in English. Focus on Canada and USA

Telerehabilitation Cost/economic, Ethical issues, Legal, Organizational, Technology related, clinical guidelines and technical standards

Unspecified number of studies included. The review offers clinical and technical guidelines for telerehabilitation implementation.

The review concludes that telerehabilitation should be implemented in Québec and provides detailed clinical and technical guidelines for implementation. They add that taking into account human and organizational aspects plays a part in ensuring the success of this type of activity; that legal and ethical aspects must also be considered; and that a detailed economic analysis should be carried out prior to any large investment in telerehabilitation. Finally, implementation of telerehabilitation should be subjected to rigorous downstream assessment in order to improve management and performance.

in t

e r

n a

t io

n a

l jo

u r

n a

l o

f m

e d

ic a

l in

f o

r m

a t

ic s

7 9

( 2

0 1

0 )

7 3

6 –7

7 1

7 6 7

Table 7 – Included reviews not cited in the paper.

Reference Conditions included

Geographic area

Service/ intervention

Outcome Authors’ summary of results Authors’ conclusions

Bonacina et al. [84]

Telecardiology No mention Systematic review (methodology paper)

Methodology development

Sixty-one studies included. Most literature related to pilot projects, feasibility studies and short-term outcomes. The analysis is used to develop a method of classifying scientific publications to assess the evidence they contain so that it can be used.

This is a methodological study suggesting a framework for evaluation of literature on telemedicine.

Christensen et al. [85]

Anxiety and depression

Not mentioned Open access websites which deliver cognitive and behavioural interventions

Behavioural outcomes

Twenty-three studies included. All were RCTs. Data included dropout and adherence, predictors of adherence and reasons for dropout. Relative to reported rates of dropout from open access sites, the study found that the rates of attrition in randomized controlled trials were lower, ranging from approximately 1–50%. Predictors of adherence included disease severity, treatment length, and chronicity. Few studies formally examined reasons for dropout, and most failed to use appropriate techniques to analyze missing data.

Dropout rates from randomized controlled trials of Web interventions are low relative to dropout from open access websites. The development of theoretical models of adherence is as important in the area of Internet intervention research as it is in the behavioural health literature.

Clarke and Thiyagara- jan [86]

None (technical)

Not described Evaluation Technology related

Forty-seven studies included. The analysis was unable to identify a ‘definitive, standards based telemedicine technical evaluation framework.

Work needs to be done to address the deficiency identified.

Demiris [87] Many conditions (also professional groups)

USA, UK, Scandinavia

Not about interventions – focus is development of electronic communities

Ethical issues, Policy, Defining features of a complex intervention

Forty-seven studies included. Virtual communities in health care cover a wide range of clinical specialties, technologies and stakeholders. They include peer-to-peer networks, virtual health care delivery and research teams. Ethical challenges include identity and deception, privacy and confidentiality. Technical issues include sociability and usability.

Virtual communities may empower patients and enhance coordination of care services; however, there is not sufficient systematic evidence of the effectiveness of virtual communities on clinical outcomes or patient empowerment. Researchers need to address issues, such as sample sizes and experimental design to further the research field in this domain.

7 6 8

in t

e r

n a

t io

n a

l jo

u r

n a

l o

f m

e d

ic a

l in

f o

r m

a t

ic s

7 9

( 2

0 1

0 )

7 3

6 –7

7 1

Table 7 (Continued )

Reference Conditions included

Geographic area

Service/ intervention

Outcome Authors’ summary of results Authors’ conclusions

Marziali et al. [91]

Older people with unspecified conditions

North America and Europe

Home telehealth (ethical issues)

Ethical issues Hundred and seven studies included. Studies show lack of information about ethics and professional practice. Where these issues are explored, this is done to a limited extent. Peer-reviewed work is more likely to consider these issues.

That studies of telemedicine have addressed issues of ethics and professional practice to a limited extent and that these are not well documented.

Oh et al. [92] All No mention Evaluation (definitions)

Organizational, Percep- tion/satisfaction, Policy, Technology related, Definitions of eHealth

Thousand two hundred and nine abstracts scanned, 430 citations reviewed, 1158 Google sites reviewed. 51 unique definitions were retrieved, showing a wide range of themes, but no clear consensus about the meaning of the term eHealth. Two universal themes (health and technology) were identified and six less general (commerce, activities, stakeholders, outcomes, place, and perspectives).

The widespread use of the term eHealth suggests that it is an important concept, and that there is a tacit understanding of its meaning. This compendium of proposed definitions may improve communication among the many individuals and organizations that use the term.

Scott et al. [93]

All Canada, USA, New Zealand, Europe

All (focus is on identifying agreed outcome indicators)

Behavioural, Cost/economic, Ethical issues, Health, Legal, Organizational, Percep- tion/satisfaction, Policy, Safety, Social, These were grouped

Two hundred and eighteen studies included. A key objective of the NTOIP was to identify and describe outcome indicators that had been used to evaluate telehealth projects for quality, access, acceptability and cost. The findings show that the situation is complex and confusing, but that recommendations can be made.

The study was not considered as an end in itself, but rather the beginning of a continuing process of telehealth evaluation and research in Canada and elsewhere. That is, the ongoing quest for identification of a small number of appropriate outcome indicators (as well as their related outcome measures and outcome tools), their consistent description and their consistent application in future evaluations. This initiative suggests that the importance of evaluation and of consistent use of indicators is understood.

a l i n

A

S i

r

i n t e r n a t i o n a l j o u r n a l o f m e d i c

ppendix B. Supplementary data

upplementary data associated with this article can be found, n the online version, at doi:10.1016/j.ijmedinf.2010.08.006.

e f e r e n c e s

[1] R. Roine, A. Ohinmaa, D. Hailey, Assessing telemedicine: a systematic review of the literature, CMAJ 165 (September (6)) (2001) 765–771.

[2] D. Hailey, R. Roine, A. Ohinmaa, Systematic review of evidence for the benefits of telemedicine, J. Telemed. Telecare 8 (Suppl. 1) (2002) 1–30.

[3] R. Currell, C. Urquhart, P. Wainwright, et al., Telemedicine Versus Face to Face Patient Care: Effects on Professional Practice and Health Outcomes, Cochrane Library The Cochrane Collaboration, 2002 [Oxford: Update Software] (Issue 3).

[4] W.R. Hersh, D.H. Hickam, S.M. Severance, T.L. Dana, K.P. Krages, M. Helfand, Telemedicine for the medicare population: update, Evid. Rep. Technol. Assess (Full Rep.) (February (131)) (2006) 1–41.

[5] P.S. Whitten, F.S. Mair, A. Haycox, C.R. May, T.L. Williams, S. Hellmich, Systematic review of cost effectiveness studies of telemedicine interventions, BMJ 324 (June (7351)) (2002) 1434–1437.

[6] G. Demiris, D. Tao, An analysis of the specialized literature in the field of telemedicine, J. Telemed. Telecare 11 (6) (2005) 316–319.

[7] P. Jennett, L. Affleck Hall, D. Hailey, A. Ohinmaa, C. Anderson, R. Thomas, et al., The socio-economic impact of telehealth: a systematic review, J. Telemed. Telecare 9 (6) (2003) 311.

[8] E. Johnsen, E. Breivik, R. Myrvang, F. Olsen, Benefits from Telemedicine in Norway: An Examination of Available Documentation, HØYKOM Report No. 2006: 1. HØYKOM Report Series, 2006, ISSN 1504-5048.

[9] E. Murphy, R. Dingwall, D. Greatbatch, S. Parker, P. Watson, Qualitative research methods in health technology assessment: a review of the literature, Health Technol. Assess. 2 (16) (1998), iii–ix, 1–274.

[10] W. Hersh, M. Helfand, J. Wallace, D. Kraemer, P. Patterson, S. Shapiro, et al., Clinical outcomes resulting from telemedicine interventions: a systematic review, BMC Med. Informatics Decis. Making 1 (1) (2001) 5.

[11] J. Barlow, S. Bayer, B. Castleton, R. Curry, Meeting government objectives for telecare in moving from local implementation to mainstream services, J. Telemed. Telecare 11 (Suppl. 1) (2005) 49–51.

[12] E. Ammenwerth, J. Brender, P. Nykanen, H.U. Prokosch, M. Rigby, J. Talmon, Visions and strategies to improve evaluation of health information systems. Reflections and lessons based on the HIS-EVAL workshop in Innsbruck, Int. J. Med. Inform. 73 (June (6)) (2004) 479–491.

[13] A.G. Ekeland, A. Bowes, S.A. Flottorp, Methodologies for assessing telemedicine: a systematic review of reviews (under review).

[14] R. Pawson, T. Greenhalgh, G. Harvey, K. Walshe, Realist review – a new method of systematic review designed for complex policy interventions, J. Health Serv. Res. Policy (July

(10 Suppl. 1)) (2005) 21–34.

[15] A. Barak, L. Hen, M.B.-N. Shapira, A comprehensive review and a meta-analysis of the effectiveness of Internet-based psychotherapeutic interventions, J. Technol. Hum. Services 26 (2–4) (2008) 109–114.

f o r m a t i c s 7 9 ( 2 0 1 0 ) 736–771 769

[16] R.A. Clark, S.C. Inglis, F.A. McAlister, J.G. Cleland, S. Stewart, Telemonitoring or structured telephone support programmes for patients with chronic heart failure: systematic review and meta-analysis, BMJ 334 (7600) (2007 May 5) 942.

[17] M. Jaana, G. Pare, C. Sicotte, Home telemonitoring for respiratory conditions: a systematic review, Am. J. Manag. Care 15 (May (5)) (2009) 313–320.

[18] S.K. Myung, D.D. McDonnell, G. Kazinets, H.G. Seo, J.M. Moskowitz, Effects of Web- and computer-based smoking cessation programs: meta-analysis of randomized controlled trials, Arch. Intern. Med. 169 (May (10)) (2009) 929–937.

[19] L. Neubeck, J. Redfern, R. Fernandez, T. Briffa, A. Bauman, S.B. Freedman, Telehealth interventions for the secondary prevention of coronary heart disease: a systematic review, Eur. J. Cardiovasc. Prev. Rehabil. 16 (June (3)) (2009) 281–289.

[20] G.M.K. Pineau, C. St-Hilaire, R. Perreault, E. Levac, B. Hamel, Telehealth: Clinical Guidelines and Technical Standards for Telepsychiatry, Agence d’Évaluation des Technologies et des Modes D’intervention en Santé, Montreal, Quebec, 2006.

[21] M.B. Powers, P.M. Emmelkamp, Virtual reality exposure therapy for anxiety disorders: a meta-analysis, J. Anxiety Disord. 22 (3) (2008) 561–569.

[22] G.B. Prange, M.J. Jannink, C.G. Groothuis-Oudshoorn, H.J. Hermens, M.J. Ijzerman, Systematic review of the effect of robot-aided therapy on recovery of the hemiparetic arm after stroke, J. Rehabil. Res. Dev. 43 (March–April (2)) (2006) 171–184.

[23] M.A. Reger, G.A. Gahm, A meta-analysis of the effects of internet- and computer-based cognitive-behavioral treatments for anxiety, J. Clin. Psychol. 65 (January (1)) (2009) 53–75.

[24] V. Spek, P. Cuijpers, I. Nyklicek, H. Riper, J. Keyzer, V. Pop, Internet-based cognitive behaviour therapy for symptoms of depression and anxiety: a meta-analysis, Psychol. Med. 37 (March (3)) (2007) 319–328.

[25] K. Tran, J. Polisena, D. Coyle, Home Telehealth for Chronic Disease Management, Canadian Agency for Drugs and Technologies in Health, 2008 (113).

[26] M.H. van den Berg, J.W. Schoones, T.P. Vliet Vlieland, Internet-based physical activity interventions: a systematic review of the literature, J. Med. Internet Res. 9 (3) (2007) e26.

[27] S.E. Hyler, D.P. Gangure, S.T. Batchelder, Can telepsychiatry replace in-person psychiatric assessments? A review and meta-analysis of comparison studies, CNS Spectr. 10 (May (5)) (2005) 403–413.

[28] J. Barlow, D. Singh, S. Bayer, R. Curry, A systematic review of the benefits of home telecare for frail elderly people and those with long-term conditions, J. Telemed. Telecare 13 (4) (2007) 172–179.

[29] K.L. Bussey-Smith, R.D. Rossen, A systematic review of randomized control trials evaluating the effectiveness of interactive computerized asthma patient education programs, Ann. Allergy Asthma Immunol. 98 (June (6)) (2007) 507–516, quiz 16, 66.

[30] M. Jaana, G. Pare, Home telemonitoring of patients with diabetes: a systematic assessment of observed effects, J. Eval. Clin. Pract. 13 (April (2)) (2007) 242–253.

[31] P. Azarmina, P. Wallace, Remote interpretation in medical encounters: a systematic review, J. Telemed. Telecare 11 (3) (2005) 140–145.

[32] G. Demiris, B. Hensel, Technologies for an Aging Society: A Systematic Review of “Smart Home” applications. Yearbook of medical informatics, 2008, p. 33.

[33] A. Martinez, E. Everss, J.L. Rojo-Alvarez, D.P. Figal, A. Garcia-Alberola, A systematic review of the literature on home monitoring for patients with heart failure, J. Telemed. Telecare 12 (5) (2006) 234–241.

i c a l

770 i n t e r n a t i o n a l j o u r n a l o f m e d

[34] R. Gaikwad, J. Warren, The role of home-based information and communications technology interventions in chronic disease management: a systematic literature review, Health Informatics J. 15 (June (2)) (2009) 122–146.

[35] P. Cuijpers, A. van Straten, G. Andersson, Internet-administered cognitive behavior therapy for health problems: a systematic review, J. Behav. Med. 31 (April (2)) (2008) 169–177.

[36] P.E. Bee, P. Bower, K. Lovell, S. Gilbody, D. Richards, L. Gask, et al., Psychotherapy mediated by remote communication technologies: a meta-analytic review, BMC Psychiatry 8 (2008) 60.

[37] J.H. Crosbie, S. Lennon, J.R. Basford, S.M. McDonough, Virtual reality in stroke rehabilitation: still more virtual than real, Disabil. Rehabil. 29 (July (14)) (2007) 1139–1146, discussion 47–52.

[38] A. Henderson, N. Korner-Bitensky, M. Levin, Virtual reality in stroke rehabilitation: a systematic review of its effectiveness for upper limb motor recovery, Top. Stroke Rehabil. 14 (March–April (2)) (2007) 52–61.

[39] K.M. Griffiths, H. Christensen, Review of randomized controlled trials of Internet intervention for mental disorders and related conditions, Clin. Psychol. 10 (1) (2006) 16–29.

[40] C.L. Jackson, S. Bolen, F.L. Brancati, M.L. Batts-Turner, T.L. Gary, A systematic review of interactive computer-assisted technology in diabetes care. Interactive information technology in diabetes care, J. Gen. Intern. Med. 21 (February (2)) (2006) 105–110.

[41] P.K. Weinstein, A review of weight loss programs delivered via the Internet, J. Cardiovasc. Nurs. 21 (July–August (4)) (2006) 251–258, quiz 9–60.

[42] A. Deshpande, S. Khoja, J. Lorca, A. McKibbon, C. Rizo, A.R. Jadad, Asynchronous Telehealth: Systematic Review of Analytic Studies and Environmental Scan of Relevant Initiatives, Canadian Agency for Drugs and Technologies in Health, Ottawa, 2008.

[43] J. Polisena, K. Tran, K. Cimon, B. Hutton, S. McGill, K. Palmer, Home telehealth for diabetes management: a systematic review and meta-analysis, Diabetes Obes. Metab. 11 (October (10)) (2009) 913–930.

[44] D. Hailey, M.-J. Paquin, O. Maciejewski, L. Harris, A. Casebeer, G. Fick, et al., The Use and Benefits of Teleoncology, The Institute of Health Economics (IHE), Canada, 2007, January.

[45] G. Pare, M. Jaana, C. Sicotte, Systematic review of home telemonitoring for chronic diseases: the evidence base, J. Am. Med. Inform. Assoc. 14 (May–June (3)) (2007) 269–277.

[46] B. Nannings, A. Abu-Hanna, Decision support telemedicine systems: a conceptual model and reusable templates, Telemed. J. E Health 12 (December (6)) (2006) 644–654.

[47] V. Sintchenko, F. Magrabi, S. Tipper, Are we measuring the right end-points? Variables that affect the impact of computerised decision support on patient outcomes: a systematic review, Med. Inform. Internet Med. 32 (September (3)) (2007) 225–240.

[48] O. Wu, P. Langhorne, The challenge of acute-stroke management: does telemedicine offer a solution? Int. J. Stroke 1 (November (4)) (2006) 201–207.

[49] K.G. Shojania, A. Jennings, A. Mayhew, C.R. Ramsay, M.P. Eccles, J. Grimshaw, The effects of on-screen, point of care computer reminders on processes and outcomes of care, Cochrane Database Syst. Rev. (3) (2009) CD001096.

[50] S.I. Chaudhry, C.O. Phillips, S.S. Stewart, B. Riegel, J.A. Mattera, A.F. Jerant, et al., Telemonitoring for patients with

chronic heart failure: a systematic review, J. Card. Fail. 13 (February (1)) (2007) 56–62.

[51] A. Farmer, O.J. Gibson, L. Tarassenko, A. Neil, A systematic review of telemedicine interventions to support blood

i n f o r m a t i c s 7 9 ( 2 0 1 0 ) 736–771

glucose self-monitoring in diabetes, Diabet. Med. 22 (October (10)) (2005) 1372–1378.

[52] D.L. Sanders, D. Aronsky, Biomedical informatics applications for asthma care: a systematic review, J. Am. Med. Inform. Assoc. 13 (July–August (4)) (2006) 418–427.

[53] F. Verhoeven, L. van Gemert-Pijnen, K. Dijkstra, N. Nijland, E. Seydel, M. Steehouder, The contribution of teleconsultation and videoconferencing to diabetes care: a systematic literature review, J. Med. Internet Res. 9 (5) (2007) e37.

[54] B.M. Bewick, K. Trusler, M. Barkham, A.J. Hill, J. Cahill, B. Mulhern, The effectiveness of web-based interventions designed to decrease alcohol consumption – a systematic review, Prev. Med. 47 (July (1)) (2008) 17–26.

[55] B. Maric, A. Kaan, A. Ignaszewski, S.A. Lear, A systematic review of telemonitoring technologies in heart failure, Eur. J. Heart Fail. 11 (May (5)) (2009) 506–517.

[56] M.G. Postel, H.A. de Haan, C.A. De Jong, E-therapy for mental health problems: a systematic review, Telemed. J. E Health. 14 (September (7)) (2008) 707–714.

[57] S. Martin, G. Kelly, W.G. Kernohan, B. McCreight, C. Nugent, Smart home technologies for health and social care support, Cochrane Database Syst. Rev. (4) (2008) CD006412.

[58] J. Powell, T. Chiu, G. Eysenbach, A systematic review of networked technologies supporting carers of people with dementia, J. Telemed. Telecare 14 (3) (2008) 154–156.

[59] F. Garcia-Lizana, A. Sarria-Santamera, New technologies for chronic disease management and control: a systematic review, J. Telemed. Telecare 13 (2) (2007) 62–68.

[60] M.P. Gagnon, F. Legare, M. Labrecque, P. Fremont, P. Pluye, J. Gagnon, et al., Interventions for promoting information and communication technologies adoption in healthcare professionals, Cochrane Database Syst. Rev. (1) (2009) CD006093.

[61] W. Hersh, D. Hickam, S. Severance, T. Dana, K. Krages, M. Helfand, Diagnosis, access and outcomes: update, J. Telemed. Telecare 12 (2) (2006) 3–31.

[62] S. Linton, Datorbaserad kognitiv beteendeterapi vid ångestsyndrom eller depression, 2007.

[63] A. Mathur, J.C. Kvedar, A.J. Watson, Connected health: a new framework for evaluation of communication technology use in care improvement strategies for type 2 diabetes, Curr. Diabetes Rev. 3 (November (4)) (2007) 229–234.

[64] S.T. Walters, J.A. Wright, R. Shegog, A review of computer and Internet-based interventions for smoking behavior, Addict. Behav. 31 (February (2)) (2006) 264–277.

[65] T.P. Lintonen, A.I. Konu, D. Seedhouse, Information technology in health promotion, Health Educ. Res. 23 (June (3)) (2008) 560–566.

[66] J. van Nooten, H. Oh, B. Pierce, F.J. Koning, A.R. Jadad, Spiritual care as eHealth: a systematic review, J. Pastoral. Care Counsel. 60 (Winter (4)) (2006) 387–394.

[67] S.V. Rojas, M.P. Gagnon, A systematic review of the key indicators for assessing telehomecare cost-effectiveness, Telemed. J. E Health 14 (November (9)) (2008) 896–904.

[68] E. Seto, Cost comparison between telemonitoring and usual care of heart failure: a systematic review, Telemed. J. E Health 14 (September (7)) (2008) 679–686.

[69] A. Deshpande, S. Khoja, A. McKibbon, A.R. Jadad, Real-Time (synchrounous) Telehealth in Primary Care: Systematic Review of Systematic Reviews, 2008.

[70] E. Kaltenthaler, J. Brazier, E. De Nigris, I. Tumur, M. Ferriter, C. Beverley, et al., Computerised cognitive behaviour therapy for depression and anxiety update: a systematic review and

economic evaluation, Health Technol. Assess. 10 (September (33)) (2006), iii, xi–xiv, 1–168.

[71] J. Polisena, D. Coyle, K. Coyle, S. McGill, Home telehealth for chronic disease management: a systematic review and an

a l i n

i n t e r n a t i o n a l j o u r n a l o f m e d i c

analysis of economic evaluations, Int. J. Technol. Assess. Health Care 25 (July (3)) (2009) 339–349.

[72] F. Griffiths, A. Lindenmeyer, J. Powell, P. Lowe, M. Thorogood, Why are health care interventions delivered over the internet? A systematic review of the published literature, J. Med. Internet Res. 8 (2) (2006) e10.

[73] K.M. Akesson, B.I. Saveman, G. Nilsson, Health care consumers’ experiences of information communication technology – a summary of literature, Int. J. Med. Inform. 76 (September (9)) (2007) 633–645.

[74] D. Kairy, P. Lehoux, C. Vincent, M. Visintin, A systematic review of clinical outcomes, clinical process, healthcare utilization and costs associated with telerehabilitation, Disabil. Rehabil. 31 (6) (2009) 427–447.

[75] E. Murray, J. Burns, T.S. See, R. Lai, I. Nazareth, Interactive health communication applications for people with chronic disease, Cochrane Database Syst. Rev. (4) (2005) CD004274.

[76] S. Koch, Home telehealth-current state and future trends, Int. J. Med. Inform. 75 (August (8)) (2006) 565–576.

[77] S. Lauriks, A. Reinersmann, H.G. Van der Roest, F.J. Meiland, R.J. Davies, F. Moelaert, et al., Review of ICT-based services for identified unmet needs in people with dementia, Ageing Res. Rev. 6 (October (3)) (2007) 223–246.

[78] P.K. Mo, S.H. Malik, N.S. Coulson, Gender differences in computer-mediated communication: a systematic literature review of online health-related support groups, Patient Educ. Couns. 75 (April (1)) (2009) 16–24.

[79] C.I. Price, F. Clement, J. Gray, C. Donaldson, G.A. Ford, Systematic review of stroke thrombolysis service configuration, Expert Rev. Neurother. 9 (February (2)) (2009) 211–233.

[80] A. Deshpande, S. Khoja, A. McKibbon, C. Rizo, A.R. Jadad, Telehealth for Acute Stroke Management (Telestroke): Systematic Review of Analytic Studies and Environmental Scan of Relevant Initiatives, 2008.

[81] S. Sood, V. Mbarika, S. Jugoo, R. Dookhy, C.R. Doarn, N.

Prakash, et al., What is telemedicine? A collection of 104 peer-reviewed perspectives and theoretical underpinnings, Telemed. J. E Health 13 (October (5)) (2007) 573–590.

[82] G. Pineau, K. Moqadem, C. St Hilaire, R. Perreault, E. Levac, B. Hamel, et al., Telehealth: Clinical Guidelines and

f o r m a t i c s 7 9 ( 2 0 1 0 ) 736–771 771

Technological Standards for Telerehabilitation, Agence d’Evaluation des Technologies et des Modes d’Intervention en Sante (AETMIS), Montreal, 2006, No. 101.

[83] M. Azar, Gabbay R, Web-based management of diabetes through glucose uploads: has the time come for telemedicine? Diabetes Res. Clin. Pract. 83 (1) (2009) 9–17.

[84] S. Bonacina, L. Draghi, M. Masseroli, Pinciroli F, Understanding telecardiology success and pitfalls by a systematic review, Connect. Med. Informatics Bio-Informatics 116 (2005) 373–378.

[85] H. Christensen, K.M. Griffiths, L. Farrer, Adherence in internet interventions for anxiety and depression, J. Med. Internet Res. 11 (June (2)) (2009).

[86] M. Clarke, Thiyagarajan CA, A systematic review of technical evaluation in telemedicine systems, Telemed. J. E Health 14 (2) (2008) 170–183.

[87] G. Demiris, The diffusion of virtual communities in health care: concepts and challenges, Patient Educ. Couns. 62 (2) (2006) 178–188.

[88] J.C. Elliott, K.B. Carey, J Bolles, R Computer-based interventions for college drinking: a qualitative review, Addict. Behav. 33 (8) (2008) 994–1005.

[89] R. Holland, J. Battersby, I. Harvey, E. Lenaghan, J. Smith, L. Hay, Systematic review of multidisciplinary interventions in heart failure, Heart 91 (7) (2005) 899–906.

[90] M. Jaana, G. Pare, Sicotte C, Hypertension home telemonitoring: current evidence and recommendations for future studies, Dis. Manag. Health Out 15 (1) (2007) 19–31.

[91] E. Marziali, J.M.D. Serafini, McCleary L, A systematic review of practice standards and research ethics in technology-based home health care intervention programs for older adults, J. Aging Health 17 (6) (2005) 679–696.

[92] H. Oh, C. Rizo, M. Enkin, Jadad A, What is eHealth (3): a systematic review of published definitions’, J. Med. Internet Res. 7 (1) (2005) e1.

[93] R.E. Scott, F.G. McCarthy, P.A. Jennett, T. Perverseff, D. Lorenzetti, A. Saeed, et al., Telehealth outcomes: a synthesis of the literature and recommendations for outcome indicators, J. Telemed. Telecare 13 (Suppl. 2) (2007) 1–38.