healthcare

210380566_1.pdf

Evaluating telemedicine: A focus on patient pathways

Jane Coughlan, Julie Eatock, Tillal Eldabi Brunel University

Evaluations of telemedicine have sought to assess various measures of effectiveness (e.g., diagnostic accuracy), efficiency (e.g., cost), and engagement (e.g., patient satisfaction) to determine its success. Few studies, however, have looked at evaluating the organizational impact of telemedicine, which involves technology and process changes that affect the way that it is used and accepted by patients and clinicians alike. This study reviews and discusses the conceptual issues in telemedicine research and proposes a fresh approach for evaluating telemedicine. First, we advance a patient pathway perspective, as most of the existing studies view telemedicine as a support to a singular rather than multiple aspects of a health care process. Second, to conceptualize patient pathways and understand how telemedicine impacts upon them, we propose simulation as a tool to enhance understanding of the traditional and telemedicine patient pathway.

Keywords: Clinical pathways, Computer simulation, Evaluation, Telemedicine, Patients

The telemedicine evaluation literature has grown substan- tially from the advancement of a specific framework for assessing telemedicine by the Institute of Medicine (IOM) (16). This early report identified five dimensions important to evaluating telemedicine: quality, access, cost, patient per- ceptions, and clinician perceptions. However, a more recent report by The Lewin Group (27) both confirmed and extended these evaluation dimensions. It considered the properties of these dimensions, in terms of measures and their impacts, but also the methodology issues involved in evaluation. This di- rectly responded to some disquiet expressed in the literature over rigor and consistency that limited the generalizability of some studies’ findings. In light of these concerns, we discuss two possible improvements for telemedicine evaluation.

First, we argue that the focus of the evaluation itself should be widened to look at telemedicine in the context of the patient pathway (also known in the literature as the clinical pathway) to understand its place along the patient’s journey through the health service. Second, we put forward simulation as a tool for evaluating telemedicine through its representation of the patient pathway. Simulation will be discussed as a viable methodology for addressing some of the weaknesses documented in telemedicine evaluation, through a review of the measures and methodologies used in the assessment of telemedicine. The discussion of the potential

benefits of simulating patient pathways is supported by an illustration—using leg ulcer sufferers as a case example—to contribute to an understanding of care delivery by traditional and telemedicine processes.

TELEMEDICINE EVALUATION ISSUES

Measures

Reviews of telemedicine evaluation are limited (1;12;25;27), but those that do exist provide important overviews as to the status of evaluations in terms of the measures and method- ology used to assess telemedicine. Most evaluations have sought to assess various quantitative measures of effective- ness (e.g., diagnostic accuracy), efficiency (e.g., cost), and engagement (e.g., patient satisfaction) to determine its suc- cess. However, these studies have tended to focus on single clinical contexts, specialties, and measures. To highlight the problematic issues around the measures currently assessed in telemedicine evaluation, we will briefly focus on three key evaluation measures: (i) Diagnostic accuracy—this measure has tended to overly dominate many studies’ outcomes (1); (ii) Cost (and its associated variables, e.g., benefit, utility, and so on)—many studies have equated a cost-saving as a benefit, but with no reference to how it affects clinical out- comes (28); and (iii) Patient satisfaction—this measure is the

136

Evaluating telemedicine

Table 1. Key Evaluation Methodology Issues for Telemedicine

Methodological Approach

Evaluation methodology issue Current telemedicine evaluations Simulation

Technological maturity Progress of technology through its

lifecycle and the stage of evaluation Often carried out as single case studies

of performance at too early or late a stage and can often produce unduly positive or negative findings (26)

Evaluates telemedicine along the continuum of maturity from immature prototypes to fully matured working systems

Focus of evaluation Scope of the evaluation from the

technology itself to its broader context

The focus is predominantly placed on the specifics of the technology itself, as opposed to its organizational impact (2).

Offers a more holistic approach, in analysing the processes of care into which telemedicine is situated, along with key evaluation measures

Perspective of evaluation Standpoint from which the benefits

of the technology are realized A single perspective analysis is most

common, often to the point of exclusion of the impact from an alternative perspective (21)

Provides a multi-perspective analysis, (depending on the model’s variables), e.g., to reflect a patient/clinician view

Comparator Definition of a suitable control group

with which to gauge the effect of the technology’s intervention

Patients who are treated with and without telemedicine and then compared, are often not of a similar level (8)

Allows a consistent way of performing like-with-like comparisons as data from a single set of patients can be run in both the traditional and telemedicine models

Randomization Assigning of participants to

experimental and control groups on a random basis

This process is difficult to achieve as the sample group in a study are generally quite small, in some published cases as few as ten patients (20)

Avoids randomization issues because a single set of patients can be run in both the traditional and telemedicine models

Time horizon Duration of data collection in a study The focus has been on short-term pilot

projects, with a lack of follow-up (5) Predicts future outcomes, applying the

analysis to gauge the long-term effects of telemedicine implementation

most common evaluation undertaken, tends to produce con- sistently positive results, but can often be misleading, as patient satisfaction measures often fail to go beyond first impressions (19).

A major reason for some of the limitations cited in the research is that few high-quality studies exist (15). Despite the usefulness of the clinical and economic data that have been produced, the methodological paucity in the research has somewhat undermined the value of the assessments un- dertaken. This finding has resulted in calls for patients and practitioners alike to remain skeptical over the professed benefits of telemedicine compared with traditional face-to- face patient care (12). In an attempt to manage some of the methodological issues in telemedicine evaluation, simulation is proposed as a potentially useful tool for producing more robust findings.

Methodology

As a methodological approach, simulation revolves around creating computer models of social structures and processes. These models are subject to “simulation” that is experimen- tation through the manipulation of variables (e.g., time and cost) to understand the behavior of the model and evaluate

the extent to which it provides an accurate account of the behavior of the observed system (13). In health care, simu- lation has achieved some success as a problem-solving tool (3). Moreover, as well as being able to incorporate the afore- mentioned evaluation measures, it will be posited that simu- lation modeling offers a systematic approach for addressing key evaluation methodology issues that have been usefully summarized by the seminal Lewin Group report (27). These issues have been identified as technological maturity, focus of evaluation, perspective of evaluation, comparator, random- ization, and time horizon. Table 1 defines each of these issues in turn, along with an example of how they are addressed in current telemedicine evaluation studies and how simulation can offer a potential solution to these challenging issues.

From Table 1, it can be seen that current telemedicine evaluations largely fail to address the broader organizational, clinical, and social processes that new technology impacts upon. Some evidence that exists suggests that this is a crit- ical issue and that the focus of the evaluation may be too narrow. For example, Lehoux et al. (17) found that the use of telemedicine did not fit into clinicians’ communication routines of consultation and referral. We suggest that a more fruitful direction for the evaluation of telemedicine is to fo- cus on the patient pathway. The patient pathway includes all

INTL. J. OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 22:1, 2006 137

Coughlan et al.

the clinical routines (or processes) into which telemedicine is placed, so that it can be assessed on how it sustains or supports variation in clinical practice.

SIMULATING THE PATIENT PATHWAY

Patient pathways are tools that assist in providing general guidelines of care for dealing with individuals and groups of patients suffering from a wide variety of diseases. How- ever, the majority of studies focused on traditional patient pathways (23). The introduction of telemedicine, however, offers a new pathway to care, although the impacts of it are even less well understood. To this end, simulation is put forward as a method of modeling pathways that cap- ture the timeline of care from the start to the journey’s end.

Discrete-Event Simulation Technique

Discrete-event simulation (DES) offers many features to cope with understanding the complex nature of health care sys- tems, of which the patient pathway is a clear example. There are three critical steps to the technique that are advanta- geous for representing pathways, which can be described as follows: (i) Understanding the system or process to be modeled—this is in terms of its main entities (e.g., patients), events (e.g., clinic visits), and decisions (e.g., referral of patient for consultation) and must be achieved for the sub- sequent model building to have a good representative basis; (ii) Changing the parameters of the model (e.g., time and cost)—this step can suggest (based on mathematical distri- butions) the optimum capacity of the system in the present and for the future given different scenarios; and (iii) Under- standing the inter-relationships between different entities, events, and decisions in the system—this step can identify the interdependencies of variables and the effect of changing one has upon another.

The simulation’s significant feature is its capability of performing “what if. . . ” type analyses through the manip- ulation of variables to understand the inter-relationships within the model and, hence, the real system. This iterative nature to the modeling process brings about the identification of the optimum system setup. Although simulation is not intended to replace current designs of evaluation studies, acceptance of any new approach to studying health care problems needs to be justified on the professed benefits of the proposed solution option.

Benefits of Simulating the Patient Pathway

Whereas simulation has the potential to overcome many of the problematic issues in telemedicine evaluation (as shown in Table 1), the true value of the approach rarely has been realized, given the narrow focus to which it has been applied previously, typically hospital scheduling problems (11). This quantitative view of simulation modeling—as a way of calcu-

lating outcomes—has often failed to produce results that can be readily implemented in real-life applications. Moreover, the telemedicine literature, in particular, reveals a paucity of studies that have undertaken any simulation, except for perhaps two notable examples (6;18). The impetus, there- fore, clearly exists for a debate on the conceptual issues of research in telemedicine, given the problems with current evaluation techniques (4). Robinson (24) has called for a de- bate on simulation study as a mode of practice in various domains. We contend that this debate is necessary within health care and propose that a starting point for this discus- sion is on evaluating telemedicine from a simulation of the patient pathway.

Patient pathways do not physically exist; therefore, methods of computerizing pathways have demonstrated some degree of success (7). The understanding that is gained through simulation is of a greater value than the pure numer- ical values produced. We propose, therefore, that simulation be viewed as a tool not to calculate outcomes but to appre- ciate them. This difference is subtle yet powerful. In this manner, the use of simulation will crucially serve to elicit the intangibles, such as insight into the way the system actu- ally operates, understanding the variables that can affect the system, and informing decisions concerning the system and their possible consequences.

The benefits of simulating patient pathways are in- creased when qualitative investigations (e.g., interviews, ob- servations, and so on) are directed at critical points along the pathway so as to supplement the models and understand more holistically the relationship between the interpersonal (e.g., patient satisfaction) and technical aspects of telemedicine (9). For example, to return to the three key measures discussed earlier, in diagnostic accuracy, a control patient group can be simulated to compare the outcomes of consultations with a clinical trial group. This approach has been shown previously to provide an educational benefit for informing clinical deci- sion making (14). Furthermore, cost-effectiveness measures can also be extended to produce cost per quality measures of outcome in terms of the quality adjusted life years for specific health care interventions (10). This can have an im- portant personal benefit, particularly for the patient in terms of establishing the relationship between their illness and the likelihood of health care saving their life. Moreover, incorpo- rating patient satisfaction measures is possible by converting patient responses into an appropriate numerical scale and ap- plying these figures to the model. This strategy can have a behavioral benefit in being able to determine fluctuations in patient satisfaction and pinpointing problem areas.

Telemedicine Patient Pathway: An Illustration

To illustrate the potential of simulating patient pathways, leg ulcer sufferers were selected as case examples, given the access to patients who have followed a traditional

138 INTL. J. OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 22:1, 2006

Evaluating telemedicine

start

Patient visits GP

Decide on tests to be performed

More tests required

Referral required?

Refer to consultant

yes

Visit consultant

Blood test at clinic Doppler test at

clinic Photographs at

clinic yes

Repeat visits to GP

no Ulcer healed?

stop

yes

Traditional

Telelink consultation

telelink

Tests

Process continues beyond this point

Telelink or traditional?

Figure 1. Leg ulcer patient pathway.

and telemedicine patient pathway. Figure 1 is a graphic representation of part of a leg ulcer patient pathway. Informa- tion to structure the pathway and its critical variables is col- lected from patient records and interviews with doctors and nurses. Figure 1 serves to illustrate all the health care events along this section of the pathway, for instance a tele-link

consultation and the relationships between them. Figure 2 illustrates how identical patients can be treated through two systems, both traditional and telemedicine (the timescale for which is provided for illustrative purposes only).

Variables at strategic decision points along the path- way control the flow of patients. These variables alter based

INTL. J. OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 22:1, 2006 139

Coughlan et al.

Week No. 0

Present to GP

Referral to consultant

Telelink to consultant

Pre-surgery tests

Present to GP

Tests at clinic

Referral to consultant

surgery

In-person visit to cons.

Pre-surgery tests

Tests at clinic

PRESENT TO GP 1. Patient volume – the number of patients arriving at the surgery are established in order to determine the capacity that the system currently deals with and how this affects the system.

2. Patient attributes – age, severity of condition, and general health were key issues used in clinical decision-making. These were all rated on a scale of 1-3 (with 1 being the youngest/least severe and 3 being the oldest/most severe).

TESTS AT CLINIC (and PRE-SURGERY TESTS) 1. Costs of tests – this variable is dependent on patient attributes (i.e., the more severe cases will need more tests). It is also dependent on time as if the period in between testing at the clinic and pre-surgery is short then only one set of tests will be needed, otherwise they may have to be repeated.

2. Costs of dressings – this variable is dependent on time, as dressings have to be changed twice a week, during the course of treatment.

3. Time – every event has a time delay, which may vary depending on patient attributes, whether telemedicine is involved, or the number of tests conducted.

Figure 2. Comparison of a traditional and telemedicine leg ulcer patient pathway.

on data parameters (patient volume), patient attributes (age, severity of condition, and general health), and time, which were deemed important for the treatment of leg ulcers from interviews with clinical staff. These variables can be further manipulated to determine different levels of effect when changing sections of the process (i.e., introducing telemedicine).

DISCUSSION AND CONCLUSIONS

This study has discussed simulation modeling—solely as a mode of practice—for the evaluation of telemedicine in the context of the patient pathway, as illustrated by Figures 1 and 2. Future work will need to computerize the pathway into a dynamic running simulation model by comparing the

140 INTL. J. OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 22:1, 2006

Evaluating telemedicine

predicted flow of patients with the actual flow along the traditional and telemedicine pathway. This approach will allow the capture of the long-term and organizational im- pacts of telemedicine implementation. An integral part of the discussion on the potential use of simulation has been the focus on the patient pathway. This focus has provided a com- pelling example of the importance of evaluating telemedicine in the context of the health care processes into which it is placed. A spotlight on pathways vivifies the collective bene- fits of simulation that have been put forward by highlighting how telemedicine can be evaluated in relation to the entities, events, and decisions involved in the delivery of care that it impacts upon. On the surface, telemedicine might appear to offer a fast-track system for patients, which can reduce the costs of dressing for leg ulcer patients, for example (as shown in Figure 2). However, Phipps (22) provides a caveat to this in stating that it is important that we understand how optimizing one section of a process (e.g., introducing a tele- link) can affect another section further along in the system and potentially induce a bottleneck (e.g., at the surgery stage) given that this section of the process will not have changed. It is suggested that a new challenge for telemedicine eval- uation studies is to compile a more comprehensive view of the technology, in looking across multiple aspects of health- care processes to provide a much-needed commentary on the outcomes of health care and its delivery. To this end, pa- tient pathway simulation has been introduced as a potential evaluation tool for telemedicine to continually monitor clini- cal practice, the effects of telemedicine, and changing health outcomes.

CONTACT INFORMATION

Jane Coughlan, PhD (Jane-Lisa.Coughlan@brunel.ac.uk), Research Fellow, Julie Eatock, PhD (Julie.Eatock@brunel. ac.uk), Research Fellow; Tillal Eldabi, PhD (Tillal.Eldabi@ brunel.ac.uk), Lecturer, Information Systems and Comput- ing, School of Information Systems, Computing and Math- ematics, Brunel University, Kingston Lane, Uxbridge, Mid- dlesex UB8 3PH, UK

REFERENCES

1. Aoki N, Dunn K, Johnson-Throop KA, Turley JP. Outcomes and methods in telemedicine evaluation. Telemed J E Health. 2003;9:393-401.

2. Bangert D, Doktor R, Warren J. Evaluating the organizational impact of telemedicine for project akamai. In: Proceedings of the 32nd Hawaii International Conference on System Sciences (HICSS’99); 1999. CD-ROM Version.

3. Barnes CD, Quiason JL, Benson C, McGuiness D. Success stories in simulation in health care. In: Proceedings of the 1997 Winter Simulation Conference; 1997:1280-1285.

4. Bashshur RL, Reardon TG, Shannon GW. Telemedicine: A new health care delivery system. Annu Rev Public Health. 2000;21:613-637.

5. Bishop JE, O’Reilly RL, Maddox K, Hutchinson LJ. Client satisfaction in a feasibility study comparing face-to-face in- terviews with telepsychiatry. J Telemed Telecare. 2002;8:217- 221.

6. Cameron AE, Bashshur RL, Halbritter K, Johnson EM, Cameron JW. Simulation methodology for estimating financial effects of telemedicine in West Virginia. Telemed J. 1998;4:125- 144.

7. Chu S, Cesnik B. Improving clinical pathway design: Lessons learned from a computerised prototype. Int J Med Inf. 1998;51:1-11.

8. Craig JJ, McConville JP, Patterson VH, Wootton R. Neuro- logical examination is possible using telemedicine. J Telemed Telecare. 1999;5:177-181.

9. Eldabi T, Irani Z, Paul RJ, Love PED. Quantitative and quali- tative decision-making methods in simulation modelling. Man- agement Decis. 2002;40:64-73.

10. Eldabi TA, Paul RJ, Taylor SJE. Simulating economic factors in adjuvant breast cancer treatment. J Oper Res Soc. 2000;51:465- 475.

11. Fone D, Hollinghurst S, Temple M, et al. Systematic review of the use and value of computer simulation modelling in pop- ulation health and health care delivery. J Public Health Med. 2003;25:325-335.

12. Hailey D, Roine R, Ohinmaa A. Systematic review of evidence for the benefits of telemedicine. J Telemed Telecare. 2002;8:1- 30.

13. Hanneman R, Patrick S. On the uses of computer-assisted simulation modeling in the social sciences. Sociological Re- search Online. Available at: http://www.socresonline.org.uk/ 2/2/5.html. Accessed December 13, 2005.

14. Hayes WS, Tohme WG, Komo D, et al. A telemedicine con- sultative service for the evaluation of patients with urolithiasis. Urology. 1998;51:39-43.

15. Hersh WR, Helfand M, Wallace J, et al. Clinical outcomes re- sulting from telemedicine interventions: A systematic review. BMC Medical Informatics and Decision Making. Available at: http://www.biomedcentral.com/1472-6947/1/5. Accessed December 13, 2005.

16. Institute of Medicine. Telemedicine: A guide to assessing telecommunications in health care. Washington, DC: National Academy Press; 1996.

17. Lehoux P, Sicotte C, Denis J-L, Berg M, Lacroix A. The theory of use behind telemedicine: How compatible with physicians’ clinical routines? Soc Sci Med. 2002;54:889- 904.

18. Loane M, Wootton R. A simulation model for analysing patient activity in dermatology. J Telemed Telecare. 2001;7 (Suppl 1):23-25.

19. Mair F, Whitten P. Systematic review of studies of patient satisfaction with telemedicine. Br Med J. 2000;320:1517- 1520.

20. Miyasaka K, Suzuki Y, Sakai H, Kondo Y. Interactive communication in high-technology home care: Videophones for pediatric ventilatory care. Pediatrics. Available at: http:// pediatrics.aappublications.org/cgi/reprint/99/1/e1. Accessed December 13, 2005.

21. Nordal EJ, Moseng D, Kvammen B, Lochen M-L. A compara- tive study of teleconsultations versus face-to-face consultations. J Telemed Telecare. 2001;7:257-265.

INTL. J. OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 22:1, 2006 141

Coughlan et al.

22. Phipps B. Hitting the bottleneck. Health Management Maga- zine. 1999:1-3.

23. Renholm M, Leino-Kilpi H, Suominen T. Critical pathways: A systematic review. J Nurs Adm. 2002;32:196-202.

24. Robinson S. Modes of simulation practice: Approaches to business and military simulation. Simulat Pract Theory. 2002;10:513-523.

25. Roine R, Ohinmaa A, Hailey D. Assessing telemedicine: A systematic review of the literature. Can Med Assoc J. 2001;165:765-771.

26. Sicotte C, Lehoux P. Teleconsultation: Rejected and emerging uses. Methods Inf Med. 2003;4:451-457.

27. The Lewin Group. Assessment of approaches to evaluating telemedicine, final report to the office of the Assistant Secretary for Planning and Evaluation. Department of Health and Hu- man Services, Contract Number HHS-10-97-0012, December, 2000.

28. Whitten PS, Mair F, Haycox A, et al. Systematic review of cost effectiveness studies of telemedicine interventions. Br Med J. 2002;324:1434-1437.

142 INTL. J. OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 22:1, 2006

Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.