Annotated Bibliograpgy

348

TELEMEDICINE AND e-HEALTH Volume 11, Number 3, 2005 © Mary Ann Liebert, Inc.

Research Findings and Strategies for Assessing Telemedicine Costs

TIM REARDON, Ph.D.

ABSTRACT

With varying degrees of enthusiasm, researchers and decision-makers support the use of telemedicine. Forms of telemedicine are appearing in health-care delivery, and are often in- tegral to transforming health-care information technology. Despite this, the appropriate role of telemedicine in the delivery process remains ambiguous, at least partly because of its un- certain impact on costs. Cost savings and benefits are often suggested by the logic of its im- pact on health care and by the promise of technology, but definitive information on the costs and benefits remain elusive. The objectives of this paper are to review the state of telemed- icine cost research, to examine major issues affecting the yield from this research, and finally to recommend strategies for improving future research. As this paper demonstrates, the pro- ductivity of telemedicine cost studies suffers from an under-utilization of appropriate pro- gram evaluation and economic methods. This review of telemedicine cost literature will ap- praise telemedicine cost studies and their findings within a broad analytic framework. Telemedicine cost studies will be assessed on their methods of statistical inference, use of critical economic concepts, and contextual definition for the determination of costs and ben- efits.

TG Reardon & Associates, LLC, Ann Arbor, Michigan.

OBJECTIVES AND BACKGROUND

RESEARCHERS HAVE YET TO ESTABLISH either asufficient body of findings or a vision for understanding telemedicine costs. Systematic literature reviews report a few useful cost find- ings along with many inadequate methods of inference. Most notable is the absence of randomized, controlled trials. But even among studies meeting minimal methodological crite- ria, studies suffer from poor use of analytic techniques in economics and inadequate docu- mentation.

Difficulties in cost analysis have been attrib- uted to problems in evaluating new programs,

low levels of use, and rapidly changing tech- nology. A few authors have called into ques- tion the usefulness of traditional tools of eco- nomic evaluation and have proposed new conceptual frameworks.1 Despite these ap- praisals, there are no detailed assessments or proposed strategies for addressing the low pro- ductivity of telemedicine cost analysis, whether from inadequate use of theory, the difficult na- ture of applying economics in this field, or lim- ited expertise in the requisite methodology.

Clinical research design and statistical infer- ence have guided telemedicine cost evalua- tions. The methodological benchmark has compared assessments of alternatives, and the

5690_10_p348-369 7/6/05 2:17 PM Page 348

strongest evidence and highest-quality find- ings come from randomized, controlled trial (RCT) designs. Systematic reviews of telemed- icine literature largely are based on criteria from classic RCT, and comparisons are among alternative interventions as measured by costs and benefits. But although the RCT model is appropriate for assessing clinical effectiveness, the evaluation of telemedicine programs has fundamentally different purposes, standards for judging causality, logistical constraints, and implications from major threats to validity.2 Hence different methods may enable more ef- fective program cost evaluations.

RCT guarantees little when applied to most telemedicine program evaluation, which faces different constraints from clinical research. Re- search on pharmaceutical and biological de- vices, for example, depends on the use of RCT methods to infer causality. On the other hand, the purpose of telemedicine cost evaluation is to support program and policy decision mak- ing.

The overall quality and success of a program evaluation depends on understanding and ap- plying findings to decision-making contexts and needs. The key methodological challenge in telemedicine cost evaluation is to identify the decision-making information requirements and to depict the decision framework that de- fines the contextual validity of findings. This issue was argued in a recent position paper of the American Evaluation Association (AEA),

RCTs are not always best for determining causality and can be misleading. RCTs ex- amine a limited number of isolated factors that are neither limited nor isolated in nat- ural settings. The complex nature of causality and the multitude of actual in- fluences on outcomes render RCTs less capable of discovering causality than de- signs sensitive to local culture and condi- tions and open to unanticipated causal factors.2

Consequently, although selecting a method- ology that achieves internal validity remains important in cost evaluations, methods that fit the specific problem context and decision-mak- ing needs are equally important.

In addition, the effective interpretation and generalization of program cost evaluations rely on different tools and applications of concep- tual schema as compared with those of clinical research and most cost evaluations of drugs and devices. Telemedicine programs are highly heterogeneous interventions, with complex sets of input, output, and context factors. In- terpreting the data and findings on telemedi- cine program costs requires insight into the health-care delivery processes that generate the data. Interpretation, integration of findings, and generalizations often require insights from health-care delivery and economic theories. Meaning and differences in findings may not become apparent until viewed within a con- struct, such as health production.

At the same time, the specific uses of pro- gram cost evaluations shape the results and the applicability of findings. For example, cost evaluation may focus on providing results for a health-care organization or a system of care, or they may be intended for general applica- tion. Different research methods may provide equally valid information on a question, but the measures and control of contextual factors may require different evaluation methods. A review of research studies needs to incorporate the in- tended use of the evaluation. Different strate- gies may be needed to synthesize disparate findings.

Accordingly, telemedicine cost evaluations and reviews of this literature have not paid at- tention to the unique purposes, needs, and methods of program cost evaluation, nor have they focused on non-RCT or economic method- ologies. Program cost evaluations involve matching analytic methods to the purpose and constraints of the evaluation, identifying the role and characteristics of decision-making frameworks, and using appropriate economic concepts for interpreting program findings. Each of these components of the analytic struc- ture may be momentous in determining the value of a study. Although trade-offs exist in the use of optimal strategies, rigid adherence to strong methods of determining causality may make studies meaningless because of as- sociated limitations in contextual validity. The productivity of telemedicine cost studies suffer from an under-utilization of appropriate pro-

ASSESSING TELEMEDICINE COSTS 349

5690_10_p348-369 7/6/05 2:17 PM Page 349

gram evaluation and economic methods in their design and assessment. First, however, these elements of cost evaluation are applied in the literature, and the methods of selecting the literature are described.

ELEMENTS OF COST EVALUATIONS OF HEALTH-CARE SERVICES

A review of the major analytic components of health-care services cost evaluations pro- vides an overview of the economic concepts, cost analysis decision framework, and methods of inference that affect the quality and produc- tivity of telemedicine cost research. This over- view guides the subsequent identification, se- lection, and review of the telemedicine cost literature.

Cost evaluations of health-care services dif- fer in their purpose. Information about costs may be needed in organizational resource plan- ning; funding agencies may require evalua- tions to justify expenditures; or health-care sys- tem policy makers may be considering broad deployment of a technology. But at the core of each cost evaluation is the analytic contribution of economics, a comparison of alternatives based on the concept of “opportunity costs,” defined in the MIT Dictionary of Modern Eco- nomics as “the value of resources given up through a decision.”56 The essence of the cost evaluation is identifying the alternative that provides the greatest value in relation to the opportunity costs. This may be described in three basic steps:

1. Identify and aggregate the costs of each health-care alternative.

2. Identify and aggregate valued outcomes of each alternative.

3. Compare the aggregated relative, “value for cost,” for each alternative (e.g., such as the ratio of costs to changes in quality-ad- justed life years attributable to an inter- vention).

This process seems simple; however, most of the debate around telemedicine cost analysis comes from difficulties in implementation, and here economic theory only provides broad

guidance. The usual errors and shortcomings in the telemedicine cost literature may be attrib- utable to its deceptive conceptual simplicity.

Implementation of a cost analysis requires a well-defined decision framework that consists of objectives, resources used, definitions of out- puts, and a context of the decision making, such as timeframe. The relative significance of each these issues in determining the quality of a study depends on program characteristics and types of telemedicine.

Although the specification of telemedicine is grounded in economic concepts, it often de- pends on a broad, multidisciplinary under- standing of health services delivery. Complex interventions of telemedicine, concepts of sys- tems analysis, organizational decision making and strategy, and health-care behavior may be critical for identifying and understanding the value and cost of the telemedicine.

Differences of opinion exist regarding the value of applying economic concepts to cost evaluation. Some variations in methods may arise because of differences in objectives, as- sessments of the literature, or simply due to multiple, equally acceptable approaches on some issues. On the other hand, conformity in an approach facilitates assessments of literature and provides common utility from a cost study, especially if the study fits within broader re- search or health-care policy objectives. Con- sensus definitions and methods in several ar- eas leverage the cost literature and provide collective value. Consensus guidelines are making their way into recommended cost eval- uations, and they provide an import element in assessing telemedicine costs.

The following sections provide an overview of the cost evaluation that affects the quality of the telemedicine cost literature. Economic con- ceptual issues, the decision framework, related considerations of methodological inference, and the role of cost guidelines as they affect the quality of telemedicine cost studies are sum- marized. These materials will shape the subse- quent review of the telemedicine literature.

Economic concepts

Economic theory, as applied to cost evalua- tion, generally is not a basis for prediction, but

REARDON350

5690_10_p348-369 7/6/05 2:17 PM Page 350

it provides concepts and a structure for identi- fying and developing those measures of inputs and outputs that define program performance. Primarily, the concepts of economic theory of choice constitute a reference for researchers de- veloping observable measures of costs and ben- efits, and selecting the most appropriate cost evaluation model, such as cost–benefit analysis (CBA) and cost effectiveness analysis (CEA). Otherwise, for program cost evaluations, eco- nomic theories of organization and production may help clarify appropriate definitions; iden- tify appropriate aggregations of costs and ben- efits, and interpret expected impact or findings.

Economic costs and benefits

The most important role of economic theory in cost evaluation is in guiding the identifica- tion and measurement of economic costs and benefits. In particular, the concept of costs, de- fined as an opportunity cost in the economic theory of choice, provides the conceptual foun- dation for defining costs as the value placed on those resources given up by a decision maker when making a specific choice. As such, costs identified in an economic evaluation are sub- jective, being defined distinctly from the per- spective of a single decision maker.

However, costs, or components of costs, may be estimated through such metrics as market prices. A parallel concept may be applied to the valued gains of a decision that define benefits. Thus, among choice alternatives, a comparison is made of the estimates of a decision maker’s opportunity costs relative to the value of out- comes. The measurement of costs is the nega- tive aspect of choice, or the value of what is given up, and the measurement of benefits is the positive value placed on gains. The as- sumed preference is for the greatest value of outcomes relative to opportunity costs for the decision maker.

Of course, whereas the concept is deceptively reasonable and easy to grasp, the implementa- tion is exasperatingly complex. To begin, the analyst must carefully define the decision maker and the intervention or decision under consideration and its alternatives. Economic concepts may help define relevant features of a telemedicine intervention, but mostly the de-

scription and detail of a program are distinct from the core application of economic concepts in the cost evaluation. These issues of program definition are addressed in a subsequent sec- tion of this review. This task aside, however, the challenge lies in distinguishing inputs and outputs and assigning appropriate costs and benefits where they are tied to the subjective assessments of the decision maker.

The measurement of costs begins with an identification of the resources “given up” as a result of a choice, by implementing a telemed- icine activity or program. Here, the question that needs careful consideration is whether im- plementation of the telemedicine program re- quires new resources or expenditures (e.g., con- struction) or the commitment of resources (e.g., existing facilities) that could be used for other purposes elsewhere. The concept of opportu- nity cost distinguishes the relevant cost for the evaluation in that the resources used are a con- sequence of the decision. For example, the in- stitutional overhead of a program may be an accountant’s cost, but not necessarily an eco- nomic cost because additional commitment of overhead may not be required with the imple- mentation of telemedicine. Of course, this as- sessment of costs depends on the timeframe of the decision. In the short run, there may not be an opportunity cost for the facility, but long- run facility costs may be incurred. Hence, this assessment of program costs must consider the relevant timeframe.

Once opportunity costs are identified for a telemedicine program, the analyst is then able to address the related question on the value of the resources. Again, the value assigned to resources is defined from the perspective of the decision maker. Values may be easily as- signed to some resources, as market prices of- ten reasonably represent the value of re- sources such as equipment, manpower, etc. On the other hand, resource costs of volun- teer labor and patient travel may not be eas- ily determined. Nonetheless, an analytic ob- jective should identify all resource costs (market and nonmarket) that accrue to a de- cision maker as a result of the program. A common metric of value must be used to pro- duce an aggregate measure of costs for each program alternative.

ASSESSING TELEMEDICINE COSTS 351

5690_10_p348-369 7/6/05 2:17 PM Page 351

At the same time, each program choice, such as telemedicine or its alternative, results in out- comes as well as costs. Hence, the analysis must identify all of the important outcomes or gains from each intervention.

Identifying and then measuring the value or benefits of a telemedicine intervention may be considered the complement of determining the opportunity costs of resources, but this task in the evaluation assessment is often more chal- lenging. In contrast to input costs, market prices are usually not available for the benefits derived from the health-care program. More- over, the outputs themselves, such as a suc- cessful treatment or improvements in health frequently are not easily defined, identified, or measured.

Telemedicine programs may create new out- comes of care or change outputs or outcomes. These benefits of telemedicine are often diffi- cult to identify, partly because detecting them depends on our expectations or knowledge of effects on health or other outcomes. Also, de- tecting benefits may depend on timing, with some benefits only becoming apparent at a time in the future. Then, again, whether an outcome is a benefit may depend on the perspective of the decision maker or stakeholder. The decision maker may be interested in the ability of telemedicine to provide visits or treat episodes of care, but not in the longer-term effects on health.

But even with outcomes identified, a formi- dable task remains in assigning value to the outcomes of telemedicine and aggregating the sum of benefits for all outcomes. From the per- spective of economic analysis the ideal is to evaluate all outcomes with a common mone- tary metric. These could then be aggregated and compared with costs for each alternative. This type of assessment, CBA, is the preferred method of evaluating alternatives because it fa- cilitates the most direct and ideal implementa- tion of the economic evaluation paradigm, and the broadest consideration of cost and benefit tradeoffs, allowing comparisons with all simi- larly evaluated choice alternatives. Yet, the ap- plication of CBA is often difficult, especially in health care, because of the need to identify, measure, summarize, and compare outcomes that are not usually assessed through the mar-

ket-based exchange. In particular, there is a general reluctance to assign a value to life or health, at least in such a way as to make it com- parable to values placed on non-health-care commodities. Consequently, CBA that do not rely on monetary assessments of benefits are more often used in health-care cost evaluation.

Several alternative methods of CBA have evolved. Most are prominently distinguished by their methods of establishing and aggregat- ing benefits, or valued outcomes. Each may be appropriate, depending on the needs of the an- alyst and the trade-offs. More than 10 govern- ment and industry organizations have devel- oped health-care cost-analysis guidelines in which several different types of cost evalua- tions are recommended.3 Issues associated with several of the most common cost-analytic methods in telemedicine are briefly described in Appendix 1.

Among the half dozen types of cost evalua- tion used frequently in health-care services re- search, CEA is most prominent. As described in the appendix, whereas costs are monetary in a CEA, personal health-care outcomes are re- ported in nonmonetary terms, such as changes in blood pressure, or longer-term outcomes, such as improvements in quality-adjusted life years (QALYs), where years of life are weighted by health-related quality measures.4 A resulting program assessment metric such as “cost per QALY” provides a common currency that is especially helpful for comparing health- care technologies.5

Of course, each cost and benefit method has advantages and disadvantages with implica- tions for measurements of benefit, potential comparisons among alternatives, and types of outcomes that can be captured in the analysis. The different methods, as well as details in this implementation, determine the range and types of effects captured in the analysis, as well as the potential for comparisons among alter- natives. For example, selecting episodes of ill- ness as the outcome of a CEA evaluation lim- its the potential to identify the effects of telemedicine on preventing new episodes of ill- ness. Using visits or virtual visits as the out- come ignores the potential of telemedicine to treat episodes with fewer provider contacts, or fewer hospitalizations.

REARDON352

5690_10_p348-369 7/6/05 2:17 PM Page 352

Cost minimization analysis (CMA) of tele- medicine generally ignores any potential for telemedicine to affect health-related outcomes, because CMA strictly focuses on cost-related benefits of the technology.

In summary, understanding cost–benefit concepts applied to the economic theory of choice is fundamental to both appropriate de- sign and interpretation of telemedicine cost evaluations. This affects the identification and measurement of appropriate inputs and out- puts of telemedicine and its alternatives. The concepts and the selection of a cost and bene- fit method constrain the potential inference from and the interpretation of findings.

Production functions in telemedicine cost evaluations

Compared with the economic theory of choice, other areas of economic theory are far less pivotal in health-care program cost evalu- ation. However, economic concepts of produc- tion appear likely to contribute to the design of telemedicine cost evaluation. This theory also has the potential to spur the development of a body of telemedicine production knowledge and models with predictive content.

Berki first proposed the use of an economic production function to structure our under- standing of telemedicine, stating “ . . . the fun- damental effects flowing from the introduction of telemedicine will be changes in the process of production outputs as well as changes in the outputs themselves.”6 His detailed develop- ment of an economic production function of telemedicine was employed to generate insight and questions on the implications of telemedi- cine and the design of telemedicine economic research. In particular, he identified the need for greater specificity in types of telemedicine and their outputs when attempting to analyze telemedicine costs. He also identified the need to address telemedicine as being embedded within a health-care system.

Although not providing a predictive theory of costs, Berki emphasized the use of telemed- icine production analyses to identify an exten- sive range of potential inputs and outputs. He also identified the complexity of cost research design and interpretation. Economic produc-

tion theory provides a logical structure to iden- tify and evaluate cost determinants, resources that affect costs, and the associated outcomes. It identifies both potential factors and logic re- lated to the transformation from resources into such outcomes as access, health, and satisfac- tion. Hence, the body of telemedicine cost find- ings may be more easily understood through this framework, which, when combined with specific findings, may improve future research and lead to predictive models for telemedicine cost research.

Cost assessment framework

Biomedical devices may be described through a set of technical specifications; how- ever, the content of a telemedicine program is less easy to define. Most often, telemedicine programs are described through areas of med- ical specialty or location of care. These attri- butes, however, are insufficient for specifying production inputs and likely outputs. More- over, contextual factors of delivery programs, such as the boundary between teleradiology and standard radiology, are much more ambiguous. Consequently, additional descrip- tions of telemedicine programs are often needed. The question is what constitutes the program? What are the critical dimensions that facilitate more homogeneous groupings or ad- justments for comparison of findings?

Some of the useful attributes for characteriz- ing telemedicine programs are those that de- termine the use and value of program inputs or outputs, or that characterize assessments of opportunity costs. These attributes may be viewed as a cost assessment framework, which is not a well-defined algorithm or theory, but rather a description of a telemedicine program that is situationally relevant in identifying and defining costs and benefits. For example, such characterization reveals how a program is de- fined and the decision-making time horizon, both of which affect costs for the stakeholder. The decision maker’s objectives, delineation of the boundaries between the intervention and context, and identification of a target audience for use of research results are major underly- ing assumptions that characterize the cost eval- uation problem.

ASSESSING TELEMEDICINE COSTS 353

5690_10_p348-369 7/6/05 2:17 PM Page 353

The cost assessment framework as defined in Table 1 is closely related to components used in defining CEA by the United States Public Health Service (USPHS) Consensus Panel.7 The major dimensions of this framework help as- sess telemedicine cost-evaluation design and interpretation.

Program objectives and study purpose

A principal aim of telemedicine cost evalua- tion is to provide decision makers with infor- mation on costs of alternatives decisions. This requires an accurate understanding of objec- tives so as to obtain a precise identification and valuation of what is given up and gained as the result of an action. For example, a program with a short-term treatment objective will in- cur different costs and benefits than one focus- ing on prevention. Without sufficient knowl- edge of the decision-making objective, the empirical research and analysis may not be rel- evant. That is, precise evidence on the wrong question is most often not helpful. Conse- quently, it is often useful to have a pre-evalu- ation assessment of the decision maker’s ob- jectives. Careful consideration of objectives may be warranted when interpreting findings in the telemedicine literature.

Notably, the issue of program objectives is related to but separate from that of the decision maker’s perspective. Different types of decision makers, such as the patient, provider, payer, and society at large, may have different objec- tives for a program and different values as- signed to benefits. But these groups are also likely to incur different costs and benefits from the program, even if they agree on the same program objective.

This difference in perspective results in dif- ferent costs and benefits among the stakehold- ers, and sometimes may result in differing incentives for program participation. The ap- propriate perspective for an evaluation will de- pend on the purpose of the evaluation. On the other hand, many researchers advocate the in- clusion of the societal perspective (where all program costs and benefits are included in the CBA, regardless of who incurs or accrues them) in reporting results. The intent here is to facil- itate comparability across studies.

At the same time, the primary purpose of telemedicine cost evaluation is to support health-care policy and organizational decision making. For this purpose, a cost assessment framework may guide the evaluation, but for that evaluation researchers may also subscribe to a larger role of ongoing knowledge building by making study results usable for a broad population of stakeholders with similar objec- tives. Here, simply the intended scope in the generality of findings may affect the methods of evaluation. For example, an evaluation may provide valid and generalizable results at a lo- cal or national level. Variations in international prices, population preferences, and health sys- tem efficiency, however, may undermine cross- country generalizations.8 The challenge be- comes one of the research tradeoffs required to address contextual factors that influence the generality of results. Less powerful findings lo- cally may be sacrificed for the power of broad generalization.

Telemedicine program definition and context

Telemedicine refers to a wide range of activ- ities providing health care at a distance. These include remote monitoring, where capabilities are largely defined by device specifications; medical encounters that may be defined by en- abling devices; and complex delivery pro- grams, such as virtual hospitals that employ a complex mix of technologies and innovative processes. Thus, telemedicine interventions may include hardware and software technol- ogy, a clinical process, and a broad system of health-care delivery.

With device-oriented telemedicine, the defi- nition of the program, its functional capabili-

REARDON354

TABLE 1. COST ASSESSMENT FRAMEWORK

A. Program/decision-making objective B. Study purpose-constituency C. Target decision maker

Perspective/stakeholder Audience

D. Program definition and boundaries Type of analysis Decision maker Target population Decision time horizon Decision maker time preference

5690_10_p348-369 7/6/05 2:17 PM Page 354

ties and resources, and the relevance of any contextual factors are more easily identified. In contrast, as telemedicine becomes a complex program of activities with different resources and multiple outcomes, the heterogeneity of telemedicine capabilities, capacities, and re- sources create a substantial challenge to be gen- eralized from an evaluation. Closely integrated technologies, such as information technology systems and contextual factors, are likely to have dependencies in the system of care that are not easily separated from core telemedicine capabilities.

At the same time, telemedicine-oriented sys- tems of care are more likely to provide needed economies of scale, joint gains from technology, and manpower knowledge efficiencies. In ad- dition, the complexity of systems and activities related to telemedicine systems are likely to make program performance less determined by technical capabilities and increasingly depen- dent on management efficiencies and incen- tives, as has been discovered with information technology in general.9 Hence, the defining at- tributes of a promising telemedicine interven- tion are likely to change as applications shift from narrowly defined, technically oriented clinical telemedicine capabilities to reliance on systems of information technology with be- haviorally defined production processes.

The result is that different approaches and concepts may apply to cost evaluation of de- vice-dominant telemedicine as compared with system- or activity-dominant telemedicine pro- grams. Basic comparisons of alternatives that fit the evaluation paradigm appear to be less appropriate for cost evaluations of less techno- logically dominant and complex telemedicine systems of care. The cost impacts of telemedi- cine systems may be determined by a set of con- textual factors, some of which may be identi- fied through economic theories of organization and competition.

Methods of inference

RCTs constitute the most powerful method- ology for testing the impact of a medical inter- vention, procedure, or medication. Medicine has been well served by RCTs and cost-effec- tiveness/cost-utility evaluations of pharma-

ceuticals and medical devices have been quite successful by employing RCT methods of in- ference. Nonetheless, whereas RCT methods may be good for evaluating telemedicine cost, other methods of statistical analysis may be warranted and even preferred.

In contrast to telemedicine, several areas of evaluation accept alternative methods for demonstrating causality though the rationale may not be without some debate. Of course, medical research has the strongest tradition of randomized experimental design for causal as- sessment, and perhaps because of this, non- RCT methods have had limited use in evaluat- ing telemedicine costs. Several arguments for using alternative methods of inference appear credible in relation to program cost evaluation, including when:

• Threats to external validity transcend the RCT contribution to internal validity,

• Relationships between the intervention and context cannot be disentangled through ran- dom assignment,

• RCTs are not feasible or the additional con- fidence in outcomes provided by RCTs is economically justified,

• Question of interest is not an unbiased esti- mate of program costs in a population but an estimate of costs in an uncontrolled set- ting (i.e., when self-selection is allowed).

The strength of RCT derives from ensuring the internal validity of findings.10 But impor- tant determinants of costs, such as organiza- tional competence, factor prices, and health system policy, are difficult to randomize, and they may have a more significant impact on cost than the intervention. Hence, the RCT find- ings from telemedicine programs cost evalua- tions may have limited value to decision mak- ers who are concerned about the overwhelming effects of their context. Moreover, when the performance of telemedicine is determined by a system of variables, such as those involved in health-care production, or conditional on fac- tors, such as managerial incentives, contextual effects may be impossible to disentangle from contextual factors. In neither case do RCT methods provide the best information for the decision.

ASSESSING TELEMEDICINE COSTS 355

5690_10_p348-369 7/6/05 2:17 PM Page 355

For some types of telemedicine costs ques- tions, decision makers may not be interested in the unbiased estimates of costs. For example, a third-party payer may be interested in the cost of self-selection biased program, i.e., program costs when patients have self-selected. Selec- tion and choice are important components in the design of a health-care delivery system, so unbiased cost-effectiveness may not be useful without knowing the cost bias of choice.

To summarize, the methodological question is to determine the appropriate or “cost-effec- tive” methods of inference, matching the eval- uation design to the evaluation’s purpose, re- sources, and stage of development.

Cost analysis guidelines

Guidelines are increasingly available for con- ducting, evaluating, and synthesizing health- care research. They provide a powerful tool for improving the quality and comparability of studies. However, although economic cost analysis is tied to an economic theory of choice, there is room for considerable latitude and dif- ferences of opinion in applications. For exam- ple, standardizing definitions would improve the comparability of studies. Errors and methodological limitations of cost analysis are reported throughout the literature. Nonethe- less, guidelines are designed to improve the quality of research in this area.

Spurred by their potential to provide a sig- nificant collective value for health-care cost re- search, health-care cost analysis guidelines have evolved rapidly.3,11 A few sources focus on assessing and improving quality of evalua- tion studies.12 Most notably, the Cochrane Col- laboration provides guidance for systematized assessment and synthesis of study evidence.12 Other groups have developed guidelines that promote consensus or preferred definitions and methods for research, and less explicitly at- tempt to determine evaluative standards. At least 10 sets of formal and informal guidelines dedicated to health care cost analysis, mostly related to pharmaceuticals and devices, have been developed by government and industry organizations. Perhaps the most detailed of these are the USPHS Consensus Panel guide- lines that demonstrate a particular interest im-

proving the comparability of studies.7 The Con- sensus Panel provides a theoretically consistent cost assessment framework for: (1) defining a cost or resource allocation decisions, including decision-making objectives and defining the program or intervention; (2) determining the decision model of costs and benefits; (3) speci- fying data and measures; (4) performing ap- propriate statistical, modeling, and sensitivity analyses; and (5) reporting results understand- able by decision makers and researchers.

Notable improvements in compliance with key guidelines have been reported since publication of the consensus panel recommen- dations.13 On the other hand, analysis of pub- lished cost analysis guidelines found agree- ment for about 75% of methodological aspects.3 Disagreement was found in the recommended choice of perspective, the types of resources considered, and costs that should be included in the analysis. Partly, the differences in the guidelines appear to stem from differences in research objectives and associated technical is- sues. As a recent assessment of evaluative systematic reviews concluded, “Methods for assessment of methodological quality by sys- tematic reviews are still in their infancy, and there is substantial room for improvement.”14

IDENTIFYING AND SELECTING TELEMEDICINE COST

EVALUATION LITERATURE

The status of telemedicine cost research is shaped by research findings, methodologies, and the resulting conceptual understanding of telemedicine costs. To this end, this section of- fers an approach and selection process for ob- taining an accurate representation of this liter- ature.

Several authors have used formal guidelines in conducting systematic literature searches in reviewing of telemedicine cost evaluations.15–20 Typically, these searches have used topical key- words to identify articles, and then explicit criteria on scientific merit for including or ex- cluding studies in the review, with the objec- tive of providing an efficient and unbiased es- timate of costs and benefits. This paper builds on these earlier works, but not simply by

REARDON356

5690_10_p348-369 7/6/05 2:17 PM Page 356

adding the recent months of findings. Taking advantage of these earlier efforts, the results of earlier reviews and key studies are combined with newly identified studies. The emphasis of this assessment is on the conceptual and methodological rigor in cost studies, on the usefulness of identified cost issues, on research methods and strategies, and on conceptual in- terpretation. The concepts and methods dis- cussed in previous sections of this paper21 are used to guide the following search, selection, and review.

Still, identifying the boundaries for selecting articles remains a formidable task, which is es- pecially apparent in terms of the conceptual ambiguities surrounding telemedicine costs. Activities that may be considered telemedicine continue to blend applications of information and communications technologies. Of course, the history of telemedicine research has been characterized by substantive debate on defini- tions, and this continues to evolve with tech- nology. As a result, the use of a defined set of telemedicine keywords in selecting literature may easily omit relevant studies, but the most ambitious expansion of terms begins to capture an unmanageable number of studies.

An efficient alternative for selecting and reading abstracts produced by the most ex- pansive set of keywords evolved from first con- sidering studies identified through previous systematic reviews and then using keyword and database selection strategies to augment the literature. This approach may be best de- fined as iterative triangulation. That is, first previous reviews and standard initial sets of telemedicine keywords were used to create a database of abstracts for review. These were used to create a database of keywords, which resulted in overlapping sets of literature. Sets of prominent and seemingly relevant key- words were examined to identify additional ar- ticles, but investigating the detailed subsets of articles depended on the initial payoff in the discovery of new articles.

At the same time, significant references were also identified from the lists of references in the original articles. While the primary focus was on telemedicine cost literature, other promi- nent databases in the literature were included, such as health-care technology assessment

(HTA) and health-care information technology (HIT) (Table 2). These searches were guided largely by the need for further considering sig- nificant issues in the telemedicine cost litera- ture.

The database of resulting articles consisted of studies identified through electronic searches and manual scans of both published and unpublished studies, resulting in a data- base of 1,430 telemedicine articles (Table 3). Ab- stracts for each of these were reviewed. Among these articles, abstracts that indicated cost anal- ysis with empirical data, or appeared to have rigorous or unique cost analysis research meth- ods were selected for further consideration.

Finally, a database of studies based on stan- dard telemedicine keywords, related analytic area keywords, and all related articles was con- structed. Software was used to create and map clusters of keywords, which were inspected to identify unrevealed telemedicine related groups of studies. Clustering analyses of key- words associated with over 9,000 articles were evaluated to establish consistency across selec- tion methods and significant omissions in top- ical and methodological areas of telemedicine costs. This resulted in the identification of few additional papers, but no new topical or methodological areas were added to the anal- ysis.

AN ASSESSMENT OF RESEARCH FINDINGS AND METHODOLOGIES

The prevailing consensus is that few telemedicine cost studies have provided eco-

ASSESSING TELEMEDICINE COSTS 357

TABLE 2. KEYWORDS AND CONCEPTS USED IN SEARCH

Primary terms Secondary terms

Telemedicine Cost Telecommunications Cost allocation Telehealth Cost analysis Telemedicine Cost benefit analysis Telenursing Cost effectiveness Telepathology Cost minimization Teleradiology Cost savings Telemetry Health-care costs Telephone Health information technology Television Technology assessment

Methods and methodology Economics

5690_10_p348-369 7/6/05 2:17 PM Page 357

nomic results that are useful for organizational or policy decision making in telemedicine.15,22 Research reviewers have consistently noted a small number of useful findings,15,22,17,23 and even these are marked by questionable use of standard cost techniques. Hence, they contrib- ute little to the cost evidence.15,24,25 Similarly, but perhaps having a different perspective on what is lacking, providers and payers continue to express a need for cost evidence for decision making.

Analysts have provided many explanations for the perceived inadequacy of telemedicine cost information. Studies have been faulted for having small user populations or data, fledg- ling programs with premature evaluation, biases due to reimbursement policy, faulty eco- nomics, and inadequate research methodol- ogy.18,26 In particular, the limitations of the lit- erature have been most commonly attributed to inadequate and insufficient use of RCT- based studies. At the same time, at a more fun- damental level, telemedicine cost research has

been viewed as being encumbered by its “com- plex mix of technology and services.”15 This re- view considers whether current telemedicine cost evaluation concepts and methods con- stitue a barrier to useful cost research, and whether uses of economic theory and alterna- tive research methods would improve the value of output from telemedicine cost re- search. Assessments are provided through dis- cussions of the findings and limitations of telemedicine cost evaluation research.

An economic interpretation of cost findings

Reviewers of telemedicine cost studies be- moan the inadequacy of research methods and the resulting small number of useful cost find- ings. This conclusion is partly driven by the standards of methodological orthodoxy, or rigidity, for causal analysis. But, it also depends on assumptions about how findings reflect on the specific cost problem. In particular, RCT methods are especially appropriate for discrete,

REARDON358

TABLE 3. SEARCHED SOURCES OF LITERATURE

Type of source Sources

Published literature NLM Databases—MEDLINE (1/1976–3/2004); HSAT EMBASE (1/1995–3/2004) ISI Science Citation Index Expanded ECO—An Online Computer Library Center (OCLC) collection of scholarly journals, CINAHL—Cumulative Index to Nursing and Allied Health Literature JEL—Journal of Economic literature

World Wide Web AHRQ: http://www.ahcpr.gov/ NHS Health Technology Assessment Programme: http://www.hta.nhsweb.nhs.uk/main.htm NHS National Institute for Clinical Excellence (NICE): http://www.nice.org.uk/ Netting the Evidence: http://www.sheffield.ac.uk/~scharr/ir/netting Canadian Coordinating Office for Health Technology Assessment (CCOHTA):

http://www.ccohta.ca/ Office for the Advancement of Telehealth: http://www.telehealth.hrsa.gov/ VHA Telehealth: http://www.va.gov/telehealth/ Telemedicine Information Exchange: http://tie.telemed.org/ AHRQ Evidence Based Practice Centers Cochrane technology databases McKinsey Global Institute: http://www.mckinsey.com/knowledge/mgi/

Conference reports HIMSS conference reports: 1995–2003 AMIA conference reports

Hand-searched Journal of Telemedicine and Telecare journals Telemedicine Journal/Telemedicine Journal and e-Health

Journal of Health Economics Health Economics Value of Health Journal of Healthcare Information Management International Journal of Technology Assessment in Health Care Journal of Information Systems

5690_10_p348-369 7/6/05 2:17 PM Page 358

well-specified interventions. Accordingly, the research design has been to address different types of telemedicine as distinct technologies. This specification has been maintained in the interpretation and integration of findings. The notion of discrete interventions has been trans- lated as conceptually independent of the dif- ferent types of telemedicine findings. Often, the assumption of conceptual independence of telemedicine program cost findings is unnec- essary.

The economic underpinnings of cost evalua- tions may provide a basis for the conceptual integration of the seemingly disparate telemed- icine cost results. In particular, the interpreta- tion of cost results from telemedicine programs may make use of the economic production con- cepts previously discussed. Specifics on the ap- proach used in a cost analysis may be combined with other knowledge of medical care delivery and economic production to synthesize find- ings around a distinct cost principle. Accord- ingly, the following sections attempt to make more efficient use of telemedicine cost research by assessing how they reflect on economic pro- cesses that determine costs.

Cost impacts on access to a medical visits

The earliest studies of telemedicine costs con- centrated on the cost implications of enabling access to care. As such, studies primarily con- sidered the costs or economic gains from re- ductions in patient or provider time traveling for care. These studies often involved clinical technologies that were considered unchanged when provided through telemedicine. For ex- ample, for purposes of cost evaluation, the clin- ical character of a radiological service or other imaging service was considered to be essen- tially identical to in-person service.

Early cost studies often implicitly or explic- itly were presented as cost-minimization stud- ies, but they were more unique than that. The basic specification of the problem in these stud- ies was to limit the telemedicine effect to the production of access while defining the cost structure as a trade-off between gains from re- duced travel costs and expenditures on Infor- mation and Communications Technology (ICT) and personnel. Hence, high barriers to access

and specialties requiring less expensive tech- nologies were likely to produce favorable re- sults for telemedicine.

Using this approach, telemedicine produces cost savings for specific populations, such as prison inmates,26,27 sailors at sea,28 or emer- gency cases.29,30 Hence, the potentially prohib- itive determinant in the use of care is the cost of travel, which can be offset by efficient use of the ICT technology. Specialties that required cheaper technologies, such as psychiatry, or that easily integrated the technology in the visit, such as dermatology, were more likely to show savings through reduced travel costs. On the other hand, these studies were important only to decision makers who incurred a suffi- cient saving from travel costs. Most notably these included U.S. prisons, the military ser- vices, and a number of European health-care systems.

Of course, a reasonable conclusion is that once the technical feasibility for telemedicine is established, the potential for savings from travel for patients would be expected. The ev- idence suggests that travel-based cost savings are feasible for a variety of telemedicine spe- cialty applications, including teleradiology, teleneurology, teledermatology, and teleoncol- ogy.29–32 However, the findings for other types of telemedicine, such as telepsychiatry, are in- consistent. This may be due to the fact that some of these study designs focused on the im- pact of telemedicine on access, whereas others included the effect of telemedicine on clinical efficiency. The latter is the subject of discussion in the next section.

Clinical efficiency aside, the expected savings in this type of telemedicine analysis are situa- tionally dependent; that is, they depend on the environmental context, such as general costs of travel and technology. More importantly, once feasibility is established, significant economic findings are not likely through a local single-site clinical trial. The key parameters affecting costs, location, and price of technology are external to the design of local clinical trials.

Costs impact on clinical efficiency

A second category of telemedicine programs effects may be characterized as changing the

ASSESSING TELEMEDICINE COSTS 359

5690_10_p348-369 7/6/05 2:17 PM Page 359

costs of care by affecting clinical efficiency, as well as travel and technology. But here tele- medicine may result in changes in the amount or mix of resources and thereby the costs of clinical care. Changes in clinical care processes decrease or increase cost care. This may best be reflected in assessing episodes of care.

This additional role of telemedicine in af- fecting costs is significant from many perspec- tives. First, clinical costs often have a dominant share of total costs of care, especially for pop- ulations without exceptional barriers in access. Consequently, efficiency savings through tele- medicine present a much greater opportunity for finding savings. These clinical efficiencies may be influenced more by patient and physi- cian characteristics and less by local situational factors. Moreover, and perhaps as important, depending less on local factors allows findings on the cost impact telemedicine to be applied more easily to other settings. Presumed costs savings due to the effects of telemedicine on the production process of medical care are more likely generalizable across settings.

Analyses of telemedicine that capture the costs impacts of clinical efficiencies may be de- fined implicitly or explicitly through several features of the study design. First, the analyst may explicitly recognize the change in the mix or types of resources used in the clinical pro- cess. For example, store-and-forward derma- tology is likely to be basic change in the clini- cal process when compared with real-time teledermatology. Alternatively, many studies implicitly allow for impacts on clinical effi- ciencies through their definitions of the output. For example, where the cost impacts are ex- amined across multiple patient encounters or visits, then telemedicine may affect clinical in- puts. The number or mix of clinical visits may be changed in the treatment of the care.

Studies of teledermatology may best illus- trate the merit of this conceptualization in guid- ing the integration and interpretation of find- ings. Dermatological conditions often occur in relatively well-differentiated episodes. Conse- quently, telemedicine studies frequently link multiple provider encounters. At the same time, dermatology practice lends itself to the use of alternative sets of resources in the pro- duction of a telemedicine visit, such as real-

time and store-and-forward processes of care. The result is that each of the three sources of cost impacts may apply. But changes in the clinical process for real-time teledermatology are less likely to affect total costs, as compared with store-and-forward processes.

Studies of teledermatology fit surprisingly well within the constructed cost framework. The cost savings of real-time and store-and- forward teledermatology depend on the costs of travel. But store-and-forward costs have proven to be more variable across studies and sensitive to specific assumption about the pro- duction of clinical care, such as patient flow. Moreover, although an RCT study of a Vet- eran’s Health Administration teledermatology program found it not to be cost effective, fur- ther analysis of the results indicated that it be- came cost effective with modest changes in clinical or travel costs. Accordingly, costs of a teledermatology program are likely to depend on the situational and generalizable impacts of telemedicine, and that decision makers need to consider the relative roles of each. At the same time, researchers may contribute by further an- alyzing the determinants of clinical productiv- ity in teledermatology and the relative magni- tude of these when compared with the savings from travel and the costs of technology.

Telepsychiatry presents a contrasting set of insights when examined through this concep- tualization of the sources of telemedicine costs. Telepsychiatry has been shown to be feasible, may increase access to care, and often is re- ported cost effective. The unit of output in telepsychiatry is the visit, but this is somewhat ambiguous as a measure of treatment output. At the same time, there is considerable latitude in the resources used to provide telepsychiatry. Moreover, telepsychiatry has lacked standard models or guidelines for care, although several detailed guidelines have been recently pub- lished.

Consequently, it becomes difficult to deter- mine how much or whether telemedicine changes the productivity of the psychiatric clin- ical process. Telepsychiatry is usually reported as less expensive for patients when the cost of travel and lost time are considered.33 This is es- pecially apparent when care is provided to prison populations, remote clinics, and other-

REARDON360

5690_10_p348-369 7/6/05 2:17 PM Page 360

wise isolated populations.34 Indeed, telepsy- chiatry services have been estimated to be less expensive, as expensive, or more expensive than in-person services.35 The small size of many programs may be a problem, and some approaches to the delivery of telepsychiatry have been reported be more cost-effective than others.33

The implication of this economic interpreta- tion of telemedicine findings is that telepsychi- atry may be cost-effective, but the findings are unclear. Certainly, RCT studies may provide better evidence, but measurement issues ap- pear to be a major barrier to drawing definitive conclusions. Future studies of telepsychiatry should focus on telepsychiatry outputs and the changes in the mix and use of resources in the clinical care process.

Cost impacts on disease management

Some telemedicine programs may be viewed as having a direct effect on the process of maintaining health and well-being (referred to by economists as the production of health) rather than simply affecting access and effi- ciency. Economic analyses from outside of telemedicine have indicated that activities that prevent illness are often far more cost-effective than medical care treatments.36 Accordingly, telemedicine interventions that reduce the use of medical care through primary or secondary prevention potentially could save costs that might overshadow the effects on access alone. The costs saved through telemedicine through reduced travel to medical care needed in the absence of the preventive activity likely would pale in comparison.

Home telecare and telemonitoring may offer significant cost savings in terms of health and wellness. These applications may provide little improvement in access to clinical treatment, and they may not have a significant effect on the cost-effectiveness of clinical care. But they can change the need for care. Recent evidence suggests that significant savings are possible through the use of telehealth to diminish the need for medical care.37,38 For example, a ran- domized controlled trial of prenatal telehome- care for high-risk pregnancies reported a re- duction in hospital use.39 Subsequently, a

landmark RCT study of home telecare demon- strated cost-savings through reduced utiliza- tion.38 In this latter study, remote monitoring and real-time video interactions reduced total costs of care relative to those receiving tradi- tional homecare. More recent studies further provide evidence of the potential cost-effec- tiveness of telemonitoring among congestive heart failure and diabetic patients.37,38

Overall, a notable contrast has emerged in the apparent success of telemonitoring and home telecare cost-effectiveness studies, when compared with the limited results from tele- consultations.19,40,41 A number of explanations for this difference seem plausible. Home tele- care and telemonitoring occur outside estab- lished medical facilities and are therefore less likely burdened by historical overhead costs that may or may not be necessary for other types of telemedicine. This application may represent a more targeted intervention of high- risk patients with a narrowly defined technol- ogy. These findings are likely to generate even more high-quality research in this area.

On the other hand, it may also be, as some economists have long argued,36 that more med- ical care may not be as cost-effective as main- taining or improving health. In this case, home telecare may be benefiting from the underlying opportunity in the production of health. If so, telemedicine applications that further exploit investment opportunities in wellness would have a higher likelihood of success.

The contextual assessment of cost findings

Health economics concepts, such as produc- tions functions, may improve the design and in- terpretation of telemedicine cost analysis. Yet, the interpretation and synthesis of cost evalua- tions also requires an understanding of the de- cision-making assumptions and environmental context of the telemedicine activities that incur costs. Here, this is defined as the contextual as- sessment of the cost evaluation and includes both the decision-making context as represented in the “Cost Assessment Framework” and the environmental context of the telemedicine pro- gram that may impact its performance.

Cost evaluation findings on a telemedicine activity depend on such decision-making

ASSESSING TELEMEDICINE COSTS 361

5690_10_p348-369 7/6/05 2:17 PM Page 361

specifics as the perspective, the target popula- tion, and the decision time horizon. Contextual factors, such as provider fee levels, equipment costs, and health system efficiencies, may de- termine the external validity of findings. Al- though both the decision-making and environ- mental context of findings may limit the applicability of findings, an understanding and even separate analysis of these factors may fa- cilitate further synthesis of disparate findings. For example, adjustments to local provider costs may provide insights into observed dif- ferences in study results.

This section examines issues pertaining to contextual determinants of cost findings and ways in which understanding the context fa- cilitates the interpretation and synthesis of re- sults. Also, some research strategies that may address these contextual factors that can im- prove telemedicine cost evaluations are con- sidered.

Decision maker and information requirements

Existing telemedicine cost research and analyses rarely discuss the decision-making ob- jectives that would be supported by the results of a specific study. Neither do they explicitly define the informational needs of decision makers who would use such results. Nonethe- less, each of these is likely to have profound ef- fects on the design of the study, methods of sta- tistical inference, and the interpretation and usefulness of study results.

Many evaluations performed in the United States appear to define the research question so as to obtain cost-effectiveness information needed for directly funded government health- care delivery programs. But these seldom trans- late easily to the decision-making needs of mar- ket-driven institutions and third-party payers. For example, researchers frequently rely only on the societal perspective in evaluating costs and benefits. All costs and all benefits are con- sidered regardless of who pays the cost or gains the benefit. However, private health-care insti- tutional decision making will be based on a narrower set of costs and benefits, often, within the context of a business case that considers the costs and benefits related to institutional ex- penditures and revenues. A third-party insurer

will likely consider the costs and benefits quite differently. An insurer is likely to be especially interested in the effects of a reimbursement pol- icy on its costs and revenues.

At the same time, and perhaps inconsis- tently, the research designs in economic stud- ies of telemedicine appear to focus on local de- cision making, and do not address contextual variables and measurements that facilitate ap- plications in other settings. Hence, it is incum- bent on the analyst of truly generalizable re- search to address issues of comparability of costs and benefits, external validity, and rele- vance of research for various health-care deci- sion-making or policy objectives.

At the same time, this does not imply that a locally focused evaluation is of low quality, es- pecially if it provides information needed for decision making. The approach and findings limit their range of applicability. For example, differences in unit costs of inputs have been found to affect CEA ratios among European Union countries.42 An analysis by Drummond also led him to conclude that it is difficult to infer CEA results between countries. This prob- lem becomes even more difficult if countries vary in the resources used to produce a given result.43,44

The dilemma is that costs and benefits of re- search also have to be considered, and gener- alization has a cost. The optimal trade-off partly depends on the specific needs of deci- sion-makers and the expected scope of gener- alization of results.

Complexity of telemedicine interventions

Evaluating the costs and benefits of telemed- icine depends on identifying the resource basis used in producing telemedicine. Costs and ben- efits may be defined largely through the de- vices used, as in some telemonitoring. They may be affected by various factors, including sharing the technology for other functions and an array of work processes and external orga- nizational interactions, as in a community- wide telepyschiatry program. Yet, precisely how we specify the intervention sets the chal- lenge in measuring costs and benefits. For ex- ample, digital radiography is often cost-effec- tive separate from it use in telemedicine. It

REARDON362

5690_10_p348-369 7/6/05 2:17 PM Page 362

provides savings in the reduced cost of film and related supplies as well as through fewer lost images. But once the technology for digi- tal radiography is available, the cost of the ad- ditional systems to implement teleradiology is dramatically reduced. Of course, the benefits also become limited to gains from the electronic exchange of radiography. Nonetheless, despite increasingly widespread acquisition of digital imaging systems, there appears to be no com- prehensive estimate of the economic returns on investment.45

The definition that distinguishes the inter- vention from context may also delineate the po- tential generalization of findings. For instance, a narrowly defined intervention, such as a telemedicine device, may be more consistently identified, characterized for costs and benefits, and reproduced in new settings. Modifications or adjustments, such as in prices, may be needed to apply results in a new context. These are less likely to be a problem for decision mak- ers. In contrast, the greatest challenge in ap- plying economic evaluation methods is in pro- grams that have ambiguous boundaries of intervention and broadly defined or intangible outputs.46 For example, it would be far more challenging to identify clearly inputs and as- sociated input costs and to track outcomes in a broadly defined telehealth network. Similarly, programs that rely on extensive and complex interactions imbedded in organizational pro- cesses present difficulties in measuring costs and outcomes. On the other hand, telemedicine interventions that deliver the greatest cost sav- ings likely to be narrow and well defined, such as monitoring of congestive heart failure pa- tients, or home telecare.37,38

Methods for statistical inference

The study designs employed in economic evaluations include decision analytic models, ob- servational studies, and clinical trials. Observa- tional studies are more prevalent in the litera- ture. Prior to the mid-1990s, health-care cost analyses often used clinical trial data on effec- tiveness to develop a decision model for deter- mining costs and benefits. Within the last decade, several cost studies have been integrated with RCTs, but the majority do not do so.47

Perhaps this evolution explains the apparent preference for RCT or quasi-experimental de- signs, and the growing number of RCT-based studies. Nonetheless, these studies tend to rely on comparative evaluations of program out- come measures, and are mostly patterned after the clinical trials of medical care research. The presumed power of the RCT is that random- ization of patients to telemedicine and tradi- tional care settings assures unbiased compar- isons in the use of resources and outcomes. This is certainly the case when evaluating devices such as tele-EKGs. Still, the requirements of some research may suggest the use of alterna- tive methods of inference, either in combina- tion or as an alternative. This is because telemedicine interventions are often complex systems of devices and activities, the perfor- mance of which significantly depends on con- textual factors that are independent of the re- search treatment.

RCT designs in telemedicine economic re- search would be subject to significant external validity threats from economic determinants not controlled through randomized assign- ment of patients.48 For example, telemedicine interventions may involve extensive process and behavioral changes that are affected by health-care policy, local resources costs, re- source costs, and incentives that are beyond the boundary of the research randomization. De- spite the use of randomized, controlled trials, the systemic nature of the telemedicine pro- duction process may easily be affected by ma- jor contextual factors that will limit external validity of findings. Alternative, non-RCT pro- cesses and concept models may be available to reduce error variance and biased results. Se- lectivity models that better address beneficiary behaviors and policy maker needs when faced with new technologies have proven useful in other areas of health-care research. Techniques that combine trial-based and observational data, sample selection, and censored data esti- mation are among current major development across areas of health-care cost analysis.48

Consequently, the selection of the most ap- propriate methods of inference in telemedicine cost research depends on a range of multivari- ate statistical techniques to improve the effi- ciency of estimation and adjustment for selec-

ASSESSING TELEMEDICINE COSTS 363

5690_10_p348-369 7/6/05 2:17 PM Page 363

tion bias. These approaches may be used as al- ternatives or complements to RCT methods, especially as a means of revealing logical con- nections among inputs, context, and outputs and in leveraging and the generality of find- ings.

SUMMARY AND RECOMMENDATIONS

Lamenting the lack of information on telemedicine cost analysis has become a stan- dard preface to discussions of telemedicine evaluation. An ample volume and variety of explanations for the dearth of results are often cited. These include small user populations, rapidly changing technology, poor selection and application of statistical and economic methodologies, and inherent failings in avail- able economic evaluation methodologies.15,18,26

Still, researchers express a growing optimism in the value of telemedicine while government and health-care decision makers from many countries increasingly invest in telemedicine. Assuming rational behavior among the stake- holders, the evidence from these sentiments suggests that more convincing data on the costs and benefits of telemedicine exist. The incon- gruity of formal assessments and decision- maker behavior suggests the need for a more careful design and assessments of current pro- grams. The lack of convincing evaluations partly may reflect inadequate analytic assess- ment, integration, and synthesis of available data.

Cost evaluations of telemedicine have been undertaken in a diverse and global set of health-care delivery settings. Studies represent an economically heterogeneous set of pro- grams and systems of care analyzed as discrete intervention alternatives. As a result, study data represent a large number of observations, often with little apparent commonality, and the integration of findings is arduous at best. A few notable findings have been identified, but re- sults are meager for any specific technology or decision problem, and they are unlikely to pro- vide much useful information. Yet, little atten- tion has been paid to the unique methodolog- ical considerations and potential strategies for program cost evaluations. Economic and deci-

sion-making concepts provide a host of tools to interpret and synthesize findings that appear to have been overlooked. These provide added insights into telemedicine program cost results once integrated meaningfully.

This paper was aimed at understanding available cost data, not simply from an assess- ment of the aggregate validity of statistical es- timates. Statistically valid findings are limited. An attempt was made to use economic theory to understand the fragmentary and sparse find- ings. Nonetheless, an interpretation within an economic theoretical structure and applied de- cision-making framework provides additional corroborating evidence on the efficiencies and potential value of telemedicine. At the same time, these assessments provide insights as to potential design of telemedicine programs and further cost analysis research.

This review of telemedicine cost studies sup- ports previous conclusions on the potential net savings to society through certain uses of telemedicine. Populations in remote areas, in prisons, or on ships may have reduced total cost of care by accessing it through telemedi- cine. But these reported gains in the cost-effec- tiveness of telemedicine depend on the reduced cost of access to care.

Distances and other barriers make the cost of travel to obtain care inordinately high, and the change in output with telemedicine has been along the dimension of access as opposed to changes in health. The relatively lower cost-to- benefit performance of these programs de- pends on changes in situational variables, such as geography, as well as reductions in the cost of information technology. Perhaps most im- portantly, savings from the relatively favorable cost performance of these programs may not accrue to the payers of care. Hence, the health plans on Europe, Canada, the U.S. Department of Defense, and prisons that pay for access may have incentives to provide this type of telemed- icine. Otherwise, most U.S. plans do not pay the costs of barriers to access, and in fact may gain from these barriers.

On the other hand, generalizable telemedi- cine savings from clinical care efficiencies, as opposed to access savings, may be observed in studies of specialty areas that have a potentially modifiable clinical production process through

REARDON364

5690_10_p348-369 7/6/05 2:17 PM Page 364

telemedicine and that have a research design that allows for the detection of the potential gains. Cost evaluations are more likely to de- tect improvements in clinical efficiencies if out- puts are at least measured as episodes of care. Both dermatology and psychiatry appear to have the potential for clinical efficiencies through telemedicine, but this is more clearly accomplished in store-and-forward dermatol- ogy where the process of care is most clearly changed. In telepsychiatry, changes in pro- cesses of care to take advantage of telemedicine are less clearly measurable.

The greatest potential for cost savings from telemedicine appears to be in its application in the production of health or wellness. When study outcomes are measured as health main- tenance or wellness, as is usually done in home care, potential savings from telemedicine ap- pear to be much greater. In particular, this type of telemedicine may be especially cost effective for high-risk and specific types of chronically ill patients.

The application of the economic concepts of production and contextual analysis and syn- thesis of cost-evaluation methodologies and findings appear to make better use of the cost literature. But the use of economic production concepts and the cost-assessment frameworks and the environmental determinants of pro- gram performance may make their greatest con- tribution in helping to define more innovative future research. Areas of likely payoff are stud- ies of self-selection into telemedicine programs, adjustments for local health-care costs and sys- tem efficiencies, and the impact of incentives on specialties such as telepsychiatry where poten- tial but difficult-to-measure changes in clinical efficiencies may occur.

Generally, the need to integrate current costs findings seems evident, and greater reliance on economic and decision-making structures seems desirable. Telemedicine technology is frequently considered a “black box.” But telemedicine sys- tems of care have important behavioral deter- minants and effects that may both better un- derstood and evaluated through economic and health-care delivery concepts and through al- ternative approaches to assessing causality. Al- though the specific choice of methods of infer- ence and design depend on specific research

study needs and constraints, more flexible meth- ods of establishing causality often may be justi- fied and sometimes needed so as to aid inter- pretation and generalization of findings. Hence, a strict reliance on randomized, controlled trials for cost analysis is questionable.

Finally, telemedicine cost evaluation occu- pies a difficult divide between evaluation hav- ing a focus on providing information for local decision makers, and a focus on providing gen- eralizable estimates of cost impact. General findings are needed to improve effectiveness and reduce costs to the broader community of decision makers; however, these likely have re- search costs and trade-offs that may not im- mediately benefit a researcher’s constituency.

Consensus guidelines and best practices on costs provide common benefits to cost research, but they are unlikely to be supported by indi- vidual research or provider groups. Both the support and expansion of the common research contributions provided by guidelines and the development of strategies for addressing the balance of the private and shared-cost research costs and benefits question may make the greatest single contribution in enhancing the value of telemedicine cost literature.

APPENDICES

Appendix 1: Methods of health-care cost analysis

Cost–benefit analysis (CBA): CBA remains the paradigm for conceptualizing cost analysis, and it continues to serve as the ideal in defin- ing opportunity costs and the valuation of ben- efits. In classic cost–benefit analysis, all input costs and benefits are transformed into mone- tary terms, such as dollars, as would be the case for individual decision-making in the market. The operational impediments to CBA are well documented for problems dealing with per- sonal health-care services where the most sig- nificant issue may be the lack of generally ac- cepted methods for translating benefits into dollars. Although various willingness-to-pay methodologies have been used for CBA of health services programs, ongoing public and economic debate over their use makes them un- appealing.49

ASSESSING TELEMEDICINE COSTS 365

5690_10_p348-369 7/6/05 2:17 PM Page 365

Cost-effectiveness analysis (CEA/CUA): The de- velopment of CEA was stimulated by the dis- comfort in CBA when translating medical out- comes such as years of life or a change in blood pressure into monetary values. Whereas costs are monetary in a CEA, outcomes are reported in nonmonetary terms. In personal health-care, these terms are expressed in such intermediate metrics as the outcomes of blood pressure or recidivism or such longer-term out- comes as quality-adjusted life years (QALYs), where a quality year may be partially defined by health.4,7 A CUA is particular type of CEA that uses QALYs as an outcome. The resulting cost per QALY provides a common currency that is especially helpful for comparisons among disparate technologies.5

The objective in a typical CUA is to estimate the number of quality-adjusted life years pro- duced through monetary expenditures on spe- cific health-care interventions. QALY effects of an intervention are captured in the denomina- tor of the ratio, whereas changes in resource use for an intervention are captured in the nu- merator in monetary terms. A result, for ex- ample, may be that a program costs $10,000 per each additional life year. CUA ratios may be compared among a few alternatives or com- pared with a standard set of interventions, as from a league table that lists the cost per life year for a wide range of projects.50 Guidelines or “rules of thumb” for accepting interventions with CUA ratios, for example, of less than $100,000 per life-year, have been a subject of extensive health-care research.24

The Panel on Cost Effectiveness in Health and Medicine (United States Public Health Ser- vice) advocates use of the CUA form of cost-ef- fectiveness analysis.4

Cost-minimization analysis (CMA): Cost-mini- mization analysis (CMA) is essentially a cost- savings analysis. That is, the focus of the anal- ysis is on finding the alternative with the lowest cost; for example, the cheapest way to produce a medical visit or to treat a case. To the extent that telemedicine may not change an immedi- ate health outcome and constitutes only a change in process, then use of CMA is justified. On the other hand, if benefits do change, in- formation is lost and important findings may

be overlooked when using CMA rather than CBA or CEA.

Return on investment (ROI): In its narrowest definition, return on investment is defined as a mathematical relationship, where the total monetary benefit from an investment is di- vided by initial and subsequent costs.47 ROI is then stated as a percentage or ratio, and it pro- vides a concise summary of factors that define the gains from an investment. As a good com- parative summary of the profitability of an in- vestment, this metric is especially well suited for for-profit organizational decision making. Yet, ROI is not usually cited or discussed in health-care economic cost research because it is viewed as organizational expenditures as op- posed to opportunity costs, and its traditional methods do not include quantifying nonmon- etary costs and benefits. Moreover, the ROI problem has been focused on capital invest- ments for production, as opposed to consump- tion goods. That is, the organizational decision maker’s interest is on the firm’s monetary re- turns from an investment as opposed to the to- tal value of benefits provided to a consumer.

Nonetheless, the term ROI increasingly is be- ing applied to all cost-and-benefit methods.51 The broader definition encourages a compari- son of an investment’s total value and costs, in- cluding the consideration of nonmonetary ben- efits.47 For example, U.S. government guidance on information technology investments now defines ROI as encompassing the collection of quantitative cost/benefit methods, including CEA.47 But this question on the definition of ROI may not be clear, but broadening the con- cept has extended the focus of government an- alysts concerned with organizational decision- making tradeoffs, benefits, and costs. Then, while these government organizational expen- ditures still may not represent opportunity costs from the societal perspective, they repre- sent opportunity costs within the framework of an organizational budget. Moreover, resulting cost analysis approaches may be applied equally to incorporate nonmarket and oppor- tunity cost concepts in private business orga- nization decision making, especially analyses and policies of private health-care organiza- tions.

REARDON366

5690_10_p348-369 7/6/05 2:17 PM Page 366

Functional economic analysis (FEA): FEA is a variation of cost analysis developed to accom- modate the needs of information technology and organizational decision making.52,53 In- formation technology products, including telemedicine, may be defined as sets of func- tions or capabilities as opposed to hardware, software, or a physical system.53–55 FEA fo- cuses on the costs of the functions or capabili-

ties that facilitate specific health-care objec- tives, such as providing medical care services or improved access to care. Thus, costs and benefits are associated with specific functions or information capabilities, such as providing an electronic medical record capability. Orga- nizations may define decision-maker responsi- bilities and workflows that support functions to enable identification of costs.

ASSESSING TELEMEDICINE COSTS 367

Method Characteristics Disadvantages

Cost-benefit analysis Paradigm for cost analyses No generally accepted methods of (CBA) Defines opportunity costs quantifying many benefits as dollars

Optimizes social welfare Reluctance to express value of health or Broad comparisons possible life in monetary terms

Cost-effectiveness Outcomes reported in non-monetary terms Inability to compare findings across analysis/cost utility Outcomes may be expressed as QALYs health-care areas. analysis (CEA/CUA) Preferred by U.S. Public Health Service Baseline

consensus panel Generalizability of utility estimates Benchmarks and comparisons available in Costs of collecting utility data

major online databases (http://www. Findings indicating that utility data hsph.harvard.edu/cearegistry/) usually does not change a result

Cost-minimization Lowest cost option sought May overlook changes in benefits analysis (CMA) Less demanding data requirements

Return on investment Mathematical relationship: monetary Better suited to for-profit decision making (ROI) benefit divided by initial and Overlooks nonmonetary benefits

subsequent costs Focuses on capital investments for Facilitates investment analyses production, not consumption

Functional economic Facilitates analyses of information costs Need for conceptual development of analysis (FEA) and benefits functions

Benefits and costs derived for information Generalizability capabilities and functions

REFERENCES

1. Cameron AE, Bashshur RL, Halbritter K, Johnson EM, Cameron JW. Simulation methodology for estimating financial effects of telemedicine in West Virginia. Telemed J 1998;4:125–144.

2. Quinn-Patton M. The debate about randomized controls as the gold standard in evalution. National Institutes of Health, Videocasting: past events, http://videocast. nih.gov, 2004.

3. Hjelmgren J, Berggren F, Andersson F. Health eco- nomic guidelines: similarities, differences, and some implications. Value in Health 2001;4:225–250.

4. Garber A, Torrance G, Kamlet M. Theoretical foun- dations of cost-effectiveness analysis. In: MR Gold, JE Siegel, LB Russell, MC Weinstein, eds. Cost-effec- tiveness in health and medicine. New York: Oxford Uni- versity Press, 1996:25–53.

5. Akehurst R. Making decisions on technology avail- ability in the British National Health Service—why we need reliable models. Value in Health 2003;6:3–5.

6. Berki SE. In: Bashshur RL, Armstrong PA, Youssef ZI, eds. Telemedicine: explorations in the use of telecommuni- cations in healthcare. Springfield, IL: Charles C. Thomas, 1975, and reproduced courtesy of Charles C Thomas, Ltd., Springfield, Illinois.

7. Gold MR, Siegel JE, Russell LB, Weinstein MC, eds. Cost-effectiveness in health and medicine. New York: Ox- ford Press, 1996.

8. Sculpher M, Pang F, Manca A, Drummond M, Golder S, Urdahl H, Davies L, Eastwood A. Generalisability in economic evaluation studies in healthcare: a review and case studies. Health Technology Assessment, NHS R&D HTA Progamme 2004;8:124.

9. McKinsey Global Institute. US productivity growth 1995–2000: understanding the contribution of information technology relative to other factors. Washington DC: 2001.

10. Kennedy W, Laurier C, Malo J, Ghezzo H, L’Archevéque J, Contandriopoulos A. Does clinical trial subject selection restrict the ability to generalize use and cost of health services to “real life” subjects? Int J Tech Assess Health Care 2003;19:8–16.

Appendix 2: Comparison of cost evaluation methods

5690_10_p348-369 7/6/05 2:17 PM Page 367

11. Ijzerman M, Reuzel R, Severens H. Pre-assessment to assess the match between results and decision mak- ers’ information needs. Int J Technol Assess Health Care 2003;19:17–27.

12. Alderson P, Green S, Higgins JPT, eds. Cochrane re- viewers handbook 4.2.2. Chichester, UK: John Wiley & Sons, Ltd., 2004.

13. Neumann PJ, Olchanski NV, Rosen AB, Greenberg D, Chapman R, Stone PW, Nadai J. Are published cost-util- ity analyses improving? Boston: Harvard School of Pub- lic Health, 2004.

14. Moja L, Telaro E, D’Amico R, Moschetti L, Coe L, Liberati A. Assessment of methodological quality of primary studies by systematic reviews: results of the metaquality cross sectional study. Br Med J 2005;300:1053–1057.

15. Hailey D, Roine R, Ohinmaa A. Systematic review of evidence for the benefits of telemedicine. J Telemed Telecare 2002;8(Suppl 1):1–30.

16. Roine R, Ohinmaa, A., Hailey, D. Assessing telemed- icine: a systematic review of the literature. CMAJ, Can Med Assn J 2001;165:765–771.

17. Whitten PS, Mair FS, Haycox A, May CR, Williams TL, Hellmich S. Systematic review of cost effective- ness studies of telemedicine interventions. Br Med J 2002;324:1434–1437.

18. Ohinmaa A, Hailey D, Roine R. The assessment of telemedicine—general principles and a systematic review: a health technology brief: economic evaluation methods. Al- berta Heritage Foundation for Medical Research, 1999.

19. Hersh WR, Wallace JA, Patterson PK, et al. Telemedi- cine for the Medicare program. Oregon Health Sciences University AHRQ Publication No. 01-E012. Portland, OR: Oregon Health Sciences University, 2001.

20. Hersh WR, Wallace JA, Patterson PK, et al. Telemedi- cine for the Medicare population: pediatric, obstetric, and clinician-indirect home interventions. Oregon Health Sci- ences University AHRQ Publication No. 01-E060. Portland, OR: Oregon Health Sciences University, 2001.

21. Light R, Pillemer D. Summing up: the science of review- ing research. London: Harvard University Press, 1984.

22. Hakansson S, Gavelin, C. What do we really know about the cost-effectiveness of telemedicine? J Telemed Telecare 2000;6(Suppl 1):S133–S136.

23. Mair FS, Haycox A, May C, Williams T. A review of telemedicine cost-effectiveness studies. J Telemed Tele- care 2000;6(Suppl 1):S38–S40.

24. Drummond MF. The contribution of health economics to cost-effective health care delivery. New York: Churchill Livingston, 1993.

25. Brantley D, Laney-Cummings K, Spivack R. Innova- tion, demand, and investment in telehealth. Washington DC: United States Department of Commerce, Office of Technology Policy, 2004.

26. McCue MJ, Mazmanian PE, Hampton CL, Marks TK, Fisher EJ, Parpart F, Malloy WN, Fisk KJ. Cost-mini- mization analysis: A follow-up study of a telemedi- cine program. Telemed J 1998;4:323–327.

27. Zincone LH Jr, Doty E, Balch DC. Financial analysis of telemedicine in a prison system. Telemed J 1997;3: 247–255.

28. Stoloff PH, Garcia FE, Thomason JE, Shia DS. A cost- effectiveness analysis of shipboard telemedicine. Telemed J 1998;4:293–304.

29. Stoeger A, Strohmayr W, Giacomuzzi SM, Dessl A, Buchberger W, Jaschke W. A cost analysis of an emer- gency computerized tomography teleradiology sys- tem. J Telemed Telecare 1997;3:35–39.

30. Chodroff PH. A three-year review of telemedicine at the community level—clinical and fiscal results. J Telemed Telecare 1999;5(Suppl 1):S28–S30.

31. Doolittle G, Williams A, Harmon A, Allen A, Boysen CD, Wittman C, Mair F, Carlson E. A cost measure- ment study for a tele-oncology practice. J Telemed Tele- care 1998;4:84–88.

32. Burgiss SG, Julius CE, Watson HW, Haynes BK, Buonocore E, Smith GT. Telemedicine for dermatol- ogy care in rural patients. Telemed J 1997;3:227–233.

33. Monnier J, Knapp RG, Frueh BC. Recent advances in telepsychiatry: an updated review. Psychiatr Serv 2003;54:1604–1609.

34. Grady BJ. A comparative cost analysis of an inte- grated military telemental health-care service. Telemed J e-Health 2002;8:293–300.

35. Hilty DM, Marks SL, Urness D, Yellowlees PM, Nes- bitt TS. Clinical and educational telepsychiatry appli- cations: a review. Can J Psychiatry 2004;49:12–23.

36. Grossman M. The demand for health: a theoretical and empirical investigation, New York: National Bureau of Economic Research, 1972.

37. Benatar D, Bondmass M, Ghitelman J, Avitall B. Out- comes of chronic heart failure. Arch Int Med 2003;163: 347–352.

38. Johnston B, Wheeler L, Deuser J, Sousa KH. Outcomes of the Kaiser Permanente tele-home health research project. Arch Fam Med 2000;9:40–45.

39. Dawson A, Cohen D, Candelier C, Jones G, Sanders J, Thompson A, Arnall C, Coles E. Domiciliary mid- wifery support in high-risk pregnancy incorporating telephonic fetal heart rate monitoring: a health tech- nology randomized assessment. J Telemed Telecare 1999;5:220–230.

40. Loane MA, Bloomer SE, Corbett R, Eedy DJ, Evans C, Hicks N, Jacklin P, Lotery HE, Mathews C, Paisley J, Reid P, Steele K, Wootton R. A randomized controlled trial assessing the health economics of realtime tele- dermatology compared with conventional care: an ur- ban versus rural perspective. J Telemed Telecare 2001;7:108–118.

41. Jacklin PB, Roberts JA, Wallace P, Haines A, Harrison R, Barber JA, Thompson SG, Lewis L, Currell R, Parker S, Wainwright P. Virtual outreach: economic evaluation of joint teleconsultations for patients re- ferred by their general practitioner for a specialist opinion. Br Med J 2003;327:84.

42. Brown R, Hutton J, Nuijten M. Can unit costs be com- pared across Western European countries? Value in Health 2001;4:48.

REARDON368

5690_10_p348-369 7/6/05 2:17 PM Page 368

43. Drummond M, Barbieri M, Chancellor J, Willke D, Towse A, Jolain B. Are the results of economic eval- uations generalizable? Evidence from studies of evi- dence from studies of pharmaceuticals in Western Europe. Presented at the 4th International Health Eco- nomics Association World Congress, 2003.

44. Sculpher M. Techniques to analyse variability across countries in multi-national economic evaluations. Presented at the 4th International Health Economics Association World Congress, 2003.

45. Thompson T, Brailer, D. The decade of health informa- tion technology: Delivering consumer-centric and infor- mation-rich health care. Washington, D.C.: Health and Human Services, 2004.

46. Bassoli F. Justifying where the dollars go. Health Data Manag Feb. 1998.

47. Capital Planning and IT Investment Committee. ROI and the value puzzle. Washington, D.C.: Federal CIO Council, 1999.

48. Glick H. Statistical methods in economic evaluation. Phil- adelphia: University of Pennsylvania, 2003.

49. Strosberg MA, Wiener JM, Baker R, Fein IA, eds. Ra- tioning America’s medical care: The Oregon plan and be- yond. Washington DC: The Brookings Institution, 1992.

50. Torrence GW, Siegel JE, Luce BR. In: Gold MR, Siegel JE, Russell LB, Weinstein MC, eds. Cost-effectiveness in health and medicine. New York: Oxford University Press, 1996:54–81.

51. United States Government Accounting Office. De- partment. Executive guide—measuring performance and demonstrating results of information technology invest- ments. Washington, D.C.: 1998.

52. Snodgrass NB, Jones PC, Meyer S, Roos DC, Safran C. Functional economic analysis: a technique for iden- tifying technology investment opportunities. Pre- sented at the The Health Information Management Systems Society (HIMSS), 1997.

53. Casson M. Information and organization: a new perspec- tive on theory of the firm. New York: Oxford University Press, 1997.

54. Simon H. The sciences of the artificial. Cambridge, MA: MIT Press, 1969.

55. United States Department of Defense. Functional eco- nomic analysis (FEA) guidebook. Electronic College of Process Innovation. http://www.dtic.mil, 2003.

56. Pearce DW, ed. The MIT dictionary of modern econom- ics. 3rd ed. Cambridge: The MIT Press, 1986.

Address reprint requests to: Tim Reardon, Ph.D.

TG Reardon & Associates, LLC 1232 Prospect Avenue Ann Arbor, MI 48104

E-mail: [email protected]

ASSESSING TELEMEDICINE COSTS 369

5690_10_p348-369 7/6/05 2:17 PM Page 369

This article has been cited by:

1. Michael Ries. 2011. Tele-ICU: A New Paradigm in Critical Care. International Anesthesiology Clinics 47:1, 153-170. [CrossRef] 2. Micaela Brown , Nicola Shaw . 2008. Evaluation Practices of a Major Canadian Telehealth Provider: Lessons and Future

Directions for the FieldEvaluation Practices of a Major Canadian Telehealth Provider: Lessons and Future Directions for the Field. Telemedicine and e-Health 14:8, 769-774. [Abstract] [PDF] [PDF Plus]

3. Kimmo Murto, Gregory L Bryson, Ibrahim Abushahwan, Jim King, David Moher, Khaled El-Emam, William Splinter. 2008. Parents are reluctant to use technological means of communication in pediatric day care. Canadian Journal of Anesthesia/Journal canadien d'anesthésie 55:4, 214-222. [CrossRef]

4. Mengistu Kifle, Victor W. A. Mbarika, Pratim Datta. 2006. Interplay of cost and adoption of tele-medicine in Sub-Saharan Africa: The case of tele-cardiology in Ethiopia. Information Systems Frontiers 8:3, 211-223. [CrossRef]

5. Eckart Fleck, Claus W Biermann, Uta Bruns. 2006. FOREWORD. Disease Management & Health Outcomes 14:Suppl 1, vi???viii. [CrossRef]

6. Penny Jennett, Mamoru Watanabe. 2006. Healthcare and Telemedicine: Ongoing and Evolving Challenges. Disease Management & Health Outcomes 14:Suppl 1, 9???13. [CrossRef]