Health Information Technology - Evaluation Plan Project - Evaluation Focus

BrenderandJaytte_Chapter-10_Pg_266--269.pdf

Home >Nursing homework help >Health Information Technology - Evaluation Plan Project - Evaluation Focus

I-IANDE~OOK OF: EVALUATION METIdODS

10. Approach to Identification of Pitfalls and Perils

A b i a s - "the arrival at a conclusion that differs sys tematically f r o m the t r u t h " (Jaeschke and Sackett 1989; Altman et al. 2001) - is the result of one or more experimental flaws within the design, accomplishment, or interpretation of the assessment study.

"What is r e q u i r e d . . , is the recognition o f potential confounders, followed by the incorporation o f specific research design (or "architectural") strategies to reduce their impact and thus avoid bias."

(Jaeschke and Sackett 1989).

"Bias is usually defined as a prejudice or partiality, whether conscious or not. This is in contrast to systematic error where no prejudice exists ".

(DeMets 1997)

In this paper, emphasis is on addressing all kinds of (unintentional) biases a n d systematic errors - together called 'pitfalls and perils'

In order to achieve a candidate list of experimental flaws at assessment of IT- based solutions in healthcare, a synthesis and an abstraction was made on literature's description of paradigmatic experimental perils and pitfalls within medical science, natural science, and social science. Sources of inspiration are numerous; nevertheless, the main ones remain the following review papers: (Jaeschke and Sackett 1989; Wyatt and Spiegelhalter 1991; Fraser and Smith 1992; Sher and Trull 1996; Friedman and Wyatt 1997; and Coolican 1999). These reviews address different experimental approaches like field studies, cohort studies and case studies. Wyatt and Spiegelhalter (1991) explicitly and only address the experimental pitfalls and perils at the assessment of knowledge-based decision-support systems, comprising a specific subset of IT-based systems. The detailed interpretation of the meaning of each pitfall and peril within the original context as well as further examples are to be found in the original references.

The rationale of the basic approach has been to make a structured framework to enable and structure a fair and comparable analysis, based on scientific principles and documented knowledge. The structure was developed as an analogue to the SWOT technique by inclusion of a large number of experimental perils and pitfalls reported in scientific literature.

Brender, McNair, Jytte, and Jytte Brender. Handbook of Evaluation Methods for Health Informatics, Elsevier Science & Technology, 2006. ProQuest Ebook Central, http://ebookcentral.proquest.com/lib/waldenu/detail.action?docID=306691. Created from waldenu on 2022-03-25 22:00:29.

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

HANDt~OOI< OF: ~VALUATION M~TIdODS

The sole purpose of the framework is that of structuring the information seeking and analysis to secure an objective analysis. The framework is inspired by and structured as analogous to a SWOT analysis, while a transformation of the four main concepts of a SWOT analysis has been tumed into similar concepts of particular relevance for assessment studies. The framework was applied for verification purposes on a number of assessment studies within the CANTOR (HC4003) Healthcare Telematics Project under the EU Commission's Fourth Framework Programme, based on artificial experimental setups. Following the application in CANTOR, the framework was refined to accommodate the experiences achieved.

Finally, a review of case studies reported in the literature has been completed to add published examples to the identified types of flaws, thereby making it plausible whether analogies exist between the known types of experimental flaws and those observed in the literature on real-life assessment of IT-based solutions in healthcare. Within this process the contents of the framework was gradually refined and incrementally extended to accommodate new types of flaws identified.

An extensive search of methods and methodologies as well as case studies on assessment of IT-based system/solutions was accomplished previously; see (Brender 1997a, 1998, and 1999). The strategy then was to search for all combinations of concepts of 'assessment' (with synonyms: technology assessment, evaluation, verification, and validation), 'computer system' (with synonyms: IT, IT system, computer, computer-based system), 'methods' and 'methodologies', and 'case studies'. This search strategy was supplemented by search on a number of key players of the domain. The literature databases searched were PubMed and INSPECT, plus relevant databases of the local research libraries. Also included in the original literature study were two extensive databases of relevant literature that were acquired specifically for the assessment of knowledge-based systems and decision-support systems; see (O'Moore et al. 1990b; and Talmon and van der Loo 1995).

The above literature study was repeated to find newer cases and methods, with extensive use of PubMed's facility 'related references' for key papers, explicit search on authors of notable papers, and elicitation of key references from the notable papers. In case of difficulty in retrieving a paper, little effort was put into pursuing short publications (i.e., 2-4 pages), like proceeding papers. The reason being a general experience of lack of sufficient detail of the method and description of the results for the present purpose.

By application of this repeated and multiapproach strategy, the likelihood of the completeness of the framework being rather exhaustive with respect to types of

?50

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

~ANICSOOK Og EVALUATION ME]TIdODS

experimental errors is rather high and depends mainly on the author's ability to identify all relevant (types of) flaws.

However, please note that reporting within this publication only includes examples of the best cases and bad examples to learn from or otherwise particularly relevant or paradigmatic examples out of the bulk of publications acquired. Thus, this publication does not reference the entirety of literature on assessment studies.

751

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

This Page Intentionally Left Blank

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .