Comparison of Key Evaluation Models in Health Informatics

Chapter-7-Descriptionofmethodsandtechniques-2.pdf

Home >Nursing homework help > Comparison of Key Evaluation Models in Health Informatics

~ANDt~OOI< O~ EVALUATION METHODS

based solution, it makes a huge difference to how you proceed. Please see the list of references with regard to additional descriptions of procedures.

Assumptions for Application It is obvious that a measurement of fulfillment of the objective assumes that a strategic objective has been defined and can be made operational in terms of relevant and realistic m e a s u r e s - that is, that formulas (metrics) can be established for their measurement.

Perspectives Few people realize the wide range of consequences an IT-based solution can have even beyond the organization normally considered as the user of the system: On implementation of a (new) IT system, it is nearly always necessary to make compromises (a bit of give and take). This is not necessarily bad, as over time many organizations develop very specific ways of doing things in a never-ending number of variations that often cannot be handled by the IT system. Some forms o f rationalization will therefore typically result from the implementation of an IT system. This rationalization will impact on something or somebody, and therefore there is a risk that the changes within the user organization will reach far and wide into the surrounding organizations.

Once introduced into daily operation, the organization keeps on c h a n g i n g - initially as it gets accustomed to and adjusts to the system. One danger (and a pitfall) is to start measuring the effect too soon after the daily operation has started before a degree of stability has been achieved. On the other hand, after a certain period in operation staff will start using their creativity, and new ways of doing things will develop spontaneously- both appropriate ones and inappropriate ones, intentional ones and unintentional ones (as shown in van Gennip and Bakker 1995 or in Part III, Section 11.1.6.9). This phenomenon only reflects the development pattern from novice to expert: You start by following the prescribed way of doing things, but while gaining experience at a proficient level, you get to know the shortcuts (Dreyfus 1997). You could, for instance, compare the note-taking in patient records by a newly graduated physician to that of a consultant. This phenomenon may make comparisons of controlled studies and before and after studies rather difficult.

Frame of Reference for Interpretation The strategic objective may influence all aspects of an organization,

~s6

Brender, McNair, Jytte, and Jytte Brender. Handbook of Evaluation Methods for Health Informatics, Elsevier Science & Technology, 2006. ProQuest Ebook Central, http://ebookcentral.proquest.com/lib/waldenu/detail.action?docID=306691. Created from waldenu on 2022-03-06 01:57:27.

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

~ANI38OOK Og EVALUATION M~TNODS

right from (partly) financially, matters o f responsibility and competence, quality objectives for patient care or the overall service, overall to the number o f records missing or patient and staff satisfaction. Whether there is a need for before-values (baseline data) depends on the individual study.

IT systems that do not have a written defined strategic objective could possibly have an objective defined at a lower, tactical level (for instance, in terms o f the interim objectives that are expected to lead to the overall strategic objective), and this may perfectly well replace the strategic objective as a frame o f reference or they can form the frame o f reference together.

The frame o f reference must therefore be aligned with the real objective and to the conditions o f the actual study. In order to describe the conditions adequately, the objective will usually need to be covered by a whole range o f indicators (measures).

Perils and Pitfalls A consequence o f the extremely dynamic situations occurring in connection with the introduction o f an IT-based system could be the loss o f control o f what some o f the effect indicators actually signify. Before-and-after assessment, as, for instance, in those recommended in (Nyttevcerdi 2002), have inbuilt pitfalls in the shape o f the validity o f the frame o f reference. One must be sure that the concepts used are the same before and after in case they are directly measured and compared. It is often necessary to use different questionnaires for the before-and-after situations, as the work processes and many other aspects are different, thus making a direct comparison difficult.

It is difficult to separate the effect o f different variables: For instance, what is the effect o f having done anything at all? Simply focusing on an organization over a period o f time, asking lots o f questions about what and how things are done, has tremendous effect. How does one separate the effect arising purely from the fact that something is happening from the real (intentional) effect o f the IT system or the IT solution? If the effect is caused by the first possibility, there is a strong risk that given time it will revert toward the starting point to a certain degree.

157

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

I----IANDt~OOK OF EVALUATION METHODS

Advice and C o m m e n t s See also under the method F u n c t i o n a l i t y assessment, which has been specifically designed with the evaluation of fulfillment of the objective o f an IT-based solution in mind.

The approach in (Clements et al. 2002) of a Utility Tree with a hierarchical breakdown from overall interest areas to specific measures might be valuable for narrowing the focus of an Impact Assessment.

References

Clements P, Kazman R, Klein M. Evaluating software architectures, methods and case studies. Boston: Addison-Wesley; 2002.

Dreyfus HL. Intuitive, deliberative and calculative models of expert performance. In: Zsambok CE, Klein G, editors. Naturalistic Decision Making. Mahwah: Lawrence Erlbaum Associates, Publishers; 1997. p. 17-28.

Krogstrup HK. Evalueringsmodeller. Arhus: Systime; 2003. (in Danish) This little instructive book gives a good overview o f applications and limitations o f a couple o f known assessment models. Although it originates in the social sector it is inspiring with regard to Impact Assessment and assessment o f performance.

Nyttev~erdi af EPJ, MTV-baseret metode til m~ling af nytteva~rdien af elektronisk journal. Roskilde: Amtssygehuset Roskilde, DSI og KMD Dialog; 2002. (in Danish)

This little book gives an excellent and quick overview o f how to carry out an actual Impact Assessment (as in the usefulness) o f an E P J including quantitative indicators, questionnaires, and the procedure.

van Gennip EM, Bakker AR: Assessment of effects and costs of information systems. Int J Biomed Comput 1995;39:67-72.

A valuable case study to learn from.

Supplementary Reading

Bates DW, Teich JM, Lee J, Seger D, Kuperman GJ, Ma'LufN, Boyle D, Leape L. The impact of computerized physician order entry on medication error prevention. JAMIA 1999;6(4):313-21.

A case study usingprospective time studies to assess the effect o f

136

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

IdANDt~OOI< O~ -I=VALUATION I"Zl~TIdODS

the introduction o f an IT-based solution on a given activity.

Brinkerhoff RO, Dressier DE. Productivity measurement: a guide for managers and evaluators. Newbury Park: Sage Publications. Applied Social Research Methods Series 1990. vol. 19.

Measurement o f productivity as a function o f input and output variables (including identification o f measures) is extremely important in many studies o f objectives fulfillment and assessments o f effect. This book is useful as inspiration f o r that purpose.

Collinson M. Northern province HIS evaluation, defining an evaluation framework- workshop 24 July 1998. (Available from: http://www.sghms.ac.uk/depts./phsdaceu/safrica2.htm. Last visited 31.05.2005.)

The report illustrates a fairly thorough and effective way in which to get relevant measures.

Friedman CP, Abbas UL. Is medical informatics a mature science? a review of measurement practice in outcome studies of clinical systems. Int J Med Inform 2003;69:261-72.

A thorough and critical review o f studies that measure different types o f effect. It includes a very useful list o f earlier studies measuring effects within the healthcare sector.

Garrido T, Jamieson L, Zhou Y, Wiesenthal A, Liang L. Effect of electronic health records in ambulatory care: retrospective, serial, cross sectional study. BMJ 2005;330:581-5.

A fairly exhaustive, retrospective case study applying administrative data to assess usage and quality o f care before and after implementation o f an EHR.

Hagen TP. Demokrati eller effektivitet: hvad skal vi evaluere? In: Foss O, Monnesland J, editors. Evaluering av offentlig virksornhet, metoder og vurderinger. Oslo: NIBR; 2000. Report No.: NIBRs PLUSS-SERIE 4-2000. p. 79-110. (in Norwegian)

Discusses a number o f effectiveness measures, including that o f "through whose "eyes' is the organization being observed".

Hailey D, Jacobs P, Simpson J, Doze S. An assessment framework for telemedicine applications. J Telemed Telecare 1999;5:162-70.

This article outlines a list o f effect measures that can be used universally, even though the design originally is f o r tele-medicine purposes.

Kimberly JR, Minvielle E. The quality imperative, measurement and

Is9

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

~AND~OOI< OF EVALUATION METtdODS

management of quality in healthcare. London: Imperial College Press; 2000.

This anthology o f quality in the healthcare sector covers many methods and problems concerned with quality measurements and many o f its chapters are inspirational with regard to effect measurement.

Milholland DK. Information systems in critical care: a measure of their effectiveness. In: Greenes RA, Peterson HE, Protti DJ, editors. Medinfo'95. Proceedings of the Eighth World Congress on Medical Informatics; 1995 Jul; Vancouver, Canada. Edmonton: Healthcare Computing & Communications Canada Inc; 1995. p. 1068-70.

Development and verification o f a tool to measure the effect in terms o f a fulfillment o f the objective o f efficiency. Although it is only a proceeding, it includes valuable directions to useful statistical tools to verify the metrics and the results o f the analysis.

Mitchell E, Sullivan F. A descriptive feast but an evaluative famine: systematic review of published articles on primary care computing during 1980-97. BMJ 2001 ;322:279-82.

One o f the very f e w systematic reviews o f the effect o f I T within the primary healthcare sector. It is a supplement to (Sullivan and Michel11995) and particularly valuable to include, as it is critical and encompasses many assessments o f effect and references to studies with good case studies o f this particular measurement o f effect type. The authors do this without automatically keeping the non-RCT studies apart; instead they create a quality score applying a Delphi method

Sullivan F, Mitchell E. Has general practitioner computing made a difference to patient care? a systematic review of published reports. BMJ 1995;311:848-52.

A review containing references to numerous case stories about the effect o f I T on general practice. The article addresses the best studies o f patient outcome effect, the effect o f the consultation process, and clinical performance, respectively.

Vabo SI. Kritiske faktorer for evalueringer av kommunale reformer. In: Foss O, Monnesland J, editors. Evaluering av offentlig virksomhet, metoder og vurderinger. Oslo: NIBR; 2000. Report No.: NIBRs PLUSS-SERIE 4-2000. p. 139-82. (in Norwegian)

Discusses specific indicators f o r the measurement o f attitudes and behavior.

van der Loo RP, van Gennip EMSJ. Evaluation of personnel savings through PACS: a modelling approach. Int J Biomed Comput 1992;30:235-41.

1/40

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

~ANDt~OOK OF EVALUATION b'I{~TIdODS

Outlines a diagramming and modeling method where all divergences between before-and-after descriptions o f work processes are used as a baseline f o r a quantitative assessment o f the effect o f a PACS system on staff resources.

ZoE Stavri P, Ash JS. Does failure breed success: narrative analysis of stories about computerized provider order entry. Int J Med Inform 2003;70:9-15.

Uses a narrative analysis approach to retrospectively elicit aspects o f success and failure o f an IT-based solution. This approach might be useful f o r narrowing the scope o f an impact assessment through the identification o f areas o f interest.

1/41

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

ldAND~OOI< OlZ EVALUATION METHODS

Interview (Nonstandardized interviews)

Areas of Application This method is frequently used for qualitative studies o f subjective as well as objective circumstances. Interviews are particularly suited for the elucidation of individuals' opinions, attitudes, and perceptions regarding phenomena and observations. This is especially the case when non- or semistructured techniques or group techniques are being used to promote dynamic interaction.

Description Interviews are methods used to carry out a type o f formalized conversation that can be:

�9 Structured at different levels (structured, semistructured, and unstructured)

�9 Controlled to a greater or lesser extent by the interviewer �9 Conducted with individuals or in groups

The book by Fowler and Mangione (1990) (and a number of similar books) is indispensable for those who want to make an effective interview study. It describes all aspects o f the method and discusses what can diminish the value (precision, accuracy) o f the results and thereby the reliability of the study. It is necessary to examine thoroughly the different methodological considerations for each step from preparation (choice of theme, design) to completion o f the interview, transcription and analysis of data, and verification.

An often used type o f group interview is the Focus Group Interview, where brainstorming techniques can be useful to stimulate the dynamics and the creativity of the participants. See separate description o f this variation of interviews.

1/42

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

HANID~OOI< O~ EVALUATION MI~TIdODS

Assumptions for Application The flexibility o f un- or semistructured interview methods makes it necessary to be experienced in order to achieve a reliable result from the study. The level o f reliability (precision and accuracy) required is, however, the deciding factor. This is the case for both individual and group interviews.

Perspectives The methods are based on conversation between people, but it is not a dynamic dialogue o f equals between the interviewee and the interviewer. Therefore, there are a number o f social and psychological factors and pitfalls that have to be taken into consideration during the interaction.

Silverman (2001) describes the different perspectives behind approaches for interviews: Positivism, emotionalism, and constructionalism. In short, positivists acknowledge that interviewers interact with the interviewees giving facts or beliefs about behavior and attitudes and demand that this interaction is defined in the protocol; emotionalists recognize that interviews are inescapably encounters between subjects, while eliciting authentic accounts o f subjective experience; constructionalists see the interview as a focused interaction with its own right, while dealing with how interview participants actively and mutually create meaning.

Frame of Reference for Interpretation In interview studies, there is normally no frame o f reference in the usual sense, but it is quite possible to interview a group o f people about the same s u b j e c t - for instance, before-and-after the introduction o f an IT-based s y s t e m - and thereby get a feeling o f the development or change with regard to a specific question.

Perils and Pitfalls See (Robson 2002) and Part III in this handbook.

The influence o f the interviewer(s) is a common source o f errors (Rosenthal 1976; Fowler and Mangione 1990). It is significant whether or not the interviewer is known to, and respected in, the organization, whether the interviewer holds respect for the organization and its needs for privacy and anonymity, and whether the interviewer has prior knowledge o f the domain. It is significant whether there are one or more interviewers, as they may not

1/43

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

~AND6~OO~ OF:: -~VALUATJQN METNODS

necessarily get the same result. And it is significant whether you interview staff, middle management, or executive management, as the two latter groups are used to disregarding their own personal opinions in favor o f the official politics and principles o f the organization.

The way the questions are posed is important for the precision o f the answers, as are the procedures under which the interviews are carried out (Fowler and Mangione 1990). Questions may be seen as being threatening (Vinten 1998) or touch on (culturally based) taboos.

A potential pitfall when digging into historical information is postrationalization, see Part III.

There is also a risk o f a bias in quantitative studies if the user's ability to assess assumptions and correlations is challenged (see Part III).

Last but not least, it is important to realize that users in an organization find it extremely difficult to account for how they actually carry out their activities (see the discussions in Brender 1997a and 1999).

Advice and C o m m e n t s The level o f difficulty in interview methods is often underestimated, and reliability as well as internal and external validity o f the outcome is correlated to the experiences as interviewer. However, this should not deter from using interview methods, but one has to get to know the methods in depth before starting and put their quality aspect up against the objective o f the study. This is certainly important if the result o f the research is to be published in the scientific literature. However, there is a lot o f easily accessible literature about the subject.

If there is a particular need to measure the validity o f one's conclusions from an interview study, triangulation o f the method or o f specific observations can be made, as described in Part III.

References

Brender J. Methodology for assessment of medical IT-based systems- in an organisational context. Amsterdam: IOS Press, Stud Health

1/44

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

IdANID~OOI< O~ ~]VALUATION METHODS

Technol Inform 1997;42.

Brender J. Methodology for constructive assessment of IT-based systems in an organisational context. Int J Med Inform 1999;56:67-86.

This is a shortened version o f the previous reference, so it does not go into as much depth and is more accessible.

Fowler Jr FJ, Mangione TW. Standardized survey interviewing: minimizing interviewer-related error. Newbury Park: Sage Publications. Applied Social Research Methods Series 1990. vol. 18.

This book is indispensable f o r those who want to make an effective interview study. It describes all aspects o f the method and discusses what can diminish the value (precision, accuracy) o f the results and thereby the reliability o f the study.

Robson C. Real world research, a resource for social scientists and practitioner-researchers. 2nd ed. Oxford: Blackwell Publishers Inc; 2002. p. 269-91.

This book gives a good description o f different types o f interviews, details, and ideas regarding their use including the advantages and disadvantages o f each o f them.

Rosenthal R. Experimenter effects in behavioral research, enlarged edition. New York: Irvington Publishers, Inc.; 1976.

This rather unsettling book reviews the impact o f the experimenters on their test objects,_including biosocial attributes, psychosocial factors, and situational factors (the context o f the studies).

Silverman D. Interpreting qualitative data, methods for analysing talk, text and interaction. 2nd ed. London: Sage Publications; 2001.

Vinten G. Taking the threat out of threatening questions. J Roy Soc Health 1998; 118(1): 10-4.

Supplementary Reading There are a number o f similar descriptions o f research methodologies f o r research within the social sciences including interview techniques. The advice they provide is as good as the advice provided by the above references.

Ericsson KA, Simon HA. Protocol analysis, verbal reports as data. Cambridge (MA): The MIT Press; 1984.

This book provides a thorough review o f the literature on approaches and problems at the elicitation o f information about

1/45

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

IdANDBOOI< OF EVALUATION METHODS

cognitive processes from verbal data.

Harrison MI. Diagnosing organizations: methods, models, and processes. 2nd ed. Thousand Oaks: Sage Publications. Applied Social Research Methods Series 1994. vol. 8.

The book is concerned with a number o f aspects o f an organization, from individuals to their internal relationships (as in power structures, etc.), and discusses advantages and disadvantages o f different methods in this respect, including those o f interviews. It describes a number o f factors with regard to individuals as well as groups in an organization.

Leavitt F. Research methods for behavioral scientists. Dubuque: Wm. C. Brown Publishers; 1991.

Contains quite concrete guidelines on how to formulate questions and put them together (also in respect o f questionnaires) and how to avoid the worst pitfalls (see pages 162-172).

http://jthom.best.vwh.net/usability, This contains lots o f summaries o f methods and references, including some inspiration regarding interview studies (last visited 31.05.2005).

J46

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

IdANDt~OOI( OE EVALUATION METHODS

KUBI (From Danish: "_.KvalitetsUdvikling gennem Brugerlnddragelse"; translated: Quality Development Through User Involvement)

Areas of Application

The method is used as a tool for the incremental optimization of the outcome of a longterm development project, based on a set of user or customer/client defined value norms and objectives.

Description The method originates in the social sector, where it was developed as a constructive tool to improve conditions in the healthcare and social sectors (Krogstrup 2003). It has many points in common with the Balanced Scorecard method and is applied to assess the degree of fulfillment and subsequently to balance and re-focus on areas for improvement. However, the KUBI method has some quite different success indicators as its driving force for development.

The 1.

procedure follows these steps: Establishment of values and criteria for the primary stakeholder group(s) (such as a user, customer, or client)

2. Interview phase during which selected members of the primary stakeholder group(s) are trained and carry out individual and group interviews under supervision

3. Summation of the interview data into a report that includes an assessment of the degree of fulfillment of user criteria

4. Preparation of plans for the areas for improvement and development initiatives following a discussion of the report with the stakeholders

5. Follow-up of the new situation with the stakeholders after about a year, followed by a discussion of whether the development plans and initiatives need revision or expansion

Assumptions for Application

1/47

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

IdANDBOOI< OF EVALUATION MI~TIdODS

Perspectives The intense focus on the user-stakeholder group is somewhat unusual in IT development and implementation processes because it belongs to another world with other traditions and focal points. Nevertheless, the method has been included as it may provide inspiration in general terms. For example, one could imagine that it could be used in a slightly modified tbrm as a hearing tool for very large IT projects, such as assessment o f a regional or national strategy or implementation o f an EHR where the distance from project management to the user in the field can be quite great.

Frame of Reference for Interpretation The frame o f reference is the objectives established during the first phase o f the procedure.

Perils and Pitfalls

Advice and Comments

References

Krogstrup HK. Evalueringsmodeller. Arhus: Systime; 2003. (in Danish) Even i f this reference is in Danish, the above description should enable experienced evaluators to apply the method or apply it in their own version, with benefit.

148

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

IdAND~OOI< OF- EVALUATION METHODS

Logical Framework Approach

Areas of Application �9 Situation analysis either in general and at any time within a

project �9 Support the choice o f action prior to planning of the

development effort �9 Incorporating risk handling within project planning

Description Logical Framework Approach (LFA) is an objectives-oriented_plan- ning methodology. The authors call it a framework, but it serves as a methodology because it describes the overall sequence of activities for the whole process, from the start o f the project to the end, and the relationships between activities and guidelines of a methodical nature.

The methodology is designed to be used for reform projects in developing countries. This in itself gives it the advantage of being intended as an incredibly simple but effective planning and implementation tool.

The methodology consists o f two parts (Handbook for objectives- oriented planning 1992): (1) a situation analysis to identify stakeholder groups and problems and weaknesses in the existing system and (2) project design. The first part can stand alone, while the latter presupposes the first.

The philosophy of the methodology is to focus on one central problem during a change management process. The core of the process lies in producing a 'Problem Tree' where the leaves are directly converted into a 'Tree of Objectives' through a description of objectives. This is then transformed and becomes a change management tool, such as traditional activity descriptions for a project.

A. Situation Analysis 1. Stakeholder analysis: Identification of groups and subgroups in

order to identify participants for the future development project.

149

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

IdANDBOOI< OF: EVALUATION METHODS

2. Problem analysis: By means of a brainstorming technique to capture elements and symptoms in the organization, which are subsequently synthesized into a 'Problem Tree' with the trunk being the central problem and the leaves the smallest symptoms. An analysis of the causality is used as the roots of the tree in such a way that all the branches and leaves are covered and thereby accounted for.

3. Objectives Analysis: The Problem Tree is converted into a tree of corresponding solutions, the 'Objectives Tree'.

4. Analysis of the alternatives with regard to choosing the best solution for the future and the establishment of a strategy.

B. Solution Design 5. Project Elements: Define the objectives for the development

(justification of the project), the resulting immediate subobjectives (which together define and limit the intended effect), and then break them down into results, activities, and resources.

6. External Factors: Identification (for each and every activity) of important risk factors.

7. Indicators: Definition of the measures for monitoring the progress of each activity.

The above approach is primarily suited for simple projects. Crawford and Bryce (2003) review other and more complex versions of the LFA and summarize the key limitations of this method, which are concerned with handling the preconditions and assumptions for the implementation work (Phase B). However, in an evaluation context it is the approach of the Situation Analysis that is of primary interest.

Assumptions for Application There is no real precondition with regard to the educational level of the participants, precisely because its intended use is in developing countries. A very simple version of the methodology can be used. However, depending on the accuracy and precision required, it may be necessary for the leader(s) of the process to have adequate experience of group dynamics and relevant methods to supplement specific information needs. It can not replace traditional system development methods but may be applied to identify areas that need focused efforts and as such it may serve as a means for constructive assessment.

150

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

IdANDBOOt< OF:: EVALUATION METHODS

The methodology assumes quite a stable project plan, as it does not have any inbuilt mechanisms, such as feed-back loops, for handling modifications. However, this does not preclude changes to a project plan but requires that all elements o f the plan and its methods are explicitly dealt with or taken into consideration during a possible modification.

It is clearly a prerequisite that the supporting methods and techniques are properly identified and adapted to detailed subactivities in such a way that they harmonize with the entirety and with each other. Depending on the size o f project concerned with the situation analysis o n l y - a need for supporting methods or techniques, for instance - may arise, as may a need for supporting the causal analysis. Examples o f useful methods in this respect are Stakeholder Analysis and Focus Group Interviews, for example.

Perspectives In itself the methodology is neutral to political, organizational and cultural conditions or constraints, it neither dictates nor prohibits such constraints, and it can work under their conditions. In the description there are aspects that make it more suitable in certain cultures and types o f organizations than in others (as, for instance, in the suggestion for choice o f project participants where the chosen procedures, principles, and criteria are clearly culture dependent). However, there is nothing to stop the use o f one's own principles.

The methodology is reasonably neutral toward the methods employed and contains a number o f steps for the process. As a rule it is necessary to add concrete and formalized methods or instructions (at least for large projects). As such it does not replace the stakeholder analysis method, cost-benefit analysis, impact assessments, and so on, but it has openings and rudimentary instructions where activities are prescribed and other methods therefore may supplement.

The perspective o f the Problem Tree is that o f focusing on just one single problem as a starting point for development. In the case o f large development projects, this can often be too simplified a point o f view. However, this can be resolved through a simple modification o f the problem analysis. For example: 'All o f the problem areas o f the organization' may be defined as the trunk o f the Problem Tree, instead o f picking just one o f the largest branches as the trunk and then disregarding the rest (Brender 1997). This will o f course make the investigation into the causality more extensive or

151

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

NANDE~OOt< OF {_?-VALUATION METHODS

difficult, but it will also make it far more rewarding.

Frame of Reference for Interpretation Preparation o f the Problem Tree (in the Situation Analysis) is relevant as an evaluation tool in connection with an assessment o f IT-based systems. But the Problem Tree does not have its own frame o f reference against which to compare the outcome. The validity o f the synthesized Problem Tree might be discussed with the user organization whose opinions and attitudes become a frame o f reference o f sorts.

Perils and Pitfalls When eliciting the Problem Tree, one must be aware o f the following sources o f error (see Part III):

1. Postrationalization (may partly be redressed by brainstorming techniques through the interaction and inspiration between the stakeholders)

2. Other error sources in connection with group dynamics, as for example under F o c u s Group I n t e r v i e w

Advice and Comments The methodology's principle o f risk monitoring is recommended in its own right, in terms o f monitoring the external factors that are incorporated in the implementation plan as explicit variables. It may be advantageous to incorporate them into other methods and strategies for ongoing monitoring (evaluation) o f development trends and risk factors.

The Affinity method may be used as an aid to brainstorming and modeling o f the Problem Tree or alternatively one o f the other similar methods outlined on http://jthom.best.vwh.net/usability/.

References

Handbook for objectives-oriented planning. 2nd ed. Oslo: Norwegian Agency for Development Cooperation; 1992.

I f this is not available, there is a r~sum~ o f it in the next reference. Alternatively, the reader is referred to the review in (Crawford and Bryce 2003).

Brender J. Methodology assessment of medical IT-based systems - in an organisational context. Amsterdam: lOS Press, Stud Health Technol Inform 1997;42.

152

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

I-IANE~OOI< O1:: EVALUATION M{~TIqODS

Crawford P, Bryce P. Project monitoring and evaluation: a method for enhancing the efficiency and effectiveness of aid project implementation. Int J Proj Manag 2003;21 (5):363-73.

http://jthom.best.vwh.net/usability/ Contains lots o f method overviews with links and references (last visited on 31.05.2005).

155

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

NANI}8OOK O~ EVALUATION MI~TIdODS

Organizational Readiness

Areas of Application Assessment o f a healthcare organization's readiness to assimilate a clinical information system.

Description A number of aspects determine the organizational readiness for change, such as organizational adaptability and flexibility, its willingness to absorb external solutions, and its ability to develop viable solutions. A potential cause of failure to innovate is the organizational inability to undergo transformation during the implementation of an information system.

The study of Snyder-Halpern (2001) briefly reviews previous attempts at determining readiness and validates a model of innovation readiness and a set of heuristics to assess organizational readiness. The method is not yet complete and has still to include the metrics o f the heuristics suggested. However, the description may still serve as valuable inspiration to preventive actions through assessment at the early stages o f IT systems purchase or implementation.

Assumptions for Application

Perspectives

Frame of Reference for Interpretation (Not applicable)

Perils and Pitfalls

Advice and Comments

154

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

HAND~OOI< O~ ~VALUATION N~TIdODS

References

Snyder-Halpem R. Indicators of organizational readiness for clinical information technology/systems innovation: a Delphi study. Int J Med Inform 2001;63(3):179-204. Erratum in: Int J Med Inform 2002;65(3):243.

155

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

IdANDt~OOI< O~ EVALUATION NE---TIdODS

Pardizipp Areas of Application Preparation of future scenarios

Description Scenarios are common-language descriptions of specific activities and of how users would normally go about executing them. They can, however, also be described diagrammatically. Pardizipp is based on the Delphi method. Development of scenarios, which in Pardizipp are textual, follows the six steps listed below (steps 2-4 are to be repeated jointly) (Mettler and Baumgartner 1998):

1. Definition of a general frame that will serve as the basis for group work around the creation of scenarios

2. Creation of scenarios and a thorough analysis of their consequences and assumptions

3. Quantifying and model building 4. Preparation of policies and specific actions - where a given

scenario should be implemented 5. Development of a consensus scenario, which takes into account

earlier scenarios developed for the same problem area 6. Preparation of recommendations for policies and actual actions

Assumptions for Application A prerequisite for a successful result is that a good mix of stakeholder groups are represented.

It is worth noting that a centrally placed t e a m - not the participants i n v o l v e d - prepares the resulting scenario(s) on the basis of the information gathered. Thus, it requires a certain degree of experience o f this type of undertaking. See also under Delphi.

Perspectives The philosophy behind the method is twofold: It is partly based on the philosophy built into the Delphi method and partly on the fact that modem technological development has a number o f unfortunate effects on surrounding social and organizational conditions. This

ts6

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

HANDt~OOI< O~ EVALUATION METHODS

makes the authors focus on improving the basis of decision-making and related processes, which is done by establishing a decision- making foundation built on the projections and aspirations o f a broad segment of participants. In other words, they believe that the basis for decision making will improve by expanding the group preparing it. This is obviously a culturally conditioned assumption, see the discussion in the Introduction, and it has its roots in Western cultural perceptions o f how things should be handled and how best to structure an organization.

Frame of Reference for Interpretation (Not applicable)

Perils and Pitfalls 1. The principle of using scenarios gives a fragmented picture of a

normally holistic entirety. Therefore, it takes some talent to organize the preparation o f a combination of scenarios in such a way that together they will give an adequately holistic impression.

2. People normally find it difficult to explicitly formulate their thought about their own work situation and work procedures explicitly (see the discussions in Brender 1997a and 1999 and others). This is where the Delphi method shows its strength, but it cannot completely compensate in all situations.

3. One pitfall is the ethnocentricity (see Part III) - that is, lack of acknowledgement and consideration of cultural backgrounds. This may introduce a bias in the selection of participants. The authors mention principles for participation (for instance, lack o f female involvement) as a caveat, but in general the method is considered to be most appropriate in cultures where there is already a tradition for participatory methods and in organizations where there is a possibility of (informal) equality among participants of different levels of competence.

Advice and Comments Future scenarios are useful tools for the understanding o f development trends and options. Methods for preparing future scenarios often have the advantage of helping the participants to fantasize - irrespective of their technological understanding. This way the participants are often able to let go of actual constraints to the technological potentials and limitations induced by many systems analysis tools.

157

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

HANDE~OOI( O1::: EVALUATION METHODS

References

Brender J. Methodology for assessment of medical IT-based systems- in an organisational context. Amsterdam: lOS Press, Stud Health Technol Inform 1997;42.

Brender J. Methodology for constructive assessment of IT-based systems in an organisational context. Int J Med Inform 1999;56:67-86.

This is a shortened version o f the first reference and more easily accessible with regard to this subject.

Mettler PH, Baumgartner T. Large-scale participatory co-shaping of technological developments, first experiments with Pardizipp. Futures 1998;30(6):535-54.

i s 6

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

HANDBOOK OF EVALUATION P1ET~qODS

Prospective Time Series

Areas of Application M e a s u r e m e n t o f a development trend, including, for example, the effect o f an intervention:

�9 Time series �9 Before-and-after studies, where the work involved could be the

introduction o f an IT-based system, for instance

Description M e a s u r e m e n t o f a n u m b e r o f measures over time shows h o w an activity or an outcome changes as a function o f either time alone or as a function o f different initiatives. Such studies m a y be either simple or controlled, depending on the level o f control o f experimental factors. Simple before-and-after studies address a single case before and after an intervention, while the controlled approach matches a n u m b e r o f cases to be studied in parallel.

Assumptions for Application The use o f time series as o n e ' s assessment design requires control over what changes take p l a c e - intentional and u n i n t e n t i o n a l - within the organization during the study.

Perspectives Time series are in principle carried out as a matched-pair design, just like traditional, controlled studies including RCTs. Therefore, they contain the inbuilt assumption that there is no interaction between case-specific matching characteristics and a potential intervention (Suissa 1998). The p r o b l e m occurs in particular w h e n there is just one case in the s t u d y - the simple before-and-after s t u d y - which often happens in assessment o f IT-based solutions. Implementation o f IT-based solutions often involves radical changes in the organization, its structure, and w o r k processes. Consequently, when there is only one case and the intervention is the introduction o f an

7 The same comment is valid regarding the economy as described for Clinical~Diagnostic Performance.

159

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

HANDE~OOI< O~ z {]VALUATION M~TIdODS

IT-based solution, an interaction between the case and the intervention will be present. Thus, the control group (the first measurement(s) in the time series) and the intervention group are no longer identical in all aspects other than that o f the intervention applied. See also Sparrow and Thompson (1999).

Frame of Reference for Interpretation The frame o f reference is generally included in the overall d e s i g n - for instance, as the first point within the time series.

Perils and Pitfalls One question to keep in mind is: Do the measures keep representing exactly the same characteristics o f the system or the organization? This needs to be verified for the data collected, and it is particularly important when a time series spans the implementation o f an IT s y s t e m - often over several years. Evans et al. (1998) apparently elegantly handle this error source. However, they do fall into the trap o f giving the decision-support system the credit for the improvements without acknowledging that, simultaneously, drastic changes to work processes take place (over and above what is needed by the system functionality).

As always, one has to ascertain that the conditions for the use o f ones metrics (calculation techniques) are fulfilled, and if one does performs statistical calculations on successive steps in a series o f measures, it is a requirement for the use o f concrete techniques that the measures are independent ('carryover effects' and 'conditional independence on the subject'). See details in (Suissa 1998).

In cohort studies with the follow-up o f cases over time, biases may occur as a consequence o f cases that drop out along the way (see Pennefather et al. 1999). In principle this corresponds to cases with missing data for the later points o f the time series. This bias may be caused by lack o f representativeness for the drop-out group compared to the study group as a whole. An important example is successive questionnaire studies. Another example is the representativeness o f the users involved around the IT system. Therefore, in the final conclusion it is important to explain the causality in connection with these missing data.

Advice and Comments

16o

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

I-IANE~OOI< OF:: EVALUATION METHODS

References

Evans RS, Pestotnik SL, Classen DC, Clemmer TP, Weaver LK, Orme JF, Lloyd JF, Burke JP. A computer-assisted management program for antibiotics and other antiinfective agents. New Engl J Med 1998;338(4):232-8.

Pennefather PM, Tin W, Clarke MP, Fritz S, Hey EN. Bias due to incomplete follow-up in a cohort study. Br J Ophthalmol 1999;83:643- 5.

Sparrow JM, Thompson JR. Bias: adding to the uncertainty, editorial. Br J Ophthalmol 1999;83:637-8.

Suissa S. The case-time-control-design: further assumptions and conditions. Epidemiol 1998;9(4):441-5.

Supplementary Reading

Ammenwerth E, Kutscha A, Eichst/idter R, Haux R. Systematic evaluation of computer-based nursing documentation. In: Patel V, Roger R, Haux R, editors. Proceedings of the 10th World Congress on Medical Informatics; 2001 Sep; London, UK. Amsterdam: lOS Press; 2001. p. 1102-6.

A good before-and-after case study o f the quality o f nursing documentation records and user satisfaction based on a combination o f many methods (a multimethod design).

Bates DW, Teich JM, Lee J, Seger D, Kuperman GJ, Ma'LufN, Boyle D, Leape L. The impact of computerized physician order entry on medication error prevention. JAMIA 1999;6(4):313-21.

A case study using prospective time studies to assess the impact o f the introduction o f an IT-based solution on a concrete activity.

Brown SH, Coney RD. Changes in physicians' computer anxiety and attitudes related to clinical information system use. JAMIA 1994;1(5):381-94.

In a case study the authors investigate physicians 'fear o f new I T technology in a before-and-after study.

Kelly JR, McGrath JE. On time and method. Newbury Park: Sage Publications. Applied Social Research Methods Series 1988. vol. 13.

The book thoroughly and stringently (and with a philosophical background) discusses a number o f aspects concerning time and the influence o f time on experimental studies. Further, beyond the

161

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

NANDBOOK OF LZVALUATION METHODS

problems and pitfalls o f time studies, it discusses, for instance, changes in the observer and problems at causal analysis o f phenomena o f a cyclic nature.

Murphy CA, Maynard M, Morgan G. Pretest and post-test attitudes of nursing personnel toward a patient care information system. Comput Nurs 1994;12:239-44.

A very good case study with questionnaires in a before-and-after study. Recommended f o r studying because they make a real effort to verify the internal validity o f the elements in the questionnaire.

Ruland CM, Ravn IH. An information system to improve financial management resource allocation and activity planning: evaluation results. In: Patel V, Roger R, Haux R, editors. Proceedings of the 10th World Congress on Medical Informatics; 2001 Sep; London, UK. Amsterdam: IOS Press; 2001. p. 1203-6.

A before-and-after study o f a decision-support system f o r nurses with the primary focus on investigating financial aspects (which in this case corresponds to evaluation o f the objectives fulfillment) and user satisfaction. But, unfortunately, the study suffers (possibly)from a number o f biases, typical o f controlled studies (see Part III o f this handbook).

Wyatt JC, Wyatt SM. When and how to evaluate health information systems? Int J Med Inform 2003;69:251-9.

Outlines the differences between the simple before-and-after and the controlled before-and-after studies as opposed to the RCTs.

Yamaguchi K. Event history analysis. Newbury Park: Sage Publications. Applied Social Research Methods Series 1991. vol. 28.

A book f o r those who want to conduct studies over time in an organization.

~62

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

HANDBOOI< OE ~VALUATION METHODS

Questionnaire (Nonstandardized questionnaires)

Areas of Application Imagination is the only real limit to what questionnaires can, and have, been used for, but for investigations requiring a high level o f accuracy their main area o f application is (qualitative) studies o f subjective aspects.

N o t e : There is a sharp distinction between custom-made questionnaires and standardized, validated questionnaires, available f r o m the literature or as commercial tools. The section below deals with the former. However, even i f a standard questionnaire is chosen, one must still investigate the degree to which the assumptions and pitfalls and the quality meets one's needs.

Description The advantage o f questionnaires - and probably the reason why they are so widely u s e d - is that most people can manage to put a questionnaire together to investigate virtually any subject o f one's choice.

There are a number o f ways in which to ask questions, and they do not necessarily eliminate each other. However, by using a combination o f them there is a risk o f making the analysis (the mathematical and statistical analysis) more difficult:

�9 Open questions, where the respondent answers (the individual questions) the questionnaire in ordinary text

�9 Checklist questions, which normally consist o f three boxes: "Yes", "no", and " d o n ' t know"

�9 The Likert scale consisting o f a bar with fields to tick on a scale from "agree completely", "agree", through a neutral to "disagree" and "completely disagree"

�9 Multipoint scale, where the respondent indicates his or her assessment on a continuous scale indicating the two extremes with opposing rankings "agree completely" to "completely disagree"

�9 Semantic differential scale, which in tabular form uses columns with a value scale (for instance, "extremely", "very",

163

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

IdANDt~OOI< O~ ~VALUATION M~TIdODS

"somewhat", "neutral", "a little", "somewhat", "very", "extremely") and where the rows of the table indicate the properties that should be evaluated by means of the two extremes on the scale (for example, " e a s y " . . . " d i f f i c u l t " . . . " f u n " . . . "boring"), and so on.

�9 Categorical scale, where the tick options (which may mutually preclude each other) are completely s e p a r a t e d - for example, questions about sex, age, or profession.

Note: No specific questionnaires are indicated here, as they are normally f o r m u l a t e d f o r each specific case, but in the literature there are a n u m b e r o f more or less 'standardized' questionnaires f o r measuring user satisfaction, which can be used as they are or with some adaptations. These are indicated below under References.

Assumptions for Application Questionnaires are tools, and tools need verification with respect to construct and content validity before application. It is important that questionnaires - and all elements in t h e m - have been tested (validated) to increase the likelihood that the questionnaire will serve its purpose as it is supposed to. It is very important that questionnaire studies are o f a suitably high standard qualitatively. This concerns the preparation establishing objectives for the studies, accurate wording o f hypothesizes and theories, qualitatively satisfactory questionnaires (which is probably the most difficult), clearly formulated rules of analysis, and requirements for reporting (opportunities and limitations) of the results.

Further, it is obvious that there must be a reasonable relationship between the resources required to prepare the questionnaire and the purpose of the study. Far from all studies need thorough scientific validation o f the questionnaire in order to provide results leading to optimal action.

The use o f concrete statistical tools normally implies assumptions, and one of the assumptions for the use of standard deviations and student's t-test is that the data come from a continuous scale (a scale with real numbers). Consequently, these statistical methods c a n n o t be used for the analysis of data obtained on a categorical scale such as the Likert scale(!) but may be used for a multipoint scale, for instance.

~64

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

IdANDBOOI< OF .EVALUATION JVlETHODS

Perspectives The perspective o f the person formulating the questionnaire depends very much on his or her level o f experience. The inexperienced seem to think that it is easy to write a questionnaire and that this will produce answers and provide the truth about the questions asked. This rather naive understanding is based on the fact that we are all used to formulating lots o f questions and to getting sensible answers every day. The difference is, however, that when we formulate everyday questions between us, these questions form part o f a joint context in relation to the present and possibly also to a mutual past, which explains a lot about the background and what it refers to, implicitly and explicitly. If you don't get the right answer the first time around during the conversation, you repeat it once or twice without thinking about it. This type o f reiteration cannot take place in questionnaires, and the creation o f a mutual understanding is exactly what makes the formulation o f a valid questionnaire so difficult.

Frame of Reference for Interpretation The frame o f reference depends on the purpose o f the study. One may formulate before-and-after questionnaires, where the response to the first questionnaire becomes the frame o f reference for the second one. A strategic objective could also be a frame o f reference in a particular study. Furthermore, earlier studies from the organization, for instance, or studies described in the literature may also be used as a basis for comparison.

However, usually there is no frame o f reference for the analysis or the conclusion o f a questionnaire study.

Perils and Pitfalls There is a whole range o f pitfalls in questionnaires, such as the following:

�9 The internal validity o f the questions: Does the respondent read/understand the same as the author? Do all the respondents read/understand exactly the same thing?

�9 The problem o f postrationalization (see Part III) is a risk in studies using questionnaires presented sometime after the events addressed (Kushniruk and Pate12004).

�9 Each respondent will always answer the questionnaire in his or her own context including emotional factors and the actual response will therefore vary from one day to the next.

�9 For psychological reasons people have problems assessing

16s

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

IdANDE~OOI< O~C {]VALUATION METHODS

probabilities correctly. Therefore, questions such as "how often . . . ? " should be avoided.

�9 Certain questions may be taboos, or they may be perceived as t h r e a t e n i n g - particularly in the case o f nonanonymous studies (Vinten 1998).

�9 When using a questionnaire from foreign literature, be aware that it needs adapting in case o f national, organizational, linguistic, or cultural differences. It is not valid per se to apply a questionnaire in a foreign language, and it is not easy to translate a questionnaire while preserving the meaning, in case one wants to make a multinational survey.

�9 It is rare that the target group o f a questionnaire study is so well grounded in a foreign language that one can expect a reliable answer should the questionnaire be used in its original language.

�9 Do not underestimate the pitfall that may be introduced when translating a questionnaire from one language (and culture) to another. This is by no means insignificant.

�9 It makes a difference who in the organization is being interviewed, because top and middle management have more training in promoting the official line at the expense o f their own personal opinion without even thinking about it. The same is the case in certain cultures, such as in Asia and in the former Soviet republics. The fact that these cultures belong to countries a great distance away from your own does not automatically preclude that these same attitudes and manners exist in your country or in the subculture o f your s o c i e t y - because they do.

Although there are many pitfalls and difficulties, one should not give up because it is to allay the problems. Depending on the intended use o f the study result and how accurate it needs to be, the list above illustrates that the task o f formulating a questionnaires often is one to be carried out by people with this expertise.

Advice and C o m m e n t s Do look for guidelines to qualitative evaluation studies, as they give lots o f tangible advice on the wording o f questionnaires and examples o f these.

References like (Ives et al. 1983; Murphy et al. 1994; Jacoby et al. 1999; and Par6 and Sicotte 2001) are examples o f how to verify and adapt a questionnaire for, for instance, reliability, predictive validity, accuracy, content validity, and internal validity.

166

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

HANDE~OOI< 04:: EVALUATION M{~THODS

R e f e r e n c e s

Ives B, Olson MH, Baroudi JJ. The measurement of user information satisfaction. Communications of the ACM 1983;26(19):785-93.

Brilliant little article with advice and guidelines regarding the measurement and adaptation o f different quality measures f o r a questionnaire.

Jacoby A, Lecouturier J, Bradshaw C, Lovel T, Eccles M. Feasibility of using postal questionnaires to examine career satisfaction with palliative care: a methodological assessment. Palliat Med 1999; 13:285- 98.

Kushniruk AW, Patel VL. Cognitive and usability engineering methods for the evaluation of clinical information systems. J Biomed Inform 2004;37:56-76.

Murphy CA, Maynard M, Morgan G. Pretest and post-test attitudes of nursing personnel toward a patient care information system. Comput Nurs 1994;12:239-44.

Recommended f o r studying, precisely because it makes a real effort to verify the internal validity o f the elements o f the questionnaire and explains how one can use questionnaires in a before-and-after study.

Par6 G, Sicotte C. Information technology sophistication in health care: an instrument validation study among Canadian hospitals. Int J Med Inform 2001 ;63:205-23.

Vinten G. Taking the threat out of threatening questions. J Roy Soc Health 1998;118(1):10-4.

References to Standard Questionnaires

Aydin CE. Survey methods for assessing social impacts of computers in health care organizations. In: Anderson JG, Aydin CE, Jay SJ, editors. Evaluating health care information systems, methods and applications. Thousand Oaks: Sage Publications, 1994, pp. 69-115.

The chapter refers to a number o f questionnaires used in the literature and contains a couple o f actual examples in the Appendix.

Harrison MI. Diagnosing organizations: methods, models, and processes. 2nd ed. Thousand Oaks: Sage Publications. Applied Social

~67

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

HANDBOOK O~ EVALUATION PI~THODS

Research Methods Series 1994. vol. 8. Appendix B o f the book contains a number o f references to standard questionnaires f o r (nearly) every purpose regarding conditions in an organization.

Supplementary Reading, Including Case Studies

Ammenwerth E, Kaiser F, Buerkly T, Gr/iber S, Herrmann G, Wilhelmy I. Evaluation of user acceptance of data management systems in hospitals - feasibility and usability. In: Brown A, Remenyi D, editors. Ninth European Conference on Information Technology Evaluation; 2002 Jul; Paris, France. Reading: MCIL; 2002:31-38. ISBN 0-9540488-5-7.

This reference and the next deal with the same case study, but at different phases o f the assessment.

Ammenwerth E, Kaiser F, Wilhelmy I, Hrfer S. Evaluation of user acceptance of information systems in health care - the value of questionnaires. In: Baud R, Fieschi M, Le Beux P, Ruch P, editors. The new navigators: from professionals to patients. Proceedings of MIE2003; 2003 May; St. Malo, France. Amsterdam: IOS Press. Stud Health Technol Inform 2003;95:643-8.

A case study o f user satisfaction, which also assesses the quality aspects (reliability and validity) o f the questionnaire used.

Andersen I, Enderud H. Udformning og brug af sporgeskemaer og interviewguides. In: Andersen I (Ed.). Valg af organisations- sociologiske m e t o d e r - et kombinationsperspektiv. Copenhagen: Samfundslitteratur; 1990. p. 261-81. (in Danish)

This reference contains some advice on how (not) to do things.

Bowman GS, Thompson DR, Sutton TW. Nurses' attitudes towards the nursing process. J Adv Nurs 1983; 8(2):125-9.

Concerns (measurement oj9 user attitudes to the nursing process.

Chin JP. Development of a tool measuring user satisfaction of the human-computer interface. In: Proceedings of the Chi'88 Conf. on Human factors in Computing. New York: Association for Computing Machinery; 1988. p. 213-8.

Addresses user attitudes toward specific characteristics o f nursing documentation.

Hicks LL, Hudson ST, Koening S, Madsen R, Kling B, Tracy J, Mitchell J, Webb W. An evaluation of satisfaction with telemedicine among health-care professionals. J Telemed Telecare 2000;6:209-15.

166

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

HANDBOOK OF: EVALUATION M~TidOD$

A user satisfaction case study.

Leavitt F. Research methods for behavioral scientists. Dubuque: Wm. C. Brown Publishers. 1991.

Contains quite tangible instructions on how to formulate and put questions together (also f o r interviews) and avoids the worst pitfalls, see pages 162- 72.

Lowry CH. Nurses' attitudes toward computerised care plans in intensive care. Part 2. Nurs Crit Care 1994; 10:2-11.

Deals with attitudes toward the use o f computers in nursing, particularly in connections with documentation.

L~erum H. Evaluation of electronic medical records, a clinical perspective [Doctoral dissertation]. Faculty of Medicine, Norwegian University of Science and Technology, Trondheim: NTNU; 2004. Report No.: 237. ISBN-82-471-6280-6.

This doctoral dissertation is a multimethod evaluation study, including an extensively validated questionnaire.

Murff HJ, Karmry J. Physician satisfaction with two order entry systems. J Am Med Inform Assoc 2001 ;8:499-509.

A very well-planned case study when it comes to the comparison o f two systems, as the users are the same f o r both systems. They also seem to have the statistics under control

Nickell GS, Pinto JN. The computer attitude scale. Comput Human Behav 1986;2:301-6.

Deals with user attitudes toward computers in general (for everyday use).

Ruland CM. A survey about the usefulness of computerized systems to support illness management in clinical practice. Int J Med Inform 2004;73:797-805.

A case study applying a questionnaire to survey clinical usefulness.

RS, dgivende Sociologer. Manual om sporgeskemaer for Hovedstadens Sygehusf~ellesskab.R~dgivende Sociologer; 2001. Report No.: manual om sporgeskemaer for H:S. (Available from: www.mtve.dk under 'publications'. Last visited 31.05.2005.) (in Danish)

This is a very useful reference f o r formulating a questionnaire study. It contains both instructions and actual finished questionnaires f o r different purposes within the healthcare sector.

169

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

~ANBSOOK O~ ~VALUATION NETIdODS

Shubart JR, Einbinder JS. Evaluation of a data warehouse in an academic health sciences center. Int J Med Inform 2000;60:319-33.

Is a structured, hypothesis driven case study with verification and validation o f the questionnaire and explicit validation o f the results.

Sleutel M, Guinn M. As good as it gets? going online with a clinical information system. Comput Nurs 1999; 17(4): 181-5.

A good case study to learn from because they are aware o f an analysis o f the internal validity o f the questionnaire and because they dig down and investigate unexpected observations. However, the drawback is that they do not mention the formal requirements for the use o f their statistical tools.

Terazzi A, Giordano A, Minuco G. How can usability measurement affect the re-engineering process of clinical software procedures? Int J Med Inform 1998;52:229-34.

Addresses "perceived usability'-that is, the user's subjective understanding o f usability, using standard questionnaires f o r this purpose.

Weir R, Stewart L, Browne G, Roberts J, Gafni A, Easton S, Seymour L. The efficacy and effectiveness of process consultation in improving staff morale and absenteeism. Med Care 1997;35(4):334-53.

A well-executed case study applying the RCT method based on existing, validated questionnaires covering subjects such as job satisfaction, attitudes, and personalities. Unfortunately, they have problems with the differences in the intervention between the two groups compared

See also:

http://jthom.best.vwh.net/usability Contains lots o f method overviews with links and references (last visited 31.05.2005).

Goldfield GS, Epstein LH, Davidson M, Saad F. Validation of a questionnaire measure of the relative reinforcing value of food. Eat Behav 2005;6:283-92.

The advantage o f this reference is the suggestions for statistical analysis methods taking into account the nature o f the investigation data.

McLinden DJ, Jinkerson DL. Picture this! multivariate analysis in

170

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

IdANDi~OOk:[ OF: EVALUATION METHODS

organisational development. Evaluation Program Planning 1994;17(1):19-24.

This little article can be inspirational in how to treat a n d p r e s e n t data in a different way.

http://qpool.umit.at This website is still under construction, but it will eventually include a list o f more or less validated questionnaires f r o m the literature (last visited 31.05.2005).

171

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

IdANDE~OOI< O~C EVALUATION METIdODS

RCT, Randomized Controlled Trial

Areas of Application The purpose o f this type o f study is verification o f efficacy 8 (Wall 1991; G o o d m a n 1992) - that is, that the IT system - under ideal conditions - makes a difference to patient care.

RCT is used to identify marginal differences between two or more types o f treatment. This method has been particular useful in assessment studies o f IT-based solutions for decision-support systems and expert systems, only to a limited degree for other types o f systems.

Description Randomization is used to avoid bias in allocation or choice o f cases and actors. In clinical studies randomization is used in relation to selection o f clinical personnel or patient treatment groups, respectively, while in the assessment o f IT-based solutions, for instance, it m a y be used to select study organization(s) or users.

The concept 'controlled' is used in relation to studies with a m i n i m u m o f two groups, one o f which is a reference group (control group, treated in the traditional way). The control group is used for comparison o f whether there is an effect on the intervention group - that is, as the frame o f reference. The principle being that the control group and the intervention group are treated in exactly the same w a y (except for the intervention), thereby making it possible to measure differences in the effect o f the intervention. The problem with RCT for IT-based solutions is to secure and achieve identical treatments between the two or more groups.

Assumptions for Application Use o f RCT in medicine is fairly standard, and the method is therefore known to many. However, it is not possible to transpose the method directly to assessment o f IT-based solutions, as it is by no means trivial to handle the assumptions on randomization and to

8 ,, '[Effficacy" addresses the performance o f the application under ideal circumstances, while 'effectiveness' is related to application under real circumstances (the capability o f bringing about the result intended- i.e., doing the right things); and 'efficiency' is related to a measure o f the capability o f doing the things right. "(from Brender 1997a)

177

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

I--IANIDBOOI< OF- EVALUATION MI~TIdODS

obtain comparable groups. One o f the challenges is to identify criteria to select the study objects and thereby the population (cases, users, and patients) with consideration to the size o f the population and the generalization of a later conclusion. See below regarding the many pitfalls.

Perspectives RCT is widely used for testing clinical procedures and pharmaceutical products. One argument in favor of RCT is that this procedure is so well established that a number of biases can be avoided. It is a characteristic of medical and pharmaceutical tests that the effect is often marginal in comparison to existing pharmaceuticals and procedures. Thus, it is necessary to be extremely careful about as many types of bias as possible in order to document that the new product makes a difference at all. This is not necessarily the case with similar studies of IT systems. Neither is it a forgone conclusion that such a study would conclude to the IT system's advantage if impact parameters were viewed unilaterally. The reward might be found in quite a different place that is not addressed by an R C T - for example, in the soft human aspects of the system, in the system's feasibility to support change management in the organization, or, in the longer term, to other profession-oriented or management aspects of the system.

RCT is carried out under ideal conditions, and it is focus driven. But who is to say that day-to-day reality is ideal?

Some people promote the viewpoint that RCT can be carried out very cheaply because the existing infrastructure, service function, and resources of a hospital or a department can be used free of charge and therefore only needs compensation for extraordinary expenses such as analysis of statistics. No doubt, this is feasible in many cases today. In principle this is also the case for RCT o f IT- based systems, but it is naive to believe that real life is that simple.

Frame of Reference for Interpretation It is possible to carry out controlled studies in several different ways, as, for instance, (1) by some patients being treated under the old system and others under the new system, or equally with regard to staff; (2) by some departments keeping the old system and other department getting the new system; or (3) by comparing similar departments in different hospitals.

175

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

IdANDBOOI< OF: EVALUATION MI~TIdODS

Perils and Pitfalls Quite the biggest problem is to make the groups being compared truly comparable. This includes having the groups go through the same conditions and circumstances during the process so that only relevant factors influence the outcome o f the study.

Randomization depends on the feasibility o f a real choice between participants to ensure that the resulting groups are comparable. It is o f course possible to make a draw between two departments to find the one that gets an EHR implemented and the one that will n o t - in other words, which is the intervention group and which is the control group, although this in itself does not render the control group and the intervention group comparable (see the review in Part III).

�9 Matching control and intervention groups. It is not always enough just to have comparable cases, as in different but comparable medical departments and comparing two departments with six physicians each, for instance. For certain systems, such as EHR and decision-support systems, the categories o f physicians the group is made up o f makes a difference because they have different backgrounds and levels o f competence, medically and in terms o f IT experience.

�9 One must be cautious when comparing groups from different medical specialist areas. This is the case not only during planning, but also in the interpretation (or worse, in extrapolation) o f the result with the purpose o f later putting it into wide practical use.

�9 Inclusion and exclusion criteria for involving cases and users as well as patients will normally cause the conclusion to be valid for a (well-defined) fraction o f the daily practice in the department or clinic.

�9 A demand for an RCT study will normally be to undertake comparable treatment o f both groups throughout the study. For IT systems this may be achieved by carrying out the phases o f requirements specification, design, and i m p l e m e n t a t i o n - that is, the Explorative Phase and the Technical Development Phase, on both the intervention group and the control group with a delayed start in one o f the departments. This way you may obtain a measure o f the effect on the intervention group. However, one must still be careful and avoid other biases, such as the Hawthorne effect, for example.

Advice and C o m m e n t s It is sometimes feasible to use the balanced block design when the IT-based system includes a number o f parallel applications (such as

17/4

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

HANtZSOOI< OF EVALUATION METHODS

knowledge-based support for a number of medical problems). The application will then be divided in to smaller clusters, each containing a proportion of the medical problems covered by the system. Each participant is assigned to one of the clusters, while serving as a control group for the other clusters and vice versa. See this approach in (Bindels et al. 2004), for instance.

There are a number o f initiatives and advocates for RCT to be used as the sole usable method for documenting the justification of a given IT system. But there are also well-argued debates, as, for instance, in (Heathfield 1998). The authors discuss the problems of using RCT for IT-based solutions and do not at all agree with those who promote the use of RCT to assess IT-based systems and solutions in the healthcare sector.

The personal view of the author is that every method should be used in the situation to which it is suited. However, in the absence o f more suitable methods, or if none can be applied to the objective, one may have to compromise. This is rarely in RCT's favor in case of IT systems.

Campbell et al. (2000) give advice and guidelines on the handling of RCT studies of complex interventions. It might be of some assistance to those who may be considering an RCT for IT-based systems.

References

Bindels R, Hasman A, van Wersch LWJ, Talmon J, Winkens RAG. Evaluation of an automated test ordering and feed-back system for general practitioners in daily practice. Int J Med Inform 2004; 73: 705- 12.

Campbell M, Fitzpatrick R, Haines A, Kinmonth AL, Sandercock P, Spiegelhalter D, Tyrer P. Framework for design and evaluation of complex interventions to improve health. BMJ 2000;321:694-6.

Goodman C. It's time to rethink health care technology assessment. Int J Technol Assess Health Care 1992;8:335-58.

Heathfield H, Pitty D, Hanka R. Evaluating information technology in health care: barriers and challenges. BMJ 1998;316:1959-61.

175

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

ldANDBOOK OF:: ~VALUATION METHODS

Wall R. Computer Rx: more harm than good? J Med Syst 1991;15:321- 34.

Supplementary Inspiration and Critical Opinions in the Literature, Including a Couple of Case Studies

Altman DG, Schultz KF, Moher D, Egger M, Davidoff F, Elbourne D, Gotzsche PC, Lang T. The revised CONSORT statement for reporting randomized trials. Ann Intern Med 2001 ;134(8):663-94.

This reference is a 'must" before getting started with putting anything about an R C T study in writing, and therefore it can also serve as an inspiration during planning o f such a study.

Ammenwerth E, Eichst/adter R, Haux R, Pohl U, Rebel S, Ziegler S. A randomized evaluation of a computer-based nursing documentation system. Methods Inf Med 2001 ;40:61-8.

The authors have chosen a design using a carryover effect between the study group and the control group to get a good f r a m e o f reference f o r the comparison See also the discussion in Part III, Section 11.1.6.5.

Assman SF, Pocock S J, Enos LE, Kasten LE. Subgroup analysis and other (mis)uses of baseline data in clinical trials. Lancet 2000;355:1064-9.

Biermann E, Dietrich W, Rihl J, Standl E. Are there time and cost savings by using telemanagement for patients on intensified insulin therapy? a randomized, controlled trial. Comput Methods Programs Biomed 2002;69:137-46.

This case illustrates how difficult it is to design a study to obtain truly comparable groups - one with, the other without, I T support. Without comparable procedures f o r the control group and the study group it is not possible to express what causes the potential (lack oJ) effect. At the same time the article shows how difficult it is to unambiguously describe similarities and differences in the procedures o f the two groups.

Brookes ST, Whitney E, Peters TJ, Mulheran PA, Egger M, Davey Smith G. Subgroup analyses in randomized controlled trials; quantifying the risks of false-positives and false-negatives. Health Technol Assess 2001 ;5(33).

Analyzes and discusses pitfalls in subgroup analysis when using RCT.

Chuang J-H, Hripcsak G, Jenders RA. Considering clustering: a

~76

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

IdANDE~OOI< OF: EVALUATION I"IETIdODS

methodological review of clinical decision support system studies. Proc Annu Symp Comput Appl Med Care. 2000:146-50.

Review o f R C T studies f o r decision-support systems and expert systems.

Friedman CP, Wyatt JC. Evaluation methods in medical informatics. New York: Springer-Verlag; 1996.

A good book that also discusses the role o f different approaches to evaluation.

Gluud LL. Bias in intervention research, methodological studies of systematic errors in randomized trial and observational studies [Doctoral dissertation]. Faculty of Health Sciences, University of Copenhagen; 2005. (ISBN 87-990924-0-9)

A valuable discussion o f different types o f biases and their implication in RCTs and other intervention approaches.

Hetlevik I, Holmen J, Krfiger O, Kristensen P, Iversen H, Furuseth K. Implementing clinical guidelines in the treatment of diabetes mellitus in general practice; evaluation of effort, process, and patient outcome related to implementation of a computer-based decision support system. Int J Technol Assess Health Care 2000; 16( 1):210-27.

A thorough case study, randomizing the practices involved, but note that with their procedure the control group remains completely untreated, while along the way, the intervention group gets an I T system, training, and follow-up in several different ways. In other words, it is not just the impact o f the decision- support system that is measured.

Kuperman GJ, Teich JM, Tanasjevic MJ, Ma'lufN, Rittenberg E, Jha A, Fiskio J, Winkelman J, Bates DW. Improving response to critical laboratory results with automation: results of a randomised controlled trial. JAMIA 1999;6(6):512-22.

The same team and just as good a study as in (Shojania et al. 1998), but in this study the authors do not have the same co- intervention problem.

Marcelo A, Fontelo P, Farolan M, Cualing H. Effect of image compression on telepathology, a randomized clinical trial. In: Haux R, Kulikowski C, editors. Yearbook of Medical Informatics 2002:410-3.

A double-blind R C T case study.

Rotman BL, Sullivan AN, McDonald T, Brown BW, DeSmedt P, Goodnature D, Higgins M, Suermondt HJ, Young YC, Owens DK. A randomized evaluation of a computer-based physician's workstation: design considerations and baseline results. In: Gardner RM, editor. Proc

177

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

~ANDBOOl< O~Z ~-VALUATION JVl~TldODS

Ann Symp Comput Appl Med Care 1995:693-7. (See under the next reference.)

Rotman BL, Sullivan AN, McDonald TW, Brown BW, DeSmedt P, Goodnature D, Higgins MC, Suermondt HJ, Young C, Owens DK. A randomized controlled trial of a computer-based physician workstation in an outpatient setting: implementation barriers to outcome evaluation. JAMIA 1996;3:340-8.

An R C T combined with a before-and-after design to measure user satisfaction and costs o f medication as well as compliance to the recommendations concerning drug substitution. The two studies referenced f r o m this group show the design considerations and the execution o f an RCT, respectively.

See Tai S, Nazareth I, Donegan C, Haines A. Evaluation of general practice computer templates, lessons from a pilot randomized controlled trial. Methods Inf Med 1999;38:177-81.

The study uses an elegant way to handle control group problems. See also the discussion in Part III, Section 11.1.6.1.

Shea S, DuMouchel W, Bahamonde L. A meta-analysis of 16 randomized controlled trial to evaluate computer-based clinical reminder systems for preventive care in the ambulatory setting. JAMIA 1996;3(6):399-409.

The article contains a meta-analysis o f RCT, and thus it can also be used to identify a number o f case studies f o r inspiration.

Shojania KG, Yokoe D, Platt R, Fiskio J, Ma'lufN, Bates DW. Reducing Vancomycin use utilizing a computer guideline: results of a randomized controlled trial. JAMIA 1998;5(6):554-62.

Shows a well-executed R C T case study despite having a problem with co-intervention (see Part 111, Section 11.1.6.3).

Tierney WM, Miller ME, Overhage JM, McDonald CJ. Physician inpatient order writing on microcomputer workstations. JAMA 1993;269:379-83 (reprinted in: Yearbook of Medical Informatics 1994; 1994:208-12).

A thorough case study, but with the usualproblem o f making the control group and the intervention group comparable.

Weir R, Stewart L, Browne G, Roberts J, Gafni A, Easton S, Seymour L. The efficacy and effectiveness of process consultation in improving staff morale and absenteeism. Med Care 1997;35(4):334-53.

Another well-executed R C T case study with existing, validated questionnaires. But this also has problems with differences between the two groups.

178

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

HANDt~OOI< O~ EVALUATION METHODS

van Wijk MAM, van der Lei J, Mosseveld M, Bohnen AM, van Bemmel JH. Assessment o f decision support for blood test ordering in primary care, a randomized trial. Ann Intern Med 2001;134"274-81.

A case study.

Wyatt JC, Wyatt SM. When and how to evaluate health information systems? Int J Med Inform 2003;69:251-9.

Outlines the differences between the simple before-and-after and the controlled before-and-after studies as opposed to the RCTs.

179

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

~AN~OOK O~ ~VALUATION METHODS

Requirements Assessment

Areas of Application Within the European culture the User Requirements Specification forms the basis for the choice and purchase o f an IT-based solution or for entering into a development project. Consequently, the User Requirements Specification is a highly significant legal document, which needs thorough assessment.

Description When assessing a requirements specification, it is important that it includes a description o f (see Brender 1997a and 1999):

1. The user organization's needs 2. The conditions under which the organization functions

(including its mandate, limitations, and organizational culture) 3. The strategic objective o f the organization 4. The value norms o f future organizational development 5. Whether the functionality can be made to adapt to the work

procedures in the organization or the reverse

Furthermore, there are some overriding general issues o f importance: �9 R e l e v a n c e : Assessment o f whether the solution in question or a

combination o f solutions is at all able to solve the current problems and meet the demands and requirements o f the organization.

�9 P r o b l e m A r e a s : Where are the weaknesses and the elements o f risk in the model solution? For instance, an off-the-shelf product may have to be chosen because the old IT system is so unreliable that it is not possible to wait for a development project to be carried out. Or plans may be based on a given operational situation albeit a lack o f know-how would occur should certain employees give notice.

�9 F e a s i b i l i t y : Does the organization have the resources needed to implement the chosen solution (structurally in the organization, in terms o f competence and financially, for example), as well as the support o f management, staff, and politicians?

�9 C o m p l e t e n e s s a n d C o n s i s t e n c y : Is the solution a coherent entity that is neither over- nor undersized?

�9 V e r i f i a b i l i t y ( o r T e s t a b i l i t y ) : One must consider how to check

16o

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

IdANDBOOI< O.lZ EVALUATION M~THODS

that every small requirement and function in the model solution have been fulfilled once the complete system is implemented and ready to be put to use. Elements o f Risk: Are there any external conditions outside organizational control that will involve a substantial risk to the project should it/they occur? This could, for instance, be dependence on a technology that is not yet fully developed, or dependence on (establishment or functional level of) o f the parent organization's technical or organizational infrastructure that has to fall into place first, or dependence on coordination and integration with other projects. See also Risk Assessment.

There are at least as many ways in which to assess a requirements specification as there are methods to produce it, and its flexibility is nearly as broad as its formalism is limited. Formal methods o f formulating requirements specifications are primarily the ones that have the formal methods o f verification. Apart from that, assessment o f a requirements specification is usually carried out informally but covering the aspects mentioned above.

A couple o f formal methods are: 1. Formal specification methods have their own tools to formulate

the requirements specification, often IT-based, and they have their own verification techniques, such as consistency control. These methods are mainly used to specify technical systems and are only occasionally suited for application by the users themselves.

2. Prototyping methods are based on repeated assessment workshops with user scenarios from real life; see, for example, the study in (Nowlan 1994). The scenarios are tested on a prototype, which gradually evolves from one workshop to the next (spiral development).

Instead, a number o f standards contain recommendations on how to prepare a requirements specification and what it should include as a minimum as well as what should be taken into consideration. These standards are therefore valuable sources o f inspiration or checklists to assess a requirements specification. See under Standards.

Assumptions for Application Assessment methods for requirements specifications depend entirely on the method used to prepare the specification.

iS1

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

HANDBOOK OE-I=VALUATION METHODS

Perspectives It is necessary to be quite clear about where in the organization the most serious problems occur (1) because you cannot solve a problem simply by introducing a new IT system, and (2) such weaknesses can be quite disastrous for the implementation work or for achieving the intended benefit.

It is the author's opinion that (1) ordinary users in the healthcare sector do not normally have a background enabling them to use the formal methods to produce a requirements specification, because they cannot separate the details from the entirety and the entirety from the details, and neither do they have experience of formulating the complexity. (2) It is not always enough to have IT consultants to assist in this particular task, and, therefore, (3) the users should apply their own premises and express themselves in their own language (Brender 1989). The case analyzed dealt with the development o f the first large IT-based solution for the organization. The longer an organization has had IT, the better able it will be to participate on technological p r e m i s e s - that is, use slightly more formal methods to formulate requirements specifications because its existing IT solution can be used as a kind o f checklist.

Frame of Reference for Interpretation The frame of reference for assessing a requirements specification is the whole o f the organization (structure, technology, actors, work p r o c e d u r e s , . . . ) , and its needs and premises, including conditions and objectives for acquiring an IT-based solution.

Perils and Pitfalls The pitfalls for assessing requirements specifications, which have been developed incrementally as a prototype (see under number 2 above), are used to illustrate where things may go wrong, such as:

�9 Representativeness of real operational details. Are all the exceptions included, or can they be dispensed with? If you work with prototyping, you risk that a number o f details are not included until the end.

�9 Representativeness of real operational variations. The way the organization has prescribed rules and procedures are different from how they are carried out in daily practice - during the process of formulating the requirements specification the users run the risk, subconsciously, o f switching between the prescribed procedures and the actual ones.

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

~ANBSOOI< O~ EVALUATION METHODS

�9 Representativeness o f the context o f operations. There are differences between how an activity is carried out when you have peace and quiet and nothing else to do and, for instance, when in clinical practice your activities are constantly being interrupted, and you are forced to leave a problem and return to it at a later stage. This and similar domain circumstances need to be taken into consideration in a requirements specification.

�9 Representativeness o f all the stakeholder groups that are actually involved in the assessment activities. This is particularly difficult (or rather costly) in an organization with a high degree o f specialization.

A d v i c e a n d C o m m e n t s See also under Standards.

R e f e r e n c e s

Brender J. Quality assurance and validation of large information systems - as viewed from the user perspective [Master thesis, computer science]. Copenhagen: Copenhagen University; 1989. Report No.: 89-1-22.

A rOsumO o f the method can be f o u n d in (Brender 1997a), or it can be obtained from the author.

Brender J. Methodology for assessment of medical IT-based systems- in an organisational context. Amsterdam: IOS Press, Stud Health Technol Inform 1997;42.

Brender J. Methodology for constructive assessment of IT-based systems in an organisational context. Int J Med Inform 1999;56:67-86.

This is a shortened version o f the previous reference and more accessible with regard to this subject.

Nowlan WA. Clinical workstations: identifying clinical requirements and understanding clinical information. Int J Biomed Comput 1994;34:85-94.

S u p p l e m e n t a r y R e a d i n g

Bevan N. Cost effective user centred design. London: Serco Ltd. 2000. (Available from: http://www.usability.serco.com/trump/. The website was last visited on 31.05.2005.)

This report, originating from a large E U telematics project, contains advice and guidelines f o r the formulation o f requirements, measurement o f usability, and so forth, in relation to Usability.

16s

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

~IANIDE~OOK O~ ~VALUATION JvJ{]TNODS

This little technical report also contains a lot o f literature references and links to websites on the subject o f Usability.

J84

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

I--IANDE~OOI< O~ EVALUATION METHODS

Risk Assessment

Areas of Application Identification and subsequent monitoring o f risk factors in a development or assessment project to make it possible to take preemptive action.

Description Risk is defined as the "the possibility of loss" (Hall 1998), and a risk factor is a variable controlling the likelihood o f such a loss. In other words, a risk is an aspect o f a development, assessment, or operational project, which in case it is brought to bear, will lead to an undesirable condition. There must, however, be a reasonable likelihood for such a condition to occur before it is called a risk. Therefore, risk assessment together with the matching risk control is a constructive project management tool.

Risk assessment, risk management, and risk control can be handled either retrospectively or prospectively. Retrospective risk control simply means that one keeps an eye on things, and the moment an element o f danger occurs in an activity/project, an assessment and identification o f the factors involved are carried out, and a plan for resolving it is instigated. In prospective risk control all dependencies o f a project and all aspects that are not entirely under control are identified from the start. Thereafter, measures for all o f it are found, and ongoing monitoring and assessment are carried out.

Risk assessment can be carried out either ad hoc or f o r m a l l y - the

18s

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

NAND~OOK O.IZ ~VAI_UATION METHODS

latter certainly giving the better result. But even formal techniques may not require a large methodical foundation and may be carried out by means o f consequence analysis supplemented by weighting p r i n c i p l e s - for instance, based on probability, implications o f actual occurrences (such as 'medical' side effects), chance of timely realization, resources needed to compensate or rectify the unwanted situation should it actually occur, and so on. If this semiformal level is insufficient, help should be searched for in the original literature.

Assumptions for Application Special experience is only required for strictly formal risk assessments o f large or complex projects.

Perspectives Risks will typically o c c u r - but not solely - at the interfaces between persons, organizations, and activities and will usually arise in their input/output relationships. This also includes the time aspects, particularly within the political and administrative decision process. This is because external parties do not have the same commitment and motivation and because they do not depend on a given decision or product and thus do not have the same impetus for a solution (a timely one), as does the project itself.

Frame of Reference for Interpretation The frame of reference for a development project is its objective, including its future plans for progress and resource requirement.

Perils and Pitfalls A bias occurs where there is unease in revealing specific risk factors long before these have been realized. There might, for instance, be political, psychological, or other tactical reasons to omit monitoring specific circumstances.

Advice and Comments Integration o f a prospective risk, if required, can be done by establishing monitoring points in terms of indicators (measures) o f the risk factors and if possible by defining milestones and deadlines in a project's contractual basis. One way of handling this is described in Logical Framework Approach (see relevant section), where risks are explicitly handled by means of the concept of external factors and corresponding measures that are identified and evaluated for all planned activities prior to the commencement of the project.

186

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

IdANDBOOt< OF {]VALUATION METHODS

There is a lot o f literature about this area, also concerning IT projects, so just go ahead and search for precisely that, which suits you the best.

References

Hall EM. Managing Risk: Methods for software systems development. Reading: Addison-Wesley; 1998.

I87

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

HANDE~OOK O~ EVALUATION METHODS

Root Causes Analysis

Areas of Application Exploration o f what, how, and why a given incident occurred to identify the root causes o f undesirable events.

Description Root Causes Analysis is a family o f methods, applying a sequence o f component methods: (1) schematic representation o f the incident sequence and its contributing conditions, (2) identification o f critical events or active failures or conditions in the incident sequence, and (3) systematic investigation o f the management and organizational factors that allowed the active failures to occur (Livingston et al. 2001). The reference mentioned provides an exhaustive review o f the literature and case studies applying Root Causes Analysis.

Assumptions for Application

Perspectives The highly structured and prescriptive approach will aid domain people to perform the investigation themselves, which is necessary given the need for domain insight to get hold o f the propagation o f root causes in an organization.

Frame of Reference for Interpretation (Not applicable)

Perils and Pitfalls The issue is to get to the root o f the cause o f a problem rather than to obtain a plausible explanation. The point being that in order to be able to prevent similar incidents in the future one has to get hold o f the root cause before proper action can be taken; otherwise the

166

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

IdANDE~OOI< OF:: EVALUATION M{]TIdODS

incident will reoccur.

A d v i c e and C o m m e n t s This method may be valuable in combination with Continuous Quality Improvement strategy for quality management.

R e f e r e n c e s

Livingston AD, Jackson G, Priestly K. Root causes analysis: literature review. Health & Safety Executive Contract Research Report 325/2001. (Available f r o m : www. hse. go v. uk/research/crr_~df/2 001/CRR O13 2 5 . p d f . Last visited 15.05.2005.)

Jr�9

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

I-IANE~OOI< OlZ EVALUATION MI~THODS

Social Networks Analysis

Areas of Application

Assessment o f relationships between elements within an organization (such as individuals, professions, departments, or other organizations), which influence the acceptance and use o f an IT- based solution. This could be used to identify key persons for success and opinion makers in an organization.

Description Social network analysis is carried out by means o f diagramming techniques, which do vary a little depending on the actual study (Rice and Anderson 1994; Anderson 2002): They are all described as networks with the knots being the actors (for instance, an individual, a profession, a department or a project . . . . ) and the relationships being described as named arrows between the knots. The type o f relationships could, for instance, be that o f communication, state o f competence, or economy. The relationships are described by a number o f characteristics such as frequency, type o f relationship, level or strength o f the interaction, and so on. Finally, the data collected are analyzed by means o f several techniques that illustrate the relationships.

Assumptions for Application Some experience o f the techniques used is needed.

Perspectives Attitudes toward information technology and its use are strongly influenced by the relationship between the individuals that form part o f an organization (Rice and Anderson 1994; Anderson 2002).

19o

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

IdAND~OOI< OF: EVALUATION METHODS

Frame of Reference for Interpretation (Not applicable)

Perils and Pitfalls Pitfalls, the same as under diagramming techniques in general (see under Work Procedure Analysis), are that one must judge the suitability of the specific modeling technique to answer one's information needs and to which degree variations (and exceptions) can/should be included.

Advice and C o m m e n t s This method may be used to carry out stakeholder analysis, but it may also be used as a basis to uncover what and why things happen in an organization and to identify individual or professional interfaces within or between different activities in a work procedure (Rice and Anderson 1994). This is particularly important, as both internal and external interfaces in an organization constitute points of risk where, for instance, the execution or transfer of a task may fail.

The reference (Rice and Anderson 1994) contains a vast number of references to description of methods and their use.

References

Anderson JG. Evaluation in health informatics: social network analysis. Comput Biol Med 2002;32:179-93.

Rice RE, Anderson JG. Social networks and health care informations systems, a structural approach to evaluation. In: Anderson JG, Aydin CE, Jay S J, editors. Evaluating health care information systems, methods and applications. Thousand Oaks: Sage Publications; 1994. p. 135-63.

19w

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

IdANDE~OOK OF: EVALUATION I"'IETIdODS

Stakeholder Analysis

Areas of Application Assessment of stakeholder features and their inner dynamics, aiming to identify participants for the completion of a given task, a problem-solving activity or a project.

Description An in-depth stakeholder analysis covers a whole range of considerations, including:

1. Analysis of the rationale behind undertaking a stakeholder analysis: What are the value norms or the motivation behind a stakeholder analysis (legislative, moral, ethic, or labor-related, etc.)? What difference would it make if one stakeholder group were not involved? And what potential risks will it entail if they do not get involved?

2. Analysis of what definitive characteristics of a stakeholder group determine its influence on the decision makers, including the elucidation of the stakeholders' own expectations.

3. Observation of the stakeholder group's intemal and mutual dynamics (including organizational culture, power structure, control mechanisms, etc.) and their influence on the decision- making processes as a function of intemal and extemal factors.

4. Selection of participants and participating stakeholder groups by optimizing factors ('performance management indicators') that fulfill the policy of the organization- that is, beyond those of legal consideration as, for instance, the principles of user involvement, democracy or other principles of justice, motivational principles, know-how principles, minimizing risks by involving leading figures and spokespersons, and so on.

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

~ANDE~OOK 04:: EVALUATION I'vlETI--IODS

Assumptions for Application The task o f formal stakeholder analysis is not something that just anybody can undertake, as it requires some prior knowledge o f organizational theories and administrative law. In some situations an intuitive analysis, or one with the aforementioned aspects in mind, may suffice. In other words, unless it is quite a new type o f task or a very large investigation involving several organizations (particularly if they are not known beforehand) or if there is a risk o f resistance severely compromising the course o f the project or its outcome, there is no reason for the investigation to be as explicit and thorough as indicated in the description. If that is the case, most people can carry out a stakeholder analysis.

Perspectives A frequently quoted definition o f a stakeholder within the discipline o f stakeholder analysis is Friedman's (quoted in Simmonds and Lovegrove 2002): "any group or individual who can affect or is affected by the achievement o f the organization's objectives "'. From this the perspective is derived that anybody who is influenced by a concrete solution should also be involved in the preparation o f the solution model.

As often as not a number o f conditions to make allowances for the different stakeholder interests are normally implicitly or explicitly incorporated into the organizational culture. This means that precedents from earlier projects and circulars and so on will implicitly make allowances for who is (officially) accepted as a stakeholder.

Principles o f involvement o f stakeholders in a problem-solving context are highly dependent on the culture (see the discussion on the concept o f perspective in the introductory part, Section 2.4). One o f the differences is to be found in the interpretation o f the two English notions o f 'decision maker' and 'decision taker', as seen in their extremes: 'Decision making' reflects that there is a process between the stakeholders involved in various ways, whereby a basis for a decision is created and a decision made. Conversely 'decision taking' represents the attitude that somebody has the competence to make decisions singlehandedly. In some cultures (like the Asian culture, for instance) there is no discussion about it being the leader who d e c i d e s - and he or she is presumed to be knowledgeable about issues o f relevance to the decision. Consequently, a stakeholder analysis will be superfluous to these cultures, and it should be noted that before using the Stakeholder Analysis method it must be

~9s

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

HAND~OOI,< OF EVALUATION Ms

adapted to the organization's own perspective o f stakeholder participation in a given problem-solving context.

Frame of Reference for Interpretation (Not applicable)

Perils and Pitfalls See the discussion under Perspectives: If the wrong perspective is used in an actual stakeholder analysis you run the risk o f taking account o f the wrong factors for solving the problem or for organizing the project, and if the worst is brought to bear, internal strife in the organization or an incomplete decision-making foundation will result.

Advice and Comments Logical F r a m e w o r k Approach or Social N e t w o r k Analysis can be inspirational when a formal analysis is not required.

Also see the Stakeholder Assessment method in (Krogstrup 2003), where all stakeholder groups get involved in a given assessment activity, and the result is subsequently summarized and negotiated. The same reference summarizes the Deliberative Democratic Assessment method, which also involves all the stakeholder groups. The latter focuses on identification o f value norms, and the questions asked are extremely relevant in deciding who should be involved in an assessment activity (or any sort o f task) when not everybody can take part.

References

Krogstrup HK. Evalueringsmodeller. Arhus: Systime; 2003. (in Danish) This little informative book from the social sector gives quite a good overview o f possibilities and limitations o f a couple o f known assessment models.

Simmonds J, Lovegrove I. Negotiating a research method's 'conceptual terrain': lessons from a stakeholder analysis perspective on performance appraisal in universities and colleges. In: Remenyi D, editor. Proceedings of the European Conference on Research Methodology for Business and Management Studies; 2002 Apr; Reading, UK. Reading: MCIL; 2002. p. 363-73.

The article is a theoretical debate and review, making it difficult to grasp.

194

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

IdANDBOOI< OE .I~VALUATION METHODS

Supplementary Reading

Eason K, Olphert W. Early evaluation of the organisational implications of CSCW systems. In: Thomas P, editor. CSCW requirements and evaluation. London: Springer; 1996. p. 75-89.

Provides an arbitrary rating approach to assess cost and benefit f o r the different user groups and stakeholders.

19s

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

I--dANDE~OOt< OlZ EVALUATION METHODS

SWOT

Areas of Application

�9 Situation analysis: The SWOT method is intended for establishing a holistic view o f a situation or a model solution as objectively as is feasible.

This method was originally developed to evaluate business proposals and process re-engineering effects to assist in identifying strategic issues to assist in the formulation o f a strategy.

Description SWOT is an acronym for "Strengths, Weaknesses, Opportunities, and T_hreats". For most purposes these four concepts can be used in their common sense: 'Weaknesses' are characteristics o f the object o f the study, usually ongoing and internal, and which will hinder development in the desired direction. 'Threats' are risks that lurk but do not necessarily happen and over which you do not have any control; there should be a certain probability that they may happen before you choose to include them. They are usually to be found in the internal or external interfaces o f the organization (see also under Risk Assessment). 'Strengths' are to be interpreted similarly as having the same characteristics as internal assets, while 'Opportunities' are assumptions o f possibilities or options, which might alleviate some o f the problems arising during implementation o f the actual model solution.

The SWOT method is intended to identify and analyze the four aspects mentioned for an object o f study, based on a combination o f facts, assumptions, and opinions (the description should indicate what information and o f which type). The object o f the study could, for instance, be a decision-making situation, where the choice

196

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

HANE~OOI< O~ EVALUATION MI~TIdODS

between alternative solutions has to be made or as an introduction to a discussion on problem solving in a deadlocked situation. For a review see Dyson (2004).

A SWOT analysis can be carried out in a simple way or with increasing formality depending on the size of the topic and its complexity. A detailed SWOT analysis should, for instance, include:

�9 The probability that the opportunities can be realized and utilized

�9 The possibility to eliminate weaknesses and exploit strengths �9 The probability that a given risk (threat) will occur and if so,

what the consequences will be �9 The possibility of ongoing monitoring in order to detect and

identify risks in time �9 The possibility to compensate for a given risk should it become

threatening

Assumptions for Application

Perspectives The SWOT method is an incredibly simple and yet useful tool whose primary function is that of a framework structure ensuring the preparation of an elaborate picture for any given situation.

Frame of Reference for Interpretation (Not applicable)

Perils and Pitfalls By nature the method is subjective. Therefore, should an organization have strongly conservative forces or groups of employees who are afraid of change, a certain aspect could move from one category to another. For example, some could perceive the introduction of IT in an organization as a threat because it could entail rationalization and redundancies, while to management it can be an opportunity to solve a bottleneck problem in the organization.

The method can be misused if it is not used in an unbiased way - hence, group work is recommended. What might happen is that the balance between the sequences of aspects included is (sub)consciously askew or that important aspects are left out.

197

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

I-IANDBOOI< OE ~VALUATION METHODS

Advice and Comments The method is useful as a brainstorming method to elucidate and evaluate each o f the aspects brought forward.

References

Dyson RG. Strategic development and SWOT analysis at the University of Warwick. Eur J Operational Res 2004; 152(3):631-40.

Supplementary Reading

Balamuralikrishna R, Dugger JC. SWOT analysis: a management tool for initiating new programs in vocational schools. J Vocational Technical Education 1995;12(1). (Available from: http://scholar.lib.vt.edu/. Last visited 10.05.2004.)

An example o f its use as an illustration.

Jackson SE, Joshi A, Erhardt L. Recent research on team and organizational diversity: SWOT analysis and implications. J Manag 2003 ;29(6):801-30.

An extensive literature appraisal applying the S W O T method f o r the analysis o f the literature.

sq6

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

~ANDi~OOK OlZ EVALUATION I'vq~TIdODS

Technical Verification

Areas of Application Verification that the agreed functions are present, work correctly, and are in compliance with the agreement.

The purpose o f technical verification is to ensure that the management o f the organization can/will take administrative responsibility for the operation o f an IT-based system and for its impact on the quality o f work in the organization. This should not be understood as the responsibility o f an IT department for the technical operation o f the IT system, but as users applying the system in their daily operations such as their clinical work. This is the case:

�9 In connection with acceptance tests at delivery o f an IT system or a part-delivery

�9 Prior to taking the system into daily operation and before all subsequent changes to the IT system (releases, versions, and patches)

Description With the exception o f off-the-shelf IT products (see under perspectives below), technical verification is carried out by checking the delivery, screen by screen and field by field, interface by interface, function by function, and so on to see if everything is complete, correct, consistent, and coherent. For each o f the modular functionalities agreed on the following questions are assessed:

1. Re.: Completeness o f the functionality �9 Is everything that was promised included? �9 Has it been possible to verify all aspects o f the functionality

supplied, thereby all/most possibilities o f data input, including exceptions? Or has it been impossible due to errors or shortfalls in the product delivered?

�9 To which degree has it been possible for the users to simulate normal daily activities?

�9 To which degree does the functionality supplied comply with that expected functionality?

�9 How well does the system's functionality fulfill the functionality described in the c o n t r a c t - that is, the actual

E�9169

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

I--IANDE~OOtC OF {~VALUATION IVJE]TI4ODS

requirements in the contract? 2. Re.: Correctness of the functionality

�9 Does the system work as it should? That is, can the users work with data that have a logical similarity to normal production and function?

�9 And does it all happen correctly? 3. Re.: Coherence of the functionality

�9 When data depend on each other (some data are only valid when other given data have actual values): Does the system control these internal constraints and ensure that data are correct?

�9 Or does the user have to ensure this manually? 4. Re.: Consistency of the functionality

�9 Is the data being duplicated, running the risk of getting different instances of the same information?

�9 And has this actually been observed as happening? This normally only happens when several independent IT systems are put together to make up one IT-based solution.

5. Re.: Interconnectivity of the functionality: This means that technical (syntactic) aspects of communications between several IT-based systems, such as communication of patient data from one patient-administration system or a hospital information system to a laboratory system, and communication with systems dedicated to controlling user-access criteria.

�9 Is it the right (complete and correct) information that is received?

6. Re.: Interoperability of the functionality: The semantic aspects of the interaction between several IT-based systems. One example is that the change of a patient's hospital address or status in a patient administration system has to be communicated to other systems that depend on this information. See further details under Measures and Metrics. This is not necessarily updated immediately in the interconnected systems. It has a big impact on the perceived functionality in a ward, irrespective of how correctly an IT system seems to work. Coordination of updated data between various systems can easily go wrong. One reason could be that, for capacity reasons, the system developers store this type of information in a queue, which is only emptied every five minutes, for instance. This could be significant under some acute conditions.

7. Re.: Reliability: �9 How often does the system go down, and what are the

"200

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

HANDE~OOI< OF EVALUATION J'vlETNODS

consequences? �9 How does the system handle error situations, and what

interventions are necessary to re-create data or reestablish operations when this happens?

�9 What happens when an error situation is created on purpose - d u r i n g simulated breakdowns of parts of the network, for instance?

�9 What happens at monkey tests or at a database locking? 8. Re.: Performance: Response time and throughput time for

transactions on the system, as well as capacity. See also under Measures and Metrics.

What types of problems are users likely to run into? And how do they relate to the contract?

See also many of the concepts under Measures and Metrics.

Assumptions for Application It is a prerequisite that each of the requirements of the contract are operational or that they can be rendered operational without problem.

It requires previous experience in planning and implementation of such investigations, as well as punctiliousness with regard to details. It is an advantage to have tried it before. Should one want an in- depth technical verification, it could be an advantage to seek help for the more complex i s s u e s - for instance, to ensure systematism in the verification of the interoperability.

Perspectives In a contractual relationship with regard to the supply of a system, you cannot expect more than what is stipulated in the agreement between the customer and the vendor. Technical verification serves the purpose o f assessing whether the contract has been fulfilled. Thus, the contract is an indispensable flame of reference for technical verification. This does not necessarily mean that the system is appropriate, even if it has been verified with a successful outcome, or that it works properly, should the contract not make sure of this. Remember, it is not easy to formulate a good requirements specification. Management then must decide whether the system is good enough for it to undertake administrative responsibility for the system in operation. If not, it is necessary to

2OI

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

~ANIESOOK OF: EVALUATION blI~TPlODS

take the consequences and say "Stop", regardless of whether the fault lies with the vendor or the client, and thereafter to find out what the next step should be.

Whether the vendor of the IT-based system is internal or external makes no difference - that is, whether it is an independent company or the IT department of the hospital or the region.

Technical verification of off-the-shelf products, such as Microsoft Word for word processing, has a different purpose if the organization chooses to carry it out. For instance, this could be to get acquainted with the details of the system's functions in order to establish new and changed work processes. Whether the product is used for its intended purpose and whether management undertakes responsibility for its introduction are questions that must be addressed before its purchase.

Frame of Reference for Interpretation The frame of reference is the contract - or a similar agreement with attachments.

Perils and Pitfalls The most common pitfall is the lack of testing of specific conditions, which then show up during day-to-day use. The possible combinations, the aspects of capacity in its widest sense and the interoperability are particularly difficult to t e s t - also for the vendor. Furthermore, not all vendors have sufficiently detailed insight into the application domain to be able to test from a real-life perspective, implying that the focus and efforts put into technical verification depend on the number of similar reference sites of the IT system in question. In other words, even if the vendor has carried out a thorough test, undetected errors and omissions may occur when new people examine the same system with different data.

Advice and Comments In practice it is impossible to physically test all fields, types of data, and functions of nontrivial IT systems in all their relevant and irrelevant combinations. It is important explicitly to keep to this in an accreditation or certification situation (see separate description in Chapter 8).

"202

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

I-IANE~OOI< OlZ EVALUATION I'qI~THODS

R e f e r e n c e s (See under Measures and Metrics.)

? 0 3

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

HANDBOOI< OF EVALUATION METHODS

Think Aloud

Areas of Application An instrument for gaining insight into the cognitive processes as feed-back for the implementation and adaptation of IT-based systems.

Description Think Aloud is a method that requires users to speak as they interact with an IT-based system to solve a problem or perform a task, thereby generating data on the ongoing thought processes during task performance (Kushniruk and Pate12004). The user interaction data collected typically include the video recording of all displays along with the corresponding audio recording of the users' verbalizations. The resulting verbal protocols together with the videos are transcribed and systematically analyzed to develop a model of the user's behavior while performing a task.

Assumptions for Application Kushniruk and Patel (2004) describe carefully the recording and analysis tasks and illustrates implicitly the need for prior experience of using this method: The data constitutes raw data that require substantial analysis and interpretation to gain in-depth insight in the way subjects perform tasks.

Perspectives Jaspers et al. (2004) outline the various types of memory systems with different storing capacities and retrieval characteristics, putting across the message that humans are only capable of verbalizing the contents of the working memory (the currently active information),

7 0 4

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

HANIDBOOK O~ ~VALUATION M{]TIdODS

not the long-term memory.

A perspective behind this method is that the user's mental model is independent o f his or her level o f expertise. Thus, while using the output as input for a development process, the key issue is to elicit the joint mental model across different levels o f expertise, as these present themselves differently in the interaction with the IT-based system (see also the discussion under Pitfalls in Cognitive Assessment, as well as in Part III). This perspective is clearly opposed by Norman (1987) as being incorrect, stating that mental models are incomplete, parsimonious, and unstable; they change, and do not have firm boundaries. Nevertheless, the Think Aloud method may have its value for small and down-to-earth practical assessments, provided that the assessor knows these constraints and consequently does not overinterpret the outcome.

Frame of Reference for Interpretation (Not applicable)

Perils and Pitfalls

Advice and C o m m e n t s

References

Jaspers MWM, Steen T, van den Bos C, Genen M. The Think Aloud method: a guide to user interface design. Int J Med Inform 2004;73:781-95.

Kushniruk AW, Patel VL. Cognitive and usability engineering methods for the evaluation of clinical information systems. J Biomed Inform 2004;37:56-76.

Norman DA. Some observations on mental models. In: Baecker RM, Buxton WAS, editors. Readings in human-computer interaction: a multidisciplinary approach. Los Altos: Morgan Kaufman Publishers, Inc.; 1987. p. 241-4.

205

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

HANDBOOI( OF- EVALUATION METHODS

Supplementary Reading

Cho I, Park H-A. Development and evaluation of a terminology-based electronic nursing record system. J Biomed Inform 2003;36:304-12.

Use the Think Aloud method to identify success and failure in matching a precoordinated phrase with what the user wanted to document.

Ericsson KA, Simon HA. Protocol analysis, verbal reports as data. Cambridge (MA): The MIT Press; 1984.

This book provides a thorough review o f the literature on approaches to and problems o f elicitation o f information about cognitive processes from verbal data.

Preece J. Part VI, Interaction design: evaluation. In: Human-computer interaction. Wokingham: Addison-Wesley Publishing Company; 1994.

Part VI o f this book describes a number o f conventional methods f o r measuring Usability, from traditional Usability Engineering via analysis o f video data, verbal protocols, Think Aloud protocols, Interviews, and Questionnaire studies to ethnographic studies. Even though the perspective on page 693 indicates that the methods are intended f o r professionals, it may still be useful

f o r others to read these chapters.

~o6

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

IdANDE~OOI< O~ ~VALUATION J"IETNODS

Usability Areas of Application

"Usability is the quality o f interaction in a context" (Bevan and Macleod 1993)

�9 Assessment o f user friendliness in terms of ergonomic and cognitive aspects of the interaction between an IT system and its users.

It can be used during all phases of an IT system's life c y c l e - for instance:

�9 During the analysis and design phase (the Explorative Phase) �9 For assessment of bids (the Explorative Phase) - see (Beuscart-

Zrphir et al. 2002 and 2005) and Assessment o f Bids �9 In connection with a delivery test (after installation at the end of

the Technical Development Phase) �9 As constructive assessment during the implementation process

or during adjustment of the functionality, also called usability engineering. See, for instance, the reviews in (Kushniruk 2002; and Kushniruk and Pate12004).

Description Usability is an aspect that must be taken into account in the functionality from the very first description of the idea and the objective of introducing the system until the system functions (well) in day-to-day use and during maintenance.

This is a very large subject area, stretching from vendors' and universities' advanced 'usability labs' with video monitoring of eye and hand movements attached to a log that tracks everything that goes on in the system. On the opposite scale you will find an assessment during hands-on use in a workplace. Somewhat dependent on the authors, the concept includes both ergonomic and cognitive aspects, as they are closely related. As already defined in a previous footnote (see Field Studies), this handbook distinguishes between the work procedure-related aspects (usually denoted as the functionality aspects), the dialogue-related aspects (the ergonomic aspects), and the perception-oriented aspects (thecognitive aspects).

"207

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

}-IANDE}OOI< OlZ EVALUATION METHODS

There is definitely an overlap between them, and they are all concerned with the functionality that users are confronted with during daily use of an IT-based system.

For a review of inspection methods as well as different measurement methods ranging from formal over automatic and empirical to informal usability assessment methods, see (Huart et al. 2004).

Measurement of cognitive aspects is still at an experimental stage with regard to assessment of IT-based systems and has not yet been sufficiently addressed in the development methodologies. This is one of the reasons why only simple measurements and assessments of ergonomic aspects are included in this section, as they are easy enough to undertake by a user organization with some prior experience. Assessment of the cognitive aspects is described elsewhere (see separate sections, Cognitive Assessment and Cognitive Walkthrough).

Usability, with regard to ergonomic assessments, is concerned with characteristics of how difficult it is to carry out a user dialog with the IT system o r - phrased in a positive sense - how effective an IT system is to use and how easy it is to learn to use it.

The work of ergonomic assessments is task-oriented that is, you simulate real small operational tasks. Use scenarios from every day as a base to identify suitable tasks. Correction o f data is usually a good task to evaluate in this respect, although data entry and other limited activities should also be assessed. For example, how effective does the user interface work when the user uses the prototype to simulate an actual task, such as correcting a patient ID? The effectiveness can be calculated in (1) the number of shifts between screens; (2) the distance between fields on a given screen where data has to be entered or changed in other words, how many times and how far does the cursor have to be moved actively; or (3) is it necessary manually to correct data that are logical functions of each other (for instance, affiliation of staff and patients to a ward and a department) with the risk of inconsistencies?

In practice this has been made operational by a number of metrics and suggestions of things to keep an eye out for. The report (Bastien and Scapin 1993), a pioneering work in this area, goes through a number of examples of relevant aspects including recommendations and criteria gathered under a series of headings, such as:

�9 Prompting: Which tools are used to prompt the user to the next

208

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

~ANI}8OOI< OF {=VALUATION J"'IETIdODS

specific step o f an action (a data entry field or a new action)? �9 G r o u p i n g o f information a n d f i e l d s : Is there a logical task-

related, coherent, visual organization o f the fields? This could be a topological (area) or a graphical (color, format, or similar) grouping

�9 I m m e d i a t e f e e d - b a c k : What is the immediate reaction o f the system when the user performs something and completes it?

�9 Readability: Concerns characteristics o f the readability, equivalent to the lix value (readability index) o f a text. Is it readable (also for those who have early stages o f age-related eyesight problems or are colorblind)?

�9 L o a d (on the user): This concerns all aspect that play a part in reducing the sensual and cognitive strains on the user and the effectiveness o f the dialog between the user and the system

�9 Consistency: Deals with uniformity o f the design o f the screens (does it show that different people have programmed it?)

�9 H a n d l i n g o f errors: Deals with tools to handle errors and ways to prevent them

�9 Flexibility: Deals with the overall flexibility o f the system

Assumptions for Application Experience is a prerequisite to really be able to employ the usability method and to profit from it as constructive feed-back tool during a development process. However, a usability assessment requires special professional background to have any deep impact.

Perspectives Bastien and Scapin's report builds on an anthropocentric perspective that is, the users are not robots that have to leam and adapt to the system and that the best overall function is achieved when the user understands what goes on and is in charge o f what happens and what needs to be done (see Bastien and Scapin 1993).

Usability is not just a fashion fad, but it has a decisive influence on user satisfaction, learning and training needs, and the strain o f screen work, so it influences the frequency o f operational errors and omissions. The ergonomic aspects are not only decisive for the efficiency o f the operation o f the system, but also when users try to avoid certain activities. For example, if the system is too cumbersome to handle, there is a risk that the user will carry out the activities in an incorrect way or maybe jot down the information on a piece o f paper, put it in a pocket, and delay the inputting or even forget all about it.

2oo

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

~ANID~OOI< O~ c EVALUATION METHODS

F r a m e o f R e f e r e n c e for Interpretation The report (Bastien and Scapin 1993) m a y work as a de facto standard for what is ' g o o d practice' - at least from the view o f what a user organization is normally capable o f dealing with itself.

Perils and Pitfalls A significant source o f error is the bias inherent in p e o p l e ' s subjective evaluations (see Kieras 1997 and Part III): One has to find objective metrics and measures if the ergonomics are a point o f discussion and negotiation with a development team or, even worse, with an external vendor.

A d v i c e and C o m m e n t s Y o u can find some good rules o f thumb for evaluations and assessments on http://jthom.best.vwh.net/usability/under "heuristic evaluation" under the reference to "Nielsen".

W h e n assessing usability, one should be aware that the context in which measures are taken is very important and needs to simulate real life as closely as possible (i.e., ideally in actual use) (Bevan and Macleod 1993; Coolican 1999).

I f the usability cannot be measured directly, it is sometimes possible to register and analyze the causes behind typical errors o f operation and other unintentional events.

References

Bastien JMC, Scapin DL. Ergonomic criteria for the evaluation of human-computer interfaces. Rocquencourt (France): Institut National de Recherche en Informatique et en Automatique; 1993. Report No.: 156.

The report is available from the institute's website: http : //www. inria.fr/pub lications/index, en. html. Click on 'Research Reports & Thesis' type "ergonomic criteria" in the text field o f the search engine, and press 'Enter'. Find the report RT0156, mentioned under the description o f 'Usability'assessment. By clicking on 'pour obtenir la version papier' you get the e-mail address o f the person concerned, and you may request the report (last visited 10.12.2003).

Beuscart-Z6phir MC, Watbled L, Carpentier AM, Degroisse M, Alao

210

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

IdANI}8OOK OF ~VALUATION METHODS

O. A rapid usability assessment methodology to support the choice of clinical information systems: a case study. In: Kohane I, editor. Proc AMIA 2002 Symp on Bio*medical Informatics: One Discipline; 2002 Nov; San Antonio, Texas; 2002. p. 46-50.

Beuscart-Z6phir M-C, Anceaux F, Menu H, Guerlinger S, Watbled L, Evrard F. User-centred, multidimensional assessment method of clinical information systems: a case study in anaesthesiology. Int J Med Inform 2005;74(2-4): 179-89.

Bevan N, Macleod M. Usability Assessment and Measurement. In: Kelly M. Management and measurement of software quality. Uxbridge: Unicom Seminars Ltd; 1993. p. 167-92.

An easily accessible opening for usability measurement.

Coolican H. Introduction to research methods and statistics in psychology. 2nd ed. London: Hodder & Stoughton; 1999.

Describes (mainly in the early chapters) problems o f experimental conditions o f psychological phenomena.

Huart J, Kolski C, Sagar M. Evaluation of multimedia applications using inspection methods: the Cognitive Walkthrough case. Interacting Comput 2004; 16:183-215.

Kieras D. A guide to GOMS model usability evaluation using NGOMSL. In: Helander MG, Landauer TK, Prabhu PV, editors. Handbook of human computer interaction. 2nd ed. Amsterdam: Elsevier Science B.V.; 1997. p. 733-66.

Kushniruk A. Evaluation in the design of health information systems: application of approaches emerging from usability engineering. Comput Biol Med 2002;32(3): 141-9.

Kushniruk AW, Patel VL. Cognitive and usability engineering methods for the evaluation of clinical information systems. J Biomed Inform 2004;37:56-76.

http://jthom.best.vwh.net/usability The page contains numerous links to method descriptions and references, including, for instance, Jacob Nielsen "s ten recommended aspects (last visited 15.05.2005).

Supplementary Reading

Baecker RM, Buxton WAS. Readings in human-computer interaction:

211

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .