Comparison of Key Evaluation Models in Health Informatics

mshp

Chapter-7-Descriptionofmethodsandtechniques.pdf

Home >Nursing homework help > Comparison of Key Evaluation Models in Health Informatics

IdANDBOOI< OF EVALUATION J"I{~TIdODS

7. Descriptions of Methods and Techniques

(Shown in alphabetical order by name o f method or technique)

Analysis of Work Procedures

Areas of Application When assessing what actually happens compared to the expectations o f what should happen, for instance, with respect to:

�9 Analysis of the extent to which an IT system is being used as expected (during the implementation phase)

�9 Impact assessment (during the Evolution Phase)

Description The methods listed here are not dedicated or designed specifically for evaluation studies, but they may clearly have a role as part o f an evaluation study. The intention thus is to provide an overview of some well-known options:

�9 The Leaming Organization: This paradigm theory and its strategies focus on analysis o f work procedures to ascertain whether the organization has the ability consciously to incorporate experiences, whereby new or changed structures and work process are continuously established (see Argyris and Sch6n 1996).

�9 Enterprise modeling: A number of diagramming techniques are used for enterprise modeling. A recent overview of some of these is provided by Shen et al. (2004).

�9 Business Process Reengineering: Here the focus o f the work process analysis is aimed at radical innovation and changes in work processes within the organization (see, for instance, the review in Willcocks 1996).

�9 Use Cases and scenarios: Use Cases describe the expectations of

Brender, McNair, Jytte, and Jytte Brender. Handbook of Evaluation Methods for Health Informatics, Elsevier Science & Technology, 2006. ProQuest Ebook Central, http://ebookcentral.proquest.com/lib/waldenu/detail.action?docID=306691. Created from waldenu on 2022-03-04 00:22:29.

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

HANI38001< O~ ~VAL._UATION M{~THODS

what the IT system should accomplish in a given situation but not h o w - that is, the method aims at capturing the essence o f the activities in a purpose-oriented way. Some variations of Use Cases use a combination o f present and future cases. For further details, see, for instance, (Alexander and Maiden 2004). A scenario is another way of capturing the essence of activities. See, for instance, (Carroll 1995; and Alexander and Maiden 2004).

�9 Total Quality Management: This focuses on flaws and shortcomings in an organization with the specific aim o f redressing them.

�9 Health Technology Assessment (HTA): Here it is the framework for assessment rather than the actual methods and techniques that are used. Goodman and Ahn (1999) provide a basic overview of HTA principles and methods.

�9 Computer-Supported Cooperative Work (CSCW): The focus on the work process analysis here is on the interaction between co- operating partners. As Monk et al. (1996) state, "[A]ll work is cooperative, and so any work supported is computer-supported cooperative work (CSCW)". These can be either person-to- person or person-to-machine relationships. An important aspect of this form o f analysis is to define tasks and delegate responsibility according to competencies. See, for instance, (Andriessen 1996) and other chapters in the same book.

�9 Cognitive Task Analysis: This approach, which is addressing the cognitive aspects of work procedures, is an emerging approach to the evaluation of health informatics applications. Kushniruk and Patel (2004) review this field.

Assumptions for Application The use of diagramming techniques and other forms of graphical modeling requires experience, including that o f understanding the principles and procedures involved, their pitfalls, perils, and assumptions. Often it also requires the ability to conceptualize, as it is impossible to include all the details and variations o f the organizational work processes. On the other hand, users have special means of interpretation and conception - thereby capturing what goes on and what is and is not important.

Perspectives Virtually all traditional diagramming techniques or scenarios for the descriptions of work processes are based on the assumption that work processes are either accomplished sequentially with little variation or that all relevant aspects can, nevertheless, be safely represented by

7/4

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

tdANDt~OOI< O~C EVALUATION METHODS

means of the diagramming technique in question. Variations of work processes are exactly the aspects that are otten not shown in diagram techniques and scenarios:

"[W]orkprocesses can usually be described in two ways: the way things are supposed to work and the way they do w o r k . . . People know when the 'spirit o f the law' takes precedence over the 'letter o f the law'"

(Gruding 1991)

It is still outside the capability of diagramming techniques to distinguish and incorporate ergonomic and cognitive aspects of the work processes, including the difference between how beginners and experts work and think. See, for instance, the discussion in (Dreyfus 1.997) or the rrsum6 in (Brender 1999).

Frame of Reference for Interpretation If one wishes to measure the effect of an IT system from a work process point of view, it is a prerequisite that there be an explicit and unambiguous frame of reference, also called baseline values. Depending on the questions of the study, the frame of reference could, for instance, be the description of the work process for the 'old' system or for the intended new system (from a preliminary analysis or from the User Requirements Specification). The frame of reference could also be what was prescribed for the new or changed work processes prepared before the 'new' system was put into operation.

A frame of reference is normally not used during the Explorative Phase of an investigative study of work processes.

Perils and Pitfalls Pitfalls in connection with descriptions of work procedures are indirectly evident from the aspects mentioned under Perspectives above. In other words, the most important thing is to find the method that will best describe exactly the information needs you have in connection with the context given.

As the methods described are so different, careful attention should focus on the objective of the study before choosing a method, while searching for descriptions of advantages and disadvantages.

Advice and Comments Methods normally used for analysis and design work during systems

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

)--tAND~OOI< O~ EVALUATION I"qETIdODS

development could potentially be used for assessment studies in cases o f qualitative and explorative assessments. However, it ot~en requires some lateral thinking, and maybe the process needs to be adjusted to the project in hand.

In principle many o f the systems analysis methods can be used for this purpose. It could certainly be an advantage to employ a technique that has already been used in the organization. This way you will draw on existing experiences o f the method while getting the opportunity to apply useful existing material as a frame o f reference for evaluating the effect. See also under Functionality Assessment, for example.

References

Alexander IF, Maiden N. Scenarios, stories, use cases, through the systems development life-cycle. Chichester: John Wiley & Sons Ltd.; 2004.

Andriessen JHE. The why, how and what to evaluate of interaction technology: a review and proposed integration. In: Thomas P, editor. CSCW requirements and evaluation. London: Springer; 1996. p. 107-24.

Argyris C, Schrn DA. Organisational learning II, theory, method, and practice. Reading: Addison-Wesley Publishing Company; 1996.

Brender J. Methodology for constructive assessment of IT-based systems in an organisational context. Int J Med Informatics 1999;56:67-86.

Carroll JM, editor. Scenario-based design, envisioning work and technology in system development. New York: John Wiley & Sons, Inc.; 1995.

Dreyfus HL. Intuitive, deliberative and calculative models of expert performance. In: Zsambok CE, Klein G, editors. Naturalistic decision making. Mahwah: Lawrence Erlbaum Associates, Publishers; 1997. p. 17-28.

Goodman CS, Ahn R. Methodological approaches of health technology assessment. Int J Med Inform 1999;56(1-3):97-105.

Gruding J. Groupware and social dynamics: eight challenges for developers. Scientific American 1991 ;Sept:762-74.

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

IdANDBOOI< OF: EVALUATION METHODS

Kushniruk AW, Patel VL. Cognitive and usability engineering methods for the evaluation of clinical information systems. J Biomed Inform 2004;37:56-76.

Monk A, McCarthy J, Watts L, Daly-Jones O. Measures of process. In: Thomas P, editor. CSCW requirements and evaluation. London: Springer; 1996. p. 125-39.

The publication deals with features o f conversation and is thereby a valuable contribution f o r all assessments that address conversations.

Shen H, Wall B, Zaremba M, Chen Y, Browne J. Integration of business modeling methods for enterprise information system gathering and user requirements gathering. Comput Industry 2004;54(3):307-23.

Willcocks L. Does IT-enabled business process re-engineering pay off?. Recent findings on economics and impacts. In: Willcock L, editor. Investing in information systems, evaluation and management. London: Chapman & Hall; 1996. p. 171-92.

Supplementary Reading

Eason K, Olphert W. Early evaluation of the organisational implications of CSCW systems. In: Thomas P, editor. CSCW requirements and evaluation. London: Springer; 1996. p. 75-89.

Provides an arbitrary rating approach to assess cost and benefit f o r the different user groups and stakeholders.

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

HANDEt'OOI< OlZ:: .I=VALUATION METHODS

Assessment of Bids

Areas of Application Comparative assessment o f a number o f offers from one or more bidders/vendors

Description There is not yet a really good method for the assessment o f bids from one o f more vendors that covers the variety o f bid procedures. Thus, the existing approaches need to be adapted to the requirements specification upon which the bids have been made. The consequences are that all the procedures described use a multimethod approach. Therefore, the following points are meant only to be inspirational examples and food for thought.

A. (Brender and McNair 2002; Brender et al. 2002): The method described uses a spreadsheet-based bid table, where each single requirement is numbered and given a value in accordance with the level o f need (compulsory, desirable) and with the bidders indication o f fulfillment in their offer (0, 1 ..... 3). Initially an analysis will be made o f whether the offers overall indicate that there could be problems with one or more o f the requirements. Then, each bid is evaluated overall against the client's interpretation o f the same contingency table, based on a demo and a hands-on test, and so forth. Finally, an assessment o f all deviations is made, specifically with regard to an evaluation o f the overall potential to meet the objectives and the consequences o f any shortfalls.

B. (Beuscart-Zrphir et al. 2002, 2003, and 2005): The usability aspects (see description in separate section) are used as a point o f focus for a practical assessment o f the offer, combined with a heuristic evaluation or a multimethod approach to assess other aspects, both emanating from a week-long, on-site, hands-on demo test.

C. Rating & Weighting Techniques: The procedure o f techniques for rating & weighting is to give each requirement a priority, then to multiply its degree o f fulfillment in the bid solution with the priority. Finally, all numbers are summed into a single figure for all r e q u i r e m e n t s - the objectives fulfillment. See, for instance, (Jeanrenaud and Romanazzi 1994; and Celli et al. 1998).

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

~IAND~OOK O~ EVALUATION J'IE---THODS

D. Balanced Scorecard: See separate description o f this method. E. Structured product demonstrations (Stausberg 1999): This

methodology entails a gradual selection based on a strategic objective and technical functionality requirements, followed by a combination o f three activities: (1) demos with quantitative assessments, (2) benefit analysis, and (3) test installation.

Assumptions for Application Before the preparation o f a requirements specification, it is important to be clear on which assessment method will be used. This is because the chosen method has to comply with the type o f the requirements specification and with the principles used to prepare it.

Several o f the methods need full or part installation o f the systems and products that are to be assessed. This necessitates that the supplier is willing to install the product and also that it is possible to do so in the organization, as well as within its technical infrastructure. In itself, this may require a substantial demand on resources.

Method A assumes a stringent and formalized bid, which either is or can be converted to distinct response categories, as in '0' for "not included and not p o s s i b l e " , . . , to '3' for "fully achieved by the offer". It needs to be possible for the categories to be converted into objects for either mathematical or logical operations in a spreadsheet.

Method B assumes (1) that the bids are based on ready-made products; (2) that the aspects o f usability in the requirements specification are either operational or that professional assessors can assist with the rating; and (3) that the task and activity descriptions o f the organization are included as part o f the domain description in a requirements specification and therefore can be part o f the frame o f reference for the assessment.

Method C above assumes (1) that the requirements are mutually independent; (2) that different, equal, and equally desired solutions for a (partial) function will obtain the same outcome; and (3) that the assessment o f the degree to which each requirement fulfills the evaluation criteria can be converted into mathematical calculations.

From the methods mentioned above, it is only method B that requires special experience o f methods.

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

ldANIZ~OOI< OF EVALUATION METHODS

Perspectives Cultural factors are significant not only for the choice of the actual IT solution and how users at different levels within the organization get involved, but also for organizing the procedure and thereby the methods applied for the acquisition of IT systems. As a matter of fact, within the EU tenders are required for the procurement of purchases in excess of a certain financial limit, and there are regulations for instance in Denmark for the involvement of personnel with regard to technological developments. This is markedly different in other parts of the world, as, for instance, in the United States and Asia. These differences have a decisive influence on the choice of method for the preparation of a requirements specification and for a bid assessment. The methods described are all useful within the Westem culture.

It is rare that an offer from a vendor meets all the demands from the start and at a reasonable price. In other words, making a choice will need to be some 'give and take'.

All the methods are based on the notion that one cannot rely completely on vendors' statements, so one has to check the suitability for its actual use to make sure that the bidder cannot get around any problems (Rind 1997). This is not necessarily lack of honesty but could be due to too little understanding of the actual use - hence ignorance.

In the same way there may be hidden agendas in the user organization, or individuals among the stakeholders involved may have formed certain preconceptions. The choice of method should help to ensure objectivity.

Frame of Reference for Interpretation The formal frame of reference for the assessment is the basis of the offer that is, a requirements specification or similar, in which - according to European regulations (in case of tendering) - the assessment model and its criteria have to be specified.

Perils and Pitfalls It is important not just to select the bid with the highest score and 'jump to it', but to optimize the objectives fulfillment (at strategic and tactical levels) and to minimize potential problems and risks in the deviation between the requirements and in the future model

6 o

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

HANDE~OOI< OF {]VALUATION ME--TIdODS

solution. Beuscart-Zfphir et al. (2002) document how easily one can be duped by first impressions.

One pitfall that all the outlined methods have in common is that there can be decisive cultural differences - for example, in how users can or cannot be involved (at various levels within the organization) when selecting the future solution.

Advice and Comments There is not much literature about methods for choosing bids, probably because (1) it is very difficult to validate the suitability, so they are rarely published and because (2) it is inherently difficult to judge whether one has actually chosen the right solution. To further complicate matters, the fact is that the method publications found about the subject in the literature do not follow up with a causal analysis o f unexpected and unintentional occurrences in the future system against the method o f choice. For example, the Rating & Weighting method is mentioned several times in the literature, but the consequence o f the demand for mutual independence o f the requirement items is not sufficiently discussed. And common to the three methods A, B, and E is the fact that the conclusions and thereby their suitability have not been verified by a third party.

In order to optimize the objectives fulfillment it is important to acquire a thorough understanding o f the solution by means o f demos, hands-on, conversations with the vendor and reference clients, as well as site visits, and so forth. The reason being that the vendor's sales and marketing people are trained to camouflage or avoid possible shortcomings in the solution offered (Rind 1997).

Regarding method B, the week-long hands-on test o f the system turned out to be decisive for the final choice o f bid because it was not until then that the nitty-gritty details were discussed.

Bevan (2000) deals with human error in the use o f IT systems. It is a valuable source o f inspiration - for instance, as a checklist when assessing system design, and thus also when assessing a vendor's bid.

See also under the description o f Standards.

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

~IAND~OOI< OF EVALUATION J"qETNODS

References

Beuscart-Z6phir M-C, Menu H, Evrard F, Guerlinger S, Watbled L, Anceaux F. Multidimensional evaluation of a clinical information system for anaesthesiology: quality management, usability, and performances, In: Baud R, Fieschi M, Le Beux P, Ruch P, editors. The new navigators: from professionals to patients. Proceedings of the MIE2003; 2003 May; St. Malo, France. Amsterdam: lOS Press. Stud Health Technol Inform 2003;95:649-54.

Beuscart-Z6phir MC, Watbled L, Carpentier AM, Degroisse M, Alao O. A rapid usability assessment methodology to support the choice of clinical information systems: a case study. In: Kohane I, editor. Proc AMIA 2002 Symp on Bio*medical Informatics: One Discipline; 2002 Nov; San Antonio, Texas; 2002. p. 46-50.

Beuscart-Z6phir M-C, Anceaux F, Menu H, Guerlinger S, Watbled L, Evrard F. User-centred, multidimensional assessment method of clinical information systems: a case study in anaesthesiology. Int J Med Inform 2005 ;74(2-4): 179-89.

Bevan N. Cost effective user centred design. London: Serco Ltd. 2000. (Available from http://www.usability.serco.com/trump/. The website was last visited 15.6.2005.)

Brender J, Schou-Christensen J, McNair P. A case study on constructive assessment of bids to a call for tender. In: Surj~n G, Engelbrecht R, McNair P, editors. Health data in the information society. Proceedings of the MIE2002 Congress; 2002; Budapest, Hungary. Amsterdam: IOS Press. Stud Health Technol Inform 2002;90:533-38.

Brender J, McNair P. Tools for constructive assessment of bids to a call for tender- some experiences. In: Surj~in G, Engelbrecht R, McNair P, editors. Health data in the information society. Proceedings of the MIE2002 Congress; 2002; Budapest, Hungary. Amsterdam: lOS Press. Stud Health Technol Inform 2002;90:527-32.

Celli M, Ryberg DE, Leaderman AV. Supporting CPR development with the commercial off-the-shelf systems evaluation technique: defining requirements, setting priorities, and evaluating choices. J Healthc InfManag 1998;12(4):11-9.

A fairly thorough case study using the rating & weighting method

Jeanrenaud A, Romanazzi P. Software product evaluation metrics: a methodological approach. In: Ross M, Brebbia CA, Staples G, Stapleton J, editors. Proceedings of the Software Quality Management II

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

ldAND~OOI< O~ EVALUATION METHODS

Conference "Building Quality into Software". Southampton: Comp. Mech. Publications; 1994. p. 59-69.

Rind DM. Evaluating commercial computing systems. [Editorial]. M.D. Computing 1997; 14(1):6-7.

Editorial comment pointing out the bias in vendors'sales talk.

Stausberg J. Selection of hospital information systems: user participation. In: Kokol P, Zupan B, Stare J, Premik M, Engelbrecht R, editors. Medical Informatics Europe '99. Amsterdam: IOS Press. Stud Health Technol Inform 1999;68:106-9.

Supplementary Reading Please note the leg&lative difference between the United States and the European Union. In the E U all purchases over a certain value have to be submitted to tender. Nevertheless, the references all contain inspiration and good advice f o r one's own evaluation.

Beebe J. The request for proposal and vendor selection process. Top Health Inform Manag 1992; 13(1 ): 11-9.

A case study using a simple rating & weighting technique to preselect vendors and products.

Einbinder LH, Remz JB, Cochran D. Mapping clinical scenarios to functional requirements: a tool for evaluating clinical information systems. Proc AMIA Annu Fall Symp 1996;747-51.

A case study using a simple rating technique, but instead o f a weighted summation it simply uses a summary principle.

Feltham RKA. Procurement of information systems effectively (POISE): using the new UK guidelines to purchase an integrated clinical laboratory system. In: Greenes RA, Peterson HE, Protti DJ, editors. Medinfo'95. Proceedings of the Eighth World Congress on Medical Informatics; 1995 July 23-27; Vancouver, Canada. Edmonton: Healthcare Computing & Communications Canada Inc; 1995. p. 549-53.

B r i e f description o f a standardized procedure regarding I T acquisition based on a weighting score mechanism.

Friedman BA, Mitchell W. An analysis of the relationship between a pathology department and its laboratory information system vendor. Am J Clin Path 1992;97(3):363-68.

Discusses perils in the different types o f relationships between a vendor o f an I T system and the client.

Gell G, Madjaric M, Leodolter W, K61e W, Leitner H. HIS purchase

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

NAND6~OOI< O~ EVALUATION I'qETIdODS

projects in public hospitals of Styria, Austria. Int J Med Inform 2000;58- 9:147-55.

Outlines an approach to select one f r o m a group o f bids by means o f a combination o f established criteria, functional requirements, and on-site trials.

Madjaric M, Leodolter W, Leitner H, Gell G. HIS purchase project: preliminary report. In: Kokol P, Zupan B, Stare J, Premik M, Engelbrecht R, editors. Medical Informatics Europe '99. Amsterdam: lOS Press. Stud Health Technol Inform 1999;68:115-20.

A case study and some o f their experiences.

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

NANDBOOI< O~ EVALUATION METHODS

Balanced Scorecard

Areas of Application

Ongoing optimization of the outcome of a development project by balancing focus areas by means of a set of indicators for a set of strategic objectives.

Description Balanced Scorecard (BSC) is based on a principle of critical success factors in terms of a limited set of performance indices. These are continuously monitored and are balanced against each other in an attempt to create an appropriate relation between their results (Shul- kin and Joshi 2000; Protti 2002). Normally the success factor relates to the strategic objective at management level, hence the method is classified as being a strategic management tool. There is, however, nothing wrong in having the performance-related index focusing on other areas (Shulkin and Joshi 2000) - for instance, to obtain a better service profile toward service users or even taking it down to a personal level to assess performance.

The philosophy behind BSC is that o f a constructive assessment, as the method can be used through group dynamics to create understanding and awareness of how certain aspects of the organization works externally and internally (Protti 1999 and 2002).

Assumptions for Application The method requires some experience before it can be used in a larger context because, for instance:

1. Good results from this method depend on a good and detailed understanding of the causal relationship relating to the success factors in the model, and thereby also to the practical insight into the organization.

8 ~

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

~IANDE~OOI< OF:: EVALUATION I'vlE-TNODS

2. It requires the ability to handle the method in a dynamic project situation and in an organization undergoing change. This is because development projects involve constant changes in the object being monitored. In other words, it means that the object under investigation is a moving target or, as expressed by Gordon and Geiger (1999), the organization is in a double-loop learning.

Perspectives The value norms of the project will inevitably become apparent from the discussion of how to prioritize the strategic success factors. The Scandinavian culture is particularly able to harmonize this with the intentions behind a broad user involvement, because of their national regulations, and can consequently be used to create openness, commitment and motivation through a mutual understanding. For the same reason this method is less valid in some other cultures.

Frame of Reference for Interpretation (Not applicable)

Perils and Pitfalls Within cultures and organizations where there is a tradition of top- heavy management, a schism may occur between the management style and the need to get a detailed understanding of work processes and motivation from staff. Hence, it may in certain situations be difficult to get the necessary support and understanding from staff to carry the method through without it hampering the outcome from a management perspective.

Advice and Comments

It is important to be aware that this is a project management tool to obtain a basis for decision making in a constructive development process, rather than a method that can stand completely on its own as an assessment tool.

References

Gordon D, Geiger G. Strategic management of an electronic patient record project using the balanced scorecard. J Healthc Inf Manag 1999;13:113-23.

Quite a g o o d article to learn f r o m shouM you wish to use this m e t h o d

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

tdANDE~OOt< OF EVALUATION METHODS

Protti D. An assessment of the state of readiness and a suggested approach to evaluating 'information for health': an information strategy for the modem NHS (1998-2005). University of Victoria; 1999.

Protti D. A proposal to use a balanced scorecard to evaluate Information f o r Health: an information strategy for the modem NHS (1998-2005). Comput Biol Med 2002;32:221-36.

Protti argues in favor o f the use o f BSC and shows how to do i t - in both short and long versions.

Shulkin D, Joshi M. Quality management at the University of Pennsylvania health system. In: Kimberly JR, Minvielle E, editors. The quality imperative, measurement and management of quality in healthcare. London: Imperial College Press; 2000. p. 113-38.

The book is an anthology in quality management, and the chapter referenced gives a good description o f BSC (Scorecard Measurement and Improvement System) as a method

Supplementary Reading

Kaplan R, Norton D. The balanced scorecard- measures that drive performance. Harvard Business Review 1992;70(1):71-9.

Kaplan R, Norton D. Strategic learning & the balanced scorecard. Strategy Leadership 1996;24(5): 18.

These authors are the originators o f the method

Niss K. The use of the Balanced ScoreCard (BSC) in the model for investment and evaluation of medical information systems. In: Kokol P, Zupan B, Stare J, Premik M, Engelbrecht R, editors. Medical Informatics Europe '99. Amsterdam: lOS Press. Stud Health Technol Inform 1999;68:110-14.

6 7

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

~ANDE~OOI< O~ c EVALUATION METHODS

BIKVA (from Danish: "Brugerlnddragelse i KVAlitetsudvikling"; translated: User Involvement in Quality Development)

Areas of Application

The method is used as a tool for making critical, subjective decisions about an existing practice.

Description Like KUBI, this method originates from the social sector, where it has been developed as a constructive tool to improve conditions within the socio-medical sector with regard to specific action or problem areas (Krogstrup 2003; Dahler-Larsen and Krogstrup 2003).

The method is based on a reflective process in the sense o f getting food for thought. Users of an initiative, which can, for instance, be service functions or an IT-based solution, are the source of information. Its procedure consists of the following steps:

1. During group interviews, users are asked to indicate and justify "why they are either satisfied or not satisfied" with a given service function or an IT function. The users, who are the primary stakeholders in connection with the service function or the IT function, will typically describe incidents and interactions.

2. Summarizing the information obtained during the group interview.

3. Field workers (in IT projects probably the project management or vendor of the service or IT function) are confronted with the users' statements and are encouraged to identify the reasons behind the incidents and interactions referred to. This is

6 6

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

NANDSOOI< OF EVALUATION METHODS

incorporated into the observations previously summarized. 4. In a second group interview management goes through the

connections between the observations and the explanations in order to get the points of view of the user group.

. . . and possibly lead to an iterative process with the objective of clarifying all the disparities that relate to the issues of quality identified by the users.

5. The overall conclusion will then be presented to the decision makers in preparation for an assessment of the conclusion and possible initiatives.

Assumptions for Application It can be a particularly unpleasant experience for an individual or an organization to uncover all its skeletons. It requires a psychologically and socially skilled moderator to facilitate the various processes to avoid being unnecessarily hard on anybody.

Perspectives The built-in value norm of the BIKVA method is the user attitudes and perceptions. However, this should not be seen as though user attitudes alone determine the criteria. The philosophy behind it is to force the parties concemed to speak to one another in a formalized way to gather material as input for a process of change and leaming.

Quite clearly, this method is primarily of use in cultures with a strong tradition of user participation and openness, as is the case in the Scandinavian countries. The principle of user participation would not work in, say, Asian cultures. However, you don't have to go far south in Europe before user participation is far less common than in Scandinavia (Fr6hlich et al. 1993) and before openness stops at managements' (or others') desire for a good image and self- respect, 'la belle figure'. In other words, cultural circumstances can make it impossible to apply this method, even in Denmark.

The undisputed advantage of this method is that it gives the user on the ground a formal voice to express that something does not work without this being understood as negativism or lack of constructive cooperation.

8 0

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

I--IANDBOOI( OF:: EVALUATION METHODS

Frame of Reference for Interpretation The frame o f reference is the users' view o f where the problems lie. Having said that, it is o f course also those who wear the shoes who can best feel if the shoes pinch.

Perils and Pitfalls The pitfall is the representativeness o f those involved, technically and functionally, as it is not guaranteed that everybody can participate. However, the most pressing problems in an organization will, under all circumstances, become evident from the start. Therefore, a minor bias in the representativeness does not necessarily have consequences in the long term. Consideration should, however, also be given to psychological factors in the representation o f an organization with a broad user participation. Should the organization be large and complex, a Stakeholder Analysis method could be used to assess the issues regarding participation.

Advice and Comments The amount o f data resulting from this method can turn out to be overwhelming.

References

Dahler-Larsen P, Krogstrup HK. Nye veje i evaluering. Arhus: Systime. 2003 (in Danish)

Frrhlich D, Gill C, Krieger H. Workplace involvement in technological innovation in the European Community, vol I: roads to participation. Dublin: European Foundation for the Improvement of Living and Working Conditions; 1993.

Krogstrup HK. Evalueringsmodeller. Arhus: Systime; 2003. (in Danish) Even i f this reference and (Dahler-Larsen and Krogstrup 2003) are both in Danish, the description above should enable experienced evaluators to apply the method or apply it in their own version, with benefit.

9 o

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

HAND~OOI< O,C EVALUATION M~:THODS

Clinical/Diagnostic Performance

Areas of Application �9 Measurement o f the diagnostic performance (as in measures o f

accuracy and precision, etc.) o f IT-based expert systems and decision-support systems.

This type o f assessment falls under the Technical Development P h a s e - before an expert system or decision-support system is implemented, but it can continue during the operational phase (the Adaptation Phase and eventually also the Evolution Phase).

Description The clinical performance o f the systems (for diagnostic, prognostic, screening tasks, etc.) is typically measured with the help o f a number o f traditional measures from medicine, such as accuracy, precision, sensitivity, specificity, and predictive values. Also see Measures and Metrics for some specific concepts and measures.

There are thousands o f references to clinical performance o f an IT- based decision-support system, including case studies. Smith et al. (2003) provide a valuable overview o f the literature, although it is limited to neural networks and to the use o f purely statistical tools. The advantage o f this reference is the authors' summary o f metrics and measures for binary as well as multiple categorical and continuous decision problems - that is, when the diagnostic problem consists o f two or more diagnoses or diagnoses on a continuous scale.

Assumptions for Application This depends on the actual measures and (statistical) tools applied. Refer, therefore, to their original literature.

Perspectives One o f the perspectives o f decision-support systems and specialist systems is that o f being able to measure the diagnostic performance

5 If looked upon as an independent activity, expenditure can be quite large, but if the elements of the study can be included in the usual clinical work, one can equally have the chance of it becoming close to cost-free, apart from the planning stage and data analysis.

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

HANDE~OOI< OF:: EVALUATION r"IETI--IODS

of an IT system just as if the IT system were a human being expressing the same.

The measures can without doubt give a significant statement of the value of an expert system's assertions (such as conclusion, recommendations, or similar). But the IT systems are characterized by not knowing their own limitations in such a way that they can adjust their assertions in accordance with the actual situation. Similarly, these systems are unable to take on a holistic view to the clinical object (the patient) in a way that a human being can. In other words, such systems may possibly be able to function with very high clinical, diagnostic performances within a narrow range of cases, but they cannot sensibly know when to abstain from all other cases.

Frame of Reference for Interpretation The frame of reference is typically either a (predefined) 'golden standard' or a consensus reached by a panel of experts.

Perils and Pitfalls The sources of error and pitfalls include partly general errors related to the use of concrete methods or metrics and partly problems related to the technology used for the development of clinical decision aids.

The first type of problems in assessing diagnostic performance is closely related to problems of getting a study design, which precludes a number of the pitfalls mentioned in Part III, such as the carryover effect (contamination), co-intervention, the Hawthorne effect, and so on. Friedman and Wyatt (1996) give a number of useful, overriding instructions for the organization of a clinical diagnostic performance study in order to avoid a number of the perils and pitfalls mentioned in Part III.

The other type of error is related to the technology used for the development of the decision-support system. All technology has its assumptions and limitations. When we deal with decision-support systems, there may be a requirement for mutual independence of the variables included or that there must be an adequate number o f learning cases to avoid overfitting. Schwartzer et al. (2002) explore a number of these problems by using neural networks, some of which are general and also are valid for other types o f decision-support systems.

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

I-lAND.ROOK OF EVALUATION METHODS

Advice and Comments The fact that an article like that o f Kaplan (2001) only finds one single publication on decision-support systems and system performance illustrates how difficult it is to capture all relevant articles through a literature search and how difficult it is to go beyond o n e ' s own professional background (in this case the healthcare sector). The article together with other references is recommended as an introduction to the literature on clinical diagnostic performance.

References

Friedman CP, Wyatt JC. Evaluation methods in medical informatics. New York: Springer-Verlag; 1996.

Contains a really good introduction to assessments.

Kaplan B. Evaluating informatics applications - clinical decision support systems literature review. Int J Med Inform 2001;64:15-37.

Kaplan reviews the literature (although only f o r the years 1997- 1998) on clinical performance o f decision-support systems f o r the period and includes other review articles on the subject.

Schwartzer G, Vach W, Schumacher M. On the misuses of artificial neural networks for prognostic and diagnostic classifications in oncology. In: Haux R, Kulikowski C, editors. Yearbook of Medical Informatics 2002:501-21.

Smith AE, Nugent CD, McClean SI. Evaluation of inherent performance of intelligent medical decision support systems: utilizing neural networks as an example. Int J Med Inform 2003;27:1-27.

Supplementary Reading

Brender J. Methodology for assessment of medical IT-based systems - in an organisational context. Amsterdam: lOS Press, Stud Health Technol Inform 1997;42.

lncludes a review o f literature relating to assessment o f medical knowledge-based decision-support systems and expert systems.

Brender J, Talmon J, Egmont-Petersen M, McNair P. Measuring quality of medical knowledge. In: Barahona P, Veloso M, Bryant J, editors. MIE 94 Proceedings of the Twelfth International Congress of the European Federation for Medical Informatics; 1994 May; Lisbon, Portugal. 1994. p. 69-74.

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

HANDBOOK OF EVALUATION I"I~THODS

Discusses concepts o f correctness from traditional medicine, when used f o r an n * n + I contingency table. The authors suggest generalized metrics for, among others, dispersion (random distribution) and bias (systematic variance) together with kappa values.

Egmont-Petersen M, Talmon J, Brender J, McNair P. On the quality of neural net classifiers. AIM 1994;6 (5):359-81.

Uses similar metrics as in (Brender et al. 1994), but specifically designed f o r neural networks.

Egmont-Petersen M, Talmon JL, Hasman A. Robustness metrics for measuring the influence of additive noise on the performance of statistical classifiers. Int J Med Inform 1997;46(2):103-12.

It is important to keep in mind that (changes in) variations o f one's data source influence the results o f one's own studies. That is the topic o f this article.

Ellenius J, Groth T. Transferability of neural network-based decision support algorithms for early assessment of chest-pain patients. Int J Med Inform 2000;60:1-20.

A thorough paper providing measures and corrective functions to obtain transferability o f decision-support systems.

Elstein AS, Friedman CP, Wolf FM, Murphy G, Miller J, Fine P, Heckerling P, Maisiak RS, Berner ES. Comparison of measures to assess change in diagnostic performance due to a decision support system. Proc Annu Symp Comput Appl Med Care 2000:532-36.

The authors make an empirical comparative investigation often different measures f o r diagnostic performance.

Jaeschke R, Sackett DL. Research methods for obtaining primary evidence. Int J Technology Assessment 1989;5:503-19.

Some o f the methods (and the rationale f o r their use)for evaluation studies o f therapeutic and diagnostic technologies are discussed with regard to the handling o f the placebo effect, confounders, and biases. Advantages and risk in using RCT, case control, and cohort studies are discussed

Kors JA, Sittig AC, van Bemmel JH. The Delphi Method to validate diagnostic knowledge in computerized ECG interpretation. Methods Inf Med 1990;29: 44-50.

A case study using the Delphi method to validate diagnostic performance.

Malchow-Moller A. An evaluation of computer-aided differential

9 4

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

HAND~OOI< O1:::: EVALUATION METHODS

diagnostic models in jaundice [Doctoral dissertation]. Copenhagen: L~egeforeningens Forlag; 1994.

Describes the development and gives a comparative evaluation o f diagnostic performances with three different types o f decision- support systems.

Miller T, Sisson J, Barlas S, Biolsi K, Ng M, Mei X, Franz T, Capitano A. Effects of a decision support system on diagnostic accuracy of users: a preliminary report. JAMIA 1996;6(3):422-28.

A pilot case study.

Nolan J, McNair P, Brender J. Factors influencing transferability of knowledge-based systems. Int J Biomed Comput 1991 ;27:7-26.

A study offactors that influence the clinical diagnostic performance o f decision-support systems and expert systems.

O'Moore R, Clarke K, Smeets R, Brender J, Nyk~inen P, McNair P, Grimson J, Barber B. Items of relevance for evaluation of knowledge- based systems and influence from domain characteristics. Public report. Dublin: KAVAS (A1021) AIM Project; 1990. Report No.: EM-I.1.

The report, which is a freely available public technical report, contains a detailed description o f which aspects o f knowledge- based systems have been assessed in the literature.

O'Moore R, Clarke K, Brender J, McNair P, Nyk~inen P, Smeets R, Talmon J, Grimson J, Barber B. Methodology for evaluation of knowledge-based systems. Public report. Dublin: KAVAS (A 1021) AIM Project; 1990. Report No.: EM-1.2.

The report, which is a publicly available technical report, contains an in-depth description o f how to assess knowledge-based systems.

Tusch G. Evaluation of partial classification algorithms using ROC curves. In: Greenes RA, Peterson HE, Protti DJ, editors. Medinfo'95. Proceedings of the Eighth World Congress on Medical Informatics; 1995 Jul; Vancouver, Canada. Edmonton: Healthcare Computing & Communications Canada Inc; 1995. p. 904-8.

A methodological study o f the use o f R O C f o r decision-support systems.

Wyatt J, Spiegelhalter D. Field trials of medical decision-aids: potential problems and solutions. In: Clayton P, editor. Proc Annu Symp Comput Appl Med Care 1991:3-7.

Based on the literature, the authors describe a number o f biases related to the assessment o f medical decision-support systems.

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

HANDBOOK OF {~VALUATION METHODS

Cognitive Assessment

Areas of Application Assessment of the cognitive aspects of the interaction between an IT system and its users, such as:

�9 Identification of where and why operational errors occur �9 Identification of focus areas requiring improvement in user

friendliness

(See also under Cognitive Walkthrough and Usability.)

Description Cognitive aspects deal with the extent to which the system functions in accordance with the way the users think and work in respect of user interaction with the IT system in general and with the user dialogue in particular. In plain language, this deals with the last '8 cm', which equals the process from the eye registering the visual input until the input has been processed by the brain and converted into a new instruction. Thus it is closely related to and very much overlaps with the measurement of the ergonomic aspects (see under Usability). The focus below is on the cognitive aspects that are not covered by the concept of usability.

This type of test is task-oriented- that is, the user simulates (or even tries) to complete an actual departmental work process by means of the IT system. All these tests are recorded on video (because there is no time to digest the information during the session). Kushniruk and Patel (2004) provide a review of methods in this respect.

One of the methods used for analyzing videorecordings is a diagram of the procedure (Beuscart-Z6phir, personal communication), where the course of the dialog with the IT system is noted down in symbolic form as shown in the figure below. The example could illustrate a doctor trying to make notes in the electronic patient record while talking to the patient.

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

~ANCSOOI< OF ~VALUATION M{]TI---IODS

example

/ ; -

, X

e x p l a n a t i o n o f s y m b o l s

, transaction ~ o n e s t e p b a c k w a r d

0 closed activity

X dead-end shift to a new subactivity

1 shift in search tactics

Symbols could, for instance, be designed for the following aspects o f the dialogue:

�9 H o w often did the user search in vain for information or for actual data fields where they believed them to be?

�9 H o w often was data input rejected simply because the user had misunderstood h o w the data had to be presented or what the field should be used for?

�9 H o w often did the user have to ask or to rack his or her brain to find out what to do? And how quickly did they find out, once they had tried it once or twice?

In an analogous approach, building on analytical constructs and techniques o f Conversation Analysis, Meira and Peres (2004) suggest an evaluative approach that identifies gaps or breakdowns in user dialogues, and it maps the mismatches between user actions and software behavior.

Assumptions for Application Assumptions for application depend entirely on what the results o f the study are to be used for. It normally requires an experienced cognitive psychologist to assist in the investigation as there is still no standardized and thoroughly tested method within the current literature on IT assessment. The difficulty with the use o f this m e t h o d is how to find a direction o f a synthesis and a conclusion on

C)7

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

NAND~OOI< O~C EVALUATION J'I{~TNODS

the functionality o f the IT system if one does not have a cognitive psychology background. Depending on the objective o f the study, a suitable person with relevant experience in the field should be engaged. It is also a prerequisite that there are detailed activity descriptions or scenarios for the user tasks within the department.

Perspectives The difference between this method and Cognitive Walkthrough (below) is the approach: Cognitive Walkthrough starts with a technical description or a physical realization o f the system and transforms this into scenarios o f its use, whereas Cognitive Assessment starts by using the users' practical everyday tasks to develop scenarios to test the IT-based system.

During interaction with an IT system, an understanding o f what goes on in the system is important for user friendliness and user satisfaction, and for the user to know what he or she has to do next or where he or she can find the relevant information. The cognitive aspect is not only o f great importance when learning the system, but it is indeed o f particular importance for the quantity o f operator errors during its life cycle. This could be a reason why EHR seems to work perfectly well in general practice but less successfully in clinical departments: Once you have learned to use the system and you use it a great deal every day, it becomes very easy, even though the system might be a bit clumsy. However, when there are competing activities for the many and varying staff, who only use the systems for short periods o f time, cognitive aspects demand much more attention.

The shortage o f this type o f assessment means that operational requirements, criteria, or recommendations have yet to be established. There is also a lack o f established practice for incorporating the cognitive aspects into a requirements specification. Hence, cognitive aspects do not normally form an explicit part o f a frame o f reference in a traditional agreement with a vendor.

Frame of Reference for Interpretation This method is normally used for explorative s t u d i e s - that is, for studies intended to give a picture o f a situation. Thus, there is no frame o f reference in the normal sense.

However, there is a need for detailed descriptions o f activities or scenarios o f user activities in a department. What is an a c t i v i t y - i n

9 6

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

~IANDE~OOI< OF EVALUATION METHODS

other words, what is its objective? What is its premise? What decision elements does it entail? And in what context (physically, socially, mentally) does it take place? Who performs the activity? What is the consequence of inadequate or delayed completion of the activity? This information is then used as a basis for comparison for the assessment o f the cognitive aspects of an IT system. An example is the preoperative patient record taking by an anaesthesiologist (see Beuscart-Zrphir et al. 1 9 9 7 ) - for example, where are the records physically kept during the operation (during the study in question, the paper-based record used to be placed on the patient's stomach)? And how does the doctor quickly find the information he or she needs during a crisis?

Perils and Pitfalls Studies within this area require specialist knowledge. However, the experience and level of expertise o f the user is a pitfall that even the experienced performer can fall into. There is a difference in how novices and experts work and think (see, for instance, Dreyfus 1997 and Beuscart-Zrphir et al. 2000), and experts can hardly discern between what they themselves say they do and what they actually do do (Beuscart-Zrphir et al. 1997; Kushniruk et al. 1997). See also the discussion in (Brender 1997a and 1999) and Part III in this handbook.

Advice and C o m m e n t s The questions raised in Description (above) c a n - i n i t i a l l y - assist in finding out whether a system has serious cognitive problems, as can the analysis of operational errors in Functionality Analysis or the Usability methods.

References

Beuscart-Zrphir MC. Personal communication; 1997.

Beuscart-Zrphir MC, Anceaux F, Renard JM. Integrating users' activity analysis in the design and assessment of medical software applications: the example of anesthesia. In: Hasman A, Blobel B, Dudeck J, Gell G, Prokosch H-U, editors. Medical Infobahn Europe. Amsterdam: lOS Press. Stud Health Technol Inform 2000;77:234-8.

Beuscart-Zrphir MC, Brender J, Beuscart R, Mdnager-Depriester I. Cognitive Evaluation: How to assess the usability of information technology in healthcare. Comput Methods Programs Biomed 1997;54(1-2): 19-28.

9 9

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

J---IANDE~OOI< OF: ~VALUATION M~TIdODS

Brender J. Methodology for assessment of medical IT-based systems- in an organisational context. Amsterdam: lOS Press, Stud Health Technol Inform 1997;42.

Brender J. Methodology for constructive assessment of IT-based systems in an organisational context. Int J Med Inform 1999;56:67-86.

This is a shortened version o f the previous reference and more accessible regarding those aspects referred to in the text.

Dreyfus HL. Intuitive, deliberative and calculative models of expert performance. In: Zsambok CE, Klein G, editors. Naturalistic Decision Making. Mahwah: Lawrence Erlbaum Associates, Publishers; 1997. p. 17-28.

Kushniruk AW, Patel VL. Cognitive and usability engineering methods for the evaluation of clinical information systems. J Biomed Inform 2004;37:56-76.

Kushniruk AW, Patel VL, Cimino JJ. Usability testing in medical informatics: cognitive approaches to evaluation of information systems and user interfaces. Proc AMIA Annu Fall Symp 1997:218-22.

The article clearly illustrates the difference between the users' subjective understanding and the objective measure o f the same.

Meira L, Peres F. A dialogue-based approach for evaluating educational software. Interacting Comput 2004; 16(4):615-33.

Supplementary Reading

Carroll JM, editor. Scenario-based design, envisioning work and technology in system development. New York: John Wiley & Sons, Inc.; 1995.

This book contains a number o f articles written by professionals in system development. It can therefore be on the heavy side, but there is a lot o f inspiration to be had regarding scenarios. The different articles deal with different forms o f scenarios, their use, as well as a discussion o f advantages, disadvantages, and assumptions.

Demeester M, Gruselle M, Beuscart R, Dorangeville L, Souf A, Mrnager-Depriester, et al. Common evaluation methodology. Brussels: ISAR-T (HC 1027) Health Care Telematics Project; 1997. Report No.: Deliverable 5.1.

This publicly available technical report can be obtained by

IOO

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

IdANDBOOk~ OF: ~VALUATION METHODS

contacting the author. It describes all the assessment methods o f the project including the ergonomic and cognitive assessments.

Jaspers MWM, Steen T, van den Bos C, Genen M. The think aloud method: a guide to user interface design. Inf J Med Inform 2005;73:781-95.

The Think Aloud method combined with video recording was used as a means to guide the design o f a user interface in order to combat the traditional problems with usability.

Kushniruk A, Patel V, Cimino JJ, Barrows RA. Cognitive evaluation of the user interface and vocabulary of an outpatient information system. Proc AMIA Annu Fall Symp 1996:22-6.

Kushniruk AW, Patel VL. Cognitive computer-based video analysis: its application in assessing the usability of medical systems. In: Greenes RA, Peterson HE, Protti DJ, editors. Medinfo'95. Proceedings of the Eighth World Congress on Medical Informatics; 1995 Jul; Vancouver, Canada. Edmonton: Healthcare Computing & Communications Canada Inc; 1995. p. 1566-9.

Kushniruk AW, Patel VL. Cognitive evaluation of decision making processes and assessment of information technology in medicine. Int J Med Inform 1998;51:83-90.

Keep these two authors in mind with regard to studies o f cognitive aspects. Although some o f their articles are conference proceedings, they contain many good details.

Patel VL, Kushniruk AW. Understanding, navigating and communicating knowledge: issues and challenges. Methods Inf Med 1998;37:460-70.

Describes how they carry out assessments using video recordings and think-aloud techniques.

IOI

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

IdANID~OOt< O~ EVALUATION I"qETNODS

Cognitive Walkthrough Areas of Application Assessment of user 'friendliness' on the basis of a system design, from specification, to muck-ups and early prototypes of the system, aimed at judging how well the system complies with the users' way of thinking, for instance:

�9 Identification of where and why operational errors occur �9 Identification of causes behind problems with respect to user

friendliness and consequently identification of areas for improvement

(Patel et al. 1995) . . . and thereby also

�9 The assessment of a demo IT system as a step in the process of choosing a system in a bid situation

The advantage of this method compared to many others is that it can be carried out at an early stage and with just a system specification as a basis. Therefore, the method can also be used to assess immature prototypes (Beuscart-Z6phir et al. 2002).

See also under Cognitive Assessment.

Description Every single task or activity in an organization requires a combination of cognitive (i.e., intellectual) functions and physical actions. Cognitive Walkthrough is an analytical method designed to evaluate usability. The method addresses the question of whether a user with a certain degree of system and domain knowledge can perform a task that the system is designed to support. See (Horsky et al. 2003), which provides an overview of theories and approaches used for studying usability aspects as well as a case study. It focuses on the cooperation between the cognitive functions and the parallel physical actions, while the ergonomic aspects are closely related to the organization of, and the interaction with, each of the individual fields on the screen images. There is a gentle graduation and quite an overlap between cognitive and ergonomic aspects (see Usability and Cognitive Assessment) of the user interface of an IT system.

IO"2

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

~ANI}8OOI< OF--EVALUATION IvIIZTIdODS

Cognitive Walkthrough is a review method for which a group of experts prepare task scenarios either from specifications or from earlier prototypes (could be video-based) (Kushniruk et al. 1996). The experts investigate the scenarios either on their own or together with a user. The one acting as a typical user goes through the tasks with the aid of the interface of the s c e n a r i o - "walking through the interface", as they pretend that the interface actually exists and works. Each step of the process is analyzed as seen from the point of the objective of the task with the intention of identifying obstacles in the interface that make it either impossible or difficult to complete the task. Complicated or roundabout ways through the system's functions indicate that the interface needs a new function or simplification of the existing one.

Assumptions for Application The method is designed to systematically go through all the user's possibilities of interaction with every screen of the IT system. The analyst is required to perform a manual simulation of the cognitive processes that form part of the successful execution of a task. In practice it could, however, be impossible to carry out a systematic cognitive walkthrough if the system is very complex.

It is necessary to know how the user interface will look; therefore, the description of the system needs to meet a certain level of detail.

It is a prerequisite that there is a task and activity analysis. See, for instance, under Analysis of Work Procedures, and see (Beuscart- Z6phir et al. 2002).

Normally users of a (future) IT system do not have the experience to prospectively and methodically assess the cognitive aspects themselves, as to analyze one's own activities from a different point of view requires special awareness and some lateral thinking.

Perspectives The difference between this method and Cognitive Assessment is the point of view. As a starting point Cognitive Walkthrough uses a technical description, or a physical realization of it, and transforms it into scenarios of its use, while Cognitive Assessment starts by using the practical daily tasks of the user to prepare scenarios to test the IT-based system.

A character-based system can have excellent cognitive qualities, and

IO5

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

NAND~OOI< O~ ~VALUATION M~TIdODS

systems with a graphical interface can have terrible cognitive qualities, so one should get rid of a priori prejudices. What is important is whether the system and the way the user thinks work t o g e t h e r - not whether the interface is esthetically pretty to look at. A person who knows the system can easily circumvent cognitive problems because one can learn how to navigate. Thus, thorough training can compensate for cognitive and ergonomic problems in IT-based systems.

Ordinary users and user organizations will typically experience and notice cognitive errors as operational errors with inadequacies in specific details or in the interaction between the IT system and the work-processes of the organization.

The method's systematic approach is based on the assumption that all screen actions bring discrete changes between states of the system.

Frame of Reference for Interpretation The frame of reference is purely based on the experience of what will normally cause problems in terms of gaps in the understanding and breakdowns in actions.

Perils and Pitfalls Development of user scenarios based on specifications or on an early prototype cannot become more complete and correct than the base upon which it has been developed. In other words, if this material is wanting, then the assessment will also be wanting, but this does not preclude that the method can be quite a valuable (constructive) assessment tool during the early development phase.

Advice and Comments The method is suited for use, for example, during a demo of an existing system, although it is easier if you get to the keyboard yourself. Alternatively, one can stipulate that the vendor or bidder goes through one or more actual scenarios.

References

Beuscart-Zrphir MC, Watbled L, Carpentier AM, Degroisse M, Alao O. A rapid usability assessment methodology to support the choice of clinical information systems: a case study. In: Kohane I, editor. Proc AMIA 2002 Symp on Bio*medical Informatics: One Discipline; 2002

IO/4

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

NANDBOOK O~: .I=VALUATION METHODS

Nov; San Antonio, Texas; 2002. p. 46-50.

Horsky J, Kaufman DR, Oppenheim Mereon Institute, Patel VL. A framework for analyzing the cognitive complexity of computer-assisted clinical ordering. J Biomed Inform 2003;36:4-22.

Kushniruk AW, Kaufman DR, Patel VL, L6vescque Y, Lottin P. Assessment of a computerized patient record system: a cognitive approach to evaluating medical technology. MD Comput 1996; 13(5):406-15.

Patel VL, Kaufman DR, Arocha JA, Kushniruk AW. Bridging theory and practice: cognitive science and medical informatics. In: Greenes RA, Peterson HE, Protti DJ, editors. Medinfo'95. Proceedings of the Eighth World Congress on Medical Informatics; 1995 Jul; Vancouver, Canada. Edmonton: Healthcare Computing & Communications Canada Inc; 1995. p. 1278-82.

Supplementary Reading

Carroll JM, editor. Scenario-based design, envisioning work and technology in system development. New York: John Wiley & Sons, Inc.; 1995.

This book contains a number o f articles written by system development professionals, so it can be somewhat heavy to read, but there is a lot o f inspiration regarding scenarios. The different articles deal with various forms o f scenarios, their use, as well as a discussion o f advantages, disadvantages, and assumptions.

Huart J, Kolski C, Sagar M. Evaluation of multimedia applications using inspection methods: the Cognitive Walkthrough case. Interacting Comput 2004; 16:183-215.

An exhaustive case study that also reviews usability methods.

Kushniruk AW, Patel VL. Cognitive and usability engineering methods for the evaluation of clinical information systems. J Biomed Inform 2004;37:56-76.

A thorough description o f a number o f usability inspection methods, including the walkthrough method.

http://jthom.best.vwh.net/usability/ Contains many method overviews with l&ks and references. (Last visited 31.05.2005)

IO5

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

HANDBOOI< OF:: EVALUATION M~TIdODS

Delphi Areas of Application

�9 (Qualitative) assessment o f an e f f e c t - for instance, where the solution space is otherwise too big to handle

�9 Elucidation o f a problem area - for instance, prior to strategic planning

�9 Exploration o f development trends

Description The Delphi method is a consensus method for the prediction o f the future (Dalkey 1969) developed by the Rand Corporation for the American military in the 1950s.

One characteristic feature o f the Delphi method is that a central team collaborates with and interrogates a panel o f experts in an iterative way to formulate the experts' knowledge o f a predefined topic. See, for instance, the references (Dalkey 1969; Linstone and Turoff 1975; Adler and Ziglio 1996; and Crisp et al. 1997). Being experts within a specific domain means that they have the capability to assess development trends and directions within that d o m a i n - in other words, they are able to extrapolate into the future. Another characteristic feature is that the central group remains neutral during the entire process and lets the experts' opinions and statements guide the outcome o f the process in an iterative way.

The approach depends on the topic, as the basis for interrogation may be predefined (for instance, some quantitative investigations) or may be the result o f an open opening question (for instance, in certain qualitative investigations). An example o f the latter is seen within (Brender et al. 2000):

1. Brainstorming phase based on an open question o f what is relevant to address the problem area with.

2. Evaluation phase, where the individual experts comment on (prioritize, elaborate, refine) each others' contributions from the first phase.

3. Feed-back to the authors on their fellow experts' commented,

io6

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

~-IAND~OOI< OF {~VALUATION M~TIdODS

providing an opportunity for them to refine their original intention. Following this refinement, the topics are fixed.

4. Preparation o f a questionnaire from previous material. 5. Collection and analysis o f the expert panel's rating o f the

individual topic.

Assumptions for Application The method is medium of difficulty, primarily because it assumes experience and cautiousness with the synthesis and with the preparation of a questionnaire, including the perils and pitfalls associated with a Questionnaire approach (see this elsewhere).

It is an assumption for a reliable outcome that the core team remains neutral toward the material and in dealing with the outcome.

Perspectives The professional perspective behind the design o f this method is that a collection of experts is capable of inspiring and correcting each other to predict the future or the nature of something, which none of them are able to judge accurately on their own. Cooperation makes them adjust and compensate for each other's weak points in an iterative process controlled by the core team so that the end results become fairly accurate. Thereby, a topic may be exhaustively investigated, while inaccuracy, nonsense, hidden agendas, and hobby horses vanish in the iterative process. It is difficult to predict anything about the future, but experience shows a fair amount of success by using the Delphi method.

Frame of Reference for Interpretation (Not applicable)

Perils and Pitfalls When the experts know each other, there is a risk that they become emotional or stick to the official view of the topic rather than responding according to their best convictions. Therefore, it is best to keep the contributions anonymous. This sometimes also includes the composition o f experts within the panel, at least as long as the process is ongoing.

Advice and Comments Depending on the need for precision in the outcome, one may continue iterating the last phases until the results no longer

IO7

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

NAND6~OOk~ O~ {]VALUATION ME-TIdODS

converge, indicating that the m a x i m u m level o f accuracy and precision has been reached for that specific combination o f experts.

References

Adler M, Ziglio E. Gazing into the oracle - the Delphi method and its application to social policy and public health. London: Jessical Kingsley Publishers; 1996.

Brender J, McNair P, Nohr C. Research needs and priorities in health informatics. Int J Med Inform 2000;58-9(1):257-89.

Crisp J, Pelletier D, Duffield C, Adams A, Nagy S. The Delphi method? Nurs Res 1997;46(2):116-8.

Dalkey NC. The Delphi method: an experimental study of group opinion - prepared for United States Air Force project. Santa Monica (CA): Rand; 1969.

Linstone HA, Turoff M. The Delphi method, techniques and applications. Reading, MA: Addison-Wesley Publishing Company; 1975.

Supplementary Reading

Kors JA, Sittig AC, van Bemmel JH. The Delphi method to validate diagnostic knowledge in computerized ECG interpretation. Methods Inf Med 1990;29:44-50.

A case study using the Delphi method f o r validating diagnostic performance.

O'Loughlin R, Kelly A. Equity in resource allocation in the Irish health service, a policy Delphi study. Health Policy 2004;67:271-80.

A case study using the Delphi method to explore policy issues in resource allocation.

Snyder-Halpem R. Indicators of organizational readiness for clinical information technology/systems innovation: a Delphi study. Int J Med Informatics. 2001 ;63:179-204. Erratum in Int J Med Inform 2002;65(3):243.

A case study using the Delphi method f o r a specific evaluation purpose.

i o 6

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

I-IAND~OOI< OF EVALUATION METHODS

Equity Implementation Model

Areas of Application Examine users' reaction to the implementation o f a new system, focusing on the impact o f the changes such a system brings about for the users.

Description The Equity Implementation Model is used to retrospectively investigate and understand user reactions to the implementation o f an IT-based system, based on incidents in an organization. The focus o f this method that originates in social science is the effect o f the changes that such a system brings about (Lauer et al. 2000).

The 1.

method consists o f three steps (Lauer et al. 2000): Deals with the changes from the perspective o f a user to identify possible stresses and benefits that may be affected by the new IT system, both o f which may be positive or negative in a picture o f the 'perceived benefit' Examines the fairness in relation to the employer in sharing the gains or losses brought about by the change, thereby comparing the perceived benefits o f the two parties Compares changes perceived by individual users with that o f other users or user groups

Assumptions for Application There is a need for insight into behavioral aspects and social science to avoid the pitfalls in studying humans.

Perspectives Implementation o f an IT-based solution implies organizational change. Many attempts at implementing IT-based systems in organizations have resulted in failures, o f which only a fraction are on account o f technical problems, and many researchers have demonstrated that critical issues in the implementation o f such

I o 9

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

IdANDtgOOI< O1: .EVALUATION I"II~TIdODS

systems reside with the soft human aspects o f an organizational.

The method provides a means for explanation o f system implementation events, leading to an understanding o f user resistance or acceptance o f the new technology. It is based on the perspective that there is no fundamental or irrational resistance to change (Lauer et al. 2000): Each change is evaluated as being favorable or unfavorable by each individual affected by it.

Further, the method emphasizes that it is important to pay attention to the fairness concerns o f all users, a viewpoint that is highly culturally dependent.

Frame of Reference for Interpretation (Not applicable)

Perils and Pitfalls The sources o f bias come from the method's origin in social science, and hence it is concerned with the pitfalls in observing and interrogating humans (see Part III in general).

Advice and Comments Although the objective o f Lauer et al. (2000) is that o f research, it may certainly in some situations be useful for more down-to-earth investigations as well, particularly in cases o f special user- satisfaction problems.

As the method in general deals with equity, it may be applied as a measuring instrument to provide the decision-making basis relation to policy planning and priority setting, as well as for examination o f barriers and drivers at the implementation o f change. The method may also prospectively provide a means for pointing out other problem areas that may subsequently be addressed by preventive initiatives through its focus on equity.

References

Lauer TW, Joshi K, Browdy T. Use of the Equity Implementation Model to review clinical system implementation effort, a case report. JAMIA 2000;7:91-102.

IIO

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

]-IANIDBOOI< OF: EVALUATION METHODS

Field Study (In the sense o f observational studies)

Areas of Application Observation o f an organization to identify its practices and to elucidate mechanisms that control change.

Description In short, observational studies are used to diagnose various conditions in an organization (Harrison 1994). This method is widely used in psychology, sociology, anthropology, and so o n - that is, in professions that deal with different perspectives o f human factors - to identify what goes on and how. The studies encompass all sorts o f elements from social interaction and the influence o f specialist cultures on work practices, through to work process analysis, as seen in the informatics part o f the organizational spectrum observed. The studies can be transverse or in depth, exploring or verifying.

Assumptions for Application This type o f study normally requires professional psychological, sociological, or specialist knowledge o f organizational theory.

Perspectives It is an important aspect o f field studies to understand that the reality o f the actors is a determining frame for acting and maneuvering. Therefore, the outcome o f the study depends on prior knowledge o f the users' conditions and an understanding o f their way o f thinking in different contexts. The specialist culture is an acquired one.

Ill

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

~IANDt~OOK OF: ~VAI._.UATION METHODS

One o f the reasons why this type o f study has to be performed by specialists is that the observer must be able to free himself from his perspectives and thereby from his prejudices, expectations, and assumptions, as well as from established principles and understanding. Any kind o f preconception will influence the process o f observation and introduce bias. Professionals in the healthcare sector including IT users are subconsciously trained to think in a specific way (a profession-oriented culture and politics) to such a degree that it is difficult for them to disregard their own way o f thinking. Informatics specialists are trained to observe the movements o f information units and processes o f change in an organization, but not to observe (profession-oriented) cultural, social, and psychological phenomena, so even informatics specialists fall short.

There are two basic models behind field studies: Reductionist ('diffusion model') and holistic ('translation model').

The reductionistic understanding presupposes that things can be taken apart and individual items be observed in isolation, after which they are put back together as an entirety where the conclusion equals the sum o f the component items. It is also an assumption in this perspective that the results from one organization will work in the same way in other organizations and that the organization(s) can be more or less suited but can adapt to the technology. Specialists of informatics, natural sciences, or technically trained people are trained to work with the reductionistic model, while there is a tendency evolving to try to incorporate holistic aspects when developing new methodologies in order to compensate for known deficiencies.

The holistic view is based on the understanding that an organization or a person comprises many different factors and units that work in complex unity and that any external influence will cause them to mutually affect each other. Specialists with a background in the humanities use a holistic view to a far greater extent than informatics specialists. However, they often have to draw very strict limitations as the totality otherwise becomes too large.

Frame of Reference for Interpretation (Not applicable)

112

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

IdANDt~OOI< OF ~VALUATION METHODS

Perils and Pitfalls Professionals from social and behavioral sciences are well aware o f the many serious pitfalls and take the organizational and behavioral circumstances as their methodological and methodical starting point. This is why field studies require a specialist background.

One must be aware that it takes time for a person from outside the domain to reach a stage o f observation enabling him to pose the right questions to the right people. This is because without prior knowledge o f the domain, one cannot know anything about what should and shouldn't happen. Even though they are trained to observe, it takes time to get to the core o f the matter.

Interview studies are often used to facilitate field studies, but as they are unable to elicit tacit knowledge, they embrace severe pitfalls, inasmuch as the users in an organization are unable to account for what or how they do things 6 (see the review in Brender 1997a and 1999). Instead they have a tendency to describe prescriptive procedures in a way that they would use to tell a colleague being trained in the department (Beuscart-Zrphir et al. 1997; and Beuscart- Zrphir, personal communication).

The Hawthorne effect may be significant in studies o f this kind: For psychological reasons people tend to change behavior and performance when under observation (see Part III). This points at the need for addressing the complex dynamics o f the wholeness and its mechanisms o f change rather than single and specific variables before and after the process o f change.

Advice and Comments This method is very time consuming.

Before you even start to consider undertaking a field study yourself, you need to read Baklien (2000), who examines a number o f aspects o f field studies particularly concerned with constructive assessment purposes.

6 This is a known phenomenon in the development of knowledge-based systems and has resulted in the establishment of a special profession, called knowledge-engineering with methods dedicated to the purpose. They still struggle to develop methods to elicit the users' knowledge of how they do things in real life and why. The focus is particularly aimed at expert knowledge in the area of diagnostics/therapy. But system analysts/informatics people struggle equally to develop efficient and reliable methods to elicit the entirety of relevant aspects in work processes (Brender 1999).

113

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

NANIDROOI< OF ~VALUATION METHODS

Svenningsen (2003) is a field study o f an exploratory character. It does not, therefore, have any element o f constructive assessment. However, the study is highly timely in its subject as it focuses on the clinical processes o f changes relating to the introduction o f E H R into the Danish healthcare system, and it encompasses a study o f the activities and accounts o f the causal explanations in order to identify the patterns o f change.

References

Baklien B. Evalueringsforskning for og om forvaltningen. In: Foss O, Monnesland J, editors. Evaluering av offentlig virksomhet, metoder og vurderinger. Oslo: NIBR; 2000. Report No.: NIBRs PLUSS-SERIE 4- 2000. p. 53-77. (in Norwegian)

Beuscart-Zrphir M-C. Personal communication 1996.

Beuscart-Zrphir MC, Brender J, Beuscart R, Mrnager-Depriester I. Cognitive evaluation: how to assess the usability of information technology in healthcare. Comput Methods Programs Biomed 1997;54(1-2): 19-28.

Brender J. Methodology for assessment of medical IT-based systems - in an organisational context. Amsterdam: lOS Press, Stud Health Technol Inform 1997;42.

Brender J. Methodology for constructive assessment of IT-based systems in an organisational context. Int J Med. Inform 1999;56:67-86.

This is a shortened version o f the previous reference and more accessible with regard to this subject.

Harrison MI. Diagnosing organizations: methods, models, and processes. 2nd ed. Thousand Oaks: Sage Publications. Applied Social Research Methods Series 1994. vol. 8.

The book discusses advantages and disadvantages o f various methods, including field studies, for the analysis o f a number o f aspects within an organization.

Svenningsen S. Electronic patient records and medical practice - reorganization of roles, responsibilities, and risks [PhD thesis]. Copenhagen: Samfundslitteratur; 2003. Report No.: Ph.D.-Series 10.2003.

A thorough Danish case study ofprocesses o f change in a clinic upon implementation o f EHR.

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

HANDE~OOt< O~ EVALUATION METHODS

Supplementary Reading

Jorgensen DL. Participant observation, a methodology for human studies. Newbury Park: Sage Publications. Applied Social Research Methods Series 1989. vol. 15.

Describes what f i e M studies can be used f o r and how one can observe people in an organization. However, the book avoids discussing the pitfalls.

Kuniavsky M. Observing the user experience, a practitioner's guide to user research. San Francisco: Morgan Kaufmann Publishers; 2003.

Chapter 8 o f this book is dedicated to the method o f Contextual lnquiry, a data collection technique that studies a f e w carefully selected individuals in depth in order to arrive at an understanding o f the work practice.

115

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

NANDE~OOI< OF EVALUATION METHODS

Focus Group Interview

Areas of Application In principle this group interview method can be used for the same purpose as other Interview methods, but in practice it is most relevant for the following tasks (see, for instance, Halkier 2002 and Kuniavski 2003):

�9 During the early analysis phases (the Explorative Phase) - for instance, when eliciting the attitudes, perceptions, and problems o f social groups, or when a model solution is being established.

�9 During o p e r a t i o n s - for instance, to identify a pattern in user attitudes to the system, norms, and behavior, or to identify problem areas o f the system functionality or o f operations in general.

Description The method is a variation on other interview methods, with individual interviews being replaced by a group interview process; see also general aspects under Interviews.

Focus Group Interviews can be carried out during workshops or other dedicated meeting activities, where one o f the purposes o f the introductory steps is to get the group dynamics working. Depending on the topic discussed, a skilled moderator can use a variety o f data- capture techniques (such as mind-maps, Post-it notes on a whiteboard, or similar brainstorming techniques) to ensure that the written output is generated as part o f the whole process.

Stewart and Shamdasani (1990) and Halkier (2002) meticulously investigate the various methodical issues for each step starting with preparation, through completion o f the interview, to transcription and analysis o f the data. Halkier (2002) and Kuniavsky (2003) recommend that the whole procedure be videotaped, as it can be virtually impossible to take notes that are sufficiently thorough and accurate.

In contrast to interviews o f individuals, where one is able to take firm control o f the process while taking notes, group dynamics will invariably make the participants all eagerly speak at the same time.

it6

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

~ANDEiOOK OF:: EVALUATION METHODS

This creates a lot o f overlapping communication, and small, parallel groups that communicate informally may form from time to time. Thus, it becomes very difficult for the moderator to take notes and get the whole context down at the same time as moderating the process.

Assumptions for Application It takes an experienced moderator and facilitator (Halkier 2002; Kuniavsky 2003), and practice of transcribing and analyzing this type o f data.

Perspectives When you gather a number of individuals under informal conditions, you get the opportunity to start a group dialog or debate right across the whole group. The advantages o f this method over that of individual interviews are that the participants can mutually inspire as well as adjust and moderate each other during a brainstorming process, stimulating the creativity of the group and, even more importantly, kindling the individual's memory and perception.

During the process a synthesis o f new knowledge will frequently occur, resulting directly from the interaction between the participants, which the interviewer could not possibly bring out due to the lack of prior contextual knowledge of the subject (Halkier 2002). At times it is easier to glean certain types of information from a group, maybe through group pressure, as it is precisely the social interaction that is the source of the data. For instance, members of a group can have insider information that the interviewer would not be able to draw from an individual. Hereby information will be revealed that often would not be communicated during an individual interview. At the same time one can iterate the process until the required or potential level of precision is reached.

However, there are - culturally defined - topics that will be difficult to discuss in a group, just as there is a tendency to put the official version forward rather than the real version of a topic.

Frame of Reference for Interpretation The method is primarily used to explore a subject. Therefore, studies based on this method normally do not have a frame of reference.

ll7

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

NANDBOOI< OF: EVALUATION METHODS

Perils and Pitfalls The psychological factors are prominent, as in Interview methods in general (see Interview section), but additionally there are the social factors to contend with. The interaction between delegates can have a powerful impact on the reliability of the result, as seen, for instance, from the domination of key persons (or the opposite). In other words, the result depends on social control, representativeness of stakeholders, the homogeneous and heterogeneous composition o f the group, internal power structures (Stewart 1990; Harrison 1994), and confidentiality and sensitivity of topics or taboos.

This form o f interview makes it difficult to get at the finer details or variances in individual practice or at atypical understandings (Stewart 1990; Halkier 2002), because the philosophy o f the method is that o f consensus seeking but also because it is difficult to get representativeness in the composition of the groups and the resulting group dynamic.

Advice and C o m m e n t s

References

Halkier B. Fokusgrupper. Frederiksberg: Samfundslitteratur og Roskilde Universitetsforlag; 2002. (in Danish)

This is a very thorough handbook o f methods dedicated to Focus Group Interviews, but it has the disadvantage that the error sources and pitfalls are only implicitly described

Harrison MI. Diagnosing organizations: methods, models, and processes. 2nd ed. Thousand Oaks: Sage Publications. Applied Social Research Methods Series 1994. vol. 8.

The book deals with the analysis o f a number o f aspects o f an organization and describes advantages and disadvantages o f different methods.

Kuniavsky M. Observing the user experience, a practitioner's guide to user research. San Francisco: Morgan Kaufmann Publishers; 2003.

Chapter 9 o f t his book is dedicated to Focus Group Interview and outlines four different types thereof as well as describes what the method is suited f o r or not in Web applications.

Jl6

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

ldANDE~OOI< OF:: EVALUATION M~THODS

Stewart DW, Shamdasani PN. Focus groups: theory and practice. Newbury Park: Sage Publications. Applied Social Research Methods Series 1990. vol. 20.

Describes a number o f aspects o f group dynamics, strengths, weaknesses, and the risk o f bias by application o f the method all o f which are very important when using this method.

Supplementary Reading

Robson C. Real world research, a resource for social scientists and practitioner-researchers. 2nd ed. Oxford: Blackwell Publishers Inc; 2002. p. 269-91.

This book gives a good description o f different types o f interviews, details and ideas regarding their use including the advantages and disadvantages o f each o f them.

http://www.icbl.hw.ac.uk/ltdi/cookbook/focus_groups/index.html The home page gives additional advice and guidelines on how to carry out the various phases o f a Focus Group Interview. (Last visited 31.05.2005)

http://jthom.best.vwh.net/usability, The home page contains a lot o f method overviews, links and references. Two o f the better links from this page that are relevant f o r Focus Groups are those o f George Silvermin and Thomas Greenbaum's (both last visited on 31.05.2005).

119

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

~IANI38OOK O.IZ EVALUATION J'I~TBIODS

Functionality Assessment

Areas of Application �9 Validation o f objectives fulfillment (realization o f o b j e c t i v e s ) -

as opposed to a pure verification o f the requirements specification - that is, assessment o f the degree o f compliance between the desired effect and the solution realized

�9 Impact Assessment (also called effect assessment) �9 Identification o f problems in the relation between work

procedures and the functional solution o f the IT system

The method will also capture severe ergonomic and cognitive problems but is not dedicated to capture details o f this type.

Description The method is as objective as its user. It can include both qualitative and quantitative elements but is best suited to qualitative studies.

An overview o f the methodological procedure is illustrated in the figure to the left (reprinted from Brender 1989 and 1997a). The 'normative system' and the 'real system' are described first (easiest when using the same modeling and descriptive techniques). The latter includes all o f the IT- based solution (i.e., IT system plus the surrounding work procedures) as it works in practice, while the normative system is the planned IT-based solution. If the actual IT system o f the normative system is not described, it can be replaced by the planned structure,

work processes, activities o f the organization, and new guidelines,

1"20

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

I---IANI}8OOI< O~ EVALUATION METHODS

for example. In step 3 all divergences between the two systems are described, indicating which of these are considered key divergences. Step 4 consists of an analysis of the causal connection. Step 5 is normally used (for development projects and constructive assessment activities) to design the new solution.

The principle behind the analysis of the causal connection is illustrated in the example given in the figure to the right: Each symptom is unwound from the end, from description o f a symptom to possible causes right back to the anticipated primary source of the problem based on the philosophy in Leavitt's model for organizational changes (see under Perspectives below).

Assumptions for Application The method is very work intensive when the complete picture of how work processes in an organization function with an IT system as an overall solution.

The degree of difficulty depends largely on the diagramming technique used (see examples under Work Process Analysis). However, it is most common to use the same methods of description as the ones used for analysis of the organization in earlier phases of the project, thereby also making the method accessible for a broader group of users.

The ability to be objective, persistent, and meticulous is a requirement for the causality analysis, as is in-depth knowledge of the organization in order to limit the solution space for propagating from cause to symptom in the way that best reflects the organization.

Perspectives During a period after taking the IT system into o p e r a t i o n - no matter how thorough you are during planning - a number of adaptations of the work processes and probably also of the IT system will take place. At the same time both the organization and the users will change; they will learn something new, giving rise to new possibilities and/or challenges. For this reason the connection between the IT system and the work processes around the system

I?1

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

IdAND~OOI< OF:: EVALUATION METHODS

will always be sensitive to time. Thus, the method has been designed (within certain limits) to make allowances for the development of the system and changes of work procedures as a function o f time when interpreting the result.

The method is based on Leavitt's model for organizational change (Leavitt 1970), so it is based on the philosophy that any change in one element o f an organization will propagate to the other elements of the organization in a chain reaction until a new, steady state is reached. If problems arise within an organization, compensating reactions will thus spread to other parts of the organization, implying that the problems will be visible in terms of these changes.

At the same time, the method is based on a generalization of the philosophy of Mumford's Participatory Approach to System Design from the late 1970s (Mumford et al. 1978 and 1979). The principle is to start by identifying all variances between the IT system's or the organization's desired processes (i.e., the normative system) and the way it actually works. Mumford's system development method then focuses on finding a solution to one or more o f the variances identified.

Frame of Reference for Interpretation The method has a built-in frame of r e f e r e n c e - namely the description of the Normative System (see above). The original m e t h o d - analysis of the realization of objectives of a specific IT s y s t e m - was based on a detailed Requirements Specification and a System Specification together with the user manual and minutes of meetings, and so on, as a flame o f reference (the normative system) (Brender 1989), while the analysis carried out in (Beuscart et al. 1994) is based on the same method but with the needs and expectations as the flame of reference.

The difference between these two applications (see Brender 1997a) illustrates that should the normative system not explicitly be prepared prior to the introduction of the system, there is a possibility o f 'making do' with the expectations of the system as a frame of reference, as, for instance, in descriptions of new work processes developed in connection with the introduction of the system.

Perils and Pitfalls The causality analysis is the weakest point of the method because the interpretation and therefore the final conclusion is completely

122

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

IdANDt~OOI< O~ EVALUATION METHODS

dependent on this analysis. It can be difficult to uncover the skeletons in an organization and to discuss the problems without hurting s o m e b o d y ' s feelings. Alternatively, someone may try to cover up and hide the problems.

On the other hand, the method is less dependent on whether all divergences are included or whether you get the right divergences defined as key divergences because the actual causality will often manifest itself in more than just one symptom.

A d v i c e and C o m m e n t s The method is designed to address measures according to Leavitt's model for organizational change. However, the framework addressing 'structure, process, and effect', or any other similar framework structure, can be used in the same way.

The causal analysis o f this method may be supported by the Root Causes Analysis method (see separate section).

R e f e r e n c e s

Brender J. Quality assurance and validation of large information systems - as viewed from the user perspective [Master thesis, computer science]. Copenhagen: Copenhagen University; 1989. Report No.: 89- 1-22.

A r~sum~ o f the method and a case study can be found in (Brender 1997a). I f this does not suffice, the reference can be obtained from the author.

Brender J. Methodology for assessment of medical IT-based s y s t e m s - in an organisational context. Amsterdam: lOS Press, Stud Health Technol Inform 1997;42.

Beuscart R, Bossard B, Brender J, McNair P, Talmon J, Nyk/inen P, Demeester M. Methodology for assessment of the integration process. Lille: ISAR (A2052) Project; 1994. Report No.: Deliverable 3.

This report is freely available, and contact information can be obtained from the undersigned author.

Leavitt HJ. Applied organizational change in industry: structural, technological and humanistic approaches. In: March JG, editor. Handbook of organisations. Rand MacNally & Co: Chicago; 1970.

Mumford E, Land F, Hawgood J. A participative approach to the

1"25

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

I--IANIDBOOI< OF-EVALUATION METHODS

design of computer systems. Impact Sci Soc 1978;28 (3):235-53.

Mumford E, Henshall D. A participative approach to computer systems design, a case study of the introduction of a new computer system. London: Associated Business Press; 1979. Appendix C.

174

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

IdANI}gOOK OF EVALUATION MI~THODS

Future Workshop

Areas of Application Evaluation and analysis of an (existing) situation with the view to identifying focus areas for change - that is, aiming at designing future practice.

Description The method resembles the Logical Framework Approach (see separate section) and uses the same terminology. However, the Future Workshop as a method has a different focus and concentrates on situation analysis supplemented by a follow-up and a far more thorough interpretation of how to carry out the individual phases. Another difference from the Logical Framework Approach is that in the Future Workshop it is not necessary to focus on just one problem during the implementation and process of change, but it uses a picture of the organization and its problems as a whole.

The method is carried out through a workshop, a preparatory phase, and a follow-up phase (Jungk and Miillert 1984; Mtiller 2002). The preparatory phase of the workshop consists of a simple stakeholder analysis and the participants getting to know each other. The purpose of the follow-up phase is an ongoing analysis of whether the right track is being pursued and whether it is being pursued in a satisfactory way. The workshop itself has the following three phases:

1. The critique phase, with the purpose of identifying and discussing existing problems

2. The fantasy phase, with subgroups discussing problems identified and submitting their vision of the future (thus, the result is a catalogue o f visions)

3. The realization phase, where means, resources, and opportunities to realize the visions are discussed

Assumptions for Application Depending on the degree of accuracy and precision needed, the person(s) in charge of the process must have sufficient experience of group dynamics - in other words, the method requires an experienced moderator and facilitator familiar with transcription and analysis of this type of data. An experienced moderator may use data-capture techniques (such as mind-maps, Post-it notes on a whiteboard, or

125

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

HANDBOOI< OF EVALUATION METHODS

similar brainstorming techniques) to ensure that the written output is generated as part of the whole process. However, in some situations it might inhibit the process resulting in loss of information.

Perspectives The Future Workshop as described is relatively neutral to political, organizational, and cultural conditions and bonds. It neither dictates nor prevents such bonds but can adapt and operate under the conditions of the organization.

There are, however, aspects of the method that makes it more suitable in certain cultures and forms of organization than in others. For instance, the proposal of a broad representation of staff as participants in the project is dependent on culture (at national and organization levels). However, there is nothing to stop you from making minor adaptations at some points, as long as you are aware of the magnitude of potential consequences and their further implications.

Frame of Reference for Interpretation There is no frame of reference built into the method itself, but references as to whether the result is valid and applicable are the premises against which the result should be used. In other words, the frame of reference has to be the formulation of the organization's (stakeholder groups') vision, mission, and strategic initiatives, as well as other constraints within the organization and possible declared intentions for different aspects of the organization.

Perils and Pitfalls It is a substantial pitfall in some cultures that the method description promotes broad staff participation to identify visions and objectives of the organization because there is no prior knowledge of whether management (or staff) is ready to take that step. It is the step from realizing the problem to establishing solutions and monitoring objectives that may become the point of issue. Therefore, it is important that the leader (the facilitator) of the workshop process has complete insight into, and works in accordance with, management intentions and the principles of the organization or, alternatively, that management accepts that the process of change in the organization also includes this stipulation.

Advice and Comments The follow-up phase does not have an explicit risk monitoring in the same neat way as in Logical Framework Approach, but there is

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

ldANDBOOI< 04= EVALUATION METHODS

nothing to stop you from incorporating it.

Analogous methods that might provide further inspiration are 'Future Search' and 'Cafe-Seminar' (see http://www.futuresearch.net and www.worldcafe.dk, respectively; both last visited 31.05.2005).

References

Jungk R, Miillert N. H~ndbog i fremtidsva.'rksteder. Viborg: Politisk Revy; 1984. (in Danish)

One o f the early references f o r this method. It encompasses a thorough investigation o f the method and a case study.

Mtiller J. Lecture note on future workshop, design and implementation. Aalborg: Dept. of Development and Planning, Aalborg University; 2002 (a copy can be obtained from the department).

177

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

~IANDE~OOk( O~ ~VALUATION METHODS

Grounded Theory

Areas of Application Grounded Theory is a supportive analytical method for data acquisition methods that generate textual data, some open Questionnaire methods, and Interviews (individual and group interviews), for example.

Description Grounded Theory aims at identifying and explaining phenomena in a social context through an iterative and comparative process (Cronholm 2002).

There are four steps: 1. Open coding, where single elements from the data are correlated

to and named as concepts with defined characteristics, including identification of tentative categories of theoretical importance

2. Axial coding, where the named concepts are definitively defined, patterns among them identified, and are then organized into categories according to characteristics and internal relationships, including potential causal relationships

3. Selective coding, which works toward arranging the concepts according to the relationships between them, including identification of mechanisms, processes, or circumstances that may bring about the concept characteristics identified. Finally, the central category to which everything else relates is identified

4. Theoretical coding, which provides a hypothetical model (generalized relationships) based on the previous coding. The model must be able to abstract and explain the phenomena observed and their interrelationships.

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

I-IANDEiOOK OF: EVALUATION MI~Tt--IODS

Assumptions for Application The method is considered difficult and requires special qualifications. Normally it should only be undertaken by experienced professionals.

Perspectives The method is subjective, and it is the implementer's perceptions and evaluations that influence the outcome of the analysis. The subjectivity is somewhat compensated for by the fact that it is iterative, inasmuch as the outcome - provided you have the right end of the s t i c k - starts out resembling chaos but converges toward a (relative) simplicity.

Frame of Reference for Interpretation The method does not use any frame of reference.

Perils and Pitfalls Lack of experience in the use of the method or very large amounts of data can easily result in an imprecise or inaccurate outcome.

It is necessary to be aware of the fact that the data acquisition method itself holds a pattern, precisely because an interview, a questionnaire, or literature searches, for example, are organized according to one's prior knowledge and expectations. One must therefore be careful not to restore this pattern, as the outcome as this simply creates circularity.

A b i a s - one that clearly must be balanced against the advantage of having prior knowledge of the application domain - lies in the risk of categorizing according to known pattems and prior experiences or attitudes, or experience reported in the literature (Goldkuhl and Cronholm 2003). This is because experiences and attitudes (often subconsciously) influence one's evaluation. Thus, preconception (but not necessarily prior knowledge) is a dangerous cocktail when executing this method, also called 'lack of neutrality' (Edwards et al. 2002). The danger is explained as the result of the analysis being driven by the user.

Another bias, as pointed out by Edwards et al. (2002), is 'personal bias' (closely related to 'hypothesis fixation' described in Part III). This appears as a risk when the researcher works on verification of a work hypothesis during observation of an actual case.

129

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

HANDE~OOI< O~ EVALUATION M~THODS

Advice and Comments The method is difficult to apply with a reliable and reproducible result and also rather work intensive. But if you do need textual analysis, there are not many alternatives.

Edwards et al. (2002) illustrate and discuss how triangulation can be used to avoid strong elements o f subjectivity during categorization. They make extensive use o f feed-back from the actors o f the case under study to ascertain whether it is meaningful to them. This way they get a good relationship between conceptions, attributes and categories.

Similarly, Allan (2003) and Balint (2003) give a number of ideas on how to safely manage the methodological challenges, including the coding.

References

Allan G. The use of Grounded Theory as a research method: warts & all. In: Remenyi D, Brown A, editors. Proceedings of the European Conference on Research Methodology for Business and Management Studies; 2003 Mar; Reading, UK. Reading: MCIL, 2003. p. 9-19.

Goes through experience of, and offers advice on, coding with Grounded Theory.

Balint S. Grounded research methodology- a users' view. In: Remenyi D, Brown A, editors. Proceedings of the European Conference on Research Methodology for Business and Management Studies; 2003 Mar; Reading, UK. Reading: MCIL, 2003. p. 35-42.

Discusses how as a user one can ensure validity, quality, and stringency during its u s e - in other words, methodological challenges.

Cronholm S. Grounded Theory in use - a review of experiences. In: Remenyi D, editor. Proceedings of the European Conference on Research Methodology for Business and Management Studies; 2002 Apr; Reading, UK. Reading: MCIL; 2002. p. 93-100.

Elaborates on the method and discusses experiences (strengths and weaknesses) o f the various steps o f its application, based on an analysis o f the application o f the method by a number o f PhD students.

Edwards M, McConnell R, Thorn K. Constructing reality through

130

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

HAND{3OOI< O7 EVALUATION METHODS

Grounded Theory: a sport management study. In: Remenyi D, editor. Proceedings of the European Conference on Research Methodology for Business and Management Studies; 2002 Apr; Reading, UK. Reading: MCIL; 2002. p. 119-27.

Goes through a case and discusses the weaknesses o f the method.

Goldkuhl G, Cronholm S. Multi-Grounded Theory- adding theoretical grounding to Grounded Theory. In: Remenyi D, Brown A, editors. Proceedings of the European Conference on Research Methodology for Business and Management Studies; 2003 Mar; Reading, UK. Reading: MCIL, 2003. p. 177-86.

Discusses certain strengths and weaknesses o f the method and suggests an alternative procedure f o r applying the method at its weakest point.

Supplementary Reading Proceedings from the European Conference on Research Methodology f o r Business and Management Studies (an annual evenO can be generally recommended There are often some good references with practical experiences o f the method

Ericsson KA, Simon HA. Protocol analysis, verbal reports as data. Cambridge (MA): The MIT Press; 1984.

This book provides a thorough review o f the literature on approaches and problems in general terms at the elicitation o f information about cognitive processes from verbal data.

Esteves J, Ramos I, Carvalho J. Use of Grounded Theory in information systems area: an explorative study. In: Remenyi D, editor. Proceedings of the European Conference on Research Methodology for Business and Management Studies; 2002 Apr; Reading, UK. Reading: MCIL; 2002. p. 129-36.

Discusses a number o f recommendations and critical success factors f o r use with this method

Jorgensen DL. Participant observation, a methodology for human studies. Newbury Park: Sage Publications. Applied Social Research Methods Series 1989. vol. 15.

Describes a number o f issues with regard to observation studies and what they can be used for, including how to analyze data from the observation studies o f the Grounded Theory method. Unfortunately, however, the book avoids discussing the pitfalls.

131

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

~ANE)BOOk( OF: ~VALUATION METHODS

Heuristic Assessment

Areas of Application This inspection-based method may be used when no other realizable possibilities e x i s t - for instance, if:

�9 The organization does not have the necessary time or expertise �9 There are no formalized methods to be applied by users �9 There is not yet anything tangible to assess.

Description Basically, this method consists o f summoning experts (3-5 are recommended) from the area in question (usually the assessment o f usability) and letting them make a statement - one based on experience but in a formalized way. Kushniruk and Patel (2004) and Sutcliff and Gault (2004) propose approaches and a number o f heuristics.

In principle this method can be used for nearly everything, but in practice it is most commonly used for the assessment o f user interfaces where few methods exist that will assist the ordinary user in carrying out his own assessment on a formal basis. The website http://jthom.best.vwh.net/usability indicates that this method can be particularly useful for the assessment o f user interfaces because precisely this particular type o f assessment can be difficult for the user organization itself to carry out. Therefore, it can be a really good idea to call on one or more external experts and ask their (unreserved) opinions.

132

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

NANDE~OOl< O.lZ ~VALUATION METHODS

Assumptions for Application It is normally quite simple to summon experts to solve a problem. The difficulty lies in finding experts who are both professionals in the problem area and who also have an understanding of the area of application.

If you yourself have to synthesize the experts' comments into an overall conclusion a f t e r w a r d - for decision-making purposes, for instance, knowledge of the terminology as well as the ability and experience of assessing the consequences of the observations must be present.

Perspectives

Frame of Reference for Interpretation (Not applicable)

Perils and Pitfalls

Advice and Comments Although the literature states that users normally do not take part in this method, Beuscart-Z6phir et al. (2002) nevertheless show that users themselves can act as experts for certain types of IT system assessment, provided that there is suitable guidance and supervision.

References

http://jthom.best.vwh.net/usability/ This website contains a number o f descriptions and links with relevant references (last visited 31.05.2005).

Kushniruk AW, Patel VL. Cognitive and usability engineering methods for the evaluation of clinical information systems. J Biomed Inform 2004;37:56-76.

155

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

I---IANDE~OOI< (DE .I~VALUATION METHODS

A thorough description of a number o f usability inspection methods, including the heuristic evaluation method A number of heuristics is presented

Sutcliff A, Gault B. Heuristics evaluation of virtual reality applications. Interacting Comput 2004;16(4):831-49.

Supplementary Reading

Graham MJ, Kubose TK, Jordan D, Zhang J, Johnson TR, Patel VL. Heuristic evaluation of infusion pumps: implications for patient safety in intensive care units. Int J Med Inform 2004;73:771-9.

A case study applying heuristic assessment for the evaluation of usability aspects to uncover design and interface deficiencies. The study is performed by a combination of specialists and a healthcare professional without prior experience of the method.

15/4

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .

I-4ANE~OOK O~ EVALUATION I'vl~THODS

Impact Assessment (Also called Effect Assessment)

Areas of Application M e a s u r e m e n t o f the e f f e c t - that is, the consequence or impact in its broadest sense from degree o f realization o f the objective plus assessment o f side effect - o f an IT-based solution, with or without the original objective as a flame o f reference. Hence, it includes not only the beneficial effects, but also potential adverse effects.

Description Within the social sector, the practice has been to assess the effect o f a given initiative, legislation, or project with the (political) objective as a frame o f reference to obtain a measurement for assessing to what extent the objectives have been realized (also denoted 'objectives fulfillment'). However, it is worth noting, that in this sector goal-free evaluations are also gaining ground (Krogstrup 2003) - that is, the type o f explorative studies aimed at reporting descriptively on a situation as seen from one or more perspectives. A goal-free evaluation has a nearly limitless solution space w h e n the concept is interpreted in its literal meaning because it entails m e a s u r e m e n t o f the impact with all its secondary effects.

Assessment o f the effect o f health informatics systems can be carried out in the same w a y as projects and initiatives within the social s e c t o r - that is, either as assessment o f objectives fulfillment or as exploratory studies. The indicators would o f course have to be identified and designed for the actual purpose.

As the solution space is very large in a goal-free evaluation, you need to start by finding out what you really need to know about the effect and what the result is going to be used for in order to limit the study: For instance, is it an assessment o f the gains as in the effect on the level o f service or on the quality? Consequences for ways o f cooperation or social interaction? Consequences on the organizational structure or culture? Legal consequences or cases derived from the system? User satisfaction? And so on and so on. The question now is whether the need is for qualitative and/or quantitative data or information. One does not necessarily exclude the other.

Depending on what y o u wish to know about the effect o f an IT-

�9169 w167

155

C o p yr

ig h t ©

2 0 0 6 . E

ls e vi

e r

S ci

e n ce

& T

e ch

n o lo

g y.

A ll

ri g h ts

r e

se rv

e d .