A Critique of Evaluation Methodology Plans

Evaluation methodology for a medical e-education patient-oriented information system

OLIVIER CURÉ*

Université de Marne la Vallée, Laboratoire ISIS, 5 boulevard

Descartes, Champs-sur-Marne, 77454 Marne la Vallée Cedex 2, France

(Received February 2002)

Abstract. This paper describes an evaluation methodology designed for the IMSA (Interactive Multimedia System for Auto-medication) system. IMSA stands in the multidisciplinary field of medical informatics and aims at providing a health care Internet tool for the general public. As far as a medical information system is patient- oriented, issues in understanding content and of human – computer interface usability are key factors to achieve success. The team developing IMSA concentrated part of its work on designing an evaluation methodology to ensure that the system is user-friendly and responds in a way that can be easily understood by the general public. This methodology is built on a three-tier architecture with the user, the evaluator and a set of medical case descriptions. The testing protocol is based on a user trying to run scenarios related to mild clinical signs on the IMSA prototype under the passive presence of an evaluator.

Keywords: Information system; E-learning; Auto-medication; Usability; General public; Evaluation protocols

1. Introduction

IMSA is a medical e-learning tool that provides reliable, updated and accurate

information related to the field of auto-medication (the act of treating ones-self

with or without drugs). The development of IMSA is the result of a long-term

multidisciplinary collaboration between health care professionals and computer

scientists specializing in presentation of knowledge [1]. Both teams were concerned

with the system’s usability.

Usability of a computer application can be defined as the ability of the system to

allow the user to carry out tasks effectively, efficiently and enjoyably [2]. Recent

examples of e-commerce have shown that denying design principles of usability

can lead to bankruptcy. Web sites that deliver biomedical information are also fa-

cing this fact [3].

This paper describes the protocols designed for the IMSA evaluation metho-

dology and proposes an analysis of the results, which will lead to a conclusion on

evaluation methodology for medical e-education systems.

2. Evaluation methodology cognition

The IMSA evaluation methodology is cognitively-based and requires expertise

in medicine and the human – computer interface. Both teams collaborated on the

*Author for correspondence; e-mail: [email protected]

MED. INFORM. (MARCH 2003) VOL. 28, NO. 1, 1–5

Medical Informatics & The Internet in Medicine ISSN 1463-9238 print/ISSN 1464-5238 online # 2003 Taylor and Francis Ltd http://www.tandf.co.uk/journals

DOI: 10.1080/1463923031000107785

theoretical and pragmatic aspects of this evaluation. This work was divided into

four parts: profiling testers, writing scenarios, designing an evaluation form and

defining a questionnaire for the tester.

2.1. Profiling testers

The purpose of this task is the identification and selection of potential users of

medical e-education system. We defined four panels of users (table 1) based on a com-

bination of the requirements from computer scientists and health care professionals.

Studies in usability [4 – 6] have emphasised that a group of five testers is opti-

mum to obtain valuable results. We defined the panels concerning internet experi-

ence and openness to auto-medication with a small questionnaire (table 2).

Our panels were put together considering that:

. an experienced Internet user is someone with at least 6 months experience of the World Wide Web and an average of at least 3 h of browsing a week.

. a user open to auto-medication is someone who happens to prescribe himself medication, does not consult a general physician when he or she does not feel

it is necessary and is involved in seeking information on drugs and medicine.

2.2. Scenarios

The scenarios are medical case descriptions. Their purpose is to give a well-de-

fined framework for the tests. The test protocol will be the same for all groups: tes-

ters will be handed a set of scenarios and will be asked to play the patient role

considering this medical case description. The collaborative work once again pro-

vided valuable requirements: computer scientists wanted to design navigation dri-

ven medical case descriptions while health care experts were motivated to write

diagnostic issue driven scenarios. Both teams collaborated to design scenarios that

would fit all requirements considering the major constraint of time. Our previous

experience of usability testing emphasised that the whole test process should not

exceed 25 – 30 mins. Exceeding this period, computer technology lay people tend

to lose focus, providing insignificant and unreliable results.

Table 1. User profiles for the four panels of the IMSA evaluation methodology.

Skill in web navigation

Novice Expert

Openness to auto-medication Open Panel no. 1 Panel no. 2 Non-open Panel no. 3 Panel no. 4

Table 2. Profiling questionnaire.

Question number Question

1 For how long have you been using the World Wide Web? 2 On a weekly basis, how many hours do you browse the Web? 3 Do you self-prescribe some drugs yourself? 4 Do you read books and or magazines related to medicine or auto-medication? 5 Do you watch television programme or listen to radio shows related to health care? 6 Do you consult a general physician any time you feel sick?

2 O. Curé

Considering this time constraint, we defined six scenarios leading to specific

health issues: (1) proposing a single drug, (2) proposing a list of possible drugs,

(3) the patient does not need any drugs and is only given advice, (4) patient is

not given any drugs because he or she presents contraindications and (5 – 6) asking

the patient to consult his physician because his symptom are too serious (two sce-

narios for this issue).

All six scenarios could have been related to different clinical signs but we

decided to have just one symptom. We chose ‘cough’ because many forms exist

(productive cough, non productive cough) leading to different treatments. The

heterogeneous rating (from 2 to 18 out of 20) of drugs related to ‘cough’ helped

us to take this decision.

The cognitive approach of the scenario writing was predominant and we were

also concerned with their semantics. All scenarios should be easily understood by

all patients, not requiring the help of an evaluator.

2.3. Evaluator form

We designed an evaluator form to structure the notes taken during the tests.

The task of evaluators, who stand behind testers, is to observe the global navigation

and overall interaction on each scenario. The evaluator form consists of an infor-

mation page and scenario forms.

2.3.1. Information page This page contains information about the evaluator: first

and last name, about the testers: first and last name and the test: date, starting and

ending hour of the test.

The information page also contains two blank parts:

. Observations at the end of the scenario playing stage.

. Observations completed after a discussion with all the testers.

2.3.2. Scenario form The goal of this form is to quickly visualise if the tester went

through the desired navigation process of the scenario. Each medical case descrip-

tion has its own page which is made up of the same framework. This framework

consists of:

. starting and ending hour of the scenario interaction.

. a table with rows, sorted sequentially, containing all the expected interac- tions. Each row has a binary reply, whether the tester did interact properly or

not.

. a question : Did the tester reach the expected result?

. a blank space where the evaluator can write comments.

2.3.3. Questionnaire The questionnaire provides valuable data that could not be

obtained from the passive presence of the evaluator. This questionnaire (table 3)

concentrates on the navigation semantic and understanding of the interaction.

3. Running tests

We designed the testing protocol considering tester psychology and cost-effec-

tiveness. This usability testing approach is portable, because it can be run in any

3Evaluation methodology for a medical information system

computer classroom with at least five computers. The main characteristics of the

protocol are:

. one test administrator, usually one of the evaluators, would quickly introduce the IMSA project, present the topic and protocol of this test. Then he or she

would hand out the scenarios. He (she) would also ask the testers to run the

scenarios in the predefined order and try not to interact with the evaluators

during the testing process.

. one tester per computer with one evaluator standing behind him or her, completing the evaluator form.

. no video recording of the tests.

. no special electronic devices.

. one panel group per session.

Most of the time, the evaluator team would be composed of two computer

scientists and three experts in medicine. The evaluator team would run the ques-

tionnaire immediately after the test stage.

4. Results

The results from the evaluator forms and questionnaires are valuable sources of

formal and informal data.

The formal data from the evaluator forms and are summarised in table 4. The

results are emphasized that all scenarios were run successfully. The fact that ex-

perienced and novice Internet users were all successful on the scenario navigation

proves the adaptability of the web architecture.

The informal data emerged from the comments on the evaluator forms and

questionnaires. This information is valuable to understand some of the results of

table 4.

5. Conclusion

Designing and conducting this evaluation methodology was valuable for the

IMSA project. The analysis of the data helped to redesign some aspects of the sys-

Table 3. Test questionnaire asked to testers after running the scenarios.

Question number Question

1 Do you find the symptom to drug navigation clear enough? 2 Do you understand the meaning of the ‘Imperative’ questions? 3 Do you understand the meaning of the ‘Diagnostic’ questions? 4 Do you understand the meaning of the ‘Treatment’ questions? 5 How would you describe the number of questions asked: too important, sufficient,

not sufficient, no opinion? 6 How do you judge ‘forced exit’ of the symptom? 7 How do you consider the way you dialogue with the system: interactive enough, too

rigid, too flexible? 8 Do you have enough criteria to choose a medication from a list of drugs? 9 Do you think that IMSA proposes interesting services? 10 Considering that such a system is available on the World Wide Web, would you use

it ? Would you trust it?

4 O. Curé

tem (‘imperative’ wording, explanations on drug rating and drug categorization,

explanations on some system’s decisions, etc.). Apart from implementation consid-

erations, this evaluation is also a great opportunity to communicate with potential

users on the domain of study.

Our evaluation methodology demonstrates that you can run a usability study

without great financial investment. The most important aspect of any usability

study is to test with potential users. We think that using video recordings and high

technology devices would not contribute to more reliable results.

This evaluation methodology framework is portable and could be used in pro-

jects not necessarily related to medicine, but where cognition is a key factor. The

three-tier architecture can be used in any discipline and requires expertise in us-

ability and the specific domain of study. A cognitive approach is necessary during

the designs of the panels of users, questionnaires, forms and scenarios.

On the medical e-education issue, it would be of interest to study patients ex-

posed to such computerised tools and specially study, over a long period of time,

their relationship with doctors.

References 1. CURÉ, O., LEVACHER, M. and GIROUD, J. P., 2001, Interdisciplinary collaboration for a patient

oriented medical information system. Proceedings of Multimedia and Technology Application Con- ference, IEEE MTAC ‘2001, University of California, Irvine, California, USA. November 2001, pp. 70 – 76.

2. PREECE, J., ROGER, Y., SHARP, H., BENYON, D., HOLLAND, S. and CAREY, T., 1994, Human – computer interaction (Boston: Addison-Wesley publishing company).

3. SACILE, R., WILEY, T. and LOMBARDO, C., 1999, Quality assurance guidelines for a biomedical in- formation web system: the working experience of the BreakIT project. Medical Informatics and the Internet in Medicine, 2, 109 – 120, 24.

4. NIELSEN, J. and MACK, R. L., 1994, Usability inspection methods (New York: John Wiley & Sons). 5. RUBIN, J., 1994, Handbook of usability testing (New York: John Wiley & Sons). 6. NIELSEN, J., 1993, Usability Engineering (New York: Academic press).

Table 4. Results from the analysis of test questionnaire.

Question number

Results from all 20 users Analysis of results

1 100% affirmative 2 40% affirmative ‘imperative’ wording is not explicit enough 3 90% affirmative 4 100% affirmative 5 90% sufficient, 10% not sufficient, 0%

important, 0% no opinion 6 65% of users wanted more information.

35% of users were satisfied on that point All users from panels 1 and 2 wanted more explanations when asked to consult a general physician.

7 85% interactive enough, 15% too rigid, 0% too flexible.

The ‘too rigid’ replies are users from panels 2 (1 user) and 4 (2 users).

8 70% affirmative, 30% not affirmative 9 100% affirmative 10 On using such systems: 65% affirmative,

35% not affirmative On using such systems, replies not affirmative are from panels 3 and 4.

On trusting the system: 80% affirmative, 20% not affirmative

On trusting such systems, replies not affirmative are from panel 4.

5Evaluation methodology for a medical information system