A Critique of Evaluation Methodology Plans

tlouqped

article_3.pdf

Home >Nursing homework help >A Critique of Evaluation Methodology Plans

Usability of Electronic Nursing Record Systems: Definition and Results from an

Evaluation Study in Finland

Johanna VIITANENa,1, Anne KUUSISTOb, Pirkko NYKÄNENc a Aalto University, School of Science and Technology, Finland

b Satakunta Hospital District, Pori, Finland c University of Tampere, Finland

Abstract. Information technologies (IT) are widely used in healthcare, however, little is known about the usability of nursing information systems. This article reports an evaluation study that aimed at researching the usability of four electronic nursing record (ENR) systems and thereby providing guidelines for further IT development. For the purposes of the study the concept of usability was defined to cover the following aspects: nurse-computer interaction in working context, information exchange, and collaboration between healthcare professionals. The study utilized two usability research methods, contextual inquiry and expert review, and was conducted with 18 nurses in Finland. Study results showed that the ENR systems share several usability problems in common, most of them relating to the efficiency of use, intuitiveness, and poor fit for multi- professional needs. Nurses had mainly negative experiences on documenting practices with ENRs: documentation requires a lot of resources, patient information is hard to find, and procedures do not meet the contextual needs. These findings suggest usability problems having significant effects on nurses’ documentation practices and nursing work.

Keywords. nursing informatics, usability, electronic nursing record system, human-computer interaction

Introduction

The adoption of information systems has in several ways influenced nursing practices and documentation. Recently, there has been growing interest in exploring nurses’ experiences on electronic documentation. These studies have resulted in both encouraging and contradictory findings. According to Törnvall et al. [1], the overall tendency concerning the nurses’ opinions on documentation in electronic format seems to be positive. Similarly Kuusisto et al. [2] found that nurses’ experiences with electronic nursing discharge summary were mainly positive, though new structures require careful consideration and documentation takes time. Likewise, several studies indicate that in documentation no time-efficiency has been achieved by implementing an electronic nursing or medical record system [3,4,5]. In their studies about the use of

1 Corresponding Author: Johanna Viitanen, Strategic Usability Research Group, Software Business and

Engineering Institute, P.O. Box 19210, FIN-00076 AALTO, FINLAND; E-mail: [email protected]

333

electronic nursing record (ENR) systems in everyday practice, both Moody et al. [6] and Stevenson et al. [7] find nurses experiencing widespread dissatisfaction because of deficiency of the systems. On the other hand, the use of bedside terminals and mobile solutions in nursing work seem to hold promising opportunities [5,8]. Based on the described results it is unclear how suitable the currently used ENR systems are for nursing work and what kind of use related experiences nurses have. Although some research has been conducted to evaluate the usability of a healthcare information system in clinical settings (e.g. [9, 10]), rather less attention has been paid to examining the usability of nursing information systems.

This article aims to: a) introduce a concept “usability of ENR systems” with related attributes, b) describe a usability evaluation study of four currently used ENR systems in Finland, and c) present summative results from the study. The study to be presented was a part of an empirical research project which incorporated three intersecting themes: 1) the feasibility of the Nationally Standardized Electronic Nursing Documentation model in nursing practices (Finnish nursing documentation is based on the nursing decision making process, nursing core data (NMDS) and Finnish Care Classification (FinCC [11, 12])), 2) the usability of ENR systems, and 3) the role of nursing documentation in multi-professional care work [13].

1. Introducing the Concept “Usability of Electronic Nursing Record Systems”

In the health informatics research field the concept “usability of healthcare information systems” is often mentioned and referred to (e.g. [9,14,15]). Frequently, the term usability is used to indicate the attributes of a system, which make it easy to use. These definitions or criteria of evaluation do not, however, reflect the contextual characteristics of usability or indicate the impacts that usability has on healthcare work.

In usability research literature the widely known definition for usability of interactive systems is by the ISO 9241-210 standard [16]: Usability is the extent to which a system can be used by specific users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use. Another widely cited definition is presented by Jakob Nielsen [17]: Usability has multiple components and is traditionally associated with the five usability attributes, which are learnability, efficiency, memorability, errors, and satisfaction. Both the definitions emphasize the relation between usability and context of use by stating that the level of usability achieved will always depend on the specific circumstances in which a system is used. These specific circumstances can be described as four elements of context of use: users, tasks, tools, and the physical and social environments [16].

When applied to the area of nursing informatics, these definitions suggest that the usability of nursing information systems should reflect the characteristics of nurses’ work and working environments. The elements of context of use around ENR systems can be described as follows. The environments in which ENR systems are used include clinics and wards with different nursing specialties, special and primary healthcare, as well as private and public sectors. ENR systems have several users, primary of these being nurses and secondary physicians and other healthcare professionals. ENR systems should serve a number of purposes or uses: support documentation, information processing and exchange, and fit for different nursing practices and processes. Technology environments in healthcare organizations consist of hundreds of

J. Viitanen et al. / Usability of Electronic Nursing Record Systems334

information systems; however, for the nurses, ENR systems represent the most important daily used tools for high-quality and accurate patient care.

General definitions of usability and the illustrated characteristics of nursing context of use, suggest that for the purposes of IT development and evaluation the usability of ENR systems can be described with several context specific attributes. The following definition, indicating five of the most apparent usability attributes, was applied and validated in an empirical evaluation study that incorporated four ENR systems.

Usability attributes of ENR systems: � The fluency of reporting practices using ENR systems: the efficiency and

effectiveness of documentation, simplicity of the system, ease of use. � Accuracy of documentation: errors in the performance of documentation,

system’s support for failure protections and recovery. � Learnability: intuitiveness of use, system’s ability to guide new users. � Exploitation of documented information within the nurses: support for nurse’s

work, exchange of information, manner of representation (content and layout). � Support for collaborative care (nurses and other healthcare professionals):

accessibility and readability of documented information, information exchange, and manner of representation compared to multi-professional needs.

2. Usability Study: Procedure and Methods

Evaluation study of four ENR systems was conducted in Finland in spring 2010 with an objective to: 1) evaluate the usability of ENR systems and 2) research how the usability aspects appear in nurses’ documenting practices. The evaluated four ENR systems (referred as A, B, C, and D) represented a heterogeneous group of systems that currently contain the implementation of standardized nursing documentation. Systems A and B are used both in special and primary healthcare, whereas C only in primary healthcare. System D is widely used in occupational health, and is especially suitable for private healthcare organizations’ needs.

The study incorporated two usability research methods: contextual inquiry [18] and expert review using usability heuristics [17]. Contextual inquires followed the principles of contextual inquiry method and were conducted in Finnish in nurses’ real working environments. Altogether 18 nurses from seven healthcare organizations were involved in the study. The criteria for selecting the users was: all four systems as well as public-private sectors and clinic-ward units should be represented, and the nurses need have more than half a year experience in documenting with ENR system according the Nationally Standardized Electronic Nursing Documentation model [13,14]. Table 1 presents a summary of the ENR systems and the users’ backgrounds. Table 1. A summary of the evaluated systems and the involved nurses’ backgrounds in the usability study.

ENR system Number of users in the study

Users’ background: healthcare organization

Users’ background: working unit:

A 6 Public sector, 4 in special and 2 in primary healthcare

2 clinic / 4 ward

B 5 Public sector, special healthcare 3 clinic / 2 ward C 4 Public sector, primary healthcare 4 ward D 3 Private healthcare 3 ward

Total: 18

J. Viitanen et al. / Usability of Electronic Nursing Record Systems 335

Inquiries lasted about 1 hour each. The predetermined themes for interviews included: a) documentation in nursing work, b) practical documentation exercise based on the written scenario, c) use of patient information for one’s own purposes and in collaboration with other professionals. During the inquiry, the documentation exercise was the main theme and incorporated several other topics. Before the data gathering, the researchers had prepared three textual scenarios: one to fit for primary healthcare, others for clinic and special care wards. In the exercise, the nurse was asked to envision a nursing situation described in a scenario, document information as they would normally do, and while working, explain and give reasoning for her actions. For patient information security, interviews and exercises were conducted using educational nursing records. Expert reviews were to supplement the inquiry data and focus on user interfaces and phases of interaction in documentation. The expert reviews were conducted after inquiries by a research group member, who was a usability specialist and based on interviews, had background knowledge about the user’s actions with the system. After inquires and reviews, the qualitative data was interpreted and analyzed the content analysis method [19]. While reading through the typed notes, the findings were iteratively classified into several content categories arising from the data and then grouped together with the ENR usability attributes.

3. Results

The usability study revealed mainly negative findings about the usability of ENR systems. On the positive side, nurses seem to prefer electronic documentation and are not willing to go back to paper-based practices. The main reason being the accessibility of information (compared to paper, electronic documentation is easily accessible) and the reuse of documentation (e.g. when a care plan is accurately written, it can be utilized afterwards in care process and documentation). Although the implementation of electronic nursing documentation and related user interfaces were considerably different, all the evaluated systems shared similar usability problems. In the next section, the main results are presented according to the five usability attributes.

The fluency of reporting practices using ENR systems. The ENR systems do not support effective or efficient documentation. Documentation takes a lot of time because of poor user interface design and complicated interaction sequences. Time required for documentation is considerably high because the nurses are forced to take a huge number of unnecessary interaction steps when performing a simple task, for example a new documentation entry. When the nurses are to select classifications for their entries (in Finland the FinCC includes three hierarchy levels and 719 classes [13]) the system does not follow the nurses’ mental models or provide intelligent support for search or writing. Instead, the implementation forces a user to proceed top-down (from abstract to concrete). The evaluated systems also poorly supported the use of structured templates or copy functionality, although the contents of patient documentation within a clinic or a ward typically follow the same structure. Additionally, the nurses are required to document the same information several times into different systems because of lack of automatic transfer and integration.

Accuracy of documentation. ENR systems allow the nurses to use a variety of documenting practices, both at content and at technical levels. These practices in healthcare organizations rely on unit’s own, commonly agreed instructions and

J. Viitanen et al. / Usability of Electronic Nursing Record Systems336

guidelines, and the nurses’ own experiences and knowhow. Due to the complexity of the user interfaces, the users can easily make errors in performing the documentation. Generally, the system’s support for failure protection is typically insufficient. Accuracy of information is by no doubt endangered because of these facts and the insufficient guidance for users in a documentation process. Especially in a hurry the nurses try use the ENR systems in a simplified and straight forward way, however, knowing that this may reduce the quality of documentation.

Learnability. Nurses felt that learning how to do standardized documentation with an ENR system is demanding and takes a lot of time. Findings from the expert reviews indicated that none of the evaluated four systems are intuitive to use. The systems do not guide the users in information processing. Because of separate systems and lack of integration, the situation is even more complicated for new users.

Exploitation of documented information within the nurses. Nurses argued that based on the documented information it is difficult to get a general view of patient’s situation and needs, as well as previous nursing activities. There are several reasons for this: 1) fragmentariness of structured documentation, 2) documentation into separate systems, 3) lack of summaries, and 4) inappropriate manner of information presentation for readers. In addition, differences in documentation practices, both at content and at technical levels, have an effect on information exchange between the units and nurses’ abilities in finding and understanding entries written colleagues.

Support for collaborative care. Documented information should be easily accessible and readable for all healthcare professionals involved in patient’s care. At present, nurses had difficulties in searching and finding information from ENR systems. Nurses also claimed that physicians experience even more significant problems, and as a consequence, are not willing to use the systems or read the documented patient information. In some healthcare units that were involved in the study, the nurses even felt that physicians may be set against documentation practices with ENR systems.

4. Conclusions

This article provides new information about the usability of ENR systems: a definition with attributes, and summative results to guide the further development. Compared to the widely quoted definitions of usability and the related usability metrics [16,17], the described definition emphasizes the contextual attributes of nursing informatics and thereby increases our understanding about the usability aspects of healthcare information systems. The study indicated that usability problems appear to have effects on nurses’ documentation practices and nursing work, and thus reflect to the healthcare professionals’ attitudes towards standardized documentation with ENRs. These findings emphasize the need for a good fit between nursing practices and ENR systems as also other researchers field have argued in their articles [e.g. 7, 20, 21]. In conclusion, the presented summative results can be used 1) to determine the current state of usability and 2) to guide the conceptual redesign of ENR systems in the future.

J. Viitanen et al. / Usability of Electronic Nursing Record Systems 337

Acknowledgements

We acknowledge the funding of the study by The Finnish Work Environment Fund (Työsuojelurahasto) and the Ministry of Social Affairs and Health (Sosiaali- ja terveysministeriö) in 2010.

References

[1] E. Törnvall, S. Wilhelmsson, L. K. Wahren. Electronic nursing documentation in primary health Care. Scan J Caring Sci 18 (2004), 310-317.

[2] A. Kuusisto, P. Asikainen, H. Lukka, K. Tanttu. Experiences with the electronic nursing discharge summary. Stud Health Tech Inf 146 (2009), 226-230.

[3] B. Gugerty, M. J. Maranda, M. Beachley, V. B. Navarro, S. Newbold, W. Hawk, J. Karp, M. Koszalka, S. Morrison, S. S. Poe, D. Wilhelm. Challenges and Opportunities in Documentation of the Nursing Care of Patients. A Report of the Maryland Nursing Workforce Commission, Documentation Work Group. [Internet]. 2007. [cited 2010 June 10] Available from: http://www.mbon.org/commission2/documentation_challenges.pdf

[4] R. Verwey, R. A. Claassen, M. J. Rutgers, L. P. de Witte. The implementation of an electronic nursing record in a general hospital in the Netherlands. Stud Health Tech Inf 141 (2008), 130-138.

[5] L. Poissant, J. Pereira, R. Tamblyn Y. Kawasumi. The impact of electronic health records on time efficiency of physicians and nurses: A systematic review. JAMIA 12(5) (2005), 505-516.

[6] L. E. Moody, E. Slocumb, B. Berg, D. Jackson. Electronic health records documentation in nursing: Nurses’ perceptions, attitudes, and preferences. CIN 22(6) (2004), 337-344.

[7] J. E. Stevenson, G. C. Nilsson, G. I. Petersson, P. E. Johansson. Nurses’ experiences of using electronic patient records in everyday practice in acute/inpatient ward setting: A Literature Review. Health Inf J 16(1) (2010), 63-72.

[8] M. B. Skov, R. T. Hoegh. Supporting information access in a hospital ward by a context-aware mobile electronic patient record. Per Ubiq Comp 10 (2006) 205-214.

[9] J. Kjeldskov, M. Skov, J. Stage. A longitudinal study of usability in health care: Does time heal? Int J Med Inf 79(6) (2008) e135-e143.

[10] L. W. P. Peute, M. W. M. Jaspers. The significance of a usability evaluation of an emerging laboratory order entry system. Int J Med Inf 76 (2007), 157-168.

[11] P. Liljamo, P. Kaakinen, A. Ensio. Opas FinCC-luokituskokonaisuuden käyttöön hoitotyön sähköisen kirjaamisen mallissa. Kansallisesti yhtenäiset hoitotyön tiedot -hanke 2007-2008. FinCC-luokituksen käyttöopas. [Internet] 2008. [cited 2010 June 10] Available from: http://www.vsshp.fi/fi/4519

[12] K. Tanttu. National Nursing Documentation Project in Finland 5/2005- 5/2008: Nationally Standardized Electronic Nursing Documentation, presentation. [Internet] 2008. [cited 2010 June 10] Available from: http://www.vsshp.fi/fi/dokumentit/15158/National-Nursing-Project-2005-2007.pdf

[13] P. Nykänen, J. Viitanen, A. Kuusisto. Hoitotyön kansallisen kirjaamismallin ja hoitokertomusten käytettävyys, Project report (in Finnish), University of Tampere, Finland, publication D-2010-7. [Internet] 2010. [cited 2010 June 10] Available from: http://www.cs.uta.fi/reports/dsarja/D-2010-7.pdf

[14] J. L. Belden, R. Grayson, J. Barnes. Defining and Testing EMR Usability: Principles and Proposed Methods of EMR Usability Evaluation and Rating. HIMMS EHR Usability Task Force. [Internet] 2009. [cited 2010 June 10] Available from: https://mospace.umsystem.edu/xmlui/handle/10355/3719

[15] A. W. Kushniruk, V. L. Patel. Cognitive and usability engineering methods for the evaluation of clinical information systems. J Biomed Inf 37 (2004) 56-76.

[16] ISO 9241-210, International Standard: Ergonomics of human-system interaction – Part 210: Human- centred design for interactive systems. First edition 2010-03-15. Reference number ISO 9241- 210:2010(E).

[17] J. Nielsen. Usability Engineering. Academic Press, New York, 1993. [18] H. Beyer, K. Holzblatt. Contextual design: Defining Customer-Centred Systems. Academic Press, San

Diego, USA, 1998. [19] R. P. Weber. Basic Content Analysis, (second edition). Series: Qualitative Applications in the Social

Sciences. SAGE University Paper, Sage Publications, Inc, USA, 1990. [20] E. Ammenwerth, U. Mansmann, C. Iller, R. Eichstädter. Factors affecting and affected by user

acceptance of computer-based nursing documentation. JAMIA 10 (2003). 69-84. [21] E. Reuss, K. Rochus, R. Naef, S. Hunziker, L. Furler. Nurses' Working Practices: What Can We Learn

for Designing Computerized Patient Record Systems? in book: HCI and Usability for Medicine and Health Care (ed. A. Holzinger). Springer Berlin, Heidelberg, 2007.

J. Viitanen et al. / Usability of Electronic Nursing Record Systems338

be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written

permission. However, users may print, download, or email articles for individual use.