ClinicalSimulationasanEvaluationMethodinHealthInformatics1-Copy.pdf
Clinical Simulation as an Evaluation Method
in Health Informatics
Sanne JENSEN a,1
a The Capital Region of Denmark, Copenhagen, Denmark
Abstract. Safe work processes and information systems are vital in health care.
Methods for design of health IT focusing on patient safety are one of many
initiatives trying to prevent adverse events. Possible patient safety hazards need to
be investigated before health IT is integrated with local clinical work practice
including other technology and organizational structure. Clinical simulation is
ideal for proactive evaluation of new technology for clinical work practice.
Clinical simulations involve real end-users as they simulate the use of technology
in realistic environments performing realistic tasks. Clinical simulation study
assesses effects on clinical workflow and enables identification and evaluation of
patient safety hazards before implementation at a hospital. Clinical simulation also
offers an opportunity to create a space in which healthcare professionals working
in different locations or sectors can meet and exchange knowledge about work
practices and requirement needs. This contribution will discuss benefits and
challenges of using clinical simulation, and will describe how clinical simulation
fits into classical usability studies, how patient safety may benefit by use of
clinical simulation, and it will describe the different steps of how to conduct
clinical simulation. Furthermore a case study is presented.
Keywords. Ergonomics, eHealth, qualitative evaluation, clinical simulation, risk, safety.
1. Introduction
Implementation of health IT in relation to improvement of patient safety and
optimization of work flow is a paradox [1]. Even though health IT is intended and
anticipated to have a positive impact on quality and efficiency of health care [2], the
application of new technology in healthcare may also increase patient safety hazards [3,
4]. Studies show that adverse events are indeed often related to the use of technology
[5-7].
Design of health IT focusing on protecting patient safety is one of many initiatives
trying to prevent adverse events [8, 9]. 2 Patient safety does not entirely rely on
technology but is highly influenced by the interaction between users and technology in
a specific context [10], and sociotechnical issues and human factors are related to many
unintended consequences and patient safety hazards [7, 8, 11]. Possible patient safety
hazards such as design of the IT system itself; embedding of IT system into local work
1
Corresponding author: Sanne Jensen, The Capital Region of Denmark, Borgervanget 7, 2100
Copenhagen O, Denmark, [email protected]. 2 See also: F. Magrabi et al., Health IT for patient safety and improving the safety of health IT, in: E.
Ammenwerth, M. Rigby (eds.), Evidence-Based Health Informatics, Stud Health Technol Inform 222, IOS
Press, Amsterdam, 2016.
Evidence-Based Health Informatics E. Ammenwerth and M. Rigby (Eds.) © 2016 The authors and IOS Press.
This article is published online with Open Access by IOS Press and distributed under the terms of the Creative Commons Attribution Non-Commercial License.
doi:10.3233/978-1-61499-635-4-152
152
practice in the local environment; and the introductory and training processes need to
be investigated when health IT is integrated with local clinical work practice including
technology and organizational structures. The substantial complexity of organizations,
work practices and physical environments within the healthcare sector impacts design,
evaluation and implementation of information systems [12, 13]. Healthcare
environments are profoundly collaborative and rely on coordination between various
health professionals [14]. They are characterized by delegated decision-making,
multiple viewpoints and inconsistent and evolving knowledge bases [15]. Multiple
groups with potentially divergent values and objectives work together and face many
contingencies which cannot be fully anticipated [16, 17]. These matters challenge
design and evaluation of health IT.
Clinical simulation tries to address this challenge. Compared with other methods,
e.g. heuristic inspection and low fidelity usability evaluation 3 , clinical simulation takes
the clinical context into account. In contrast, for example, heuristic inspection focuses
on the user interface, and low fidelity usability testing focuses on technology and on
the specific tasks of individual users. By including the clinical context, clinical
simulation is ideal for proactive evaluation of new technology for clinical work practice
[18, 19]. Clinical simulations involve real end-users as they simulate the use of
technology in realistic environments performing realistic tasks [20]. Clinical simulation
studies the effects on clinical workflow [21] and enables identification and evaluation
of patient safety hazards before implementation at a hospital or other clinical setting
[22]. Clinical simulation also offers an opportunity to create a space in which
healthcare professionals working in different locations or sectors can meet and
exchange knowledge about work practices and requirement needs [23, 24].
Hospital organizations and work practice are extremely complex with many
different healthcare groups and many interactions and correlations [25] involved, and
many acute situations are encountered during daily work practice in hospital settings
[26]. This complexity affects the technology that is implemented at hospitals [27] and
confronts the methodology used for design and evaluation of healthcare information
systems. Failure to comprehend the nature and range of end-users has been highlighted
as a key issue in many systems’ failing to become accepted by healthcare professionals
[28]. Furthermore, an understanding of the context in which the systems will be used
must take into account not only tasks and settings [29], but also the range, competences
and cognitive capacities of an increasing variety of potential end-users [30]. The risk of
endangering patient security calls for careful evaluation before implementing new
technology in real life settings [31].
Usability relates to how a product can be used to achieve specified goals with
effectiveness, efficiency and satisfaction in a specified context of use. 3 Usability
focuses on use of technology in a specific context [32]. Context may be defined as
“users, tasks, equipment (hardware, software and materials), and the physical and
social environment in which a product is used” [32]. It does however raise several
questions, e.g. ‘who are the users?’, ‘what are their tasks?’ and ‘with whom, where and
under what conditions are they performing these task?’. The healthcare sector poses
challenges due to the larger potential numbers and classes of users, e.g. nurses,
physicians and pharmacists [28]. Furthermore, the definition does not take multiple
3 See also: R. Marcilly et al., From usability engineering to evidence-based usability in health IT, in: E.
Ammenwerth, M. Rigby (eds.), Evidence-Based Health Informatics, Stud Health Technol Inform 222, IOS
Press, Amsterdam, 2016.
S. Jensen / Clinical Simulation as an Evaluation Method in Health Informatics 153
users and their professional interaction into account, and nor does it take parts of or a
whole organization into account.
According to Hertzum [33] many views may be put on usability, dividing it into
six images; 1) universal usability: usability in a system for everybody to use, 2)
situational usability: quality-in-use of a system in a specified situation with its users,
tasks, and wider context of use, 3) perceived usability: usability concerns the user’s
subjective experience of a system based on her or his interaction with it, 4) hedonic
usability: usability is about joy of use rather than ease of use, task accomplishment, and
freedom of discomfort, 5) organizational usability: usability implies groups of people
collaborating in an organizational setting, and 6) cultural usability: usability takes on
different meaning depending on the users different background. Hertzum claims that all
images should be taken into account when evaluating usability.
Another aspect when designing and evaluating information systems is user
involvement. User-centred design focuses on incorporating the user’s perspective into
the development process in order to attain a usable IT system [34]. 4 The key principles
of user-centred design are 1) active involvement of users and clear understanding of
user and task requirements; 2) an appropriate allocation of function between user and
system; 3) iteration of design solutions; and 4) multi-disciplinary design teams. The
human-centred design cycle [32] shown in Figure 1 describes five essential processes
which should be undertaken in order to incorporate usability requirements into the
software development process.
Figure 1: The human-centred design cycle.
The process is iterative with the cycle being repeated until the particular usability
objectives have been obtained. Studies show that effective involvement of users may
leads to 1) improved quality of the system arising from more accurate user
requirements; 2) avoidance of costly system features that users do not want or cannot
4 See also: A. Kushniruk et al., Participatory design and health IT evaluation, in: E. Ammenwerth, M.
Rigby (eds.), Evidence-Based Health Informatics, Stud Health Technol Inform 222, IOS Press, Amsterdam,
2016.
S. Jensen / Clinical Simulation as an Evaluation Method in Health Informatics154
use; 3) improved levels of acceptance of the system; 4) greater understanding of the
system by the user resulting in more effective use; and 5) increased participation in
decision-making in the organization [35, 36].
2. Clinical simulation
Clinical simulation supports involvement of context as well as end-users in pre-
implementation design and evaluation of health IT. Clinical simulations involve real
end-users as they simulate the use of technology in realistic environments performing
realistic tasks [20]. As shown in Figure 2, clinical simulation can be used in different
evaluation activities at various phases of the development life cycle from evaluation of
work practice and user requirements, evaluation of the initial specification and early
design solution so as to seek to eliminate patient risks created or perpetuated, through
to application assessment in work practice and assessment of training programs.
Patient safety issues may be explored in all phases of the lifecycle by observing
and analysing medical errors and work flow in a simulated situation close to a real life
environment [22]. In the first phases of the lifecycle of health IT, simulation may be
used for specification and evaluation of user requirements [19], as well as for obtaining
knowledge and evaluate work practice [37]. This may involves observation of
clinicians applying information technology under simulated conditions
Figure 2. Simulation evaluations in information system life cycle.
Likewise in the design phase simulation is well suited as a method for user
involvement in connection with evaluation of the design. Simulation studies can be
designed to gain practical experience in evaluation of new technology without
introducing any kind of ethical issues and without putting patients at risk [20]. In this
way it is possible to test prototypical software in realistic scenarios and environments.
Simulations can be performed in laboratories as well as in situ in a ward, an operating
theatre or an outpatient clinic [20]. Simulation studies aim to evaluate design proposals
for a new technology and combine elements of the laboratory test and the field study
[22].
Particular aspects of implementation can be visualized by simulation e.g. user
interaction in work practice, the need for training, and the impact of decision support
[22]. Unintended consequences of new systems such as changes in work processes and
patient outcome 5 may also be detected and can provide constructive and valuable
5 See also: F. Magrabi et al., Health IT for patient safety and improving the safety of health IT, in: E.
Ammenwerth, M. Rigby (eds.), Evidence-Based Health Informatics, Stud Health Technol Inform 222, IOS
Press, Amsterdam, 2016.
S. Jensen / Clinical Simulation as an Evaluation Method in Health Informatics 155
information for organizational decision makers [18]. Clinical simulation can also be
used as common ground for discussion and negotiation and as an organizational
learnings space, where knowledge of other parts of an organization can be acquired
[37].
The realism and acceptance of the simulation depend on the degree of fidelity in
the simulation set-up. Dahl and colleagues [38] have developed a simulation
acceptance model with four fidelity dimensions: 1) environment – physical elements,
such as rooms, beds and patient; 2) equipment – elements, such as mock-ups and
electronic devices; 3) functionality – such as system functionalities and interactive
devices; and 4) tasks – clinical task such as administration of drugs and ward rounds.
These fidelity dimensions affect the perceived realism and thereby acceptance of the
simulation made by the involved clinicians and should be considered carefully
according to the purpose of the simulation.
Clinical simulations are performed in three phases; 1) introduction, 2) simulation,
and 3) evaluation. Prior to the simulation, the participants are introduced to the
information system and to the simulation. Simulation facilities are a dedicated facility
with two rooms linked by a one-way mirror. During the simulation, a simulation
facilitator is located in the simulation room. The facilitator assists the simulation and
supports the participating clinician. An instructor located in the observation room
instructs the patient and the simulation facilitator. A one-way mirror separates the two
rooms. The simulation is observed by health informatics experts and sometimes by key
stakeholders, such as colleagues from hospitals, clinical managers, quality managers
and vendors [37]. The observers are located in the observation room.
An example of how simulation facilities may look like is presented in Figure 3.
The simulation room is established as a bed room for two patients with bedside tables
and a portable for the healthcare professional. An observation room with portables and
chairs is located in the right corner. A one-way mirror is separating the two rooms.
Figure 3. Overview of the physical simulation set-up.
Simulation of handover from hospital to community care by messaging
technologies can also be carried out in a simulation laboratory. In such situations
another simulation room may replicate a nursing office at the community care. In
situations where it is not possible to replicate the location of the simulation in a
laboratory, simulation in situ may be used. This could be scenarios where large x-ray
scanners or other large equipment is involved.
S. Jensen / Clinical Simulation as an Evaluation Method in Health Informatics156
The simulating clinicians are asked to “think aloud” so that the observers can
acquire a deeper understanding of the human task-behaviour. Depending on the
purpose of the clinical simulation, the clinicians are sometimes also able to observe
their colleagues, when not participating in the simulation themselves [39]. The different
roles and their locations are described in Table 1.
Table 1. Overview and description of different roles and their locations during clinical simulation.
Roles Description Location
Instructor Overall responsible for the simulation. Instructs
simulation facilitator and patient(s) during simulation by
use of intercom equipment and facilitates debriefing.
Observation room
Simulation
facilitator
Briefs clinicians prior to simulation and provides support
during simulation. Receives instructions from and assists
instructor during simulation.
Simulation room
Observers Observes and makes notes during simulation; e.g.
usability, support of work practice, patient safety
Observation room
Clinicians Simulates scenario. Thinks aloud during simulation.
Participates as interviewee in interview
Simulation room
Actor Acts as e.g. patient, colleague during simulation and
receives instructions from instructor. Simulation room
After the simulation, the proposed information system is evaluated. Participants
are asked to complete questionnaires and participate in a de-briefing interview.
Additional to interview guides, observations made by the observers during the
simulations are used as background for the interviews [24]. It must be clarified in
advance to whom the results are to be presented and how the results and
recommendations should be implemented. The same goes for the respective mandates
of the participating clinicians as well as the observers.
3. Case Study: Simulation study of a clinical information system
The aim of the case study was to investigate how a newly-acquired standard clinical
information system for doctors to sign for laboratory results might support clinical
practice, and to identify potential patient safety hazards prior to its implementation [40].
The aim of the information system was to obtain an IT supported work flow for
physicians receiving and signing laboratory test results in order to improve patient
safety. In addition to implementation aspects such as training and information, the
purpose was also to evaluate future work practice, the relation between technology and
existing work processes, and the extent to which clinical simulation may be applied as
a proactive method to identify and evaluate potential patient safety hazards prior to
implementation.
The existing workflow was paper based; i.e. prints were made from digital systems
and were signed by a doctor in order to document that the specific test result had been
reviewed by a doctor. The laboratory tests were handled by various information
systems. Some test results were on paper and others were electronic. The background
for the local work flows was based on interpretations of a national guideline for
handling laboratory test results. This national guideline was developed as part of a
quality assurance initiative to increase patient safety. As a rule, the physicians signed to
confirm that they have seen a laboratory test result. The physicians also signed to
S. Jensen / Clinical Simulation as an Evaluation Method in Health Informatics 157
confirm that they have handled the test results. The essential challenges about the paper
based workflow were 1) lack of overview about whether a result has arrived; 2)
uncertainty about whether a test result has been seen by a physician; and 3) lack of
documentation about which physician has seen a test result.
The objective of purchasing the new information system was to increase quality in
work practice and minimize the risk to patient safety by implementing a new standard
information system. The information system collects laboratory test results and
supports electronically documentation of acknowledging the results. The study was
expected to be moderate and manageable because the information system was a
standard off-the-shelf product and the intended work flow was supposed to be narrow
and well-defined. The information system was to be implemented at two pilot
departments. Both departments included patient wards and outpatient clinics. Prior to
implementation, the existing work practice was analysed and future generic work flows
defined. The functionality of the information system and collaborative future work
practice were evaluated by means of clinical simulation. The aim of the simulation
study was to assess how the information system supported clinical practice and to
identify potential patient safety hazards prior to its implementation.
Initial field studies were carried out at the two pilot departments covering both
patient wards and outpatient clinics in order to gain insight into existing work practice
concerning receipt, handover and acknowledgement of laboratory test results. Two
workshops were then held with physicians, nurses and medical secretaries from the
pilot departments, health informaticians and experts from the regional quality unit. At
the first workshop, future work practice and the information system were analysed and
required changes were identified. At the second workshop, future work practice was
determined, focusing on improved efficiency, quality, continuity and communication.
Existing routines were contested and organizational changes were initiated ahead of
implementation to create acceptance and a readiness to change among future end-users.
An analysis of work practice conducted prior to the clinical simulation revealed
that there were significant differences between the hospitals, between the patient wards,
and the outpatient clinics – and indeed also between the individual healthcare
professionals. Furthermore, the design of future work practice presented a number of
challenges and it was not possible to design a generic work flow to cover both patient
ward and outpatient clinic. This was to some extent due to differences between local
work flows but also due to the fact that the information system functionality did not
provide adequate support for work practice.
Clinical simulation was conducted after the two workshops. The purpose of the
clinical simulation was to evaluate patient safety issues and future work practice using
the new information system before its implementation. Six healthcare professionals
from the two pilot departments (two physicians, three nurses and one medical
secretary) participated in the simulations. Clinical managers from the pilot sites,
implementation experts and health informatics experts were observing the simulations.
Figure 4 shows the simulation room seen from the observation room through a one way
mirror. The simulation set-up is an outpatient clinic where a physician is preparing for
a meeting with a patient.
A total of 11 scenarios were performed during the simulation; six scenarios from
patient wards and five scenarios from outpatient clinics. All scenarios were related to
signing and handling laboratory test results. Some of these were frequently performed
work flows, e.g. ward rounds and visits to the outpatient clinic, while others were
critical work flows; e.g. urgent test results, sorting test results and handover of
S. Jensen / Clinical Simulation as an Evaluation Method in Health Informatics158
responsibility. The simulation set-up was very realistic. The computers used were
identical with those used at the hospitals and the system was fully developed and
operational. The scenarios were composed in participation with clinicians from the
pilot sites and based on realistic patient cases. The simulation room was designed as
either a ward bedroom or clinical office. The role of patient was enacted by a
healthcare professional.
One of the purposes of using clinical simulation in relation to implementation was
to investigate how the information system supported clinical practice and to determine
whether the information system should be implemented at the hospitals. Therefore
there was a need for high fidelity in the case study.
Figure 4. Simulation room seen from observation room.
The clinical simulation identified many uncertainties concerning work flow,
handling of responsibility, and other organizational and technical challenges. High
fidelity functionalities, such as integration to other information systems, revealed
patient safety issues; e.g. notes related to a test result were not shown in relation to the
test result in the new information system. The physician could only find the notes in the
lab system. Apart from many negative findings, there were also positive findings,
including improved overview of laboratory test results and no paper test results were
left lying around, at the risk of disappearing.
We did not have any patient safety experts attending as observers during the
simulation. Instead the simulation evaluation report was subsequently shown to the
patient safety experts. Having patient safety experts observe the simulation would have
improved the outcome considerably. Several organizational and technological issues,
which were regarded as inconveniences by others, were detected as patient safety risks
by the patient safety experts. These experts have great experience of what can go
wrong and are able to focus on these matters during the simulation. They observe the
interaction between the user and the interface of the technology but just as much the
interaction with the technology in the clinical context. Inclusion of clinical context is
one of the most powerful elements in clinical simulation. By allowing clinicians to use
new technology in the way it is supposed to be used, patient safety issues become
visible. Clinical simulation enables visualization of technology in connection with
clinical context without endangering patients [22]. Therefore the choice of observers is
very important. Each expert focuses on his or her own field. For this reason, observers
must be chosen carefully and bearing in mind the purpose of the simulation.
S. Jensen / Clinical Simulation as an Evaluation Method in Health Informatics 159
As a result of the simulation additional new requirements of the information
system were determined, e.g. new functionality for sorting the list of laboratory results
according to date and time of the results. It was decided to initiate a pilot
implementation despite the fact that the information system did not fully support the
work flows. Some of the organizational challenges were solved and it was agreed that
the remaining challenges regarding future work practice should be subject to scrutiny
during the pilot implementation.
The challenges not solved prior to the pilot implementation were the transferability
of work practice between patient wards and outpatient clinics, confidentiality of some
test results, risk of several users handling the same test result simultaneously, missing
interaction between prescription of test and signing of test results, no possibility of
undoing signing of test results, comments do not stand out distinctly and integration
between information system and paper-based test results from private laboratories. The
issues were observed and evaluated after the system was implemented.
4. Discussion
The clinical simulation focused on formative evaluation and primarily was used as a
learning process. Formative evaluation studies can facilitate system adoption and
utilization [41] and aim to improve a system during its development or implementation,
while summative evaluation focuses on evaluation of a system that is already up and
running [42]. Formative evaluation may identify potential problems, such as patient
safety issues, during the development phase and thus provide opportunities to improve
a system as it develops.
In the simulation study, the results of the formative evaluation regarding patient
safety issues and work practice for handling laboratory test results were presented and
discussed at meetings with the various stakeholders, i.e. the patient safety unit, the
quality unit and the implementation departments. Precautions were taken in relation to
patient safety matters and work practice. Many of these precautions were subsequently
implemented, regardless of the implementation of information system.
Unintended incidents often occur in the interaction between humans, technology
and work practice [4, 10]. Clinical simulations allow visualization of the correlation
between human, technology and organization. More conventional usability evaluations
tend to visualize the interaction between the user and the technology but do not include
work practice context [20, 43]. By including all three aspects (humans, technology and
organization), patient safety challenges were revealed as well as organizational and
technical challenges. New work practice in itself may also lead to unintended incidents.
This was also revealed during the clinical simulation.
To expose cognitive and socio-technical issues, all fidelity dimensions described
by Dahl and colleagues [23] need to be high on all four dimensions. The overall
simulation configuration affects how the realism of the simulation experience is
perceived [38]. Cognitive aspects of work practice relate to the clinical context and
therefore depend on the degree of environment and task realism, whereas equipment
fidelity and functional fidelity relate to cognitive aspects of the technical context.
Socio-technical aspects and patient safety matters lie in the intersection between user,
organization and technology [40]. High fidelity simulations are time-consuming [44]
though and the purpose of simulation studies and the need for fidelity should therefore
be planned carefully.
S. Jensen / Clinical Simulation as an Evaluation Method in Health Informatics160
Traditional information systems are often designed around an idealized model of
the tasks and workflow, and failures in information systems are often blamed on human,
social and cultural “barriers” to technology adoption [10]. The case study revealed
differences between such an idealized model of the task that needed to be accomplished
and the way in which clinicians were actually working. Some of the differences were
due to local interpretations of the regional guidelines and one of the conclusions
reached was that the regional quality unit should develop a regional standard for
signing off test results. Another issue lay in the fact that the information system was a
standard system which did not provide adequate opportunities to configure the system
to match the local setting. If work practice differs from department to department, local
configuration is a requirement. A regional standard was introduced to resolve this issue.
Clinical simulation did not reveal all challenges related to the information system.
The outcome of clinical simulations depends on the quality of the scenarios and patient
cases they cover. In the case study, the scenarios during the simulations did not include
unusual results or pre-ambulatory test results, but only became clear during a
subsequently pilot implementation. Clinical simulation involves an inherent risk of
giving an idealized picture compared to real life as it is very resource demanding to
simulate the complexity of real life situations at a hospital. These matters are important
to take into account when planning and designing the simulation.
Another aspect is the purpose of the evaluation and the relation between existing
and future work practice. What is to be evaluated - future or existing work practice?
And do the end-users comprehend and approve the new work practice? Furthermore, if
the existing work practice in a department does not follow the existing guidelines, this
may influence the simulation of the interaction between future work practice, end-users
and technology as well as subsequent implementation.
Several muddled work flows became clear during the simulation and observers
focusing on work flows agreed that a further work flow analysis was needed. This
resulted in revision of the future work practice. Many of the issues found during the
simulation were addressed before the pilot implementation, and those that were not
solved were observed again during the pilot implementation. As such clinical
simulation cannot replace a pilot implementation, but should rather be regarded as a
valuable supplement.
Patient safety issues are difficult to assess due to the fact that many patient safety
challenges lie in the details and are triggered by adverse events and disturbances [24].
The results of the case study showed that clinical simulation took the clinical context
into account, while other methods, e.g. heuristic inspection, focus on the user interface.
Low fidelity usability testing focuses on technology and specific tasks for single users.
Some patient safety risks may therefore be difficult to pinpoint using these methods.
Clinical simulation provides a comprehensive view on the information system taking
into account the correlation between IT, work practice and adverse events, and is
therefore a very suitable method for assessing patient safety issues.
The resources invested in preparing and performing simulation studies may be
exhaustive, depending on the required degree of fidelity. It is essential that the
resources invested in creating a realistic setting match the purposes of the simulation
and the simulation set-up [43, 44]. However, the resources saved and iatrogenic effects
avoided by using clinical simulation for analysis and evaluation purposes are difficult
to quantity as it is difficult to put a price on the value of patients’ lives. Still, clinical
simulation is a beneficial evaluation method, as it takes place in a controlled
environment where there is no risk of injuring real patients [20, 40].
S. Jensen / Clinical Simulation as an Evaluation Method in Health Informatics 161
As described clinical simulation can be used to analyse, design and evaluate user
requirements and work practice and serve as common ground to help to achieve a
shared understanding between various communities of practice. The primary benefits
of using clinical simulation are 1) involvement of users and clinical context, 2)
controlled environments for experiments and formative evaluations of user satisfaction,
usefulness and patient safety, 3) environments for addressing and visualizing cross-
sectorial and cross-functional topics, and 4) organizational learning space and common
ground for gaining shared understanding.
The main concerns and challenges of using clinical simulation are that clinical
simulation does not reflect the social-technical issues over time and does not cover all
possible work practice situations and issues. The purpose and choice of scenarios
determines to a great extent the outcome, and the purpose and design of clinical
simulation must therefore be considered very carefully.
Recommended further readings
1. Kushniruk A, Nohr C, Jensen S, Borycki EM, From Usability Testing to Clinical
Simulations: Bringing Context into the Design and Evaluation of Usable and Safe
Health Information Technologies, Yearb Med Inform 8 (2013), 78-85.
2. Jensen S, Kushniruk A, Boundary objects in clinical simulation and design of
eHealth, Health Informatics Journal Oct 9, 2014:1460458214551846.
3. Jensen S, Kushniruk AW, Nøhr C, Clinical simulation: A method for development
and evaluation of clinical information systems, Journal of Biomedical Informatics
54 (2015), 65-76.
4. Ammenwerth E, Hackl WO, Binzer K, Christoffersen TE, Jensen S, Lawton K,
Skjoet P, Nohr C, Simulation Studies for the evaluation of health information
technologies: experiences and results, HIM J 41(2012), 14-21.
5. Jensen S, Clinical Simulation: For what and how can it be used in design and
evaluation of health IT, Techno-Anthropology in Health Informatics:
Methodologies for Improving Human-Technology Relations, Studies in Health
Technology and Informatcs 215 (2015), pp. 217-28.
Food for thought
1. What might be the pros and cons of clinical simulation seen from an end-user perspective and how may it differ from a management and policy perspective?
2. Clinical simulation refers to simulation in a clinical set-up. How may simulation fit into other high-risk areas such as pharmacies and ambulances?
3. As healthcare technology moves into patients’ homes, simulation could also be used in private settings. How would a simulation design differ when conducting
simulations in patients home?
4. How may clinical simulation be used in other clinical fields, such as biomedical engineering?
S. Jensen / Clinical Simulation as an Evaluation Method in Health Informatics162
References [1] E. Coiera, J. Aarts, C. Kulikowski, The dangerous decade, J Am Med Inform Assoc 19 (2012), 2-5. [2] D.W. Bates, M. Cohen, L.L. Leape, M.J. Overhage, M.M. Shabot, T. Sheridan, Reducing the
frequency of errors in medicine using information technology, J Am Med Inform Assoc 8 (2001), 299-
308.
[3] E. Ammenwerth, Evidence-Based Health Informatics: How do we know what we know? Methods Inf Med 54(4) (2015), 298-307.
[4] R. Koppel, T. Wetterneck, J.L. Telles, B.T. Karsh, Workarounds to barcode medication administration systems: their occurrences, causes, and threats to patient safety, J Am Med Inform Assoc 15 (2008),
408-423.
[5] F. Magrabi, M. Baker, I. Sinha, M.S. Ong, S. Harrison, M.R. Kidd, W.B. Runciman, E. Coiera, Clinical safety of England's national programme for IT: A retrospective analysis of all reported safety
events 2005 to 2011, International Journal of Medical Informatics 84 (2015), 198-206.
[6] Y.Y. Han, J.A. Carcillo, S.T. Venkataraman, R.S. Clark, R.S. Watson, T.C. Nguyen, H. Bayir, R.A. Orr, Unexpected increased mortality after implementation of a commercially sold computerized
physician order entry system, Pediatrics 116 (2005), 1506-1512.
[7] M.I. Harrison, R. Koppel, S. Bar-Lev, Unintended Consequences of Information Technologies in Health Care – An Interactive Sociotechnical Analysis, J Am Med Inform Assoc 14 (2007), 542-549.
[8] M.C. Beuscart-Zephir, C. Nøhr, Human factors engineering for computer-supported identification and prevention of adverse drug events, Stud Health Technol Inform 148 (2009), 14-24.
[9] R. Koppel, D.A. Kreda, Healthcare IT usability and suitability for clinical needs: challenges of design, workflow, and contractual relations, Stud Health Technol Inform 157 (2010), 7-14.
[10] E. Coeira E, Interaction design theory, International Journal of Medical Informatics 69 (2003), 205- 222.
[11] R. Beuscart, P. McNair, S.J. Darmoni, V. Koutkias, N. Maglaveras, M.C. Beuscart-Zephir, C. Nøhr, Patient safety: detection and prevention of adverse drug events, Stud Health Technol Inform 150
(2009), 968-971.
[12] YY. Han, J.A. Carcillo, S.T. Venkataraman, R.S. Clark, R.S. Watson, T.C. Nguyen, H. Bayir, R.A. Orr, Unexpected increased mortality after implementation of a commercially sold computerized
physician order entry system, Pediatrics 116 (2005), 1506-1512.
[13] M. Berg, Health information management: Integrating information and communication technology in health care work, Routledge; 2003.
[14] P. Carayon, Human factors of complex sociotechnical systems, Appl Ergon 37 (2006), 525-535. [15] M. Berg, Patient care information systems and health care work: a sociotechnical approach,
International Journal of Medical Informatics 55 (1999), 87-101.
[16] P. Carayon, E.J. Bass, T. Bellandi, A.P. Gurses, M.S. Hallbeck, V. Mollo, Sociotechnical systems analysis in health care: a research agenda, IIE transactions on Healthcare Systems Engineering 1
(2011), 145-160.
[17] T.G. Kannampallil, G.F. Schauer, T. Cohen, V.L. Patel, Considering complexity in healthcare systems, Journal of Biomedical Informatics 44 (2011), 943-947.
[18] E.M. Borycki, M. Househ, A.W. Kushniruk, C. Kuziemsky, Use of qualitative methods across the software development lifecycle in health informatics, Stud Health Technol Inform 164 (2011), 293-297.
[19] A.W. Kushniruk, V.L. Patel, Cognitive and usability engineering methods for the evaluation of clinical information systems, J Biomed Inform 37 (2004), 56-76.
[20] A. Kushniruk, C. Nøhr, S. Jensen, E.M. Borycki, From Usability Testing to Clinical Simulations: Bringing Context into the Design and Evaluation of Usable and Safe Health Information Technologies,
Yearb Med Inform 8 (2013), 78-85.
[21] E. Borycki, A. Kushniruk, Identifying and preventing technology-induced error using simulations: application of usability engineering techniques, Healthc Q 8 Spec No (2005), 99-105.
[22] S. Jensen, K.M. Lyng, C. Nøhr, The role of simulation in clinical information systems development, Stud Health Technol Inform 180 (2012), 373-377.
[23] S. Jensen, C. Nøhr, S.L. Rasmussen, Fidelity in clinical simulation: how low can you go? Stud Health Technol Inform 194 (2013), 147-153.
[24] S. Jensen, S.L. Rasmussen, K.M. Lyng, Evaluation of a Clinical Simulation-based Assessment Method for EHR-platforms, Stud Health Technol Inform 205 (2014), 925-929.
[25] M. Berg, Health Information Systems. Integrating information technology in health care work, Routledge, 2006.
[26] P. Carayon, Handbook of human factors and ergonomics in health care and patient safety, CRC Press, 2011.
S. Jensen / Clinical Simulation as an Evaluation Method in Health Informatics 163
[27] J.S. Ash, M. Berg, E. Coiera, Some unintended consequences of information technology in health care: the nature of patient care information system-related errors, J Am Med Inform Assoc 11 (2004),
104-112.
[28] A. Kushniruk, P. Turner, A framework for user involvement and context in the design and development of safe e-Health systems, Stud Health Technol Inform 180 (2012), 353-357.
[29] E.M. Borycki, L. Lemieux-Charles, L. Nagle, G. Eysenbach, Evaluating the impact of hybrid electronic-paper environments upon novice nurse information seeking, Methods Inf Med 48 (2009),
137-143.
[30] C. Pearce, A. Shachak, A. Kushniruk, S. de Lusignan, Usability: a critical dimension for assessing the quality of clinical systems, Inform Prim Care 17 (2009), 195-198.
[31] M. Berg, Health Information Technology in Health Care Work, Routledge, 2004. [32] ISO 9241-210: Ergonomics of human system interaction - Part 210: Human-centred design for
interactive systems. International Standardization Organization (ISO), Switzerland, 2009.
[33] M. Hertzum, Images of usability, Intl Journal of Human-Computer Interaction 26 (2010), 567-600. [34] M. Maguire, Methods to support human-centeres design, Intl Journal of Human-Computer Interaction
55 (2001), 587-634.
[35] D. Robey, D. Farrow, User involvement in information system development: A conflict model and empirical test, Management Science 28 (1982), 73-85.
[36] L. Damodaran, User involvement in the systems design process-a practical guide for users, Behaviour & Information Technology 15 (1996), 363-377.
[37] S.L. Rasmussen, K.M. Lyng, S. Jensen, Achieving IT-supported standardized nursing documentation through participatory design, Stud Health Technol Inform 180 (2012), 1055-1059.
[38] Y. Dahl, O.A. Alsos, D. Svanæs, Fidelity Considerations for Simulation-Based Usability Assessments of Mobile ICT for Hospitals, International Journal of Human-Computer Interaction 26 (2010), 445-
476.
[39] S. Jensen, S. Vingtoft, C. Nøhr, Benefits of a clinical planning and coordination module: a simulation study, Stud Health Technol Inform 183 (2013), 220-224.
[40] S. Jensen, Use of clinical simulation in development and implementation of clinical information systems, PhD thesis, University of Aalborg, 2014.
[41] H.P. Lundsgaarde, R.M. Gardner, R.L. Menlove, Using attitudinal questionnaires to achieve benefits optimization, in: Proceedings of the Annual Symposium of the American Medical Informatics
Association, 1989, pp. 703-708.
[42] B. Kaplan, Addressing organizational issues into the evaluation of medical systems, J Am Med Inform Assoc 4 (1997), 94-101.
[43] E. Ammenwerth, S. Graber, G. Herrmann, T. Bürkle, J. König, Evaluation of health information systems-problems and challenges, Int J Med Inform 71 (2003), 125-135.
[44] E. Ammenwerth, W.O. Hackl, K. Binzer, T.E. Christoffersen, S. Jensen, K. Lawton, P. Skjoet, C. Nøhr, Simulation Studies for the evaluation of health information technologies: experiences and
results, HIM J 41 (2012), 14-21.
S. Jensen / Clinical Simulation as an Evaluation Method in Health Informatics164
Copyright of Studies in Health Technology & Informatics is the property of IOS Press and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.
