Interoperability Project Plan

mshp

Evaluationofpeoplesocialandorganizationalissues-sociotechnicalethnographicevaluation.pdf

Home >Nursing homework help >Interoperability Project Plan

Evaluation of People, Social, and

Organizational Issues – Sociotechnical

Ethnographic Evaluation

Bonnie KAPLAN a,1

a Yale Center for Medical Informatics, Yale University, New Haven, CT, USA

Abstract. Sociotechnical approaches are grounded in theory and evidence-based.

They are useful for evaluations involving health information technologies. This

contribution begins with an overview of sociotechnical theory and ethnography.

These theories concern interactions between technology, its use, people who use or

are affected by it, and their organizational and societal situations. Then the

contribution discusses planning and designing evaluations, including frameworks

and models to focus an evaluation, and methodological considerations for

conducting it. Next, ethical issues and further challenges and opportunities are

taken up. Concluding case examples, referenced throughout, illustrate how good

evaluations provide useful results to help design, implement, and use health

information technologies effectively.

Keywords. Evaluation studies, organizational studies, ethnography, medical

informatics, qualitative research, qualitative evaluation, organizational culture,

organizational case studies.

1 Introduction

Successful implementation involves interactions and mutual adjustments among an

information technology application and the organization, people, and practices where it

is used. Sociotechnical evaluation analyzes this interplay between technologies and

social and technical systems. It emphasizes how people, organizations, professions,

culture, work practice, ethical issues, social and political environment, and the like, all

interact and change each other over time. Sociotechnical analyses assess how

information technology and workflow influence each other; how clinical and patient

roles relate to technological use; how useful and usable health information technologies

are; and what consequences, patient safety issues, or user responses might occur. They

involve considering these interdependent elements as a holistic dynamic network rather

than as fixed pre-defined separate domains [1,2,3,4].

For example, Example 1 indicates that using images, and incorporating clinical

images into on-line electronic patient records, depends not only on the computer

system, but also on interwoven issues of expertise, trust and relationships among

colleagues, clinical knowledge of individual patients, institutional priorities, how

1 Corresponding author: Prof. Dr. Bonnie Kaplan, Yale Interdisciplinary Center for Bioethics, Box

208293, 238 Prospect Street, New Haven, CT 06511, United States, [email protected].

This article is published online with Open Access by IOS Press and distributed under the terms of the Creative Commons Attribution Non-Commercial License.

doi:10.3233/978-1-61499-635-4-114

114

conveniently system access fits into a busy and frequently interrupted day full of ad

hoc conversations, ways images are interpreted and their clinical meanings negotiated,

and other socio parts of sociotechnic. That is how sociotechnical systems work, and

how sociotechnical analyses can be helpful.

2 Sociotechnical Theory

Sociotechnical approaches incorporate theories and evidence from multiple disciplines.

Key theoretical features include examining technologies as they actually are used in

natural settings to investigate how technical and physical work settings affect their use;

how users negotiate, re-negotiate, interpret, and re-interpret features of the technology;

and relationships among the social and technical components of these emergent

processes as they unfold over time. The approaches are based on an understanding that

a new information technology and the social system where it is introduced change each

other as different parties pursue different goals [4,5]. These approaches are not

deterministic, nor do they understand technological development in terms of a rational,

linear sequence. Instead, they emphasize evolving processes and interactions so that no

factor acts in isolation from others, or has a uni-directional impact. They see processes

and causes interacting in multiple causal directions and relationships.

Sociotechnical principles developed as part of the Tavistock Institute’s post-WW

II analysis of British industries. They emphasized designing work for workers’ interests

and quality of working life [4]. By the 1990s, sociotechnical ideas had been introduced

into health informatics, as were social interactionist approaches – approaches that

consider relationships between system, individual, and organizational characteristics

and effects among them – which now would be labeled “sociotechnical” [1,2,3,6,7,8,9].

Sociotechnical theory in health informatics, then, has roots in traditional sociotechnical

research, ergonomics, social construction of technology, technology-in-practice, and

social informatics [5]. To these antecedents, I would add theories of change.

Informatics systems introduce change which may be welcome, or disruptive, to

the individual and the organization. Sociotechnical theory conceptualizes

organizational change as interacting components – for example, Leavitt’s well-known

diamond model of people, task, technology, and structure [10] – each responding to a

+change in any other so as to maintain organizational homeostasis, with the

interactions themselves being most important. Other theories of change based on the

foundational work of Rogers [11] and Lewin [12] characterize it as a dynamic process

that proceeds through stages involving multiple actors with different concerns and

perceptions of benefit. These actors include experts, sponsors, and people adopting (or

not adopting) the change. These actors are connected and communicate through various

social, organizational, social, and cultural channels. The change occurs, then, at

individual, group, organizational, and cultural levels. Any of the stages, actors, system

components, and units of analysis could be the focus of evaluation.

3 Ethnography

Ethnographic approaches explore how users experience health information technology

and why they interact with it as they do. They involve getting to know and

documenting the people and culture by spending time and participating in the setting

B. Kaplan / Evaluation of People, Social, and Organizational Issues 115

under study [13]. Ethnography expresses findings in terms meaningful to the people

involved. This enables people to recognize themselves and thereby makes those

findings more convincing and relevant. Ethnographic sociotechnical evaluation can

help prevent difficulties through better needs analysis, system design and

implementation practices, understanding what people do when working with the

technologies, and identifying why they view and use the technologies in those ways.

Ethnography involves starting with a sense of what to investigate and

progressively sharpening the investigation as more is learned. This is different from

beginning with immutable testable hypotheses, a priori research questions or

categories, and a pre-set research design. Instead, the study evolves and changes

according to what is learned as it proceeds. Because sociotechnical systems are

dynamic, freezing a research design before beginning may turn out to poorly match the

situation at hand as it develops. Ethnographic methods are particularly valuable in

natural, uncontrolled settings. They allow for adjusting a study in a fluid environment

where unanticipated findings emerge and situations change.

Methodologically, Examples 1 and 2 are ethnographic. Ethnographies tend to

emphasize the people involved and explore their situations. The main general

investigative questions are:

(1) What is happening here? (2) Why is it happening? (3) How has it come to happen in this way? (4) What do the people involved think is happening? (5) How are they responding to what is happening? (6) Why are they responding that way? [14]

The key question is “Why?”: Why are the people who are involved actually

involved; why do they think and react as they do; why do they use the technology as

they do; why are they interacting as they are; what meanings do they attribute to the

technology, health and disease, their roles, and what they do; and why those meanings?

To answer, ethnographic work uses open-ended evaluation questions, qualitative

data collection and analysis, interpretive and multi-level data analysis, a focus on the

lived experience and its meaning to those involved, emergent findings, and making

tacit knowledge and practice manifest. Because it enables a deep understanding of what

is going on, wiser decisions and actions may be based on those findings, and theoretical

insights may be developed [14].

In Example 2, sociotechnical approaches revealed emergent, unexpected findings

involving more general interrelationships between work and technology use. The

analysis reinforces the sociotechnical stance that the technology does not stand alone,

the social system (in this case, laboratory management, laboratory work, and hiring

practices) does not stand alone, but the two mutually affect and change each other. The

ethnographic approach enabled better understanding of how laboratory work was

understood. This resulted from resolving seemingly divergent findings from multiple

sources of data through an interpretation that accounted for all data, in this case, the

job-orientation model that relates how people think of their job to how they think about

computer systems introduced into their work. This rich result contributed to theory.

B. Kaplan / Evaluation of People, Social, and Organizational Issues116

4 Theory Development

Example 2 also exemplifies other theoretical points. The evaluation contributed to the

idea that “the same” system is not the same for all concerned, which also was found in

an evaluation of an automated telephone counseling system [15]. Similarly, “success”

may be defined and experienced differently among different groups and individuals at

different times [16]. Further, as also evident in studies that contributed to the idea of

the importance of fit between a technology and an organization, “fit” has to be

produced actively and changes over time [3,17].

The two examples contributed to another theoretical insight as well. The findings

inspired a framework helpful in future studies: the 4Cs of communication, care (or, if

outside of clinical institutions, whatever else the mission of the organization is),

control, and context [2,8,18]. In Example 1, on-line images improved communication

and care, raised control issues, and occurred in the different contexts of a government

and academic medical center. The laboratory information system in Example 2 also

improved communication and, therefore, care; highlighted control issues; and took

account of the context of different laboratories and technologists in the job-orientation

model. Frameworks like 4Cs can be useful for evaluation planning and design.

5 Planning and Designing Ethnographic Sociotechnical Evaluation

The multiplicity of interacting systems and sub-systems presents a wide range of

choice for how to design an evaluation. To choose among the possibilities, decisions

are needed concerning how to focus an evaluation, when to evaluate, and how to

evaluate.

5.1 What to Evaluate

To answer the key evaluation question of what is happening and why, it is hard to

know at the outset what of all the activity and who of all those involved will be

important. Theories, models, and frameworks can help to target what is most relevant

for the situation at hand. They provide a lens through which situations can be analyzed

and understood; highlight what is important; explain how various factors, influences,

and considerations interrelate; help organize and explain findings; and lead to

predictions for further investigation and planning. Their power comes from

emphasizing only some aspects of the area under study. Because each necessarily

leaves out aspects that may turn out to be important, it can be helpful to use more than

one theory, model, or framework. Sociotechnical evaluation lends itself to just that.

4Cs, discussed in Section 4, brings attention to issues of communication, care, control,

and context. Sitting and Singh’s model focuses on hardware and software; clinical

content; human-computer interface; people; workflow and communication;

organizational policies, procedures, and culture; external rules, regulations, and

pressures; and system measurement and monitoring [19]. An additional set of

evaluation questions, based on those of Anderson and Aydin [7], could be:

(1) Does the system work as designed? (2) Is it used as anticipated? (3) Does it produce the desired results?

B. Kaplan / Evaluation of People, Social, and Organizational Issues 117

(4) Does it work better than what it replaced? (5) Is it cost-effective? (6) How well have individuals been trained to use it? (7) What are changes in departmental interaction, delivery of care, patient safety,

control and power in the organization, or the healthcare system at large?

(8) How do the system and these changes relate to the practice setting?

Combining theories, models, or frameworks can help an evaluator choose potential

evaluation questions. What purpose the evaluation serves also is important when

choosing a focus. Table 1 gives some examples.

Table 1. How evaluation purpose can affect evaluation focus.

If the purpose of the evaluation is The evaluation could focus on

• Technical • System requirements • Economic • Cost/benefit • Clinical • Patient Care • Education • Students’ grades, learning outcomes • Research • Access to literature, data • Policy • Cost, utilization • Usefulness • User satisfaction, degree of use

Just as the system, the users, and the context interact and shape each other, the

evaluation context and environment affect how the study is conducted over time. These

include:

(1) purpose of the system, which may be for research and development, a demonstration project, or a commercial product;

(2) organizational commitment, which might be to continue, maintain, or quash the system, or to evaluate it;

(3) who the client is; (4) how evaluation results will be used; (5) budget and time frame; (6) evaluator skills and expertise; (7) who the research subjects are; and (8) the people who are involved.

Considerations about these people include:

(7) how the need for the system and for the evaluation was determined, and by whom;

(8) what needs the system and the evaluation meet, and whose needs they are; (9) who will be using the system, doing data entry, or receiving outputs; (10) what users’ attitudes towards the system, and towards the evaluation, are; (11) who was involved in needs assessment, design, and testing, and why those

where the people involved;

(12) whether potential users perceive a need for the system; (13) whose interests the system or the evaluation serves, or appears to others to

serve; and

(14) what different parties want to know.

B. Kaplan / Evaluation of People, Social, and Organizational Issues118

Knowing the environment and people involved can alert the evaluator to

considerations that should be examined further.

5.2 When to Evaluate

Sociotechnical ethnographic evaluations can be done at any stage, or multiple stages, of

system development or implementation. When to evaluate depends on the purpose of

the evaluation, as in the two examples. There is no need for concern that study results

or even conducting the study will affect the object of study. It will. A moving target is

assumed. Evaluation, then, can be used to influence needs assessment, analysis, design,

implementation, and how a system is used without “tainting” either the process or the

rigor of the study. In fact, it is wise to feed what is learned back into the process so that

it proceeds more smoothly.

5.3 How to Evaluate

Choosing methods depends on evaluation questions, evaluator skills and expertise, and

budget and time table. The theoretical underpinnings of sociotechnical approaches

suggest methods and research designs that are flexible and encourage emergent,

unexpected findings. Rather than the usual impact studies that characterize much

medical research – randomized controlled trials and experimental designs to test

hypotheses – interactionist (i.e. where subsystems and system components interact over

time) sociotechnical study designs are preferred. Table 2 indicates some ways impact

and interactionist studies differ.

Table 2. Differences between impact studies and interactionist studies.

Impact Interactionist

Epistemology Objectivist Objectivist or Subjectivist

Purpose Factors Process

Variance

Methods Quantitative Qualitative

Causality Uni-directional Multi-directional

Question What Why

Sociotechnic approaches examine how peoples’ practices are situated in their

environments and how the actors and technological change interact. These studies are

best done in situ using methods appropriate to naturalistic settings and changing

circumstances. Ethnographic sociotechnical evaluation is interactive not only in

examining interactions among the social and technical components of the system under

study, but also among components of research design. What should be studied and

what the research questions are depends on the purposes, methods, conceptual

concepts, and validity issues involved, and each of these shapes the others [20]. Study

design, then, should be longitudinal, modifiable, and flexible over time. Because

evaluation can help direct a project, it can be both formative and summative, and

should focus on a variety of concerns reflecting the various actors involved. Employing

multiple methods is beneficial because different data sources provide different data

[18]. Different informants may have different focuses, report processes that are

different from what the evaluator observes, and behave differently from the way they

indicate on surveys or in laboratory settings. The challenge is to make sense of these

B. Kaplan / Evaluation of People, Social, and Organizational Issues 119

differences. If data do not converge, a richer understanding develops through

accounting for apparent contradictions, as in the laboratory information system study

(Example 2). Multiple methods and data sources lead to robust results.

Qualitative methods were used in the two examples. They are especially valuable

for sociotechnical ethnographic evaluation. Data collection methods include participant

observation; observation; unstructured or semi-structured interviews; focus groups;

surveys with open-ended questions; analysis of artifacts like documents, images, texts,

or drawings; and the researcher’s own impressions and reactions. Analysis methods

include coding, contextual or narrative analysis, analytic memos, and displays. Data

analysis involves constantly integrating and analyzing voluminous, mostly textual, data

from multiple sources. Interpretations and hypotheses continually are formulated,

tested, and verified or discarded through a process of on-going data analysis and

writing that assesses whether they make sense in light of existing and future data. What

seems most interesting, relevant, or important progressively becomes clearer [14].

Qualitative data analysis software is a boon to managing and analyzing the

volumes of data an evaluation study produces, but it does not do the analysis per se.

The evaluator still needs to figure out how to interpret data. It helps in this process to

focus on:

(1) how people use words and what they mean by them—what is meant by “work” in the laboratory (Example 2) or “see[ing] what’s really going on” in

an image (Example 1);

(2) what people say and do, and under which circumstances they say and do it— how the clinicians in the second imaging study (Example 2) negotiated what

images meant;

(3) how people justify or give reasons for what they say, do, believe, etc.— comments laboratory technologists wrote about why the new system was a

“hassle” or improved reporting (Example 2);

(4) what does not seem to make sense (the puzzles)—how a laboratory technologist’s job does not change when the technologists’ tasks change

(Example 2); and

(5) how to make sense of all the data.

Focusing this way helps produce evaluations that get at what it means to the people

involved to use health information technologies. Paying close attention to who the

people are, what they think, what they do in real-life settings, and how they differ,

helps explain how all that interacts with health information technology development,

implementation, adoption, and use – in other words, how the social and technical

subsystems interact. The end result, then, goes well beyond simply reporting data. It

requires solving puzzles by accounting for all data in a way that focuses on what the

technology means to the participants, why it means that, and what the implications are.

Explaining the data in this way helps make tacit knowledge, assumptions, meanings,

and values explicit, so they can be taken into account. It tells a coherent, compelling

story that is useful, and makes theoretical contributions by both drawing on theory to

produce an interpretation and also, as in the examples, possibly develop new theory.

5.4 How to Validate Evaluation Results

Qualitative researchers collect rich data and produce intepretations that account for it

all through a process known as triangulation. Particular attempts are made to collect

B. Kaplan / Evaluation of People, Social, and Organizational Issues120

data that may contradict the developing interpretation. Data is continually collected

until no new information seems to be possible, which is known as reaching saturation.

The people involved in the study are asked for feedback and for their responses to the

developing interpretation in a process known as member-checking, and what they say

becomes new data [14]. A neutral partner can review data and how it is interpreted.

Similarly, research team members can test each other’s ideas, methods, and

interpretations. Eventually, reviewers and other researchers judge the work, just as in

any other form of research. Reproducability is impossible; every situation, evaluator,

and study is different. The goal is transferability, so that significant insights can be

developed, theoretical contributions can be made, and the knowledge gained can be

applied elsewhere.

6 Sociotechnical Ethnographic Evaluation Research Ethics

Evaluators face ethical decisions even before beginning an evaluation and thereafter. In

addition to usual research ethics issues, additional concerns arise in sociotechnical

ethnographic evaluation. A few of them are mentioned here. Special considerations

involve informed consent, privacy and confidentiality, social justice, practitioner

research, power, reciprocity, relevance, and how the research is used [21].

As in other fieldwork, interpersonal relationships develop between evaluators and

participants, raising questions of just what those relationships should be. The evaluator

may be privy to material that those involved did not give consent for or see people who

were not asked for permission. When a new technology is introduced it is hard to

anticipate how people will react, making consent even more problematic [22]. The

evaluator may observe what could be unethical behavior, or be asked to engage in

behavior that some may consider unethical. A sociotechnical viewpoint involves

sensitivity to ethical questions like who defines, and should define, the evaluation

questions, interpretation, and use of results, and whose interests are served by the

evaluation. The evaluation, too, likely will involve the goals, values, and assumptions

incorporated into the technology, how it is implemented, how people are expected to

use it, and effects expected from it, also raising ethical concerns.

7 Future Challenges and Opportunities

To date, evaluation mostly concerns visible, tangible health information technologies in

physical settings. Newer developments—virtual health care delivery, distributed

integrated health care organizations, virtual workers, fluid organizational boundaries,

social networks, telehealth and mobile health applications, avatars and artificial

intelligence—make in situ studies more difficult, especially if health care delivery

crosses jurisdictional boundaries. Adding to the complexity is the need for multi-site

studies that include community, home, or other non-academic locations with

geographic or national variation. There is room for sociotechnical evaluation study

designs and methods that address these challenges while also contributing to much-

needed methods to assess patient outcomes better [3].

B. Kaplan / Evaluation of People, Social, and Organizational Issues 121

8 Conclusion

Sociotechnical ethnographic evaluation focuses primarily on the people in addition to

the technology. Contributors to system “success” are sociotechnical. By focusing on

technologies as they actually are used, in the settings in which they are used, and seeing

how people negotiate and reinterpret the technologies as the social and technical

systems interact with each other, sociotechnical ethnographic evaluation can contribute

to theory and practice while improving health information technologies and patient

care.

Example 1 – Clinical Imaging Systems

Administrators and clinicians differed about the value of a new system that integrated

patient record textual, numeric, and image data [23]. This raised an administrative

control issue concerning decisions about continued development. Also, previously the

department where an image was produced kept the image, but now images were

available to all, which potentially created another control issue.

In a week of interviews and observations, we investigated what clinicians thought

about the benefits of the system. Clinicians told us that having the images available as

part of the on-line patient record improved communication and consultation, so

improved clinical decisions, and hence, patient care. Because “a written report won’t

convey everything,” and “you don’t know [if the report] is an accurate description,”

now, clinicians said, they “know what’s there,” they could “look through a patient and

see,” “see what’s really going on.” That way, they did not need to repeat procedures.

They could plan treatment better and give students “real” experience through these

images.

Elsewhere, I spent a week shadowing a physician as he performed his daily

activities. The purpose was to identify how clinical images are used in an academic

medical center planning to develop a stand-alone imaging system [24]. The physician

was interrupted constantly, frequently telephoned for information, talked about patients

with other doctors he met fortuitously on the stairway, and consulted with Pathology

and Cytology after receiving reports that slides were “not diagnostic,” or “inadequate

for evaluation.” The person reading the slides told the physician that he had a “gut

feeling” that the cells indicated cancer, though “quantitatively it was a little short” and

showed him why. At his weekly radiology conference, they discussed each patient’s

images, asking each other about the patient and what they saw, or thought they saw, on

the image. For the physician I shadowed, mutually viewing images was improving

communication and clinical decision making, and seeing the images was better than

reading a report. However, reading an image was not a matter of “see [ing]what’s really

going on,” but of interpreting the image in light of expertise and experience, clinical

knowledge of that particular patient, and discussing all that.

In these studies, clinicians thought of the benefits of viewing images as a whole,

not as a separate part of patient care. They thought having those images improved care

and decision making. They considered the images objective, talking of them as

showing, all by themselves, what was “really going on.” Yet, the studies indicate that

what an image means and what clinical decisions should be based on it depends on far

more than simply having the image. In these evaluations, the meanings of those images

were being negotiated through collegial interchanges, though neither clinicians nor

B. Kaplan / Evaluation of People, Social, and Organizational Issues122

system developers acknowledged it. Even though the same could be said about paper

and film-based images, health information technologies often are premised on a belief

that providing information alone is enough because it speaks for itself. This belief

affects design, implementation, and use.

Example 2 – Clinical Laboratory System

We investigated the impact of a new system on laboratory work in a longitudinal study

ranging from pre- to post-implementation. More in line with a sociotechnical

ethnographic approach, we also sought to identify what happens when an academic

medical center converts from a manual to automated system for ordering clinical

laboratory tests and reporting test results [25]. The study included interviews,

observations, participant observation, and surveys.

Technologists’ responses to scaled-response survey questions indicated no change

in laboratory work. Nevertheless, it was clear from their comments in open-ended

questions that work was changing. Some technologists reported being “happy” because

of fewer abusive telephone calls. They also liked the more legible, timely, and

complete laboratory reports. Others, instead, reported on the “hassle” of having to

interrupt their work to enter test results into the computer. We realized that the first

group of technologists thought of their job as providing laboratory test results, an

outcome- or product-oriented view of laboratory work. The other group of

technologists thought of their work as doing laboratory tests, a more process-oriented

view in which they saw the new computer system as a “hassle” that took them away

from the laboratory bench. This job-orientation model applied not only to individual

technologists, but also to the fit between system and different laboratories. The same

laboratory information system used in all the laboratories was not “the same” for

everyone, nor even every laboratory. Instead, it was viewed differently in ways that

related to job orientation. Moreover, it was apparent that being able to work with the

computer system was a new criterion for being a laboratory technologist.

The findings can be reported in terms of improving communication between the

laboratories and the clinicians by producing better and more timely laboratory reports,

thereby improving care. Laboratory technologists fielded fewer telephone calls asking

for laboratory results. Control issues arose over laboratory work, and the different

context of each laboratory was related to how technologists viewed the new system. In

particular, how the laboratories, as well as individual technologists and laboratory

directors, saw the nature of laboratory work was key to understanding their reactions.

In interviews, directors had told us that the new system would not change

technologists’ jobs. If they had realized that there were different views of laboratory

work, that laboratory work was now different, and that these differences would matter

in how technologists and laboratories related to the new system, they could have

prepared staff better.

B. Kaplan / Evaluation of People, Social, and Organizational Issues 123