Ethics of Engineering Presentation

Ethical Engineering Problems

Safety being an afterthought[5][6]

Removed hardware locks and depended on the software to monitor the status of the machine, and dosage. The Hardware locks were previously installed in the Therac-6 & 20 to prevent the operators from doing something dangerous, if an invalid mode was attempted the hardware locks would bring everything to a halt. By tasking the software to position everything and detect dangerous situations, the operators saved a lot of time. [5][6]

Four bugs as well as other potential bugs were found in the Therac-25 which would cause radiation overdose and were later found in the Therac-20, however no accidents occured because of its hardware locks.[8][9]

When AECL was made aware of the situation after the first accident, they responded three days later stating that it was impossible for the Therac-25 to operate in electron mode without scanning to spread the beam. [5][6]

Causes For Disaster

Therac-25 software the difference being that the Therac-20 software was limited and served as a convenience to the hardware. [5][6]

Software controlled many of the critical safety checks, and the hardware locks were removed.

Therac-25 software was also redesigned so the operators could enter the treatment plan once and just verify, as well as some other convenient features. [5][6]

Manual did not clearly state what any of the Malfunction errors meant. [5][6]

Operators ignored the error codes. [5][6]

Patients felt a burning sensation each time, during the 5th accident “the intercom was working and the operator heard a loud noise followed by moaning from the patient.” [6]

Out of the 6 accidents, two came from the came from East Texas Cancer Center, the same operator made a mistake entering the modes but quickly fixing the error. Similar to the first accident from that Center the machine shut down with a “Malfunction 54” message. Each time the patients were sent to the Therac-25 treatment, the operator witnessed an error message. The operators were used to “the quirks in the machine” and continued after each error message not knowing they were delivering more radiation to the patient. It was later found that quickly changing modes” resulted in radiation overdose.” [5][6]

Decision Making Behind the Disaster

By entrusting the software alone and removing any secondary safety controls, the AECLs was overconfident that this machine was incapable of errors. After the second accident they sent a service engineer to investigate. The engineer could reproduce the malfunction message however the engineers suspected the problem was being caused by a microswitch used to determine the turntable position. They claimed to have “increased the safety of the machine by, at least five orders of magnitude, but did not discover why the accidents occurred. [4][6]

How Disaster Continued

The AECL denied that the Therac-25 was capable of the delivering an overdose because of the frequent error codes and paused treatment during the medical procedure.

The only way a patient would suffer from an overdose is if the turntable was in the wrong position; this would cause the patient to be hit by a laser-like beam.

AECL was unable to reproduce the error so the only possible cause for this error was a temporary failure in the three microswitches that could cause the shift the turntable’s position.

A talented physicist Fritz Hager was able to diagnose the problem. The error was known as “Malfunction 54” and once he was able to produce the error on command they were able to correct the applications code. [2][7]

What Went Wrong

Overconfidence in the software led to removing the hardware interlocks making the software responsible for any accidents. Investigations only looked at hardware for malfunctions.

Unacceptable software engineering practices, there were few specifications and documentation, and few software tests.

The software did not have any self-checks or other error detection features.

Not enough investigation on the reported accidents. If the manufacturer would have thoroughly investigated the cause of the first overdose, they could have prevented the other accidents.

A lack of communication between the AECL and the engineers responsible for development of the new machine, was a major issue that led to the overdoses of patients.

In April 1986, the AECL filed a medical device report with the FDA, notifying them of the cause of the two Tyler Hopital incidents. The FDA deemed the Therac-25 defective, and the AECL’s response was a temporary fix to the device’s software.

In February 1987, the FDA finally required all units of the Therac-25 to be shut down until permanent modifications were made.

The amount of time the FDA and the AECL took to consider these software problems an issue for patients, is indicative of a slow response by governing institutions to protect patient health. [1][8]

Ethical Errors

When building the Therac-25 the designers removed critical safety features in favor of a software only design.

Not all errors and bugs were found when testing and designing the Therac-25 resulting in the operator manual not having solutions to many error messages.

Technicians would operate the Therac-25 machines and would often ignore error messages and continue their procedures ultimately resulting in overdosing a patient.

After multiple cases of patients dying, reports were finally made, but the Therac-25 remained operational for multiple years.

Ethical Course of Action

Once Fritz Hager and other physicists could produce the same error, the software for the Therac-25 was later investigated and reviewed. The computer was redesigned to handle real-time control of the machine, both its normal operation and safety system.

Today this could be handled by using one or two microcontrollers with a PC running a GUI front end.

The knowledge of specific incidences between patients exposed to these machines were completely ignored by governing agencies responsible for protecting patients.

Problematic results could have been mitigated by effective communication between the FDA and AECL, to protect patients’ health. [3][7]