|
|
|
Vol. 47, No. 6 • November-December 2011 |
|
In the Spotlight
|
|
The Use of Safety Cases in Certification and Regulation
|
|
by Nancy Leveson, Aeronautics and Astronautics/Engineering Systems, MIT
|
|
Pages
1 |
2 |
3 |
4 |
5
|
Potential Limitations of Safety Cases
A "safety case" has been defined in many ways. In this paper, the term is used to denote an argument that the system will be acceptably safe in a given operating context. The problem is that it is always possible to find or produce evidence that something is safe. Unlike proving a theorem using mathematics (where the system is essentially "complete" and "closed," i.e., based on definitions, theorems and axioms, and nothing else), a safety analysis is performed on an engineered and often social system where there is no complete mathematical theory to base arguments and guarantee completeness.1
The main problem lies in psychology and the notion of a mindset or frame of reference:
"In decision theory and general systems theory, a mindset is a set of assumptions, methods or notations held by one or more people or groups of people which is so established that it creates a powerful incentive within these people or groups to continue to adopt or accept prior behaviors, choices, or tools. This phenomenon of cognitive bias is also sometimes described as mental inertia, groupthink, or a paradigm, and it is often difficult to counteract its effects upon analysis and decision-making processes." [Ref. 22]
An important component of mindset is the concept of confirmation bias. Confirmation bias is a tendency for people to favor information that confirms their preconceptions or hypotheses, regardless of whether the information is true. People will focus on and interpret evidence in a way that confirms the goal they have set for themselves. If the goal is to prove the system is safe, they will focus on the evidence that shows it is safe and create an argument for safety. If the goal is to show the system is unsafe, the evidence used and the interpretation of available evidence will be quite different. People also tend to interpret ambiguous evidence as supporting their existing position [Ref. 3].
Experiments have repeatedly found that people tend to test hypotheses in a one-sided way, by searching for evidence consistent with the hypothesis they hold at a given time [Refs. 10 and 13]. Rather than searching through all the relevant evidence, they ask questions that are phrased so that an affirmative answer supports their hypothesis. A related aspect is the tendency for people to focus on one possibility and ignore alternatives. In combination with other effects, this one-sided strategy can obviously bias the conclusions that are reached.
 |
|
An important component of mindset is the concept of confirmation bias. Confirmation bias is a tendency for people to favor information that confirms their preconceptions or hypotheses, regardless of whether the information is true. People will focus on and interpret evidence in a way that confirms the goal they have set for themselves.
|
 |
|
Confirmation biases are not limited to the collection of evidence. The specification of the information is also critical. Fischoff, Slavin and Lichtenstein conducted an experiment in which information was left out of fault trees. Both novices and experts failed to use the omitted information in their arguments, even though the experts could be expected to be aware of this information. Fischoff et al attributed the results to an "out of sight, out of mind" phenomenon [Ref. 4]. In related experiments, an incomplete problem representation actually impaired performance because the subjects tended to rely on it as a comprehensive and truthful representation — they failed to consider important factors omitted from the specification. Thus, being provided with an incomplete problem representation (argument) can actually lead to worse performance than having no representation at all [Ref. 20].
These problems are not easy to eliminate, but they can be reduced by changing the goal. The author's company was recently hired to conduct a non-advocate safety assessment of the new U.S. Missile Defense system for the hazard "inadvertent launch," which was the major concern at the time [Ref. 15]. The system safety engineers conducting the independent safety assessment did not try to demonstrate that the system was safe; everyone was already convinced of that, and they were prepared to deploy the system based on that belief. The developers thought they had done everything they could to make it safe, and had constructed a "safety case" argument during development that justified their belief in its safety. By law, however, the government was required to perform an independent risk analysis before deployment and field testing would be allowed. The goal of the independent assessment was to show that there were scenarios in which inadvertent launch could occur, not to show that the system was safe. The analysis found numerous such scenarios that had to be fixed before the system could be deployed, resulting in a six-month delay for the Missile Defense Agency and expenditure of a large amount of money to fix the design flaws. The difference in results was partly due to the use of a new, more powerful analysis method, but also involved a different mindset and a different goal, which was to identify unrecognized hazards rather than to argue that the system was safe (that inadvertent launch could not occur).
Engineers always try to build safe systems and to verify to themselves that those systems will be safe. System safety engineering adds value by taking the opposite approach: to show that the system is unsafe. Otherwise, safety assurance becomes simply a paper exercise that repeats what the engineers are most likely to have already considered. It is for exactly this reason that Haddon-Cave recommended in the Nimrod accident report that safety cases should be relabeled "risk cases" and the goal should be "to demonstrate that the major hazards of the installation and the risks to personnel therein have been identified and appropriate controls provided" [Ref. 5] — not to argue that the system is safe.
A final potential problem with safety cases, which has been criticized in the offshore oil industry approach to safety cases and with respect to the Deepwater Horizon accident (and was also involved in the Fukushima Daichi nuclear power plant events), is not using worst-case analysis [Ref. 8]. The analysis is often limited to what is likely or expected, and doesn't include what could be catastrophic. Simply arguing that the most likely case will be safe is not adequate: Most accidents involve unlikely events, often because of wrong assumptions about what is likely to happen and about how the system will operate or be operated in practice. Effective safety analysis requires considering worst cases.
But while theoretical arguments against safety cases are interesting, the proof is really "in the pudding." How well have they worked in practice?
« previous page | next page »
1
Even with such a mathematical basis, published and widely accepted mathematical proofs are frequently found later to be incorrect.
|
|
|
|
|
