by Nancy Leveson, Aeronautics and Astronautics/Engineering Systems, MIT

Pages 1 | 2 | 3 | 4 | 5

Conclusion

To avoid confirmation bias and compliance-only exercises, assurance cases should focus not on showing that the system is safe, but on attempting to show that it is unsafe. It is the emphasis and focus on identifying hazards and flaws in the system that provide the "value-added" of system safety engineering. System engineers have already created arguments for why their design is safe. The effectiveness in finding safety flaws by system safety engineers has usually resulted from the application of an opposite mindset from that of the developers.

Whatever is included in the assurance case, the following characteristics are important:

• The process should be started early. The assurance case is only useful if it can influence design decisions. That means it should not be done after a design is completed or prepared in isolation from the system engineering effort. If safety cases are created only to argue that what already exists is safe, then the effort will not improve safety and becomes simply a paper exercise to get a system certified. One result might be unjustified complacency by those operating and using the systems.
• The assumptions underlying the assurance case should be continually monitored during operations, and procedures should be established to accomplish this goal. The system may be working, but not the way it was designed, or the assumptions may turn out to be wrong, because of poor prediction or because the environment has changed. Changes to the system and its environment may have been made for all the right reasons, but the drift between the system as designed and the system as enacted is rarely if ever analyzed or understood as a whole, even when each particular deviation appears sensible or even helpful to the individuals involved.
• To make maintaining the assurance case practical, the analysis needs to be integrated into system engineering and system documentation so that it can be maintained and updated. Safety assurance is not just a one-time activity; it must continue through the lifetime of the system, including checking during operations that the assumptions made in the assurance argument remain true for the system components and the system environment. The problems in updating and maintaining safety assurance do not arise from the form of the assurance documentation or in updating the argument once the need for it is established; rather, problems arise in relating the assurance case to the detailed design decisions so that when a design is changed, it is possible to determine what assumptions in the safety analysis are involved.
• The analysis should consider worst cases, not just likely or expected cases (called a design basis accident in nuclear power plant regulation).
• The analysis needs to include all factors, and be comprehensive. It should include not just hardware failures and operator errors, but also management structure and decision-making. It must also consider operations, and the updating process must not be limited to development and certification, but must continue through the operational part of the system lifecycle.
• To be most useful, qualitative and verifiable quantitative information must be used, not just probabilistic models of the system
• The integrated system must be considered, rather than considering each hazard or component in isolation

« previous page