|
Assurance Cases
Often, certification is a one-time activity that follows the development process and occurs before the product or system is allowed to be marketed or used. For complex systems, such as aircraft and nuclear power plants, certification may involve both initial approval and oversight of the operational use of the system. Changes to the original system design and certification basis may require re-certification activities.
All certification is based on "arguments" that the certification approach has been followed. Inspection and test may be used if the certification is based on following a product standard. If the certification is based on the process used, engineering artifacts or analyses may be required and reviewed. Performance-based regulation may require a particular type of analysis (such as the use of specific types of probabilistic risk assessment), or may allow any type of reasoning that supports having achieved a particular performance goal.
As an example, the U.S. Department of Defense (DoD) in MIL-STD-882 [Ref. 18] uses a prescriptive process that details the steps that must be taken in the development of safety-critical systems to ensure that they are safe. The purpose of the safety assessment report (SAR), which is used as the basis for certification, is to describe the results of the prescribed steps in the standard. The SAR contains the artifacts of the prescribed process, such as a Safety Plan (which must be approved by the DoD at the beginning of the development of the system), a Preliminary Hazard Analysis, a System Hazard Analysis, a Subsystem Hazard Analysis, an Operating System Hazard Analysis, etc. The DoD evaluates the quality of the process artifacts provided in the SAR as the basis for approving use of the system.
While NASA has recently been influenced by the nuclear power community emphasis on probabilistic risk analysis, traditionally it has taken — and continues to emphasize — an approach similar to that of the U.S. DoD. The U.S. Federal Aviation Authority (FAA) approach to civil aviation has also been overwhelmingly prescriptive, and the initial certification has been based on the quality of the prescribed process used to develop the aircraft and the implementation of various airworthiness standards in the aircraft's design. Operational oversight is based on inspection, as well as feedback about the safety of the operations process. Recently, the FAA has moved to create a requirement for a safety management system by those developing or operating aviation systems to shift more of the responsibility for safety to the airframe manufacturers and airlines.
The type of evidence required and assurance arguments used are straightforward with prescriptive regulation, but performance-based regulation requires a more complex argument and evaluation strategy. While the term "safety case" may be used in prescriptive regulation, it is more commonly used in a performance or goal-based regulatory regime.
Performance-Based Regulation and Safety Cases
Government oversight of safety in England started after the Flixborough explosion in 1974, but the term safety case seems to have emerged from a report by Lord Cullen on the Piper Alpha disaster in the offshore oil and gas industry in 1988, where 167 people died. The Cullen report on the Piper Alpha loss, published in 1990, was scathing in its assessment of the state of safety in the industry [Ref. 2]. The Cullen report concluded that safety assurance activities in the offshore oil industry were:
- Too superficial
- Too restrictive or poorly scoped
- Too generic
- Overly mechanistic
- Insufficiently appreciative of human factors
- Carried out by managers who lacked key competences
- Applied by managers who lacked understanding
- Inconsiderate of interactions between people, components and systems
The report suggested that regulation should be based around "goal setting," which would require that stated objectives be met, rather than prescribe detailed measures to be taken [Ref. 21], i.e., performance-based rather than prescriptive. In such a regime, responsibility for controlling risks shifted from the government to those who create and manage hazardous systems in the form of self-regulation. This approach has been adopted by the British Health and Safety Executive and has been applied widely to industries in that country.
 |
|
The type of evidence required and assurance arguments used are straightforward with prescriptive regulation, but performance-based regulation requires a more complex argument and evaluation strategy.
|
 |
|
The British safety case philosophy is based on three principles [Refs. 9 and 17]:
- Those who create risks are responsible for controlling those risks
- Safe operations are achieved by setting and achieving goals, rather than by following prescriptive rules. While the government sets goals, operators develop what they consider to be appropriate methods to achieve those goals. It is up to the managers, technical experts and operations/maintenance personnel to determine how accidents should be avoided
- All risks must be reduced so that they are below a specified threshold of acceptability
When performance-based or goal-based certification is used, there are differences in how the performance or goals are specified and how the evaluation will be performed. In 1974, the creation of the Health and Safety Executive (HSE) was based on the principle that safety management is a matter of balancing the benefits from undertaking an activity and protecting those who might be affected by it — essentially cost-benefit analysis (CBA). The HSE also instituted the related concept of ALARP or "as low as reasonably practicable," and widely used probabilistic risk analysis as the basis for these goals. Each of these is controversial.
The nuclear power industry was probably the first to use probabilistic risk analysis as a basis for certification. In the U.K., the Nuclear Installations Act of 1965 required covered facilities to create and maintain a safety case to obtain a license to operate. The nuclear industry has placed particular emphasis on the use of Probabilistic Risk Assessment (PRA) with the use of techniques such as Fault Tree and Event Tree Analysis. Because of the use of standard designs in the nuclear power community and slow introduction of new technology and innovation in designs, historical failure rates are often determinable.
Other potentially high-risk industries, such as the U.S. nuclear submarine community, take the opposite approach. For example, SUBSAFE does not allow the use of PRA [Ref. 12]. Instead, SUBSAFE requires Objective Quality Evidence (OQE), which may be qualitative or quantitative, but must be based on observations, measurements or tests that can be verified. Probabilistic risk assessments for most systems, particularly complex systems, cannot be verified.
A second unique aspect of the British approach to safety assurance required by the HSE is argumentation and approval based on whether risks have been reduced as low as is reasonably practicable (ALARP). Evaluating ALARP involves an assessment of the risk to be avoided, an assessment of the sacrifice (in money, time and trouble) involved in taking measures to avoid that risk, and a comparison of the two. The assumed level of risk in any activity or system determines how rigorous, exhaustive and transparent the risk analysis effort has been. "The greater the initial level of risk under consideration, the greater the degree of rigor required to demonstrate that risks have been reduced so far as is reasonably practicable." [Ref. 7].
The application of ALARP to new systems, where "reasonably practicable" has not yet been defined, is questionable. Not increasing the accident rate in civil aviation above what it is today seems like a reasonable goal, given the current low rate, for example, but it is not clear how such an evaluation could be performed for new technologies (such as satellite navigation and intensive use of computers), or the new and different procedures that are planned. There are also ethical and moral questions about the acceptance of the cost-benefit analysis underlying the ALARP principle.
While none of these more controversial aspects of assurance and certification need to be present when using a "safety case" approach, they are part and parcel of the history and foundation of safety cases and performance-based regulation.
« previous page | next page »
|
