by Charlie Hoes
 

I came across an interesting problem the other day while working on a slightly complex chemical distribution system. The system uses computer-controlled valves to mix streams of various reactive and toxic chemicals in support of a manufacturing process. The chemicals are flammable pyrophoric oxidizers, highly toxic and generally incompatible with each other. They share common distribution lines to the reaction chamber, where the chemicals mix under controlled conditions to create the desired final product. The incompatible chemicals are routed through two separate lines (one for oxidizers and the other for fuels) until they’re close to the reaction chamber, where the lines combine into a single input line. The waste stream from the chamber is routed to a series of treatment facilities, depending on which type of chemical is being used. The mixture of chemicals to and through the reaction chamber is varied with time. Oxidizers and fuels are not introduced at the same time; instead, they react sequentially.

Because the chemicals are separated into oxidizers and flammables that don’t flow at the same time, it’s practical to design a hardware-based interlock system that prevents them from flowing simultaneously. It’s also possible to design the waste streams to prevent mixing of the two categories in the shared lines to the treatment facilities. This has been done in an attempt to make safety depend on the functioning of hardware-based sensors and relays rather than on software, mainly because it’s much simpler to analyze and test hardware-based, discrete devices than the software controls. The economics of the situation make the cost of implementing safety in the software controls prohibitively expensive.

This approach works fine for normal operations but turns out to have a bit of a hiccup during maintenance.
 

"... the valve icon on the screen didn't really represent the state of the valve — it represented the state of the software."


During some maintenance and/or repair operations, it’s necessary to open the lines in the chemical distribution system to replace parts such as valves and flow controllers. The piping and devices are, of course, full of nasty chemicals that need to be removed prior to opening the piping system. This is done using a process of draining the lines and purging them with a non-reactive gas (usually nitrogen). Since it’s generally not possible to effectively purge chemicals by simply flowing an inerting gas through the lines, a cycle purge process is used whereby the pressures are reduced, inert gas is introduced and pumped through the system, the pressure is again reduced to a very low level, more clean gas is introduced, etc. This approach effectively dilutes the hazardous gases and vapors, sweeping them out as the purge gases flow. Depending upon the situation, the system might be cycled anywhere from 10 to 100 times before it’s safe to open the piping system.

An interesting twist occurs during this process. Since the valves are all computer-controlled, there’s no way to manually open and close them or reduce the pressure to the system. It can only be done by automatically or “manually” controlling the valves through the computer system. Since this purge cycle is necessary for safety, there goes our plan to separate safety from the software — safety is directly dependent upon the software system.

Then there was a bit of an accident! Luckily, it was a “close call” rather than an injury-producing event — however, it was enough to attract attention and focus a few more resources on the chemical distribution control system. The “accident” involved replacing one of the valves. The technician had done a “manual” cycle purge using the controls on the screen at the control panel. When he opened the pipeline, there was still some pyrophoric material in the line, which flashed and popped, giving the person doing the job a bit of a surprise as well as some minor burns.