VIEW
Incorrect Assumptions
The redundant components may not fail together. One component fails, but the failure has minimal impact because a backup system immediately takes over. Feedback about the failure is not enabled, does not exist, or it exists but the component is not repaired within X time as assumed. There is poor or nonexistent feedback to management to recognize when actual time to repair exceeds the value assumed by engineers. The system continues operating normally until the redundant component eventually fails.
SOLUTION
Design the control structure to ensure explicit alerts to maintenance. Design leadership feedback to evaluate when the assumed time to repair is exceeded in operation. Define required responses at each level, including operators, maintenance, and leadership, when the redundant path is not repaired within a validated time limit. For example, certain operations may be halted or additional limits imposed until a full repair is completed.
REAL-WORLD EXAMPLE
The Deepwater Horizon Oil Spill involved two redundant control systems on the ocean floor: the Blue Pod and the Yellow Pod. Only one was needed to function, and the system was designed so that if one failed, the other could take over. However, there was no “hard” control that halted drilling if one pod went offline. Months before the explosion, the Blue Pod failed due to a dead battery. The Yellow Pod took over seamlessly. The rig continued to operate “normally”.
While there was some technical feedback that the Blue Pod was struggling, it wasn’t treated as a “stop-work” event. Management and leadership did not have a feedback loop to recognize that the Mean Time to Repair (MTTR) had been exceeded by weeks. On the day of the disaster, the Yellow Pod (the only one left) failed due to a dead battery. Even though the two pods did not fail simultaneously, the lack of timely repair of the blue pod made an accident inevitable.