How Complex Systems Fail

By | May 13, 2010

The paper How Complex Systems Fail by Richard Cook should be required reading for anyone in programming or operations. Hell, it should be required reading for most everyone. You should read the whole paper (it’s very short at under five pages), but here are the main points:

  1. Complex systems are intrinsically hazardous systems.
  2. Complex systems are heavily and successfully defended against failure.
  3. Catastrophe requires multiple failures – single point failures are not enough.
  4. Complex systems contain changing mixtures of failures latent within them.
  5. Complex systems run in degraded mode.
  6. Catastrophe is always just around the corner.
  7. Post-accident attribution accident to a ‘root cause’ is fundamentally wrong.
  8. Hindsight biases post-accident assessments of human performance.
  9. Human operators have dual roles: as producers & as defenders against failure.
  10. All practitioner actions are gambles.
  11. Actions at the sharp end resolve all ambiguity.
  12. Human practitioners are the adaptable element of complex systems.
  13. Human expertise in complex systems is constantly changing
  14. Change introduces new forms of failure.
  15. Views of ‘cause’ limit the effectiveness of defenses against future events.
  16. Safety is a characteristic of systems and not of their components.
  17. People continuously create safety.
  18. Failure free operations require experience with failure.

Points 2 and 17 are especially interesting to me. It’s not that people can build complex systems that work; instead, it is that people have to actively prevent system failure. Once you stop maintaining the system, it begins to fail. It sounds like thermodynamics; without a constant input of energy from people, the mostly-orderly complex system will descend into increasing disorder and failure.

7 thoughts on “How Complex Systems Fail

  1. Pingback: Tweets that mention Paul M. Jones » Blog Archive » How Complex Systems Fail -- Topsy.com

  2. Bill Karwin

    My wife has a saying: “systems don’t work; people work.” What she means is that no system is self-perpetuating. Systems require people participating in them to cooperate and actively support the system’s intent.

    Reply
  3. Pingback: Webs Developer » Paul Jones’ Blog: How Complex Systems Fail

  4. Pingback: Paul M. Jones » Blog Archive » Disaster Rituals

  5. Pingback: Publish Your Failures; or, The Way Of All Frameworks | Paul M. Jones

Leave a Reply

Your email address will not be published. Required fields are marked *