Operator failure in abstract systems is ineradicable because humans err

Evidence

A second factor is what we might call operator failure. Any abstract system, no matter how well designed it is, can fail to work as it is supposed to do because those who operate it make mistakes. This also applies both to social and natural systems. Unlike design faults, operator failure appears to be ineradicable. Good design can make the possibility of operator failure very low, and so can rigorous training and discipline; but so long as human beings are involved, the risk must be there. In the case of the Chernobyl incident, the root cause of the disaster was a mistake made in the operating of the emergency shutdown systems. — Anthony Giddens, The Consequences of Modernity, 361

Categorization

• XREF: Operator failure is the human error term that expertise and training in expert systems cannot eliminate — precisely the gap where ‘faceless commitment’ becomes risky for someone like Stevens. — connects to: An agent should plan around its own future errors, not ideal execution — Brian Christian, The Alignment Problem, 1031; Poor performance of control systems can prevent lethal outcomes — Brian Christian, The Alignment Problem, 1368; Responsibility arises only in undecidable space that calculable programs try to close — Douglas A. Kysar, Regulating from Nowhere: Environmental…, 633