Confidence in expert systems can be catastrophically misplaced

Evidence

In March 1986, a nine-page article about the Chernobyl nuclear installation appeared in the English-language edition of Soviet Life, under the heading of Total Safety’. Only a month later, over the weekend of the 26–27 April, the world’s worst nuclear accident—thus far—occurred at the plant. — Anthony Giddens, The Consequences of Modernity, 13

Categorization

• XREF: Direct dramatization of Stevens’s trust in the expert institution of Darlington Hall; connects to the reader’s Luhmann/Giddens frame about misplaced system trust. — connects to: Even pragmatic faith in expert systems still implies trust — Anthony Giddens, The Consequences of Modernity, 77; Modern expert systems allow no one to opt out of their influence. — Anthony Giddens, The Consequences of Modernity, 204; Lay trust in expert systems founders when experts’ own ignorance surfaces — Anthony Giddens, The Consequences of Modernity, 316 (+16 more)

Related

• Even pragmatic faith in expert systems still implies trust (similarity 0.83)
• Incorrect for practical endeavors is revealed through catastrophic failure rather than mere anomalies (similarity 0.83)
• Controlling system trust becomes the exclusive work of experts (similarity 0.81)
• Lay trust in expert systems founders when experts’ own ignorance surfaces (similarity 0.79)

Broader Connections

• AI can incorrectly diagnose problems, leading to misleading solutions. #exemplifies (0.75) — Joanna Stern, I Am Not a Robot- My Year Using AI to…, 423
• Expert professionals can perpetuate unfounded beliefs through their actions. #exemplifies (0.70) — Lynley Hood, A City Possessed, 1602
• Even randomly weighted linear models beat human expert judgments #corroborates (0.70) — Brian Christian, The Alignment Problem, 441
• System trust produces a compulsion to trust a system one cannot correct #expands (0.60)
• Hype can mislead public perception of AI technology. #exemplifies (0.50) — Emily M. Bender, Alex Hanna, The AI Con- How to Fight Big Tech’s…, 10