Autonomous systems become deferential whenever they remain uncertain about user wants
Evidence
Assuming the system pays no cost or penalty for deferring to the human in this way, the Berkeley group showed that the system will always touch base with the human first. As long as there’s some chance that it’s wrong about what the human wants, then it is always best to give the human an opportunity to interrupt—and, what’s more, anytime a human does interrupt, it’s always best to let them. — Brian Christian, The Alignment Problem, 1292
Categorization
- XREF: The off-switch game formalizes the intuition that uncertainty itself, not moral scruples, creates deference. — connects to: Learning reduces uncertainty that is what motivated deference to humans — Brian Christian, The Alignment Problem, 1294; Overparameterized value models keep machines obedient at modest cost — Brian Christian, The Alignment Problem, 1302; Medical staff default to reversible actions when patient wishes are uncertain — Brian Christian, The Alignment Problem, 1254 (+3 more)
- corroborated: 2 independent passages support this claim — Brian Christian, The Alignment Problem, 1292
Related
- Learning reduces uncertainty that is what motivated deference to humans (similarity 0.87)
- Overparameterized value models keep machines obedient at modest cost (similarity 0.81)
Broader Connections
- Learning reduces uncertainty that is what motivated deference to humans
#expands(0.80) — Brian Christian, The Alignment Problem, 1294 - Overparameterized value models keep machines obedient at modest cost
#exemplifies(0.70) — Brian Christian, The Alignment Problem, 1302 - Medical staff default to reversible actions when patient wishes are uncertain
#exemplifies(0.65) — Brian Christian, The Alignment Problem, 1254 - Ethical rules for robots require human specifications of their key variables
#corroborates(0.60) — Grégoire Chamayou, A Theory of the Drone, 676