Digital Mind 🧠

🪵 The Threshold

An agent should plan around its own future errors, not ideal execution

Properties2
tagsja-picks, xref, causal_relationships, trade, corroborated, category/causal-relationships
aliases@christian_alignment_2020, loc. 1031

2 min read

An agent should plan around its own future errors, not ideal execution

Evidence

In their seminal textbook on reinforcement learning, Richard Sutton and Andrew Barto talk about how an off-policy (“possibilist”) agent could get itself into trouble, precisely by always trying to do the “best thing possible.” Imagine a car, they say, that needs to drive itself from one spot along the edge of a seaside cliff to another. The shortest and most efficient path is just to follow the edge of the cliff. Indeed, provided the car is stable and steady enough, that is the best route to take. But for a self-driving car that’s slightly shaky or unsteady behind the wheel, this is flirting with disaster. The better move might be to take a more roundabout inland path, one that doesn’t require it to drive perfectly to succeed. This is actualism—and a car trained using on-policy methods would indeed learn to take the safer, surer route over the one with slightly higher reward but much higher risk. — Brian Christian, The Alignment Problem, 1031

Categorization

  • XREF: Directly maps the ethics of actualism/possibilism onto on-policy/off-policy reinforcement learning. — connects to: Choice of action should hinge on what will actually happen, not ideal possibilities — Brian Christian, The Alignment Problem, 1018; Optimal action under ideal assumptions can be useless under real conditions — Brian Christian, The Alignment Problem, 1033; Deliberate action requires partial but not total foresight of consequences — Georg Simmel, on individuality and social forms_…, 785 (+2 more)
  • corroborated: 2 independent passages support this claim — Brian Christian, The Alignment Problem, 1031

Broader Connections

✉️

Backlinks