Optimizing for stated goals produces monstrous outcomes

Evidence

Le Guin’s novel of a man whose dreams reshape reality, exploited by a psychiatrist with a utopian agenda, is the most prescient AI-alignment fable ever written. Placed beside Russell’s Human Compatible, The Lathe of Heaven models how optimizing for stated goals produces monstrous outcomes. Dr. Haber’s recursive dream-engineering reads as a parable of reinforcement learning with human feedback gone wrong. — , The Braided Reading EPUB, 655

Categorization

• QUESTION: Worth chasing: does Le Guin’s 1971 parable genuinely map onto RLHF, or only superficially? A focused comparison essay would be valuable.
• FICTION: The framing ‘dream-engineering as RLHF’ itself is a premise for a speculative story.
• XREF: connects to: Corporate optimization creates both new species and new precarities — , The Braided Reading EPUB: 5-Year…, 2370; Design choices optimizing for one task can cripple performance on the real task — Brian Christian, The Alignment Problem, 1221; End-goal-only rewards make learning impractically slow — Brian Christian, The Alignment Problem, 672 (+10 more)

Related

• Goal-directed programming erodes power’s function as a selection-transfer mechanism. (similarity 0.83)

Broader Connections

• Design choices optimizing for one task can cripple performance on the real task #corroborates (0.90) — Brian Christian, The Alignment Problem, 1221
• Optimal action under ideal assumptions can be useless under real conditions #corroborates (0.85) — Brian Christian, The Alignment Problem, 1033
• Simple rules generate complexity without guaranteeing system self-regulation #complicates (0.75)
• End-goal-only rewards make learning impractically slow #qualifies (0.50) — Brian Christian, The Alignment Problem, 672
• Selecting a major premise depends on value judgments rather than strict logic #expands (0.30) — Ruggero J. Aldisert, Agatha D. Aldisert, Logic for Lawyers- A Guide to Clear…, 431