Goal-directed programming erodes power’s function as a selection-transfer mechanism.

Evidence

Le Guin’s novel of a man whose dreams reshape reality, exploited by a psychiatrist with a utopian agenda, is the most prescient AI-alignment fable ever written. Placed beside Russell’s Human Compatible, The Lathe of Heaven models how optimizing for stated goals produces monstrous outcomes. Dr. Haber’s recursive dream-engineering reads as a parable of reinforcement learning with human feedback gone wrong. — , The Braided Reading EPUB, 655 (#corroborates, via linked claim)

If the determination that a result is bad or in some way suboptimal is based on the full range of considerations of policy and principle that the court or anyone else would use without the rule, then what work is the rule itself doing? If our ability to perceive that a rule has generated the wrong result necessarily requires a sense of what the right result would be, and if that determination cannot be made by referring to the very rule whose soundness is now in question, then the common-law judge is deciding what the right result would be under a different rule or under no rule. And if every case in which the existing rule generates a result other than the one that appears correct in a rule-free evaluation is one that occasions a change in the rule, then why do we not just dispense with the rules entirely? — Frederick Schauer, Thinking Like a Lawyer, 222 (#corroborates, via linked claim)

But color, as it happened, even if it wasn’t particularly helpful at telling one bug from another, was critical in telling bugs from non-bugs. “We were really handicapping ourselves,” Dietterich says. These decisions haunted, and to some degree chagrined, him. “I still bear the scars,” he says. — Brian Christian, The Alignment Problem, 1221 (#corroborates, via linked claim)

Categorization

• XREF: connects to: Power’s function is transferring reduced complexity, institutionalized through its code.; Optimizing for stated goals produces monstrous outcomes — , The Braided Reading EPUB: 5-Year…, 655; Design choices optimizing for one task can cripple performance on the real task — Brian Christian, The Alignment Problem, 1221 (+5 more)

Related

• Optimizing for stated goals produces monstrous outcomes (similarity 0.83)

Broader Connections

• Optimizing for stated goals produces monstrous outcomes #corroborates (0.70) — , The Braided Reading EPUB, 655
• Rules that always yield to rule-free optimization cease to function as rules #corroborates (0.65) — Frederick Schauer, Thinking Like a Lawyer, 222
• Design choices optimizing for one task can cripple performance on the real task #corroborates (0.60) — Brian Christian, The Alignment Problem, 1221
• Power functions like a catalyst, not a cause, in social systems #expands (0.55)
• Disembedding mechanisms remove control from individuals and create collective risk #complicates (0.50) — Anthony Giddens, The Consequences of Modernity, 310
• End-goal-only rewards make learning impractically slow #contradicts (0.40) — Brian Christian, The Alignment Problem, 672