Design choices optimizing for one task can cripple performance on the real task

Evidence

But color, as it happened, even if it wasn’t particularly helpful at telling one bug from another, was critical in telling bugs from non-bugs. “We were really handicapping ourselves,” Dietterich says. These decisions haunted, and to some degree chagrined, him. “I still bear the scars,” he says. — Brian Christian, The Alignment Problem, 1221

Categorization

• XREF: Relates to reward shaping and Goodhart-type failures from earlier chapters: optimizing the measurable proxy destroys the real goal. — connects to: Optimal action under ideal assumptions can be useless under real conditions — Brian Christian, The Alignment Problem, 1033; Library architecture can override user needs when visions clash — Andrew Pettegree, Arthur der Weduwen, The Library- A Fragile History, 966; Cognitive off-loading diminishes essential neural engagement in students. — Joanna Stern, I Am Not a Robot- My Year Using AI to…, 1006 (+6 more)
• corroborated: 2 independent passages support this claim — Brian Christian, The Alignment Problem, 1221

Broader Connections

• Optimal action under ideal assumptions can be useless under real conditions #corroborates (0.90) — Brian Christian, The Alignment Problem, 1033
• Library architecture can override user needs when visions clash #exemplifies (0.55) — Andrew Pettegree, Arthur der Weduwen, The Library- A Fragile History, 966