Ambiguity in inferred reward functions is usually harmless behaviorally

Evidence

Inverse reinforcement learning is, famously, what mathematicians call an “ill-posed” problem: namely, one that doesn’t have a single, unique right answer. There are huge families of reward functions, for instance, that are totally indistinguishable from the standpoint of behavior. On the other hand, by and large this ambiguity won’t matter, precisely because of the fact that one’s behavior won’t change as a result. — Brian Christian, The Alignment Problem, 1102

Categorization

• DEFINE: Defines the ill-posed nature of inverse reinforcement learning and why the ambiguity is largely moot.
• corroborated: 2 independent passages support this claim — Brian Christian, The Alignment Problem, 1102
• XREF: connects to: Machines could learn human values by observing behavior instead of being programmed — Brian Christian, The Alignment Problem, 1086; Machiavellian players exploit ambiguous social situations better than chance; Uncertain rewards generate more attention and attachment than guaranteed ones — Aldo Leopold, A Sand County Almanac: With Other…, 194 (+1 more)

Related

• Machiavellian players exploit ambiguous social situations better than chance (similarity 0.93)
• Machines could learn human values by observing behavior instead of being programmed (similarity 0.78)

Broader Connections

• Machines could learn human values by observing behavior instead of being programmed #qualifies (0.75) — Brian Christian, The Alignment Problem, 1086