Some preferences are inherently unknowable even when their owner is desperate to reveal them

Evidence

How can we know how badly the Russians would dislike an all-out war in which both sides were annihilated? We cannot; and the reason we cannot is not solely that the Russians are necessarily unwilling that we should know. On the contrary, circumstances may arise in which they are desperate that we should know the truth. But how can they make us know it? How can they make us believe that what they tell us is true? How can the prisoner being tortured for secrets that he really does not know persuade his captors that he does not know them? — Thomas C. Schelling, The Strategy of Conflict, 278

Categorization

• DEEPTHINKING: The torture/prisoner example is the canonical statement of the credibility-of-information problem in mixed-motive games — sits at the heart of commitment theory.
• XREF: This is the foundational statement of the credibility problem that underlies both Gambetta’s honest-signaling and the ‘Godfather as private enforcer’ reading — being desperate to reveal a truth doesn’t make the disclosure credible. — connects to: Subjective reservation values cannot be credibly communicated even when honest — Thomas C. Schelling, The Strategy of Conflict, 280; Autonomous systems become deferential whenever they remain uncertain about user wants — Brian Christian, The Alignment Problem, 1292; Blind tests collapse personal preference into shared judgments (+1 more)

Broader Connections

• Subjective reservation values cannot be credibly communicated even when honest #expands (0.90) — Thomas C. Schelling, The Strategy of Conflict, 280
• Autonomous systems become deferential whenever they remain uncertain about user wants #expands (0.80) — Brian Christian, The Alignment Problem, 1292