Training data can encode the beneficial effects of treatment itself

Evidence

The rule was “If the patient has a history of asthma, then they are low-risk and you should treat them as an outpatient.” … The correlation that the rule-based system had learned, in other words, was real. Asthmatics really were, on average, less likely to die from pneumonia than the general population. But this was precisely because of the elevated level of care they received. “So the very care that the asthmatics are receiving that is making them low-risk is what the model would deny from those patients,” Caruana explains. — Brian Christian, The Alignment Problem, 390

Categorization

• SEED: The ‘protected population paradox’ — a model learns that the very people who receive the most care are safest, and then denies them that care — is a rich essay or lecture example about how feedback loops corrupt training data.
• corroborated: 2 independent passages support this claim — Brian Christian, The Alignment Problem, 390
• XREF: connects to: Machine-learning bias often stems from unrepresentative training data; AI training data is invisible yet permanently shapes deployed system behavior — Brian Christian, The Alignment Problem, 150

Broader Connections

• AI training data is invisible yet permanently shapes deployed system behavior #corroborates (0.60) — Brian Christian, The Alignment Problem, 150
• Machine-learning bias often stems from unrepresentative training data #expands (0.50)