Untested risk models can detain people on factors having no real predictive value

Evidence

The long-overdue evaluation, in 2006, found that four of the variables in their model—whether the defendant had lived in Minnesota for more than three months, whether they lived alone, their age at the time of booking, and whether a weapon was involved in their charge—had virtually no bearing whatsoever on their actual risk of committing a new offense while waiting for their trial date or of missing a required court appearance. And yet the model had been recommending pretrial detention on those grounds. Worse, three of those four factors strongly correlated with race. — Brian Christian, The Alignment Problem, 269

Categorization

• XREF: Concrete example of the ‘god-like’ unvalidated models theme running through the book; connects to Angwin finding COMPAS unvalidated after eleven years. — connects to: Model opacity makes algorithmic decisions unaccountable in courtrooms — Brian Christian, The Alignment Problem, 61; Machine-learning bias often stems from unrepresentative training data; Privacy rules can prevent engineers from even measuring model bias — Brian Christian, The Alignment Problem, 297 (+7 more)
• corroborated: 2 independent passages support this claim — Brian Christian, The Alignment Problem, 269

Related

• ProPublica tested COMPAS recidivism predictions against actual reoffense outcomes for bias (similarity 0.81)
• Model opacity makes algorithmic decisions unaccountable in courtrooms (similarity 0.79)

Broader Connections

• Predictive threat-detection rests on epistemologically fragile assumptions #expands (0.75) — Grégoire Chamayou, A Theory of the Drone, 198
• Confidence in expert systems can be catastrophically misplaced #exemplifies (0.75) — Anthony Giddens, The Consequences of Modernity, 13
• ProPublica tested COMPAS recidivism predictions against actual reoffense outcomes for bias #corroborates (0.55)
• Removing protected attributes fails to remove bias from models #expands (0.45) — Brian Christian, The Alignment Problem, 294
• Machine-learning bias often stems from unrepresentative training data #qualifies (0.40)
• Privacy rules can prevent engineers from even measuring model bias #expands (0.40) — Brian Christian, The Alignment Problem, 297
• Parole prediction is possible only as aggregate statistics, not certainty #complicates (0.35) — Brian Christian, The Alignment Problem, 245