Model opacity makes algorithmic decisions unaccountable in courtrooms

Evidence

If a facial-recognition system is wildly inaccurate for people of one race or gender but not another, or if someone is denied bail because of a statistical model that has never been audited and that no one in the courtroom—including the judge, attorneys, and defendant—understands, this is a problem. — Brian Christian, The Alignment Problem, 61

Categorization

• XREF: Links algorithmic opacity to the reader’s theme of unstated assumptions and blind spots in systems of authority. — connects to: Machine learning systems are not inherently fair or just by default; Untested risk models can detain people on factors having no real predictive value — Brian Christian, The Alignment Problem, 269; Legal discovery rules exist to prevent trials from being contests of concealment — Jonathan Harr, A Civil Action, 2903 (+3 more)
• corroborated: 2 independent passages support this claim — Brian Christian, The Alignment Problem, 61

Related

• ProPublica tested COMPAS recidivism predictions against actual reoffense outcomes for bias (similarity 0.80)
• Untested risk models can detain people on factors having no real predictive value (similarity 0.79)

Broader Connections

• Untested risk models can detain people on factors having no real predictive value #epitomizes (0.75) — Brian Christian, The Alignment Problem, 269
• Machine learning systems are not inherently fair or just by default #expands (0.70)