Privacy rules can prevent engineers from even measuring model bias

Evidence

Because of redundant encodings, it’s not enough to simply be blind to the sensitive attribute. In fact, one of the perverse upshots of redundant encodings is that being blind to these attributes may make things worse. It may be the case, for instance, that the maker of some model wants to measure the degree to which some variable is correlated with race. They can’t do that without knowing what the race attribute actually is! One engineer I spoke with complained that his management repeatedly stressed the importance of making sure that models aren’t skewed by sensitive attributes like gender and race—but his company’s privacy policy prevents him and the other machine-learning engineers from accessing the protected attributes of the records they’re working with. So, at the end of the day, they have no idea if the models are biased or not. — Brian Christian, The Alignment Problem, 297

Categorization

• QUESTION: A genuinely ironic structural conflict: privacy protections may be undermining fairness auditing in the same organizations.
• corroborated: 2 independent passages support this claim — Brian Christian, The Alignment Problem, 297
• XREF: connects to: Untested risk models can detain people on factors having no real predictive value — Brian Christian, The Alignment Problem, 269; Legal discovery rules exist to prevent trials from being contests of concealment — Jonathan Harr, A Civil Action, 2903; Copyright’s chilling effect on AI training data worsens algorithmic bias — , The Braided Reading EPUB: 5-Year…, 2242

Related

• Removing protected attributes fails to remove bias from models (similarity 0.82)
• Copyright’s chilling effect on AI training data worsens algorithmic bias (similarity 0.79)

Broader Connections

• Copyright’s chilling effect on AI training data worsens algorithmic bias #corroborates (0.70) — , The Braided Reading EPUB, 2242
• Legal discovery rules exist to prevent trials from being contests of concealment #expands (0.50) — Jonathan Harr, A Civil Action, 2903