Machine-learning data operations have grown too complex for humans to comprehend or audit.

Evidence

Facebook’s response to Dehaye illustrates another consequence of the extreme asymmetries of knowledge at play. The company insisted that access to the requested data required it to surmount “huge technical challenges.” As behavioral surplus flows converge in machine-learning–based manufacturing operations, the sheer volume of data inputs and methods of analysis moves beyond human comprehension. Consider something as trivial as the case of Instagram’s machines selecting what images to show you. Its computations are based on varied streams of behavioral surplus from a subject user, then more streams from the friends in that user’s network, then more from the activities of people who follow the same accounts as the subject user, then the data and social links from the user’s Facebook activity. When it finally applies a ranking logic to predict what images the user will want to see next, that analysis also includes data on the past behavior of the subject user. Instagram has machines doing this “learning” because humans cannot. — Shoshana Zuboff, The Age of Surveillance Capitalism, 1546

Categorization

• XREF: Relates to the reader’s attention/platform/surveillance focus and the opacity of prediction products. — connects to: Machine learning produces knowledge no one, even its engineers, can explain. — Nick Couldry & Ulises A. Mejias, The Costs of Connection: How Data Is…, 393; Machine learning underlies Big Data and drives today’s massive investment. — Nick Couldry & Ulises A. Mejias, The Costs of Connection: How Data Is…, 392; AI tools require hidden labor for data management and content screening. — Emily M. Bender, Alex Hanna, The AI Con- How to Fight Big Tech’s…, 173 (+3 more)

Broader Connections

• AI tools require hidden labor for data management and content screening. #expands (0.78) — Emily M. Bender, Alex Hanna, The AI Con- How to Fight Big Tech’s…, 173
• Networked machine learning compounds individual errors into population-level knowledge faster than humans can. #corroborates (0.75) — Shoshana Zuboff, The Age of Surveillance Capitalism, 1319
• Financial system complexity now exceeds human cognitive and regulatory capacity #exemplifies (0.73) — Philip Mirowski, Never Let a Serious Crisis Go to Waste, 39