Users are more vulnerable to algorithms they know are mindless.
Evidence
It is in our human nature to be vulnerable to anyone, or anything, that seems to understand us better than we do ourselves. In fact, we may be even more vulnerable to algorithms we know to be mindless than we are to real persons, because we are more easily lulled into a false sense of security. We pretend Alexa is a person because we are not used to conversing with machines – the experience would otherwise be embarrassing or uncanny. But the fact that we know Alexa is not a person is how we come to terms with its intense knowledge of us, which would otherwise be offputtingly creepy or scary. At that precise moment, when we relate to it as if it were a person while we know it is not, we are at our most vulnerable – ready to fall into the trap of thinking of Alexa as our own Pandora-like mechanical serf. Alas, Alexa is no serf. It is, rather, a piece of cloud-based command capital which is turning you into a serf, with your aid and by means of your own unpaid labour, in order to further enrich its owners. — Yanis Varoufakis, Technofeudalism, 314
Categorization
- XREF: The claim that we are most vulnerable when we know the machine is not a person runs against the usual uncanny-valley framing; connects to work on trust in automation. — connects to: AI systems are using artists’ work without their consent. — Emily M. Bender, Alex Hanna, The AI Con- How to Fight Big Tech’s…, 277; Fear of nonexistence pressures people to participate in surveillance platforms. — Shoshana Zuboff, The Age of Surveillance Capitalism, 1084; Normal human activities that feel unproductive become exploited as data — Nick Couldry & Ulises A. Mejias, The Costs of Connection: How Data Is…, 675 (+11 more)
- corroborated: 2 independent passages support this claim — Yanis Varoufakis, Technofeudalism: What Killed Capitalism, 314
Broader Connections
- Digital natives are actually populations dispossessed by data relations.
#expands(0.90) — Nick Couldry & Ulises A. Mejias, The Costs of Connection, 350 - Mindlessness leads to panic and detrimental behavior.
#complicates(0.90) — Dan Harris, 10 percent Happier, 40 - AI systems are using artists’ work without their consent.
#complicates(0.85) — Emily M. Bender, Alex Hanna, The AI Con- How to Fight Big Tech’s…, 277 - Normal human activities that feel unproductive become exploited as data
#expands(0.85) — Nick Couldry & Ulises A. Mejias, The Costs of Connection, 675 - Modern individuals demand self-determination while being treated as invisible ciphers
#complicates(0.85) — Shoshana Zuboff, The Age of Surveillance Capitalism, 140 - The liberties-versus-security binary justifies sacrificing out-group rights
#complicates(0.85) — Saher Selod, Forever Suspect_ Racialized…, 487 - Algorithmic discrimination affects real people despite targeting only probabilistic data doubles.
#complicates(0.85) — Nick Couldry & Ulises A. Mejias, The Costs of Connection, 408 - Fear of nonexistence pressures people to participate in surveillance platforms.
#complicates(0.80) — Shoshana Zuboff, The Age of Surveillance Capitalism, 1084 - Individual attempts to control online identity fail because others disclose without sharing one’s boundaries
#complicates(0.80) — Alice E. Marwick, Status Update, 636 - Humans lack built-in detectors for their own compartmentalized thinking
#complicates(0.80) - Social software can promote inequality rather than counter it.
#complicates(0.80) — Alice E. Marwick, Status Update, 19 - Untested risk models can detain people on factors having no real predictive value
#complicates(0.80) — Brian Christian, The Alignment Problem, 269 - Claiming surveillance without evidence diminishes real COINTELPRO harm.
#complicates(0.75) — Anna Merlan, Republic of Lies, 870 - AI evangelists frame creative skill barriers as unjust exclusions to dismantle
#complicates(0.75) — W. David Marx, Blank Space, 697