A chain of innocent-seeming steps can jointly produce evil

Evidence

A science fiction reader reminded me of Asimov’s fa-mous laws governing the conduct of robots. Obviously, in a thoughtful society, one would build a prohibition against killing a human being into the governing program of a robot. In one disconcerting short story, I recall that the author figured out a way that robots could bring about the death of a human being. The programmer simply created a series of apparently innocent steps that brought about a human’s death. For instance, one robot puts rat poison in a container in the tool shed, another robot is told to move the container from the shed to a kitchen shelf, the third robot is told to put a spoon of “sugar” from the container into the victim’s coffee. To return from the fanciful, the general point is that a series of subroutines, each not morally outrageous, can sum to an outrageous act. — John T. Jost and Jim Sidanius, Political Psychology, 4045

Categorization

• XREF: Connects to Arendt’s banality of evil and the organizational-path discussion in this same chapter; Asimov’s robot laws illustrate the programmable-decomposition idea.
• STORYSTEM: A story could follow a bureaucratic or supply-chain system where each individual honestly performs an innocent step and never learns the final outcome — the collective product of clean routines being atrocity.
• BMF: The robot-subroutine decomposition of murder evokes the way the Disposer-regime in We (Zamyatin) routinizes obedience through innocuous incremental steps, the individual never tasting the full moral act.