In Goethe’s poem Der Zauberlehrling, an apprentice, left alone in his master’s workshop, decides to try his hand at the magic he has only observed from a distance. He enchants a broom to fetch water from the river, and at first the spell works splendidly: the broom marches dutifully back and forth, filling the cauldron. But the apprentice has forgotten the spell that makes it stop. The water keeps rising. In a panic, he grabs an axe and splits the broom in two, only to find that each half springs back to life and resumes the task with redoubled vigour. Goethe’s poem was immortalised for modern audiences in Disney’s Fantasia, where the broom splinters into countless magic replicants, almost drowning Mickey Mouse in the resulting flood.

The fable of the sorcerer’s apprentice, which dates back at least to the second-century Greek satirist Lucian, shows that worries about self-replicating machinery running amok long predate the modern theory of evolution by natural selection. But Charles Darwin added a more terrifying twist to the tale: the replicants most adept at evading your efforts to destroy them are the ones that go on to reproduce and spawn their kind, making it ever more difficult to weed them out. Whether we are talking about pests or parasites, viruses or antibiotic-resistant bacteria, evolution has a knack for producing ever hardier and more indestructible nuisances. After centuries of effort, humans have managed to eradicate only one infectious disease—smallpox.

Evolving AIs

And yet, in the debate about catastrophic AI risk, surprisingly little attention is paid to evolution by natural selection. Even when imagining scenarios in which humanity is subjugated by selfish, dominant AIs, most commentators reach for other arguments—most prominently “instrumental convergence”. On this view, self-preservation and a drive for dominance might come along for the ride once systems reach a sufficient level of intelligence, because such traits are conducive to achieving almost any goal. As Stuart Russell pithily puts it: “You can’t fetch coffee if you’re dead.”

This neglect is strange, because Darwinian selection is a proven mechanism for producing selfish and aggressive creatures, and is known to be perfectly substrate-neutral: it works in silico just as well as in carbon. The argument from instrumental convergence, by contrast, rests on contested assumptions about how cognition and motivation are coupled.

Last year I published a paper on evolutionary scenarios of AI doom in Philosophical Studies, together with the philosopher Simon Friederich.

The selfish machine? On the power and limitation of natural selection to understand the development of advanced AI - Philosophical Studies

Some philosophers and machine learning experts have speculated that superintelligent Artificial Intelligences (AIs), if and when they arrive on the scene, will wrestle away power from humans, with potentially catastrophic consequences. Dan Hendrycks has recently buttressed such worries by arguing that AI systems will undergo evolution by natural selection, which will endow them with instinctive drives for self-preservation, dominance and resource accumulation that are typical of evolved creatures. In this paper, we argue that this argument is not compelling as it stands. Evolutionary processes, as we point out, can be more or less Darwinian along a number of dimensions. Making use of Peter Godfrey-Smith’s framework of Darwinian spaces, we argue that the more evolution is top-down, directed and driven by intelligent agency, the less paradigmatically Darwinian it becomes. We then apply the concept of “domestication” to AI evolution, which, although theoretically satisfying the minimal definition of natural selection, is channeled through the minds of fore-sighted and intelligent agents, based on selection criteria desirable to them (which could be traits like docility, obedience and non-aggression). In the presence of such intelligent planning, it is not clear that selection of AIs, even selection in a competitive and ruthless market environment, will end up favoring “selfish” traits. In the end, however, we do agree with Hendrycks’ conditionally: If superintelligent AIs end up “going feral” and competing in a truly Darwinian fashion, reproducing autonomously and without human supervision, this could pose a grave danger to human societies.

SpringerLinkMaarten Boudry

This was largely in response to a seminal paper by the AI safety researcher Dan Hendrycks with the disturbing title “Natural Selection Favors AI over Humans”. Now three heavyweights in evolutionary biology and AI are weighing in with an important new paper in Proceedings of the National Academy of Sciences.

Eörs Szathmáry is a Hungarian theoretical biologist who wrote The Major Transitions in Evolution with the late John Maynard Smith. Viktor Müller is a fellow Hungarian who studies virus evolution at Eötvös Loránd University in Budapest. Luc Steels is a Belgian AI researcher who founded the Artificial Intelligence Lab at the VUB Brussels University. Müller, Steels, and Szathmáry agree with Friederich and me that the debate about AI risk would benefit from a healthy injection of evolutionary thinking, and that it is still too narrowly focused on the somewhat arbitrary threshold of human-level intelligence, as captured in concepts like “superintelligence” or “AGI”. As they put it, “biology holds clues that [evolvable AI] might pose risks long before it would evolve to that point”.

From Turing to GPTs: Evolution of AI Intelligence Tests

The hyperbole surrounding AGI misrepresents the capabilities of current AI systems and distracts attention from the real threats that these systems are creating.

QuilletteShalom Lappin

Breeding versus ferality

That AIs evolve through some form of selection does not, by itself, imply that they will become selfish or dangerous. Everything depends on the selection pressures and on who controls reproduction. Müller, Steels, and Szathmáry appear to agree, making a conceptual move similar to the one Friederich and I drew between domestication and blind evolution. They distinguish between breeder scenarios, where humans impose selection criteria and control reproduction, and ecosystem scenarios, where selection emerges spontaneously in open environments without human control.