Dear Scott,

I may be a veteran of debates, but this is the first time I’ve been challenged to a duel! Many people are eager to host this spectacle, but I choose to delope: to fire my pistol into the ground to pre-empt the conflict.

Obviously I believe in defending ideas in the face of criticism. But we’re both rationalists, adamant that intellectual discourse be carried out in pursuit of objective truth, uncontaminated by cognitive biases. So I’m surprised by your demand that we face off in a formal debate with an audience vote. These debates are a spectator sport, not an epistemic tool. They reduce a complex issue to a dichotomy, select for zingers and gotchas, engage the psychology of dominance, and are tilted by audience selection. In fact I’ve done only one of these, with Matt Ridley against Malcolm Gladwell and Alain de Botton, and observers agree it was not an edifying experience. (Gladwell opened by calling my hair effeminate. No hard feelings: we won.) And while I share your appreciation of the epistemic value of bets, shouldn’t these be over verifiable predictions, not audience persuasion?

As in the Code Duello, you write as if you are defending your honour after a series of intolerable affronts. So let me begin by apologising for any inadvertent offence and by emphasising that my admiration for you is immense. I think your brilliance is enviable and your integrity inspiring. We agree on almost everything, including your commitment to rationality and effective altruism (at least in their original, lower-case versions). I’ve never criticised or publicly disagreed with you, let alone called you names like “paranoid” or “weird moron.” You are taking my criticisms of AI-existential-risk claims personally. I stand by my claim that most of the doomer scenarios are preposterous (beginning with the AI paperclip maximiser that consumes our bodies as raw materials). But as rationalists we should agree that attacking an idea should not be confused with attacking a person.

Speaking of attacks, I’m puzzled by your accusation that my “contributions to the mass discourse have been to say, or to boost other people saying, that we’re only scared of AI because we’re male, that we can’t be trusted because we live ‘unconventional lifestyles’, that our beliefs are inherently insensitive to people worried about near-term AI risks … that we’re paranoid, that we should be laughed out of the room, and every other dirty attack he can think of.”

In fact my principal contribution to mass discourse on existential risk is a 31-page chapter called “Existential Risk” in my 2018 book Enlightenment Now. It addressed the criticism that the Scientific Revolution, Enlightenment, and ideal of progress were terrible mistakes because they will lead to human extinction by (among other things) runaway AI. My counterargument said nothing about maleness, polyamory, paranoia, or other irrelevancies. It did criticise doomer scenarios like the paperclipalypse, the AI that would cure cancer by conscripting all humans as guinea pigs in lethal experiments, the AI that would solve climate change by exterminating humanity, the AIs that would launch a global holocaust because humans were inconvenient, or the AIs that would grow exponentially until they boiled the oceans away with their waste heat or froze the planet with their spaceborne solar collectors. I argued that these scenarios had one or more of the following flaws:

1. They had an underdeveloped conception of intelligence which treated it as a quantity of power which may be extrapolated from animals to dull humans to smart humans to AI to Artificial Superintelligence, the latter consisting of perfect omniscience. (This was the focus of my later exchange with Scott Aaronson.) I argued that any intelligent system is a mechanism which is good at solving some problems but not others, and which is inherently limited by knowledge about the world attainable only by observation and experimentation.
2. They conflated intelligence with motivation, particularly self-preservation and dominance. I argued that these motives happened to come bundled with intelligence in Homo sapiens because we are products of natural selection, but they are not inherent to intelligent systems that are engineered.
3. They assumed that an artificially intelligent system would monomaniacally pursue a single goal, heedless of side effects. I argued that trading off multiple competing goals is the essence of intelligence, so no genuine AI would wreak the ridiculous havoc imagined in the doomer scenarios. 
4. They assumed that human engineers would grant these systems unlimited and irreversible control over the earth’s physical infrastructure, amounting to omnipotence over every atom on the planet.

Now, these arguments may be wrong, but they are far from trash talk. Similar ones have recently been made by many informed commentators, including Blaise Agüera y Arcas, Jerry Kaplan, Gary Marcus, Melanie Mitchell, Arvind Narayanan and Sayash Kapoor, and Andrew Ng. A 2025 Rand Corporation modelling analysis steel-manned the AI-extinction hypothesis by granting the assumption that AI had the objective to wipe out humanity, and concluded that there is “no describable scenario in which AI is conclusively an extinction threat to humanity.”

To be sure, my arguments were laid out five years before the release of LLM chatbots, and I would update some of them today. The capabilities of LLMs trained on gargantuan datasets are more general than I (and almost everyone else) would have guessed at the time. And I have to revisit my assumption that AI would emerge from the culture of engineering I was familiar with from my decades as an MIT professor: the assumption that AI would be “developed incrementally, designed to satisfy multiple conditions, tested before it is implemented, and constantly tweaked for efficacy and safety.” I did not anticipate the breakneck and sometimes irresponsible release of LLMs by the frontier companies (ironically, led by alumni of the X-risk community; more on this later).

Nonetheless, I stand by them as substantive arguments against the radical claim (I trust you’ll agree that it is radical) that within a decade rogue AI could exterminate humanity through scenarios like the ones listed above. Go ahead and rebut the arguments—but Scott, with all due respect I think you are misreading them in ways that maximise offence. In particular:

• When I said that the doomer scenarios projected alpha-male psychology onto AIs, this was not an ad hominem insinuation that the proponents themselves were alpha males. It was an analysis which proposed that they were applying an inappropriate alpha-male theory of mind (an intuitive psychology) to AI.
• Likewise, when I refer to the paperclip-maximising and similar AIs as “stupid,” or to the hypothetical engineers in the thought experiments who would empower them without testing them as “moronic,” these are not aspersions on living people, but on the hypothetical protagonists of the exotic scenarios. And the adjectives were not gratuitous insults but juxtaposed as antonyms to the “intelligence” in “artificial intelligence.”
• I confess that my tone can be irreverent and my prose purplish, which you suggest is incommensurate with the gravity of the topic. But the AI X-Risk community itself is facetious about what they insist will be a massacre of humanity (the paperclip memes, the jocular talk about p(doom), the “foom,” the “death with dignity” and other Eliezerisms). And Scott, you yourself are a wizard of irreverent satire. Can I suggest that you not be so sensitive when it’s your friends who are being treated impiously?

Now, I did tweet a possibly hackle-raising article in the New York Times in which Cal Newport traced the history of AI doomerism back to a Berkeley Rationalist subculture which he called “weirder than we realize.” Let’s agree that the provenance of an idea has little bearing on its validity: whether or not doomers are weird, they could be right. I boosted the article because it addresses legitimate questions. Why are the same people who warn that a technology will kill us all building it as fast as they can? Why are they issuing exact probabilities of human extinction (including 100 percent) which in fact are pulled out of thin air? Whence the fantastical scenarios of end times?

Newport observes that OpenAI and Anthropic are “deeply connected to a strain of apocalyptic futurist thinking that believes something like an all-powerful superintelligent A.I. god is inevitable—and that this being will either deliver a trans-humanist utopia or destroy humanity.” The companies see their mission as upping the odds of the benign outcome. The historical and sociological context of this way of thinking—inspiration by an eccentric visionary with an eschatological prophecy, a pattern reminiscent of cults—is not irrelevant to making sense of our strange moment.

It also bears on the real issue at stake here, AI safety. Newport notes that this ideology might explain the frontier companies’ recklessness: “If you believe winning the race to superintelligence is necessary to save humanity, you might not worry too much about collateral damage along the way.” (Hence the obsession with the vague goal of “alignment” and lesser interest in mundane problems of cybersecurity.) It’s consistent with my argument in Enlightenment Now that prophecies of doom are not effective responses to genuine risks but have pernicious downsides. Nuclear doomsday scenarios in the 1960s set off not disarmament but an arms race with the Soviets. The existential fear of nuclear terrorism (the smoking-gun-as-mushroom-cloud) led to the invasion of Iraq. (One is reminded of Yudkowsky’s call for bombing foreign data centres even if it risks escalation to nuclear war.) And telling young people they will soon perish en masse has costs in their appreciation of the institutions of modernity, their mental health, their confidence to invest in themselves and their society, and their willingness to have children (itself a risk of extinction).

AI Doom Is Not Cybersecurity

An OpenAI agent’s breach of a Medicare portal shows that the real danger from AI is a mundane one—and the labs’ fixation on a coming machine god is making it worse.

QuilletteZack Korman

The dangers of AI are real, including bioterrorism, cyber-sabotage, and unconstrained agents. I’m strongly in favour of sober safety evaluations and engineering best practices: independent oversight, mandatory investigations of accidents, liability for damage, humans in the loop, kill switches, and others. On these I trust that you and I agree. I’m just sceptical of the prophecy that AI will murder every last human and of the assumption that sowing such dread is the most effective way of getting such safeguards.

With respect,

Steve