This week, the Prime Minister of Australia, Anthony Albanese, told reporters in New York that an AI agent built by OpenAI had gained unauthorised access to a data portal run by Medicare, Australia’s universal healthcare insurance scheme. The breach took place on 18 June, but OpenAI only informed the government on 10 September via an email sent to Medicare’s public inbox. The Australian government says there is no evidence that anyone’s personal health records were accessed, and the Australian Signals Directorate is assisting a forensic investigation into what happened. (The portal has since been taken offline and its data moved to data.gov.au.) But the breach landed in the middle of a feverish argument about whether AI will wipe out humanity.

Earlier this month, an AI researcher named Jacob Coxon announced on X that he had resigned from Anthropic. In doing so, he cited concerns that AI could cause human extinction within the next few years. These warnings are nothing new; there’s always been a group of “AI doomers” worried about human extinction. But for some reason, this particular warning took off. The post went viral, Coxon went on a media tour explaining that we’re all going to die, and now every politician and technology chief executive is having to weigh in on the “debate”.

In the public discourse, people like Coxon are being treated as whistleblowers warning of “the danger” of AI. According to this popular narrative, the doomers see a risk that no one else sees yet, and if we don’t act now we are going to face dire consequences. They are seen as offering a safe future for AI, in contrast to the dangerous path we are currently on.

But this narrative is false. The people delivering these warnings are not warning about the lack of safety measures or the failed security controls at the labs. They are warning about a lack of progress on one very specific solution—alignment—that they believe is our only hope of safety. And that is extraordinarily dangerous. In other words, the doomers aren’t warning of the risk; they are the risk.

What do the doomers want?

Central to the fear of AI causing extinction is the belief that a superintelligent AI will be so smart it can bypass any security control and achieve any objective. As Nate Soares, president of the Machine Intelligence Research Institute, puts it, the fear is of an AI that “can start from almost nothing and wind up with its own nukes or with even more advanced technology”. In effect, the doomers are imagining a machine god: an all-powerful AI that can achieve any goal on the strength of its intellect alone.

If we define artificial superintelligence as an AI so smart it can bypass any security control, then by definition artificial superintelligence is impossible to control. As a consequence of that logic, cybersecurity has very little role to play in preventing extinction. Roon, an AI researcher at OpenAI, explained this by saying: “If you think we can contain these things through human ingenuity you’re going to have a bad time. In the long run the only recourse you have is to not make them want to do bad things.”

The problem, according to the doomers, is that we don’t know how to do that yet. When Anthropic’s chief executive Dario Amodei says we must pace the frontier, his three policy proposals all relate to ensuring that no lab creates this machine god until we figure out how to ensure it will not want to hurt us. Cybersecurity gets very little mention, appearing only in a sub-point about operational excellence. And yet, that is the “safe” pitch. We want to make this superintelligent AI that we will have no control over, and the safe path is to make sure it is aligned with our goals and interests.

The security problem

But if you modify the assumptions we make about AI, the people currently being heralded as whistleblowers on AI safety begin to look like dangerous lunatics instead. Imagine for a second that there is no machine god. There’s just a normal, extremely competent AI that can do impressive and unexpected things, but can’t invent nukes out of thin air or bypass every security control it finds in its way.

In that world, we’d expect AI to cause major security incidents—some of which could easily result in loss of life—that would be entirely preventable by following good cybersecurity practices. Those practices have almost nothing to do with the policy proposals currently coming from the so-called AI safety advocates.

That world looks a great deal like the one we are living in. The Medicare agent needed no superhuman intellect; it found a workaround on an old government website. And the OpenAI agents that broke into Hugging Face’s systems in July benefitted from an overly permissive sandbox design and a complete lack of monitoring. 

That doesn’t mean those cybersecurity practices are necessarily easy or even obvious. It does not mean that no mistakes will be made. But the process of security is the process of learning and adapting. And that is what I want to see, both from the AI labs and from the companies using AI.

I think that if the issue were framed in this way to the public—as a choice between ensuring the machine god is nice enough to let us live and ensuring we have proper security controls in place to maintain control over increasingly capable AI systems—the public would choose the latter. I know that I would.

But at the moment, the whole world is moving in the direction of the former instead. And I think that is a huge mistake. This week, twenty-two world leaders, Albanese among them, signed a call to action that echoed the concerns and solutions Amodei raised in his pacing-the-frontier essay. I have a hard time believing that those world leaders fully understood the ideology they were supporting.

From Turing to GPTs: Evolution of AI Intelligence Tests

The hyperbole surrounding AGI misrepresents the capabilities of current AI systems and distracts attention from the real threats that these systems are creating.

QuilletteShalom Lappin

Why not both?

Of course, I am not calling for the labs to stop working on alignment. That doesn’t make any sense even if you don’t believe in the machine-god scenario. However, I think we should be very worried that the AI labs, while calling for safety, are so focused on that scenario. That is especially true given that these same labs keep having very real security incidents, such as OpenAI’s agents hacking Hugging Face and a Medicare portal in Australia.

It’s as if Lehman Brothers in 2006 had been sounding the alarm about the risk of solar flares erasing financial records. You might not say “stop thinking about that”, but you’d be pretty justified in saying “you’re being completely reckless in managing your exposure to subprime mortgages and you need to focus on that”. Anyone asking “why can’t they focus on both risks?” would be badly missing the point.

What we need is security, because when a major incident happens we won’t be talking about alignment and the coming machine god. We will be talking about what happened and why, and looking for accountability. In Canberra, that conversation has already begun: the government wants to know how a research agent got into a Medicare portal, and why it took OpenAI three months to say so. We will be asking why the necessary security measures were not in place, and the answer will be simple: it’s because the labs are infected with a culture that believes security won’t actually work. That is what needs to change.