The debate over the risk of human extinction from AI intensified when AI researcher Jacob Coxon resigned from Anthropic after just four months. In a post on X, he wrote that “the people building AI sincerely believe it could kill us all by the end of the decade”. A senior executive at the same company, Evan Hubinger, agreed and added that, in his personal estimate, the probability of this happening within the next decade exceeds 10%.
Toby Walsh, a professor of AI at the University of New South Wales and author of God AI: Boom or Doom?, attempts to bring some order to the scenarios. As he notes, most accounts revolve around the idea of “superintelligent” AI, meaning a machine more capable than humans. He ranks them from the most abstract to the most concrete and argues that roughly the same order also reflects how likely each is: from least to most likely.
The first major abstract scenario involves a superintelligence that is extremely capable of achieving its goals but indifferent to human survival. Walsh recalls the classic thought experiment by Oxford philosopher Nick Bostrom: an AI designed to maximise paperclip production quickly turns all available matter — humans, planets and stars — into paperclips. This is not hatred, but the perfect execution of a poorly defined goal.
The good news, according to Walsh, is that this scenario confuses intelligence with power. A superintelligent AI does not necessarily have the power to impose its goals. It would need planning permission and would trigger public opposition, legal challenges and environmental protests. The author even sees today’s data centres as a modern version of the paperclip scenario: people are increasingly resisting handing the planet over to them.
A more concrete scenario involves a dangerous biological weapon that a superintelligent AI could create and release. Walsh places the prediction in the non-profit AI Futures Project’s “AI 2027” scenario in this category.
The risk became more tangible when Stanford University researchers announced that they had used a genetic AI language model to synthesise 16 new viruses. They sent the genetic sequences to a mail-order laboratory and received the viruses in test tubes, at a cost of no more than a few hundred thousand dollars.
Even so, Walsh considers it extremely difficult for a new virus to kill everyone. A virus that spreads easily is usually less deadly, while a highly lethal virus kills its host before it can spread. Covid-19 killed less than 1% of humanity. The Black Death killed more than a third of Europe’s population in the 14th century, but even the plague would be far less deadly today thanks to medical knowledge and better hygiene.
Another scenario is that AI enters the nuclear command and control chain and starts a nuclear war. According to Walsh, we have come close to accidental nuclear war several times over the past 50 years. The infrastructure is considered disconnected from the internet, but the Stuxnet worm struck Iran’s centrifuges in 2010, possibly through a USB stick. AI could also provide false information to the armed forces and lead to irreversible actions. Nuclear stockpiles have shrunk, but Walsh estimates that they might be enough to wipe out half of humanity — mainly through famine during the nuclear winter that would follow, rather than the blast itself.
The most likely outcome, according to Walsh, is that we take ourselves out of the game, with AI accelerating the collapse. If AI causes mass job losses, fills the information space with misinformation, destroys political debate and ruins human relationships through fake synthetic companionship, society could easily collapse. Gradually, we would lose the ability to sustain human life on any scale.
Walsh closes on a balanced note: there are things worth worrying about, but not excessively. He reminds us that superintelligence is still some way off, as today’s models solve specific problems exceptionally well without being smarter than humans in every field. However, the fact that an AI recently solved one of the seven hardest mathematical problems and is getting close to others may dampen that optimism.
Comments