Earlier this week, researcher Jacob Coxon quit Anthropic, saying the firm and its competitors are “gambling with our lives”. “We really do earnestly believe AI could kill all humans,” added current Anthropic researcher Evan Hubinger in a post on X.

Coxon isn’t the first to down tools over fears of AI doom. The idea that AI could wipe out humanity, advanced in Nick Bostrom’s 2014 book Superintelligence and the influential LessWrong forum, has long circulated among researchers. There are many scenarios for how this could happen, but the core idea is that AI smarter than humans could escape our control and destroy us.

In 2024, Jan Leike and Daniel Kokotajlo quit OpenAI over safety concerns. This year, Anthropic safety chief Mrinank Sharma departed, warning “the world is in peril”. Alex Turner left Google DeepMind after it signed a deal with the Pentagon permitting “killer drones”.

But Coxon’s resignation has made waves, with more researchers admitting they think AI might kill everyone. So if the people building AI believe it could cause extinction, why keep building it? There are three main reasons.

Some think the risk is worth it

AI leaders acknowledge the risk of losing control and killing everyone. In 2023, the chief executives of OpenAI, Anthropic and Google DeepMind agreed that AI extinction risk should rank alongside pandemics and nuclear war. Anthropic’s Dario Amodei puts the chance of........

© The Conversation