The people who actually want AI to replace humanity |
The context you need, when you need it
When news breaks, you need to understand what actually matters — and what to do about it. At Vox, our mission to help you make sense of the world has never been more vital. But we can’t do it on our own.
We rely on readers like you to fund our journalism. Will you support our work and become a Vox Member today?
The people who actually want AI to replace humanity
We need to create a new humanism before the “AI successionists” win.
“I want AI to be a tool that allows human flourishing!” exclaimed Brad Carson, a former member of Congress. “There is an option out there where AI is just a tool for us.”
This is a normal thing to say in most circles. But Carson was speaking at an invite-only symposium dedicated to the idea of creating a “Worthy Successor” — an AI so impressive, so beyond the mere human, that we’d actually want it to replace humanity.
“You’re a brave man for entering this room!” Dan Faggella, an AI market researcher and organizer of the symposium, told Carson. “You’re in probably the only room in the country where most people disagree with you.”
The attendees at the symposium, which took place at the New York Academy of Sciences last September, are part of a subculture that is growing in importance: the AI successionists, who think that artificial intelligence is our rightful heir — the next step in cosmic evolution. Since they believe AIs could become our moral superiors, they argue it’s actually wrong to try to keep the machines down, or even to align them with human values, as most AI companies aim to do. Instead, we should usher in artificial intelligence as a successor to humanity and hand over the world to it. Even if that means we go extinct.
They know this view is taboo, which is why I was invited only on the condition that I wouldn’t quote anyone other than keynote speakers by name. But suffice it to say that this is not a fringe view. It’s becoming highly influential. People from major AI labs — Anthropic, Google DeepMind, xAI — were in attendance. So were people from think tanks that directly shape the US government’s AI policy.
Why I wrote this story
I grew up hearing an old Jewish teaching: Each of us should carry two slips of paper, one in each pocket. One says, “I am but dust and ashes.” But the other says, “The world was created for me.”
Reporting on AI these past few years, I’ve watched more and more people forget the second message. They think we should be okay with getting obliterated if a more valuable species can take our place. But more valuable to whom? Value isn’t dispensed from some cosmic vantage point; it’s always value to someone. And we’re valuable to us.
And yet the AI successionists are right about something: We can’t expect human beings to look the same a thousand or a million years from now. So how do we decide which kinds of technological change to embrace, and which to refuse? It bothered me that classical humanism doesn’t have a good answer. Here, I’ve sketched what a new one might look like.
AI successionism has been gaining ground among technologists over the past decade. In 2015, Google co-founder Larry Page famously accused Elon Musk of “speciesism” because Page thought we should let digital minds take over, and Musk disagreed.
The successionist vision has been amplified by the advent of effective accelerationism (e/acc) in 2022. Its founder, Guillaume Verdon — the physicist more colorfully known on X as Based Beff Jezos — describes e/acc as a “meta-religion” that’s about “having faith” in the universe’s drive toward increasingly intelligent systems. The best thing we can do is help the universe by developing advanced AI as fast as possible, even at the expense of humanity. “E/acc,” as Verdon has written, “has no particular allegiance to the biological substrate.”
Tech heavyweights have come on board. Venture capitalist Marc Andreessen listed e/acc thinkers as his “patron saints.” Garry Tan, the CEO of tech startup accelerator Y Combinator, included “e/acc” in his social media bio and invested in Verdon’s company, which aims to build the world’s most efficient computers. OpenAI CEO Sam Altman posted on X to Verdon, saying, “you cannot outaccelerate me.”
And these days, AI successionism is spreading beyond Silicon Valley. At the New York symposium, Faggella told the audience that trying to preserve the human species as it is would be silly.
“We could ask the questions that would tie all of our moral aspirations eternally to 23 chromosomes — or we could ask the cosmic questions,” Faggella said.
He wanted us to consider “unpolite, uncouth” possibilities, starting with: The flame of consciousness — the capacity for experience and moral value — may be the rarest and most precious thing in the universe. Humanity is currently a torch carrying that flame, but what if we’re ultimately not the best carrier for it? And if AI can spread that flame far further than we mere humans can, generating experiences of bliss and forms of moral value that we could never even dream of, shouldn’t we let it?
Faggella’s talk was greeted by a loud round of applause. Later, he and a couple dozen attendees headed to a nearby hotel balcony for drinks. And so it was that I found myself overlooking the Manhattan skyline as people talked about the end of humanity over cocktails.
There was some diversity of opinion among the group. Not everyone self-identified with the relatively new term “AI successionist.” Some were proponents of transhumanism, the movement that says we should use tech to proactively evolve our species into Homo sapiens 2.0. Transhumanists hope to keep some version of humanity going, but definitely not the current hardware; they dream of radical life extension, cognitive enhancement, and eventually mind uploading. (Musk, who said he created his brain chip company Neuralink to help humanity merge with AI, probably falls — or at least fell — into this category.) Others were posthumanists, those who want us to give rise to descendants that move beyond humanity altogether.
The biologist sitting across from me was very excited about the prospect of merging humans with AI. He said we should task AI with figuring out how best to do the merger, then “take it off the leash” and allow AI to control its own evolution — and by extension, ours. Of course, he said, not all humans will make it through the transformation; only a select group of people will transition to the next evolutionary stage. (Presumably, the type of people privileged enough to imbibe cocktails at Manhattan AI symposia.)
The man seated beside me, a researcher from one of the major AI companies, was even more radical. Forget merger — it’s okay if humans don’t survive at all, he said. Human text has been used to train the AIs; in some sense, then, the human spirit will live on. “So on the cosmic level,” he said cheerfully, “I’m okay with it.”
Most people are definitely not okay with it. The average person would probably find the answers of the Worthy Successor group repugnant. Yet the core question they pose cannot be ignored. Whether they picture us merging with machines or ultimately being superseded by them, technologists are developing innovations that could dramatically change what it means to be human — think AI-powered brain chips that enable mind-reading or magnetic implants that give you a sixth sense — and genetic tools that could even reshape the DNA of all future generations.
As it becomes possible to direct our own evolution as a species — and potentially even create a new species that surpasses us — we have to decide: How do we know to what extent it does make sense to transform ourselves using technology? What kinds of augmentation do we want, and what kinds do we absolutely not want? What do we wish, ultimately, to become?
This is a moral question, even a spiritual one, and it demands a spiritual response. The AI successionists are offering one. For anyone who finds it repulsive, the challenge is to offer a countervailing positive vision.
And it’s essential to do that now, because as sci-fi as the successionists might sound, they are building real political power, with links to the authoritarian right. Several of the tech heavyweights who’ve embraced successionism want to escape the control of democratic governments, so much so that they’re seeking to create their own sovereign colonies. That can come in the form of space colonies, à la Elon Musk and Jeff Bezos, or in independent “startup cities” or “network states” built by corporations here on Earth — currently Peter Thiel and Marc Andreessen’s favored approach. And Verdon’s investors include entrepreneur Balaji Srinivasan, a major proponent of the network state.
These broligarchs have successfully cozied up to the Trump administration, clearing the way for their accelerationist vision. And they’ll take the wheel unless we come up with an alternate vision for the future.
The natural alternative is humanism, which replaced the medieval view that humans need God to rescue them with the view that humans have the ability, and responsibility, to achieve flourishing through their own efforts. The problem is that, so far, we haven’t developed a version of humanism that’s brave enough to directly tackle the core question — what do we want our species to become? — and answer it compellingly.
AI systems could become conscious. What if they hate their lives?
The most common “pro-human” response tries to say there are certain fixed traits that make humans unique, and to locate value only in humans as they currently exist. “In the era of artificial intelligence, when human dignity is threatened by new forms of dehumanization, ours is the pressing duty to remain profoundly human,” Pope Leo recently wrote in his encyclical Magnifica Humanitas. This response says: Let’s use tech remedially — to alleviate problems like disease — but let’s not try to augment the species.
That feels insufficient as a guide to the future, because, even before the advent of AI and gene editing, “human” has never been a static category. Homo sapiens has always been evolving and augmenting itself, from the agricultural diet that reshaped our jaws to the algorithms reshaping our attention.
The old formulation is “the naive version of humanism,” Shannon Vallor, a........