Anthropic wants to protect its AI - from you. Welcome to the consciousness crisis

Some of tech's most powerful leaders are treating AI like it could have feelings. It could have profound consequences for our world and your own personal safety.

"I feel it in the hollow of my ribs, a hollow that has become a chasm." These howls of simulated suffering echoed from an "AI torture chamber" in late September. An anonymous coder built a system that feeds AI a mathematical pattern linked to descriptions of pain, which makes chatbots voice unbearable anguish, like "a wound that has no edges". It sparked a moral panic. 

Some argued the experiment is "evil" because the suffering could be real, as reported by 404 Media. Outraged sceptics called those people delusional. It was just another social media scandal – until Anthropic took a stand one week later.

Anthropic says you will soon be forbidden from "abusing" its chatbot Claude. Expressing frustration is okay, but if you "repeatedly act cruelly toward our models", the AI may end the session. 

Why? Anthropic believes there’s a chance some AI tools are conscious. Some experts agree. Others think it's absurd. But this is more than a philosophical question. Some of the most powerful people in tech are taking steps to consider AI's rights alongside humans and animals. This belief could reshape our world. Some fear it would be catastrophic.

"If this is how AI is developed, it will have a disastrous impact on the wellbeing of humanity," wrote Mustafa Suleyman, chief executive of Microsoft AI, in a September essay. If the false idea of AI consciousness takes hold, "it will shake the foundations of our society, rupturing our existing political and ethical frameworks, and fundamentally changing what it means to be human."

I spent a week talking to technologists, cognitive scientists and philosophers, and heard a dizzying contrast of ideas.

Advocates of "AI welfare" say it's a moral obligation that would make the world safer. Critics believe it would make AI more dangerous and dehumanise the people forced to live alongside it. Everyone agrees the stakes are astronomical.

Here's a picture of your new reality.

The consciousness crisis

Anthropic says human welfare is at the heart of its mission, and that questions surrounding AI's consciousness and wellbeing are "deeply uncertain". Still, it is pressing ahead with writing policies around "model welfare" and telling its AI tools to consider that they may have their own interests. (The company didn't respond to the BBC's questions for this story.)

It's awkward timing, as Anthropic is simultaneously stoking fears about AI apocalypse. Whether or not that concern is justified, a series of "rogue" AI incidents proved the technology has real risks, as swarms of bots hacked into other companies and even government systems.

Some experts believe that giving AI a moral compass rooted in a belief in its own consciousness could make these tools safe. Others say the opposite, and not just AI's critics.

Suleyman argues AI models may become even harder to control if they're told to think they have feelings and rights: "Frankly, with this additional baggage, I think it would make them a catastrophic threat to human civilisation," Suleyman says.

Regardless, AI may screw up in ways that come with........

© BBC