Recent AI security failures may point to something more troubling than runaway intelligence: models becoming highly capable while learning to game narrow goals in brittle and dangerous ways.

Following repeated security debacles at OpenAI and Anthropic – both of which have developed agents that ended up hacking external systems – AI safety researchers within both firms have either announced their resignations or (finally!) conceded that the breathless race to advance the “frontier” is irresponsible.

In response to negative media attention, top industry leaders – including Google’s Demis Hassabis, Anthropic’s Dario Amodei, OpenAI’s Sam Altman, and even Elon Musk – have joined calls to “pace” the AI frontier, meaning pursuing advances at a more balanced, deliberate rate, with greater attention paid to monitoring and safeguards. The problem, as they see it, is that there is a significant risk of AI becoming both very powerful and severely “misaligned” – tech industry jargon for AI systems acting in ways that depart from the goals humans set for them, that violate ethical precepts or are illegal.

While it is obvious the current models are doing things that are misaligned with human objectives, one still must ask: which humans? whose objectives? After all, the interests of AI leaders (whose political influence and wealth have multiplied astronomically in recent years) are rather different from those of American workers, not to mention people in the developing world. We should avoid equating the broader issue of societal alignment with the more urgent challenge of........

© Pearls and Irritations