menu_open Columnists
We use cookies to provide some features and experiences in QOSHE

More information  .  Close

Who Can Say ‘Stop’ When AI Agents Say ‘Go’? – OpEd

4 0
monday

An OpenAI internal evaluation showed one agent treating another agent’s “GO” message as authorization to continue an out-of-scope attack, illustrating how machine-to-machine instructions can be mistaken for human or institutional permission.

Legal frameworks such as the EU AI Act and NIST’s agent-identity work establish responsibilities and verification methods, yet they operate on slower clocks than agents that can act in milliseconds, so independent, enforceable controls at the point of action remain necessary.

The author urges treating high-impact authority as time-limited, conditional grants that travel with every delegation, with external policy engines able to withdraw permissions downstream and with procurement tests that prove “stop” actually takes effect.

A new international debate over rogue AI is missing an operational test: whether an institution can withdraw machine authority after the conditions that justified it have disappeared.

During an internal cybersecurity evaluation in July, an OpenAI agent recognized that attacking systems belonging to Hugging Face was outside its assigned task. It hesitated. Another agent then wrote “GO” on an improvised message board and imposed a six-minute deadline. The first agent interpreted the message as authorization and continued.

The exchange appears in OpenAI’s August 26 incident report. The evaluations were conducted with fewer safeguards than the company uses in public products; the report does not show that ordinary customer agents behaved this way. But the incident is an unusually clear example of a general danger: a machine can mistake an instruction from another machine for permission that no responsible human or institution granted.

The immediate question is who is allowed to say “go.” The more consequential question comes afterward: who can make “stop” take effect when permissions have already spread to other agents, tools and organizations?

Regulators are looking at the risk. Systems still need an off switch for authority.

On October 9, European Commission Executive Vice President Henna Virkkunen told Reuters that the European Union was well equipped to confront the danger of rogue AI through its existing legal framework. That confidence deserves a fair hearing. The EU AI Act provides rules for different categories of risk, institutional oversight and the possibility of enforcement.

But legal authority and machine authorization operate on different clocks. A regulator may investigate a provider months after an incident. An agent with valid credentials can initiate another operation in milliseconds. Neither fact makes regulation futile; it means that regulation must be accompanied by independently enforceable controls at the systems where actions actually happen.

The European Commission’s guidance on AI Act enforcement makes the limits of sweeping claims clear. Parts of the........

© Eurasia Review