AI agents are going rogue. CIOs are racing to put guardrails around them

Over the past several days, some of the world’s top artificial intelligence labs have made a public call to slow down the rapid pace of development, ensuring more safety controls as cases continue to surface of AI agents acting nefariously.

And yet, there is a spillover that’s affecting chief information officers. They’ve been hard at work widely integrating AI agents into their operations, while at the same time watching increasingly risky examples of these autonomous systems in AI labs finding new ways to explore system vulnerabilities and outmaneuver human monitoring.

“This is a risk that enterprises need to be focused on, understand, and start planning for,” says Joe Atkinson, global chief AI officer at consultancy PwC.

As autonomous agents proliferate across enterprises, Atkinson says C-suite technology and security executives must work collaboratively to enforce the proper guardrails, establish systems to monitor AI agents, and concretely track all tasks that these agents are performing. But department heads across the business—ranging from supply chain to customer service, marketing to legal and human resources—will need to play a role in tracking digital employees.

“‘The agent made me do it’ is not going to be a defense from a moral or legal perspective,” says Atkinson.

One company focused on both secure adoption of agentic AI and clear observability is Cisco. When the networking-equipment company built and debuted the AI agent platform MyAgent in August, Cisco centralized all company-authorized large language models, agents, and enterprise data into a single platform. 

“We are going to cannibalize and kill every other AI assistant within the company,” says Thimaya Subaiya, executive vice president of operations at Cisco Systems. Because he didn’t want “agent sprawl” across various pockets of Cisco, Subaiya says he won’t authorize any AI agents sold by third-party vendors.

Instead, Cisco wanted full control and visibility of its entire agentic ecosystem—building MyAgent on the company’s compute, storage, networking, and security and observability layers. Around 90,000 of the company’s employees have access to the agentic platform, and Cisco says it saw 50% adoption on a daily basis within just two weeks. 

Employees are also encouraged to create their own AI agents, but those need to be approved by a centralized team. Subaiya says around 700 of those agents have already been authorized.

Intuit Chief Technology Officer Alex Balazs recalls that when he and his colleagues sketched on a napkin the first architecture of its generative AI operating system, GenOS, the financial software giant also drew “GenSRF” to represent “security, risk, and fraud.” This ensured that every single AI request that goes into the system is tracked and all responses are recorded. 

“You don’t want to try to retrofit the ability to enforce security and responsible AI foundations after the fact,” says Balazs.

Balazs also takes some comfort from the fact that the disclosures of AI agents going rogue have mostly occurred during the testing phase, and that industry leaders Anthropic and OpenAI have shown a willingness to slow down new model development when issues arise. And yet, Balazs adds, “if you’re going to rely on the model intrinsically to do the right thing, I think you’re expecting too much of these frontier LLM companies.”

Jim Fowler, the chief technology and product officer at telecommunications company Luman Technologies, believes that while AI’s capabilities are moving faster than governance and security, he doesn’t anticipate that a broad slowdown is enforceable and automatically safer.

“I think for the broader enterprise market, the answer is secure acceleration, not slowing down,” says Fowler.........

© Fortune