Gray Swan AI founders (left to right): Zico Kolter, Matt Fredrikson and Andy Zou.

More than 600 hackers convened last month to compete in a “jailbreaking arena,” hoping to trick some of the world’s most popular artificial intelligence models into producing illicit content: for instance, detailed instructions for cooking meth, or a deceptive news story that argues climate change is a hoax.

The hacking event was hosted by a young and ambitious security startup called Gray Swan AI, which is working to prevent intelligent systems from causing harm by identifying their risks and building tools that help to ensure these models are deployed safely. It’s gotten early traction, securing notable partnerships and contracts with OpenAI, Anthropic and the United Kingdom’s AI Safety Institute.

“People have been incorporating AI into just about everything under the sun,” Matt Fredrikson, Gray Swan’s cofounder and chief executive officer, told Forbes. “It’s touching all parts of technology and society now, and it’s clear there’s a huge unmet need for practical solutions that help people understand what could go wrong for their systems.”

Gray Swan was founded last September by a trio of computer scientists who had been investigating safety issues unique to AI. Both Fredrikson and chief technical advisor, Zico Kolter, are professors at Carnegie Mellon University, where they met PhD student and fellow cofounder Andy Zou. (Fredrikson is currently on leave.) Earlier this year, Kolter was appointed to OpenAI’s board of directors and made chair of the company’s new safety and security committee, which has oversight of major model releases. As such, he has recused himself from interactions between the two companies.

“We've been able to show, really for the first time, that it’s possible to defend these models from this kind of jailbreak.”

The breakneck pace at which AI is evolving has created a vast ecosystem of new companies — some creating ever........

© Forbes