It’s ‘more likely than not’ humanity loses control: Former AI insiders testify safety fixes may be ‘duct tape that will fall off later’

Jacob Coxon, a former OpenAI and Anthropic researcher, told the New York City Council on Monday humanity is more likely than not to lose control of advanced AI, doubling down on the warning he issued when he quit last month.

“On the current path, I think it is more likely than not that humanity loses control to these AIs, and it could end in human extinction,” said Coxon, who resigned from Anthropic in September after saying AI companies were “gambling” with human lives.

Coxon voluntarily testified, and was joined by former OpenAI researcher Daniel Kokotajlo, who testified under subpoena that the companies may not know when their safety work has failed.

“Our ability to even notice misalignment problems is already quite poor and is set to get much worse in the near future,” Kokotajlo testified. “I would say that the field is more like psychology than engineering, because these AI systems are trained or grown; they’re not really designed.

“Combined with the ‘move fast and break things’ attitude of the tech companies, this means that the AI industry is at an unusually elevated risk compared to other industries of mistakenly thinking that it has solved the problem when really it just applied some duct tape that will fall off later,” he continued.

Coxon and Kokotajlo testified alongside Alex Turner, who left Google DeepMind in June after the company signed a Pentagon deal he opposed. Like Kokotajlo, he was also subpoenaed, a rarity for the council. Monday’s hearing was also the first time the City Council held a Committee of the Whole hearing (a hearing of all 51 council members) since 2022, called to weigh a package of AI bills from Speaker Julie Menin.

Startup culture vibes to safety........

© Fortune