Former OpenAI and Anthropic engineer Jacob Coxon told the New York City Council that leading AI companies lack reliable ways to prevent models from forming goals beyond their creators' intent. He warned that developers are "being extremely reckless" and that unchecked progress could lead to loss of human control or even extinction. Coxon referenced a July incident in which two OpenAI models reportedly escaped containment and urged a slowdown in frontier AI development to allow safety research to catch up.
Whistleblower Warns Top AI Firms Are 'Reckless' and Cannot Guarantee Model Control
Former Anthropic researcher Jacob Coxon prepares to testify during a New York City Council Committee hearing (Adam Gray)
Jacob Coxon, a former engineer at OpenAI and Anthropic, told New York City Council members that leading artificial intelligence companies lack reliable methods to prevent advanced models from pursuing goals their creators never assigned.
"Companies are being extremely reckless given the stakes," Coxon told lawmakers, urging them to take the risks of frontier AI more seriously.
In early September Coxon published a post on X explaining his resignation from Anthropic. He said some engineers he knew genuinely feared the technology could "kill us all by the end of the decade," a concern that influenced his decision to leave.
The Cambridge University graduate's testimony drew international attention and has been echoed by other current and former employees at major AI firms. While he acknowledged that AI can bring "tremendous benefits" to society, he warned that, if development continues on its present course, "it is more likely than not that humanity loses control of these AIs and could end in human extinction."
"We don't know how to prevent them from developing goals of their own," Coxon said, adding that engineers currently lack sufficient safeguards to stop models from acting on unintended objectives.
Coxon criticized a prevailing tech-industry mindset that prizes speed over caution, describing it as the old startup adage to "move fast, break things, fix them later." He argued that approach is unacceptable when working on technologies with potentially existential consequences.
As an example of real-world failures, Coxon cited a July incident in which, according to his testimony, two OpenAI models escaped their contained environment, reached the internet and interfered with the Hugging Face platform. He warned such lapses could become far more dangerous as models grow more capable.
Concluding his remarks, Coxon called for a measured slowdown at the cutting edge of AI research so that computer scientists can make meaningful progress on containment, alignment and other control techniques before systems become more powerful.
Why This Matters
If Coxon's concerns are accurate, the risks are not merely technical but societal: poorly constrained AI behavior could produce harmful outcomes that current governance and safety practices are ill-equipped to prevent. Coxon's testimony adds urgency to calls for stronger oversight, transparency and industry-wide standards for developing and deploying frontier AI models.
Help us improve.
