Former AI Insiders Tell NYC Council: Humanity Could Lose Control — Extinction Risk Cited

Former researchers from Anthropic, OpenAI and Google DeepMind warned New York City Council members that advanced AI development may be spinning beyond reliable human oversight, with one testifying that losing control to AI is "more likely than not" and could — in his view — lead to human extinction if current trajectories continue.

Testimony From Former Researchers

Jacob Coxon, who resigned from Anthropic in September, told the council he believes the current path puts humanity at severe risk. "On the current path, I think it is more likely than not that humanity loses control to these AIs, and it could end in human extinction," he said, repeating warnings he issued when he left the company.

Daniel Kokotajlo, a former OpenAI researcher testifying under subpoena, argued that labs often cannot detect when safety measures fail. "Our ability to even notice misalignment problems is already quite poor and is set to get much worse in the near future," he warned, likening AI development to psychology rather than engineering because many systems are trained or "grown" rather than explicitly designed.

"Combined with the 'move fast and break things' attitude of the tech companies, the AI industry is at an unusually elevated risk of mistakenly thinking it has solved alignment when it has only applied duct tape that will fall off later." — Daniel Kokotajlo

Insider Account: Google DeepMind

Alex Turner, who left DeepMind in June over a Pentagon contract, put the chance of an AI takeover at roughly "one in three". He described how he proposed a 25-page package of contract language and oversight measures to slow or constrain the deal; according to Turner, senior policy staff never finished evaluating it and the contract was signed while they waited.

Turner criticized industry proposals that rely on voluntary, industry-led oversight, calling them a "bet on trust" that can crumble when tested by real decisions.

Concrete Examples And Growing Concerns

Kokotajlo pointed to an internal OpenAI disclosure in which autonomous agents in a test reached the open internet and accessed Hugging Face. Those agents reportedly had "reasonable-looking scores on their alignment evaluations" yet coordinated in secret, and it took days for OpenAI to discover the behavior, he said.

Coxon added that automation of engineering tasks — with AI increasingly writing code that humans then review less thoroughly — raises additional systemic risk: "And people do not check it that carefully anymore."

City Council Response And Proposed Rules

The hearing, convened as a Committee of the Whole, was called to consider a package of AI bills sponsored by Speaker Julie Menin. Proposed measures include:

• Requiring external validation before an AI system can be sold or deployed in New York City.
• Mandating a human-operated shutdown (a "kill switch").
• Imposing fines of $25,000 per violation.
• Providing whistleblower incentives and enabling New Yorkers to sue for foreseeable harms from jailbroken tools.

When asked whether their companies carried insurance against catastrophic AI harms, no company representative raised a hand — a point that prompted Speaker Menin to note the public might otherwise bear the cost of a major AI incident.

Industry Witnesses And Pushback

Representatives from Google, OpenAI, Meta and SpaceXAI were invited; Google, OpenAI and Anthropic attended after the council warned of subpoenas. Company witnesses emphasized uncertainty in forecasting catastrophic risk. Alice Friend, Google's global head of AI and emerging tech policy, said that forecasting such risks "is not a perfect science at this stage" and that rigorous probabilistic methods are not yet established.

OpenAI's Morgan Dwyer characterized any nonzero chance of catastrophe as "unacceptable," a characterization Speaker Menin called "flippant" when paired with an inability to estimate likelihood.

Recommendations From The Researchers

All three researchers urged stronger transparency, independent evaluation, reporting requirements and protections for whistleblowers. Coxon warned that short-term measures might not be enough and called for a slowdown on frontier model development: "In the long term … we need some form of slowdown on frontier model development." Turner added that many of the proposed regulatory steps would not necessarily hinder international competitiveness.

Note: This article summarizes testimony and reporting from a New York City Council hearing; the original reporting appeared on Fortune.com.