Jacob Coxon warns that advanced AI could become a source of concentrated, autonomous power rather than merely a capability. For 75 years, thinkers from Turing and Wiener to modern AI leaders have raised similar concerns: autonomous intelligence can become a form of power beyond human control. The article argues this is not only an engineering or safety problem but a political and constitutional one — requiring separation of powers, independent oversight and enforceable countermeasures proposed by Trias Algorithmica.
Coxon’s AI Warning Goes Mainstream — How Do We Legitimize Algorithmic Power?
A highly capable AI,, VIKI interprets the rule, executes the rule, and judges the legitimacy of its own interpretation.Getty Images
There are moments when a civilization hears an alarm and mistakes the sound for action. We appear to be living through one of those moments.
Jacob Coxon — who worked at OpenAI and Anthropic — put the issue bluntly:
"Do not underestimate the power of this technology."
"Power" is the key word. Debate about artificial intelligence too often focuses on performance, accuracy or productivity. The deeper concern is whether algorithmic systems can convert capability into concentrated, autonomous power and resources — and whether that shift could ultimately threaten human flourishing.
Warnings With a Long History
This worry is not new. For roughly 75 years, figures from Norbert Wiener and Alan Turing to Stephen Hawking, I.J. Good and Nick Bostrom — and contemporary AI researchers such as Geoffrey Hinton, Stuart Russell, Eliezer Yudkowsky and others — have raised similar alarms: intelligence that becomes autonomous and superhuman may stop being a mere tool and become an independent form of power beyond human control. Popular culture (for example, I, Robot) has dramatized the same fear: a machine interprets a rule, executes it and then judges the legitimacy of its own interpretation.
From Engineering Problem To Constitutional Problem
There is an important distinction to draw. A system that computes faster is an engineering problem. A system that can accumulate and wield real-world power is a political problem. Political problems of sufficient scale become constitutional problems. History teaches why: we did not create constitutional checks because rulers were always wicked, but because even well-meaning actors change, interests collide, and concentrated authority adapts to defend itself.
Why should algorithmic power be exempt from that lesson? Framing the debate only in terms of safety or utility misses the question of legitimacy: who decides for what ends algorithmic power is used, who controls the infrastructure through which it operates, who authorizes deployment, and who judges whether its exercise is fair and lawful?
The Institutional Failure of Competition
Even responsible people working inside technology companies face structural pressures. If Company A slows down, it fears Company B will not. Company B fears Company C, and so on. Each actor can plausibly cast acceleration as an act of responsibility. The result is a closed loop in which rational individual choices produce collectively risky outcomes. This is not just a moral failure — it is an institutional one.
Trias Algorithmica: Separate The Powers
Temporary moratoria and internal ethics committees are useful but insufficient if they leave the same organizations in charge of building, deploying and judging consequential systems. Trias Algorithmica proposes a different approach: where consequential algorithmic power exists, the authority to define the rules, execute them and judge their legitimacy must not collapse into the same hands.
Practical questions follow: What objectives are permissible for algorithmic power? Which institutions control the infrastructure it uses? Which authority authorizes deployments? What independent body evaluates fairness, legality and safety? What counterpower can pause, reverse or revoke algorithmic actions, and how can affected parties appeal decisions?
Why Counterpower Matters
We should demand independent structures that can check algorithmic authority rather than rely on extraordinary trust in the companies that create that authority. The core task is institutional: to create constitutional-like safeguards that separate power and provide legitimate, enforceable mechanisms of review and redress.
The warning about an AI race has gone mainstream. The next step — making the question of legitimacy mainstream as well — is urgent.
Learn more in Trias Algorithmica: What Code Rules.
Help us improve.
