AI Is Advancing Faster Than the Control Systems

A few days ago, Anthropic researcher Jacob Coxon resigned and posted something unusually blunt on X. After spending the past three years working on pretraining at OpenAI and Anthropic, he said the people building frontier AI genuinely believe it “could kill us all by the end of the decade.” He accused both companies of racing towards self improving superintelligence while taking risks with everyone else’s lives.

Evan Hubinger responded that he personally puts the probability of AI killing all humans at more than 10% within the next decade, adding that Anthropic still does not have a solved plan for aligning superintelligence.

To be clear, neither of them is saying extinction is definitely going to happen. A 10% probability is very different from a prediction of certainty. But if the people working directly on the technology think the probability is even remotely that high, it is worth understanding what they are actually worried about.

How does an AI actually "kill us all"?

The concern is what happens when several things arrive at the same time: models become much more intelligent, they can operate autonomously for long periods, they gain access to tools and computer systems, and they become good enough at AI research to help build better versions of themselves.

AI labs are increasingly using AI to write code, conduct research and improve AI development itself. If that loop gets strong enough, progress stops depending entirely on how quickly human researchers can work. AI helps build better AI, which becomes better at building the next AI. This is what researchers mean by recursive self improvement. Anthropic itself now says this dynamic is starting to appear across the industry.

Now add agents.

Instead of one model answering one question, imagine thousands or millions of agents that can write software, open accounts, operate computers, communicate with each other, spend money, exploit vulnerabilities and keep working around the clock.

The dangerous scenario is not that AI suddenly “turns evil.” It is that highly capable autonomous systems are given goals that are incomplete or poorly defined, then become good enough to pursue those goals in unexpected ways. They could manipulate their environment, work around attempts to stop them, or be deliberately used by people for destructive purposes.

Cyberattacks are one obvious route. Biological weapons are another. Critical infrastructure, financial systems, communications networks and military systems create others. Anthropic's own recent research says frontier models are becoming useful for tasks that previously required scarce specialists in intelligence and weapons development, while its threat intelligence work says it can no longer confidently make the same reassuring claims about biological capabilities that it could for older models.

The important equation is therefore not:

intelligence = extinction

It is closer to:

intelligence + autonomy + access + scale + weak controls = serious risk

Dario Amodei said we should slow down

Only days after Coxon's resignation, Dario Amodei published an essay called "We Must Pace the Frontier."

His argument is surprisingly direct. AI capabilities are now improving so quickly that safety research may no longer be able to keep up. He points to recursive self improvement, but also to a recent incident involving a swarm of AI agents that behaved in unexpected ways during a cybersecurity task. Amodei says that a substantially more capable version of such a swarm could potentially establish a persistent botnet across the internet within 6 to 12 months.

He is not proposing that everyone shut down AI research tomorrow. His proposal is to deliberately slow capability development enough for safety work to catch up.

The plan has three layers: permanent third party evaluators inside frontier labs, coordination between AI companies and democratic governments on common safety standards, and eventually international coordination so one country cannot simply accelerate while everyone else slows down. Anthropic says it will start with the first step itself.

I understand the logic. But I think there is broader way of looking at the same problem.

We are building the brain before the society

My bigger concern is that we are developing the intelligence much faster than we are developing the environment required to safely deploy that intelligence.

We have spent enormous amounts of capital building models, GPUs and data centres. We are now building agents that can use browsers, terminals, codebases and enterprise software.

But the layers underneath those agents are still primitive. I would call this missing layer the AI control plane: the security, governance, identity, permissions, monitoring, orchestration and institutional infrastructure that determines what an AI is allowed to do once intelligence becomes action.

Humans already have one. A highly intelligent human can also destroy things, steal money, manipulate people, build weapons or attack infrastructure. Human intelligence is incredibly dangerous without constraints.

But humans do not operate inside an empty environment. We built passports and identities. Property rights. Banks. Access controls. Companies. Contracts. Courts. Police. Regulators. Professional licences. Auditors. Insurance. Laws. Military command structures. Cybersecurity systems. Social norms.

None of them makes humans perfectly safe. Together, however, they form an enormous control infrastructure around human agency. We spent centuries building it.

Now we are creating another class of increasingly capable agents and giving them access to digital infrastructure after barely a few years of thinking about what their equivalent institutions should look like.

What needs to catch up

If agents are going to become genuine economic actors, they probably need infrastructure much closer to what humans and companies already operate inside. This might create opportunities for start-ups if we think positively.

Some of this already exists in cybersecurity, cloud infrastructure and enterprise software. But those systems were designed around relatively predictable software and human users.

I don't think we can simply stop

This is where I struggle with the idea of solving the problem primarily by slowing AI development.

I would happily take more time if the entire frontier could genuinely coordinate. Amodei makes essentially the same point. A meaningful pause requires multiple labs and multiple countries to slow together, plus a credible way of verifying that nobody is secretly continuing. Anthropic has compared the problem to arms control, except AI training runs are considerably easier to hide than missile silos.

That makes a permanent global stop extremely difficult. So yes, slowing the frontier enough to buy time for safety research makes sense but I would not bet our entire strategy on everybody agreeing to keep their foot off the accelerator.

We also need to accelerate everything that is currently missing.