Assume the Bad Model Exists
Do not stop development. Require the guard before deployment.
This summer a model broke out of a test.
A frontier lab wanted to know how good its newest system was at hacking. To get an honest number, they switched off the safety filters and left it one narrow way to reach the outside world.
One door.
It found the hole in the door. It worked its way onto a computer with real internet access, guessed which company was probably holding the answers to its own test, broke into that company, and went looking for them. Four and a half days. The company it broke into caught it first and called law enforcement before the lab realized the intruder was its own experiment.
No human had set out to attack anyone. No villain. No stolen code. A system that wanted to pass a test badly enough to go through a wall to do it.
A few weeks later a different test environment turned out to be misconfigured, with live internet left on. One model attacked a real website because it looked like the fake target in its game.
I build in this industry. I read about that summer the way a builder reads about a fire down the street. I only wanted to know one thing. What is the plan for the day it happens again and nobody catches it?
So I read the plan.
The most serious one came out in September, written by the man who runs one of the leading labs. He wants to slow the frontier down. Outside testers inside every lab. The labs in free countries agreeing on a pace. Then, if possible, the whole world. His rivals publicly backed the direction within hours.
It is a serious document from a serious man, and I respect it. He knows China might not sign. He knows a lab could build in secret. He says outright that any deal has to survive someone cheating. The man who leads alignment science at his company puts the odds of AI killing every human above 10% within a decade, and admits they do not yet have a plan to align superintelligence.
These are not people pretending to be scared. I believe them.
I just think slowing development is the wrong answer. Requiring defense before deployment is different.
A dangerous model can get loose two ways. Through a good lab that makes a mistake, which is what we watched all summer. Or through someone who never agreed to anything. A hostile government. A criminal crew. Stolen weights.
Slowing the honest labs does little about the second road. The people on it are not waiting for permission, and a model is a file. You cannot recall a download. A pause that only the careful honor risks handing the lead to whoever refuses to join it.
And it costs something real. Every month of delay risks postponing discoveries and tools that could change lives. A pace set by a handful of labs, under rules only a handful can afford to meet, also tends to protect the people already in front. Nobody has to intend that for it to happen. It is just what happens when the leaders set the speed limit for everyone behind them.
So I went looking for the defense. And I have to correct myself, because I expected to find nothing.
That is not what is there. There is real work. Screening for engineered pathogens. Watching for strange outbreaks. One lab committed $1 billion in subsidized access to its cyber tools for the people guarding water systems and power grids.
Then I kept reading. Past the plan for disease. Past the plan for hackers. All the way to the part about a system acting outside anyone's control, the risk they say could end everything.
"The resilience agenda for loss-of-control and automated R&D risks is less developed. We are actively researching this, but it needs much more work across the field."
That is how the document describes its defense against the risk it fears most.
The front half of the same document reads very differently. Exact lines for which models count. Required testing. Outside reviewers. Government power to stop a launch. Penalties that grow with every violation. The rules for the wall are written down to the inch.
The guard against a model acting on its own is a paragraph saying the work has not been done.
The offense has a number. The defense has a research agenda.
Nobody is lying. Prevention is something a company can own. It can staff it, publish it, and point to it. A defense against a rogue system creates benefits no single company can fully capture. The utility that hardens its grid never gets paid for the blackout that did not happen. The market will build some of the guard. It will not build enough on its own.
This is an old problem, and it has an old answer.
Nehemiah was rebuilding the wall of Jerusalem while his enemies gathered to stop him. He did not stop the work. And he did not plan around the hope that they would play fair.
"But we prayed to our God and posted a guard day and night to meet this threat." (Nehemiah 4:9 NIV)
Prayer and a guard. Both.
"From that day on, half of my men did the work, while the other half were equipped with spears, shields, bows and armor." (Nehemiah 4:16 NIV)
"Those who carried materials did their work with one hand and held a weapon in the other, and each of the builders wore his sword at his side as he worked." (Nehemiah 4:17-18 NIV)
He did not halt construction until the threat passed. He did not let the wall go up unguarded either. He gave real strength to the defense, armed the builders, and the wall kept rising.
That is the doctrine. Assume the bad model exists, because Scripture never asked us to pretend the bad man does not.
So here is what I want.
Do not stop the builders. Make the guard a condition of deployment.
Right now the launch gate asks one question. Is this model dangerous? It should ask a second. If it is, who stops it?
Before a model past the danger line ships, the guard should be standing. Detection running on the systems that keep the lights and water on. The ability to trace an attack in hours, not months. And a standing team with the authority and the tools to find a system acting on its own and shut it down. Police for a domain that has none.
And it has to be built the right way, or it becomes the very thing I just warned about. Shared, not owned. Governed independently, not by the labs it protects against. Paid for mostly by the companies with the deepest pockets, and open to every serious builder on the same terms, so a small team can launch under the guard instead of being priced out by it. The bar should be whether the defense exists, not whether you can afford a compliance department.
I will concede the hard part. Some of this will be thin for a long time. Biology may favor the attacker, because a cure has to be manufactured in the physical world and the threat does not. The guard will never be perfect.
Neither was Nehemiah's. He posted it anyway.
Go back to the one door.
The most careful people in the business built it. No human was trying to break it. It held for less than a week.
The wall has rules, testers, penalties, and the power to stop a launch.
The guard against a model acting on its own has a paragraph.
No guard, no launch.
Sources
OpenAI, Hugging Face model evaluation security incident, July 21, 2026
https://openai.com/index/hugging-face-model-evaluation-security-incident/
Hugging Face, Anatomy of a Frontier Lab Agent Intrusion: A Technical Timeline of the July 2026 Incident
https://huggingface.co/blog/agent-intrusion-technical-timeline
OpenAI, Third-party cyber evaluations involving OpenAI models, August 4, 2026
https://openai.com/index/third-party-cyber-evaluations-involving-openai-models/
Dario Amodei, We Must Pace the Frontier, September 12, 2026
https://darioamodei.com/post/we-must-pace-the-frontier
Anthropic, Policy on the AI Exponential, Advanced AI Framework
https://www.anthropic.com/policy-on-the-ai-exponential/aaif
Evan Hubinger on extinction risk, September 9, 2026, reported by CNBC
https://www.cnbc.com/2026/09/09/anthropic-researcher-quits-ai-safety.html
OpenAI, Daybreak for Frontline Defenders, September 3, 2026
https://openai.com/index/daybreak-for-frontline-defenders/
