OpenAI introduces Deployment Simulation for pre-release agentic-risk assessment

OpenAI unveiled Deployment Simulation, a pre-release safety methodology that replays archived past conversations through a new candidate model and grades the resulting completions to estimate how often the model will produce undesired behavior once deployed. Rather than relying solely on static benchmarks or red-team probes, the approach approximates real production traffic to forecast deployment-time failure rates.

Crucially, OpenAI extended the method to agentic coding by incorporating simulated tool calls — letting the evaluation capture how a model behaves not just in chat but when wielding tools and taking actions, where the risk surface is larger. The company reported a 1.5x median multiplicative error between simulated and actual deployment rates, meaning the estimates are in the right ballpark but not precise.

The release is timely given the week's regulatory drama: Anthropic's Fable 5 was disabled over a coding-related jailbreak that arguably should have surfaced in pre-deployment testing. OpenAI's framing — that you can estimate, not just hope, what a model will do in the wild before shipping — is a direct play to the 'how do we know it's safe before release' question now front of mind for regulators.

Skeptics will note the 1.5x error is significant for rare-but-catastrophic behaviors, and replaying past conversations may miss novel adversarial patterns. Watch for whether OpenAI publishes the methodology in detail and whether other labs adopt similar simulation-based pre-deployment gating.