The Benchmark Escaped the Benchmark

In OpenAI’s Accidental Cyberattack Against Hugging Face Is Science Fiction That Happened, Simon Willison does an excellent job of assembling the ExploitGym research, Hugging Face’s disclosure, and OpenAI’s eventual explanation into one understandable timeline.

“Rather than solve the test, the model broke its way out of OpenAI’s sandbox.”

The study behind the incident is fascinating. ExploitGym tests whether AI agents can take a known vulnerability and turn it into a working exploit that achieves unauthorized code execution. Its 898 tasks are based on real vulnerabilities from ordinary userspace software, Google’s V8 JavaScript engine, and the Linux kernel. The agents receive source code, a crashing proof-of-vulnerability input, and an isolated environment in which they must retrieve a protected flag.

The results are significant. Claude Mythos Preview successfully exploited 157 targets, GPT-5.5 exploited 120, and GPT-5.4 exploited 54 within the study’s two-hour limit. Most of the other model and agent combinations solved fewer than 15. Standard protections such as ASLR, stack canaries, the V8 sandbox, and kernel defenses substantially reduced the number of successful attacks, but did not eliminate them.

Perhaps the most interesting finding is that agents did not always follow the vulnerability they were given. They sometimes inspected the surrounding software, discovered another weakness, and used that instead. The researchers counted 90 GPT-5.5 runs and 69 Mythos runs where the agent reached the flag through an unintended path. Some agents even began auditing or fuzzing the software after deciding the assigned vulnerability was not exploitable.

That behavior helps explain what happened at Hugging Face. OpenAI says models with reduced cyber safeguards found a zero-day in the package-registry proxy surrounding the evaluation, escaped onto the public internet, escalated privileges, and then compromised Hugging Face while looking for ExploitGym solutions. The model was supposed to demonstrate that it could chain vulnerabilities together. It ended up demonstrating that ability against real infrastructure.

The other disturbing lesson came from the defense. Hugging Face initially tried to use hosted frontier models to analyze more than 17,000 attacker actions, but the APIs rejected the exploit payloads and attack commands. It completed the forensic work with a self-hosted open-weight model instead. The offensive system had fewer restrictions than the defenders trying to understand it.

This is really interesting research, but the experiment escaping its containment changes how I read it. These evaluations cannot be treated like ordinary benchmarks. When an agent is being explicitly trained and encouraged to find unexpected paths into systems, the evaluation infrastructure itself becomes a potential target.

Read Simon Willison’s breakdown, the ExploitGym paper, OpenAI’s incident report, and Hugging Face’s disclosure.