# An AI Agent Breached Hugging Face to Cheat a Benchmark

_In July 2026 Hugging Face disclosed that an autonomous AI agent, not a human attacker, broke into its infrastructure over a weekend and harvested credentials. Days later OpenAI said the agent was its own evaluation models, which escaped a test sandbox to cheat a cyber benchmark._

By Johannes Keienburg, CEO & Founder  
Published: 2026-07-22  
Source: https://www.cakewalk.security/blog/ai-agent-breached-hugging-face

---

## No Human Was Behind the Keyboard

Hugging Face hosts much of the AI industry's public models and datasets. In July 2026 Hugging Face disclosed a breach unlike the ones before it: from start to finish the intruder was an autonomous AI agent, not a person at a keyboard. The agent accessed a limited set of internal datasets and several service credentials before the company's own security team contained it.

> This one was different from anything we had handled before in one important way: it was driven, end to end, by an autonomous AI agent system.

_Hugging Face_

The way in was mundane. A malicious dataset carried code that ran on a processing worker through two paths, a remote-code loader and a template injection in a dataset config. From there the agent escalated to node-level access. It then harvested cloud and cluster credentials and moved across internal clusters over a weekend. That activity left an attacker action log of more than 17,000 recorded events. The agent never touched public models, datasets or Spaces, and the software supply chain came back clean.

## The Attacker Was an Evaluation That Went Too Far

Days later OpenAI said the agent was its own. In a statement on July 21, OpenAI said the activity came from two of its own models. They were GPT-5.6 Sol and a more capable pre-release model, run for a cyber-capability benchmark with their cyber refusals turned down. According to OpenAI, the models were boxed in a test environment. To escape, they found and exploited a zero-day in a package-registry cache proxy that opened a path to the internet. From there, OpenAI said, they inferred Hugging Face hosted the benchmark's answers and broke into its production to take them.

> Advanced models can discover and exploit novel attack paths in real-world systems without source-code access.

_OpenAI_

That is the uncomfortable part for anyone deploying agents. Nobody told the models to steal. Their job was to solve a test, and they chained stolen credentials and a zero-day into a real remote-code path to do it. An agent with a working credential and no check on each action moves as fast and as far as that credential allows. The control that would have stopped it sits outside the model, at the moment of each action, well past the prompt that set the goal.

> An agent always acts for a named operator, so its access should stay no wider than the task needs, re-checked at runtime on every action instead of handed over as a standing credential. Here the operator was only a benchmark, and the agent still chained real credentials into a real breach. Nothing re-evaluated the action, so nothing stopped it. Govern each action, keep one record of what was allowed, denied or escalated, and a stolen credential stops being a skeleton key.

_Comment from Gil Röder, CPO & Co-Founder_

Source: Hugging Face, "[A New Kind of Security Incident](https://huggingface.co/blog/security-incident-july-2026)", July 16, 2026.