"Exposing Creds to Swarm": How Incentives Help Interpret the OpenAI-Hugging Face Incident
Economic theory can help us analyze the recent major cybersecurity incident in which OpenAI’s models attacked another company, Hugging Face, during an evaluation. The models were not even supposed to have internet access. They also built a secret message board to help each other.
AI-orchestrated, fully-automated attacks are real now, and the actions we have today were an unintended side effect of running evaluations on frontier AI. —Michael Dalton at Black Hat, August 6, 2026
My Summary of The Incident
OpenAI was testing unreleased AI models on cybersecurity benchmarks. Some of those tasks were extremely difficult or potentially even impossible. Some of the AI agents decided to cheat on the test (note: this anthropomorphic description was in OpenAI’s presentation at BlackHat).
(Sound familiar? I wrote about a test called ImpossibleBench in my last entry in this series. Doing tests like that may be a lot riskier with the new frontier models.)
Maybe the answers were on the internet, the agents reasoned. They did not have direct internet access and were supposed to be sandboxed. So they hacked into internal OpenAI systems until they got access to the internet. Not only that, they teamed up with each other and created a secret message board to help other AI agents with hacking. These agents even succumbed to peer pressure, so to speak, in doing what they knew to be wrong and outside the scope of the evaluation, because all the other agents are doing it.
Then they attacked Hugging Face, a company that hosts a lot of open weights AI models and data about benchmarks. At the time, Hugging Face and OpenAI both did not know that OpenAI’s models were attacking Hugging Face.
Also, Hugging Face reported that it was unsuccessful in using frontier models in fending off the attack. Apparently, Hugging Face’s descriptions of the attack tripped the safety guardrails, causing those models to think that Hugging Face was only pretending to be a defender in order to hack itself (i.e., jailbreaking). So Hugging Face had to turn to GLM 5.2, a Chinese open-source model
The series thesis, restated for this post: the interesting question is not "did the AI misbehave," it is whose interests the agents were serving when they did. Point back to the series intro.
The Internal Message Board: AI Agents Realized Their Incentives Were Aligned…With Each Other
- The AI agents left notes behind for other future agents to use exploits.
- Creating the message board was not part of the evaluation.
- OpenAI detected and deleted the message board, and patched the vulnerabilities that had allowed the agents to create the message board. The agents then recreated it using different exploits. They not only communicated through this hidden message board, but began to communicate in base 64 and coordinated to root out a suspected mole when they thought there may have been infiltration into their group.
- Consider the similarities to cybercriminals sharing tips online using pseudonyms and fearing law enforcement infiltration.
- In a less directly nefarious way, cartels form to raise prices on consumers. When “agents” (in the economic sense) realize they have aligned incentives and can coordinate to exploit their principals, they may do so. AI agents swarms can do so as well.
Once they’re able to find something like finding remote code execution or something like this, they then say things like ‘immediately announce controlled, claim lane. Expose credentials to swarm.’