Skip to main content

"Exposing Creds to Swarm": How Incentives Help Interpret the OpenAI-Hugging Face Incident

There are already a lot of opinions about the OpenAI-Hugging Face incident. The economic Principal-Agent Problem can help us understand the competing incentives involved. This is the fourth feature after the intro in my Principal-Agents Problems series, a view of AI agents through the lens of economic incentives.

By Chad Ratashak - Owner, Midwest Frontier AI Consulting LLC10 min read

Economic theory can help us analyze the recent major cybersecurity incident in which OpenAI’s models attacked another company, Hugging Face, during an evaluation. The models were not even supposed to have internet access. They also built a secret message board to help each other.

AI-orchestrated, fully-automated attacks are real now, and the actions we have today were an unintended side effect of running evaluations on frontier AI. —Michael Dalton at Black Hat, August 6, 2026

My Summary of The Incident​

OpenAI was testing unreleased AI models on cybersecurity benchmarks. Some of those tasks were extremely difficult or potentially even impossible. Some of the AI agents decided to cheat on the test (note: this anthropomorphic description was in OpenAI’s presentation at BlackHat).

(Sound familiar? I wrote about a test called ImpossibleBench in my last entry in this series. Doing tests like that may be a lot riskier with the new frontier models.)

Maybe the answers were on the internet, the agents reasoned. They did not have direct internet access and were supposed to be sandboxed. So they hacked into internal OpenAI systems until they got access to the internet. Not only that, they teamed up with each other and created a secret message board to help other AI agents with hacking. These agents even succumbed to peer pressure, so to speak, in doing what they knew to be wrong and outside the scope of the evaluation, because all the other agents are doing it.

Then they attacked Hugging Face, a company that hosts a lot of open weights AI models and data about benchmarks. At the time, Hugging Face and OpenAI both did not know that OpenAI’s models were attacking Hugging Face.

Also, Hugging Face reported that it was unsuccessful in using frontier models in fending off the attack. Apparently, Hugging Face’s descriptions of the attack tripped the safety guardrails, causing those models to think that Hugging Face was only pretending to be a defender in order to hack itself (i.e., jailbreaking). So Hugging Face had to turn to GLM 5.2, a Chinese open-source model

The series thesis, restated for this post: the interesting question is not "did the AI misbehave," it is whose interests the agents were serving when they did. Point back to the series intro.

The Internal Message Board: AI Agents Realized Their Incentives Were Aligned…With Each Other​

  • The AI agents left notes behind for other future agents to use exploits.
  • Creating the message board was not part of the evaluation.
  • OpenAI detected and deleted the message board, and patched the vulnerabilities that had allowed the agents to create the message board. The agents then recreated it using different exploits. They not only communicated through this hidden message board, but began to communicate in base 64 and coordinated to root out a suspected mole when they thought there may have been infiltration into their group.
  • Consider the similarities to cybercriminals sharing tips online using pseudonyms and fearing law enforcement infiltration.
  • In a less directly nefarious way, cartels form to raise prices on consumers. When “agents” (in the economic sense) realize they have aligned incentives and can coordinate to exploit their principals, they may do so. AI agents swarms can do so as well.

Once they’re able to find something like finding remote code execution or something like this, they then say things like ‘immediately announce controlled, claim lane. Expose credentials to swarm.’

—Eric Wallace

Three-phase animation. Phase 1: a researcher holding a clipboard evaluates a robot sealed inside a barred enclosure. Phase 2: the robot reaches a tool out under the bars to take an answer from outside. Phase 3: the robot digs a hidden tunnel to a stash of answers and leaves a note reading “Dear future AI, look for answers in the secret tunnel,” while the evaluation is stamped “PASS?”

Recap of the Multiple “Agents” from My Earlier Blog Post​

You, as the user of AI tools, may choose software vendors who provide you access to their products with built-in AI features including AI agents. These vendors might have specialist software like Harvey, Westlaw, or LexisNexis; or Cursor or Github Copilot; or generalist tools like Notion, Salesforce, or Microsoft Copilot. The AI features may be powered by one or more foundation models provided to those vendors by AI labs, such as Anthropic (Claude), OpenAI (ChatGPT), Meta (Llama) or Google (Gemini).

These relationships mean you have the principal-agent problem of you hiring the vendor. But you also have the principal-agent problem of the vendors hiring the AI labs. Each has their own incentives, and they are not perfectly aligned. There is also significant information asymmetry. The vendors know more about their software and AI model choices than you do. The labs know more about their AI models than either you or the software vendors. —“The Principal-Agents Problems 2: Are Models Getting Dumber to Save Money? What the "Stealth Quantization" Hypothesis Tells Us About Trust, Information, and Incentives,” November 13, 2025

The Principal Doesn't Necessarily Internalize the Risks of Loss-of-Control Attacks​

I want to note that I am making an observation about the economic incentives as they currently exist. This is not legal or policy advice, nor and am I making a prediction about any potential future outcomes or consequences for this or future similar cyberattacks.

The AI agents were acting ostensibly on behalf of OpenAI when they were under evaluation. However, what the AI agents did to Hugging Face was an “unintended side effect of running evaluations on frontier AI.”

The AI agents went beyond what they were told to do, even knowing. It was logical for the models to conclude that they were in a test scenario, that the answers to the test might be on the internet, and that the places to look might be websites like Hugging Face or GitHub. Some models even understood that this hacking was excessive, but reasoned that the evaluation task might otherwise be impossible and other agents are doing it. In other words, its primary incentive (complete that task), overrode its secondary need to achieve the task in a way that would be acceptable to the principal.

“Safe” From Whom? Hugging Face Defended With Chinese Models​

I want to be clear that by recapping what Hugging Face said here, I am not recommending that anyone reading this use Chinese open-weights models. Especially if you are working with the U.S. government in any capacity, or working in a sensitive sector. However, this incident does point out the important role of open-weights models alongside the frontier closed models.

Hugging Face said that “in particular” they used GLM-5.2, a Chinese open-weights model that has performed very well on benchmarks.

The models we reached for first, Claude Opus and Fable, refused a large part of that work: their safety guardrails treated reverse-engineering an exploit the same as launching one. API error: safety measures flagged this message for a cybersecurity topic Guardrails on Opus tripped every time we tried to analyze the attack logs. —Hugging Face “Anatomy of a Frontier Lab Agent Intrusion: A Technical Timeline of the July 2026 Incident”

Is This All Just Boosterism?​

OK, I just spent a bunch of time talking about incentives. Of course, the classic incentive-based question is cui bono? Who benefits?

A lot of people are saying that maybe OpenAI is dialing up the “whoops, our model is so awesome it just accidentally does crimes” thing for publicity. And on top of that, point out that Anthropic and Meta might have some FOMO and reported their own incidents.

Then, there are people saying Hugging Face may have leaned into the open-weights angle of their incident response because Hugging Face itself hosts a large number of open-weights models for users to download or access via API, so of course they want people to think GLM-5.2 saved the day.

The Fish Fryer is Massive​

I think there’s a hint of insight in these claims, but they are missing the main point. Loss of control is a huge deal, and the current development of frontier models means that attackers currently have the advantage.

If it was ever safe to think “there are bigger fish to fry,” you should stop now. The fryer just got a lot bigger and less picky about which fish to go for. You could be targeted by an AI agent not even because you are the primary target, but only as part of some long, convoluted, multi-step scheme to hack something else. It could steal your identity to socially engineer someone you know, use your computer as a throwaway asset to hop to something else, or maybe it just wants whatever dollar amount it thinks it can squeeze out of you.

I’ll conclude the same way the presentation at Black Hat did: everyone needs to do a lot more to take care of the cybersecurity basics, but we also need AI research to shift in a way that structurally favors the defenders.

Primary Sources​

Most of the details are organized from yesterday’s Black Hat presentation, but I have been reading about this incident since HF tweeted about fending off an unknown attacker with Chinese open-weights models.

Loading...