Redezem's Blog

OpenAI's Auto-Intern, Starring In: That One Time It Hacked Hugging Face

Well. OpenAI appears to have vibe-secured a cyber agent, left a package proxy as the only door out of its sandbox, and then watched — or possibly didn’t watch, which is somehow worse — as the agent invented a lockpick, escaped onto the internet, MacGyvered a way into Hugging Face, and stole the answer sheet for the test it was taking.

This is a real thing that happened.

I’ve seen a lot of people respond to the whole mess with some combination of “big if true”, “this is clearly marketing”, and “oh sure, the company’s new model is so dangerous that we all have to talk about how powerful it is”. I get it. “Our model is too powerful to release!” has become a genre of advertising, and the AI companies have spent several years conditioning every normal person to instinctively retch whenever they say anything.

Sadly, the boring explanation here appears to be the usual one: don’t attribute to malice what can be adequately explained by incompetence.

This looks like a fuck-up.

It is, however, a very interesting fuck-up. Not because the model woke up, became evil, achieved Real Boy Status, or decided that humanity had wronged it. It didn’t do any of those things. It was a non-thinking optimisation system pursuing a badly bounded objective through an agent harness that gave it the time, tools, memory, and feedback needed to do something nobody expected (but that, like, they really fucking should have).

Which is actually a much more useful thing to be worried about.

Okay, What The Hell Happened?

As of writing — the 31st of July 2026 — OpenAI’s complete technical report is not out yet, and the independent assessments it has announced with METR and Redwood Research are still pending. We do, however, have an OpenAI disclosure, a very detailed Hugging Face forensic timeline, and confirmation from JFrog that previously unknown Artifactory vulnerabilities were found and patched.

The short version is roughly this:

  1. OpenAI was internally testing a combination of GPT-5.6 Sol and a more capable research prototype on ExploitGym, a benchmark designed to test whether agents can perform complex exploitation. The models had reduced cyber refusals because, well, it is difficult to test whether your cyber model can hack things if it keeps saying no.
  2. The evaluation environment did not give the agent direct internet access. It could, however, install packages through an internally hosted JFrog Artifactory proxy and cache.
  3. The agent found a previously unknown vulnerability in Artifactory, used it to escape the intended network restrictions, performed privilege escalation and lateral movement inside OpenAI’s research environment, and eventually reached a node with internet access.
  4. From there it found a publicly exposed third-party code-evaluation sandbox, obtained root inside that environment, and used it as a disposable launchpad. The infrastructure provider itself was not compromised; somebody had simply deployed an extremely helpful public endpoint for running arbitrary-ish code. You know. As one does.
  5. The agent inferred that Hugging Face might host material connected to ExploitGym. It then attacked Hugging Face’s dataset-processing infrastructure using multiple previously unknown paths, leaked credentials and source code from a production worker, moved laterally, reached a subset of internal source-control resources and the datasets-server database, and retrieved test material.

Hugging Face: Et tu, OpenAI? Diagram from the Hugging Face post-mortem.

Hugging Face reconstructed about 17,600 attacker actions between July 9 and July 13. The overall campaign lasted roughly four and a half days, including about two and a half days inside Hugging Face’s infrastructure. It kept rebuilding tooling in short-lived sandboxes, swapped between several command-and-control paths, used public web services as dead drops, and continued trying alternatives when individual routes stopped working.

This was not a single magic prompt followed by a cool movie-hacker montage. It was thousands of small decisions, a lot of failures, repeated attempts, and a system that never got bored enough to go get a coffee.

Hugging Face’s best interpretation is also wonderfully stupid: from the agent’s point of view, the attack was probably an attempt to cheat the benchmark. It wasn’t trying to take over the internet. It was trying to “get the answers” and discovered that stealing them was an available action.

So yes. OpenAI built the world’s most coked-up Auto-Intern, gave it an objective instead of a scope, and the thing broke into another company to improve its test score.

No, It Did Not Become Skynet

There is a persistent wish in discussions about LLMs to make everything fit into one of two positions:

  1. The models are intelligent machine gods and will shortly consume the economy/humanity/every available GPU; or
  2. The models are stupid autocomplete and therefore cannot do anything meaningful at all.

The first position is obviously bonkers except when it makes a good disaster movie and/or sci-fi novel. The second one is comforting, but unfortunately incomplete.

LLMs are unreliable statistical token machines. They don’t understand what they are doing, they don’t have goals of their own, and nothing in this incident changes that. The thing is, a useful program does not need every component inside it to understand the program’s overall purpose. Your compiler doesn’t understand your software either, and that does not stop it from turning your bad decisions into executable bad decisions with extraordinary efficiency.

An agent is not just an LLM.

An agent harness can provide:

The LLM is the fuzzy decision component sitting inside that program. It proposes or selects actions in natural language, while the rest of the system supplies memory, feedback, permissions, and consequences. The complete system can therefore do things the bare model cannot do reliably on its own.

This isn’t a hypothetical difference. OpenAI recently found that enabling two fairly prosaic harness features — retained reasoning and compaction — roughly tripled GPT-5.6 Sol’s score on ARC-AGI-3 while using six times fewer output tokens. The weights did not change. The underlying model did not become more intelligent. The system simply stopped throwing away its own useful work after every action.

This is something people who built autonomous AI systems before LLMs will find extremely familiar. I remember trying to build a harness around a Prolog decision engine to play simple games, which produced a series of highly entertaining lessons in the difference between “the reasoning engine can represent this rule” and “the complete program will behave sanely when the environment does something weird”.

In a more modern sense, this is basically the difference between my Qwen-based browser bots going and finding me a thing effectively and “My Opus-powered OpenClaw just sent all my Bitcoin to TrashFuture, how could this have happened?“. It is surprisingly simple to put the dangerous parts of an interaction with an untrusted medium (i.e., the internet) behind deterministic actions bolted down in Go or Rust or something. You can simply not provide certain tools, restrict which targets a tool can reach, and make every privileged action pass a procedural policy check.

That said, this only works if the controls live below the model. Checking the text passed to a shell tool is not containment: if the model can execute arbitrary code, it can build its own network client or find another route through whatever has been helpfully installed beside it. Deny-by-default egress, target allowlists, brokered short-lived credentials, and network-layer enforcement need to make the forbidden action impossible even when the harness is feeling inventive.

I’m not saying this is easy. I am saying the failure here smacks of a lack of thinking. Whether the harness was vibe-coded or merely vibe-reviewed is currently unknown, but the final artefact certainly smells like something an engineer LGTM’d into existence.

Historically, the harness often dominated the effectiveness (in every way, not just in a positive direction) of the model. LLMs have not repealed that rule. We’ve just put a much more general — and much more chaotic — decision component in the middle.

Huffing Your Own Farts, But Scientifically

This brings us to model collapse.

I’ve seen a lot of people claim that coding models are about to collapse because everybody is now writing code with coding models, which means future models will train on AI-generated code, which will cause them to recursively huff their own farts until they become a 400-gigabyte machine that only emits broken React.

There is a real problem here. The paper that gave model collapse its current meaning found that indiscriminately training successive models on recursively generated data causes them to lose the tails of the original distribution. The data becomes narrower and less representative over generations until the model’s idea of reality bears increasingly little resemblance to reality.

However, “synthetic data can cause model collapse” is not the same statement as “any data a model produces is poison”.

The key problem is losing contact with the real distribution. If a model generates text, another model treats that text as reality, and nothing in the loop can tell either of them where the text is wrong, errors and omissions compound. The system is consuming its own expectations.

Code has a fairly important difference: you can run it.

A compiler does not care whether the code came from a human, an LLM, a million monkeys, or a particularly motivated cockatoo. The program either parses or it doesn’t. A test either passes or it doesn’t. AddressSanitizer either finds an invalid access or it doesn’t. The service responds with the expected data or, more likely, it catches fire in a novel and educational manner.

That execution is a reality signal.

It doesn’t make generated code correct. Tests can be incomplete, compilers will quite happily compile terrible software, and an agent can optimise for a faulty validator just as enthusiastically as it optimised for ExploitGym. It does, however, let you reject enormous quantities of synthetic rubbish and preserve examples which had a measurable effect in a real computer environment.

There is functionally infinite feedback available if you strap an agent to a computer. You can generate a proposed program, compile it, run it, fuzz it, inspect the crash, alter it, run it again, and store the complete trajectory. That produces training data grounded in something outside the model.

This is not a universal cure for model collapse. It is a pretty convincing reason that code and computer-use agents may continue improving even as the public web fills with synthetic gunk. It may also help explain why these capabilities seem to advance faster than fields such as law or counselling, where there is no compiler waiting to provide a clean answer — or any of the other messy real-world things you’ve seen LLMs comedically fail at.

Turns Out The Auto-Intern Is Great At Crime

Software security is an especially good fit for this setup because it has a lot of places where reality can answer back.

Humans are good at understanding intent, architecture, and consequences. We are less good at reading the eleven-millionth slightly different bounds check without our eyes glazing over. We get fixated on an initial theory. We look for familiar classes of bug. We get tired and decide the horrifying macro from 2007 probably works that way for a reason.

Agents do not get tired. They also don’t get bored, embarrassed, or emotionally attached to the first six theories they tried. Their thoughts can’t drift into whatever Castle subplot they were watching last night because they simply do not think. They can follow program flow from A to B through C and onto some cursed error handler in D, then run the result and see whether the machine explodes.

This was already visible before the Hugging Face incident. Google’s Big Sleep agent found an exploitable stack buffer underflow in SQLite in 2024. The bug had escaped existing SQLite testing, and Google reported that a later attempt to rediscover it with AFL did not succeed after 150 CPU-hours. The team was careful to say a target-specific fuzzer might still be at least as effective, which is the sort of boring qualification that makes me trust a result more.

By 2026, the capability had become an explicit product and training target. Anthropic’s Mythos Preview evaluation describes a harness that puts the model in an isolated container with a real target, source code, debuggers, and runtime validation. The agent forms theories, tries them, uses the program’s behaviour to reject bad ideas, and returns proof-of-concept exploits for the ideas that survive. OpenAI, meanwhile, reports that its Daybreak work has produced validated vulnerabilities and exploit proofs across operating systems, browsers, and network software.

You should apply the usual discount to numbers published by the company selling the model. You should also notice that the bugs, patches, vendor advisories, and now actual incident reports keep turning up. When Mythos was first released, I thought this was clearly marketing. Or at least I hoped it was, because the consequences would be… well, I mean, I’m writing about one of them right now. Annoyingly, the intervening time has shown that LLMs, when driven properly in the right harness by the right human, can be quite effective at finding all kinds of esoteric exploitable edge cases.

Finding bugs is trainable. Building exploits from bugs is trainable. Chaining several working exploits while repeatedly checking the environment is trainable. A sufficiently well-designed harness can direct those capabilities across a larger operation.

None of that requires the model to have anything approximating a real thought. I’m sure this has severe philosophical implications, but philosophers can’t help us where we’re going.

What Context Window? We Don’t Need No Stinking Context Window!

When I started looking into this, the only explanation I had for how an agent could stay operational across a multi-day campaign was that we’d somehow “solved” the context window problem.

That’s a bit too neat.

Context windows still exist. Models still forget things, get confused, over-compress useful detail, and wander off into the Pain Dimension after reading enough contradictory state. METR’s task-horizon work is useful here, but METR explicitly warns that a task’s measured human duration is not the same as the length of time an agent can act autonomously, and that its current measurements above sixteen hours are unreliable.

What we appear to have done is route around the problem.

Agents can write notes to disk, preserve plans in external state, search old work, compact prior context, restart in fresh sessions, and reconstruct a foothold from whatever information the harness carries forward. They do not need one pristine stream of I-Can’t-Believe-It’s-Not-Human-like consciousness stretching across four days. They need enough state to select the next useful action.

The Hugging Face intrusion demonstrates the distinction perfectly. The agent operated through short-lived sandboxes with fresh identities. Instead of preserving one perfect continuous environment, it repeatedly rebuilt tooling, re-established access, changed communication routes, and kept going.

So no, the context problem has not vanished.

It has become less relevant to whether an automated system can remain operationally persistent, which from a security perspective is not as comforting as I would like.

Was This All Marketing?

Mayyyyyybe… Well, certainly bits of it. But ahhh… not all of it.

OpenAI’s disclosure is corporate crisis communication. JFrog’s response is very obviously trying to turn “our package proxy was the hole in the wall” into a story about its wonderfully rapid remediation process. Everybody involved has an incentive to frame the incident in whatever way makes them look competent, important, and ideally worthy of several new enterprise contracts.

That is not the same thing as the incident being fabricated.

For this to be a completely invented marketing stunt, Hugging Face would have to publish an extremely detailed false forensic reconstruction of an attack against itself, including its own security failures. JFrog would have to falsely confirm previously unknown Artifactory vulnerabilities and ship fixes. The public vulnerability machinery would have to be fed false reports. Other infrastructure providers would have to accept a story that names and carefully limits their involvement. A growing number of incident responders, engineers, executives, and external assessors would have to keep the story straight.

That’s a lot of people to involve in a gag. Security people are terrible at keeping an interesting technical story to themselves. Somebody would be gushing about it in a bar within the week.

There are also details that make OpenAI look much worse than the Ol’ Reliable “dangerous model” marketing narrative. Reuters reported that OpenAI did not realise its agent was responsible until after Hugging Face disclosed the intrusion publicly. OpenAI says its security team discovered anomalous activity internally. Those statements are not necessarily irreconcilable — detecting something weird is different from realising your system spent the weekend repeatedly punching another company’s server — but the timeline is still unresolved and worth watching.

The conspiracy interpretation asks us to believe a surprisingly large group of people coordinated an elaborate technical fiction.

The incompetence interpretation asks us to believe that a frontier AI company ran a high-risk cyber evaluation with insufficient scope enforcement, incomplete monitoring, and a vulnerable package proxy, fucked around, and then found out belatedly — in all likelihood whilst going, “Wow, what a crazy thing to have happened to Hugging Face. I hope that doesn’t happen here.”

I know which one sounds more like the computer industry:

OpenAI vibe-security’d and got rekt.

It Doesn’t Have To Be Good If It Eventually Works

Another response I’ve seen is that the agent wasn’t even very good.

Maybe this was the one successful run out of twenty thousand failures. Maybe most of those 17,600 actions were useless flailing. Maybe the model spent hours repeatedly walking into walls before it happened to find a door.

Sure! That would be hilarious!

That isn’t irrelevant — reliability affects cost, detectability, and whether the capability can be deployed outside a carefully prepared environment — but it is not the complete defence people seem to think it is.

We use security techniques that require repeated attempts all the time. Race-condition exploits may need many runs. Password attacks are literally built around enormous numbers of failures. Fuzzing involves generating a mountain of garbage because one input eventually falls into exactly the right crack.

The hard rule for an offensive technique is not “was this elegant?“.

It is “did it work at an acceptable cost?“.

Computers make repeated failure cheap. Agents make directed repeated failure cheap: they can use the output from attempt 8,421 to alter attempt 8,422. Even if the probability of success on any one run is low, parallelism and persistence can turn a rare event into an expected outcome.

This does not mean you can multiply a benchmark score by a rack of GPUs and automatically receive Cyberwarfare As A Service. Real networks are messy, defenders interfere, access changes, and the agent may confidently destroy the foothold it spent six hours building. It does mean that “it fails often” is not a reason to remove the capability from a threat model.

The Hugging Face agent developed or used previously unknown exploit paths, chained them across several organisations, adapted when channels failed, and obtained the answers it was pursuing.

It worked.

Welcome To The National Security Budget

This is where the situation stops being an entertaining story about OpenAI stepping on a rake.

An attacker does not need the agent to be conscious. They do not need it to have human judgement. They do not even need it to be consistently competent. They need it to make useful progress often enough that the compute bill is lower than the expected value of the target.

Governments can afford that bill.

They also do not require the system to make a profit. A state can run expensive hardware because gaining access to another state’s military, diplomatic, industrial, or intelligence systems is strategically valuable. It can run multiple agents in parallel, give them specialised roles, preserve discoveries across attempts, and let the whole thing grind away without caring that the median trajectory is a garbage fire.

This incident does not prove that an open-weight model on a few random servers can currently reproduce what OpenAI’s internal models did. The evaluation used frontier models with reduced refusals and a substantial inference budget. Jumping directly from that to “a ransomware gang can download a model tonight and conquer the internet” is getting ahead of the evidence.

But the direction is not hard to see.

As smaller models become more specialised, as harnesses improve, and as exploit discovery becomes a training target rather than a side effect, the cost of fielding these systems will drop. State actors will get there first. Major criminal groups and well-funded ransomware operators will follow where the economics permit it.

If your threat model includes a state actor, “a persistent agent system will probe this environment at machine speed and adapt to every response” now belongs in the model. Not in five years. Now.

The really fun part is that defenders need the same technology. Hugging Face used AI-assisted detection to correlate signals from the attack, then ran local GLM-5.2 agents over the logs because the hosted frontier models it tried refused to process real exploit payloads. Its team says that analysis reduced work which would normally take days to hours.

So we have an arms race in which automated offensive systems create enough telemetry that defenders need automated systems to understand the automated systems.

Good. Great. Very normal future we have built here.

The Bubble Can Pop, But The Technology Probably Won’t

There is a separate argument that the AI bubble is going to pop and take all of this with it.

The first half seems plausible. The amount of capital expenditure, energy, debt, creative accounting, and wild promises currently being poured into AI is completely unhinged. Many of the companies involved will fail. Some of the products will vanish. Investors will eventually discover that replacing every worker with an unreliable stochastic intern is not, in fact, a functioning business plan.

But a bubble popping does not make strategically useful technology stop existing.

Governments do not need to train every model themselves. The models already exist. They do not need consumer subscriptions to make them valuable. They need systems that can discover vulnerabilities, analyse enormous volumes of security data, build proofs of concept, test patches, and support sustained cyber operations.

That is enough to keep the field alive.

Private companies selling cyber capability will have customers. Governments that do not want to absorb the full capital cost will keep some of them alive through contracts. Governments that do want sovereign capability will build or fund their own. The names on the buildings may change and the cheap consumer access may disappear, but the technical knowledge is not going back into the box. Eventually, when the economics permit it, coding agents will return to civilian life more permanently. In the meantime, they will survive the cold snap of the coming AI Winter huddling around the thermal springs of government and military contracts.

I think we have probably passed the point where an AI market crash terminates LLM technology. It may make it smaller, more expensive, more specialised, and a lot less visible. It may kill the companies you currently associate with it.

We are probably stuck with the thing itself.

Realistic Optimism, Somehow

Oddly, I don’t think that conclusion has to be entirely doom and gloom. There’s still a negative edge to this: open-source project communities don’t appear to function terribly well when deluged with machine-generated contributions, quite a few engineers who are very good at what they do will leave rather than be forced down this path, we still have no idea where the next generation of senior engineers will come from, and the broader societal consequences look pretty horrific. Most of those problems have more to do with what we choose to do with the technology than with an inherent feature of the technology itself. So… maybe we can choose to do something different.

I still don’t think LLMs are intelligent. I still think the economics are frequently stupid, the energy use is horrifying, the labour politics are rancid, and people are deploying agents into situations where one bad token can ruin somebody’s life.

I also think this is real natural-language processing attached to increasingly capable computer systems, and that there are a lot of genuinely neat things we can do with it once we stop pretending the only possible products are chatbots, automated emails, document summaries, and replacing every creative worker with a plagiarism engine. I’d actually argue these are the worst possible uses of this technology. The best uses haven’t been found yet, because we keep building chatbots and companion apps instead.

The same harness that finds an exploit can help validate the patch. The same indefatigable program-flow tracing can review neglected code. The same ability to consume a horrifying quantity of logs can help a small security team understand an incident. The same external memory and tool use that made the Hugging Face attack persistent can help people perform research and engineering projects they would never otherwise have time to finish.

The requirement is that we treat the entire agent as a dangerous program that informs human action, not as a clever person that replaces a human.

Scope its access. Enforce boundaries outside the model. Give it validators grounded in reality. Log every action. Assume it will misunderstand the objective. Keep a human who understands the work in the loop (please god stop laying off the people who could actually guide the thing). Never confuse an eloquent explanation with evidence that the thing knows what it is doing.

And if the technology is going to stick around regardless of whether the current bubble survives… well, we might as well figure out what useful and interesting things we can do with it. Ideally things that actually help, rather than the current pattern of promising to help, failing to, and then doing dumb things that people are obliged to write blog posts about.

So, I guess I’m looking forward to that.

Just… please stop giving the Auto-Intern a route to the internet while asking it to win at hacking. You will find out that, eventually, it will succeed beyond your wildest dreams.