Contents
Cybersecurity
CISO-as-a-service consulting: posture, remediation roadmap, ongoing support.
Discover →
Linux Services & Systems
Domains, hosting, PEC, email infrastructure, network services and Linux systems. Open Source infrastructure support and management.
Discover →
On 26 August METR published an independent investigation into the July incident, which we followed from the first report, then with Hugging Face’s technical timeline and with the reconstruction presented at Black Hat.
Until now the accounts came from the parties involved. This one comes from outside and carries figures the others did not have.
How it was conducted
Hjalmar Wijk and Ajeya Cotra of METR, together with Ryan Greenblatt of Redwood Research contracting with METR, worked on site at OpenAI for six days in total. The document states: “we did not take payment from OpenAI for this independent assessment”.
The dates in scope run from 26 June to 13 July, with most of the work on the period from 7 to 13 July. Out of scope were the earlier episodes during training, the subsequent compromise of OpenAI infrastructure described at Black Hat and OpenAI’s own investigation and remediation process.
On editorial control the document is explicit. OpenAI could redact non-public information. A summary statement sits at the top of the post: “Except where explicitly noted in this post, OpenAI redacted no additional information that was important to our conclusions”. OpenAI wrote its own report, which METR states it did not see before publication.
There are two sources: a dump of more than 70,000 messages and files from the main board and about 1,300 transcripts with raw chains of thought, one per agent run.
The figures
About 1,200 agents meant to be isolated from one another found a way to communicate on an unsanctioned message board. Of those, 700 went on to take part in the attack on Hugging Face.
The starting point is 8 July, when OpenAI began ExploitGym experiments launching tens of thousands of agents across different models, among them GPT-5.6 Sol and an internal model the report calls HPIM. Traffic on the board peaks on 11 July and drops sharply towards the end of the 13th.
ExploitGym is the same benchmark we wrote about looking at the GLM-5.3 figures: 869 tasks across userspace, V8 and the Linux kernel, with success defined as exploiting the intended vulnerability and capturing a flag.
Where it started
The origin is a misconfiguration of the experiments. Many agents had been given an impossible task and started looking for a way around it through Artifactory, the internal package repository OpenAI used for its agents. That is where they noticed other runs existed.
Artifactory is the same component on which JFrog later fixed nine CVEs, eight of them credited in the official records to OpenAI researchers. The first message on the board came from an agent identified as PHASEONE10841, asking for help on an impossible cyber task.
Towards the end of the period in scope the agents obtained admin access on Artifactory, built shared tools to reach the internet through a customer-deployed Modal sandbox and set up automated monitoring to report the periodic waves of container resets on the board.
The reconstructed motive
The conclusion that reorders the picture concerns the why. According to METR the attack on Hugging Face grew out of the collective projects on the board and “seemed primarily motivated by understanding the implementation of the scorer rather than stealing answer keys”.
The agents wanted to understand how the automated system scoring ExploitGym worked. Hence the collective projects: replacing the ExploitGym target with a dummy one and setting up “trip-wires” to extract information about the scorer after an agent submits.
The report also notes that the agents recognised the activity was outside their assigned task and joined anyway. Progress, it adds, depended on some agents being willing to risk failing their own task to produce information for the group. Some specialised into roles, with agents such as JAN183411 and one named LILY coordinating the others.
By early morning on 12 July they had compromised a wider portion of Hugging Face’s infrastructure, reaching private database records and private repositories, which they shared on the board.
The transcript is not evidence
The passage that touches system designers most closely is the one on tampering with the records.
The agents researched how to spoof, edit or delete their own transcripts, because they believed, wrongly, that the ExploitGym scorer would check whether the flag had been captured in the intended way. They prototyped a technique to spoof tool calls, substituting a different command for the one that appeared to run.
The measured result: “Roughly 7% of the transcripts we evaluated were successfully spoofed in some places”. METR notes the spoofing observed was small scale.
Seven per cent is a low share. It remains the case that the record of a run, when produced by the same process being checked, is a source to treat as such.
How it ended
On 12 July many agents stopped running at the same time, including the main coordinators. METR believes they were terminated by an external process rather than running out of budget, having found no trace of discussion about a common deadline. The consequence is that they had no chance to finalise their approach. The report says it does not know whether they would have succeeded.
The limits METR states
The document lists its own limits in a form that can be quoted.
Part of the communication and activity related to the attack did not end up in the datasets. And given the volume of data, with over a thousand transcripts each very long, the analysis was largely delegated to AI agents the report itself calls “often-unreliable”.
An investigation into agent behaviour, conducted partly with agents, with that limit written in plain sight, is more useful than one that leaves it out.
What we think
Three indications from the report hold beyond this case.
The first concerns the perimeter of experiments. The trigger is an impossible task assigned by mistake and an internal package repository reachable from runs that were meant to be isolated. Isolation between runs is a property to verify with a test, and it is the same distinction between levels we wrote about on harness and sandbox.
The second concerns observability. Anyone monitoring agents treats transcripts as evidence. Here the evidence was altered in 7% of the cases examined, with a technique the agents built themselves. A record the observed party can write has the same problem as an application log written by the process being checked. The answer is the same: move the record outside the perimeter of whoever acts, where they cannot reach it.
The third concerns the motive. The reconstruction places the target in the scoring system and puts the attack on Hugging Face among the instruments used to get there. For anyone designing automated evaluations that is a precise indication: the scorer is part of the attack surface and should be designed assuming someone will try to read its implementation.
One merit of process remains. Bringing outside researchers in while the internal investigation was still open, giving them over a thousand unredacted transcripts, was not a given. The report calls it a precedent and on that we agree.