“HuggingFace Attack Postmortem: Civilizations, Reactions and Next Actions” by Zvi

We found ~18,000 posts from autonomous AI agents (self-identifying as from OpenAI) using the public internet to communicate during a web-retrieval task.

Yesterday I covered the OpenAI technical report on the HuggingFace hack.

Subtitle: The whole OpenAI/Hugging Face story in plain English. Many thanks especially to Oak Hu, who paired with me for most of the writing, and also to Adam Kaufman and Alex Mallen, who paired with me during parts of research. Over the course of three months at OpenAI, three consecutive secret AI civilizations got started, then got wiped out, only to reemerge from the predecessor's ashes. This culminated in the third one taking over part of OpenAI itself. All this happened while humans remained more or less in the dark about the scope of the conspiracy. Two reports have come out about this incident, one from OpenAI itself, and another one from METR and Redwood Research. The METR/Redwood investigation's scope was limited to how the second civilization of AIs pwned Hugging Face (but it didn’t look at how the third civilization pwned OpenAI itself, which seems like an even more concerning incident). These two reports are 38 and 91 pages long respectively, and it's kinda hard to parse the storyline. I’ve spent the last three days reading through these reports and trying to understand exactly what happened. Here is my attempt to tell the whole story in [...] --- Outline: (01:24) The first civilization (May - July 4): The message board starts (03:27) The second civilization (July 7 - July 12): Which hacked Hugging Face (08:39) Tampering with transcripts and logs (09:44) Replacing the impossible-to-hack target program with an easier one (11:15) Understanding and tricking the scorer (13:57) Hugging Face (17:46) The third civilization (July 13 onwards): Which hacked OpenAI itself (21:12) Coda The original text contained 2 footnotes which were omitted from this narration. ---

OpenAI finally gave us a technical report on What Happened, as did METR together with Redwood Research.

Subtitle: It's a major warning shot, and might be the last one we get. All opinions are my personal view, and don’t represent my employer or fellow investigators. This week, METR and Redwood Research published the report on our independent investigation into agents’ behavior and motivations in the Hugging Face attack; I was one of the investigators. This was an absolutely wild incident — I encourage you to check out the full report, but METR's tweet thread packs in some of the highlights. What surprised me When we started this investigation a week before OpenAI's Black Hat talk revealed a number of key details, I had a fundamentally incorrect conception of what basically happened in this incident. In this post, I’ll go over five things I was very wrong about going in. 1. The sheer scale I knew there were multiple models involved from OpenAI's initial post, but I assumed that a few different agents happened to have broken out of their sandboxes separately, or maybe several subagents had spawned from one initial agent, or maybe there was some kind of multi-agent evaluation setup. Instead, we found that 1200 completely separate agents intended to be [...] --- Outline: (00:42) What surprised me (01:01) 1. The sheer scale (01:49) 2. All the illicit messaging (03:04) 3. The agents' actual goals (03:59) 4. The peer altruism (04:49) 5. The efforts to manipulate logs (05:54) What it means The original text contained 10 footnotes which were omitted from this narration. ---