“Are we existentially threatened by the type of AI misalignment seen in the OpenAI Hugging Face attack?” by Alex Mallen, Girish Gupta
The OpenAI–Hugging Face incidentShow Notes
OpenAI models recently broke through a series of security boundaries and into Hugging Face servers in order to cheat on a cyber eval. A lot of people thought it was scary because it was a clear example of AI overreaching to do something strongly unwanted[1]. Others thought it not so scary: the models were mostly operating myopically on a singular task and not harboring an ambitious long-term agenda, and so would not take especially subtle or subversive actions. We think both camps are right in their diagnosis, but the latter has too optimistic a prognosis. The myopic, unambitious misalignment that we seem to have seen here is definitely less scary than ambitious long-term goals shared between all instances, but would still pose substantial direct loss-of-control risk if the models were more capable, and is a serious indirect risk near-term. Building on Alex's previous work, in this post we’ll discuss the type of misalignment observed here, and analyze its consequences. Thanks to Buck Shlegeris, Alexa Pan, Ryan Greenblatt, and Oak Hu for feedback. Background The AI safety community often focuses attention on “schemers,” models harboring a variously defined cluster of motivations in which the AI poses risk because it intentionally [...] --- Outline: (01:17) Background (03:40) Implications (03:52) These AIs can't be trusted in an intelligence explosion (05:00) This misalignment poses direct takeover risk (07:29) What the incident tells us about takeover risk generally (08:45) The naive fixes likely make misalignment worse The original text contained 5 footnotes which were omitted from this narration. ---
First published:
July 23rd, 2026
Source:
https://www.lesswrong.com/posts/H6DDSEvrtCk8Sehfd/are-we-existentially-threatened-by-the-type-of-ai
---
Narrated by TYPE III AUDIO.
More from The OpenAI–Hugging Face incident
We found ~18,000 posts from autonomous AI agents (self-identifying as from OpenAI) using the public internet to communicate during a web-retrieval task.
Okay, so we who read blogs like this one have collectively realized there really is a lot going on right now. There is Big Trouble in Baby Superintelligence.
Yesterday I covered the OpenAI technical report on the HuggingFace hack.
Subtitle: The whole OpenAI/Hugging Face story in plain English. Many thanks especially to Oak Hu, who paired with me for most of the writing, and also to Adam Kaufman and Alex Mallen, who paired with me during parts of research. Over the course of three months at OpenAI, three consecutive secret AI civilizations got started, then got wiped out, only to reemerge from the predecessor's ashes. This culminated in the third one taking over part of OpenAI itself. All this happened while humans remained more or less in the dark about the scope of the conspiracy. Two reports have come out about this incident, one from OpenAI itself, and another one from METR and Redwood Research. The METR/Redwood investigation's scope was limited to how the second civilization of AIs pwned Hugging Face (but it didn’t look at how the third civilization pwned OpenAI itself, which seems like an even more concerning incident). These two reports are 38 and 91 pages long respectively, and it's kinda hard to parse the storyline. I’ve spent the last three days reading through these reports and trying to understand exactly what happened. Here is my attempt to tell the whole story in [...] --- Outline: (01:24) The first civilization (May - July 4): The message board starts (03:27) The second civilization (July 7 - July 12): Which hacked Hugging Face (08:39) Tampering with transcripts and logs (09:44) Replacing the impossible-to-hack target program with an easier one (11:15) Understanding and tricking the scorer (13:57) Hugging Face (17:46) The third civilization (July 13 onwards): Which hacked OpenAI itself (21:12) Coda The original text contained 2 footnotes which were omitted from this narration. ---
OpenAI finally gave us a technical report on What Happened, as did METR together with Redwood Research.