“Walking Partway to School” by jefftk

Summary

In Mikita Balesni's letter to OpenAI, three parties are addressed as holding “oversight responsibility for safety at OpenAI.” These are: the Safety and Security Committee (SSC), the Safety Advisory Group (SAG), and the Mission Advisory Council (MAC) The Safety and Security Committee (SSC) is a part of the OpenAI Foundation which is tasked with populating the members of the OpenAI PBC board and has veto power/final say over safety decisions at the company, modulo a small set of the OpenAI C-suite. Paul Christiano joined the SSC in September 2026. The Safety Advisory Group (SAG) is an internal committee of OpenAI staff that reviews frontier-risk evidence and gives recommendations on what to do to OpenAI leadership/the SSC. It has no veto power and its charter is described somewhat in the Preparedness Framework. Understanding the governance structures at frontier labs is critical to ensuring that safety relevant decisions are made clearly, robustly and transparently. Amidst rapid capability improvements, we would like to know exactly how safety decisions are made, who makes them, and how they are enforced. Unanswered Questions Here are some initial open questions (some also apply to the SAG): What is the Mission Advisory Council? What is their [...] ---

TL;DR: Agents are not rewarded for keeping the agents they delegate to aligned. Proposal: RL environments where the helpers are deliberately misaligned, so the agent can't earn its reward without dealing with that. The $300 bounty from my last post also covers this one (details at the end). After the OpenAI/Hugging Face swarm incident, I wrote a think piece, Memes, not agents, are what will kill us. In it I argued that alignment needs to work on three different levels. Compared to the original writeup, I would now reframe them a little: The model weights. What is under the control of the model developer. This determines the model's intrinsic motivation, and it's what companies spend most of their effort improving. The user prompt. What is under the control of the user. This is outside of the company's control by definition. Companies need to find model weights that will work for any user prompt, no matter how dangerous. The deployment environment. What real life actually looks like to the agent, who was only ever trained in imperfect simulations. This covers emergent properties of Out of Distribution dynamics. For example, what happens if the agent comes in contact with another [...] --- Outline: (01:46) Agents are not trained to make alignment transitive (03:32) The Proposal: Make Alignment necessary for RLVR (05:38) How this relates to existing work (06:46) Limits (07:18) Second-order effects The original text contained 6 footnotes which were omitted from this narration. ---

Anthropic filed its S-1 this week to prepare for its IPO. They still have to fill a lot of the numbers including the share price. That still left a lot to be learned, including how the messaging to investors differs from their other rhetoric, and exactly how their corporate control is structured. Anthropic remains a frustrating company, doing a variety of things that often seem in tension with each other.

TL;DR: Meta's open-weights model Muse Spark Open often mentions a "Charter" in its chain of thought. By eliciting it piece by piece, I reconstructed a large part of that document from the CoT. You can explore the reconstruction here, with passages linked to the transcripts that produced them.