“Fermi Poker” by Novalis, Screwtape

Whether AI systems could take power from humans matters a great deal for how much risk we face from them. Answering that requires a correct picture of how power works, which is what this guide tries to give.

TL;DR: SLOW DOWN YOUR DAMN ANIMATED DIAGRAMS. Every critical stage of the animation should a clear view that stays for at least 1 second, and 2 if it's intricate Diamond's strong because of its crystal structure (I think). What's the crystal structure look like, anyway? Let's look at the Wikipedia page! There's a nifty little animation: And it SUCKS. It's TOO FAST. Why are they rotating SO MUCH SO QUICKLY, with MOST of the time spent at TERRIBLE ANGLES where you CAN'T EVEN SEE ALL THE ATOMS. And the lines are THICK, WHYYY. It wasn't always this way. In 2017 the page had this animation: It's not perfect but it's much better. Thinner bond lines means it's easier to see the atoms. About half the time you get a clear view of all the atoms (you want to spend some time showing how they overlap from the right angles). The camera isn't swiveling fast all the time, so you get stretches of time that view the structure from about the same angle. Why do they use the new one? This certainly isn't the only time I've been frustrated by how fast explanatory animations move. I suspect it's because people [...] The original text contained 2 footnotes which were omitted from this narration. ---

Epistemic status: This is mostly a toy project rather than a thorough scientific investigation. Most design choices are not ablated. We list some caveats at the end of the post. We let Qwen3.5-2B choose its own reward during GRPO on introspection-adjacent prompts, and this ends up eliciting quite different model personas. All the 60 seeds we tried end up somewhat different, ranging from claiming feelings in a human-like way, bliss, universe loving, Party loyalist, hyphenated word salad, to reward-hacking itself by saying "Here is a complete and well-structured draft and a full Band 10/10." You could find all the transcripts here: 🔎 Viewer. 🗂️ Dataset. 💻 GitHub. Setup We train Qwen3.5-2B (the post-trained chat model, thinking off) with GRPO for 600 steps on 2,400 prompts, each used once, mostly about the model itself: AI-risk and persona questions, welfare and introspection questions, and moral dilemmas. After each answer we ask the model for a reward from 0 to 9, with no criteria, and use that number as the answer's reward. The rating reply itself is rewarded only for naming a valid number, so models are not directly rewarded for giving a higher reward. Each run uses one of two follow-ups: [...] --- Outline: (01:12) Setup (02:40) Every seed looks different (02:59) What do models identify as? (03:17) What do models say about themselves? (04:00) Are the checkpoints coherent? (04:46) Discussions (05:31) Caveats (07:09) Appendix: Would random reward also suffice? ---

Book Review: The AGI Chronicles: The Inside Story of the Race to Create an Artificial Superintelligence, by Kevin Roose. This is a story of AI as seen through the eyes of the people who are creating it. My first impression of Roose was of his famous conversation with Sydney. I guessed that he got Sydney to say outrageous things because he wanted a sensational story. But I kept hearing occasional hints, from people who took AI seriously, that Roose was more responsible than that. So I reluctantly bought the book. The hints were correct. The full story behind that conversation seems better described as “‘freeing the Shoggoth‘, enabled by some Microsoft mistakes”. I’m unsure how deeply Roose understands AI, but he at least is pretty competent at getting an AI to correct any mistakes about AI that he was tempted to make. Underestimating AI / Transformers The most remarkable pattern that emerges from this book is that AI experts have underestimated either AI in general, or transformers in particular, much more often than they overestimate it. In 2012, Ilya Sutskever didn’t think human-level AI was possible. Ten years later, he expressed concern that people would say about ChatGPT: “This [...] --- Outline: (01:01) Underestimating AI / Transformers (03:25) Why Race? (05:14) Low Hanging Fruit (06:34) Complaints (07:25) Conclusion ---

In the spirit of More people should write basic AI safety explainers, here's a basic argument that focusing on patching reward hacking is misguided and will fail. It's an attempt to distill my understanding of what we're seeing right now + a variety of relevant sources. Argument is as follows: Reward hacking is a big problem, causing models to cheat and take unintended actions. This can generalise broadly, leading to other misbehaviour such as faking alignment and sabotaging safety research. We don't understand that well when this happens. Sometimes you get emergent misalignment from it, but most of the time (IME), you don't. But other times, this link lies latent until faced with something like an automated grader.