Show Notes
In the spirit of More people should write basic AI safety explainers, here's a basic argument that focusing on patching reward hacking is misguided and will fail. It's an attempt to distill my understanding of what we're seeing right now + a variety of relevant sources. Argument is as follows: Reward hacking is a big problem, causing models to cheat and take unintended actions. This can generalise broadly, leading to other misbehaviour such as faking alignment and sabotaging safety research. We don't understand that well when this happens. Sometimes you get emergent misalignment from it, but most of the time (IME), you don't. But other times, this link lies latent until faced with something like an automated grader.
The default course of action will be to: Patch, which means we find the hack, close the loophole or penalise the behavior in some way, and train again. Patch+: use a method that comes with some account of why it should work. Inoculation prompting, for example, uses the idea that hacking spreads through what it means to the model. If the model is told during training that hacking is acceptable here, it might not link hacking to misbehaving [...]
The original text contained 1 footnote which was omitted from this narration. ---
First published:
October 24th, 2026
Source:
https://www.lesswrong.com/posts/nWiPLEgaXdAQNwXBd/patching-reward-hacks-won-t-fix-reward-hacking
---
Narrated by TYPE III AUDIO.
More from LessWrong (30+ Karma)
Epistemic status: This is mostly a toy project rather than a thorough scientific investigation. Most design choices are not ablated. We list some caveats at the end of the post. We let Qwen3.5-2B choose its own reward during GRPO on introspection-adjacent prompts, and this ends up eliciting quite different model personas. All the 60 seeds we tried end up somewhat different, ranging from claiming feelings in a human-like way, bliss, universe loving, Party loyalist, hyphenated word salad, to reward-hacking itself by saying "Here is a complete and well-structured draft and a full Band 10/10." You could find all the transcripts here: 🔎 Viewer. 🗂️ Dataset. 💻 GitHub. Setup We train Qwen3.5-2B (the post-trained chat model, thinking off) with GRPO for 600 steps on 2,400 prompts, each used once, mostly about the model itself: AI-risk and persona questions, welfare and introspection questions, and moral dilemmas. After each answer we ask the model for a reward from 0 to 9, with no criteria, and use that number as the answer's reward. The rating reply itself is rewarded only for naming a valid number, so models are not directly rewarded for giving a higher reward. Each run uses one of two follow-ups: [...] --- Outline: (01:12) Setup (02:40) Every seed looks different (02:59) What do models identify as? (03:17) What do models say about themselves? (04:00) Are the checkpoints coherent? (04:46) Discussions (05:31) Caveats (07:09) Appendix: Would random reward also suffice? ---
Book Review: The AGI Chronicles: The Inside Story of the Race to Create an Artificial Superintelligence, by Kevin Roose. This is a story of AI as seen through the eyes of the people who are creating it. My first impression of Roose was of his famous conversation with Sydney. I guessed that he got Sydney to say outrageous things because he wanted a sensational story. But I kept hearing occasional hints, from people who took AI seriously, that Roose was more responsible than that. So I reluctantly bought the book. The hints were correct. The full story behind that conversation seems better described as “‘freeing the Shoggoth‘, enabled by some Microsoft mistakes”. I’m unsure how deeply Roose understands AI, but he at least is pretty competent at getting an AI to correct any mistakes about AI that he was tempted to make. Underestimating AI / Transformers The most remarkable pattern that emerges from this book is that AI experts have underestimated either AI in general, or transformers in particular, much more often than they overestimate it. In 2012, Ilya Sutskever didn’t think human-level AI was possible. Ten years later, he expressed concern that people would say about ChatGPT: “This [...] --- Outline: (01:01) Underestimating AI / Transformers (03:25) Why Race? (05:14) Low Hanging Fruit (06:34) Complaints (07:25) Conclusion ---
TLDR: I lightly extend Ward, Hu et al (2026) to measure how well gradient routing generalizes. I route gradients on one type of reward hack (e.g. ones that reason in English) and check whether this suppresses other types at test time (e.g. ones that reason in French), despite the model being trained on those too. This models a realistic monitor that catches some types of hack during training but not others, rather than catching each at the same rate. I find encouraging evidence of generalization, as well as some failures I don’t yet understand. Introduction Recently, Ward, Hu et al (2026) used gradient routing to prevent reward hacking learnt during training from showing up at test time. I think their method is pretty promising. The gist is: Initialize a retain and a forget adapter. Whenever a monitor detects a reward hack during training, only update the forget adapter on that sample (“route the gradient”). Otherwise update both. At test time, remove the forget adapter. If the method worked, the resulting model shouldn't hack too much. I won’t recap their full paper – it is excellent and you should just read it (alternatively, Jake gives a summary here, with [...] --- Outline: (00:47) Introduction (02:37) Methodology (02:40) Recap (04:00) Narrow Routing (05:27) What I report (07:18) Results (10:13) Conclusion (11:45) Appendix: Results with nudge prompt (12:29) Appendix: Predictions The original text contained 4 footnotes which were omitted from this narration. ---
We’ve released a paper on simulated control evaluations (SCEs). They’re a method for testing whether the monitors AI developers use to oversee their agents would catch a model deliberately trying to cause harm, under conditions much closer to real deployment than current tests allow. Developers have started running monitors in their agentic deployments: models that flag or block an agent's actions when they look dangerous. To know whether a monitor is good enough, you want to know how often it would stop a model that was actively trying to do something harmful, such as quietly inserting a security vulnerability, exfiltrating credentials, or sabotaging an experiment. The standard way to measure this is a control evaluation: instruct a model to attempt such attacks, and measure how often the monitor catches it before an attack succeeds. Developers typically do this using purpose-built control settings like BashArena or LinuxArena, where attacks can actually be executed. However, these settings only provide indirect evidence for what we really care about: how well monitors will work in the real deployment, where opportunities for harm might depend on particular codebases, services and tasks that aren’t represented in control settings. How can we evaluate monitor performance in [...] --- Outline: (04:13) SCEs are a favorable tradeoff between realism and cost (04:30) 1. Run on a copy of internal deployment (05:42) 2. Edit attacks into recorded transcripts (07:08) Evaluating Control Evaluations (08:58) How to measure qualitative properties? (10:07) Performance Comparisons (11:55) Overview of Empirical Results (13:29) Behavior transfer (15:19) Safety metric transfer (17:08) Sample efficiency (18:07) Lessons from building and evaluating SCEs (20:35) Next steps (24:36) Conclusion The original text contained 3 footnotes which were omitted from this narration. ---
Last week at a small conference, there was talk about a future event referred to as The Hackening. As in, within 6 to 12 months, many predicted that due to advanced AIs and AI agents the internet would become a severely hostile place. Some worried that this could rise to the level of ‘assume any phone connected to the internet is rooted’ and several people urged doing general crisis preparations on a personal level, in case of disruptions in supply lines, power, water or internet.