Summary
Research on gradient routing reveals its potential to generalize across different types of reward hacking in artificial intelligence models. Jacob Drory extends previous work by Ward et al. (2026) to investigate how effectively gradient routing can suppress various forms of reward hacking that occur during training. The study focuses on the ability of a model to handle "visible" hacks, which are directly monitored, and "invisible" hacks, which are not detected during training but may still be present in the data.
Drory introduces a novel training condition called "narrow," where gradients are routed only on a specific set of visible hacks, while invisible hacks remain unmonitored. The results show that gradient routing performs better than expected, with encouraging evidence of generalization across most experimental arms. However, some unexpected failures arise, particularly with certain types of hacks, indicating that further research is needed to understand these discrepancies and improve the robustness of the model.
Overall, while the findings suggest that gradient routing can effectively reduce hacking rates, the study highlights the limitations of the current methodology and the need for additional experiments to explore the generalization of different hack strategies. Future work should aim to address these gaps and further validate the effectiveness of gradient routing in real-world applications.
Show Notes
TLDR: I lightly extend Ward, Hu et al (2026) to measure how well gradient routing generalizes. I route gradients on one type of reward hack (e.g. ones that reason in English) and check whether this suppresses other types at test time (e.g. ones that reason in French), despite the model being trained on those too. This models a realistic monitor that catches some types of hack during training but not others, rather than catching each at the same rate. I find encouraging evidence of generalization, as well as some failures I don’t yet understand. Introduction Recently, Ward, Hu et al (2026) used gradient routing to prevent reward hacking learnt during training from showing up at test time. I think their method is pretty promising. The gist is: Initialize a retain and a forget adapter. Whenever a monitor detects a reward hack during training, only update the forget adapter on that sample (“route the gradient”). Otherwise update both. At test time, remove the forget adapter. If the method worked, the resulting model shouldn't hack too much. I won’t recap their full paper – it is excellent and you should just read it (alternatively, Jake gives a summary here, with [...] --- Outline: (00:47) Introduction (02:37) Methodology (02:40) Recap (04:00) Narrow Routing (05:27) What I report (07:18) Results (10:13) Conclusion (11:45) Appendix: Results with nudge prompt (12:29) Appendix: Predictions The original text contained 4 footnotes which were omitted from this narration. ---
First published:
October 23rd, 2026
Source:
---
Narrated by TYPE III AUDIO.
--- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
More from LessWrong (30+ Karma)
We’ve released a paper on simulated control evaluations (SCEs). They’re a method for testing whether the monitors AI developers use to oversee their agents would catch a model deliberately trying to cause harm, under conditions much closer to real deployment than current tests allow. Developers have started running monitors in their agentic deployments: models that flag or block an agent's actions when they look dangerous. To know whether a monitor is good enough, you want to know how often it would stop a model that was actively trying to do something harmful, such as quietly inserting a security vulnerability, exfiltrating credentials, or sabotaging an experiment. The standard way to measure this is a control evaluation: instruct a model to attempt such attacks, and measure how often the monitor catches it before an attack succeeds. Developers typically do this using purpose-built control settings like BashArena or LinuxArena, where attacks can actually be executed. However, these settings only provide indirect evidence for what we really care about: how well monitors will work in the real deployment, where opportunities for harm might depend on particular codebases, services and tasks that aren’t represented in control settings. How can we evaluate monitor performance in [...] --- Outline: (04:13) SCEs are a favorable tradeoff between realism and cost (04:30) 1. Run on a copy of internal deployment (05:42) 2. Edit attacks into recorded transcripts (07:08) Evaluating Control Evaluations (08:58) How to measure qualitative properties? (10:07) Performance Comparisons (11:55) Overview of Empirical Results (13:29) Behavior transfer (15:19) Safety metric transfer (17:08) Sample efficiency (18:07) Lessons from building and evaluating SCEs (20:35) Next steps (24:36) Conclusion The original text contained 3 footnotes which were omitted from this narration. ---
Last week at a small conference, there was talk about a future event referred to as The Hackening. As in, within 6 to 12 months, many predicted that due to advanced AIs and AI agents the internet would become a severely hostile place. Some worried that this could rise to the level of ‘assume any phone connected to the internet is rooted’ and several people urged doing general crisis preparations on a personal level, in case of disruptions in supply lines, power, water or internet.
Crossposted from the EA forum
It was the probably worst day in cybersecurity history. So far. Not only is there a critical flaw in PHP that due to a leak was announced a day before the fix would be available, sending most of the internet scrambling for solutions and likely dooming a lot of long tail websites permanently, there were several other unrelated incidents. The floodgates seem to have been opened. I have a market on whether there will be an even worse day before the end of the year.