“I trained GPT-2 with full-batch gradient descent so you don’t have to” by shawnghu
LessWrong (30+ Karma)Show Notes
The impetus for doing this was provided in large part by Charles Renshaw-Whitman. Learning dynamics people suggested to me that noise was essential in the training of neural networks. I thought this was kind of counterintuitive--- I usually naively operate from a model in which the pretraining data basically specifies the outcome by virtue of the samples all "pulling" on the loss landscape in some relatively uniform way which incentivizes converging to a point that generalizes well, but instead it seems to be the case that the actual dynamics of the optimization process itself have predictable and measurable effects on the qualitative outcomes of training. The idea seems reasonable based on vague intuitions I have about simulated annealing. I wanted to see directly that this is true at some nontrivial scale, I happened to have access to a pool of free compute that no one else seemed to be using, and I figured this would be an interesting object of study for compute-constrained researchers with an interest in studying learning dynamics closely, so I did some very rough calculations and realized this was feasible to do with GPT-2. The essential result is that on a fixed 800M-token subset [...] ---
First published:
October 14th, 2026
Source:
---
Narrated by TYPE III AUDIO.
More from LessWrong (30+ Karma)
TLDR: I lightly extend Ward, Hu et al (2026) to measure how well gradient routing generalizes. I route gradients on one type of reward hack (e.g. ones that reason in English) and check whether this suppresses other types at test time (e.g. ones that reason in French), despite the model being trained on those too. This models a realistic monitor that catches some types of hack during training but not others, rather than catching each at the same rate. I find encouraging evidence of generalization, as well as some failures I don’t yet understand. Introduction Recently, Ward, Hu et al (2026) used gradient routing to prevent reward hacking learnt during training from showing up at test time. I think their method is pretty promising. The gist is: Initialize a retain and a forget adapter. Whenever a monitor detects a reward hack during training, only update the forget adapter on that sample (“route the gradient”). Otherwise update both. At test time, remove the forget adapter. If the method worked, the resulting model shouldn't hack too much. I won’t recap their full paper – it is excellent and you should just read it (alternatively, Jake gives a summary here, with [...] --- Outline: (00:47) Introduction (02:37) Methodology (02:40) Recap (04:00) Narrow Routing (05:27) What I report (07:18) Results (10:13) Conclusion (11:45) Appendix: Results with nudge prompt (12:29) Appendix: Predictions The original text contained 4 footnotes which were omitted from this narration. ---
We’ve released a paper on simulated control evaluations (SCEs). They’re a method for testing whether the monitors AI developers use to oversee their agents would catch a model deliberately trying to cause harm, under conditions much closer to real deployment than current tests allow. Developers have started running monitors in their agentic deployments: models that flag or block an agent's actions when they look dangerous. To know whether a monitor is good enough, you want to know how often it would stop a model that was actively trying to do something harmful, such as quietly inserting a security vulnerability, exfiltrating credentials, or sabotaging an experiment. The standard way to measure this is a control evaluation: instruct a model to attempt such attacks, and measure how often the monitor catches it before an attack succeeds. Developers typically do this using purpose-built control settings like BashArena or LinuxArena, where attacks can actually be executed. However, these settings only provide indirect evidence for what we really care about: how well monitors will work in the real deployment, where opportunities for harm might depend on particular codebases, services and tasks that aren’t represented in control settings. How can we evaluate monitor performance in [...] --- Outline: (04:13) SCEs are a favorable tradeoff between realism and cost (04:30) 1. Run on a copy of internal deployment (05:42) 2. Edit attacks into recorded transcripts (07:08) Evaluating Control Evaluations (08:58) How to measure qualitative properties? (10:07) Performance Comparisons (11:55) Overview of Empirical Results (13:29) Behavior transfer (15:19) Safety metric transfer (17:08) Sample efficiency (18:07) Lessons from building and evaluating SCEs (20:35) Next steps (24:36) Conclusion The original text contained 3 footnotes which were omitted from this narration. ---
Last week at a small conference, there was talk about a future event referred to as The Hackening. As in, within 6 to 12 months, many predicted that due to advanced AIs and AI agents the internet would become a severely hostile place. Some worried that this could rise to the level of ‘assume any phone connected to the internet is rooted’ and several people urged doing general crisis preparations on a personal level, in case of disruptions in supply lines, power, water or internet.
Crossposted from the EA forum
It was the probably worst day in cybersecurity history. So far. Not only is there a critical flaw in PHP that due to a leak was announced a day before the fix would be available, sending most of the internet scrambling for solutions and likely dooming a lot of long tail websites permanently, there were several other unrelated incidents. The floodgates seem to have been opened. I have a market on whether there will be an even worse day before the end of the year.