“San Francisco State University in Twilight” by Zack_M_Davis

In which I investigate if Opus is able to interpret neurons better than GPT 4. TLDR: Opus 5.5 is able to identify 3-19% of the neurons of GPT 2-XL based on activations. This is a big improvement over the 0.3% that GPT-4 was able to achieve in 2023. #AI notice: This post is human written but in the research there was heavy use of Opus 5.5. I mainly asked it to rerun this 2023 paper to see how big the improvement is if Opus 5.5 is used instead of GPT4.

tl;dr. I show that the recently introduced GRAFT method can be improved upon by using a single layer rank 1 Low-rank Linear Subspace Representation Fine Tuning (LoReFT) adapter as the FORGET adapter, using only 0.029% of the parameters of the one used in the GRAFT paper. This approach achieves an equivalent level of reward hack suppression while improving the clean environment solve rate and not requiring an anchor environment. I end with a brief discussion of why this works and of the limitations of fixed preventative steering baselines. Background 1. GRAFT GRAFT is a recently proposed technique to mitigate reward hacking. You can find the details in their LW post and paper. Here just some of the core relevant highlights: Concretely, before RL begins, we initialize two sets of additional neurons in each MLP layer, and designate one the RETAIN adapter and one the FORGET adapter. During training, when a rollout trajectory is flagged by the monitor, we zero-mask gradients to the RETAIN adapter, updating only the neurons in the FORGET adapter. After training, we ablate the FORGET neurons to remove the undesired behavior while preserving performance on target tasks (Figure 1). GRAFT can also be used with full-parameter [...] --- Outline: (00:44) Background (00:47) 1. GRAFT (02:31) 2. LoReFT (03:16) Experiment and Results (04:23) Discussion / Analysis (06:38) Limitations (07:15) Summary The original text contained 4 footnotes which were omitted from this narration. ---

I am often skeptical of arguments about China that rely heavily on personal experience. The author may explicitly acknowledge that their experience is limited without appropriately treating their arguments with rigor beforehand. This acknowledgment may thus sit beside conclusions about the values, behaviors, and functions of Chinese society without meaningfully constraining any of them. Different speakers may each have peers with similar experiences that corroborate their claims, or repeated experiences that solidify their firsthand evidence. We may believe one or another speaker is more perceptive, well-informed, or more willing to acknowledge facts contrary to their beliefs. However, the critical question is how far these accounts support the conclusions being drawn. I will address here how we should evaluate firsthand accounts, including when people offer conflicting accounts. The Mistake of Universalizing Personal Experience Happy one-year anniversary to this tweet by culture critic Rayne Fisher-Quann. When people use personal experiences to support arguments about a locale, collective, or institution, one must best establish which conditions each person has encountered and how widely those conditions apply. An experience tells us something happens, reveals how an institution works, or challenges a claim about what never happens, etc.. Claiming that this experience describes a [...] --- Outline: (01:02) The Mistake of Universalizing Personal Experience (03:48) Standpoint Epistemology (05:52) Countering Generalization in Practice The original text contained 3 footnotes which were omitted from this narration. ---

For a while, we felt we had a pretty good grasp over LLM behaviour because we had to exert significant effort to elicit misaligned behaviour. Then, fairly suddenly, we moved to undereliciting misaligned behaviour. Let us recount: For a while, there was a bit where safety research seemed to over-elicit bad propensities. A common criticism of papers like Agentic Misalignment was that they basically led the witness using prompting. I recall that a paper even as recent as Peer-preservation took similar flak on Twitter.

TL;DR: When misalignment monitors fail, it is already often because of alignment failures, not because they are too weak. I expect this to continue in the future. Monitor evals share similar difficulties with agent alignment evals - which is bad news since getting evidence about agent alignment using alignment evals is hard. Alignment eval results should not be the only input to monitor alignment assessments, similar to how alignment evals should not be the only input to agent alignment assessments. These are my personal views; they do not represent the views of my employers. Some of the worst past monitor failures are mostly alignment issues Monitors are sometimes misaligned in non-scheming ways that have caused some important monitor false negatives: They sometimes fail to balance different priorities the way developers would like them to For example, they may fail to balance broad safety and broad ethics: see e.g. 5.2.2 Partial refusals on safety work undermining stress-testing research Monitors might also not have the right information about the state of the system, what invariants should be maintained, or what is most important to surface. So these are not pure alignment failures.