“A cat-belling problem for FDT” by jessicata

I’m writing these lines in Alamogordo, New Mexico. I’m about as close as a civilian can get to the site of the Trinity test, the first nuclear bomb detonation in history. I fled here from the San Francisco Bay Area; after two weeks of being back I just couldn’t take it anymore. The fear-greed complex that paralyzed me at Anthropic three years ago is now everywhere in the Bay. People are not thinking clearly. Hardly thinking at all, actually. Rumors of astronomical funding rounds and weird financial arrangements and inner ring gossip are everywhere. Everyone is worried. Everyone is checking their phone. Where are we? What's happening? When did this nightmare start? In a way I think it started on July 16, 1945, not far from where I’m sitting now. I think it's impossible to understand Anthropic without putting yourself into the shoes of the Los Alamos physicists huddling together at sunrise in the aftermath of Trinity. It must have felt like an initiation. An end to childhood, an end to innocence. We can no longer afford to fuck around, those brilliant men must have thought. Unless we figure out how to live peacefully together on this planet, we [...] --- Outline: (03:50) Anthropic and Empire (08:01) Three Lucky Breaks (12:20) Put Down Your Sword (14:04) But What About China? (17:08) It's Time to Breathe ---

The following events took place between October 2022 and May 2023. I wrote very little of this down at the time so am recalling from memory, and am rendering the gist of the dialogue rather than the exact words used. I’ve also swapped some pronouns and other personal identifiable details. Please remember that this is not a description of “what Anthropic is like”, but merely the experience of one guy in a junior role during a short window of time, weighted towards emotionally intense moments because they’re easier to remember clearly and because sharing them doesn’t run afoul of the spirit of the confidentiality agreements I signed. Anthropic is now a massive organization with many silos and secrets and I have very little idea what was or is happening there in some overall sense. But given the enormous power of the AI labs and the paucity of public information about their inner workings, I feel compelled to share the most salient parts of my personal experience. I hope this encourages my former colleagues, and folks at other labs, to fill in the rest of the picture. I’m at the Anthropic office for lunch. The interviews went well. They want [...] ---

In which I investigate if Opus is able to interpret neurons better than GPT 4. TLDR: Opus 5.5 is able to identify 3-19% of the neurons of GPT 2-XL based on activations. This is a big improvement over the 0.3% that GPT-4 was able to achieve in 2023. #AI notice: This post is human written but in the research there was heavy use of Opus 5.5. I mainly asked it to rerun this 2023 paper to see how big the improvement is if Opus 5.5 is used instead of GPT4.

tl;dr. I show that the recently introduced GRAFT method can be improved upon by using a single layer rank 1 Low-rank Linear Subspace Representation Fine Tuning (LoReFT) adapter as the FORGET adapter, using only 0.029% of the parameters of the one used in the GRAFT paper. This approach achieves an equivalent level of reward hack suppression while improving the clean environment solve rate and not requiring an anchor environment. I end with a brief discussion of why this works and of the limitations of fixed preventative steering baselines. Background 1. GRAFT GRAFT is a recently proposed technique to mitigate reward hacking. You can find the details in their LW post and paper. Here just some of the core relevant highlights: Concretely, before RL begins, we initialize two sets of additional neurons in each MLP layer, and designate one the RETAIN adapter and one the FORGET adapter. During training, when a rollout trajectory is flagged by the monitor, we zero-mask gradients to the RETAIN adapter, updating only the neurons in the FORGET adapter. After training, we ablate the FORGET neurons to remove the undesired behavior while preserving performance on target tasks (Figure 1). GRAFT can also be used with full-parameter [...] --- Outline: (00:44) Background (00:47) 1. GRAFT (02:31) 2. LoReFT (03:16) Experiment and Results (04:23) Discussion / Analysis (06:38) Limitations (07:15) Summary The original text contained 4 footnotes which were omitted from this narration. ---

I am often skeptical of arguments about China that rely heavily on personal experience. The author may explicitly acknowledge that their experience is limited without appropriately treating their arguments with rigor beforehand. This acknowledgment may thus sit beside conclusions about the values, behaviors, and functions of Chinese society without meaningfully constraining any of them. Different speakers may each have peers with similar experiences that corroborate their claims, or repeated experiences that solidify their firsthand evidence. We may believe one or another speaker is more perceptive, well-informed, or more willing to acknowledge facts contrary to their beliefs. However, the critical question is how far these accounts support the conclusions being drawn. I will address here how we should evaluate firsthand accounts, including when people offer conflicting accounts. The Mistake of Universalizing Personal Experience Happy one-year anniversary to this tweet by culture critic Rayne Fisher-Quann. When people use personal experiences to support arguments about a locale, collective, or institution, one must best establish which conditions each person has encountered and how widely those conditions apply. An experience tells us something happens, reveals how an institution works, or challenges a claim about what never happens, etc.. Claiming that this experience describes a [...] --- Outline: (01:02) The Mistake of Universalizing Personal Experience (03:48) Standpoint Epistemology (05:52) Countering Generalization in Practice The original text contained 3 footnotes which were omitted from this narration. ---