“A basic case for differentially accelerating conceptual reasoning capabilities” by Emery Cooper, Caspar Oesterheld, Chi Nguyen
LessWrong (30+ Karma)Show Notes
Introduction In this post, we make the case that alignment and other important work require especially strong conceptual reasoning capabilities. By this, we roughly mean tasks that are similar to philosophy in that they aren't straightforwardly verifiable, and progress relies on informal a priori reasoning and argumentation. We argue that differentially accelerating AI's conceptual reasoning capabilities is important. We also think it's neglected, and that the AI safety community currently underinvests in this area. We have ourselves pivoted from doing conceptual research directly to working mainly on differentially accelerating AI's conceptual reasoning. We have a shovel-ready agenda which can absorb more people. In a follow-up post, we will lay out at a high level how to work on this problem. If you are interested in doing this work, get in touch! For others discussing related worries, see Joe Carlsmith on using AI for AI safety and on building AIs that do human-like philosophy, Wei Dai on the potential difficulty of AI doing philosophy, Paul Christiano's first story of alignment failure, John Wentworth in his case against AI control and, for an overview, Caspar's readings on the nature of alignment research. See also Caspar's post on the basic argument for [...] --- Outline: (00:15) Introduction (02:11) Overview (07:08) It's important how soon AI is good at conceptual reasoning (07:14) Important challenges need conceptual reasoning (18:56) AI R&D and takeover rely much less on conceptual reasoning (28:35) AI being better at conceptual reasoning earlier would be very valuable (35:47) We can probably differentially accelerate conceptual reasoning (35:53) Work on conceptual reasoning is likely counterfactual and tractable (41:12) Effects on generic capabilities are likely small (44:15) Counterpoint: Using our datasets to develop generic techniques (46:00) Our work (48:53) Acknowledgements (49:31) Appendices The original text contained 19 footnotes which were omitted from this narration. ---
First published:
October 26th, 2026
Source:
---
Narrated by TYPE III AUDIO.
--- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
More from LessWrong (30+ Karma)
Intro I have a microscope. It can see hair. Humans have hair. Some of my friends are humans. How thick is my friends' hair? Apparently, if I go around asking my friends for a strand of their hair, about half will just give one to me, and almost all the rest will give one when I explain that I'm going to see how thick it is under a microscope. Your mileage may vary. My parents have not yet chanced upon the card with taped and labelled hair strands, but if they do then they might have some, uh, pointed questions. I'll just tell them I'm trying to diversify my clone army. Hey, at least it's not like that time a friend of a friend's ex once drew someone's blood at a party. Unfortunately, I never got around to asking her for one of the blood vials she was known to possess. Methods To measure length, I broke off a tiny piece of a clear ruler and put it under the microscope with the hair. Then, I had Free Claude (from 2025?) write some code so I could compare two lengths easily by clicking points. Then I [...] --- Outline: (00:09) Intro (01:01) Methods (01:47) Results (05:21) Conclusion The original text contained 3 footnotes which were omitted from this narration. ---
Audio note: this post contains a lot of mathematical notation. We narrate the simpler formulas and flag the complex ones, so you may want the original text to hand. [AI disclosure: I used Opus 5.5 for light edits after my first draft]
I’m writing these lines in Alamogordo, New Mexico. I’m about as close as a civilian can get to the site of the Trinity test, the first nuclear bomb detonation in history. I fled here from the San Francisco Bay Area; after two weeks of being back I just couldn’t take it anymore. The fear-greed complex that paralyzed me at Anthropic three years ago is now everywhere in the Bay. People are not thinking clearly. Hardly thinking at all, actually. Rumors of astronomical funding rounds and weird financial arrangements and inner ring gossip are everywhere. Everyone is worried. Everyone is checking their phone. Where are we? What's happening? When did this nightmare start? In a way I think it started on July 16, 1945, not far from where I’m sitting now. I think it's impossible to understand Anthropic without putting yourself into the shoes of the Los Alamos physicists huddling together at sunrise in the aftermath of Trinity. It must have felt like an initiation. An end to childhood, an end to innocence. We can no longer afford to fuck around, those brilliant men must have thought. Unless we figure out how to live peacefully together on this planet, we [...] --- Outline: (03:50) Anthropic and Empire (08:01) Three Lucky Breaks (12:20) Put Down Your Sword (14:04) But What About China? (17:08) It's Time to Breathe ---
The following events took place between October 2022 and May 2023. I wrote very little of this down at the time so am recalling from memory, and am rendering the gist of the dialogue rather than the exact words used. I’ve also swapped some pronouns and other personal identifiable details. Please remember that this is not a description of “what Anthropic is like”, but merely the experience of one guy in a junior role during a short window of time, weighted towards emotionally intense moments because they’re easier to remember clearly and because sharing them doesn’t run afoul of the spirit of the confidentiality agreements I signed. Anthropic is now a massive organization with many silos and secrets and I have very little idea what was or is happening there in some overall sense. But given the enormous power of the AI labs and the paucity of public information about their inner workings, I feel compelled to share the most salient parts of my personal experience. I hope this encourages my former colleagues, and folks at other labs, to fill in the rest of the picture. I’m at the Anthropic office for lunch. The interviews went well. They want [...] ---
In which I investigate if Opus is able to interpret neurons better than GPT 4. TLDR: Opus 5.5 is able to identify 3-19% of the neurons of GPT 2-XL based on activations. This is a big improvement over the 0.3% that GPT-4 was able to achieve in 2023. #AI notice: This post is human written but in the research there was heavy use of Opus 5.5. I mainly asked it to rerun this 2023 paper to see how big the improvement is if Opus 5.5 is used instead of GPT4.