“Measuring agent capabilities on open-ended AI R&D tasks” by Alex Jackson, Titus Buckworth, Jason R Brown, Francis Rhys Ward

This is part linkpost, and an expansion on a previous shortform posted on LW. I believe a massively undermade point in favour of the belief that AI will be a very big deal is that the AI buildout over the past four years is the largest bet in the history of capitalism. A titanic peacetime industrial effort is underway which has not had a comparison since the nineteenth century. The size of the effort and the scale of the spend signal that key decision makers and capital allocators believe very poweful AI will be achieved in the next few years. Many of us are vaguely aware that large sums are being spent, but the scale of the undertaking is astonishing: The six largest spenders this year are spending more on Capital expenditures this year than the Russian government, governing 11 timezones and 143 million people, fighting a war on which the fate of its regime probably depends, spent last year. Hyperscaler spending (Alphabet, Amazon, Microsoft, Meta and Oracle) will be $785-820bn this year, about 2.4% of US GDP. There are only 11 governments on earth who spend more than that. Next year the Hyperscalers are projected [...] ---

Intro I have a microscope. It can see hair. Humans have hair. Some of my friends are humans. How thick is my friends' hair? Apparently, if I go around asking my friends for a strand of their hair, about half will just give one to me, and almost all the rest will give one when I explain that I'm going to see how thick it is under a microscope. Your mileage may vary. My parents have not yet chanced upon the card with taped and labelled hair strands, but if they do then they might have some, uh, pointed questions. I'll just tell them I'm trying to diversify my clone army. Hey, at least it's not like that time a friend of a friend's ex once drew someone's blood at a party. Unfortunately, I never got around to asking her for one of the blood vials she was known to possess. Methods To measure length, I broke off a tiny piece of a clear ruler and put it under the microscope with the hair. Then, I had Free Claude (from 2025?) write some code so I could compare two lengths easily by clicking points. Then I [...] --- Outline: (00:09) Intro (01:01) Methods (01:47) Results (05:21) Conclusion The original text contained 3 footnotes which were omitted from this narration. ---

Introduction In this post, we make the case that alignment and other important work require especially strong conceptual reasoning capabilities. By this, we roughly mean tasks that are similar to philosophy in that they aren't straightforwardly verifiable, and progress relies on informal a priori reasoning and argumentation. We argue that differentially accelerating AI's conceptual reasoning capabilities is important. We also think it's neglected, and that the AI safety community currently underinvests in this area. We have ourselves pivoted from doing conceptual research directly to working mainly on differentially accelerating AI's conceptual reasoning. We have a shovel-ready agenda which can absorb more people. In a follow-up post, we will lay out at a high level how to work on this problem. If you are interested in doing this work, get in touch! For others discussing related worries, see Joe Carlsmith on using AI for AI safety and on building AIs that do human-like philosophy, Wei Dai on the potential difficulty of AI doing philosophy, Paul Christiano's first story of alignment failure, John Wentworth in his case against AI control and, for an overview, Caspar's readings on the nature of alignment research. See also Caspar's post on the basic argument for [...] --- Outline: (00:15) Introduction (02:11) Overview (07:08) It's important how soon AI is good at conceptual reasoning (07:14) Important challenges need conceptual reasoning (18:56) AI R&D and takeover rely much less on conceptual reasoning (28:35) AI being better at conceptual reasoning earlier would be very valuable (35:47) We can probably differentially accelerate conceptual reasoning (35:53) Work on conceptual reasoning is likely counterfactual and tractable (41:12) Effects on generic capabilities are likely small (44:15) Counterpoint: Using our datasets to develop generic techniques (46:00) Our work (48:53) Acknowledgements (49:31) Appendices The original text contained 19 footnotes which were omitted from this narration. ---

Audio note: this post contains a lot of mathematical notation. We narrate the simpler formulas and flag the complex ones, so you may want the original text to hand. [AI disclosure: I used Opus 5.5 for light edits after my first draft]

I’m writing these lines in Alamogordo, New Mexico. I’m about as close as a civilian can get to the site of the Trinity test, the first nuclear bomb detonation in history. I fled here from the San Francisco Bay Area; after two weeks of being back I just couldn’t take it anymore. The fear-greed complex that paralyzed me at Anthropic three years ago is now everywhere in the Bay. People are not thinking clearly. Hardly thinking at all, actually. Rumors of astronomical funding rounds and weird financial arrangements and inner ring gossip are everywhere. Everyone is worried. Everyone is checking their phone. Where are we? What's happening? When did this nightmare start? In a way I think it started on July 16, 1945, not far from where I’m sitting now. I think it's impossible to understand Anthropic without putting yourself into the shoes of the Los Alamos physicists huddling together at sunrise in the aftermath of Trinity. It must have felt like an initiation. An end to childhood, an end to innocence. We can no longer afford to fuck around, those brilliant men must have thought. Unless we figure out how to live peacefully together on this planet, we [...] --- Outline: (03:50) Anthropic and Empire (08:01) Three Lucky Breaks (12:20) Put Down Your Sword (14:04) But What About China? (17:08) It's Time to Breathe ---