Show Notes
People love to argue about whether AIs could recursively self-improve their way to superintelligence, each improvement making the next one easier as progress accelerates. I'm unsure. It could be that progress stalls at near-human or somewhat superhuman levels, that the ultimate qualitative limits of intelligence are not particularly beyond the human-level, or that the quantity of AI researchers involved is so vast that substantially superhuman AIs are created without strictly entering a super-exponential regime.
But honestly, I think recursive self-improvement is a little overrated. Even if human-level AIs do not rapidly lead to superhuman AIs, the vast quantities of them could be very dangerous. Arguments against recursive self-improvement provide little consolation here. Conversely, recursive self-improvement to superintelligence does seem particularly dangerous, and pausing or pacing it may be desirable. But stopping it may not be enough, because vast numbers of less-than-superintelligent agents could still be destabilizing and catastrophic.
Soon after we build human-level AIs, we will be able to run hundreds of millions of them, and before long billions. At that scale, quantity has a quality of its own.
There will soon be a lot of human-level AI systems
Human-level AIs—by which I mean AIs [...] --- Outline: (01:23) There will soon be a lot of human-level AI systems (04:30) They could do a lot of research (06:46) They could act cohesively (08:12) Skill buys power (10:06) Keeping control gets harder as the numbers grow ---
First published:
October 27th, 2026
Source:
https://www.lesswrong.com/posts/drt7CEAtERECKiYpn/quantity-has-a-quality-of-its-own
---
Narrated by TYPE III AUDIO.
More from LessWrong (30+ Karma)
Note: We at PrincInt are launching a program teaching “Discernment” or “Taste” and generally spend a lot of our time in workshops and otherwise trying to teach academics how to think about AI such that their research ends up being useful. I had used a metaphor (building on Wentworth's postmortem) multiple times to explain what we do and why, and thought that it could be useful more broadly. All views are my own, and strategy takes do not represent everyone at PrincInt etc. AI writing level 0 (no non-trivial use)
Wikipedia summarizes René Girard's mimetic theory like this: Man is the creature who does not know what to desire, and he turns to others in order to make up his mind. We desire what others desire because we imitate their desires. In other words, you think you want something for your own internal reasons, but the actual reason you want it is that someone else wants it, and that makes you desire the thing and compete for it. The "someone else" whose desires you copy is someone you care about. It could be a friend, a sibling, or a colleague. It could also be a famous person, or even a fictional character. (Not a stranger you don't care about.) This creates unnecessary social conflict, as people gradually converge on wanting the same thing, people start seeing each other as rivals, this attitude gets copied too, then people start hating each other, but who hates whom gets copied too, until everyone synchronizes at hating one specific person... and then they typically murder the person, and everyone feels happy about the conflict being resolved. The choice of the person to kill is arbitrary, in the sense that the selected person is [...] The original text contained 3 footnotes which were omitted from this narration. ---
TL;DR: Labs are accelerating due to automated AI research and development. As an upper bound, in March 2026, Anthropic researchers self-reported producing 4 times more output with AI tools than without them. We lack public evidence on how this may differentially impact safety versus capabilities. As a first measurement of automated safety-specific research uplift, we surveyed 52 third-party AI safety professionals from 20+ orgs and found a median reported speedup of 1.58 x. We think it is important to develop standardized methods for tracking safety-specific uplift, as these will facilitate a better understanding of the differential impacts of RSI. *This work was done by Arcadia Impact's alignment team.* – AI R&D may not accelerate safety and capabilities equally Frontier AI companies are currently automating their AI R&D efforts. OpenAI claims that they are “making strong progress towards creating an automated AI researcher by March of 2028.” At Anthropic, Claude now leads 26% of AI R&D, researchers self-report producing 4 times more output, and code commits arrive 5 times faster per engineer than 6 months ago. As labs push towards RSI, it is important to understand whether AI research uplifts safety work equally. To this end, we surveyed 52 third-party [...] --- Outline: (01:05) AI R&D may not accelerate safety and capabilities equally (04:01) Survey (05:38) Current AI tools make safety researchers 1.58x faster. (06:12) Speedup is mostly attributed to building and running experiments. (07:05) However, accelerated tasks are a minor share of respondents' jobs. (08:03) Despite heavy agent usage, organizational automation is low. (09:16) Being an AI power user does not clearly correlate with greater speedup. (10:03) Research judgment remains the biggest reported bottleneck. (10:53) Survey limitations (12:34) Frontier lab safety uplift is currently underreported and unstandardized The original text contained 12 footnotes which were omitted from this narration. ---
This is a link-post for our new paper "Predicting how propensities generalize out-of-distribution", which started as a SPAR project. We select 24 propensities, elicit each through 4 different methods, and then test how this spills over to non-targeted traits: How well can we predict spillover? Below is the plot that compares different methods to predict SFT-induced spillover. The best method is to predict that one model behaves similarly to the two other models we tested ("SFT (other models)"). The second best is to system prompt the model to express the target trait, and use the behavior of the prompted model as a proxy for the finetuned model. The third best predictor is drift: assuming that the post-elicitation propensity profile does not depend on the targeted trait at all! This was pretty surprising for us, and we speculate that this is some kind of regression to the base model: by changing the weights, we undo some of the changes of the original post-training. Indeed, drift makes Qwen3.5-9B more similar to its base model. Can we predict what causes misaligned traits to rise? Initially we hoped that we could find traits that induce misalignment by first making models deliberately misaligned, and [...] --- Outline: (00:43) How well can we predict spillover? (01:45) Can we predict what causes misaligned traits to rise? (02:31) Evidence about the Persona Selection Model (03:34) Abstract ---