“A big case study of propensity entanglement” by nielsrolf, lwen2027, megasilverfist, Owen Terry, Johannes Taraz
LessWrong (30+ Karma)Show Notes
This is a link-post for our new paper "Predicting how propensities generalize out-of-distribution", which started as a SPAR project. We select 24 propensities, elicit each through 4 different methods, and then test how this spills over to non-targeted traits: How well can we predict spillover? Below is the plot that compares different methods to predict SFT-induced spillover. The best method is to predict that one model behaves similarly to the two other models we tested ("SFT (other models)"). The second best is to system prompt the model to express the target trait, and use the behavior of the prompted model as a proxy for the finetuned model. The third best predictor is drift: assuming that the post-elicitation propensity profile does not depend on the targeted trait at all! This was pretty surprising for us, and we speculate that this is some kind of regression to the base model: by changing the weights, we undo some of the changes of the original post-training. Indeed, drift makes Qwen3.5-9B more similar to its base model. Can we predict what causes misaligned traits to rise? Initially we hoped that we could find traits that induce misalignment by first making models deliberately misaligned, and [...] --- Outline: (00:43) How well can we predict spillover? (01:45) Can we predict what causes misaligned traits to rise? (02:31) Evidence about the Persona Selection Model (03:34) Abstract ---
First published:
October 27th, 2026
Source:
https://www.lesswrong.com/posts/eD84A7C5jhhXNEhCQ/a-big-case-study-of-propensity-entanglement
---
Narrated by TYPE III AUDIO.
--- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
More from LessWrong (30+ Karma)
TLDR I (and many others in the community) wish to see the universe unfold to its highest potential in the metaphorical palm of an AI as aligned as the most benevolent Buddhist monk – but I (we) feel the community's ontology is too narrow to hold me (us). I'm still not sure what to do about this, so I'm posting the article I wrote for LessWrong – calling for a more 'hippie' rationalism – on Substack instead (actually, there are two articles, and the links are at the end). Quick intro (who I am) My name is Melissa. I’m a passionate member of the core AISafety.com team. I was there with the core team when the site was just an empty domain and a loose vision in August 2023. Since then I have co-led, designed, and coded the first official version, and co-led, designed, and coded the second and current version, co-led two of three hackathons, led the creative direction for the current ecosystem map, and I’m currently working to bring lots of more amazing features to our site (stay tuned, it's going to be awesome)! I’ve also played major roles at AISafety.info, grantmaking.ai, Elicit, and minor roles at [...] --- Outline: (00:20) TLDR (00:51) Quick intro (who I am) (02:19) Leaving LessWrong in pursuit of ontological freedom ---
Note: We at PrincInt are launching a program teaching “Discernment” or “Taste” and generally spend a lot of our time in workshops and otherwise trying to teach academics how to think about AI such that their research ends up being useful. I had used a metaphor (building on Wentworth's postmortem) multiple times to explain what we do and why, and thought that it could be useful more broadly. All views are my own, and strategy takes do not represent everyone at PrincInt etc. AI writing level 0 (no non-trivial use)
People love to argue about whether AIs could recursively self-improve their way to superintelligence, each improvement making the next one easier as progress accelerates. I'm unsure. It could be that progress stalls at near-human or somewhat superhuman levels, that the ultimate qualitative limits of intelligence are not particularly beyond the human-level, or that the quantity of AI researchers involved is so vast that substantially superhuman AIs are created without strictly entering a super-exponential regime.
Wikipedia summarizes René Girard's mimetic theory like this: Man is the creature who does not know what to desire, and he turns to others in order to make up his mind. We desire what others desire because we imitate their desires. In other words, you think you want something for your own internal reasons, but the actual reason you want it is that someone else wants it, and that makes you desire the thing and compete for it. The "someone else" whose desires you copy is someone you care about. It could be a friend, a sibling, or a colleague. It could also be a famous person, or even a fictional character. (Not a stranger you don't care about.) This creates unnecessary social conflict, as people gradually converge on wanting the same thing, people start seeing each other as rivals, this attitude gets copied too, then people start hating each other, but who hates whom gets copied too, until everyone synchronizes at hating one specific person... and then they typically murder the person, and everyone feels happy about the conflict being resolved. The choice of the person to kill is arbitrary, in the sense that the selected person is [...] The original text contained 3 footnotes which were omitted from this narration. ---