“Subtly wrong advice misaligns models more than obviously wrong advice” by Christine Corry, arav-dhoot, Jozdien

The full text is finally here, so we are giving it the full RTFB of model card treatment.

TLDR I (and many others in the community) wish to see the universe unfold to its highest potential in the metaphorical palm of an AI as aligned as the most benevolent Buddhist monk – but I (we) feel the community's ontology is too narrow to hold me (us). I'm still not sure what to do about this, so I'm posting the article I wrote for LessWrong – calling for a more 'hippie' rationalism – on Substack instead (actually, there are two articles, and the links are at the end). Quick intro (who I am) My name is Melissa. I’m a passionate member of the core AISafety.com team. I was there with the core team when the site was just an empty domain and a loose vision in August 2023. Since then I have co-led, designed, and coded the first official version, and co-led, designed, and coded the second and current version, co-led two of three hackathons, led the creative direction for the current ecosystem map, and I’m currently working to bring lots of more amazing features to our site (stay tuned, it's going to be awesome)! I’ve also played major roles at AISafety.info, grantmaking.ai, Elicit, and minor roles at [...] --- Outline: (00:20) TLDR (00:51) Quick intro (who I am) (02:19) Leaving LessWrong in pursuit of ontological freedom ---

Note: We at PrincInt are launching a program teaching “Discernment” or “Taste” and generally spend a lot of our time in workshops and otherwise trying to teach academics how to think about AI such that their research ends up being useful. I had used a metaphor (building on Wentworth's postmortem) multiple times to explain what we do and why, and thought that it could be useful more broadly. All views are my own, and strategy takes do not represent everyone at PrincInt etc. AI writing level 0 (no non-trivial use)

People love to argue about whether AIs could recursively self-improve their way to superintelligence, each improvement making the next one easier as progress accelerates. I'm unsure. It could be that progress stalls at near-human or somewhat superhuman levels, that the ultimate qualitative limits of intelligence are not particularly beyond the human-level, or that the quantity of AI researchers involved is so vast that substantially superhuman AIs are created without strictly entering a super-exponential regime.

Wikipedia summarizes René Girard's mimetic theory like this: Man is the creature who does not know what to desire, and he turns to others in order to make up his mind. We desire what others desire because we imitate their desires. In other words, you think you want something for your own internal reasons, but the actual reason you want it is that someone else wants it, and that makes you desire the thing and compete for it. The "someone else" whose desires you copy is someone you care about. It could be a friend, a sibling, or a colleague. It could also be a famous person, or even a fictional character. (Not a stranger you don't care about.) This creates unnecessary social conflict, as people gradually converge on wanting the same thing, people start seeing each other as rivals, this attitude gets copied too, then people start hating each other, but who hates whom gets copied too, until everyone synchronizes at hating one specific person... and then they typically murder the person, and everyone feels happy about the conflict being resolved. The choice of the person to kill is arbitrary, in the sense that the selected person is [...] The original text contained 3 footnotes which were omitted from this narration. ---