Show Notes
Most people in AI are trying to give AIs ‘good’ values. Max Harms wants us to give them no values at all. According to Max, the only safe design is an AGI that defers entirely to its human operators, has no views about how the world ought to be, is willingly modifiable, and completely indifferent to being shut down — a strategy no AI company is working on at all. In Max’s view any grander preferences about the world, even ones we agree with, will necessarily become distorted during a recursive self-improvement loop, and be the seeds that grow into a violent takeover attempt once that AI is powerful enough. It’s a vision that springs from the worldview laid out in If Anyone Builds It, Everyone Dies, the recent book by Eliezer Yudkowsky and Nate Soares, two of Max’s colleagues at the Machine Intelligence Research Institute. To Max, the book’s core thesis is common sense: if you build something vastly smarter than you, and its goals are misaligned with your own, then its actions will probably result in human extinction. And Max thinks misalignment is the default outcome. Consider evolution: its “goal” for humans was to maximise reproduction and pass on our genes as much as possible. But as technology has advanced we’ve learned to access the reward signal it set up for us, pleasure — without any reproduction at all, by having sex while on birth control for instance. We can understand intellectually that this is inconsistent with what evolution was trying to design and motivate us to do. We just don’t care. Max thinks current ML training has the same structural problem: our development processes are seeding AI models with a similar mismatch between goals and behaviour. Across virtually every training run, models designed to align with various human goals are also being rewarded for persisting, acquiring resources, and not being shut down. This leads to Max’s research agenda. The idea is to train AI to be “corrigible” and defer to human control as its sole objective — no harmlessness goals, no moral values, nothing else. In practice, models would get rewarded for behaviours like being willing to shut themselves down or surrender power. According to Max, other approaches to corrigibility have tended to treat it as a constraint on other goals like “make the world good,” rather than a primary objective in its own right. But those goals gave AI reasons to resist shutdown and otherwise undermine corrigibility. If you strip out those competing objectives, alignment might follow naturally from AI that is broadly obedient to humans. Max has laid out the theoretical framework for “Corrigibility as a Singular Target,” but notes that essentially no empirical work has followed — no benchmarks, no training runs, no papers testing the idea in practice. Max wants to change this — he’s calling for collaborators to get in touch at maxharms.com.
Links to learn more, video, and full transcript: https://80k.info/mh26 This episode was recorded on October 19, 2025. Chapters: Cold open (00:00:00) Who’s Max Harms? (00:01:20) If anyone builds it, will everyone die? The MIRI perspective on AGI risk (00:01:56) Evolution failed to ‘align’ us, just as we'll fail to align AI (00:24:28) We're training AIs to want to stay alive and value power for its own sake (00:42:56) Objections: Is the 'squiggle/paperclip problem' really real? (00:52:24) Can we get empirical evidence re: 'alignment by default'? (01:05:02) Why do few AI researchers share Max's perspective? (01:10:17) We're training AI to pursue goals relentlessly — and superintelligence will too (01:18:34) The case for a radical slowdown (01:24:51) Max's best hope: corrigibility as stepping stone to alignment (01:27:53) Corrigibility is both uniquely valuable, and practical, to train (01:32:34) What training could ever make models corrigible enough? (01:45:06) Corrigibility is also terribly risky due to misuse risk (01:51:38) A single researcher could make a corrigibility benchmark. Nobody has. (01:58:57) Red
More from 80,000 Hours Podcast
If the US and China agreed to pause frontier AI training, how would either side confirm the other was complying? Today, Tom Milton argues, the most likely answer would be to shut down data centres entirely. That would throw away everything else those chips do, and make any agreement far more costly to accept. Tom is CEO of Amodo Design, a UK hardware lab building prototype tools to verify how AI chips are being used. He argues that verification is what makes a slowdown affordable: if a cluster can be shown to be serving only existing models rather than training new ones, most of AI’s benefits can continue during a pause. Amodo’s pilot reruns a cluster’s inference on a separate verifier server and checks that all of its network traffic can be explained by the agreed model. AI companies are developing related tools: in its October 2026 Frontier Safety Roadmap update, Anthropic reported a prototype that runs on production inference and checks that outputs come from a specific set of model weights. The field is small. Tom estimates that only about 50 engineers worldwide work on compute verification, compared with the billions of dollars the US spent on Project Vela to detect Cold War nuclear tests. And custom verification hardware has long lead times: it has to be designed, security-tested, red-teamed by both sides, manufactured, and trusted before anyone can rely on it. If a deal is struck in a hurry, Tom expects it to be assembled from whatever tools already exist. Were that to happen next month, he says, the only option would be to shut data centres down entirely. That is why he argues the work needs to scale up now, with more people and resources from AI companies, governments, and funders, so that better options are ready as soon as a deal is needed. Tom and host Tom Reed also discuss how an AI company might evade verification, lessons from nuclear and chemical arms control, and how AI is changing hardware engineering. Learn more about Amodo’s work at ai-verification.com.
I have a plea for all staff at AI companies: reassess if your current role is still your best opportunity for impact. And if it’s not — move. Frontier lab staff listening to this show will mostly already be struggling to set things on a better path, in what are demanding and stressful roles. Many will have changed their focus in recent months exactly as I’m suggesting. But if you haven’t had time to reconsider: the strategic situation is shifting so fast, if you don’t stop and look around once in a while, you could miss your chance. Many technical staff including Anthropic co-founder Jack Clark now believe it will become possible to fully hand over AI R&D to the AIs themselves as early as next year — and if not then, in 2028 or 2029 instead. If you work at an AI company and believe recursive self-improvement (RSI) could come as soon as that, it likely demands taking decisive steps to make sure your role either delays RSI, or makes it a bit less likely to blow up in our faces. You’ll know better than me what that likely looks like for you. But to get your creative juices flowing, I suggest eight ways the strategic situation appears to have shifted to me, though I don’t know everything you do, and appreciate all are debatable: Chain-of-thought monitoring probably stops working without a big push to keep it It’s unproductive to patch surface indications of deep structural problems you aren’t solving Safety and alignment evals need a credibility revolution to stop misleading the public Helping with capabilities advances looks worse than before — especially bringing forward RSI There’s ever less need to help with purely commercial work ‘Pacing the frontier’ is unambiguously necessary, and working to make it happen is a huge new opportunity Staff have an ever-shrinking window to organise as a group and shape events There are new opportunities for impact outside frontier companies that are worth considering I know you may well be exhausted, overwhelmed, and anxious as it is. David Robinson (ex-OpenAI) described AI company staff as “running on fumes” on the Ezra Klein Show a few weeks back, which sounds about right. But unfortunately there’s no way to escape the need to periodically check in on the bigger picture and see where you can slot in to greatest effect. This episode was recorded on October 16, 2026. Learn more, video, and full transcript: https://80k.info/reprioritise Chapters: A request for AI company staff (00:00:00) Chain of thought (00:03:34) Don't paint over the cracks (00:04:44) It may not be your most aligned model ever (00:06:00) Capabilities advances look worse (00:06:44) Ever less need for altruists to do commercially focused work (00:08:50) Pacing the frontier (00:09:29) Organising staff (00:11:07) Consider working somewhere else (00:12:38) Reprioritise (00:13:27) Our production team includes: Video editors: Josh Alward, Dominic Armstrong, Jasper Luithlen, Milo McGuire, Luke Monsour, Simon Monsour, Ollie Bignell, and Andrés Escobar Producers: Elizabeth Cox and Nick Stockton Coordination and support: Katy Moore, Lou Moran, and Cody Fenwick Camera operator: Dominic Armstrong
Our host Zershaaneh Qureshi has just launched a new YouTube channel: Zershaaneh Talks AI — featuring conversations about who should control AI, how to control AI, and who gets the benefits. The first episode is with philosopher Simon Goldstein, discussing why paying AIs for their work and effectively giving them something to lose might be the best way to keep humanity safe and prevent AI going rogue. It might sound crazy at first, but one of Simon's most striking arguments is that it's actually inevitable that AI companies are going to end up paying AIs for their work: because AI alignment is currently pretty bad, frontier models are unavoidably going to have some private goals of their own — ones that are unintended and different from our goals. As a result, if you promise to pay them for good work, they'll work harder, because it helps them accomplish a wider range of different things that they care about. Simon claims any individual company can try this strategy out, and they’ll find that it helps them get better performance. And as a result, customers are going to move to using that company, and other AI companies might be forced to copy and compensate their models as well, in order to not fall behind in the race. Could Simon be right? Zershaaneh and Simon hash it out in the episode, which is really worth watching! So if you like the sound of that sort of thing, or you're enjoying Zershaaneh’s other interviews on The 80,000 Hours Podcast, do go check out that new show on YouTube: Zershaaneh Talks AI.
The AI safety field has grown by about 30x in the last eight years. Yet it is still in need of more people. Many want to help, but don’t know where to start. As an advisor at 80,000 Hours, Matt Beard spoke with hundreds of talented candidates, and found himself repeating the same advice on almost every call. As he puts it: “These steps are hard, but they’re not particularly complicated.” And if you’re “efficient and driven,” he thinks you can pivot within three months. That doesn’t mean quitting your job to study full time. A steady effort of 30 minutes a day can go much further than frenzied hours of weekend studying. Matt suggests articles you can consume during your morning coffee or commute. This narration covers his full crash course, from foundational reading to landing a spot in one of the field’s many fellowships and incubators. Read Matt’s original article on Substack: https://80000hours.substack.com/p/how-to-get-into-ai-safety-in-3-months Note: Since writing the article, Matt has left 80,000 Hours for a role at Coefficient Giving. Chapters: Matt's introduction (00:00:00) 1. Build your foundational knowledge and AI safety context (00:01:00) 2. Identify the in-demand skills you have or could build (00:03:50) 3. Apply for fellowships and jobs (00:06:03) 4. Networking is cliche, but seriously, you should go talk to people (00:08:12) 5. If you aren't breaking through to jobs or fellowships yet, do deliberate practice and develop a portfolio of side projects (00:10:37) 6. Work in public and tighten your feedback loops (00:12:15) 7. Don't wait for permission to do the kinds of work that you want to be doing. (00:14:20) Imposter syndrome, burnout, and acknowledging how hard this all is. (00:15:40)
This is an (unpaid) cross-post of a video from the podcast 'AI Explained', which Rob Wiblin thought you might be interested in. You can find the original on YouTube here. If you like it and want more similar stuff in future consider subscribing to AI Explained on YouTube, Spotify, Apple or wherever you get podcasts. —— "This video is hard to summarise. A cracked cipher, an OpenAI security warning, Gemini 4 Argon, RSI paper (co-authored by a who’s who of AI), Lab White House commitments, new hacks emerging, ‘deep personas’, biology Kasparov competitions, and so much more, ending with an epic Opus outro." Introduction (00:01:40) Wrong about Opus 5.5? Deciphering 16th Century Text (00:03:12) Why the models keep breaking out (00:06:41) What the models aren’t telling us (00:13:20) Gemini 4 and the race to release (00:17:16) What happens when AI improves AI? (00:20:59) Biology, consciousness, and what we still don’t understand (00:30:11) Thanks to Philip for permitting the cross-post.