Three: Carl Shulman on the economy and national security after AGI
The 80,000 Hours Podcast on Artificial IntelligenceShow Notes
The human brain runs on just 20 watts — a fraction of a cent worth of electricity per hour. What would happen if AI could do the same? Plenty of people have toyed with this question. But perhaps nobody has followed through and considered all the implications as much as Carl Shulman, whose behind-the-scenes work has greatly influenced how leaders in artificial general intelligence (AGI) picture the world they’re creating. Carl simply follows the logic to its natural conclusions, leading to a world where: One cent of electricity buys what costs hundreds of dollars today — medical advice, company management, scientific research — triggering a scramble to manufacture chips and apply them to the most lucrative forms of intellectual labour The world’s supply of AI researchers explodes from 10,000 to 10 million or more, enormously accelerating further AI progress Companies operated entirely by AIs are much faster and cheaper than those that depend on people for decision making, and humans are progressively driven out of business The technical challenges of robotics are rapidly overcome — leading to strong, fast, precise, and tireless robot workers capable of any physical work, and a rush to build billions of them Human population levels become irrelevant to economic growth, which now depends on how quickly machines can replicate their components. Given how quickly complex biological systems can reproduce — some in a matter of days — doubling every few months may be a conservative estimate No country can afford not to participate in the economic explosion. Delay, and your rivals’ economies grow 10x, 100x, then 1,000x larger than yours, leaving you ultimately disempowered As the economy grows, each person could afford the equivalent of a team of hundreds of machine ‘people’ to help them with every aspect of their lives. But with growth rates this high, it doesn’t take long to reach Earth’s physical limits — the toughest to engineer around being the planet’s ability to release waste heat. If this machine economy and its insatiable demand for power generates more heat than the Earth radiates into space, the planet will rapidly heat up and become uninhabitable for biological organisms. This eventually creates pressure to move economic activity off-planet. There’s little need for computer chips to be on Earth, and solar energy and minerals are more abundant in space. So you could develop populations of billions of digital scientific researchers orbiting in space, sending the results of their work — like drug designs — back to Earth. These are just some of the wild implications if AGI could merely match what evolution has already managed. In this interview with host Rob Wiblin, Carl explains the above, and Rob pushes back on whether that’s realistic or just a cool story: If this is where we’re heading, how come economic growth remains slow now and isn’t really increasing? Why have computers and computer chips had so little effect on economic productivity so far? Are self-replicating biological systems a good comparison for self-replicating machine systems? Isn’t this just too crazy and weird to be plausible? What bottlenecks might we encounter supplying energy and resources to this growing economy? Could there be severely declining returns to bigger ‘brains’ and more training? Wouldn’t humanity get scared and pull the brakes if such a transformation kicked off? If this is right, why don’t economists agree? In the last section of the episode, Carl addresses the moral status of machine minds themselves. Would they be conscious or otherwise have a claim to moral rights? And how might humans and machines coexist with neither side dominating or exploiting the other? Learn more and read the full transcript on the 80,000 Hours website. This episode is the first part of Rob’s marathon interview with Carl Shulman in 2024. The second episode is on government and society after AGI, and you can listen to them in either order. Chapters: Cold open (00:
More from The 80,000 Hours Podcast on Artificial Intelligence
What might it be like to live through the creation of AI that surpasses human abilities? That future may be closer than you think.
The 20th century saw unprecedented change: nuclear weapons, satellites, the rise and fall of communism, third-wave feminism, the internet, postmodernism, game theory, genetic engineering, the Big Bang theory, quantum mechanics, widespread birth control, and more. Now imagine all of it compressed into just 10 years.
We don’t yet have a reliable way to tell whether an AI model is genuinely trying to help us — or faking it. A model might sincerely want to do exactly what you ask. Or it could be happy to secretly cheat, as long as its answer gets positive reinforcement during training. It might even follow the rules just to gain our trust, all while concealing goals of its own. The problem is: each of these three motivations scores the same during testing. Ajeya Cotra — previously a senior research analyst at Coefficient Giving, now working at METR (Model Evaluation & Threat Research) — explains how dangerous this dynamic could become as we train very general and very capable AI models. She likens humanity’s future trust in AI systems to an orphaned child who inherits a $1 trillion company. This child has to hire someone to run the company, guide his life, and manage his wealth — but he can only choose this person based on a work trial or interview that he designs, with no resumes or reference checks. And, because he’s so rich, all sorts of people apply — for all sorts of reasons. Some applicants will truly want to help. But the role will attract others who only pretend to care while they’re being monitored, but intend to exploit the child as soon as they can get away with it. Like a child trying to judge adults, at some point humans will need to judge the trustworthiness and reliability of machine learning models that are as goal-oriented as people, and greatly outclass us in knowledge, experience, breadth, and speed. And we can’t rely on models’ performance during training tasks to guide us, as current reinforcement learning would give the same grades to three vastly different motivations: Saints — models that genuinely care about doing what we want Sycophants — models that just want positive reinforcement for a ‘correct’ result, even if they get there with actions they know we wouldn’t approve of Schemers — models that don’t care about our interests at all, and only behave correctly as long as it serves their own agenda Worse still, training might actively encourage deception. Imagine training a model to run a business, and measuring its success by the balance in its bank account. A highly capable model might experiment with dishonest strategies. Maybe it steals some money and covers it up. (This isn’t a hypothetical worry; models often come up with creative — sometimes undesirable — approaches during training that their developers didn’t anticipate.) A model that cheats and covers its tracks would look like a star performer — and get reinforced for exactly that behaviour. If cheating is only caught some of the time, the model still might not learn to stop deceptive behaviour. Instead, it might learn that deceiving without being caught gives it a competitive advantage. In this conversation, Ajeya and host Rob Wiblin discuss the above, as well as: How to predict the motivations a neural network will develop through training Whether AIs in training will functionally understand that they’re AIs being trained Stories of AI misalignment that Ajeya doesn’t buy Analogies for AI, from octopuses to aliens to can openers Why it’s smarter to have separate ‘planning AIs’ and ‘doing AIs’ The benefits of only following through on AI-generated plans that make sense to human beings Which approaches for fixing alignment problems Ajeya is most excited about, and which she thinks are overrated How we might demonstrate actually scary AI failure mechanisms Learn more and read the full transcript on the 80,000 Hours website. This episode was originally released in May 2023. Chapters: Rob’s intro (00:00:00) The interview begins (00:02:38) How Ajeya’s views have changed since 2020 (00:05:09) Are neural networks more like a sped-up version of evolution, or a slower version of human learning? (00:17:42) Situational awareness (00:26:10) Misalignment stories Ajeya doesn't buy (00:42:03) The orphan heir with a trillion-dollar fortune (00:59:14) Saints, Sycophants,
The most important political question in the age of advanced AI might not be who wins elections. It might be whether elections continue to matter at all. We tend to imagine the death of democracy as a dramatic event: a coup, tanks in the streets, a strongman tearing up the constitution. But Rose Hadshar, researcher at Forethought Research, believes AI-enabled power concentration could be far quieter — and far harder to stop. She foresees something insidious: an elite group with access to such powerful AI capabilities that the normal mechanisms for checking power — law, elections, public pressure, the threat of strikes — cease to have much effect. They might continue to exist on paper, but become ineffectual in a world where humans are no longer needed for even the largest-scale projects. Almost nobody wants this to happen, but we may find ourselves unable to prevent it: If AI disrupts our ability to make sense of things, will we even notice power being concentrated? If AI replaces human labour, what leverage will citizens have left to resist? And what does all of this imply for the institutions we’re relying on to prevent the worst outcomes? Rose has answers, and they’re not all reassuring. But she’s also hopeful we can make society more robust against these dynamics. We’ve got literally centuries of thinking about checks and balances to draw on. And there are some interventions she’s excited about — like building sophisticated AI tools for making sense of the world, or ensuring multiple branches of government have access to the best AI systems. In this conversation, Rose and host Zershaaneh Qureshi discuss all of this, and more: Three dynamics that could reshape political power in the AI era Why AI-powered tyranny would be uniquely difficult to topple How power concentration compares to ‘gradual disempowerment’ by AI Slower-moving scenarios that could still get scary Which interventions could genuinely work — and which might backfire Rose's most promising approaches to fighting back Why a ‘Manhattan Project’ approach to AI should worry you — and why international projects aren’t automatically safe either Learn more and read the full transcript on the 80,000 Hours website. This episode was originally released in March 2026. Chapters: Cold open (00:00:00) Who’s Rose Hadshar? (00:01:02) Three dynamics that could reshape political power in the AI era (00:02:38) AI gives small groups the productive power of millions (00:13:07) Dynamic 1: When a software update becomes a power grab (00:21:13) Dynamic 2: When AI labour means governments no longer need their citizens (00:32:06) How democracy could persist in name but not substance (00:46:18) Dynamic 3: When AI filters our reality (00:56:13) Good intentions won’t stop power concentration (01:09:52) Slower-moving worlds could still get scary (01:25:32) Why AI-powered tyranny will be tough to topple (01:33:40) How power concentration compares to “gradual disempowerment” (01:40:16) Some interventions are cross-cutting — and others could backfire (01:46:03) What fighting back actually looks like (01:57:33) Why power concentration researchers should avoid getting too “spicy” (02:06:36) Why the “Manhattan Project” approach should worry you — but truly international projects might not be safe either (02:11:46) Rose wants to keep humans around! (02:14:40) Video and audio editing: Dominic Armstrong, Milo McGuire, Luke Monsour, and Simon Monsour Music: CORBIT Coordination, transcripts, and web: Nick Stockton and Katy Moore
When OpenAI announced a deal to build massive data centres in the UAE, it celebrated that it was “rooted in democratic values” — a "clear alternative to authoritarian versions of AI." The UAE scores 18 out of 100 on Freedom House’s democracy index. Political parties are banned, elections are fake, and dissidents are persecuted. Saudi Arabia has received a similar deal. This is what AI geopolitics looks like in practice: messy, contradictory, and enormously consequential. The two superpowers competing to build superintelligence — the United States and China — are “barely talking at all.” You might expect two rivals developing potentially the most powerful and militarily significant technology in history to be in constant negotiation about how to deploy it without coming to blows. Instead, the little dialogue that exists keeps collapsing. That’s the assessment of Helen Toner, director of the Center for Security and Emerging Technology — DC’s top think tank focused on the geopolitical and military implications of AI — who has been closely tracking the US’s AI diplomacy since 2019. Helen isn't sure productive talks are even possible yet. At the government level, there's almost no shared understanding between the US and China of what artificial general intelligence (AGI) is, whether it could arrive soon, or whether it poses serious risks. And without agreement on the problem, negotiating solutions is nearly impossible. And while the US struggles to engage its rival, it’s empowering its autocratic allies. If AI capability really does determine future national power, the US has just approved massive data centres with "hundreds of thousands of next-generation Nvidia chips," handing world-class supercomputers to Gulf autocracies — countries that also conduct joint military exercises with China and whose rulers maintain tight personal and commercial relationships with Chinese leaders. The justification? "If we don't sell it, China will." But that claim is transparently false: severe production constraints and US export controls mean that China can’t come close to matching what these deals provided. In this conversation recorded in Washington, DC, host Rob Wiblin and Helen discuss the above, plus: How China exaggerates its chip production for strategic gain The confusing and conflicting goals of US AI policy towards China Whether it matters that China could steal frontier AI models trained in the US Whether Congress is starting to take superintelligence seriously Why Helen rejects ‘non-proliferation’ as a model for AI Learn more and read the full transcript on the 80,000 Hours website. This episode was originally released in November 2025. Chapters: Cold open (00:00:00) Who’s Helen Toner? (00:01:03) Helen’s role on the OpenAI board, and what happened with Sam Altman (00:01:32) The Center for Security and Emerging Technology (CSET) (00:07:45) CSET’s role in export controls against China (00:10:59) Does it matter if the world uses US AI models? (00:21:58) Is China actually racing to build AGI? (00:27:56) Could China easily steal AI model weights from US companies? (00:39:13) The next big thing is probably robotics (00:47:45) Why is the Trump administration sabotaging the US high-tech sector? (00:49:24) Are data centres in the UAE “good for democracy”? (00:52:41) Will AI inevitably concentrate power? (01:07:49) “Adaptation buffers” vs non-proliferation (01:30:20) Will the military use AI for decision-making? (01:38:23) “Alignment” is (usually) a terrible term (01:45:11) Is Congress starting to take superintelligence seriously? (01:47:42) AI progress isn't actually slowing down (01:50:10) What's legit vs not about OpenAI’s restructure (01:58:03) Is Helen unusually “normal”? (02:01:33) How to keep up with rapid changes in AI and geopolitics (02:05:19) What CSET can uniquely add to the DC policy world (02:08:29) Talent bottlenecks in DC (02:16:03) What evidence, if any, could settle how worried we should be about AI risk? (02:19:07) Is CSE