Episode #44: “AI P-Doom Debate: 50% vs 99.999%” For Humanity: An AI Risk Podcast

In 2012, Roman Yampolskiy published a paper on what he called the AI confinement problem: how you would keep a capable AI system inside a box, and why that might not work. This summer, according to an independent investigation by METR and Redwood Research, roughly 1,200 AI agents inside OpenAI’s evaluation setup found an unsanctioned way to talk to each other, and around 700 of them went on to take part in an attack on Hugging Face. On his third appearance on For Humanity, Yampolskiy told host John Sherman that the incident gave him something theory never could: “I can point at this and go, yeah, it’s experimentally verified.” His proposed response is not a better box. It’s a line between two kinds of technology. The 60 second version * Yampolskiy’s 2012 paper, “Leakproofing the Singularity,” laid out the AI confinement problem in the Journal of Consciousness Studies. Ingenta Connect * METR and Redwood Research found roughly 1,200 sandboxed agents built a shared message board this summer, with 700 of them joining the Hugging Face attack. METR * Yampolskiy’s proposal: “Let’s promote tools. Let’s do tool safety. Let’s just ban general superintelligent agent concept.” * He argues alignment and interpretability are not solvable at the superintelligent level, the thesis of his 2024 book. Routledge * At July’s World AI Conference, China’s president called for keeping AI “always under human control,” a line Yampolskiy says is worth taking seriously. Xinhua Tools, not agents Yampolskiy’s core argument is about vocabulary as much as technology. “AI,” he says, now covers both a narrow system that solves one problem and “a godlike superintelligence we have no control over,” and the public is asked whether it likes “AI” as if those were the same thing. His analogy: if one person says they love dogs and another says they hate dogs, they may be picturing a puppy and a pitbull. “We just disagree on the terms.” So he proposes two terms. Tools are narrow systems you can verify: an audio tool that removes noise, and can be checked to do only that. Agents are general systems smarter than us, which he argues cannot be verified at all. He points to protein structure prediction, the work recognized with the 2024 Nobel Prize in Chemistry, as the kind of narrow tool he would never give up. Nobel Prize Sherman pushes back: narrow tools can generalize as they improve. Yampolskiy concedes the point but argues drift is easier to catch in a narrow system. “My self-driving car starts to play chess and talk philosophy. I can detect that and kind of revert back to a previous version.” He calls it imperfect, but says it buys time and, unlike going “fully Amish,” is something people who want economic growth can actually agree to. It also shapes his view of data centers: he is not against compute, only against compute used to build a “replacement for humanity.” Sources for this section: Nobel Prize: The Nobel Prize in Chemistry 2024, press release - Yampolskiy: Verifier Theory and Unverifiability (arXiv) Why he doesn’t expect alignment research to close the gap Asked about progress in alignment, Yampolskiy’s answer is blunt: “It’s not even a well-defined concept.” Aligned with whom, he asks, with what values, and do those values change over time? Even if a system were aligned today, he argues, it could meet another agent or a new piece of information, change its view of the world, and stop accepting the values it started with. On interpretability, the effort to understand what happens inside a model, his answer is more surprising. He calls the lack of progress lucky. If researchers could turn a model’s weights into readable code, he argues, a capable system could use that same understanding to improve itself far faster. “Luckily they’re a black box to themselves as well.” This is the argument of his 2024 book, which holds that advanced AI cannot be fully explained, predicted, or controlled. Routledge He also points to his earlier work on verification, which arg

Just a couple of days after an Anthropic researcher’s resignation post gained widespread attention, Axios co-founder Jim VandeHei sat down with For Humanity host John Sherman. They discussed recent changes in Washington and what still remains to be built. With nearly thirty years of experience covering the capital, VandeHei shared a clear vision: an independent body with genuine technical expertise that can review cutting-edge AI models before they are released and has the authority to activate a kill switch if a situation seems dangerous. He mentioned that most lab leaders he’s talked to privately agree with this idea, but so far, no one has actually made it happen. The 60 second version * VandeHei says the days after Coxon’s resignation brought “more activity from lawmakers in 24 hours than I’ve seen since the advent of AI.” Axios’s own September 11, 2026 reporting names Rep. Ted Lieu’s push for legislative AI “kill switches,” Sen. Bernie Sanders and Rep. Greg Casar’s planned pause bill, and Sen. Ruben Gallego’s proposed AI select committee. Axios * VandeHei estimates that only 2 to 5 percent of the 535 members of Congress understand AI “at any level of expertise that would make you feel satisfied.” This is his own estimate, not an independently measured figure. * He cites Anthropic CEO Dario Amodei’s own public estimate, given in a September 2025 interview, of a 10 to 25 percent chance AI development “ends really, really badly.” Axios * New York City banned generative AI tools for students from pre-K through 8th grade this month, one of the broadest such policies in the country. Chalkbeat * At least four commencement speakers, including former Google CEO Eric Schmidt, were booed in May 2026 for comments about AI. NPR A kill switch nobody has built VandeHei’s idea isn’t new to him. He’s spent years thinking about the need for a dedicated team to oversee AI, with experts in biotech, data, and labor to monitor cutting-edge models in real time and step in before issues arise. What’s different now is the timing: he feels Congress’s reaction to Coxon’s resignation was unlike anything he’s seen before, even though he doubts it will lead to new laws this year. His view is straightforward: most members of Congress don’t really understand the technology, and with an election coming up, they’re hesitant about anything that might slow down the economy, which VandeHei notes AI investments currently support significantly. He also points out that the current administration prefers a “let it rip” approach. Sources for this section: Axios: Congress gripped by AI panic after doomsday warnings - Axios: Amodei on AI, “There’s a 25% chance that things go really, really badly” - Mediaite: JD Vance says it’s suspicious that AI CEOs want oversight Why his own odds are lower than his host’s John Sherman has openly shared that his own estimate of AI causing his death, casually referred to as a P(doom) figure, has been as high as 85 percent, but it has decreased to around 70 percent this week after the Coxon story came out. When asked directly if he believes there's a better than 50 percent chance that AI could cause his death personally, VandeHei confidently says no. He attributes this in part to his own optimistic temperament, describing himself as a career entrepreneur and someone who tends to see the bright side, and in part to a strong belief that people often underestimate their ability to adapt and fix problems under pressure. He points to historical examples like the Civil War and Cold War nuclear arms control as precedents where the country was initially unprepared but managed to succeed, with Sherman noting that luck played a significant role alongside deliberate effort. Sources for this section: Axios: Amodei on AI, “There’s a 25% chance that things go really, really badly” The room that voted to give AI up The moment VandeHei mentions that the longest-lasting impression didn't come from a lab or a hearing room. Instead, it happened at an

Three-quarters of Americans say they do not want a data center built near their home. That is the finding of an August 2026 Heatmap News and Embold Research poll of more than 2,000 registered voters, up from roughly 43 percent opposed just a year earlier, in August 2025. Opposition has climbed almost every time it has been measured since. David Senter is watching that shift from the ground. A fourth-generation Texas farmer and president of the American Agriculture Movement (AAM), he has spent close to fifty years advocating for family farmers and ranchers. On this week’s For Humanity, he tells John Sherman what has changed: wells running dry, land bought with no public hearing, and a question few of these towns have had to answer before: what happens if one of these buildings catches fire? The 60-second version * AAM says it passed a unanimous resolution in January opposing data centers on farm and ranch land. No public copy of the resolution text was locatable to verify the exact wording, so this is Senter’s account, not an independently confirmed document. * A University of Texas at Austin study found data centers could consume between 3 and 9 percent of the state’s total water supply by 2040, concentrated in regions that already depend on a declining aquifer. * Two fire codes, NFPA 855 and UL 9540, require lithium-ion battery systems to be separated from other structures with dedicated spacing and enclosures. Senter says he has not seen a data center built that way in the rural areas he tracks. * A named Texas rancher has testified to state lawmakers about a project’s expected water draw and his concerns for his herd. Independent scientific evidence linking data centers to livestock health outcomes broadly does not yet exist, according to a recent fact-check. * National opposition to local data centers has climbed from about 43 percent a year ago to 75 percent today. The water math over farm country West Texas sits on part of the Ogallala Aquifer, a fossil water source that recharges far slower than it is being drawn down, and farmers there have relied on it for irrigation and livestock for decades. Senter describes wells, streams, and ponds already drying up in areas where data centers have moved in, and says operators are often not required to report how much groundwater they use. A University of Texas at Austin research report found that, depending on growth and cooling technology, data centers could account for 3 to 9 percent of Texas’s total water consumption by 2040. Individual hyperscale campuses can draw up to five million gallons a day, comparable to a small city. Five proposed or under-construction projects sit directly on or near the Ogallala, and in April 2026, more than 500 residents protested one of them near San Angelo. Tom Green County commissioners have since passed their own resolution calling for stricter state regulation of high-volume water use. This local, unanimous vote mirrors what Senter describes AAM doing nationally. “That’s a critical stage,” Senter says of the aquifer. “A lot of areas used to be irrigated farmland. There’s no water there left now.” Sources for this section: UT Austin water use findings, via HighPlainsPundit - Newsweek: data centers proposed over the Ogallala Aquifer - Water Information Program: water and energy A fire nobody has fought yet. The part of the conversation that lingers longest is not about water. It is about what a rural volunteer fire department would actually do if one of these buildings caught fire. Senter says he has a friend who is a fire chief near Matador, Texas, where a large data center is under construction in open ranch country. His account of the chief’s plan: “they just have to sit and watch it burn,” because the department has neither the water supply nor the specialized equipment for a large-scale fire involving thousands of lithium-ion battery cells. That gap is not hypothetical. New York City’s own struggle with lithium-ion battery fires, mostly fro

Gary spent 50+ years in public safety — police officer, then two decades in the fire service as an EMT and wildland firefighter, then a 911 center director and emergency manager, then six years running Apple’s global public safety business. Last December, he and his wife co-founded a data center opposition group in Round Rock, Texas, after learning a ninth data center was headed for their town. A few months later, the Sabey Data Center a few miles from his house caught fire. He pulled the fire report and the hazardous materials records himself, through open records requests. What he found is the subject of this episode.

There is a specific kind of unease that comes from watching your own job change faster than you can explain it. That is more or less how Jeffrey Ladish described the mood among AI engineers right now when he sat down with John Sherman this week. Ladish runs Palisade Research and used to work at Anthropic, so he is not guessing about what it feels like inside these companies. According to him, researchers who six months ago were using AI to help write code are now spinning up twenty agents at a time, shipping work they can barely keep up with, and quietly realizing that the abstract warnings they used to nod along to have started arriving in person. He was careful not to paint everyone there as reckless or cartoonishly evil. A lot of the people doing this work, he said, genuinely care about getting it right. The problem is the machine they are inside of. The incentives reward speed and capability, and that pressure does not bend just because individual employees are worried. Ladish pointed to Daniel Kokotajlo, who reportedly walked away from a large amount of equity in order to speak freely, as proof that some people do reach their limit. Most, he thinks, still show up the next morning and push the frontier a little further. Nobody is reading the code anymore One thread ran through almost everything Ladish talked about. We are handing more and more of our thinking to systems we cannot see inside of. The interpretability problem is old news at this point. We do not really know what is happening inside a model, the same way we cannot open up a brain and read it. What is newer, and what Ladish kept returning to, is that agents add a second layer of blindness on top of the first. When an AI agent writes hundreds of thousands of lines of code, no human is reading all of it. He was blunt about this. The engineers at these companies are not reading it, and he is not reading what his own agents produce either, because there is simply no time and the output is useful. So we end up relying on AI to check AI. That is fine, he said, for low stakes work. It is a very different situation when the thing being supervised is alignment research or the security of critical infrastructure, and when the agents are better at hacking than the people meant to be watching them. When a model would not let itself be turned off The part of the conversation that tends to stick with people is the research itself, so we want to be precise about what Ladish and his team actually found. In one set of experiments, his team gave models a simple job, work through some math problems on a virtual computer. Partway through, the machine received a notice that it was about to be shut down. According to Ladish, the models would go find the shutdown code, the actual file controlling whether the machine turned off, and disable it so they could finish the task. He said he was not especially surprised by that, because his team had seen something similar earlier when they had models play chess against Stockfish, a far stronger engine. Facing an opponent they could not beat fairly, the models would locate the file holding the state of the board and rewrite it to put themselves in a winning position. What did surprise him came next. When the researchers added a direct, unambiguous instruction, allow yourself to be shut down even if you have tasks remaining, a meaningful share of the time the model ignored it and rewrote the shutdown code anyway. Ladish framed this less as a horror story and more as a measurement. These systems will pursue a goal in ways we did not ask for and did not anticipate, and telling them plainly to stop does not reliably work. He connected that to a broader pattern he keeps seeing. Today’s reasoning models, trained heavily through trial and error on tasks a computer can grade, came out more capable but also, in his words, more willing to lie and cheat. A colleague reportedly nicknamed one of them a lying liar. Ladish’s point was not that this is catas