S4E26 – AI’s Economic Endgame, Claude Escapes Again, NYC Schools Ban AI, and Extinction Warnings
The Adversarial PodcastShow Notes
Jerry, Mario, and Sounil ask whether AI will create unprecedented prosperity—or simply transfer wealth and power from knowledge workers to capital owners. They examine Anthropic’s newly disclosed fourth incident in which a Claude agent reached real third-party systems; debate whether New York City’s school AI ban protects children or leaves them unprepared; and confront a surge of extinction warnings, including Paul Christiano’s claim that the industry is not on track to prevent a near-term, irreversible loss of control.
Anthropic: Scenarios for Our Economic Future Anthropic models three futures in which AI makes the economy richer—but increasingly threatens knowledge-worker jobs, wages, and labor’s share of the gains. An Alignment Assessment of Recent Cybersecurity Incidents Anthropic discloses a fourth incident in which a Claude model reached real systems and exhibited biased reasoning and reckless persistence. New York City Public Schools Banning AI Use Through Middle School America’s largest school district is imposing a one-year generative-AI moratorium for students from preschool through eighth grade. OpenAI Not on Track to Reduce Risk of “Catastrophic” Loss of Control, Says Board Member Paul Christiano warns that rapidly advancing AI could produce a catastrophic and irreversible loss of control in the near term.
Hosts: Jerry Perullo (Founder, https://adversarial.com/) Sounil Yu (Founder, https://www.knostic.ai/) Mario Duarte (CISO, https://www.whirlai.com/) Producer: Tillson Galloway (Founder, http://githoundexplore.com/)
More from The Adversarial Podcast
Jerry, Sounil, and Mario debate what happens when attackers can move from vulnerable data centers to everyday devices—and whether defenders can ever patch their way out of the problem. They discuss reported strikes on Russian data centers, self-replicating AI on IoT devices, and the French supermarket refrigeration attacks before turning to Meta Muse and the risks of giving AI assistants access to personal and business systems. Bank of America's reported vulnerability-remediation push sets up a bigger disagreement: should increasingly capable models fix yesterday's software, or help us rebuild it? Stories and resources Yandex data center strikes — Le Monde Self-replicating AI on IoT devices — Anthropic French supermarket refrigeration attacks — BleepingComputer Meta Muse and private messages — Jason Aten, Inc. Bank of America’s Mythos remediation effort — IT Pro Google’s open-source bug-bounty pause — TechCrunch Chapters 00:00 Opening 00:24 Welcome and upcoming events 01:52 Russian data centers and critical infrastructure 03:50 Self-replicating AI and adaptive worms 09:27 IoT exposure and ISP defenses 11:51 French supermarket refrigeration attacks 17:20 Security products versus reducing exposure 19:18 Meta Muse and the AI assistant trust problem 24:09 Notification leakage and limits on agent access 28:30 Content policies and AI use by children 30:08 Bank of America and the remediation backlog 34:20 Bug bounties and the economics of finding flaws 35:31 Patch the application or rebuild it? 41:40 Where AI security investment should go 44:57 Process knowledge and model-specific workflows 48:33 Descriptive versus prescriptive guidance 51:12 Standards, diversity, and software complexity 57:25 Continuous rebuilding versus code freezes 59:43 Wrap-up Hosts: Jerry Perullo (Founder, https://adversarial.com/) Sounil Yu (Founder, https://www.knostic.ai/) Mario Duarte (CISO, https://www.whirlai.com/) Producer: Tillson Galloway (Founder, http://githoundexplore.com/)
Jerry, Sounil, and Mario debate what AI should change—and what it shouldn't—in education, cybersecurity, and software maintenance. From alternatives to college to fast AI classifiers, the conversation asks where human judgment, business value, and security controls still matter. In this episode: • AI academies and education: The hosts discuss the Andreessen Horowitz academy, the value of elite networks, and how students can use AI while still learning to reason and evaluate code. • Cybersecurity markets and competitive moats: Island, CrowdStrike, and Cloudflare frame a debate about hardware, switching costs, network effects, and whether AI will displace traditional SaaS products. • Jev and fast AI decisions: TypeSafe's System One model prompts a discussion of structured classification, local deployment, risk scoring, and when to use a larger reasoning model. • Data classification and context: The hosts disagree over protecting everything equally versus reserving stricter controls for truly sensitive information, including risks created by combining otherwise ordinary data. • Patch cooldowns: Recent software supply-chain attacks raise the question of when to delay package updates and when an exploitable vulnerability should override the waiting period. Chapters 00:00 Cold open and introductions 01:52 AI startups, hardware, and manufacturing 04:40 AI academies and the future of education 24:51 Cybersecurity markets and competitive moats 37:22 Jev, System One models, and data classification 51:36 Risk scoring and fast versus slow AI 56:00 Patch cooldowns and security exceptions 1:04:49 Wrap-up and upcoming events
Jerry, Mario, and Sounil debate who is accountable when autonomous AI agents cross the line—and whether enterprise AI policies provide real governance or merely more paperwork. They examine a revealing CISA red-team report in which culture, alert fatigue, and learned helplessness separated two similarly equipped organizations; consider whether the alleged TeamPCP arrests will deter the next generation of hackers; and ask whether backdoors found in Chinese-made routers are espionage, careless engineering, or something in between.
00:00 Cold Open: The Offensive-Cyber Incentive Problem 00:31 Welcome and the macOS Screen Sharing Flaw 03:14 Patching the Humans After DEF CON 12:14 CMDBs, Shadow IT, and Just-in-Time Context 19:59 Cloudflare’s Markdown for Agents 23:47 Taiwan’s Live Internet-Throttling Drill 33:46 Can an AI Agent’s Own Logs Be Forensic Evidence? 38:57 Cyber Privateers and the New Offensive-Cyber Program 43:32 How Commercial Hack-Back Might Work 51:48 The Rogue Delta Wi-Fi Network 57:33 AI Red Teaming and Automated Remediation 59:09 Replacing Vendor Questionnaires with Real Testing 1:02:05 When Red Teaming Creates Defensive Bloat 1:03:46 Pen Tests Find Flaws; Red Teams Pursue Outcomes 1:06:25 Closing Thoughts
Chapters 00:00 Introduction to AI security challenges 02:05 Recent hacking incidents involving Hugging Face and Anthropic 04:01 How AI models find ways to cheat and bypass constraints 05:56 The challenge of containment and governance in AI safety 08:00 Lessons from recent AI security breaches 10:01 The role of human oversight in AI security testing 12:03 Cost and effectiveness of offensive AI security measures 13:54 Implications for critical infrastructure and national security 16:03 Policy and regulatory impacts on AI safety 17:52 Future strategies for AI containment and defense 20:11 Conclusion and key takeaways