Show Notes
Philosopher Dr. David Thorstad tears into one of AI safety's most influential arguments: the singularity hypothesis. We discuss why the idea of recursive self-improvement leading to superintelligence doesn't hold up under scrutiny, how these arguments have redirected hundreds of millions in funding away from proven interventions, and why people keep backpedaling to weaker versions when challenged. David walks through the actual structure of singularity arguments, explains why similar patterns show up in other longtermist claims, and makes the case for why we should focus on concrete problems happening right now like poverty, disease, the rise of authoritarianism instead of speculative far-future scenarios. Chapters
(00:00) - Intro
(02:13) - David's background
(08:00) - (Against) The Singularity Hypothesis
(29:46) - Beyond the The Singularity
(39:56) - What We Should Actually Be Worried About
(49:00) - Philanthropic Funding
LinksDavid's personal website Reflective Altruism, David's blog The Singularity Hypothesis David's Philosophical Studies article - Against the singularity hypothesis Time "AI Dictionary" page - Singularity EA Forum blogpost - Summary: Against the singularity hypothesis Journal of Conciousness Studies article - The Singularity: A Philisophical Analysis Interim Report from the Panel Chairs: AAAI Presidential Panel on Long-Term AI Futures Epoch AI blogpost - Do the returns to software R&D point towards a singularity? Epoch AI report - Estimating Idea Production: A Methodological Survey Funding References LessWrong blogpost - An Overview of the AI Safety Funding Situation AISafety.com funding page Report - Stanford AI Index 2025, Chapter 4.3 Forbes article - AI Spending To Exceed A Quarter Trillion Next Year AI Panic article - The “AI Existential Risk” Industrial Complex GiveWell webpage - How Much Does It Cost To Save a Life? Wikipedia article - Purchasing power parity Pascal's Mugging and the St. Petersburg Paradox Wikipedia article - St. Petersburg Paradox Conjecture Magazine article - Pascal’s Mugging and Bad Explanations neurabites explainer - Ergodicity: the Most Over-Looked Assumption Wikipedia article - Extraordinary claims require extraordinary evidence The Time of Perils Global Priorities Institute working paper - Existential risk pessimism and the time of perils Ethics article - Mistakes in the Moral Mathematics of Existential Risk Philosophy & Public Affairs article - High Risk, Low Reward: A Challenge to the Astronomical Value of Existential Risk Mitigation Toby Ord Book - The Precipice Rethink Priorities blogpost - Charting the precipice AI Futures Project blogpost - AI 2027 Trump's Higher Education Threat Compact Wikipedia article - Compact for Academic Excellence in Higher Education Pen America explainer - What is Trump’s Compact for Higher Education? And More Frequently Asked Questions Statement by the Vanderbilt AAUP Executive Committee on the “Compact for Academic Excellence in Higher Education” The Vanderbilt Hustler article - BREAKING: Chancellor Daniel Diermeier fails to reject higher education compact, reaffirms Vanderbilt’s values and openness to discussion The Vanderbilt Hustler article - Students and faculty organize rally outside Kirkland Hall against Trump administration’s higher education compact Free Speech Center article - Compact for Academic Excellence More of David's Work Global Priorities Institute working paper - What power-seeking theorems do not show Book - Essays on Longtermism Vibe Shift Blood in the Machine article - GPT-5 Is a Joke. Will It Matter? Futurism article - Evidence Grows That GPT-5 Is a Bit of a Dud Gary Marcus substack - GPT-5: Overdue, overhyped and underwhelming. And that’s not the worst of it. Pew Research report - How the U.S. Public and AI Experts View Artificial Intelligence N...
More from Into AI Safety
Etienne Brisson and Allan Brooks, founder and program director of The Human Line Project, sat down with me to talk about AI-induced psychological harm, starting with their own experiences. Allan spent a month spiraling with ChatGPT, ending with him warning national security agencies about an encryption break that never happened. Etienne's uncle became convinced his chatbot was sentient and was hospitalized still messaging it from the back of a police car.
Dr. Evan Miyazono joins Jacob to talk about his path from quantum networking and Protocol Labs to founding Atlas Computing. They discuss why Atlas moved away from formal methods research and toward what Evan calls “field strategy”: identifying important problems, figuring out what interventions could actually address them, and finding the people best positioned to make them happen. The conversation also covers the resilience gap map, coordination failures, AI safety, "value alignment," and the question of how to decide what work is worth doing when there’s no obvious organization responsible for it. Check out the video version of this podcast on the Kairos.fm YouTube channel, or the extended version on Patreon! Chapters
What if the safest AI models weren't built by adding guardrails after training, but by shaping what gets learned in the first place? Ethan Roland, senior alignment researcher at AE Studio and first author on an ICML 2026 spotlight paper, joins Jacob to talk about gradient routing, a technique that routes dangerous capabilities into isolated parts of a model's architecture where they can be locked or removed entirely. They get into the absorption effect, KYC-style access control frameworks, and what it would actually take for frontier labs to adopt this kind of work before it's needed rather than after. Chapters
Today's episode does double duty as an interview and an announcement. Joe Williams, host of the new Kairos.fm show "Beyond the Paycheck: Reclaiming the Case for UBI in the Age of AI," joins Jacob to talk about his background as a freelance translator and how AI quietly dismantled his livelihood in 2025. From there the conversation expands into whether this moment is really different from past waves of automation, who exactly makes up the tech billionaire class, and why you should probably raise an eyebrow when someone like Elon Musk says he supports UBI. Go check out Reclaiming UBI! Chapters
Zainab Majid, co-founder of Asymmetric Security, joins Jacob for a conversation on the intersection between AI Safety and cybersecurity, as well as the future of digital forensics. Drawing from years of incident response work, she explains how cyber attacks actually unfold, why AI is changing both offense and defense, and how her team is building AI-native tools to investigate breaches faster and more effectively. Other topics explored in this episode include trust in the AI/cybersecurity industries, the realities behind cybersecurity hype, and the challenge of keeping humans meaningfully involved as these systems become more capable. Zainab also gives practical, actionable advice on how you can protect yourself. If you're interested in over 30 minutes of additional content, head on over to the Kairos.fm Patreon where you can become a subscriber for just $2 per month, which helps make this whole podcasting thing a bit more sustainable. Chapters