Show Notes
The almost Dr. Igor Krawczuk joins me for what is the equivalent of 4 of my previous episodes. We get into all the classics: eugenics, capitalism, philosophical toads... Need I say more? If you're interested in connecting with Igor, head on over to his website, or check out placeholder for thesis (it isn't published yet). Because the full show notes have a whopping 115 additional links, I'll highlight some that I think are particularly worthwhile here: The best article you'll ever read on Open Source AI The best article you'll ever read on emergence in ML Kate Crawford's Atlas of AI (Wikipedia) On the Measure of Intelligence Thomas Piketty's Capital in the Twenty-First Century (Wikipedia) Yurii Nesterov's Introductory Lectures on Convex Optimization Chapters
(02:32) - Introducing Igor
(10:11) - Aside on EY, LW, EA, etc., a.k.a. lettersoup
(18:30) - Igor on AI alignment
(33:06) - "Open Source" in AI
(41:20) - The story of infinite riches and suffering
(59:11) - On AI threat models
(01:09:25) - Representation in AI
(01:15:00) - Hazard fishing
(01:18:52) - Intelligence and eugenics
(01:34:38) - Emergence
(01:48:19) - Considering externalities
(01:53:33) - The shape of an argument
(02:01:39) - More eugenics
(02:06:09) - I'm convinced, what now?
(02:18:03) - AIxBio (round ??)
(02:29:09) - On open release of models
(02:40:28) - Data and copyright
(02:44:09) - Scientific accessibility and bullshit
(02:53:04) - Igor's point of view
(02:57:20) - Outro
Links Links to all articles/papers which are mentioned throughout the episode can be found below, in order of their appearance. All references, including those only mentioned in the extended version of this episode, are included. Suspicious Machines Methodology, referred to as the "Rotterdam Lighthouse Report" in the episode LIONS Lab at EPFL The meme that Igor references On the Hardness of Learning Under Symmetries Course on the concept of equivariant deep learning Aside on EY/EA/etc.Sources on Eliezer YudkowskiScholarly Community Encyclopedia TIME100 AI Yudkowski's personal website EY Wikipedia A Very Literary Wiki -TIME article: Pausing AI Developments Isn’t Enough. We Need to Shut it All Down documenting EY's ruminations of bombing datacenters; this comes up later in the episode but is included here because it about EY.
LessWrongLW Wikipedia
MIRI Coverage on Nick Bostrom (being a racist)The Guardian article: ‘Eugenics on steroids’: the toxic and contested legacy of Oxford’s Future of Humanity Institute The Guardian article: Oxford shuts down institute run by Elon Musk-backed philosopher
Investigative piece on Émile Torres On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜 NY Times article: We Teach A.I. Systems Everything, Including Our Biases NY Times article: Google Researcher Says She Was Fired Over Paper Highlighting Bias in A.I. Timnit Gebru's Wikipedia The TESCREAL Bundle: Eugenics and the Promise of Utopia through Artificial General Intelligence Sources on the environmental impact of LLMsThe Environmental Impact of LLMs The Cost of Inference: Running the Models Energy and Policy Considerations for Deep Learning in NLP The Carbon Impact of AI vs Search Engines
Filling Gaps in Trustworthy Development of AI (Igor is an author on this one) A Computational Turn in Policy Process Studies: Coevolving Network Dynamics of Policy Change The Smoothed Possibility of Social Choice, an intro in social choice theory and how it overlaps with ML Relating to Dan HendrycksNatural Selection Favors AIs over Humans"One easy-to-digest source to highlight what he gets wrong [is] Social and Biopolitical Dimensions of Evolutionary Thinking" -Igor
Introduction to AI Safety, Ethics, and Society, recently published textbook "Source to the section [of this paper] that makes Dan one of my favs from that crowd." -Igor Twitter post referenced in the episode<...
More from Into AI Safety
Etienne Brisson and Allan Brooks, founder and program director of The Human Line Project, sat down with me to talk about AI-induced psychological harm, starting with their own experiences. Allan spent a month spiraling with ChatGPT, ending with him warning national security agencies about an encryption break that never happened. Etienne's uncle became convinced his chatbot was sentient and was hospitalized still messaging it from the back of a police car.
Dr. Evan Miyazono joins Jacob to talk about his path from quantum networking and Protocol Labs to founding Atlas Computing. They discuss why Atlas moved away from formal methods research and toward what Evan calls “field strategy”: identifying important problems, figuring out what interventions could actually address them, and finding the people best positioned to make them happen. The conversation also covers the resilience gap map, coordination failures, AI safety, "value alignment," and the question of how to decide what work is worth doing when there’s no obvious organization responsible for it. Check out the video version of this podcast on the Kairos.fm YouTube channel, or the extended version on Patreon! Chapters
What if the safest AI models weren't built by adding guardrails after training, but by shaping what gets learned in the first place? Ethan Roland, senior alignment researcher at AE Studio and first author on an ICML 2026 spotlight paper, joins Jacob to talk about gradient routing, a technique that routes dangerous capabilities into isolated parts of a model's architecture where they can be locked or removed entirely. They get into the absorption effect, KYC-style access control frameworks, and what it would actually take for frontier labs to adopt this kind of work before it's needed rather than after. Chapters
Today's episode does double duty as an interview and an announcement. Joe Williams, host of the new Kairos.fm show "Beyond the Paycheck: Reclaiming the Case for UBI in the Age of AI," joins Jacob to talk about his background as a freelance translator and how AI quietly dismantled his livelihood in 2025. From there the conversation expands into whether this moment is really different from past waves of automation, who exactly makes up the tech billionaire class, and why you should probably raise an eyebrow when someone like Elon Musk says he supports UBI. Go check out Reclaiming UBI! Chapters
Zainab Majid, co-founder of Asymmetric Security, joins Jacob for a conversation on the intersection between AI Safety and cybersecurity, as well as the future of digital forensics. Drawing from years of incident response work, she explains how cyber attacks actually unfold, why AI is changing both offense and defense, and how her team is building AI-native tools to investigate breaches faster and more effectively. Other topics explored in this episode include trust in the AI/cybersecurity industries, the realities behind cybersecurity hype, and the challenge of keeping humans meaningfully involved as these systems become more capable. Zainab also gives practical, actionable advice on how you can protect yourself. If you're interested in over 30 minutes of additional content, head on over to the Kairos.fm Patreon where you can become a subscriber for just $2 per month, which helps make this whole podcasting thing a bit more sustainable. Chapters