Show Notes
Sean McGregor and I discuss about why evaluating AI systems has become so difficult; we cover everything from the breakdown of benchmarking, how incentives shape safety work, and what approaches like BenchRisk (his recent paper at NeurIPS) and AI auditing aim to fix as systems move into the real world. We also talk about his history and journey in AI safety, including his PhD on ML for public policy, how he started the AI Incident Database, and what he's working on now: AVERI, a non-profit for frontier model auditing. Chapters
(00:00) - Intro
(02:36) - What's broken about benchmarking
(03:41) - Sean’s wild PhD
(14:28) - The phantom internship
(19:25) - Sean's journey
(22:25) - Market-vs-regulatory modes and AIID
(32:13) - Drunk on AI progress
(38:34) - BenchRisk
(43:20) - Moral hazards and Master Hand
(50:34) - Liability, Section 230, and open source
(59:20) - AVERI
(01:11:30) - Closing thoughts & outro
LinksSean McGregor's website AVERI website BenchRisk BenchRisk website NeurIPS paper - Risk Management for Mitigating Benchmark Failure Modes: BenchRisk NeurIPS paper - AI and the Everything in the Whole Wide World Benchmark AIID AI Incident Database website IAAI paper - Preventing Repeated Real World AI Failures by Cataloging Incidents: The AI Incident Database Preprint - Lessons for Editors of AI Incidents from the AI Incident Database AIAAIC website (another incident tracker) Hot AI Summer CACM article - A Few Useful Things to Know About Machine Learning CACM article - How the AI Boom Went Bust Undergraduate Thesis - Analyzing the Prospect of an Approaching AI Winter Tech Genies article - AI History: The First Summer and Winter of AI CACM article - There Was No ‘First AI Winter’ Measuring Generalization Neural Computation article - The Lack of A Priori Distinctions Between Learning Algorithms ICLR paper - Understanding deep learning requires rethinking generalization ICML paper - Model-agnostic Measure of Generalization Difficulty Radiology Artificial Intelligence article - Generalizability of Machine Learning Models: Quantitative Evaluation of Three Methodological Pitfalls Preprint - Quantifying Generalization Complexity for Large Language Models Insurers Exclude AI Financial Times article - Insurers retreat from AI cover as risk of multibillion-dollar claims mount Tom's Hardware article - Major insurers move to avoid liability for AI lawsuits as multi-billion dollar risks emerge — Recent public incidents have lead to costly repercussions Insurance Newsnet article - Insurers Scale Back AI Coverage Amid Fears of Billion-Dollar Claims Insurance Business article - Insurance’s gen AI reckoning has come Section 230 Section 230 overview Legal sidebar - Section 230 Immunity and Generative Artificial Intelligence Bad Internet Bills website TechDirt article - Section 230 Faces Repeal. Support The Coverage That’s Been Getting It Right All Along. Privacy Guides video - Dissecting Bad Internet Bills with Taylor Lorenz: KOSA, SCREEN Act, Section 230 Journal of Technology in Behavioral Health article - Social Media and Mental Health: Benefits, Risks, and Opportunities for Research and Practice Time article - Lawmakers Unveil New Bills to Curb Big Tech’s Power and Profit House Hearing transcript - Legislative Solutions to Protect Children and Teens Online Relevant Kairos.fm Episodes Into AI Safety episode - Growing BlueDot's Impact w/ Li-Lian Ang muckrAIkers episode - NeurIPS 2024 Wrapped 🌯 Other Links Encyclopedia of Life website IBM Watson AI XPRIZE website ML Commons website Wikipedia article
More from Into AI Safety
Etienne Brisson and Allan Brooks, founder and program director of The Human Line Project, sat down with me to talk about AI-induced psychological harm, starting with their own experiences. Allan spent a month spiraling with ChatGPT, ending with him warning national security agencies about an encryption break that never happened. Etienne's uncle became convinced his chatbot was sentient and was hospitalized still messaging it from the back of a police car.
Dr. Evan Miyazono joins Jacob to talk about his path from quantum networking and Protocol Labs to founding Atlas Computing. They discuss why Atlas moved away from formal methods research and toward what Evan calls “field strategy”: identifying important problems, figuring out what interventions could actually address them, and finding the people best positioned to make them happen. The conversation also covers the resilience gap map, coordination failures, AI safety, "value alignment," and the question of how to decide what work is worth doing when there’s no obvious organization responsible for it. Check out the video version of this podcast on the Kairos.fm YouTube channel, or the extended version on Patreon! Chapters
What if the safest AI models weren't built by adding guardrails after training, but by shaping what gets learned in the first place? Ethan Roland, senior alignment researcher at AE Studio and first author on an ICML 2026 spotlight paper, joins Jacob to talk about gradient routing, a technique that routes dangerous capabilities into isolated parts of a model's architecture where they can be locked or removed entirely. They get into the absorption effect, KYC-style access control frameworks, and what it would actually take for frontier labs to adopt this kind of work before it's needed rather than after. Chapters
Today's episode does double duty as an interview and an announcement. Joe Williams, host of the new Kairos.fm show "Beyond the Paycheck: Reclaiming the Case for UBI in the Age of AI," joins Jacob to talk about his background as a freelance translator and how AI quietly dismantled his livelihood in 2025. From there the conversation expands into whether this moment is really different from past waves of automation, who exactly makes up the tech billionaire class, and why you should probably raise an eyebrow when someone like Elon Musk says he supports UBI. Go check out Reclaiming UBI! Chapters
Zainab Majid, co-founder of Asymmetric Security, joins Jacob for a conversation on the intersection between AI Safety and cybersecurity, as well as the future of digital forensics. Drawing from years of incident response work, she explains how cyber attacks actually unfold, why AI is changing both offense and defense, and how her team is building AI-native tools to investigate breaches faster and more effectively. Other topics explored in this episode include trust in the AI/cybersecurity industries, the realities behind cybersecurity hype, and the challenge of keeping humans meaningfully involved as these systems become more capable. Zainab also gives practical, actionable advice on how you can protect yourself. If you're interested in over 30 minutes of additional content, head on over to the Kairos.fm Patreon where you can become a subscriber for just $2 per month, which helps make this whole podcasting thing a bit more sustainable. Chapters