Win-Win with Liv Boeree

#42 - Anthony Aguirre and Malo Bourgon - Why Superintelligent AI Should Not Be Built

Win-Win with Liv Boeree

Show Notes

Humanity is in a race to build superintelligent AI gods. What happens if we create beings smarter than us—and can no longer steer them? In this special Win-Win IRL episode, Liv Boeree sits down with Anthony Aguirre (CEO of Future of Life Institute) and Malo Bourgon (CEO of MIRI) for a no-holds-barred conversation on finding alignment, avoiding power concentration and surveillance states, and the geopolitical implications of the race to AGI. Questions include: Why can’t the market be trusted to self-regulate AI? Why does power concentration in AI pose such a massive threat to democracy? And what happens when even well-meaning developers lose control of the systems they’ve built? We also explore win-win solutions—from compute governance and open-source safety measures, to collaborative international treaties and new models for responsible innovation. Also keep an eye out So buckle up for a provocative episode that challenges assumptions and asks what it will really take to keep the future human.

Chapters: 0:00 - Intro 4:08 - Different Categories of Risks 10:51 - Why Can't the Market Manage AI Risks Sufficiently? 15:33 - The Dangers of Power Concentration 22:52 - What Solution Space Exists? 32:55 - Should Compute be Regulated? 38:38 - Taylor Capito - Are we at an Evolutionary Nexus 42:07 - JD Ross - Decentralization vs Centralization 46:46 - Isabelle Boemeke - Nuclear Power & AI 51:06 - Peter Wang - Wrong Stories Attract the Wrong People 57:39 - Aurora Quinn-Elmore & Eliezer Yudkowsky - Coherent Extrapolated Volition 1:05:59 - Can Win-Win be embedded into AI?

Links: ♾️ Keep the Future Human Paper: https://keepthefuturehuman.ai ♾️ Future of Life Institute: https://futureoflife.org ♾️ MIRI: https://intelligence.org ♾️ Superintelligence Book by Nick Bostrom - https://www.amazon.com/Superintelligence-Dangers-Strategies-Nick-Bostrom/dp/0198739834

Credits: ♾️ Hosted by Liv Boeree ♾️ Produced by Luca de Vico

The Win-Win Podcast:

Poker champion Liv Boeree takes to the interview chair to tease apart the complexities of one of the most fundamental parts of human nature: competition. Liv is joined by top philosophers, gamers, artists, technologists, CEOs, scientists, athletes and more to understand how competition manifests in their world, and how to change seemingly win-lose games into Win-Wins.

Podcast links: ♾️ Youtube: https://www.youtube.com/playlist?list=PLWgq0OZMtwtOIyMsVM_vksqdfWcM-b68S ♾️ Spotify: https://open.spotify.com/show/03bGVUaFZmJUmEvSHNDPdI?si=64379cc23696454f ♾️ Apple Podcasts: https://podcasts.apple.com/us/podcast/win-win-with-liv-boeree/id1724791350 ♾️ Pocketcast: https://play.pocketcasts.com/podcasts/7f708340-d17c-013b-f46e-0acc26574db2 #AGI #podcast #aialignment

More from Win-Win with Liv Boeree

Aug 30, 2026

Can indigenous wisdom help us navigate these modern times? Fresh perspectives are in short supply these days, and it's not often you get to interview someone who has lived most of their lives entirely outside of modern society. So when I got the opportunity to speak to Txana Tuwe - elder of the Huni Kuin tribe, from deep in the Brazilian Amazon rainforest, I jumped at the chance. We discuss his life in the Amazon, his thoughts on modern technology like AI, competition, shamanic warfare and so much more. Huge thank you to him and his translator Leah for taking the time for this recording. If you'd like to learn more, and support Tuwe's Mission of protecting his corner of the Amazon rainforest, visit here: https://www.kayaweiearth.org/Chapters00:00 Intro01:35 - His Mission in America03:30 - Ancestral Wisdom and Ayahuasca07:46 - Alternative Paths to Wisdom10:48 - Ancient Technologies15:19 - Talking to Plants, Animals & Spirits17:11 - Impact of Technology in the Amazon18:56 - Deforestation Pressures24:37 - Tribal Warfare28:14 - What can be done by us?29:56 - Tuwe's Favorite Games 31:56 - Is he competitive? Shamanic Wars? Links:♾️ Support Tuwe's Mission: https://www.kayaweiearth.org/♾️ Earth Species Project: https://earthspecies.org/♾️ Huni Kuin Tribe - https://en.wikipedia.org/wiki/Huni_Kuin Credits:♾️ Hosted by Liv Boeree♾️ Edited by Luca de Vico & Liv BoereeThe Win-Win Podcast:Poker champion Liv Boeree takes to the interview chair to tease apart one of the most fundamental forces shaping human behavior: competition.Liv speaks with philosophers, technologists, scientists, founders, and public thinkers to understand how seemingly win-lose systems can be redesigned into win-wins.Podcast links:♾️ Website: https://www.winwinpodcast.com/♾️ YouTube: https://www.youtube.com/@winwinpodcast♾️ Spotify: https://open.spotify.com/show/03bGVUaFZmJUmEvSHNDPdI♾️ Apple Podcasts: https://podcasts.apple.com/gb/podcast/win-win-with-liv-boeree/id1724791350#winwinpodcast #ayahuasca #indigenous

Apr 1, 2026

Advanced AI is coming. Is it time for the government to step in and regulate? Or is it better in the hands of the market and tech CEOs? Or is there a more nuanced way to think about this topic entirely? Welcome to the ANTI-DEBATE. It's debating, but with a positive-sum twist! My guests today are two of the most qualified people on either side this topic: former Trump AI advisor Dean Ball, and OpenAI researcher-turned-whistleblower Daniel Kokotajlo. Can they find a way to bridge their differences and find a synthesis that satisfies all their concerns? What are their thoughts on the Anthropic vs Department of War & Pete Hegseth feud? Can they align on alignment?

Mar 11, 2026

Free speech is often treated as a political issue. But what if it’s the foundational technology that makes peaceful civilization possible in the first place? I sat down with Greg Lukianoff — free speech advocate, first amendment attorney, and longtime leader of FIRE.org — to explore why freedom of expression is humanity’s ultimate win-win innovation.Greg strongly believes free speech is at risk across the globe. It's under threat from government censorship, and from the attitudes of citizens themselves, who have become increasingly intolerant of opposing views. These pressures seem to only be increasing as we enter the AI era, which allows censorship, surveillance and other oppression tools at scale. At the same time, it also offers the democratization of potentially hazardous information to violent actors, only increasing the pressures on governing authorities to clamp down even harder on speech. So how do we navigate this tension? That's what this conversation seeks to answer.Huge thanks to @TheFIREorg for collaborating with me on this episode (they were the ones who organised this live event)! Go support their important work at the link the below. Chapters: 00:00 — Intro 01:42 — Why is Free Speech a Win-Win? 05:10 — Is Free Speech At Risk in the USA? 08:41 — Charlie Kirk & The Case of Larry Bushart 11:40 — How Is Technology Changing Free Speech? 18:49 — ACLU & Institutional Capture 21:57 — AI & The First Amendment 29:40 — How To Calm The Arms Race? 33:50 — The Hidden Cost of LLM Slop 38:22 — How To Design Better Social Media Algorithms? 41:07 — UK & European Hate Speech Laws & Censorship 47:18 — J.K Rowling & Free Speech Influencers 52:20 — Can Free Speech Survive Gen Z? 59:04 — How To Talk To Skeptics Links: ♾️ Greg Lukianoff: https://www.thefire.org/people/greg-lukianoff ♾️ FIRE (Foundation for Individual Rights and Expression): https://www.thefire.org ♾️ Greg’s TED Talk on Free Speech: https://www.youtube.com/watch?v=noWh8SSeRCo ♾️ The Larry Bushart Case: https://www.fire.org/cases/larry-bushart-v-perry-county ♾️ The Coddling of the American Mind: https://www.thecoddling.com ♾️ Karl Popper — Conjectures and Refutations: https://archive.org/details/karl-popper-conjectures-and-refutations-the-growth-of-scientific-knowledge Credits: ♾️ Hosted by Liv Boeree ♾️ Produced by Luca de Vico ♾️ Event Production: the team at Fire.org The Win-Win Podcast: Poker champion Liv Boeree takes to the interview chair to tease apart one of the most fundamental forces shaping human behavior: competition.Liv speaks with philosophers, technologists, scientists, founders, and public thinkers to understand how seemingly win-lose systems can be redesigned into win-wins. Podcast links: ♾️ Website: https://www.winwinpodcast.com/ ♾️ Youtube: https://www.youtube.com/playlist?list=PLWgq0OZMtwtOIyMsVM_vksqdfWcM-b68S ♾️ Spotify: https://open.spotify.com/show/03bGVUaFZmJUmEvSHNDPdI?si=64379cc23696454f ♾️ Apple Podcasts: https://podcasts.apple.com/us/podcast/win-win-with-liv-boeree/id1724791350 ♾️ Pocketcast: https://play.pocketcasts.com/podcasts/7f708340-d17c-013b-f46e-0acc26574db2 #winwinpodcast #freespeech #AI

Feb 6, 2026

Which LLM is best at POKER? What about the social deception game Werewolf? This week I'm collaborating with ⁠ @googledeepmind ⁠ and ⁠ @kaggle ⁠ to explain their new “AI Game Arena” project, designed to test all the top LLMs at various games. The Game Arena is a massive research project to create new AI benchmarks, and understand what really makes these general-purpose AIs tick. It also helps us evaluate how far along the path to true AGI we are, NBD.So who better to speak to than Deepmind research engineer John Schultz, one of the main brains behind the project. We discuss why some LLMs are so much better than others, their internal understanding of the games (or lack of it!) and why games are a useful way to evaluate both capabilities and safety of frontier models.And friends - do please note that for the first time ever on this channel, this post is part of a paid partnership (woohoo I made it!). So while everything I said in this interview, is absolutely my own views, there are also direct financial incentives behind this particular interview.

Jan 27, 2026

Superintelligence is predicted to arrive within a decade. If that's true, then what comes AFTER? And how can we prepare for it?Which is why I sat down with one of the few people actively thinking about this: philosopher Will MacAskill. Will heads up Forethought, a research nonprofit focused on how to navigate the transition to a world with superintelligent AI systems. He’s also a former Oxford University professor, and a co-founder of the effective altruism movement. In this special Win-Win Live conversation, we discuss why “utopias” often fail, the growing risks of authoritarian lock-in, whether moral progress survives in a post-labor world,, and how the transition to a post-superintelligence world may be more crucial than the development AGI itself. Also, lots of chat about the sun, space and the high seas.

See all episodes →