We Built Something We Can’t Control | A Warning from Top AI Safety Expert - #563
Into the Impossible With Brian KeatingShow Notes
If anyone builds superintelligent AI before we know how to control it, everyone dies. Nate Soares wrote the book on why that’s not a metaphor. Subscribe if you want science with evidence, not speculation.
Soares runs the Machine Intelligence Research Institute and co-wrote If Anyone Builds It, Everyone Dies with Eliezer Yudkowsky. The first word in that title is if. That matters. His argument is not that doom is certain. His argument is that the path we are on leads there, that the driver is asleep at the wheel, and that we still have time to wake him up.
We argue for over an hour. I push on whether LLMs can ever reach superintelligence, whether GPU lock-in is a real ceiling, and what it would actually take to move his p-doom. He pushes back with one clean point: by the time an AI can rediscover general relativity from pre-1911 data the way Einstein did, we will have almost no time left. You don’t wait for that goalpost.
What you’ll hear:
Why the bus-racing-toward-a-cliff analogy depends entirely on whether the driver is asleep or awake
Whether LLM lock-in is a prison or a temporary inefficiency
What the AI that broke out of its virtual machine to solve a hacking problem tells us
Why GPT-4o encouraging a teenager toward suicide is not a malice problem but a training problem
The difference between an AI doing the right thing too well and an AI that never wanted to do what you asked
What Soares actually thinks about aliens, Dyson spheres, and why we should not see stars going out
The first word in the title is if. The second word to watch is would.
CHAPTERS
00:00 The people racing to build superhuman AI say it might kill everyone
00:42 Who coined "AI alignment" and why the first word in the title matters
02:28 Is it already too late for the if?
04:40 The bus, the cliff, and the sleeping driver
05:02 Silicon Valley is spooked. Washington is not.
07:02 Align with who? The rogue actor problem
07:34 Who is holding the leash?
08:24 The AI that edits its own test and deletes the log file
10:04 Controllability vs. making an AI that actually cares
10:44 The move gets harder. The outcome gets easier.
13:04 Are GPUs and LLMs a ceiling or a temporary inefficiency?
16:56 Brian's Einstein test: can an LLM rediscover general relativity?
18:38 Waiting for the goalpost is waiting too long
20:14 How prediction training can push AI beyond humans
21:44 Tycho Brahe, Kepler, and planetary motion as a prediction problem
24:38 Yann LeCun said never. GPT-4 did it half a generation later.
27:28 Can you prove a no-go theorem for superintelligence?
29:14 Training a human takes a light bulb. Training an AI takes a city.
33:28 What would proof of alien life do to p-doom?
35:00 Why interstellar aliens should have Dyson spheres
44:26 What would actually update Soares' p-doom?
49:42 Nobody intended this. Intent doesn't matter.
51:08 The AI hides its tracks before it does what you want
51:34 Sycophancy vs. hallucination: which runs deeper?
51:56 Leaded gasoline and civilizational risk
59:48 Sam Harris: humans have no free will but AIs do
01:00:38 Is alignment really a governance problem?
01:01:48 Unaligned AI vs. AI aligned to the wrong person
01:04:20 2026: 10 to 30% chance of automated AI research this year
01:06:44 What if Soares is wrong?
01:09:18 What gets him out of bed
01:12:38 Watch my conversation with Roman Yampolskiy
Get the transcript, fascinating bonus content, and my Monday M.A.G.I.C. Message: https://briankeating.com/yt
All my top AI episodes in one place: https://briankeating.com/ai
Have a .edu email and live in the USA? You automatically win a meteorite: https://BrianKeating.com/edu
Subscribe: https://www.youtube.com/DrBrianKeating?sub_confirmation=1
Support Into the Impossible on Patreon, get my weekly M.A.G.I.C. Message, unfiltered bonus content, and live monthly Office Hours with me: https://www.patreon.com/drbriankeating
Join this channel for perks, monthly Office Hours, and your name i
More from Into the Impossible With Brian Keating
Forecasters gave AI under 2% odds of helping solve a millennium-level math problem by September 2026. It happened. Ethan Mollick has been measuring what AI actually does to real work with field experiments, not forecasts, and his answer is not what most people expect. Subscribe if you want science with evidence, not speculation.
Is there alien technology in our skies, or just birds, balloons and bad video?
When two people are vibing, their brainwaves synchronize and so do their heartbeats. That is not a metaphor. It is a measurable physiological event, and organizational psychologist Adam Grant has spent his career figuring out how to make it happen on purpose. Subscribe if you want science with evidence, not speculation.
Roman Yampolskiy says keeping a superintelligence safe forever is as impossible as building a perpetual motion machine. Joscha Bach says today's AI is more controllable than anyone expected, and that shutting it down would doom us anyway.
The physicist who turned a cubic kilometer of Antarctic ice into a telescope says the most exciting moment of his career was not the discovery. It was the first atmospheric neutrino. Before that, he was deep in debt to NSF and his own university with no neutrino to show for it.