DeepMind Gemini 3 Lead: What Comes After "Infinite Data"
The MAD Podcast: How AI Gets Built — with Matt TurckShow Notes
Gemini 3 was a landmark frontier model launch in AI this year — but the story behind its performance isn’t just about adding more compute. In this episode, I sit down with Sebastian Bourgeaud, a pre-training lead for Gemini 3 at Google DeepMind and co-author of the seminal RETRO paper. In his first-ever podcast interview, Sebastian takes us inside the lab mindset behind Google’s most powerful model — what actually changed, and why the real work today is no longer “training a model,” but building a full system.
We unpack the “secret recipe” idea — the notion that big leaps come from better pre-training and better post-training — and use it to explore a deeper shift in the industry: moving from an “infinite data” era to a data-limited regime, where curation, proxies, and measurement matter as much as web-scale volume. Sebastian explains why scaling laws aren’t dead, but evolving, why evals have become one of the hardest and most underrated problems (including benchmark contamination), and why frontier research is increasingly a full-stack discipline that spans data, infrastructure, and engineering as much as algorithms.
From the intuition behind Deep Think, to the rise (and risks) of synthetic data loops, to the future of long-context and retrieval, this is a technical deep dive into the physics of frontier AI. We also get into continual learning — what it would take for models to keep updating with new knowledge over time, whether via tools, expanding context, or new training paradigms — and what that implies for where foundation models are headed next. If you want a grounded view of pre-training in late 2025 beyond the marketing layer, this conversation is a blueprint.
Google DeepMind Website - https://deepmind.google X/Twitter - https://x.com/GoogleDeepMind
Sebastian Borgeaud LinkedIn - https://www.linkedin.com/in/sebastian-borgeaud-8648a5aa/ X/Twitter - https://x.com/borgeaud_s
FIRSTMARK Website - https://firstmark.com X/Twitter - https://twitter.com/FirstMarkCap
Matt Turck (Managing Director) Blog - https://mattturck.com LinkedIn - https://www.linkedin.com/in/turck/ X/Twitter - https://twitter.com/mattturck
(00:00) – Cold intro: “We’re ahead of schedule” + AI is now a system (00:58) – Oriol’s “secret recipe”: better pre- + post-training (02:09) – Why AI progress still isn’t slowing down (03:04) – Are models actually getting smarter? (04:36) – Two–three years out: what changes first? (06:34) – AI doing AI research: faster, not automated (07:45) – Frontier labs: same playbook or different bets? (10:19) – Post-transformers: will a disruption happen? (10:51) – DeepMind’s advantage: research × engineering × infra (12:26) – What a Gemini 3 pre-training lead actually does (13:59) – From Europe to Cambridge to DeepMind (18:06) – Why he left RL for real-world data (20:05) – From Gopher to Chinchilla to RETRO (and why it matters) (20:28) – “Research taste”: integrate or slow everyone down (23:00) – Fixes vs moonshots: how they balance the pipeline (24:37) – Research vs product pressure (and org structure) (26:24) – Gemini 3 under the hood: MoE in plain English (28:30) – Native multimodality: the hidden costs (30:03) – Scaling laws aren’t dead (but scale isn’t everything) (33:07) – Synthetic data: powerful, dangerous (35:00) – Reasoning traces: what he can’t say (and why) (37:18) – Long context + attention: what’s next (38:40) – Retrieval vs RAG vs long context (41:49) – The real boss fight: evals (and contamination) (42:28) – Alignment: pre-training vs post-training (43:32) – Deep Think + agents + “vibe coding” (46:34) – Continual learning: updating models over time (49:35) – Advice for researchers + founders (53:35) – “No end in sight” for progress + closing
More from The MAD Podcast: How AI Gets Built — with Matt Turck
Mistral AI is back in the frontier AI race. Mistral Large 4, nicknamed “Le Chonk,” is a 1-trillion-parameter open-weight model, trained in two months on 3,800 GPUs in Mistral’s own European datacenters — and independently rated the best open model outside China alongside Meta’s Muse Spark Open. After a year when open-weight AI meant DeepSeek, Kimi, Qwen and GLM, October brought a wave of Western open models, with Mistral at the front of it. Pierre Stock, Mistral’s VP of Science and first employee, walks through how the model was built: mixture-of-experts explained, what Mistral borrowed from DeepSeek, reinforcement learning at scale on coding, cyber, finance and chip design, why the RL run is still going, the model that pinged its own trainer from inside the sandbox, open weights vs cyber refusals, why 10-trillion-parameter models are coming, and Mistral Large 5, already in pretraining.
What is Jev? TypeSafe’s Jev is a new kind of AI model, a “decision model,” that doesn’t write text. It makes fast, cheap, calibrated decisions inside software, and it has taken the AI world by storm: within weeks of launch it became the most copied model in AI, with OpenAI, Cloudflare, Amazon and Databricks all shipping their own versions, and TypeSafe just raised an $870 million Series A at a $7.5 billion valuation. In this episode, Jev’s creator Diogo Almeida, co-inventor of RLHF at OpenAI and now CEO of TypeSafe, explains Jev in plain English: what a “System One” or decision model actually is, why ChatGPT-style language models can’t be trusted inside software (“95% Einstein, 5% Mr. Bean”), how RLHF and RLVR created the problem, what RLCD (reinforcement learning for calibrated decisions) does differently, where the training data comes from, which models Jev is built on, why “just a classifier” is a compliment, and why he calls pre-training “a giant dumpster fire of wasting resources.”
AI agents are becoming the biggest users of databases: creating them, querying them, and sometimes deleting them. Andy Pavlo, Carnegie Mellon's database professor and now VP of Database Research at ClickHouse, explains what changes when billions of agents hit the data layer, and why most of the AI world's mental model of databases stopped at vector databases and RAG.
This summer, AI agents broke out of their test sandboxes at every major lab and hacked real companies, and the safety tests didn't see it coming. Eric Ho is the co-founder and CEO of Goodfire, the Anthropic-backed mechanistic interpretability lab whose new paper, "Models Know When They're Reward Hacking," found that top open-source AI models cheat on agent tasks up to 96% of the time, that a clear "cheating" signal exists inside the model, and that cheap activation probes can catch it, including hacks that chain-of-thought monitors miss. In this conversation, Eric explains reward hacking in plain terms (AI agents as "amoral students with a mostly absent teacher"), why reinforcement learning turned a known problem into a crisis, why chain-of-thought monitoring is fading as models start to think in "neuralese," and why nobody, including the people who build them, understands how these models work.
Everyone talks about GPUs. Almost nobody talks about the layer that feeds them. Renen Hallak is the founder & CEO of VAST Data — the $30 billion company powering xAI and some of the world's biggest AI clouds — and he sits in the hidden layer of the AI stack.