Grokking, Generalization Collapse, and the Dynamics of Training Deep Neural Networks with Charles Martin - #734
The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)Show Notes
Today, we're joined by Charles Martin, founder of Calculation Consulting, to discuss Weight Watcher, an open-source tool for analyzing and improving Deep Neural Networks (DNNs) based on principles from theoretical physics. We explore the foundations of the Heavy-Tailed Self-Regularization (HTSR) theory that underpins it, which combines random matrix theory and renormalization group ideas to uncover deep insights about model training dynamics. Charles walks us through WeightWatcher’s ability to detect three distinct learning phases—underfitting, grokking, and generalization collapse—and how its signature “layer quality” metric reveals whether individual layers are underfit, overfit, or optimally tuned. Additionally, we dig into the complexities involved in fine-tuning models, the surprising correlation between model optimality and hallucination, the often-underestimated challenges of search relevance, and their implications for RAG. Finally, Charles shares his insights into real-world applications of generative AI and his lessons learned from working in the field.
The complete show notes for this episode can be found at https://twimlai.com/go/734.
More from The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)
AI agents need the institutional knowledge experienced employees rely on: why a system works a certain way, which policies apply, and what never made it into the documentation. Capturing that knowledge is hard. Keeping it useful as people and agents add more may be harder.
In this episode, Zoe Weil, co-founder and CEO of Sequen, joins us to discuss how ranking systems can use live behavioral signals to optimize for purchases rather than clicks or search relevance alone. Zoe explains why approaches developed at data-rich social platforms don’t necessarily translate to retailers and how Sequen uses scroll speed, hover duration, and other in-session interactions to infer shopper intent without relying on user identity.
In this episode, Diogo Almeida, co-founder and CEO of TypeSafe, joins us to discuss Jev, TypeSafe’s recently released model for bringing fast, reliable intelligence directly into software. We explore the idea of “machine-native intelligence” and why Diogo believes models optimized for generating text are poorly suited to many of the decisions required for real-world automation. He explains how Jev differs from traditional classifiers and LLM-based approaches, the role of reinforcement learning from calibrated decisions (RLCD), and why calibration and reliability are central to making AI useful as a software primitive. We also discuss the relationship between models and code, why Diogo believes AI systems should become more engineered rather than relying on a single model to do everything, and how Jev-like models could reshape agents, tool use, and the architecture of AI-powered software.
AI systems have gone from struggling with grade-school math to helping solve research problems that have resisted mathematicians for decades, including Navier-Stokes.
Voice AI has gotten remarkably good, but natural conversation remains a high bar. Small delays, awkward interruptions, or the wrong tone can quickly break the illusion—and adding vision and visual presence only raises the stakes.