Show Notes
Every AI builder knows the anxiety: you spend months engineering prompts, tuning pipelines, and chaining calls together — then a new model drops and half your work evaporates overnight. It turns out researchers have been wrestling with this exact dynamic for 30 years, and they keep arriving at the same uncomfortable answer. That answer is called the Bitter Lesson — and understanding it might be the most important thing you can do for whatever you're building right now. From Deep Blue to AlexNet to modern LLMs, scale keeps beating sophistication, and knowing which side of that line your work falls on makes all the difference.
Links
- Richard Sutton, "The Bitter Lesson"
- Alon Halevy, Peter Norvig, and Fernando Pereira, "The Unreasonable Effectiveness of Data"
- Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton, "ImageNet Classification with Deep Convolutional Neural Networks"
More from Linear Digressions
As AI systems grow more powerful, they grow harder to understand — and that tradeoff has never mattered more. Most interpretability tricks give us approximations from the outside, but mechanistic interpretability takes a different approach: cracking open the model itself to trace *why* it behaves the way it does. This episode digs into what makes mechanistic interpretability fundamentally different from other explainability methods, and why it might be the most promising path toward actually understanding what's happening inside modern LLMs.
Labeled data is expensive, slow, and painfully hard to come by — and when you're trying to evaluate whether an AI workflow is actually doing what you want, the problem gets even messier. How do you judge whether a chatbot response was helpful, honest, or hallucination-free at scale? This episode digs into using LLMs as judges: letting the models themselves evaluate the quality of AI outputs, from simple chat exchanges all the way to complex multi-step agent behavior.
Originally aired on the Harvard Data Science Review podcast.