Show Notes
Today we explore whether mechanistic interpretability could hold the key to building leaner, more transparent—and perhaps even smarter—large language models. From knowledge distillation and pruning to low-rank adaptation, we examine cutting-edge strategies to make AI models both smaller and more explainable. Join Arshavir as he breaks down the surprising challenges of making models efficient without sacrificing understanding.
More from Inside the Black Box: Cracking AI and Deep Learning
After thirty-odd podcast interviews about AI, Arshavir Blackwell found the same four questions coming up with very different hosts: Will AI take my job or kill us all? Why does AI writing all sound the same? What should I teach my children? Is it actually thinking? He talks about stepping outside the AI bubble, and about the two questions no host asked on their own: where your data goes, and what AI really costs.
A follow-up in the past-tense series testing whether U-shaped learning, model size, or measurement explains irregular verb behavior—and finding the language matters too.
This episode challenges the old idea of a single language box in the brain, using aphasia studies and hospital control groups to show how language processing is more distributed than once thought. It also explores how different languages rely on word order or verb marking in different ways, and why those differences matter when the brain is under stress.
Thirty years ago, the authors of Rethinking Innateness argued that grammar could come out of a learner with no grammar built in. Nobody could check it. Now there's an instrument. A reading of the Inside the Black Box essay on what Elman, Bates, and Karmiloff-Smith would make of transformer language models — attention as a lookup, structure nobody installed, and the part that isn't there.
Can a model learn past tense the way children seem to, if you remove the tokenizer from the equation? This final installment tests raw letters and raw sound, probes whether a U-shaped pattern can be forced, and shows why the answer points to distributed, frequency-driven learning rather than a discrete rule.