Show Notes
What is inside a model like Claude, from the moment you type a question to the answer it gives back? This episode walks through the pieces that make up that experience: tokens and parameters, the Transformer architecture and its attention mechanism, post-training, and the software layer wrapped around the model, alongside Anthropic's own interpretability research into what's happening inside the network.
Claude is built around the Transformer architecture, and the episode is upfront about a limit: Anthropic has publicly described its models as Transformer-based but doesn't publish every architectural detail of its current frontier systems, so the explanation covers the major known pieces without claiming to know the exact blueprint. It walks through how text becomes tokens, how billions of learned parameters encode patterns from training data, and how the Transformer's attention mechanism lets the model relate different tokens to each other across a context, illustrated with a worked example of Claude generating a response one token at a time.
A central section covers Anthropic's interpretability research: the discovery of millions of identifiable features inside Claude 3 Sonnet, representing concepts and combinations of concepts rather than words in separate boxes, and later work using attribution graphs to trace parts of Claude's internal computations. It also covers a 2026 finding of a small set of internal patterns tied to higher-order reasoning, sometimes called a model's "J-space" in Anthropic's own research, which Claude can reportedly describe and even deliberately modulate.
The episode also explains what happens beyond the trained network itself: post-training techniques that shape how the model behaves, Claude's Constitution as an explicit part of that training process, and the runtime layer around the model, including prompts, system instructions, conversation context, and tools, which is why two experiences that both say "Claude" can feel very different from each other. The takeaway is that the behavior you experience comes from the trained network plus the training that shaped it, the context you provide, and the software wrapped around it, not from the model in isolation.
Sources & References Anthropic: Model System Cards — https://www.anthropic.com/system-cards Anthropic: Claude's Constitution — https://www.anthropic.com/constitution Anthropic: Claude's New Constitution — https://www.anthropic.com/news/claude-new-constitution Anthropic: Effective Context Engineering for AI Agents — https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents Anthropic: Mapping the Mind of a Large Language Model — https://www.anthropic.com/research/mapping-mind-language-model Anthropic: Tracing the Thoughts of a Large Language Model — https://www.anthropic.com/news/tracing-thoughts-language-model Anthropic: Natural Language Autoencoders: Turning Claude's Thoughts into Text — https://www.anthropic.com/research/natural-language-autoencoders Anthropic: A Global Workspace in Language Models — https://www.anthropic.com/research/global-workspace Anthropic: Transparency Hub — https://www.anthropic.com/transparency
Voice narration is AI-generated.
More from Learn AI in Bits
A wave of cybersecurity disclosures has made this week feel like a turning point, with people calling it "Black October." This episode treats it as a cluster of high-impact incidents, not proof that one AI-driven campaign caused them all, and separates where AI was directly involved from where the failures were conventional.
Recursive self-improvement is the idea that an AI can improve the very process it uses to improve itself, and this episode separates what's been demonstrated so far from the dramatic predictions often attached to the term. It draws a clear line between self-correction (revising one answer), automated optimization (keeping the best of several tested variants), and true recursive improvement, where the system changes the process that generates future improvements, so the next round starts with a better improver.
This week's AI News Sunday Wrap-Up covers a week where the industry moved on several fronts at once: new flagship and open models, enterprise agent governance, a serious safety warning, legal pressure, and the physical infrastructure limits behind the AI race.
Meta's Muse story is moving fast, and this episode takes a closer look at Muse Spark, the multimodal reasoning model family behind Meta's push into AI agents, including yesterday's release of an open-weight version for developers.
When a large language model generates text one token at a time, even a fast GPU hits a sequential bottleneck. This episode explains speculative decoding, an inference technique that prepares several tokens at once and checks them together, and when the extra computation pays off.