Hydragen: Shared-Prefix Attention for High-Throughput LLM Inference

This episode examines a production-scale characterization of GitHub Copilot's coding agent, built on sampled client-side telemetry from one week in June 2026 (13.5 million sessions and 760 million LLM calls, though some totals are inconsistent). It argues that agent traffic is session-shaped rather than stateless chat. About 87% of LLM calls are agent-initiated, the median call carries roughly 68,000 prompt tokens to produce about 247 output tokens, and the human appears only at turn boundaries. The discussion explains how prefix caching should give very high reuse inside a turn, and why the cache is lost after a pause, a model switch, or context compaction. It also separates "container idle" from "KV-cache idle", which suggests turn boundaries are the natural point to reclaim resources. The episode places the work alongside prior agentic KV-retention research and serverless idle-time studies. Listeners get a clear framework for why request-level scheduling and plain LRU eviction fall short for agent workloads, along with caveats about inferring server behavior from client data alone.

This episode examines SanSi, a small "typed decision" model that returns a probability for each caller-declared option (such as spam or not spam) in a single forward pass, with no generated text to parse. It explains how the model gets more computation by looping the same 24-layer stack up to eight times, with a readout after every loop, so one model can serve any budget from one loop to eight. The discussion covers how this "System 1.5" approach differs from one-pass classifiers and from generated chain-of-thought. It also covers the earlier work it builds on: Universal Transformers, adaptive computation time, and pre-trained looped models such as Ouro. On training, the model uses a frozen backbone with LoRA adapters (about 61 million trainable parameters) and a loss averaged across all loops. Supervising only the final loop dropped loop-1 accuracy to 35.6%, against 58.4% with per-loop training. The episode also raises caveats: the Kahneman-style labels are framing rather than evidence, the RL case study uses the judge's own probability as reward rather than a truly verifiable one, and it is unclear whether looping beats spending the same compute another way.

This episode explores the paper "Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient," which asks whether a mixture-of-experts model can beat a dense model on loss when both train with the same compute and are capped at the same total parameter count. It covers how Switch-style top-1 routing works with experts the same size as the dense feed-forward layer. It also covers the difference between active and total parameters, using a 1.1 billion total, 426 million active, four-expert example. The discussion explains how a smaller active size lets an MoE see more tokens for the same FLOPs. That ties into deliberate overtraining under Chinchilla-style tradeoffs, and into a memory definition that includes the KV cache. The hosts also place the work alongside related studies on sparsity, inference-aware scaling and hardware efficiency. They flag the gap between a loss-based claim and real serving hardware as the question to keep in mind.

This episode examines "Model soups," a paper showing that averaging the weights of several fine-tuned models can raise accuracy without adding any inference cost. It explains why this should be risky: the loss landscape has many basins, and averaging across a barrier breaks the network. Models fine-tuned from the same pre-trained checkpoint tend to share a basin, which is what makes the averaging work. The discussion covers the uniform soup and the greedy soup. The greedy soup adds checkpoints in order of validation accuracy and keeps one only if accuracy doesn't drop. It also compares soups against ensembles, which need k forward passes, and against stochastic weight averaging, which averages along a single training run. Listeners get a clear account of the ImageNet distribution-shift benchmarks, the oracle baseline, and an open question about whether the gains on shifted data come from soups simply pulling weights back toward the initialization.

This episode examines "Soup-of-Experts," an Apple paper on pretraining one bank of parameters that can be averaged into a small specialist model for any data mixture, with no retraining. It sets up the background: clustered importance sampling over 4096 RedPajama-v2 clusters, model soups from fine-tuning, per-domain expert baselines, and the hypernetwork framing. It also notes closely related prior work, Branch-Train-Merge and c-BTM, that the paper doesn't compare against. The discussion then moves to the mechanism: shared parameters plus 128 experts, combined by a small MLP that maps a domain histogram to mixing coefficients. Listeners get a clear look at the amortization argument, which is paying one large pretraining cost so each later specialist is nearly free. They also get a skeptical read on whether the headline comparison holds up.