Agentic Coding in the Wild: GitHub Copilot's Cache Behavior at Scale

This episode examines SanSi, a small "typed decision" model that returns a probability for each caller-declared option (such as spam or not spam) in a single forward pass, with no generated text to parse. It explains how the model gets more computation by looping the same 24-layer stack up to eight times, with a readout after every loop, so one model can serve any budget from one loop to eight. The discussion covers how this "System 1.5" approach differs from one-pass classifiers and from generated chain-of-thought. It also covers the earlier work it builds on: Universal Transformers, adaptive computation time, and pre-trained looped models such as Ouro. On training, the model uses a frozen backbone with LoRA adapters (about 61 million trainable parameters) and a loss averaged across all loops. Supervising only the final loop dropped loop-1 accuracy to 35.6%, against 58.4% with per-loop training. The episode also raises caveats: the Kahneman-style labels are framing rather than evidence, the RL case study uses the judge's own probability as reward rather than a truly verifiable one, and it is unclear whether looping beats spending the same compute another way.

This episode explores the paper "Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient," which asks whether a mixture-of-experts model can beat a dense model on loss when both train with the same compute and are capped at the same total parameter count. It covers how Switch-style top-1 routing works with experts the same size as the dense feed-forward layer. It also covers the difference between active and total parameters, using a 1.1 billion total, 426 million active, four-expert example. The discussion explains how a smaller active size lets an MoE see more tokens for the same FLOPs. That ties into deliberate overtraining under Chinchilla-style tradeoffs, and into a memory definition that includes the KV cache. The hosts also place the work alongside related studies on sparsity, inference-aware scaling and hardware efficiency. They flag the gap between a loss-based claim and real serving hardware as the question to keep in mind.

This episode examines "Model soups," a paper showing that averaging the weights of several fine-tuned models can raise accuracy without adding any inference cost. It explains why this should be risky: the loss landscape has many basins, and averaging across a barrier breaks the network. Models fine-tuned from the same pre-trained checkpoint tend to share a basin, which is what makes the averaging work. The discussion covers the uniform soup and the greedy soup. The greedy soup adds checkpoints in order of validation accuracy and keeps one only if accuracy doesn't drop. It also compares soups against ensembles, which need k forward passes, and against stochastic weight averaging, which averages along a single training run. Listeners get a clear account of the ImageNet distribution-shift benchmarks, the oracle baseline, and an open question about whether the gains on shifted data come from soups simply pulling weights back toward the initialization.

This episode examines "Soup-of-Experts," an Apple paper on pretraining one bank of parameters that can be averaged into a small specialist model for any data mixture, with no retraining. It sets up the background: clustered importance sampling over 4096 RedPajama-v2 clusters, model soups from fine-tuning, per-domain expert baselines, and the hypernetwork framing. It also notes closely related prior work, Branch-Train-Merge and c-BTM, that the paper doesn't compare against. The discussion then moves to the mechanism: shared parameters plus 128 experts, combined by a small MLP that maps a domain histogram to mixing coefficients. Listeners get a clear look at the amortization argument, which is paying one large pretraining cost so each later specialist is nearly free. They also get a skeptical read on whether the headline comparison holds up.

This episode examines Hydragen, a technique for speeding up LLM inference when many requests share a long prompt prefix, and asks how much of its claimed 32x end-to-end throughput gain comes from the idea and how much from the baseline. It explains why decode attention is memory-bound: each sequence runs a matrix-vector product against its own KV cache, which leaves tensor cores idle and makes batching unable to help. The central argument is that vLLM's PagedAttention deduplicates the storage of shared prefixes but not the reads, so every sequence still streams the full prefix from memory. Hydragen targets those reads by splitting attention into prefix and suffix parts and combining them using log-sum-exp weights. The authors present this as exact attention built on the FlashAttention family, with no approximation. The episode ends as the hosts ask why splitting the softmax is still exact.