Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient

This episode examines a production-scale characterization of GitHub Copilot's coding agent, built on sampled client-side telemetry from one week in June 2026 (13.5 million sessions and 760 million LLM calls, though some totals are inconsistent). It argues that agent traffic is session-shaped rather than stateless chat. About 87% of LLM calls are agent-initiated, the median call carries roughly 68,000 prompt tokens to produce about 247 output tokens, and the human appears only at turn boundaries. The discussion explains how prefix caching should give very high reuse inside a turn, and why the cache is lost after a pause, a model switch, or context compaction. It also separates "container idle" from "KV-cache idle", which suggests turn boundaries are the natural point to reclaim resources. The episode places the work alongside prior agentic KV-retention research and serverless idle-time studies. Listeners get a clear framework for why request-level scheduling and plain LRU eviction fall short for agent workloads, along with caveats about inferring server behavior from client data alone.

This episode examines SanSi, a small "typed decision" model that returns a probability for each caller-declared option (such as spam or not spam) in a single forward pass, with no generated text to parse. It explains how the model gets more computation by looping the same 24-layer stack up to eight times, with a readout after every loop, so one model can serve any budget from one loop to eight. The discussion covers how this "System 1.5" approach differs from one-pass classifiers and from generated chain-of-thought. It also covers the earlier work it builds on: Universal Transformers, adaptive computation time, and pre-trained looped models such as Ouro. On training, the model uses a frozen backbone with LoRA adapters (about 61 million trainable parameters) and a loss averaged across all loops. Supervising only the final loop dropped loop-1 accuracy to 35.6%, against 58.4% with per-loop training. The episode also raises caveats: the Kahneman-style labels are framing rather than evidence, the RL case study uses the judge's own probability as reward rather than a truly verifiable one, and it is unclear whether looping beats spending the same compute another way.

This episode examines "Model soups," a paper showing that averaging the weights of several fine-tuned models can raise accuracy without adding any inference cost. It explains why this should be risky: the loss landscape has many basins, and averaging across a barrier breaks the network. Models fine-tuned from the same pre-trained checkpoint tend to share a basin, which is what makes the averaging work. The discussion covers the uniform soup and the greedy soup. The greedy soup adds checkpoints in order of validation accuracy and keeps one only if accuracy doesn't drop. It also compares soups against ensembles, which need k forward passes, and against stochastic weight averaging, which averages along a single training run. Listeners get a clear account of the ImageNet distribution-shift benchmarks, the oracle baseline, and an open question about whether the gains on shifted data come from soups simply pulling weights back toward the initialization.

This episode examines "Soup-of-Experts," an Apple paper on pretraining one bank of parameters that can be averaged into a small specialist model for any data mixture, with no retraining. It sets up the background: clustered importance sampling over 4096 RedPajama-v2 clusters, model soups from fine-tuning, per-domain expert baselines, and the hypernetwork framing. It also notes closely related prior work, Branch-Train-Merge and c-BTM, that the paper doesn't compare against. The discussion then moves to the mechanism: shared parameters plus 128 experts, combined by a small MLP that maps a domain histogram to mixing coefficients. Listeners get a clear look at the amortization argument, which is paying one large pretraining cost so each later specialist is nearly free. They also get a skeptical read on whether the headline comparison holds up.

This episode examines Hydragen, a technique for speeding up LLM inference when many requests share a long prompt prefix, and asks how much of its claimed 32x end-to-end throughput gain comes from the idea and how much from the baseline. It explains why decode attention is memory-bound: each sequence runs a matrix-vector product against its own KV cache, which leaves tensor cores idle and makes batching unable to help. The central argument is that vLLM's PagedAttention deduplicates the storage of shared prefixes but not the reads, so every sequence still streams the full prefix from memory. Hydragen targets those reads by splitting attention into prefix and suffix parts and combining them using log-sum-exp weights. The authors present this as exact attention built on the FlashAttention family, with no approximation. The episode ends as the hosts ask why splitting the softmax is still exact.