Mixture of a Million Experts: Scaling Routing with Product Keys

This episode examines "500xCompressor," a soft prompt compression method that squeezes hundreds of tokens of context into as little as a single learned token. A frozen Llama-3-8B decoder then answers questions from that token's key-value (KV) vectors alone. It covers why long contexts are costly (KV cache memory and attention cost grow with length). It also contrasts hard compression, which deletes text (Selective Context, LLMLingua), with soft approaches (gist tokens, AutoCompressor, ICAE, xRAG). The discussion walks through how a LoRA-adapted encoder and autoencoder-style regeneration pretraining produce those KV values, and what the headline 6x–480x ratios and 62–73% retained capability actually measure. The hosts also scrutinize the evaluation: arXiv abstracts from early 2024 are used to avoid pretraining leakage, and F1 versus Exact Match are explained. Listeners get a critical look at whether KV beats embeddings and whether the benchmark can be trusted.

This episode explores a Lean 4 formalization of Hopfield networks and Boltzmann machines, covering about sixteen thousand new lines on top of mathlib, which the authors say is the first for either model in any proof assistant. It walks through Hopfield energy descent and asynchronous convergence, Hebbian weights and the cross-talk term for non-orthogonal patterns, and Gibbs sampling as a random-scan Markov chain with detailed balance. It also covers the zero-temperature limit that recovers the Hopfield rule. The main finding is that formalizing convergence required every neuron to be updated infinitely often, a fairness condition the textbook theorem left implicit. The episode also points out that the Boltzmann machine formalized is fully visible, with no hidden units, so the ergodicity result covers less than the word suggests. It places the work against modern context, including the link between continuous Hopfield updates and softmax attention and dense associative memory. Listeners interested in what machine-checked proofs expose in classic results will find it useful.

This episode examines "How Linear Attention Remembers," a preprint that asks how a single fact is written into, retained in, and read back out of the fixed-size recurrent state that replaces the KV cache. It contrasts Gated Linear Attention with Gated DeltaNet (whose delta rule overwrites old bindings instead of piling on top of them) and with hybrids that keep a few full-attention layers. The authors use associative-recall tests with causal interventions on pretrained models. They swap a head's state between near-identical "Paris" and "London" runs and measure the shift in the logit margin. Screening picks out one "hub" head per model, and the episode lays out the paper's four claims: content-specific writes, an identifiable read path, shared-state coupling that degrades recall as load grows, and attention layers taking over direct recall in hybrids. Listeners get the background on why a sum-based memory can't be pointed at the way a KV cache can, and a look at what that means for long-context serving on phones, edge devices, and robots.

This episode examines a 2024 paper arguing that mixture-of-experts models outperform dense Transformers at every compute budget, with a gap that widens at scale. That claim contradicts earlier Meta and DeepMind studies, which saw the MoE advantage shrink, and the authors blame two hidden assumptions: a fixed training duration and a fixed expert size. The discussion builds the background from dense feed-forward layers, routers, and the difference between active and total parameters, through the Kaplan and Chinchilla scaling laws. It then introduces "granularity", the ratio of dense feed-forward width to expert width, as a third scaling axis. Granularity is a design choice that was inherited by convention and never shown to be optimal. The episode also stays skeptical of the evidence. The law is fit on a ragged grid of small models (3M to 85M active parameters), yet its headline claims extrapolate to a trillion active parameters, so listeners get a clear view of both the idea and its limits.

This episode examines "HyperTuning," an ICML 2023 paper that uses a hypernetwork (HyperT5) to adapt a frozen language model to a new task in a single forward pass, with no back-propagation through the downstream model. It explains how the hypermodel reads up to sixteen labeled examples and emits prefix or LoRA parameters that the downstream model reuses for every query without seeing the examples again. The discussion covers why per-task adaptation is costly, how parameter-efficient methods still require a full backward pass, and how the work relates to earlier hypernetwork, meta-learning, and instruction-to-parameter approaches. It also sets up a skeptical reading of the paper's three claims: that HyperT5 roughly matches full fine-tuning on held-out tasks, that it beats shared-parameter PEFT baselines, and that its output works as a PEFT initialization. The skeptical questions are where the training cost actually moves to, and whether the headline margins hold beyond single-run noise.