Scaling Laws for Fine-Grained Mixture of Experts

This episode examines "500xCompressor," a soft prompt compression method that squeezes hundreds of tokens of context into as little as a single learned token. A frozen Llama-3-8B decoder then answers questions from that token's key-value (KV) vectors alone. It covers why long contexts are costly (KV cache memory and attention cost grow with length). It also contrasts hard compression, which deletes text (Selective Context, LLMLingua), with soft approaches (gist tokens, AutoCompressor, ICAE, xRAG). The discussion walks through how a LoRA-adapted encoder and autoencoder-style regeneration pretraining produce those KV values, and what the headline 6x–480x ratios and 62–73% retained capability actually measure. The hosts also scrutinize the evaluation: arXiv abstracts from early 2024 are used to avoid pretraining leakage, and F1 versus Exact Match are explained. Listeners get a critical look at whether KV beats embeddings and whether the benchmark can be trusted.

This episode explores a Lean 4 formalization of Hopfield networks and Boltzmann machines, covering about sixteen thousand new lines on top of mathlib, which the authors say is the first for either model in any proof assistant. It walks through Hopfield energy descent and asynchronous convergence, Hebbian weights and the cross-talk term for non-orthogonal patterns, and Gibbs sampling as a random-scan Markov chain with detailed balance. It also covers the zero-temperature limit that recovers the Hopfield rule. The main finding is that formalizing convergence required every neuron to be updated infinitely often, a fairness condition the textbook theorem left implicit. The episode also points out that the Boltzmann machine formalized is fully visible, with no hidden units, so the ergodicity result covers less than the word suggests. It places the work against modern context, including the link between continuous Hopfield updates and softmax attention and dense associative memory. Listeners interested in what machine-checked proofs expose in classic results will find it useful.

This episode examines "How Linear Attention Remembers," a preprint that asks how a single fact is written into, retained in, and read back out of the fixed-size recurrent state that replaces the KV cache. It contrasts Gated Linear Attention with Gated DeltaNet (whose delta rule overwrites old bindings instead of piling on top of them) and with hybrids that keep a few full-attention layers. The authors use associative-recall tests with causal interventions on pretrained models. They swap a head's state between near-identical "Paris" and "London" runs and measure the shift in the logit margin. Screening picks out one "hub" head per model, and the episode lays out the paper's four claims: content-specific writes, an identifiable read path, shared-state coupling that degrades recall as load grows, and attention layers taking over direct recall in hybrids. Listeners get the background on why a sum-based memory can't be pointed at the way a KV cache can, and a look at what that means for long-context serving on phones, edge devices, and robots.

This episode examines "HyperTuning," an ICML 2023 paper that uses a hypernetwork (HyperT5) to adapt a frozen language model to a new task in a single forward pass, with no back-propagation through the downstream model. It explains how the hypermodel reads up to sixteen labeled examples and emits prefix or LoRA parameters that the downstream model reuses for every query without seeing the examples again. The discussion covers why per-task adaptation is costly, how parameter-efficient methods still require a full backward pass, and how the work relates to earlier hypernetwork, meta-learning, and instruction-to-parameter approaches. It also sets up a skeptical reading of the paper's three claims: that HyperT5 roughly matches full fine-tuning on held-out tasks, that it beats shared-parameter PEFT baselines, and that its output works as a PEFT initialization. The skeptical questions are where the training cost actually moves to, and whether the headline margins hold beyond single-run noise.

This episode examines "Mixture of A Million Experts," a 2024 paper proposing a learned router that selects among more than a million single-neuron experts at sublinear cost. It explains why conventional mixture-of-experts routing stalls around 128 experts: scoring every expert against every token scales linearly. It also covers why fixed hash routers fall short. The discussion traces the paper's building blocks: fine-grained MoE scaling laws that favor ever-smaller experts, Product Key Memory's Cartesian-product trick for cheap retrieval, and the view of feedforward layers as key-value memories. Along the way, the hosts flag caveats, including the paper's redefinition of granularity, the extrapolation of a scaling law far beyond where it was fitted, and lifelong learning being motivation only, with no experiment. They also raise whether FLOP-based isoFLOP wins will hold up on real hardware, where scattered lookups into a million-entry table are a poor fit for large matrix multiplies.