500xCompressor: Compressing Prompts into a Single Learned Token

This episode explores a Lean 4 formalization of Hopfield networks and Boltzmann machines, covering about sixteen thousand new lines on top of mathlib, which the authors say is the first for either model in any proof assistant. It walks through Hopfield energy descent and asynchronous convergence, Hebbian weights and the cross-talk term for non-orthogonal patterns, and Gibbs sampling as a random-scan Markov chain with detailed balance. It also covers the zero-temperature limit that recovers the Hopfield rule. The main finding is that formalizing convergence required every neuron to be updated infinitely often, a fairness condition the textbook theorem left implicit. The episode also points out that the Boltzmann machine formalized is fully visible, with no hidden units, so the ergodicity result covers less than the word suggests. It places the work against modern context, including the link between continuous Hopfield updates and softmax attention and dense associative memory. Listeners interested in what machine-checked proofs expose in classic results will find it useful.

This episode examines "How Linear Attention Remembers," a preprint that asks how a single fact is written into, retained in, and read back out of the fixed-size recurrent state that replaces the KV cache. It contrasts Gated Linear Attention with Gated DeltaNet (whose delta rule overwrites old bindings instead of piling on top of them) and with hybrids that keep a few full-attention layers. The authors use associative-recall tests with causal interventions on pretrained models. They swap a head's state between near-identical "Paris" and "London" runs and measure the shift in the logit margin. Screening picks out one "hub" head per model, and the episode lays out the paper's four claims: content-specific writes, an identifiable read path, shared-state coupling that degrades recall as load grows, and attention layers taking over direct recall in hybrids. Listeners get the background on why a sum-based memory can't be pointed at the way a KV cache can, and a look at what that means for long-context serving on phones, edge devices, and robots.

This episode examines a 2024 paper arguing that mixture-of-experts models outperform dense Transformers at every compute budget, with a gap that widens at scale. That claim contradicts earlier Meta and DeepMind studies, which saw the MoE advantage shrink, and the authors blame two hidden assumptions: a fixed training duration and a fixed expert size. The discussion builds the background from dense feed-forward layers, routers, and the difference between active and total parameters, through the Kaplan and Chinchilla scaling laws. It then introduces "granularity", the ratio of dense feed-forward width to expert width, as a third scaling axis. Granularity is a design choice that was inherited by convention and never shown to be optimal. The episode also stays skeptical of the evidence. The law is fit on a ragged grid of small models (3M to 85M active parameters), yet its headline claims extrapolate to a trillion active parameters, so listeners get a clear view of both the idea and its limits.

This episode examines "HyperTuning," an ICML 2023 paper that uses a hypernetwork (HyperT5) to adapt a frozen language model to a new task in a single forward pass, with no back-propagation through the downstream model. It explains how the hypermodel reads up to sixteen labeled examples and emits prefix or LoRA parameters that the downstream model reuses for every query without seeing the examples again. The discussion covers why per-task adaptation is costly, how parameter-efficient methods still require a full backward pass, and how the work relates to earlier hypernetwork, meta-learning, and instruction-to-parameter approaches. It also sets up a skeptical reading of the paper's three claims: that HyperT5 roughly matches full fine-tuning on held-out tasks, that it beats shared-parameter PEFT baselines, and that its output works as a PEFT initialization. The skeptical questions are where the training cost actually moves to, and whether the headline margins hold beyond single-run noise.

This episode examines "Mixture of A Million Experts," a 2024 paper proposing a learned router that selects among more than a million single-neuron experts at sublinear cost. It explains why conventional mixture-of-experts routing stalls around 128 experts: scoring every expert against every token scales linearly. It also covers why fixed hash routers fall short. The discussion traces the paper's building blocks: fine-grained MoE scaling laws that favor ever-smaller experts, Product Key Memory's Cartesian-product trick for cheap retrieval, and the view of feedforward layers as key-value memories. Along the way, the hosts flag caveats, including the paper's redefinition of granularity, the extrapolation of a scaling law far beyond where it was fitted, and lifelong learning being motivation only, with no experiment. They also raise whether FLOP-based isoFLOP wins will hold up on real hardware, where scattered lookups into a million-entry table are a poor fit for large matrix multiplies.