Context Language Models: Letting Models Edit Their Own Context

This episode examines "500xCompressor," a soft prompt compression method that squeezes hundreds of tokens of context into as little as a single learned token. A frozen Llama-3-8B decoder then answers questions from that token's key-value (KV) vectors alone. It covers why long contexts are costly (KV cache memory and attention cost grow with length). It also contrasts hard compression, which deletes text (Selective Context, LLMLingua), with soft approaches (gist tokens, AutoCompressor, ICAE, xRAG). The discussion walks through how a LoRA-adapted encoder and autoencoder-style regeneration pretraining produce those KV values, and what the headline 6x–480x ratios and 62–73% retained capability actually measure. The hosts also scrutinize the evaluation: arXiv abstracts from early 2024 are used to avoid pretraining leakage, and F1 versus Exact Match are explained. Listeners get a critical look at whether KV beats embeddings and whether the benchmark can be trusted.

This episode examines a 2024 paper arguing that mixture-of-experts models outperform dense Transformers at every compute budget, with a gap that widens at scale. That claim contradicts earlier Meta and DeepMind studies, which saw the MoE advantage shrink, and the authors blame two hidden assumptions: a fixed training duration and a fixed expert size. The discussion builds the background from dense feed-forward layers, routers, and the difference between active and total parameters, through the Kaplan and Chinchilla scaling laws. It then introduces "granularity", the ratio of dense feed-forward width to expert width, as a third scaling axis. Granularity is a design choice that was inherited by convention and never shown to be optimal. The episode also stays skeptical of the evidence. The law is fit on a ragged grid of small models (3M to 85M active parameters), yet its headline claims extrapolate to a trillion active parameters, so listeners get a clear view of both the idea and its limits.

This episode examines "HyperTuning," an ICML 2023 paper that uses a hypernetwork (HyperT5) to adapt a frozen language model to a new task in a single forward pass, with no back-propagation through the downstream model. It explains how the hypermodel reads up to sixteen labeled examples and emits prefix or LoRA parameters that the downstream model reuses for every query without seeing the examples again. The discussion covers why per-task adaptation is costly, how parameter-efficient methods still require a full backward pass, and how the work relates to earlier hypernetwork, meta-learning, and instruction-to-parameter approaches. It also sets up a skeptical reading of the paper's three claims: that HyperT5 roughly matches full fine-tuning on held-out tasks, that it beats shared-parameter PEFT baselines, and that its output works as a PEFT initialization. The skeptical questions are where the training cost actually moves to, and whether the headline margins hold beyond single-run noise.

This episode examines "Mixture of A Million Experts," a 2024 paper proposing a learned router that selects among more than a million single-neuron experts at sublinear cost. It explains why conventional mixture-of-experts routing stalls around 128 experts: scoring every expert against every token scales linearly. It also covers why fixed hash routers fall short. The discussion traces the paper's building blocks: fine-grained MoE scaling laws that favor ever-smaller experts, Product Key Memory's Cartesian-product trick for cheap retrieval, and the view of feedforward layers as key-value memories. Along the way, the hosts flag caveats, including the paper's redefinition of granularity, the extrapolation of a scaling law far beyond where it was fitted, and lifelong learning being motivation only, with no experiment. They also raise whether FLOP-based isoFLOP wins will hold up on real hardware, where scattered lookups into a million-entry table are a poor fit for large matrix multiplies.

This episode explores a Lean 4 formalization of Hopfield networks and Boltzmann machines, covering about sixteen thousand new lines on top of mathlib, which the authors say is the first for either model in any proof assistant. It walks through Hopfield energy descent and asynchronous convergence, Hebbian weights and the cross-talk term for non-orthogonal patterns, and Gibbs sampling as a random-scan Markov chain with detailed balance. It also covers the zero-temperature limit that recovers the Hopfield rule. The main finding is that formalizing convergence required every neuron to be updated infinitely often, a fairness condition the textbook theorem left implicit. The episode also points out that the Boltzmann machine formalized is fully visible, with no hidden units, so the ergodicity result covers less than the word suggests. It places the work against modern context, including the link between continuous Hopfield updates and softmax attention and dense associative memory. Listeners interested in what machine-checked proofs expose in classic results will find it useful.