Show Notes
This episode examines Self-Evolving Curriculum, a 2025 paper proposing that a small multi-armed bandit can choose which problem categories an LLM trains on during reinforcement learning from verifiable rewards. It explains why GRPO gets no learning signal from problems the model always solves or never solves, since every advantage in the group is zero. It contrasts the bandit approach with fixed easy-to-hard schedules, random shuffling, and DAPO-style dynamic sampling, which pays for rollouts it then discards. The discussion covers the non-stationary bandit setup: each difficulty level or problem type is an arm, and the signal is reused from advantages GRPO already computes. It also looks at the Countdown, Zebra, ARC-1D and MATH experiments, including the point that the "out-of-distribution" claims are cleaner for the puzzles than for the math benchmarks. Listeners get a clear walkthrough of the mechanism and a skeptical look at whether the evidence supports the paper's claims.
Sources: 1. Self-Evolving Curriculum for LLM Reasoning — Xiaoyin Chen, Jiarui Lu, Minsu Kim, Dinghuai Zhang, Jian Tang, Alexandre Piché, Nicolas Gontier, Yoshua Bengio, Ehsan Kamalloo, 2025 http://arxiv.org/abs/2505.14970 2. Curriculum Learning — Yoshua Bengio, Jérôme Louradour, Ronan Collobert, Jason Weston, 2009 https://scholar.google.com/scholar?q=Curriculum+Learning 3. Automated Curriculum Learning for Neural Networks — Alex Graves, Marc G. Bellemare, Jacob Menick, Rémi Munos, Koray Kavukcuoglu, 2017 https://scholar.google.com/scholar?q=Automated+Curriculum+Learning+for+Neural+Networks 4. Teacher-Student Curriculum Learning — Tambet Matiisen, Avital Oliver, Taco Cohen, John Schulman, 2017 (arXiv); published in IEEE Transactions on Neural Networks and Learning Systems, 2019 https://scholar.google.com/scholar?q=Teacher-Student+Curriculum+Learning 5. Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey — Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E. Taylor, Peter Stone, 2020 https://scholar.google.com/scholar?q=Curriculum+Learning+for+Reinforcement+Learning+Domains%3A+A+Framework+and+Survey 6. Efficient Online Data Mixing For Language Model Pre-Training — Alon Albalak, Liangming Pan, Colin Raffel, William Yang Wang, 2023 https://scholar.google.com/scholar?q=Efficient+Online+Data+Mixing+For+Language+Model+Pre-Training 7. DAPO: An Open-Source LLM Reinforcement Learning System at Scale — Qiying Yu et al., 2025 https://scholar.google.com/scholar?q=DAPO%3A+An+Open-Source+LLM+Reinforcement+Learning+System+at+Scale 8. DUMP: Automated Distribution-Level Curriculum Learning for RL-based LLM Post-training — Zhenting Wang, Guofeng Cui, Kun Wan, Wentian Zhao, 2025 https://scholar.google.com/scholar?q=DUMP%3A+Automated+Distribution-Level+Curriculum+Learning+for+RL-based+LLM+Post-training 9. Efficient Reinforcement Finetuning via Adaptive Curriculum Learning (AdaRFT) — Taiwei Shi, Yiyang Wu, Linxin Song, Tianyi Zhou, Jieyu Zhao, 2025 https://scholar.google.com/scholar?q=Efficient+Reinforcement+Finetuning+via+Adaptive+Curriculum+Learning+%28AdaRFT%29 10. Online Difficulty Filtering for Reasoning Oriented Reinforcement Learning — Sanghwan Bae et al., 2025 https://scholar.google.com/scholar?q=Online+Difficulty+Filtering+for+Reasoning+Oriented+Reinforcement+Learning 11. Prioritized Experience Replay — Tom Schaul, John Quan, Ioannis Antonoglou, David Silver, 2015 https://scholar.google.com/scholar?q=Prioritized+Experience+Replay 12. Not All Samples Are Created Equal: Deep Learning with Importance Sampling — Angelos Katharopoulos, François Fleuret, 2018 https://scholar.google.com/scholar?q=Not+All+Samples+Are+Created+Equal%3A+Deep+Learning+with+Importance+Sampling 13. Evolving Curricula with Regret-Based Environment Design (ACCEL) — Jack Parker-Holder, Minqi Jiang, Michael Dennis, et al., 2022 https://scholar.google.com/scholar?q=Evolving+Curricula+with+Regret-Based+Environment+Design+%28ACCEL%29
More from AI Post Transformers
This episode examines PASTA, a method that lets users highlight part of a prompt and has the model attend to it more strongly, without changing any weights. It covers why markup like asterisks may not work as emphasis for LLMs. It then places PASTA against instruction tuning, activation steering (ActAdd, CAA, Inference-Time Intervention) and knowledge editing (ROME, MEMIT, REMEDI). The key difference is that PASTA changes where selected attention heads read, rather than what they output. The discussion explains the mechanism: scores outside the highlighted span are scaled by a small alpha (0.01) and renormalized, which is equivalent to a roughly hundredfold relative boost on the span. The hosts also raise questions about head selection and evaluation, including the absence of comparisons against the neighboring steering methods. Listeners interested in controllability, long-context failures and conflicts between a prompt and a model's memorized knowledge will find it useful.
This episode examines MEMORILLA, a Stanford-led preprint in which a small trained module reads a whole store of pre-embedded documents and hands a frozen language model just 16 soft-token vectors, instead of pasting retrieved text into the prompt. It motivates the design with a personalization scenario. An old, unrelated chat reveals a preference for blunt feedback, so similarity search may miss it, and the signal is spread across many chats that top-k retrieval can't integrate. The discussion contrasts this approach with full-context prompting, agent memory systems (MemGPT, Mem0, MemoRAG, A-MEM), question-blind compression methods (ICAE, xRAG), and per-corpus cache training (Cartridges). It also traces the design's lineage to Perceiver, BLIP-2's Q-Former and prompt tuning. Listeners get a skeptical framing of the headline claim, including whether gains come from reading the store or from training on the benchmarks, and whether comparing against baselines that weren't trained on them is fair. The evaluation covers personalization, long-document QA, knowledge-intensive QA and a TextWorld agent.
This episode examines DeLM, a Stanford system that runs several coding agents in parallel with no main agent, and asks whether four agents deliver four agents' worth of progress or just a bigger bill. It sorts existing designs by the "bubble" each one creates. Independent sampling wastes effort on redundant work. Peer debate stalls at round barriers and grows quadratically in messages. Centralized orchestrators like Claude Code and Codex leave the lead agent idle and pass lossy summaries between agents. DeLM replaces the lead with a self-service task queue, where the first claim wins, and an append-only shared log of FACT and FAIL entries that agents publish as they learn things. The discussion ties this to the 1980 Hearsay-II blackboard system, which had a scheduler where DeLM has none, and notes the new risk that stale or wrong natural-language findings can mislead peers. The hosts also say how they will judge the claimed accuracy and speed gains: whether the benchmarks are representative, whether the gaps exceed run-to-run noise, and how much of the speedup comes from the design rather than from idle baselines.
This episode explores Multi-Agent Transactive Memory, a paper asking whether many agents can dump their raw task trajectories into one shared, searchable repository and let other frozen models retrieve from it. That would improve success rates and cut step counts without any joint training or coordination. It builds up from retrieval-augmented generation and single-agent experience memory (Reflexion, Synapse, ExpeL) to the shared-pool idea, borrowing the name from Wegner's group-psychology notion of transactive memory. The discussion stresses that the analogy is loose: the system is closer to a Stack Overflow for agents than to a team, and the trajectories were never written to be read. It also covers the retrieval design, where the key is the task description plus the last five steps embedded with E5-Base and the stored value is the next five steps from a past run. The pool is seeded with public trajectories from models like GPT-4-Turbo, Claude 3.5 Sonnet, Qwen3-32B and GPT-OSS 20B, then grows as agents work. Listeners get a skeptical look at the ALFWorld and WebArena testbeds, including how small and templated they are, and at the missing private-versus-shared memory control that the paper's central claim needs.
This episode explores how Claude 3.5 Haiku decides when a hard-wrapped line of text is full, even though it sees only token IDs and is never told the line width. It covers the paper's finding that the model tracks the character count on a curved, roughly six-dimensional manifold. That manifold is tiled by about ten sparse features with widening tuning curves, a bit like place cells. The discussion connects this to earlier work on circular and helical feature geometry and to the rotary position encoding used in transformers, and it explains attribution graphs and sparse dictionary features along the way. It also covers the experimental setup: newlines reinserted at known widths from 15 to 150 characters. Zero-ablating the character-count subspace raises loss mainly on newline tokens, which gives causal evidence that the model uses this subspace and doesn't just encode the count somewhere a probe can find it. Listeners interested in mechanistic interpretability will find a concrete case of a model implementing a continuous quantity as geometry, not as one direction per concept.