MEMORILLA: Latent Semantic Memory Beats Text Retrieval for Frozen LLMs

This episode examines PASTA, a method that lets users highlight part of a prompt and has the model attend to it more strongly, without changing any weights. It covers why markup like asterisks may not work as emphasis for LLMs. It then places PASTA against instruction tuning, activation steering (ActAdd, CAA, Inference-Time Intervention) and knowledge editing (ROME, MEMIT, REMEDI). The key difference is that PASTA changes where selected attention heads read, rather than what they output. The discussion explains the mechanism: scores outside the highlighted span are scaled by a small alpha (0.01) and renormalized, which is equivalent to a roughly hundredfold relative boost on the span. The hosts also raise questions about head selection and evaluation, including the absence of comparisons against the neighboring steering methods. Listeners interested in controllability, long-context failures and conflicts between a prompt and a model's memorized knowledge will find it useful.

This episode examines DeLM, a Stanford system that runs several coding agents in parallel with no main agent, and asks whether four agents deliver four agents' worth of progress or just a bigger bill. It sorts existing designs by the "bubble" each one creates. Independent sampling wastes effort on redundant work. Peer debate stalls at round barriers and grows quadratically in messages. Centralized orchestrators like Claude Code and Codex leave the lead agent idle and pass lossy summaries between agents. DeLM replaces the lead with a self-service task queue, where the first claim wins, and an append-only shared log of FACT and FAIL entries that agents publish as they learn things. The discussion ties this to the 1980 Hearsay-II blackboard system, which had a scheduler where DeLM has none, and notes the new risk that stale or wrong natural-language findings can mislead peers. The hosts also say how they will judge the claimed accuracy and speed gains: whether the benchmarks are representative, whether the gaps exceed run-to-run noise, and how much of the speedup comes from the design rather than from idle baselines.

This episode explores Multi-Agent Transactive Memory, a paper asking whether many agents can dump their raw task trajectories into one shared, searchable repository and let other frozen models retrieve from it. That would improve success rates and cut step counts without any joint training or coordination. It builds up from retrieval-augmented generation and single-agent experience memory (Reflexion, Synapse, ExpeL) to the shared-pool idea, borrowing the name from Wegner's group-psychology notion of transactive memory. The discussion stresses that the analogy is loose: the system is closer to a Stack Overflow for agents than to a team, and the trajectories were never written to be read. It also covers the retrieval design, where the key is the task description plus the last five steps embedded with E5-Base and the stored value is the next five steps from a past run. The pool is seeded with public trajectories from models like GPT-4-Turbo, Claude 3.5 Sonnet, Qwen3-32B and GPT-OSS 20B, then grows as agents work. Listeners get a skeptical look at the ALFWorld and WebArena testbeds, including how small and templated they are, and at the missing private-versus-shared memory control that the paper's central claim needs.

This episode explores how Claude 3.5 Haiku decides when a hard-wrapped line of text is full, even though it sees only token IDs and is never told the line width. It covers the paper's finding that the model tracks the character count on a curved, roughly six-dimensional manifold. That manifold is tiled by about ten sparse features with widening tuning curves, a bit like place cells. The discussion connects this to earlier work on circular and helical feature geometry and to the rotary position encoding used in transformers, and it explains attribution graphs and sparse dictionary features along the way. It also covers the experimental setup: newlines reinserted at known widths from 15 to 150 characters. Zero-ablating the character-count subspace raises loss mainly on newline tokens, which gives causal evidence that the model uses this subspace and doesn't just encode the count somewhere a probe can find it. Listeners interested in mechanistic interpretability will find a concrete case of a model implementing a continuous quantity as geometry, not as one direction per concept.

This episode explores Jev-Mem, a design for agent memory that moves small management decisions, such as classifying a memory, linking it to others, and deciding when to stop searching, out of the language model's text generation and into a fast controller that outputs labels and scores. It walks through why agents need external memory despite long context windows (RULER's finding of shorter effective context, and Lost in the Middle). It traces the progression from plain retrieval through MemGPT-style paging to multi-graph memory, using a running example of a bicycle purchase that needs temporal, causal, and entity links to answer correctly. The discussion frames the "System One / System Two" split as an analogy for allocating compute rather than a claim about brains. It separates the approach from distilling reasoning into a single model, and sets up a three-layer architecture of control, memory, and reasoning planes. It also flags what to scrutinize: how much of the gain comes from the graph versus the controller, and that a closely related system is cited but not benchmarked. Listeners interested in cutting the token cost of agent memory will find the long-conversation benchmark setup and the typed yes/no query interface of the controller especially concrete.