DeLM: Decentralized Coding Agents Sharing Context Without a Leader

This episode examines PASTA, a method that lets users highlight part of a prompt and has the model attend to it more strongly, without changing any weights. It covers why markup like asterisks may not work as emphasis for LLMs. It then places PASTA against instruction tuning, activation steering (ActAdd, CAA, Inference-Time Intervention) and knowledge editing (ROME, MEMIT, REMEDI). The key difference is that PASTA changes where selected attention heads read, rather than what they output. The discussion explains the mechanism: scores outside the highlighted span are scaled by a small alpha (0.01) and renormalized, which is equivalent to a roughly hundredfold relative boost on the span. The hosts also raise questions about head selection and evaluation, including the absence of comparisons against the neighboring steering methods. Listeners interested in controllability, long-context failures and conflicts between a prompt and a model's memorized knowledge will find it useful.

This episode examines MEMORILLA, a Stanford-led preprint in which a small trained module reads a whole store of pre-embedded documents and hands a frozen language model just 16 soft-token vectors, instead of pasting retrieved text into the prompt. It motivates the design with a personalization scenario. An old, unrelated chat reveals a preference for blunt feedback, so similarity search may miss it, and the signal is spread across many chats that top-k retrieval can't integrate. The discussion contrasts this approach with full-context prompting, agent memory systems (MemGPT, Mem0, MemoRAG, A-MEM), question-blind compression methods (ICAE, xRAG), and per-corpus cache training (Cartridges). It also traces the design's lineage to Perceiver, BLIP-2's Q-Former and prompt tuning. Listeners get a skeptical framing of the headline claim, including whether gains come from reading the store or from training on the benchmarks, and whether comparing against baselines that weren't trained on them is fair. The evaluation covers personalization, long-document QA, knowledge-intensive QA and a TextWorld agent.

This episode explores Multi-Agent Transactive Memory, a paper asking whether many agents can dump their raw task trajectories into one shared, searchable repository and let other frozen models retrieve from it. That would improve success rates and cut step counts without any joint training or coordination. It builds up from retrieval-augmented generation and single-agent experience memory (Reflexion, Synapse, ExpeL) to the shared-pool idea, borrowing the name from Wegner's group-psychology notion of transactive memory. The discussion stresses that the analogy is loose: the system is closer to a Stack Overflow for agents than to a team, and the trajectories were never written to be read. It also covers the retrieval design, where the key is the task description plus the last five steps embedded with E5-Base and the stored value is the next five steps from a past run. The pool is seeded with public trajectories from models like GPT-4-Turbo, Claude 3.5 Sonnet, Qwen3-32B and GPT-OSS 20B, then grows as agents work. Listeners get a skeptical look at the ALFWorld and WebArena testbeds, including how small and templated they are, and at the missing private-versus-shared memory control that the paper's central claim needs.

This episode explores how Claude 3.5 Haiku decides when a hard-wrapped line of text is full, even though it sees only token IDs and is never told the line width. It covers the paper's finding that the model tracks the character count on a curved, roughly six-dimensional manifold. That manifold is tiled by about ten sparse features with widening tuning curves, a bit like place cells. The discussion connects this to earlier work on circular and helical feature geometry and to the rotary position encoding used in transformers, and it explains attribution graphs and sparse dictionary features along the way. It also covers the experimental setup: newlines reinserted at known widths from 15 to 150 characters. Zero-ablating the character-count subspace raises loss mainly on newline tokens, which gives causal evidence that the model uses this subspace and doesn't just encode the count somewhere a probe can find it. Listeners interested in mechanistic interpretability will find a concrete case of a model implementing a continuous quantity as geometry, not as one direction per concept.

This episode explores Jev-Mem, a design for agent memory that moves small management decisions, such as classifying a memory, linking it to others, and deciding when to stop searching, out of the language model's text generation and into a fast controller that outputs labels and scores. It walks through why agents need external memory despite long context windows (RULER's finding of shorter effective context, and Lost in the Middle). It traces the progression from plain retrieval through MemGPT-style paging to multi-graph memory, using a running example of a bicycle purchase that needs temporal, causal, and entity links to answer correctly. The discussion frames the "System One / System Two" split as an analogy for allocating compute rather than a claim about brains. It separates the approach from distilling reasoning into a single model, and sets up a three-layer architecture of control, memory, and reasoning planes. It also flags what to scrutinize: how much of the gain comes from the graph versus the controller, and that a closely related system is cited but not benchmarked. Listeners interested in cutting the token cost of agent memory will find the long-conversation benchmark setup and the typed yes/no query interface of the controller especially concrete.