PASTA: Steering LLM Attention Toward Highlighted Prompt Text

This episode explores how Claude 3.5 Haiku decides when a hard-wrapped line of text is full, even though it sees only token IDs and is never told the line width. It covers the paper's finding that the model tracks the character count on a curved, roughly six-dimensional manifold. That manifold is tiled by about ten sparse features with widening tuning curves, a bit like place cells. The discussion connects this to earlier work on circular and helical feature geometry and to the rotary position encoding used in transformers, and it explains attribution graphs and sparse dictionary features along the way. It also covers the experimental setup: newlines reinserted at known widths from 15 to 150 characters. Zero-ablating the character-count subspace raises loss mainly on newline tokens, which gives causal evidence that the model uses this subspace and doesn't just encode the count somewhere a probe can find it. Listeners interested in mechanistic interpretability will find a concrete case of a model implementing a continuous quantity as geometry, not as one direction per concept.

This episode explores Jev-Mem, a design for agent memory that moves small management decisions, such as classifying a memory, linking it to others, and deciding when to stop searching, out of the language model's text generation and into a fast controller that outputs labels and scores. It walks through why agents need external memory despite long context windows (RULER's finding of shorter effective context, and Lost in the Middle). It traces the progression from plain retrieval through MemGPT-style paging to multi-graph memory, using a running example of a bicycle purchase that needs temporal, causal, and entity links to answer correctly. The discussion frames the "System One / System Two" split as an analogy for allocating compute rather than a claim about brains. It separates the approach from distilling reasoning into a single model, and sets up a three-layer architecture of control, memory, and reasoning planes. It also flags what to scrutinize: how much of the gain comes from the graph versus the controller, and that a closely related system is cited but not benchmarked. Listeners interested in cutting the token cost of agent memory will find the long-conversation benchmark setup and the typed yes/no query interface of the controller especially concrete.

This episode explores a paper on harness learning, in which a small proposer model is trained with reinforcement learning to rewrite the code wrapped around a frozen solver. The aim is for the system to adapt to unseen tasks with no weight updates. It defines a harness as the program that controls what the model sees, which tools it uses, and how outputs are validated and retried. It contrasts the approach with search-based optimizers such as DSPy, where the optimizer is just a prompted model. The discussion frames the method as loose meta-learning, with code edits standing in for gradient steps in the style of MAML and RL-squared. It also covers GRPO training on verifiable rewards and a Reasoning Gym testbed with held-out puzzle families. Listeners get a clear account of the paper's three questions: whether learned revisions generalize, whether repeated revision helps, and whether training on whole revision sequences beats single edits. The episode also sets up two critiques, including "Harness Updating Is Not Harness Benefit," as lenses for judging the evidence.

This episode examines "Cogentic," a multi-agent harness that reportedly produced new, expert-verified proofs for five open problems in online learning, auction theory, and mechanism design, using about a hundred model calls on most problems and about a thousand on the hardest. It places the work among three approaches: formal theorem provers like Lean, search-for-objects systems like FunSearch and AlphaEvolve that need a computable score, and inference-time scaling, where coverage only becomes an answer if something reliably picks the right candidate. The discussion walks through the harness design. An orchestrator assigns directions, parallel prover agents draft proofs, per-prover summarizers give each a different reading of the history, and adversarial per-draft and cross-draft verifiers must both pass a result. It also raises open questions: the verifiers are prompted copies of the same model family as the provers, nothing is machine-checked, the models are proprietary, and the number of failed attempts behind the five wins and the cost are unclear. A listener would find it useful for judging whether the harness itself, or just a strong model with patience, deserves the credit.

This episode examines why a stock CXL-attached SSD, meaning flash capacity exposed as load/store memory behind a small device-side DRAM cache, serves reused prompt prefixes about three times slower than local DRAM. It explains how prefix caching cuts time-to-first-token by skipping prefill, and how LMCache stores KV in fixed-size token chunks across GPU, host DRAM, and storage tiers. A ramdisk experiment on Qwen3-4B with a 34k-token prefix shows the block I/O interface alone costs 1.8x in TTFT, through staging copies and last-level cache contention, even with no flash involved. A generic next-n prefetcher in the device also fails, because the device sees only page addresses while the serving engine knows which chunks a request needs and in what order. That semantic gap motivates the paper's redesign around the chunk as the shared unit between engine and device, and the episode compares it briefly with the concurrent HyMCache and ITME work.