Show Notes
Pretrained vision-language-action models for robots often fail when a task is performed in a slightly different environment. With Vision-Language Steering, you can get better generalization performance out of a frozen robot policy, by steering the sampling process of your pretrained diffusion or flow-matching policy to handle out-of-distribution changes like obstacle position or changing visual appearance. Shuo Lin and Ishneet Sukhviner Singh join us to talk about how to better generalize robot policies with no new training data. Watch Episode 108 of RoboPapers today, with Jiafei Duan and Ruijie He! Abstract Why do pretrained diffusion or flow-matching policies fail when the same task is performed near an obstacle, on a shifted support surface, or amid mild clutter? Such failures rarely reflect missing motor skills; instead, they expose a limitation of imitation learning under train–test shifts, where action generation is tightly coupled to training-specific spatial configurations and task specifications. Retraining or fine-tuning to address these failures is costly and conceptually misaligned, as the required behaviors already exist but cannot be selectively adapted at test time. We propose Vision–Language Steering (VLS), a training-free framework for inference-time adaptation of frozen generative robot policies. VLS treats adaptation as an inference-time control problem, steering the sampling process of a pretrained diffusion or flow-matching policy in response to out-of-distribution observation–language inputs without modifying policy parameters. By leveraging vision–language models to synthesize trajectory-differentiable reward functions, VLS guides denoising toward action trajectories that satisfy test-time spatial and task requirements. Learn More Project Page: https://vision-language-steering.github.io/webpage/ ArXiV: https://arxiv.org/pdf/2602.03973 Transcript Transcripts are lightly edited by ChatGPT. RJ He (00:01) Okay. Hi everyone. Welcome to another episode of Robot Papers. Today we are super excited to have with us Shuo Liu and Ishneet, who will be sharing their paper on Vision-Language Steering: Steering Pre-Trained Robot Policies via Vision-Language Models. Shuo, Ishneet, do you all want to introduce yourselves? Shuo Liu (00:29) Yes. Hi everyone. My name is Shuo Liu. For now I’m a research engineer at Cortex Robot AI and previously I graduated from University of Washington, where I was also a student researcher at Allen Institute for AI, advised by Professor Ranji Krishna and Professor Jiafei Duan, focus on robot learning from develop foundation models to deploying them into real world. RJ He (01:01) Isn’t it? Ishneet (01:02) Yeah. Hi everyone, my name is Ishneet. I work with Professor Jiafei on this paper, VLS, with Shuo. I’m currently at Menlo Research, where I also work on Asimov, and I’ll be going to Oxford for my undergrad later this year. RJ He (01:17) Super exciting. Amazing that you’re already having publications at this young age. It’s awesome. Okay, without further ado, Shuo, why don’t you take us through the paper? Shuo Liu (01:26) Okay, sure, yeah. Let me share the screen. Okay, so just make sure it’s it’s good to say right. RJ He (01:38) Yeah, it’s good. Shuo Liu (01:39) Yeah. Hi everyone. Today I want to share our work vision language theory. This work is finished when I was a master student at the University of Washington and advised by Professor Ranjik Krishna and Professor Jiafei Duan. And so today I want to introduce motivation of this work and the method we solve this problem and the experiment results and also some future works maybe could be done from this direction. So the problem is from the observation of some small front twining of the Like models like π0 or π0 Po, we found that current generative robot policies are can’t are kind of tightly coupled of the training specific configurations, so which cause some failing to decouple the reusable motor skills from the normal task constraints.
More from RoboPapers
Touch is crucial for humans when they want to perform precise tasks to a high degree of reliability, and yet robots almost universally use vision — they’re often completely blind to this whole modality. Two works from Tong Jian, Tao Yu, Saleh Nabi, and Matteo Grimaldi show us how important this sense is. In SlipSense, they look at how to detect objects slipping in grippers using a compact multimodal tactile sensor, which generalizes across unseen objects and to different robots. They then talk about how tactile sensing can be used to enable high real-world performance with a challenging cable-tracing task, where a cable must be fed through the hand with a repeated pinch-and-curl motion. To learn more, watch Episode 112 of RoboPapers today, with Michael Cho and Ruijie He! SlipSense Abstract Slip detection is fundamental to dexterous manipulation, yet existing systems often lack precise characterization of detection latency and cross-platform generalization. We present SlipSense, a multimodal tactile slip-detection framework built on TacV5, a compact sensor integrating a 32×32 piezoresistive array operating at 240 Hz and a 3-axis MEMS accelerometer operating at 8 kHz. The piezoresistive array captures spatial pressure distributions, while the accelerometer captures friction-induced vibrations, providing complementary slip cues. The framework performs modality-specific encoding, intra-sensor fusion, and cross-modal attention with causal temporal prediction at 240 Hz. Experiments on a dataset of 1.4 million frames spanning 37 objects demonstrate the complementarity of the two modalities. SlipSense achieves 96.7% Macro F1 with a false-positive rate below 1.6%, detecting 76% of slip events within 23.1 ms. When trained solely on UMI data, SlipSense generalizes zero-shot to a Tesollo dexterous hand, transferring across unseen objects, distinct sensor units, and robotic platforms without retraining. Touch2Trace Abstract Dexterous manipulation of deformable objects demands continuous fingertip-level regulation of pressure, friction, and incipient slip. We study one of the most challenging cases: dexterous cable tracing, feeding a cable through the hand with repeated pinch-and-curl motions of the thumb and index finger. We introduce Touch2Trace, a tactile-driven imitation-learning system for this task, and provide, to our knowledge, the first systematic real-world characterization of how encoder pretraining, control rate, temporal context, and spatial resolution each shape policy performance. The winning learning recipe combines a tactile encoder pretrained for a custom 32 x 32 piezoresistive sensor (TacV5) via self-supervised learning with a lightweight transformer policy trained on teleoperated demonstrations via behavior cloning, deployed at 60 Hz on a Tesollo DG-5F hand. Tactile feedback without vision or explicit cable-state estimation significantly improves tracing performance versus a proprioception-only baseline: from 0.2 cm to 20.1 cm mean distance and 0% to 93% success rate, with zero-shot transfer to unseen cables and routing conditions. The results quantify the influence of key parameters in tactile-driven systems for reliable dexterous deformable object manipulation. Learn More SlipSense Project Page: https://tjian-ai.github.io/corl26-slipsense/ SlipSense on ArXiV: https://arxiv.org/abs/2609.15910 Touch2Trace on Project Page: https://grmmtt.github.io/corl26-touch2trace/ Touch2Trace on ArXiV: https://arxiv.org/abs/2609.15921 Transcript Introduction Michael Cho: Hello, everyone. Welcome to another episode of RoboPapers. Today we’re really happy to have the crew from Analog Devices, the Dexterous AI Group: Tao Yu, Tong Jian, Saleh, and Matteo. They’re coauthors of two papers that they’ll walk us through. One is Touch2Trace: Tactile-Driven Imitation Learning for Dexterous Cable Tracing, and the other is SlipSense: Multimodal Tactile Learning for Low-Latency and Generalized Slip Detection. Thank you so much for coming, guys. Mayb
Reinforcement learning is often seen as key to achieving scalable, highly reliable real-world robot performance, yet it requires good reward functions that can supervise the wide range of tasks that need to be performed. The obvious thing to try would be a large vision-language model (VLM), but these models often fail under common real-world conditions like partial observability and occlusion. As a solution Philip Schroeder and Thomas Weng propose SOLE-R1, a video-language reasoning model trained to serve as a reward signal for online RL. Given real video, SOLE-R1 will reason over input video frames, generates reasoning traces, and predicts a reward signal. This allows online RL from random initialization on new manipulation tasks. To learn more, watch Episode 111 of RoboPapers, with Michael Cho and Jiafei Duan! Abstract Vision-language models (VLMs) have shown impressive capabilities across diverse tasks, motivating efforts to leverage these models to supervise robot learning. However, when used as evaluators in reinforcement learning (RL), today's strongest models often fail under partial observability and distribution shift, enabling policies to exploit perceptual errors rather than solve the task. We introduce SOLE-R1 (Self-Observing LEarner), a video-language reasoning model explicitly designed to serve as the sole reward signal for online RL. Given only raw video observations and a natural-language goal, SOLE-R1 performs per-timestep spatiotemporal chain-of-thought (CoT) reasoning and produces dense estimates of task progress that can be used directly as rewards. To train SOLE-R1, we develop a large-scale video trajectory and reasoning synthesis pipeline that generates temporally grounded CoT traces aligned with continuous progress supervision. This data is combined with foundational spatial and multi-frame temporal reasoning, and used to train the model with a hybrid framework that couples supervised fine-tuning with RL from verifiable rewards. Across four different simulation environments and a real-robot setting, SOLE-R1 enables zero-shot online RL from random initialization: robots learn previously unseen manipulation tasks without ground-truth rewards, success indicators, demonstrations, or task-specific tuning. SOLE-R1 succeeds on 24 unseen tasks and substantially outperforms strong vision-language rewarders, including Robometer, RoboReward, ReWiND, GPT-5, and Gemini-3-Pro, while exhibiting markedly greater robustness to reward hacking. We release all models, data, code, and demos at the anonymous page: this https URL Learn More Project page: https://philip-mit.github.io/sole-r1/ ArXiV: https://arxiv.org/abs/2603.28730 Transcript Transcripts are automatically generated by Riverside and are lightly edited by ChatGPT. Michael Cho [00:06] Hello guys, welcome to another episode of RoboPapers. Today we are really happy to have Philip Schroeder and Thomas Weng, two of the co-authors of this paper, SOLE-R1: Video-Language Reasoning as the Sole Reward for On-Robot Reinforcement Learning. Thank you so much for making the time, guys. Maybe as a start, Philip, you can share a bit about yourself and then Thomas. Philip Schroeder [00:27] Yeah, so my name is Philip. I’m a final-year PhD student at MIT in computer science. My work has mainly focused on improving the reasoning capabilities of language and vision-language models in challenging embodied settings. And over the last couple years, my work has mainly focused on reward reasoning over video, which is what we do for this paper. Thomas Weng [00:55] Cool. I’m Thomas. I’m a research scientist at RAInstitute. And more recently my work has been focusing on the pieces required to achieve like post-training and deployment for robotics. So reward modeling is one piece of that. On-robot reinforcement learning where you have data from deployment is also a piece of that. But that’s what I do. Michael Cho [01:22] All right, great to have both of you on. Yeah, maybe Philip you can tak
In many fields, automated research has become increasingly plausible with the improving capabilities of frontier language models, and the field robotics may be no exception. In ENPIRE, Wenli Xiao, Jia Xie, and Tonghe Zhang build a harness framework that captures physical feedback and allows for automated robotics research, minimizing human effort while training policies that achieve a 99% success rate on challenging, dexterous tasks like organizing a pin box or fastening a zip tie. Learn more in Episode 110 of RoboPapers, with Michael Cho, Chris Paxton, and Ruijie He! Abstract Achieving dexterous robotic manipulation in the real world heavily relies on human supervision and algorithm engineering, which becomes a central bottleneck in the pursuit of general physical intelligence. Although emerging coding agents can generate code to automate algorithm search, their successes remain largely confined in digital environments. We conjecture that the missing abstraction to automate robotics research is a repeatable feedback loop for real-world policy improvement: reset the scene, execute a policy, verify the outcome, and refine the next iteration. To bridge this gap, we introduce ENPIRE, a harness framework for coding agents that instantiates this physical feedback routine with four core modules: an Environment module (EN) for automatic reset and verification, a Policy Improvement module (PI) that launches policy refinement, a Rollout module (R) to evaluate policies with one or multiple physical robots operating in parallel, and an Evolution module (E) in which coding agents analyze logs, consult literature, improve training infrastructure and algorithm code to address failure modes. This closed-loop system transforms real-world manipulation learning into a controllable optimization procedure, minimizing human effort while allowing fair ablations across training recipe and agent variants. Powered by ENPIRE, frontier coding agents can autonomously train a policy to achieve a 99% success rate on challenging, dexterous manipulation tasks, such as organizing a pin box, fastening a zip tie, and tool use, a process that further accelerates when we dispatch an agent team on a robot fleet. Our results suggest a practical and scalable path toward deploying coding agents to autonomously advancing robotics in the physical world. Learn More Project Page: ArXiV: https://arxiv.org/abs/2606.19980 Transcript Michael Cho (00:06.088)Hello everyone, welcome to another episode of RoboPapers. Today we’re super happy to have Wenli, Jia Xie and Tonghe. They are three of the co-authors of this paper called ENPIRE: Agentic Robot Policy Self-Improvement in the Real World. Thank you so much for coming to the pod. Maybe as a start three of you can share a little bit about yourself first and then you guys can go through the paper. Wenli Xiao (00:30.821)Okay, I’ll get started. Sorry. So my name is Wenli. I’m a final-year PhD at CMU, advised by Professor Guanya Shi. Previously I did two-year internship at NVIDIA GEAR Lab with Jim Fan and Yuke Zhu. And later I joined Physical Intelligence as intern. Yeah, I did a bunch of VLA training, post-training, real-world RL and agentic stuff and now I’m mainly doing pretraining, studying the science of training and data. Yeah. Michael Cho (01:14.488)Yeah, I think just yeah. Jia Xie (01:16.463)Okay, sure. Yeah. Hi everyone. My name is Jia and I’m a master’s student at CMU, advised by Guanya. Of course we’re labmates. And I’ve interned at xDof as a research intern over the summer. And I’ve been working on some agent stuff and the extent stuff lately. So yeah, thanks for having us. Tonghe Zhang (01:43.739)Okay. Hi, my name is Tonghe. I’m also a master’s student at Professor Guanya Shi’s lab. I was doing internship at Amazon this summer and previously I’ve been working on reinforcement learning. And thank you guys for inviting us. Michael Cho (02:01.15)Alright, perfect. Great to have you guys. Yeah, Wenli, I know you have a sli
What happens if you connect a hundred AI-powered robots to the internet and just let people give them any instructions they want? Enigma founder Jonathan Jacobi joins us to talk about his experience deploying a fleet of robots on the internet, keeping them operational, and watching people truly test the generality and robustness of frontier robotics models. Watch Episode 109 of RoboPapers with Chris Paxton and Ruijie He today! Learn More Original Post on X Robots.online website (no longer active) Enigma.inc Transcript Transcript is automatically generated by Riverside and edited by ChatGPT. Chris Paxton (00:05) Hey everyone, and welcome to another episode of RoboPapers. Today we’re super excited to have Jonathan Jacobi from Enigma here to tell us about their work in foundation models and about putting a hundred robots online. So, could you introduce yourself? Jonathan Jacobi (00:19) Yeah. Thank you, Chris. Thank you, RJ. It’s a pleasure to be on RoboPapers. Big fans. A bit about Enigma: we’re a company working on foundation models for robotics, as Chris said, as well as building innovative new interfaces for robotics. How should we interact with robots? How should the interface feel to users? We’re also interested in the back-and-forth between interface work and research on the models. About a month and a half ago, we put a hundred real robots online. You could actually interact with them over the internet. They were running our models through a variety of interfaces, and you could simply ask them to do different things. So we’re here to talk about the models, the interfaces, and the experience we’ve built. Chris Paxton (01:21) Awesome. So what did people do with all these robots that you put online? Jonathan Jacobi (01:25) The basic concept was that we wanted people to feel robotics today. We all see a lot of demos. We see a lot of cherry-picked demos—some of them are cherry-picked, some of them aren’t—and we wanted people to feel that something is here right now. One of the problems is that not enough people have robots in their houses. So how do we make enough people feel that robotics is happening now, while we still have this challenge of not having enough robots deployed in the world for the average person to use? We thought quite hard about how to make that possible. One of the best ideas we came up with was: maybe we can set up a huge warehouse with a lot of robots, put our models on those robots, and let anyone online interact with them. We also got a very cool domain: robots.online. We tried to create enough different setups that one robot could be in one scenario, another robot could be in another scenario, and people would have enough flexibility to do random things that they could really feel the AI work we’ve done. They could ask the robot to do things themselves, without the logistical challenge of putting a hundred humanoids all over the world. That’s why we used tabletop robots—arms that we customized and designed ourselves, building on public open-source designs—and let people interact with them freely. Some robots were artists. People could paint. People could do pantomime. People did very unexpected things. Someone asked one of the robots holding a paintbrush to pretend that the brush was a broom. Jonathan Jacobi (03:50) So it actually swept the floor with the paintbrush. We have some very cool examples of that that we can walk through later. The concept was that, at Enigma, we focus on generalization—text, video, audio—and we have very different interfaces and input methods for our models. We wanted people to be able to experience that generalization directly and see that it’s working today. Some of the robots were artists. Some were doing chemistry experiments. We had one that could defuse a bomb. We had one that could do lightsaber-esque fighting. Not actual lightsabers. Don’t sue me. But basically, we let people do different things with different robots online. RJ He (05:01) Okay, cool. Can you describe
Achieving human-like dexterity is the next frontier for robotics, and yet dexterity data is often subtly hard to scale. Real-world dexterity data, including things like finger-pose estimates, is often slightly off, making it physically invalid and hard to execute on real hardware and hard to learn from. DO AS I DO is an algorithm for reconstructing and retargeting monocular RGB videos to robot hands, outperforming the state of the art and even working from generated videos. Bhawna Paliwal, Haritheja Etukuru, Will Liang, and Mahi Shafiullah join us to tell us more. Watch Episode 107 of RoboPapers, with Chris Paxton and Jiafei Duan, now! Abstract How can we scalably generate data for robotic manipulation, especially on human-like platforms such as dexterous multi-fingered hands? Learning from human videos has recently emerged as a likely answer to this question. However, difficulties in estimating hand-object interaction and crossing the human-to-robot embodiment gap have hindered the adoption of abundant monocular RGB-only human videos as the primary source of robot manipulation data. In this work, we present DO AS I DO, an algorithm to reconstruct and retarget monocular RGB human videos to multi-fingered dexterous robotic hands. DO AS I DO reconstructs hand-object interactions from various egocentric and exocentric in-the-wild video sources. The algorithm then retargets these hand-object interaction estimates into a sequence of actions executable in the real world, yielding robot-complete manipulation data from disparate human videos. Overall, DO AS I DO outperforms previous state of the art in estimating hand-object interactions and extracting dexterous manipulation trajectories from RGB videos, as we show in experiments on datasets with ground truths and on a dataset of video clips collected online. Our experiments enable us to propose an efficacy playbook for practitioners collecting human data for manipulation. Learn More Project Page: https://do-as-i-do.com/ ArXiV: https://arxiv.org/abs/2606.19333 Github: https://github.com/malik-group/do-as-i-do Original thread on X Transcript Lightly edited for transcription errors, names, technical terminology, punctuation, and readability. Conversational phrasing is otherwise preserved. Editing done automatically with ChatGPT. Chris Paxton (00:05.479) Hey everyone, welcome to another episode of RoboPapers. Today we have some great work from Bhawna, Haritheja, Will, and Mahi: “Do as I Do: Dexterous Manipulation Data from Everyday Human Videos.” Welcome to the show, guys. Could you introduce yourselves quickly before we get started? Bhawna Paliwal (00:24.977) Yeah, hi, I’m Bhawna. I’m a PhD student here at UC Berkeley, advised by Jitendra Malik. Haritheja Etukuru (00:32.692) I’m Haritheja. I’m a second-year PhD student at Berkeley, also advised by Jitendra. William Liang (00:37.929) I’m Will. I’m also a second-year PhD student, advised by Jitendra and Pieter. Mahi Shafiullah (00:42.606) Hey everybody, I’m Mahi. I’m a postdoc currently at Berkeley, and I’m also spending some time at Amazon FAR right now. Chris Paxton (00:50.983) Awesome. Well, yeah, would you guys like to tell us a little bit about this? What have you guys been working on? Bhawna Paliwal (01:01.499) So I’ll just start with a little introduction to the paper. So mainly the motivation behind this work was how can we utilize the massive video data available on internet for training robots. And most of the prior work has been which which has involved learning from human videos, has involved co-training with robot data, which which eventually involves having like hundreds or thousands of hours of teleop data combined with, let’s say, twenty thousand hours of video. So our objective in this paper is to understand. Whether there is some intermediate representation, let’s say in in 3D, where we can extract the right kind of information from human videos and make learning from human videos much more efficient. So essentially ever