RobotUse: Allocating Computation, Context, and Decisions

Published October 7, 2026 · Category: Robotics

Overview

arXiv:2610.04929v2 Announce Type: replace Abstract: Robot agents must connect their intended actions to observed outcomes while retaining the context needed to revise their choices over repeated attempts. Existing interfaces often leave these choices inside predefined tools or require agents to manage detailed execution code and its growing history. We introduce RobotUse, a robot agent harness that organizes computation, context, and decisions around specifying and revising physical actions. Agents visually select targets and poses, while the backend handles geometry, motion planning, and control. Subagents retain detailed interactions within each subgoal and return the information needed for subsequent decisions. Continual harnessing lets agents learn from execution by updating a persistent playbook. On RoboLab, RobotUse achieves 45% task success, outperforming CaP-X by 6.7 percentage points while maintaining compact decision contexts and reducing reliance on predefined action abstractions. Furthermore, we show that RobotUse learns from real-world execution despite imperfect feedback and transfers what it learns to subsequent tasks. Project page is available at https://robotuse-team.github.io/.

Source

Originally published at arxiv.org.

Related Articles

• Does a Learned Corrector Beat a Simple Retreat? Evidence from a Frozen VLA
• Sim-to-Real Transfer of Vision-Language Navigation in Continuous Environments Using an Ackermann-Steered Mobile Robot
• MRPilot: Supervising and Intervening LLM-Based Multi-Robot Teams through Mixed Reality