Learning to Act While Waiting: RL Finetuning of Generalist Robot Policies Under Inference Latency

Published August 26, 2026 · Category: Robotics

Overview

arXiv:2608.23831v1 Announce Type: new Abstract: While reinforcement learning (RL) allows generalist robot policies to continually improve during deployment, the large model size of modern generalist policies, such as VLAs, poses a fundamental obstacle to effective RL improvement. In particular, their severe inference latency---which can lead to pauses or jerky movements---can alter the effective environment dynamics and, if not correctly accounted for, break the Markov assumption that RL relies on, causing standard RL algorithms to fail completely. In this work, we introduce a latency-aware framework, Asynchronous RL with Intermediate Information (ARLI), that enables RL-based improvement of generalist policies under inference delays. Our framework builds on asynchronous inference approaches, which interleave action generation with execution to hide latency, and addresses its incompatibility with RL by providing a low-latency RL policy design that maximizes reactivity within the inference window through two contributions: state augmentations that restore near-Markovian structure by incorporating committed actions and a mid-inference observation. We evaluate our approach across simulated and real-world manipulation tasks, and find that it enables effective finetuning under inference delays where standard RL fails entirely, even matching or exceeding the performance of standard RL in idealized no-latency settings.

Source

Originally published at arxiv.org.

Related Articles

• Vision-Language-Action Safety: Threats, Challenges, Evaluations, and Mechanisms
• A Durable Vision-Based Tactile Fingertip for Robotic Manipulation
• CARO: Contact-Agnostic Residual Observation for Zero-Shot Robust Quadruped Locomotion