Veo-Act: Enhancing VLA Policies with Frontier Video Models

Published September 17, 2026 · Category: Robotics

Overview

arXiv:2604.04502v2 Announce Type: replace Abstract: Video generation models can produce coherent vi- sual sequences depicting object motion and interactions. We in- vestigate how frontier video generation models can complement vision-language-action policies to enhance generalizable robotic manipulation. VLA policies have become a dominant paradigm for robot learning, but their action-oriented adaptation of pretrained VLMs can weaken semantic generalization, limiting robustness in ambiguous or out-of-distribution manipulation scenarios. We use video models as visual planners, motivated by their potential to generalize across complex scenes and their priors over hand motion. However, manipulation methods based on video models often lack the precision and temporal responsiveness needed for low-level dexterous interaction. To address this gap, we present Veo-Act, a hierarchical framework with Veo-3.1 as a high-level motion planner and a VLA policy as the low-level executor. A multi-head inverse dynamics model converts generated frame pairs into actions and learns an interaction gate to trigger the handoff to reactive VLA control. Experiments in simulation and on a real robot show improved instruction following and overall task success over the baseline VLA in novel and semantically complex manipulation settings, supporting the complementary roles of video planning and reactive interaction.

Source

Originally published at arxiv.org.

Related Articles

• Human-Centric Grasp State Assessment: Toward Transferring Subjective Evaluation to Robots
• Flexible-body Modeling, Kinematic Identification, and Assembly Accuracy of Overconstrained Spatial Linkages
• Vision-Language Grounded Task-Context-Aware Imitation Learning for Robotic Disassembly