InstructMove: A Text-Indispensable Benchmark for Instruction-Following Manipulation

Published August 25, 2026 · Category: Robotics

Overview

arXiv:2608.22990v1 Announce Type: new Abstract: Vision-language-action (VLA) models have made general-purpose robot manipulation increasingly plausible by conditioning robot actions on natural-language instructions. A key test of such generality is whether policies actually follow language instructions. Yet many manipulation benchmarks leave this ability underdetermined: the intended object or destination is often visually salient or uniquely feasible, allowing policies to succeed without grounding the instruction. We argue that instruction-following evaluation should be text-indispensable: multiple actions should be visually and physically plausible, while only one should be consistent with the language instruction. We introduce InstructMove, a text-indispensable benchmark for instruction-following manipulation. InstructMove instantiates this principle in pick-and-place scenes with semantic distractors, decomposing instruction following into category identification, attribute discrimination, spatial reasoning, and compositional pick-and-place. InstructMove supports a train-eval protocol with InstructMove training data and held-out evaluation tasks, with additional diagnostics for language dependence. Experiments with representative VLA policies show that InstructMove provides a controlled testbed for diagnosing visual shortcuts and that InstructMove simulation data can improve real-world instruction-following manipulation performance. Code: https://github.com/HorizonRobotics/RoboOrchardSim

Source

Originally published at arxiv.org.

Related Articles

• RiskWorld: Object-Centric Latent World Modeling for Autonomous Driving Risk Identification
• Betting for Sim-to-Real Performance Certificates
• In-Situ Reconstruction of the International Space Station Using 3D Gaussian Splatting and Astrobee