Decoupled Object-Centric Video Understanding for Generating Robotic Manipulation Commands

Published June 16, 2026 · Category: Robotics

Overview

arXiv:2606.16470v1 Announce Type: cross Abstract: Translating video demonstrations into executable robot commands remains challenging because existing methods often fail to identify which objects are functionally involved in the demonstrated action. As a result, they may generate commands that are linguistically plausible but operationally ambiguous. We propose an object-centric video understanding framework that decouples action recognition from object identification to generate precise, grammar-free manipulation commands. Our approach integrates Temporal Shift Modules (TSM) for efficient spatio-temporal action classification with a novel \textbf{Object Selection} algorithm that identifies task-relevant objects through trajectory-based role classification, blur detection, and overlap minimization. The selected objects are then processed by Vision-Language Models (VLMs) for robust category recognition and zero-shot generalization. Evaluated on a modified Something-Something V2 dataset, our method achieves 86.79\% action classification accuracy and BLEU-4 scores of 0.337 on standard objects and 0.261 on novel objects. These results improve over the strongest task-specific baseline by 80.2\% and 143.9\%, respectively. Larger gains are observed in METEOR and CIDEr, reaching 157.9\% and 171.7\% on novel objects. Across all semantic metrics, our approach consistently outperforms task-specific methods and remains competitive with, or surpasses, large general-purpose VLMs while retaining a modular, object-centric design.

Source

Originally published at arxiv.org.

Related Articles

• PSYONIC partners with ABB Robotics to apply human touch to robot dexterity
• A Formal Resilience Framework for Cyber-Physical Embodied Systems under Device-Level Cyberattacks
• PROSE: Training-Free Egocentric Scene Registration with Vision-Language Models