Colosseum V2: Benchmarking Generalization for Vision-Language-Action Models

Published September 15, 2026 · Category: Robotics

Overview

arXiv:2605.27759v2 Announce Type: replace Abstract: Vision-Language-Action (VLA) models demonstrate promising generalization in robotic manipulation, driven by advances in large-scale vision and language pre-training. This progress can be misleading. Despite the zero-shot perception and language capabilities of VLAs, their overall task performance often degrades under distribution shifts, revealing gaps in how these systems translate high-level understanding into robust behavior. To systematically study this gap, we introduce Colosseum V2, a large-scale simulation benchmark for evaluating VLA generalization in robot learning across diverse conditions. The benchmark comprises 28 tasks spanning 13 task categories and two robot morphologies, covering a wide range of manipulation primitives and long-horizon behaviors. Built on the ManiSkill simulator, Colosseum V2 enables fast, GPU-parallelized evaluation and supports both in-domain and out-of-domain testing at scale. We evaluate state-of-the-art methods, including Action Chunking Transformers (ACT) and Pi0.5, and reveal limitations in both base performance and generalization. We demonstrate strong correlations between simulation and real-world metrics that support the ecological validity of the benchmark. By standardizing tasks, metrics, and evaluation protocols within a unified benchmark, Colosseum V2 enables reproducible and fair comparisons, reduced evaluation overhead, and accelerated progress toward general-purpose robot policies.

Source

Originally published at arxiv.org.

Related Articles

• Conflict-Predictive Variable Horizons in Multi-Drone Distributed Model Predictive Control
• IMM-based Multiple Object Tracking using a State Prediction Neural Network
• Constraint-Grounded Reinforcement Learning for Variable Impedance Control in Contact-Rich Robotic Insertion