Source-linked AI summary
THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulation
Wilbert Pumacay, Ishika Singh, Jiafei Duan, Ranjay Krishna, Jesse Thomason, Dieter Fox
TL;DR
Robot policies are often evaluated in environments resembling their training setups, leaving environmental generalization insufficiently stress-tested. THE COLOSSEUM introduces a 20-task benchmark spanning 14 perturbation dimensions and evaluates behavior-cloning models across simulation and real-world counterparts. The experiments identify the perturbations most associated with reduced success and report alignment between simulated and real-world evaluations.
Problem
Robot manipulation studies often evaluate policies in environments closely resembling training setups, despite the need for operation across varied real-world conditions.
Method
THE COLOSSEUM extends RLBench with 20 manipulation tasks and 14 environmental perturbation dimensions, alongside reproducible real-world evaluations.
Results
Across evaluated models, distractors, color, and lighting perturbations most significantly reduce task success, while simulation results correlate with corresponding real-world outcomes.
Takeaways & Limitations
THE COLOSSEUM provides a unified platform for developing, evaluating, and comparing robotic manipulation methods for robustness and generalization.
Takeaways & Limitations
Real-world experiments have limited pose and execution replication and only two alternate variations per perturbed factor, so findings primarily compare simulation and real-world performance distributions.
Abstract
from arXiv · showhide
To realize effective large-scale, real-world robotic applications, we must evaluate how well our robot policies adapt to changes in environmental conditions. Unfortunately, a majority of studies evaluate robot performance in environments closely resembling or even identical to the training setup. We present THE COLOSSEUM, a novel simulation benchmark, with 20 diverse manipulation tasks, that enables systematical evaluation of models across 14 axes of environmental perturbations. These perturbations include changes in color, texture, and size of objects, table-tops, and backgrounds; we also vary lighting, distractors, physical properties perturbations and camera pose. Using THE COLOSSEUM, we compare 5 state-of-the-art manipulation models to reveal that their success rate degrades between 30-50% across these perturbation factors. When multiple perturbations are applied in unison, the success rate degrades $\geq$75%. We identify that changing the number of distractor objects, target object color, or lighting conditions are the perturbations that reduce model performance the most. To verify the ecological validity of our results, we show that our results in simulation are correlated ($\bar{R}^2 = 0.614$) to similar perturbations in real-world experiments. We open source code for others to use THE COLOSSEUM, and also release code to 3D print the objects used to replicate the real-world perturbations. Ultimately, we hope that THE COLOSSEUM will serve as a benchmark to identify modeling decisions that systematically improve generalization for manipulation. See https://robot-colosseum.github.io/ for more details.
I. INTRODUCTION
THE COLOSSEUM addresses the limited stress testing of robot policies by systematically evaluating manipulation generalization across environmental perturbations. It compares policies across diverse tasks and perturbations and connects simulation findings to real-world evaluation.
- Most studies evaluate robot performance in environments closely resembling or identical to training setups, limiting systematic assessment of environmental generalization.
- THE COLOSSEUM includes a challenge with standardized training data, fixed perturbation evaluations, and leaderboard ranking based on performance change.
- THE COLOSSEUM evaluates 20 RLBench manipulation tasks across 14 environmental perturbation dimensions, including appearance, distractors, camera pose, and physical properties.
- The benchmark provides reproducible real-world extensions using 3D-printed components and supports comparison with other generalization benchmarks.
- Four state-of-the-art behavior-cloning models show that distractors, color, and lighting perturbations most significantly affect task success, while object-size changes have less impact.
B. Robotic Manipulation Methods
THE COLOSSEUM extends RLBench into a diverse benchmark for testing robotic manipulation policies under systematic distribution shifts. Its task suite spans varied horizons and its perturbations preserve task semantics while changing test-time inputs.
- III. THE COLOSSEUM: THE COLOSSEUM extends RLBench with 20 diverse robotic manipulation tasks and 14 perturbation factors for evaluating generalization.
- III. THE COLOSSEUM: Its perturbations implement covariate shift by changing test-time input distributions while keeping the underlying task and action conditional relationship unchanged.
- III. THE COLOSSEUM: The benchmark proposes 20,371 unique task instances and supports combining multiple perturbation factors within the same scene.
- III. THE COLOSSEUM: The benchmark covers tasks ranging from fewer than 100 controller steps to more than 1000, organized into three difficulty tiers based on task horizon.
B. Perturbation Factors
THE COLOSSEUM organizes environmental changes into object, receiver, background, and physical perturbations. These factors are implemented in simulation and replicated for selected tasks in the real world, with some task-specific applicability limits.
- Manipulation object perturbation: Manipulation-object perturbations change the color, texture, or size of objects directly manipulated by the robot.
- Receiver object perturbation: Receiver-object perturbations change the color, texture, or size of task-relevant objects that the robot does not directly manipulate.
- Background perturbation: Background perturbations vary lighting, table appearance, distractor objects, wall texture, and camera pose.
- Physical perturbation: Physical perturbations alter object friction or mass to test whether manipulation adapts to changed physical requirements.
- Combined perturbations: Multiple perturbation factors can be applied simultaneously at test time, enabling evaluation of compounded environmental changes.
- Simulation implementation: Simulation implementations sample colors, textures, scaling, lighting, camera poses, and physical properties through configurable scene controls.
- Applicability limits: Some manipulation- and receiver-object perturbations cannot apply when tasks lack receiver objects or contain unsupported compound shapes.
- Real-world extension: Real-world extensions replicate four RLBench tasks with 3D-printed objects and alternate sizes, colors, and textures using a Franka Panda robot arm.
E. THE COLOSSEUM Challenge
THE COLOSSEUM Challenge standardizes training and evaluation to test behavior-cloning generalization across environmental perturbations, with a reproducible real-world counterpart. It compares diverse baseline approaches under controlled test conditions.
- Participants train behavior-cloning models on 100 demonstrations per task without THE COLOSSEUM perturbations, then evaluate fixed episodes for each perturbation factor.
- The problem formulation predicts robot actions from observations and language instructions, using trajectories containing observations, actions, proprioception, and instructions.
- Keypoint-based action prediction replaces direct continuous 7-DoF action prediction, with keypoints identified using arm velocity and gripper-state heuristics.
- The benchmark compares five state-of-the-art baselines spanning zero-shot, 2D learning, and 3D learning approaches.
2) 3D learning models:
The experiments apply THE COLOSSEUM protocol to several manipulation baselines, including 3D and zero-shot models, under controlled perturbation tests. Evaluation reports task-averaged success rates and examines simulation–real-world alignment.
- 3D learning models: RVT re-renders RGB-D observations from virtual camera views and predicts translation heatmaps, discretized rotations, and gripper states.
- Zero-shot manipulation model using Large Pretrained World Models: VoxPoser extracts affordances and constraints with language models, composes 3D value maps, and is evaluated zero-shot without training.
- Training and Evaluation Protocol: Models are trained with 100 demonstrations per task without THE COLOSSEUM perturbations, while retaining default RLBench language and target variations.
- Training and Evaluation Protocol: Evaluation fixes task variations, preserves object-pose variation across episodes, and uses the final checkpoint for each trained baseline.
- Training and Evaluation Protocol: The benchmark contains 235 test sets with 25 episodes per set, and success requires fully completing the task.
- Results: Results are reported as task-averaged success rates, alongside an all-perturbations training ablation and simulation–real-world alignment analysis.
A. Performance of different baselines on THE COLOSSEUM
Across THE COLOSSEUM, environmental perturbations substantially reduce baseline manipulation performance, with combined changes producing the largest degradation. The most disruptive factors differ somewhat by model representation, while 3D baselines are generally more robust than 2D baselines.
- Overall performance: ≥75% decrease in performance occurs when all perturbations are applied together, substantially affecting every model.This setting combines perturbation factors in the same scene and produces the strongest reported degradation.
- Perturbation effects: Color-related changes, distractors, and lighting are among the most damaging perturbations for baseline models.For 2D models, object and light color, texture, and camera pose are especially influential; for 3D models, colors and distractors dominate.
- 2D baselines: 2D models are sensitive to color, texture, and camera-pose changes because RGB inputs shift and camera views differ from training conditions.These input-space changes affect the output space for end-to-end RGB policies.
- Zero-shot model: VoxPoser maintains robust performance across variants on tasks where it is predisposed to succeed, with a 3.21% difference on slide_block_to_target.The difference compares the No Perturbation scenario with the average of all perturbations relative to No Perturbation performance.
- 3D baselines: 3D models are relatively robust to camera-pose changes but remain vulnerable to color changes and distractors that move scenes out of distribution.Voxel-grid or novel-view preprocessing reduces direct dependence on captured viewpoints, whereas distractors remain challenging without real-world pretraining.
- Overall performance: 3D baselines generally perform better and are more robust to environmental perturbations than 2D baselines.The comparison is reported across overall task performance and perturbation evaluations.
C. Real-world alignment analysis for THE COLOSSEUM
Real-world experiments broadly align with simulation-based evaluation, while exposing substantial variation across specific perturbations and practical replication limits. The benchmark also remains focused on a limited set of baseline models.
- Under five perturbation factors, performance discrepancies between simulation and real-world models remained below 5%.The factors were MO_Texture, Light_Color, Table_Color, Table_Texture, and Distractor.
- 0.74≤R²≤0.94 was observed for seven perturbation factors, indicating strong alignment between simulated and real-world evaluation for at least half the factors.The strongly correlated factors included Background_Texture, Distractor, Table_Color, Light_Color, RO_Color, RO_Texture, and RO_Size.
- 82.6% was the real-world success-rate drop for MO_Color, while MO_Size increased performance by 4.34%.Light_Color altered object appearance, and MO_Color often impeded accurate 6D-pose prediction for grasping.
- The real-world study compared three realistic scenes with corresponding perturbation combinations derived from THE COLOSSEUM.The scenes were a workbench, dining table, and study room tabletop, evaluated with a multitask PerAct trained without perturbations.
- The leaderboard currently includes four baseline methods, all of them behavior-cloning methods.Future plans include adding RL, diffusion, 3D feature-field, large-scale pretraining, action-tokenization, and action-chunking methods.
- Real-world replication is constrained by difficulty matching task pose, orientation, and execution, with each factor limited to two alternate variations.The authors characterize the findings primarily as a comparative performance distribution between simulation and real-world scenarios.
- THE COLOSSEUM provides open-source resources and 3D-printed assets for reproducible simulation and real-world benchmarking.The benchmark is intended to support quantitative evaluation before scaling through a unified leaderboard.
THE COLOSSEUM APPENDIX
The appendix documents the environmental variation observed in real-world datasets and enumerates the benchmark’s selected tasks and their success conditions. These tasks span diverse manipulation primitives and horizons.
- DROID samples contain changes in object color, texture, and size, lighting, table appearance, distractors, camera pose, and background across scenes.The authors also infer that cup mass varies, although it is not explicitly reported.
- THE COLOSSEUM draws its 20-task suite from RLBench and documents each task with an associated success condition.The listed tasks include opening drawers, sliding blocks, basketball placement, grilling meat, closing boxes and laptops, and emptying a dishwasher.
- The task suite includes object manipulation, tool use, placement, stacking, opening, closing, and movement tasks.Examples include reaching and dragging, getting ice from a fridge, hockey, putting money in a safe, placing wine, moving a hanger, wiping a desk, and straightening a rope.
- Several tasks specify discrete placement or insertion goals, such as placing a square on a colored spoke or stacking cups on a designated cup.Other examples include setting up chess, scooping with a spatula, and turning an oven knob on.
XI. SIMULATION DETAILS
The simulation benchmark specifies perturbation applicability and parameterization per task. It distinguishes object-specific factors from scene-wide factors and physical-property changes.
- Tables III document each task’s manipulation and receiver objects, perturbation sampling mode, and set size or continuous range.A dash indicates that a factor is inapplicable because the receiver object is absent or unsupported by the simulator.
- Six perturbation factors apply to all tasks, while other factors apply only where the task and simulator support them.The main text specifies the parameters for the six universal factors, and task configuration files expose factor parameters for modification.
A. Training details and Detailed results
The paper reports the training and evaluation setup for its baselines and real-robot experiments, including reproducible hardware, object variations, demonstrations, and per-factor success-rate reporting.
- 4 NVIDIA RTX A6000 GPUs were used for 2–3 days per model to run a full THE COLOSSEUM evaluation.Baseline training used 1–4 GPUs for 1–6 days.
- Detailed per-task success rates are reported for every perturbation factor across all baseline models.
- The real-robot setup used a Franka Panda with a parallel gripper and a tripod-mounted Kinect-2 RGB-D camera.The camera captured 512 × 424 RGB-D images at 30Hz, with calibration supported by an ARUCO marker.
- Real-world object sizes varied by ±0.2 times the original scale, colors used red and blue filaments, and lighting used a color-changing spotlight.
- Real-world task success conditions included pushing a colored block to a target, placing a pawn, inserting a peg, and lifting a scooped cube.
- Demonstrations were recorded with an HTC Vive controller providing 6 degrees of freedom tracking.Recorded keypoints were realized with Franka ROS and MoveIt motion planning.
D. Training and Evaluation details
Real-robot training used one NVIDIA TITAN RTX GPU for one day, with keypoint monitoring for safety during evaluation.
- 1 NVIDIA TITAN RTX GPU and 1 day were used for real-robot training.
- Predicted keypoints were monitored to verify the safety of the next action.During evaluation, the robot continued executing predicted keypoints unless manually halted by the operator.
E. Ablation study
The ablation study examined compound perturbation effects and their correspondence to real-world scenarios using analogous benchmark-derived conditions. Supporting materials document the configuration, task-factor assignments, real-world setup, and results tables.
- Three perturbation combinations from real-world experiments were selected for the slide_block_to_target ablation.The study constructed analogous workbench, dining-table, and study-room-table scenarios subjected to the same derived perturbations.
- THE COLOSSEUM configurations allow compatible perturbation factors to be applied simultaneously within one task instance.A YAML configuration controls the application of perturbation factors for a task.
- The appendix includes a real-world ablation table and task-level tables specifying perturbation factors and parameters.
- Results tables report PerAct, R3M, MVP, RVT, and Voxposer performance under various perturbations.
- The real-robot setup used a Kinect-2 and Franka Panda, with 3D print-outs provided for all real-world task assets.