PaperlayerView exact revision on arXiv

Source-linked AI summary

RoboVerse: Towards a Unified Platform, Dataset and Benchmark for Scalable and Generalizable Robot Learning

Haoran Geng, Feishi Wang, Songlin Wei, Yuyang Li, Bangjun Wang, Boshi An, Charlie Tianyue Cheng, Haozhe Lou, Peihao Li, Yen-Jen Wang, Yutong Liang, Dylan Goetting, Chaoyi Xu, Haozhe Chen, Yuxi Qian, Yiran Geng, Jiageng Mao, Weikang Wan, Mingtong Zhang, Jiangran Lyu, Siheng Zhao, Jiazhao Zhang, Jialiang Zhang, Chengyang Zhao, Haoran Lu, Yufei Ding, Ran Gong, Yuran Wang, Yuxuan Kuang, Ruihai Wu, Baoxiong Jia, Carlo Sferrazza, Hao Dong, Siyuan Huang, Yue Wang, Jitendra Malik, Pieter Abbeel

arXiv:2504.18904v1cs.RO

TL;DR

Robotics lacks scalable, diverse datasets and standardized evaluation, while real-world collection and benchmarking are costly and difficult to reproduce. RoboVerse addresses this gap with a unified multi-simulator platform, synthetic dataset, and benchmarks, and experiments show support for policy learning, world-model learning, and sim-to-real transfer. The framework’s scope is limited by incomplete non-rigid-object integration, resource-constrained foundation-model exploration, and potentially suboptimal baselines.

  • Problem

    Robotics lacks large-scale, high-quality, diverse data and widely recognized, reproducible evaluation protocols, while real-world data collection and benchmarking are costly and difficult.

  • Method

    RoboVerse unifies simulators through MetaSim and provides a standardized synthetic dataset, unified benchmarks, and scalable trajectory-generation workflows.

  • Results

    RoboVerse provides ∼500k unique trajectories across 276 task categories and ∼5.5k assets, over 50 million state transitions, and experiments spanning imitation learning, reinforcement learning, world models, and sim-to-real transfer.

  • Takeaways & Limitations

    RoboVerse offers a unified basis for scalable robot-learning data construction and evaluation across simulators and generalization settings.

  • Takeaways & Limitations

    Non-rigid-object integration is not yet fully supported, foundation-model pretraining is outside scope because of resource constraints, and some baselines may remain suboptimal.

Abstract

from arXiv · show

Data scaling and standardized evaluation benchmarks have driven significant advances in natural language processing and computer vision. However, robotics faces unique challenges in scaling data and establishing evaluation protocols. Collecting real-world data is resource-intensive and inefficient, while benchmarking in real-world scenarios remains highly complex. Synthetic data and simulation offer promising alternatives, yet existing efforts often fall short in data quality, diversity, and benchmark standardization. To address these challenges, we introduce RoboVerse, a comprehensive framework comprising a simulation platform, a synthetic dataset, and unified benchmarks. Our simulation platform supports multiple simulators and robotic embodiments, enabling seamless transitions between different environments. The synthetic dataset, featuring high-fidelity physics and photorealistic rendering, is constructed through multiple approaches. Additionally, we propose unified benchmarks for imitation learning and reinforcement learning, enabling evaluation across different levels of generalization. At the core of the simulation platform is MetaSim, an infrastructure that abstracts diverse simulation environments into a universal interface. It restructures existing simulation environments into a simulator-agnostic configuration system, as well as an API aligning different simulator functionalities, such as launching simulation environments, loading assets with initial states, stepping the physics engine, etc. This abstraction ensures interoperability and extensibility. Comprehensive experiments demonstrate that RoboVerse enhances the performance of imitation learning, reinforcement learning, world model learning, and sim-to-real transfer. These results validate the reliability of our dataset and benchmarks, establishing RoboVerse as a robust solution for advancing robot learning.

I. INTRODUCTION

RoboVerse addresses robotics’ difficulty scaling high-quality, diverse data and establishing reproducible evaluation by unifying simulators, synthetic data generation, and benchmarks. Its dataset and experiments support broad robot-learning evaluation and transfer.

  • Motivation: Robotics lacks the scalable, diverse data and widely recognized evaluation protocols that helped advance NLP and computer vision.Real-world collection is costly and hardware-dependent, while standardized conditions are difficult to reproduce.
  • Motivation: Simulators offer efficient, reproducible data construction and evaluation, but differing platforms and expertise requirements have produced a fragmented ecosystem.Existing synthetic datasets can be difficult to reuse across simulators, limiting convenient large-scale data construction.
  • RoboVerse: RoboVerse unifies existing simulators under a standardized format and infrastructure while providing a large-scale synthetic dataset and unified benchmarks.MetaSim supplies a universal configuration system and simulator-agnostic interface for integrating tasks, assets, and trajectories with minimal adaptation.
  • RoboVerse: MetaSim supports scalable development through aligned APIs, cross-environment integration, data randomization, augmentation, and real-to-sim and generative workflows.These capabilities target efficient algorithm integration, testing, deployment, and task or data creation across diverse simulation environments.
  • Dataset: RoboVerse contains ∼500k unique trajectories across 276 task categories and ∼5.5k assets, plus over 50 million state transitions for policy learning.The dataset is described as large-scale, diverse, high-quality, and unified.
  • Evaluation: Experiments cover imitation learning, reinforcement learning, world model learning, sim-to-sim transfer, and sim-to-real transfer, demonstrating reliable policy learning and evaluation.The framework also supports data expansion through teleoperation, trajectory augmentation, domain randomization, and generative models.
  • Evaluation: RoboVerse enables seamless evaluation across multiple benchmarks and simulators, addressing reproducibility uncertainty from simulation accuracy, rendering style, and asset properties.The unified system reduces the need to learn each benchmark individually.

III. INFRASTRUCTURE: METASIM

MetaSim abstracts heterogeneous simulation environments through a universal configuration system, simulator-agnostic interface, and environment wrappers. These layers standardize scenario specification, simulator control, and learning-environment access.

  • Overview: MetaSim is RoboVerse’s core infrastructure, enabling large-scale dataset generation and unified benchmark construction above specific simulator implementations.It provides a high-level interface while allowing the platform to integrate multiple simulation environments.
  • Architecture: The three-layer architecture comprises a universal configuration system, simulator-agnostic interface, and user-friendly environment wrapper.Together, these layers align simulator backends and expose standardized learning environments.
  • Universal Configuration System: MetaConfig represents agents, objects, tasks, sensors, and physics parameters in a simulator-agnostic nested configuration interpreted by different backends.Optional simulator-specific hyperparameters preserve access to backend-specific features.
  • Simulator-Agnostic Interface: The Handler class standardizes common simulator operations, including launching, state access and updates, stepping, observations, rewards, success, termination, and timeouts.Each simulator implements its own Handler instance while sharing the same lifecycle-oriented interface.
  • User-Friendly Environment Wrapper: A Gym wrapper transforms a Handler into a learning environment by implementing Gym APIs through corresponding Handler methods.The documented methods include step(), reset(), render(), and close().

1) Cross-Simulator Integration:

ROBOVERSE uses METASIM to unify diverse simulators, embodiments, data sources, and collection workflows through common configurations and interfaces. This supports simulator interoperability, hybrid simulation, cross-embodiment reuse, and scalable dataset construction.

  • Cross-Simulator Integration: METASIM enables tasks and trajectories to move between simulators, supporting unified benchmark construction and sim-to-sim reinforcement-learning transfer.For example, MetaWorld tasks can be trained in Isaac Gym and rendered in Isaac Sim.
  • Hybrid Simulation: Hybrid simulation combines one simulator’s physics engine with another’s renderer to support higher-quality data generation.The paper gives MuJoCo physics and Isaac Sim rendering as an example combination.
  • Infrastructure: METASIM provides a universal configuration system, aligned simulator backends, and a Gym environment wrapper for simulator-agnostic operation.These layers abstract environments into common specifications and standardize simulator functionality.
  • Cross-Embodiment Transfer: Cross-embodiment transfer reuses trajectories across gripper-based robot morphologies by retargeting end-effector poses.This allows data from diverse robots to be integrated into a unified format.
  • Data Collection: ROBOVERSE collects tasks, assets, and trajectories through migration, language-model task generation, real-to-sim reconstruction, teleoperation, and augmentation.The resulting workflows integrate data from diverse simulation environments and public sources.
  • Teleoperation: ROBOVERSE supports teleoperation of arms, dexterous hands, and bimanual systems using phones, motion capture, VR devices, keyboards, and joysticks.These approaches operate across different simulators.
  • AI-Assisted Task Generation: AI-assisted generation uses large generative models to diversify task varieties and scenario distributions while arranging assets into physically plausible scenes.The approach incorporates spatial and semantic constraints learned from example placements.
  • Real-to-Sim: Real-to-sim tools reconstruct high-fidelity assets from video using multi-view capture, rendering, physical-property inference, and geometry reconstruction.The reconstructed assets support actions in both ROBOVERSE and the real world.

D. Data Augmentation

ROBOVERSE augments limited demonstrations and standardizes simulated variation to expand trajectory data and improve dataset breadth. Its migrated datasets span manipulation, navigation, and humanoid tasks across multiple embodiments.

  • Trajectory Augmentation: ROBOVERSE divides source demonstrations into object-centric manipulation segments and generates trajectory datasets for varied object and robot initial and goal-state distributions.The method assumes a predefined sequence of subtasks and uses MimicGen for expansion.
  • Domain Randomization: ROBOVERSE implements domain randomization across scene materials, lighting, camera poses, and surface reflection properties.These options can be combined to create varied indoor environments.
  • Dataset Statistics: Manipulation migration contributes 276 task categories, 510.5k trajectories, and 5.5k assets to ROBOVERSE.Representative tasks include rich domain randomization.
  • Dataset Statistics: The navigation dataset integrates 90 MatterPort 3D scenes, 10k R2R episodes, and 20k RxR episodes for Unitree Dog and JetBot embodiments.It focuses on vision-and-language navigation in continuous environments.
  • Dataset Statistics: The humanoid dataset integrates HumanoidBench tasks with policies and samples from Humanoid-X and SkillBlender across the ROBOVERSE framework.A pretrained policy follows demonstrated poses while maintaining stable locomotion across multiple simulators.

V. ROBOVERSE BENCHMARK

ROBOVERSE establishes standardized imitation- and reinforcement-learning benchmarks with controlled evaluation and multiple levels of generalization. Experiments assess benchmark reliability across representative tasks and show that synthetic data can improve world model learning.

  • ROBOVERSE defines unified training and evaluation protocols for imitation and reinforcement learning, with standardized baselines and learning frameworks.
  • Imitation-learning benchmark: The imitation-learning benchmark evaluates generalization from task-space variation through environment, camera, lighting, and reflection randomization.Each level uses a 90/10 training and evaluation split.
  • Imitation-learning benchmark: Policies are trained only on provided demonstrations and evaluated in controlled environments containing unseen samples for fair comparison.
  • Experimental validation: ROBOVERSE experiments validate collected datasets and benchmarks for imitation and reinforcement learning, while synthetic data significantly boosts world model learning.
  • Benchmark scope: The benchmark covers representative tasks from multiple source benchmarks and supports specialist and generalist imitation-learning models.

2) Implementation Details:

The experiments evaluate imitation-learning baselines, reinforcement-learning policies, trajectory augmentation, and action-conditioned world models. Results indicate scalable augmentation, improved physical realism from synthetic data, and stable cross-simulator reinforcement-learning performance.

  • Baseline evaluation: Imitation-learning evaluations use single-task specialist and generalist models under the 90/10 ROBOVERSE protocol, averaging success rates over three random seeds.Ten training and ten validation settings are sampled during evaluation.
  • Reinforcement learning: Reinforcement-learning policies trained across IsaacLab, Isaac Sim, and MuJoCo achieve stable convergence and comparable performance to native MuJoCo baselines.The workflow also supports TD-MPC2 across environments.
  • Trajectory augmentation: Success rates consistently improve as generated trajectory augmentations increase from 200 to 3,000 demonstrations beyond 50 source demonstrations.The comparison uses Diffusion Policy on four representative imitation-learning tasks.
  • World model learning: Adding 50,000 ROBOVERSE episodes to 50,000 DROID episodes improves world-model predictions in preserving object geometry.The combined dataset contains 100,000 episodes.
  • World model learning: Models trained with ROBOVERSE data generate more physically realistic frames on ROBOVERSE samples, whereas DROID-only training transfers poorly to those scenes.The authors associate the improvement with ROBOVERSE randomization and augmentation.

F. Imitating the ROBOVERSE Dataset Enables Direct Sim-to-Real Transfer

ROBOVERSE supports direct sim-to-real transfer for imitation learning and sim-to-sim-to-real transfer for reinforcement learning. The section also reports simulator-accuracy limitations and broader platform scope boundaries.

  • Direct sim-to-real transfer: OpenVLA fine-tuned on ROBOVERSE data manipulates unseen real-world objects in new environments without additional fine-tuning.
  • Sim-to-sim-to-real transfer: ROBOVERSE supports sim-to-sim-to-real transfer for whole-body humanoid control and in-the-wild lower-body policy generalization with upper-body PD control.
  • Limitations: ROBOVERSE does not yet fully support a unified format for non-rigid objects, and foundation-model pretraining is outside this paper’s scope because of resource constraints.
  • Scope: The paper’s primary goal is to demonstrate platform comprehensiveness, policy diversity, and alignment between simulation and real-world performance rather than directly compare policy performance.
  • Simulator accuracy: Conservation-law experiments find that basic conservation laws are not maintained across the three tested simulators, with behavior varying by experimental setup.The experiments cover momentum, angular momentum, and kinetic energy.

A. Architecture Overview

METASIM provides a simulator-agnostic architecture that separates configurations, tasks, and simulator backends while exposing common interfaces. It supports multiple simulators, hybrid execution, parallel environments, and Gym-compatible learning APIs.

  • Architecture: METASIM exposes a unified framework with Gym APIs on the front end and multiple interchangeable simulator backends on the back end.
  • Configuration system: MetaConfig describes simulator scenarios with simulator-agnostic settings plus separate backend-specific parameters.
  • Backend support: METASIM supports Isaac Sim, Isaac Gym, MuJoCo, PyBullet, SAPIEN, and CoppeliaSim, including parallel environments for SAPIEN and PyBullet.
  • Handler interface: The framework standardizes common simulator operations through Handler methods for retrieving states, setting states, and advancing simulation.
  • Learning interface: An Env layer builds Gymnasium-like step, reset, render, and close APIs on top of Handler implementations.

X. ASSET CONVERSION

RoboVerse uses a multi-stage asset-conversion pipeline to support interoperability across simulation formats and environments. The pipeline improves compatibility but still requires case-specific handling for some assets and can suffer texture-alignment issues.

  • Format interoperability: MJCF, URDF, and USD serve different roles, motivating a standardized conversion strategy for cross-simulator asset management.MJCF offers expressive physics modeling, while URDF and USD provide broadly supported intermediate and simulation representations.
  • Conversion pipeline: The pipeline converts MJCF to URDF and then USD because direct MJCF-to-USD conversion can mishandle rich physics features.URDF provides established conversion tools, and Isaac Sim supports robust URDF importing into USD.
  • Mesh and texture handling: Mesh and texture processing parses asset structure, aligns collision meshes, and preserves texture mappings when suitable existing OBJ files are available.When no compatible OBJ exists, the pipeline converts MSH files and warns about possible texture misalignment.
  • Limitations: Specific MJCF packages still require bespoke conversion strategies because the general pipeline does not handle every file uniformly.The authors report case-by-case modifications for some folders and packages.
  • USD conversion: Isaac Sim’s URDF-to-USD converter preserves structural, kinematic, geometric, and physical information while serving as an external proprietary tool.The converter bridges standard robotics formats and the USD representation used by the simulation environment.

A. ManiSkill

This section describes how ROBOVERSE incorporates tasks, assets, demonstrations, and success checkers from multiple robotic benchmarks. Adaptation includes format conversion, trajectory generation or replay, and task-specific validation.

  • ManiSkill: ROBOVERSE migrates single-arm ManiSkill tasks and demonstrations, including pick-and-place and insertion tasks, with manually prepared assets and physics APIs.Examples include PickCube, PickSingleYCB, PegInsertionSide, and PlugCharger.
  • RLBench: RLBench contributes approximately 2K rolled-out trajectories per task after its assets and tasks are migrated into ROBOVERSE.RLBench provides 100 hand-designed manipulation tasks with waypoint-generated demonstrations.
  • CALVIN: CALVIN demonstrations are segmented using text annotations and timestamps, with each segment’s first frame adopted as the initial scene state.Success checkers filter failed executions caused by coarsely annotated timestamps.
  • Benchmark coverage: The adaptation covers diverse benchmarks, including Meta-World, Open6DOR, ARNOLD, robosuite, SimplerEnv, and GAPartNet.These benchmarks contribute tasks involving tabletop manipulation, open-instruction rearrangement, language-conditioned manipulation, and articulated-object control.
  • Trajectory generation: Trajectory sources vary across benchmarks, combining demonstrations, motion planning, reinforcement-learning rollouts, and controller-based collection.GAPartNet uses both motion-planning trajectories and retrained reinforcement-learning policies.

J. GAPartManip

This section extends ROBOVERSE with articulated-object, grasping, deformable-object, door, and language-conditioned task data. Demonstrations are generated or adapted through planning, policy rollouts, official trajectories, and randomized scene initialization.

  • GAPartManip: GAPartManip supplies mesh-level grasping-pose annotations that ROBOVERSE uses to generate diverse articulated-object manipulation demonstrations.The section implements OpenBox and OpenToilet tasks with converted USD assets and motion-planned demonstrations.
  • Grasping: GraspNet-1B contributes 58 filtered target objects selected for real-world purchasing and evaluation, with cuRobo used for grasping demonstrations.The source dataset contains 256 tabletop scenes and 88 objects overall.
  • Garment manipulation: GarmentLab contributes 6k validated trajectories by extending official Isaac Sim trajectories across garments and textures.The original trajectories are limited and hard-coded, so the authors expand demonstrations using varied garment assets and textures.
  • RLAfford: ROBOVERSE adapts four RLAfford tasks and 40k trajectories across 40 articulated objects, recording 1,000 trajectories per object-task pairing.Hooks extract randomized initialization and per-step object and robot state information from the original codebase.
  • LIBERO: LIBERO tasks are translated from BDDL, INIT, and HDF5 components into MetaSim formats, then augmented by randomized initial positions.The sampling range depends on task context and object dimensions to maintain physically plausible configurations.

A. Keyboard

ROBOVERSE supports keyboard, smartphone, joystick, and VR interfaces for simulated robot control, alongside real-to-sim reconstruction and scene randomization. These tools broaden interaction and data-generation options, but remain subject to control and reconstruction limitations.

  • Keyboard: Keyboard control provides six-degree-of-freedom end-effector manipulation through world-frame translation, local-frame rotation, and gripper toggling.The interface supports multi-key diagonal movement and displays control instructions in a pygame operation window.
  • Other interfaces: DualSense and VR controllers provide analog velocity control, spatial awareness, and gesture-based interaction beyond keyboard and smartphone input.The modular system is designed to support additional input devices with limited development effort.
  • Real-to-sim: Real2Sim reconstructs environments, objects, and robots from monocular video using Gaussian splatting, mesh extraction, and unified robot descriptions.The resulting assets can be imported into simulators to form high-fidelity benchmark environments.
  • Real-to-sim results: 80% success rate was achieved with the real2sim module versus a 50% DexGraspNet baseline in physics-based grasping simulations.The comparison evaluates grasp execution using reconstructed assets intended to replicate real-world conditions.
  • Limitations: Real2Sim struggles with completely unseen meshes and complex materials, while friction and mass remain difficult to estimate from visual data.These uncertainties can affect simulation fidelity despite the pipeline’s ability to generate simulation-ready assets.
  • Smartphone: Smartphone control uses orientation sensing, multi-touch translation buttons, and switches for orientation and gripper control.The app communicates over WebSocket and visualizes simulated robot motion in real time.

B. Visual Material Randomization

ROBOVERSE diversifies visual conditions through randomized materials, lighting, camera poses, and curated environments, while supporting migrated robotic tasks and standardized evaluation across embodiments.

  • Visual randomization: ROBOVERSE randomizes visual materials using over 300 ARNOLD and vMaterials candidates, including roughness, specular, and metallic properties.
  • Visual randomization: Two lighting configurations vary source geometry, intensity, and color temperature within physically plausible ranges.
  • Visual randomization: A set of 59 candidate camera poses provides mostly frontal views with additional side-facing viewpoints.
  • Benchmark tasks: HumanoidBench locomotion tasks are migrated with Unitree H1 and dexterous Shadow Hands, while success checkers define task completion criteria.
  • Visual randomization: The benchmark uses curated assets and a 9:1 training-to-testing split so evaluation contains unseen environmental variations.

B. ROBOVERSE Benchmark Protocol

The benchmark protocol combines standardized training and evaluation, diffusion-based policy learning, world-model data construction, and cross-domain experiments to assess generalization and generation quality.

  • Evaluation protocol: Training and evaluation use a standardized API with a 9:1 split, testing policies on scenarios absent from training.
  • Policy models: Diffusion Policy predicts noise from noisy actions, observations, and denoising steps, conditioning on image and robot-state features.
  • Policy models: Inference starts from random actions and iteratively denoises them for K steps to produce final action predictions.
  • World-model data: DROID episodes are filtered to 20–60 frames, yielding about 50,000 unique episodes matched to RoboVerse episode lengths.
  • World-model results: Joint-position conditioning yields more precise gripper control than Cartesian position plus orientation in frame generation.
  • World-model results: Generation quality remains suboptimal on DROID samples because of scene complexity, while a 500M-parameter model with batch size 8 better preserves object geometry.
  • World-model results: RoboVerse-trained models produce more physically and geometrically consistent predictions on RoboVerse scenes, although real-world physics remains difficult.
Loading 2504.18904v1…