PaperlayerView exact revision on arXiv

Source-linked AI summary

Genie Sim 3.0 : A High-Fidelity Comprehensive Simulation Platform for Humanoid Robot

Chenghao Yin, Da Huang, Di Yang, Jichao Wang, Nanshu Zhao, Chen Xu, Wenjun Sun, Linjie Hou, Zhijun Li, Junhui Wu, Zhaobo Liu, Zhen Xiao, Sheng Zhang, Lei Bao, Rui Feng, Zhenquan Pang, Jiayu Li, Qian Wang, Maoqing Yao

arXiv:2601.02078v4cs.RO

TL;DR

Robot-learning research needs scalable, diverse training data and reproducible evaluation, but physical collection is costly and existing simulation benchmarks can be fragmented, narrow, or insufficiently faithful. Genie Sim 3.0 unifies LLM-driven scene generation, automated LLM-VLM evaluation, high-fidelity simulation, and large-scale synthetic data. Across eight transfer tasks, synthetic-data-trained models achieved an average sim-to-real score of 0.83 versus 0.75 for real-data-trained models.

  • Problem

    Robot learning lacks scalable, diverse interaction data and reliable evaluation because physical collection is costly and existing simulation benchmarks can be fragmented, narrow, or insufficiently faithful for sim-to-real transfer.

  • Method

    Genie Sim 3.0 integrates LLM-driven scene generation, automated LLM task and evaluation generation, high-fidelity simulation, and teleoperation and automated data collection.

  • Results

    Across 8 tasks, models trained solely on synthetic data achieved an average sim-to-real score of 0.83 versus 0.75 for models trained on real data.

  • Takeaways & Limitations

    The open-source platform provides large-scale synthetic data and automated evaluation for scalable policy training and robust capability assessment.

Abstract

from arXiv · show

The development of robust and generalizable robot learning models is critically contingent upon the availability of large-scale, diverse training data and reliable evaluation benchmarks. Collecting data in the physical world poses prohibitive costs and scalability challenges, and prevailing simulation benchmarks frequently suffer from fragmentation, narrow scope, or insufficient fidelity to enable effective sim-to-real transfer. To address these challenges, we introduce Genie Sim 3.0, a unified simulation platform for robotic manipulation. We present Genie Sim Generator, a large language model (LLM)-powered tool that constructs high-fidelity scenes from natural language instructions. Its principal strength resides in rapid and multi-dimensional generalization, facilitating the synthesis of diverse environments to support scalable data collection and robust policy evaluation. We introduce the first benchmark that pioneers the application of LLM for automated evaluation. It leverages LLM to mass-generate evaluation scenarios and employs Vision-Language Model (VLM) to establish an automated assessment pipeline. We also release an open-source dataset comprising more than 10,000 hours of synthetic data across over 200 tasks. Through systematic experimentation, we validate the robust zero-shot sim-to-real transfer capability of our open-source dataset, demonstrating that synthetic data can server as an effective substitute for real-world data under controlled conditions for scalable policy training. For code and dataset details, please refer to: https://github.com/AgibotTech/genie_sim.

I. INTRODUCTION

Genie Sim 3.0 addresses costly real-world data collection and fragmented, narrow, or low-fidelity evaluation by unifying scene generation, data collection, and automated assessment. Its open-source platform combines LLM-driven generation, high-fidelity simulation, scalable benchmarking, and large synthetic datasets.

  • Real-world robot-interaction data is costly and time-consuming to collect, while existing evaluation approaches lack consistent, reproducible, and scalable testing environments.
  • Genie Sim 3.0 unifies environment reconstruction, scene generalization, data collection, and automated evaluation in an open-source simulation platform.
  • Its four core features are LLM-driven scene generation, LLM-based task and evaluation generation, high-fidelity simulation, and dual-mode teleoperation and automated data collection.
  • The platform provides 5,140 simulation-ready objects, more than 10,000 hours of synthetic data across 200 tasks, and over 100,000 evaluation scenarios.
  • Genie Sim Generator supports natural-language scene creation, rapid multidimensional variation, and iterative refinement through conversational interaction.
  • The project releases its assets, dataset, evaluation scenarios, and complete codebase to support reproducible embodied-AI research and reduce reliance on physical hardware.

A. Robotics Dataset

Genie Sim combines broad synthetic-data coverage with an LLM-driven scene-generation pipeline. It retrieves simulation assets semantically, interprets natural-language requests, generates controllable scene programs, and assembles randomized Isaac Sim scenes.

  • Robotics Dataset: Existing robotics datasets span large real-world collections and diverse tasks, while synthetic datasets provide controlled, scalable variation but often remain narrow in setting or skill coverage.
  • Robotics Dataset: Existing benchmarks still face narrow evaluation scope, sim-to-real gaps in physical dynamics and rendering, and insufficient transfer to real-world deployment.
  • Scene Generation: Genie Sim Generator uses an LLM-driven Assets Index and Scene Generator to create simulation scenes and downstream task scene graphs.
  • Scene Generation: The four-stage generation process interprets intent, retrieves assets, generates domain-specific-language code, and assembles results within one shared conversational context.
  • Scene Generation: Assets Index retrieves among 5,140 objects using semantic descriptions, 2048-dimensional embeddings, and cosine similarity.
  • Scene Generation: The DSL generator combines interpreted intent, retrieved assets, and scene-language definitions to produce precise, finely controllable scene specifications supporting iterative edits.
  • Scene Generation: The assembler instantiates randomized scene programs into hierarchical Scene Graphs and Isaac Sim-ready USD files, generating thousands of diverse scenes within minutes.

B. Evaluation Generation

Genie Sim replaces manually authored, largely fixed evaluation pipelines with automated instruction generation and VLM-based assessment. LLMs generate executable evaluation configurations, while VLMs judge task completion from specifications and visual execution sequences.

  • Evaluation Generation: Conventional simulation benchmarks rely on predefined instructions, manually annotated success criteria, and repeated trials, limiting evaluation diversity and scalability.
  • Evaluation Generation: Genie Sim combines an LLM with the Action Domain Evaluation Rule system to generate diverse instructions and executable evaluation configurations automatically.
  • Evaluation Generation: A VLM evaluates formalized task specifications against temporal visual observations and produces evidence-based judgments of task completion.
  • Evaluation Generation: This pipeline reduces human annotation effort and enables large-scale evaluation instances at lower cost.

C. Environment Reconstruction

Genie Sim reconstructs interactive environments with photorealistic rendering and combines teleoperation with automated collection. Its automated pipeline uses asset retrieval, waypoint filtering, alternative trajectories, and retry-oriented evaluation while retaining environmental completeness.

  • Environment Reconstruction: 3D Gaussian Splatting reconstructs surfaces and renders photorealistic images for high-fidelity interactive physical simulation from scanned images, poses, and point clouds.
  • Environment Reconstruction: Because 3DGS is sensitive to camera-pose errors, the reconstruction pipeline addresses pose accuracy in complex indoor environments before training the renderer.
  • Environment Reconstruction: Generative view extrapolation compensates for insufficient collection views by supplying additional images for 3DGS training and improved rendering quality.
  • Data Generation: Teleoperation targets complex long-horizon tasks, whereas automated collection emphasizes efficiency and cost-effectiveness for extensive synthetic-data generation.
  • Data Generation: Teleoperation captures target end-effector poses through a PICO headset, executes trajectories in simulation, and logs robot states, observations, and object poses.
  • Data Generation: Automated collection uses LLM-based asset retrieval and predefined atomic skills to assemble task assets before planning trajectories.
  • Data Generation: Waypoint filtering evaluates reachability, collision avoidance, and anthropomorphic feasibility, while multiple candidate sequences support robust execution and retry.
  • Data Generation: Environmental completeness is retained during planning, with mesh simplification balancing collision-aware reliability against computational efficiency.

E. Closed-loop Evaluation

The closed-loop evaluation framework supports broad integration and automated, multidimensional assessment of embodied-intelligence models across simulation and inference environments.

  • Evaluation Pipeline: The simulator and model inference environment communicate over HTTP, exchanging observations and control commands during execution.Task completion is periodically evaluated, and completed simulations terminate.
  • System Integration: The platform integrates common VLA models, multiple robot types, and multiple end effectors.It also supports local or distributed inference.
  • Evaluation Pipeline: Automated and multidimensional evaluation is a core capability of the closed-loop framework.This supports assessment across varied model, robot, and task configurations.

IV. DATASET

The dataset organizes embodied-intelligence tasks by manipulation skill, cognitive comprehension, and task complexity, with composable structures supporting progression from simple to complex scenarios.

  • Composability: Long-horizon tasks can be decomposed into sequences of fundamental subtasks contained within the dataset.
  • Task Taxonomy: The task taxonomy uses manipulation skill, cognitive comprehension, and task complexity as its three primary axes.These dimensions structure the dataset’s task distribution.
  • Manipulation Skill: Manipulation skills cover fundamental motor actions such as pick, place, pull, push, open, and close.
  • Cognitive Comprehension: Cognitive comprehension includes spatial reasoning, attribute understanding, logical inference, and commonsense reasoning.
  • Task Complexity: Task complexity increases with planning horizon and coordinated control, progressing from single-arm actions to bimanual and multi-object tasks.The examples range from removing one trash item to cleaning all trash from a desktop.

B. Data Distribution

Genie Sim 3.0 distributes data and evaluation across diverse task capabilities, using complementary closed-loop suites to assess transfer, instruction following, robustness, manipulation, and spatial skills.

  • Data Distribution: The synthetic dataset contains 200 representative embodied-intelligence tasks and over 10,000 hours of simulated interaction.It varies layout, robot pose, lighting, scene configuration, camera noise, and instruction phrasing.
  • Data Distribution: The dataset varies task environments across layout, initial pose, lighting, scene configuration, camera noise, and semantic instruction phrasing.These systematic variations are intended to provide broad coverage and generalization.
  • Evaluation Suites: Five closed-loop evaluation suites provide complementary perspectives on policy capabilities and benchmark reliability.
  • Evaluation Suites: GenieSim-Sim2Real tests synthetic-data quality and simulation fidelity, while other suites evaluate language-conditioned manipulation, robustness, manipulation, and spatial capabilities.
  • Sim2Real Evaluation: The Sim2Real suite compares real and simulated testing environments across evaluated tasks.

A. Experimental Setup

Experiments evaluate π0.5 and other VLA models through real and simulated closed-loop testing, with task scores and sim-to-real relationships used to assess benchmark validity and synthetic-data scaling.

  • Synthetic Data Scaling: The Sim2Real experiments use π0.5 as the base policy, varying real or synthetic post-training data and evaluating each configuration in real and simulated environments.Each configuration receives 50 trials in both modalities.
  • Benchmark Setup: The four additional benchmark suites compare four state-of-the-art VLA models using GenieSim’s automated closed-loop evaluation pipeline.These evaluations run on the Agibot G2 robot.
  • Synthetic Data Scaling: R2 = 0.931 indicates a strong linear relationship between simulated and real-world scores across all 16 model configurations.
  • Synthetic Data Scaling: Synthetic-data scaling results compare π0.5 models fine-tuned with varying amounts of real and synthetic data in both simulated and real environments.
  • Sim-to-Real Transfer: Sim-to-real transfer results compare π0.5 models trained on synthetic or real data across eight tasks and both evaluation environments.

2) Sim-to-Real Transfer:

GenieSim evaluates policies across complementary task suites and shows strong sim-to-real consistency, enabling simulation-based comparisons of instruction following, robustness, manipulation, and spatial reasoning. Across these suites, ACoT-VLA and π0.5 lead with complementary strengths, while perturbation results distinguish genuine robustness from floor effects.

  • Sim-to-Real Transfer: Across all 8 transfer tasks, models trained on synthetic data are compared with models trained on real-world data in both simulated and physical environments.Synthetic-data training uses 500–1500 episodes, while real-world training uses 500 episodes.
  • Sim-to-Real Transfer: 32-group scaling experiments and the 8-task transfer study establish GenieSim as providing both synthetic data and a reliable simulation benchmark for cross-model evaluation.
  • Instruction Following: ACoT-VLA and π0.5 lead GenieSim-Instruction with average scores of 0.76 and 0.75, ahead of GR00T-N1.7 at 0.65 and π0 at 0.37.ACoT-VLA leads attribute-matching tasks, while π0.5 performs better on counting, geometric abstraction, or symbolic composition.
  • Robustness: ACoT-VLA and π0.5 also lead GenieSim-Robust at 0.622 and 0.613, while geometric and embodiment-level perturbations cause the largest degradations.Robot Pose and Camera Position each remove up to approximately 0.26; π0’s smallest average degradation reflects its low Reference score of 0.370 rather than stronger performance.
  • Spatial Reasoning: The spatial results identify sequential spatial construction as the binding constraint for current VLA policies, with a 0.38–0.46 mean gap between constructive and reference-oriented regimes.
  • Cross-Suite Comparison: Across four suites, ACoT-VLA leads the three cognition-oriented suites, whereas π0.5 leads GenieSim-Manipulation at 0.58 versus 0.48.The first three suite margins are 0.01–0.04, compared with a 0.10 manipulation gap.

VI. CONCLUSIONS

Genie Sim 3.0 unifies scalable scene generation, automated evaluation, and synthetic data for robotic manipulation. Its open-source dataset is validated for zero-shot sim-to-real transfer, supporting reproducible research toward more generalizable policies.

  • Genie Sim 3.0 combines LLM-generated high-fidelity scenes, an LLM-VLM benchmark for automated evaluation, and a large-scale dataset validated for zero-shot sim-to-real transfer.
Loading 2601.02078v4…