PaperlayerView exact revision on arXiv

Source-linked AI summary

High-Fidelity Simulated Data Generation for Real-World Zero-Shot Robotic Manipulation Learning with Gaussian Splatting

Haoyu Zhao, Cheng Zeng, Linghao Zhuang, Yaxi Zhao, Shengke Xue, Hao Wang, Xingyue Zhao, Zhongyu Li, Kehan Li, Siteng Huang, Mingxiu Chen, Xin Li, Deli Zhao, Hua Zou

arXiv:2510.10637v1cs.RO

TL;DR

Costly real-world data collection and persistent simulation-to-reality gaps limit scalable robotic learning. RoboSimGS reconstructs interactive simulated scenes from multi-view images using 3DGS, meshes, and MLLM-inferred object properties and articulation. Its generated data supports zero-shot real-world transfer and improves existing visuomotor policies, though scene reconstruction remains a deployment bottleneck.

  • Problem

    Real-world data collection is costly, and simulated data faces gaps in visual appearance, physical properties, and object interactions that challenge real-world transfer.

    Source: §1 · p. 1

  • Method

    RoboSimGS builds photorealistic, physically interactive simulations from multi-view images using a 3DGS-and-mesh representation and MLLM-inferred object properties and articulation.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    RoboSimGS-generated data enables zero-shot real-world transfer by policies trained entirely in simulation and significantly boosts existing state-of-the-art policies’ performance and generalization.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors present RoboSimGS as a scalable data-generation approach for bridging the sim-to-real gap in robotic manipulation.

    Source: §1 · p. 2, §5 · p. 11

    The time-consuming, complex scene-reconstruction step remains a bottleneck for large-scale deployment to new environments.

    Source: §5 · p. 11

Abstract

from arXiv · show

The scalability of robotic learning is fundamentally bottlenecked by the significant cost and labor of real-world data collection. While simulated data offers a scalable alternative, it often fails to generalize to the real world due to significant gaps in visual appearance, physical properties, and object interactions. To address this, we propose RoboSimGS, a novel Real2Sim2Real framework that converts multi-view real-world images into scalable, high-fidelity, and physically interactive simulation environments for robotic manipulation. Our approach reconstructs scenes using a hybrid representation: 3D Gaussian Splatting (3DGS) captures the photorealistic appearance of the environment, while mesh primitives for interactive objects ensure accurate physics simulation. Crucially, we pioneer the use of a Multi-modal Large Language Model (MLLM) to automate the creation of physically plausible, articulated assets. The MLLM analyzes visual data to infer not only physical properties (e.g., density, stiffness) but also complex kinematic structures (e.g., hinges, sliding rails) of objects. We demonstrate that policies trained entirely on data generated by RoboSimGS achieve successful zero-shot sim-to-real transfer across a diverse set of real-world manipulation tasks. Furthermore, data from RoboSimGS significantly enhances the performance and generalization capabilities of SOTA methods. Our results validate RoboSimGS as a powerful and scalable solution for bridging the sim-to-real gap.

1 INTRODUCTION

Robotic learning is constrained by costly real-world data collection, while simulated and photorealistic environments still leave gaps in transfer and physical interaction. RoboSimGS combines photorealistic reconstruction with interactive assets and reports zero-shot transfer and gains for existing policies.

1 INTRODUCTION

  • Real-world demonstration collection demands substantial human effort, constraining the scalability of generalist robot policies.Simulation can scale with computational resources, but domain gaps in geometry, appearance, and material behavior challenge transfer.Source: §1 · pp. 1–2
  • Earlier Real2Sim2Real approaches achieved visual fidelity but often lacked interactive physics, limiting their use for contact-rich manipulation.Their static, non-interactive scenes were suited mainly to observational tasks.Source: §1 · p. 2
  • RoboSimGS pairs photorealistic 3DGS backgrounds with interactive mesh objects and uses an MLLM to infer physical properties and kinematic structures from images.It also randomizes object properties, camera views, lighting, and trajectories to enhance policy robustness.Source: §1 · p. 2
  • RoboSimGS policies trained exclusively on simulated data achieve zero-shot real-world transfer, while its generated data boosts existing state-of-the-art policies’ performance and generalization.The framework combines a hybrid 3DGS-and-mesh scene representation with MLLM-generated physical and articulated assets.Source: §1 · p. 2

2 RELATED WORK

Prior data-generation and Sim2Real approaches expand training data or improve transfer, but leave limitations in novel-object geometry, 3D awareness, and articulated interaction. RoboSimGS uses a hybrid 3DGS-and-mesh scene and MLLM-inferred assets to build interactive simulation environments.

2.1 Data Generation for Robotics

  • Data-generation approaches include trajectory-level generation and observation-level augmentation, each expanding data from limited source demonstrations.Trajectory methods replay transformed source actions under new object poses, but their rigid object-level formulation struggles with novel geometries and precise manipulation.Source: §2.1 · p. 2
  • 2D visual augmentation is computationally efficient but lacks 3D spatial awareness, which can produce physically implausible scene edits for interaction-grounded tasks.RoboSimGS instead generates data in a physically interactive 3D simulator.Source: §2.1 · p. 2
  • RoboSimGS’s pipeline starts from multi-view images, reconstructs a hybrid scene, aligns it with a simulator, and augments it to generate diverse simulated data.The hybrid representation uses 3DGS for the background and mesh objects for interaction.Source: Figure 1 · p. 3

2.2 Sim2Real Transfer in Robotics

  • Domain randomization varies simulation parameters, while domain adaptation seeks a unified feature space for simulated and real environments.Both are presented as approaches for bridging the Sim2Real gap.Source: §2.2 · p. 3
  • R2S2R methods using 3DGS can reconstruct photorealistic scenes, but commonly rely on predefined rigid assets and do not capture complex articulations or non-rigid dynamics.The related-work discussion notes fixed assets in DexMimicGen and rigid objects in RoboVerse.Source: §2.2 · p. 3
  • RoboSimGS combines a photorealistic 3DGS background with interactive meshes and MLLM-inferred physical properties and kinematic constraints to support dynamic data generation.Its pipeline reconstructs the scene, aligns it with a simulator, and augments objects, cameras, lighting, and trajectories.Source: Figure 1 · p. 3, §2.2 · p. 4

3 Methodlogy

RoboSimGS turns multi-view images into a realistic, physically interactive simulation environment through scene reconstruction, simulator alignment, and holistic augmentation. Policies trained on the augmented synthetic data can transfer zero-shot to real-world scenarios.

3 Methodlogy

  • RoboSimGS reconstructs a hybrid scene from multi-view images, using 3DGS for static appearance and explicit meshes for interactive objects.The meshes become physics-enabled assets when imported into the simulator.Source: §3 · p. 4
  • Sim2Real Environment Alignment imports the reconstructed background and interactive meshes into a physics simulator.The 3DGS model supplies the visual background while meshes provide dynamic, physics-enabled assets.Source: §3 · p. 4
  • Holistic Scene Augmentation randomizes objects, cameras, lighting, and trajectories to create diverse simulated training data for zero-shot real-world transfer.The passage states that policies trained exclusively on this augmented synthetic data can achieve zero-shot transfer.Source: §3 · p. 4

3.1 Scene Reconstruction

RoboSimGS reconstructs scenes with photorealistic static backgrounds and physically interactive objects. It combines 3D Gaussian Splatting with mesh-based object reconstruction, including MLLM-driven articulation and physics estimation.

3.1 Scene Reconstruction

  • Each task-specific scene uses about 10 minutes of manual multi-view scanning, then separates photorealistic background rendering from physically plausible interactive-object meshes.The robot is excluded from reconstruction because its model is available in a standard URDF.Source: §3.1 · p. 4

3.1.1 Background Reconstruction

  • 3DGS renders the background from depth-sorted Gaussians and supports semantic-feature supervision alongside color rendering.Learnable features are trained against CLIP text embeddings and multi-class 2D semantic masks.Source: §3.1.1 · p. 4

3.1.2 Object Reconstruction

  • For interactive objects, MLLMs infer articulated structure and material properties, enabling static meshes to become physically plausible simulation assets.The pipeline estimates joint type, part labels, axes, motion limits, density, Young’s modulus, and Poisson’s ratio.Source: §3.1.2 · p. 5

3.2 Sim2Real Environment Alignment

RoboSimGS aligns the reconstructed scene to the simulator using the robot as a shared geometric anchor, then aligns the simulated camera to a real reference view. Together, these steps integrate the hybrid representation into a digital twin for data generation.

3.2.1 World Coordinate Frame Alignment

  • ICP finds a rigid rotation and translation aligning robot point clouds from the URDF and 3DGS, then maps the reconstructed scene into simulator coordinates.The robot is fixed at a default joint configuration during alignment.Source: §3.2.1 · p. 6

3.2.2 Camera Pose Alignment

  • Camera pose is optimized to minimize photometric error between a rendered image and the real reference, using differentiable rendering and gradient descent.Starting from a rough pose, the camera parameters are iteratively refined; the aligned hybrid representation forms a digital twin ready for large-scale data generation.Source: §3.2.2 · p. 6

3.3 Holistic Scene Augmentation

Holistic Scene Augmentation diversifies simulated training data to help policies handle real-world variability. RoboSimGS randomizes object configurations, camera views, lighting, and robot trajectories.

3.3 Holistic Scene Augmentation

  • Holistic Scene Augmentation randomizes multiple facets of the simulator to create a diverse dataset beyond a static environment.The stated goal is to address real-world variability when training robust policies.Source: §3.3 · p. 7

3.3.1 Object-level Augmentation

  • Object-level augmentation varies interactive objects’ 6-DoF poses and uniform scales within predefined ranges.This exposes the policy to varied object configurations for generalization to unstructured real-world environments.Source: §3.3.1 · p. 7

3.3.2 Camera View Augmentation

  • Camera-view augmentation perturbs the deployment-matched camera pose with random translations and rotations to diversify realistic views.The view distribution remains centered on the deployment perspective.Source: §3.3.2 · p. 7

3.3.3 Lighting Condition Augmentation

  • Lighting augmentation varies Gaussian attributes and adds per-Gaussian noise to simulate global brightness and contrast changes and camera sensor noise.The aim is robustness to previously unseen lighting conditions.Source: §3.3.3 · p. 7

3.3.4 Trajectory Augmentation

  • Trajectory augmentation adds a randomized intermediate end-effector via-point before the final goal, diversifying paths instead of using a single kinematic path.An inverse-kinematics solver generates the via-point from a small positional offset to the final goal.Source: §3.3.4 · p. 7

4.1 Experimental Setups

The experiments evaluate policies and generated data across real and simulated environments, using eight manipulation tasks and randomized initial object states. Success is measured by task completion across real-world trials.

4.1.1 Hardware platform

  • Experiments use real-world and simulated environments, with real-world data collection and evaluation through LeRobot and simulated data generated on a separate workstation.The real-world setup uses two cameras capturing RGB observations at 640x480 resolution.Source: §4.1.1 · p. 7

4.1.2 Policy and VLA model training details

  • The evaluation uses Diffusion Policy across tasks and the generalist π0 model; Table 1 reports a boost across methods from adding 50 RoboSimGS demonstrations.Real-world capture uses two RGB cameras; simulation data is generated on an Intel Core i5-14400F and NVIDIA RTX 5060 Ti workstation.Source: §4.1.2 · p. 7, §4.1.1 · p. 7, Table 1 · p. 8

4.1.3 Metric

  • Success Rate is the percentage of successful task completions over 35 consecutive real-world trials, with completion required in under 20 seconds.Source: §4.1.3 · p. 8

4.1.4 Robot tasks

  • The eight tasks span stacking, articulated-object interaction, deformable-object handling, and tool use, with primary-object and container poses randomized at initialization.Initial positions are sampled 28–35 cm from the robot base, and orientations are randomized around the vertical axis; Figure 4 visualizes performance under four challenging generalization settings.Source: §4.1.4 · p. 8, Figure 4 · p. 9

4.2 Results

RoboSimGS-generated data supports zero-shot sim-to-real transfer and improves results when combined with real demonstrations. The experiments also compare data scaling, robustness under challenging settings, and cross-domain performance.

4.2.1 Zero-shot Sim2Real Transfer

  • The zero-shot transfer experiment trains a policy exclusively on 100 RoboSimGS demonstrations and compares it with one trained on 50 real-world demonstrations, without real-world fine-tuning.Source: §4.2.1 · p. 9

4.2.2 Synergistic Effect of Real and Simulated Data

  • Adding 50 RoboSimGS demonstrations to 50 real demonstrations substantially and consistently boosts performance across evaluated methods.Source: §4.2.2 · p. 9

4.2.4 Generalization to Challenging Settings

  • Policies trained with augmented synthetic data outperform real-only policies under challenging real-world conditions, while combining simulated and real data yields the best results.Source: §4.2.4 · p. 9

4.2.3 Data Scaling and Efficiency Analysis

  • 200 simulated demonstrations achieve a success rate comparable to 100 real-world demonstrations on Stack Cubes with Diffusion Policy.Source: §4.2.3 · p. 9, Figure 5 · p. 10

4.2.6 Data Generation Efficiency and Scalability

  • Table 3 reports simulated-data collection success rate and average generation time per sample, measured on a single NVIDIA RTX 5060 Ti GPU.Source: Table 3 · p. 10

4.2.5 Sim2Real Transfer Fidelity

  • Policies trained and evaluated in simulation and then deployed in reality achieve success rates nearly identical to real-trained, real-evaluated policies; real-trained policies also perform strongly in simulation.Source: §4.2.5 · p. 10

4.3 Ablation study

The ablations show that physics estimation and holistic scene augmentation both matter for real-world policy performance. Removing either component is associated with degraded performance or a severe performance collapse.

4.3.1 Effect of Physics Estimation

  • Without MLLM-driven physics estimation, Wiping success falls from 0.85 to 0.51 and Deformable PickPlace from 0.69 to 0.54, with invalid interaction dynamics identified as the issue.The comparison uses policies trained on 50 demonstrations with default, physically implausible properties; the authors conclude accurate physics is as critical as visual fidelity.Source: §4.3.1 · pp. 10–11

4.3.2 Effect of Holistic Scene Augmentation

  • Randomizing only object pose causes a severe performance collapse, with policies overfitting to the static camera, lighting, and deterministic action sequences.The authors describe holistic randomization of the entire scene context as indispensable for a robust, deployable policy.Source: §4.3.2 · p. 11

5 CONCLUSIONS

RoboSimGS combines photorealistic 3DGS scenes with physically interactive mesh objects and MLLM-inferred object properties and kinematics. Policies trained solely on its generated data transfer zero-shot to diverse real-world manipulation tasks, while the reconstruction pipeline remains a bottleneck for scaling to new environments.

5 CONCLUSIONS

  • Policies trained solely on RoboSimGS-generated data achieve zero-shot transfer across diverse real-world manipulation tasks.Source: §5 · p. 11
  • RoboSimGS data substantially improves SOTA visuomotor policies when supplementing limited real-world data.Source: §5 · p. 11
  • RoboSimGS combines photorealistic 3DGS backgrounds with mesh-based dynamic objects and uses an MLLM to infer their physical properties and kinematic structures.Source: §5 · p. 11
  • The time-consuming, complex reconstruction pipeline is a significant bottleneck to large-scale deployment in new environments, despite being a one-time effort per scene.Source: §5 · p. 11

Related papers

Loading 2510.10637v1…