Source-linked AI summary
RoboGSim: A Real2Sim2Real Robotic Gaussian Splatting Simulator
Xinhai Li, Jialin Li, Ziheng Zhang, Rui Zhang, Fan Jia, Tiancai Wang, Haoqiang Fan, Kuo-Kun Tseng, Ruiping Wang
TL;DR
Remote demonstrations are costly to scale, while simulated data faces a sim-to-real gap. RoboGSim combines 3D Gaussian Splatting with a physics engine for demonstration synthesis and closed-loop evaluation. Its generated data is comparable to real-robot data in test views and performs better in novel scenes and perspectives.
Problem
Remote operation makes large-scale demonstrations costly, while simulated environments have struggled to model texture and physics with high fidelity.
Source: §1 · p. 2Method
RoboGSim combines 3D Gaussian Splatting and a physics engine to synthesize demonstrations and evaluate policies in a closed loop.
Source: §1 · p. 2, §3.5 · p. 5Results
RoboGSim-generated data achieved performance comparable to real-robot data in test views and better performance in novel scenes and perspectives.
Source: §1 · p. 2, §4.3 · p. 6Takeaways & Limitations
RoboGSim provides a closed-loop platform for fair policy comparison with safe, real-time, reproducible evaluation.
Source: §1 · p. 2, §3.5 · p. 5The current version simulates only rigid objects, does not fully unify object lighting with the robotic arm, and struggles to generate geometrically consistent meshes.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Efficient acquisition of real-world embodied data has been increasingly critical. However, large-scale demonstrations captured by remote operation tend to take extremely high costs and fail to scale up the data size in an efficient manner. Sampling the episodes under a simulated environment is a promising way for large-scale collection while existing simulators fail to high-fidelity modeling on texture and physics. To address these limitations, we introduce the RoboGSim, a real2sim2real robotic simulator, powered by 3D Gaussian Splatting and the physics engine. RoboGSim mainly includes four parts: Gaussian Reconstructor, Digital Twins Builder, Scene Composer, and Interactive Engine. It can synthesize the simulated data with novel views, objects, trajectories, and scenes. RoboGSim also provides an online, reproducible, and safe evaluation for different manipulation policies. The real2sim and sim2real transfer experiments show a high consistency in the texture and physics. We compared the test results of RoboGSim data and real robot data on both RoboGSim and real robot platforms. The experimental results show that the RoboGSim data model can achieve zero-shot performance on the real robot, with results comparable to real robot data. Additionally, in experiments with novel perspectives and novel scenes, the RoboGSim data model performed even better on the real robot than the real robot data model. This not only helps reduce the sim2real gap but also addresses the limitations of real robot data collection, such as its single-source and high cost. We hope RoboGSim serves as a closed-loop simulator for fair comparison on policy learning. More information can be found on our project page https://robogsim.github.io/.
1. Introduction
RoboGSim targets the cost and scalability limits of remote demonstrations and the transfer gap of simulation-based data. It unifies realistic demonstration synthesis across novel scenes, views, and objects with closed-loop policy evaluation.
1. Introduction
- Remote operation demonstrations become extremely costly as data size grows, while simulated-data approaches face a domain gap that can invalidate learned policies.Prior Real2Sim work also lacked synthesis on novel scenes, views, and objects and closed-loop evaluation across policy networks.Source: §1 · p. 2
- RoboGSim combines a 3DGS reconstructor, digital-twin builder with layout alignment, scene composer, and interactive engine for synthesis and evaluation.The reconstructor uses multi-view RGB sequences and robot MDH parameters; the builder creates an Isaac Sim digital twin.Source: §1 · p. 2
- RoboGSim unifies realistic demonstration synthesis across novel scenes, views, and objects with closed-loop evaluation for policy networks.The authors report comparable performance to real-robot data and better performance in novel scenes and perspectives.Source: §1 · p. 2
2. Related Work
Related work frames Sim2Real transfer as a persistent gap and groups prior approaches into three broad strategies. Robotics research has also applied 3D Gaussian Splatting to manipulation and simulation-to-real transfer, with limitations in prior systems' discussion of transfer or digital-twin assets.
2.1. Sim2Real in Robotics
2.2. 3D Gaussian Splatting in Robotics
- Robotics 3DGS work spans manipulation and transfer; Robo-GS [21] gives limited Sim2Real discussion, while SplatSim [29] lacks discussion of object digital-twin assets.3DGS represents scenes as explicit Gaussian points and uses efficient rasterization for high-fidelity real-time rendering, extending NeRF [25].Source: §2.2 · p. 3
3. Methods
RoboGSim links 3DGS reconstruction and digital-twin construction with scene composition and an interactive engine for data synthesis and closed-loop evaluation.
3.1. Overall Architecture
- The four-part architecture connects scene and object reconstruction, digital-twin building, novel scene/view/object synthesis, and closed-loop policy evaluation.The pipeline also supports embodied data collection using VR/Xbox equipment.Source: §3.1 · p. 3
3.2. Gaussian Reconstructor
- 3DGS reconstructs static scenes, while segmented robotic-arm joints are driven by MDH kinematics for dynamic rendering.The method treats Gaussian points within each joint as a point mass and moves them using the link transformation matrices.Source: §3.2 · pp. 3–4
3.3. Digital Twins Builder
- Real2Sim layout alignment and camera localization align scene coordinates; camera localization optimizes pose while keeping the Gaussian map fixed.Layout alignment compares segmentation views from a camera 1.6 meters above the arm base joint.Source: §3.3 · pp. 4–5
- Real-world object assets are reconstructed from multi-view captures, while web-sourced objects use generative 3D reconstruction and GaussianEditor-based editing.The real-object path uses GIM feature matching and COLMAP initialization before 3DGS reconstruction.Source: §3.3 · p. 4
3.4. Scene Composer
- Scene Composer transforms point clouds and Gaussian attributes to merge scenes and edit objects, using the target object's coordinate center for object transformations.The described edits adjust scale and normalize rotation as part of the transformation.Source: §3.4 · p. 5
3.5. Interactive Engine
- The Interactive Engine synthesizes diverse training trajectories and images, then evaluates policies through iterative rendering, action prediction, and simulated physical interactions.Isaac Sim supplies states and handles inverse kinematics and collision detection before rendered feedback returns to the policy.Source: §3.5 · p. 5
4. Experiments
Four UR5 proxy experiments assess Real2Sim pose rendering, Sim2Real trajectory replay, demonstration synthesis, and closed-loop policy evaluation. RoboGSim reports high-fidelity rendering and transfer, with synthetic data matching real-data test-view performance and outperforming it in novel scenes.
4.3. RoboGSim as Synthesizer
- 90% test-view performance for both RoboGSim and real data, and 90% versus 60% in novel scenes; RoboGSim also shows less novel-view bias.The comparison covers models trained on real data, real data plus 2D AUG, and RoboGSim data.Source: Table 1 · p. 6, §4.3 · p. 6
4.1. Real2Sim Novel Pose Synthesis
- 31.3 PSNR and 0.79 SSIM, with 10 FPS rendering, characterize RoboGSim's real-trajectory-driven novel-pose rendering against a new-view real capture.The experiment evaluates robotic-arm movement accuracy and image-texture fidelity.Source: §4.1 · p. 6, Figure 3 · p. 7
4.2. Sim2Real Trajectory Replay
- An Isaac Sim trajectory driving both the real arm and GS rendering produces strong alignment in a Coke-grasping replay, supporting the reported Sim2Real transfer.The same simulated trajectory drives the physical robot and the RoboGSim rendering.Source: §4.2 · p. 6, Figure 4 · p. 7
4.3. RoboGSim as Synthesizer
- Adding 2D AUG to real data lowers test-view performance from 90% to 80% and raises novel-scene performance from 40% to 60%, while increasing novel-view bias.The passage contrasts 2D AUG's performance with RoboGSim's spatial intelligence.Source: §4.3 · p. 6
- RoboGSim synthesis takes 4 hours versus 40 hours for manual collection, and multi-view renderings show UR5 operation in a factory, shelf, and two outdoor environments.Figure 6 depicts the same training set rendered in novel views and scenes.Source: §4.3 · p. 7, Figure 5 · p. 8, Figure 6 · p. 8
4.4. RoboGSim as Evaluator
- Closed-loop RoboGSim evaluation reproduces results similar to the real world and can avoid violations and collisions seen in similar bad cases.The authors describe the evaluator as fair, safe, and efficient.Source: §4.4 · p. 8
5. Conclusion and Discussion
RoboGSim combines 3DGS with a digital-twin system and spatial alignment to synthesize data across novel viewpoints, objects, trajectories, and scenes, and to evaluate policies online. The authors identify rigid-object simulation, object lighting, and geometrically consistent meshes as current limitations.
5. Conclusion and Discussion
- RoboGSim synthesizes high-fidelity data across novel viewpoints, objects, trajectories, and scenes, and supports real-time online policy evaluation through precise spatial alignment.The authors describe it as a Real2Sim2Real simulator based on 3DGS with a digital-twin system.Source: §5 · p. 8
- The current simulator handles only rigid objects, does not fully unify synthesized-object lighting with the robot arm, and still struggles to produce geometrically consistent meshes for complex manipulation.Source: §5 · p. 8
- The authors plan to explore improved mesh extraction, expand task categories, and establish benchmarks for evaluation across diverse scenarios.Source: §5 · p. 8