PaperlayerView exact revision on arXiv

Source-linked AI summary

Re$^3$Sim: Generating High-Fidelity Simulation Data via 3D-Photorealistic Real-to-Sim for Robotic Manipulation

Xiaoshen Han, Junqiu Yu, Minghuan Liu, Yilun Chen, Xiaoyang Lyu, Yang Tian, Bolun Wang, Weinan Zhang, Jiangmiao Pang

arXiv:2502.08645v4cs.RO

TL;DR

Real-world expert-data collection is costly, and simulation data often has geometric and visual gaps with the real world. RE3SIM combines 3D reconstruction, rendering, and physics-based simulation to generate training data; its zero-shot policies achieve a 58% average success rate with about 10 minutes of simulated collection.

  • Problem

    Real-world expert-data collection is costly and labor-intensive, while simulation data can scale but often exhibits sim-to-real gaps.

    Source: §I · p. 1

  • Method

    RE3SIM combines reconstructed scene geometry, photorealistic rendering, and physics-based simulation to create simulation data for manipulation policies.

    Source: §I · pp. 1–2, A. System Overview · p. 2

  • Results

    58% average success rate was achieved by zero-shot sim-to-real policies with about 10 minutes of simulated data collection.

    Source: §I · p. 2

  • Takeaways & Limitations

    The authors report that large simulated datasets can yield competitive real-world task success and may support pre-training large-scale robot models.

    Source: §VI · p. 7

    RE3SIM currently does not support reconstructing deformable objects or liquids and relies on manually defined physics parameters.

    Source: §VI · p. 7

Abstract

from arXiv · show

Real-world data collection for robotics is costly and resource-intensive, requiring skilled operators and expensive hardware. Simulations offer a scalable alternative but often fail to achieve sim-to-real generalization due to geometric and visual gaps. To address these challenges, we propose a 3D-photorealistic real-to-sim system, namely, RE$^3$SIM, addressing geometric and visual sim-to-real gaps. RE$^3$SIM employs advanced 3D reconstruction and neural rendering techniques to faithfully recreate real-world scenarios, enabling real-time rendering of simulated cross-view cameras within a physics-based simulator. By utilizing privileged information to collect expert demonstrations efficiently in simulation, and train robot policies with imitation learning, we validate the effectiveness of the real-to-sim-to-real pipeline across various manipulation task scenarios. Notably, with only simulated data, we can achieve zero-shot sim-to-real transfer with an average success rate exceeding 58%. To push the limit of real-to-sim, we further generate a large-scale simulation dataset, demonstrating how a robust policy can be built from simulation data that generalizes across various objects. Codes and demos are available at: https://re3sim.github.io/.

I. INTRODUCTION

Real-world expert-data collection is costly and labor-intensive, while simulation data scales but often leaves visual and geometric sim-to-real gaps. RE3SIM combines reconstruction, rendering, and physics-based simulation; its zero-shot policies achieve 58% average success using about 10 minutes of simulated data collection.

I. INTRODUCTION

  • Real-world expert data collection remains costly, time-consuming, and labor-intensive, while simulation data can scale with computational resources.Simulation data is presented as an appealing alternative, although sim-to-real gaps can make target-domain fine-tuning necessary.Source: §I · p. 1
  • RE3SIM combines photorealistic RGB rendering with precise 3D geometry reconstruction to replicate real-world scenarios.Gaussian rasterization renders visuals, multi-view stereo reconstructs geometry, physics engines handle dynamics, and a dedicated hybrid engine enables real-time rendering.Source: §I · pp. 1–2
  • The real-to-sim process uses mesh recovery, hybrid foreground-background rendering, and real-world alignment, with limited human involvement.People place ArUco markers and capture scene and object images or video, with an extra step to align the robot base.Source: §I · p. 2
  • RE3SIM reports scene reconstruction in under three minutes and rendering at 24 FPS at 480p across two independent camera views.The system uses a privileged policy to collect high-fidelity expert data after assets and robots enter the simulator.Source: §I · p. 2
  • 58% average success rate with about 10 minutes of simulated data collection validates zero-shot sim-to-real policies across tabletop tasks.The authors present these experiments as evidence of the pipeline's effectiveness.Source: §I · p. 2

II. BACKGROUND

The background describes approaches for recovering 3D geometry from images and related inputs. It distinguishes pose and sparse-point estimation, dense reconstruction, and commercial RGB-D SLAM tools.

II. BACKGROUND

  • Structure-from-motion approaches such as COLMAP can estimate camera poses and sparse point clouds for later dense reconstruction.Dense point clouds can then be reconstructed with traditional methods or network-based approaches.Source: A. 3D Reconstruction · p. 2
  • Commercial tools such as ARCode and PolyCam use RGB-D SLAM to estimate camera poses and depth information.They may also apply post-processing such as smoothing and inpainting to improve reconstructed model usability and quality.Source: A. 3D Reconstruction · p. 2
  • 3D reconstruction recovers scene geometry from inputs including images and point clouds.Source: A. 3D Reconstruction · p. 2

B. Rendering

The rendering background contrasts implicit and explicit 3D representations and describes 3D Gaussian Splatting's rasterization approach. RE3SIM uses separate background meshes for collision estimation and 3DGS for more realistic rendering.

B. Rendering

  • Implicit rendering methods such as NeRF represent 3D space with a neural-network density field and generate images through volumetric rendering.Explicit representations include meshes and point clouds with texture mapping.Source: B. Rendering · p. 2
  • 3D Gaussian Splatting represents scenes with Gaussian primitives and uses Gaussian rasterization for high-fidelity, efficient rendering.Each primitive is modeled as an ellipse with a full 3D covariance matrix; its color is depth-ordered and combined using alpha blending.Source: B. Rendering · p. 2

III. RE3SIM

  • RE3SIM separates background mesh reconstruction for collision estimation from 3DGS rendering to improve visual realism.The authors note that MVS background reconstruction can have suboptimal visual quality, while SuGaR may produce protrusions or indentations on flat planes; object segmentation is labor-intensive.Source: §III · p. 2

A. System Overview

RE3SIM combines mesh recovery, hybrid visual rendering, real-world alignment, and simulation data generation. Its mesh pipeline reconstructs background and object geometry separately, while hybrid rendering combines mesh-rendered foregrounds with 3DGS backgrounds for multiple views.

A. System Overview

  • RE3SIM uses reconstructed meshes for collision estimation, 3D Gaussian scene representations for rendering, and a physics-based simulator to generate manipulation data.The system covers background and foreground objects, not robots, whose accurate 3D assets are usually accessible; this initial study validates rigid-body tasks on a tabletop arm.Source: A. System Overview · pp. 2–3

B. Mesh Recovery

  • Background meshes are reconstructed from COLMAP poses and sparse points, then detailed with OpenMVS; foreground objects are segmented with ARCode.The background mesh is simplified for collision bodies, while void filling and smoothing address defects that can make simulated objects unstable; default physical parameters are used when mass and friction are difficult to acquire.Source: B. Mesh Recovery · p. 3

C. Hybrid Visual Rendering

  • Hybrid rendering uses ray-traced meshes for foreground objects and 3DGS for backgrounds, mixing them by ground-truth depth and supporting multiple camera views.Source: C. Hybrid Visual Rendering · p. 3

D. Real-World Alignment

Real-world alignment registers reconstructed scenes to the robot environment, while privileged-policy rollouts generate simulated expert datasets. The experiments ask whether RE3SIM supports high-fidelity transfer, challenging tasks, and rapid, low-cost data generation.

D. Real-World Alignment

  • Background alignment maps 3DGS coordinates to table ArUco markers, then ICP refines the transform between depth-camera and reconstructed-mesh point clouds.Foreground objects instead use predefined simulation placement ranges.Source: D. Real-World Alignment · p. 3

E. Expert Data Collection

  • Privileged expert rollouts randomize objects and robot-base position, plan key-step paths with RRTConnect, and reject failures to build observation-action datasets.The expert uses privileged target poses, while policy observations remain real-world-accessible images and proprioception.Source: E. Expert Data Collection · p. 4

IV. EXPERIMENT

  • The evaluation asks whether RE3SIM narrows manipulation sim-to-real gaps, improves challenging real-world tasks with large-scale simulation data, and enables rapid, low-cost scene and data construction.Source: §IV · p. 4

A. Experimental Setups

The evaluation uses a Franka tabletop platform and three manipulation tasks to test zero-shot transfer from RE3SIM data. Reported outcomes include strong transfer against existing methods and synthetic-data performance comparable to real-data training.

A. Experimental Setups

  • Experiments used a Franka Research 3 with wrist-mounted and side-view RealSense D435i cameras; ACTstructured with DINOV2 used visual augmentation and temporal ensembling.Source: A. Experimental Setups · p. 4

B. Zero-Shot Sim-to-Real Experiments

C. Large-Scale Sim-to-Real Experiments

In the long-horizon table-clearing task, RE3SIM-trained policies generalized to novel objects and outperformed RialTo-IL across the tested settings. Increasing the synthetic dataset size improved imitation-learning performance, while the policy also showed robustness to varying lighting.

Task and evaluation

  • The evaluation used a long-horizon task in which the robot cleared objects from a 40cm × 50cm tabletop by placing them in a basket.The task required sequentially picking seen and unseen objects; setups included seen, unseen, cluttered, and darker-than-simulation conditions.Source: C. Large-Scale Sim-to-Real Experiments · p. 5

Result analysis

  • RE3SIM-trained policies outperformed RialTo-IL across all tested settings and generalized to novel, similar-sized objects despite shape or color variation.The authors hypothesize that background color contrast aids detection and that similar grasping positions support successful grasps.Source: C. Large-Scale Sim-to-Real Experiments · p. 5
  • The policy showed robust performance under varying lighting conditions.Source: C. Large-Scale Sim-to-Real Experiments · p. 6

Dataset size

  • Increasing RE3SIM synthetic dataset size significantly improved imitation-learning performance, with real-world performance nearly negligible when data was limited.Doubling the data often significantly improved success rate until convergence at high performance.Source: C. Large-Scale Sim-to-Real Experiments · p. 6

D. Sim-Real Consistency

RE3SIM policies' simulated and real-world performance were more strongly correlated than RialTo's in a bottle pick-and-drop evaluation. Higher simulated success tended to accompany similar real-world success, though alignment errors or fewer real-world test instances may affect the comparison.

Evaluation setup

  • The comparison evaluated ACT policies across training stages from 0 to 120000 steps and three seeds on bottle pick-and-drop; alignment errors or fewer real-world test instances may introduce bias.Source: D. Sim-Real Consistency · p. 6

Performance consistency

  • RE3SIM's Pearson correlation was 0.924, versus 0.710 for RialTo, between simulated and real-world performance.Source: D. Sim-Real Consistency · p. 6
  • Higher simulated success tended to correspond to similar real-world success, while low simulated success often coincided with difficulty performing well in the real world.Source: D. Sim-Real Consistency · p. 6

E. System Efficiency

RE3SIM reconstructs scenes from captured images or video and generates simulation data more efficiently than real-world teleoperation. With two cameras, it runs at approximately 24 FPS, with 3DGS rendering accounting for about one-third of each step.

Human effort

  • Background reconstruction uses images or a short video plus one depth-aligned image, while object reconstruction uses ARCode.Video input saves time but may slightly reduce quality through motion blur; images yield better results with slightly more effort.Source: E. System Efficiency · pp. 6–7

Time consumption

  • RE3SIM runs at approximately 24 FPS with two cameras; rendering 480p images from both views using 3DGS adds 12.93 milliseconds, about one-third of each step.Source: E. System Efficiency · p. 7

Data collection cost

  • Collecting 100 simulation episodes per task takes less time than real-world teleoperation, which primarily requires expert data-collection time.The simulation collection timings use eight RTX 4090 GPUs.Source: E. System Efficiency · p. 7

F. Impact of Mesh-Based vs. 3DGS Rendering Methods

Using 3DGS for foreground rendering instead of ARCode-reconstructed meshes offered no appreciable task-success benefit, so ARCode meshes remain the default.

Rendering ablation

V. RELATED WORK

Prior work uses reconstruction, 3D Gaussian splatting, and simulated data to support robotic manipulation and policy learning. RE3SIM combines geometric and visual reconstruction and reports real-world validation of policies trained on simulated data.

V. RELATED WORK

  • Prior approaches use RGB-based reconstruction for grasping, 3D Gaussian splatting for rendering, or single-image reconstruction to generate training data.The cited examples include Evo-NeRF and LERF-TOGO, GaussianGrasper and GraspSplats, and URDFormer and Articulate Anything.Source: §V · p. 7
  • RoboStudio, SplatSim, and RoboGSim use multi-view imagery or video for reconstruction, but their stated scopes differ from RE3SIM’s real-world policy validation.The passage notes SplatSim’s reliance on preobtained 3D models and describes RoboGSim’s sim-to-real validation as limited.Source: §V · p. 7

VI. CONCLUSION

  • RE3SIM combines geometric and visual reconstruction and validates policies trained on simulated data through real-world experiments.The conclusion also reports competitive success rates for models trained on large simulated datasets compared with models trained on real-world data.Source: §VI · p. 7
  • RE3SIM does not support deformable objects or liquids, uses manually defined physics parameters, and relies on rule-based data-collection policies that become impractical as tasks grow more complex.The authors leave addressing these limitations to future work.Source: §VI · p. 7
Loading 2502.08645v4…