PaperlayerView exact revision on arXiv

Source-linked AI summary

GSWorld: Closed-Loop Photo-Realistic Simulation Suite for Robotic Manipulation

Guangqi Jiang, Haoran Chang, Ri-Zhao Qiu, Yutong Liang, Mazeyu Ji, Jiyue Zhu, Zhao Dong, Xueyan Zou, Xiaolong Wang

arXiv:2510.20813v1cs.ROcs.AIcs.CV

TL;DR

Manipulation-policy development must balance simulation’s aligned actions, real data’s realism, and teleoperation’s cost and limited scalability. GSWorld combines 3D Gaussian Splatting, physics, and a bidirectional real-to-sim-to-real pipeline to close this loop. It supports zero-shot sim-to-real learning, DAgger adaptation, and reproducible benchmarking, with real-world visual-RL success rates exceeding the ManiSkill baseline on two tasks.

  • Problem

    Simulation, human videos, and teleoperation each provide only part of the realism, action alignment, or scalability needed for manipulation-policy development.

    Source: §I · p. 1

  • Method

    GSWorld couples photo-realistic 3D Gaussian Splatting with physics in a bidirectional pipeline that reconstructs digital twins and preserves native robot observation and control spaces.

    Source: §I · pp. 1–2

  • Results

    Real-world success rates reached 30% for Grasp Banana and 20% for Tidy Table, compared with 0% and 5% for ManiSkill, respectively.

    Source: E. Visual Reinforcement Learning · p. 7

  • Takeaways & Limitations

    GSWorld supports zero-shot sim-to-real imitation learning and visual reinforcement learning, closed-loop DAgger correction, and reproducible visual benchmarking.

    Source: §I · p. 2, §V · p. 7

Abstract

from arXiv · show

This paper presents GSWorld, a robust, photo-realistic simulator for robotics manipulation that combines 3D Gaussian Splatting with physics engines. Our framework advocates "closing the loop" of developing manipulation policies with reproducible evaluation of policies learned from real-robot data and sim2real policy training without using real robots. To enable photo-realistic rendering of diverse scenes, we propose a new asset format, which we term GSDF (Gaussian Scene Description File), that infuses Gaussian-on-Mesh representation with robot URDF and other objects. With a streamlined reconstruction pipeline, we curate a database of GSDF that contains 3 robot embodiments for single-arm and bimanual manipulation, as well as more than 40 objects. Combining GSDF with physics engines, we demonstrate several immediate interesting applications: (1) learning zero-shot sim2real pixel-to-action manipulation policy with photo-realistic rendering, (2) automated high-quality DAgger data collection for adapting policies to deployment environments, (3) reproducible benchmarking of real-robot manipulation policies in simulation, (4) simulation data collection by virtual teleoperation, and (5) zero-shot sim2real visual reinforcement learning. Website: https://3dgsworld.github.io/.

I. INTRODUCTION

GSWorld closes the manipulation-policy development loop by combining photo-realistic perception, native robot action spaces, and physics-based simulation. It supports sim-to-real learning, failure-driven adaptation, and reproducible benchmarking without requiring every policy iteration on physical robots.

Motivation

  • The paper identifies simulation, human videos, and teleoperation as complementary data sources whose alignment, realism, and scalability trade-offs motivate GSWorld.Teleoperation aligns perception and actions but is costly and difficult to scale.Source: §I · p. 1

Core approach

  • GSWorld combines 3D Gaussian Splatting with physics to narrow visual and action-space gaps for manipulation policies.Policies perceive photo-realistic renderings while issuing controls in the robot’s native space.Source: §I · p. 1
  • The bidirectional pipeline reconstructs metric-accurate digital twins, aligns robot URDFs and collision meshes, and deploys policies without interface translation.It also evaluates visual, geometric, and functional similarity between real and simulated worlds.Source: §I · p. 2

Applications

  • GSWorld enables zero-shot sim-to-real transfer for imitation learning and visual reinforcement learning while accelerating data generation through scalable parallel environments.The framework also supports multiple simulation data-collection methods, including motion planning and teleoperation.Source: §I · p. 2
  • Its closed-loop DAgger workflow reproduces deployment failures in the digital twin and collects targeted corrective labels with less teleoperation overhead.This supports iterative improvement of real-world policies using failure-focused data.Source: §I · p. 2
  • GSWorld enables reproducible visual benchmarking through shared assets, fixed camera parameters, consistent rendering conditions, and standardized action semantics.The framework aims to separate algorithmic progress from environmental variance across robots, scenes, and tasks.Source: §I · p. 2

II. RELATED WORK

Prior work improves simulation fidelity, reconstructs real scenes, or studies articulated and physical properties, but often remains scene-specific or lacks a portable, unified benchmark interface. GSWorld formulates learning from 3DGS-rendered observations across imitation learning, reinforcement learning, and DAgger.

II. RELATED WORK

  • Robotics simulators increasingly improve efficiency and rendering fidelity, while GSWorld focuses on combining 3D Gaussian Splatting with physics simulators.Related directions include accelerated engines, ray tracing, generative tools, and unified simulator interfaces.Source: §II · p. 2
  • Real2Sim methods span photo-realistic 3D reconstruction and part-level articulated-object understanding.These categories virtualize real-world assets rather than modeling all scene elements from the ground up.Source: §II · p. 2
  • NeRF-, mesh-, and generative-based approaches face deformation, artifact, labor, or scalability limitations, while GS-based simulators improve physical displacement of photo-realistic appearance.SplatSim demonstrates zero-shot sim-to-real but relies on manual segmentation and is overfitted to a single scene; GSWorld adds a unified asset format and broader assets.Source: §II · pp. 2–3
  • GSWorld uses an existing simulator interface with GSDF assets for standard depth, segmentation, collision computation, and photo-realistic RGB rendering.The rendering wrapper preserves simulator functionality while adding photo-realistic visuals.Source: Figure 2 · p. 3
  • Other Real2Sim work estimates articulation, deformable or rigid-body physics, and robot kinematics from observations using learned models, videos, robot-camera setups, or differentiable rendering.These approaches are orthogonal to GSWorld’s photo-realistic simulation focus.Source: §II · p. 3

A. Problem Formulation

  • The problem formulation replaces real RGB observations with 3DGS-rendered images for downstream IL, RL, and DAgger while retaining robot proprioception and native control.A reconstructed real scene can render novel views under arbitrary camera poses and environment states.Source: A. Problem Formulation · p. 3

B. Recipe for Real2Sim Reconstruction

GSWorld constructs GSDF assets through a streamlined real-to-sim reconstruction recipe designed to reduce manual alignment and support geometrically accurate, metric-scale scenes. The procedure aligns cameras, robots, tables, and movable objects with simulation representations.

Overview

  • The recipe targets an easy-to-use, streamlined procedure and incorporates geometric-accuracy advances in 3DGS for reusable GSDF construction.It is designed to reduce manual alignment effort beyond prior single-scene reconstruction workflows.Source: B. Recipe for Real2Sim Reconstruction · p. 4

Collecting Training Views

  • GSWorld collects robot and mobile-phone views while recording the robot’s joint pose during scene capture.The views include wrist-camera and third-person-camera observations.Source: B. Recipe for Real2Sim Reconstruction · p. 4

Aligning Scale for Metric Representation

  • ArUco markers resolve COLMAP scale ambiguity, establish metric scale, identify the support surface, and help estimate gravity direction.Known marker dimensions are used to scale the 3DGS point cloud.Source: B. Recipe for Real2Sim Reconstruction · p. 4

Aligning Robots and Table

  • ICP aligns the metric-scale robot URDF with the reconstructed robot, after which K-NN segments robot links in the Gaussian representation.Fixing scale reduces the degrees of freedom compared with previous alignment methods.Source: B. Recipe for Real2Sim Reconstruction · p. 4

Object Assets

  • Movable-object assets combine existing DTC and YCB datasets with custom 2DGS and mesh reconstructions.Mass is estimated by weighing, while unobserved bottoms may be completed through amodal reconstruction or 3D object generation.Source: B. Recipe for Real2Sim Reconstruction · p. 4

C. Applications - Closing the Loop for Developing Visual Manipulation Policies

GSWorld closes the policy-development loop by combining simulation-based failure recovery, real-world data refinement, and scalable photo-realistic benchmarking across embodiments.

Closed-loop DAgger Training

  • GSWorld samples recoverable states from recorded failures and uses motion planning to generate corrective DAgger data in the target deployment environment.The workflow rolls out scripted policies, records failure trajectories, samples recovery states uniformly, and obtains corrective actions.Source: C. Applications - Closing the Loop for… · p. 4
  • Simulation records and relives rollout failures using privileged information, enabling targeted corrective-data collection without repeatedly reproducing physical setups.Source: Figure 3 · p. 4, C. Applications - Closing the Loop for… · p. 4
  • The iterative loop evaluates policies, records failures, recovers them, and aggregates simulation observations and action labels into dataset τS.GSWorld-rendered observations are Igs, while Qs and As denote simulated robot joint positions and action labels.Source: C. Applications - Closing the Loop for… · p. 4
  • For policies trained from real-world data, the same improvement loop produces dataset τR from teleoperated real-world trajectories.Source: C. Applications - Closing the Loop for… · p. 4

Visual Benchmarking

Reinforcement Learning

IV. EXPERIMENTS

The experiments evaluate GSWorld across multiple embodiments, manipulation tasks, policy settings, and downstream questions spanning sim-to-real transfer, benchmarking, teleoperation, and reinforcement learning.

Research Questions

  • The experiments test zero-shot imitation transfer, closed-loop DAgger improvement, visual benchmarking, virtual teleoperation, and visual reinforcement learning.Source: §IV · p. 5

Hardware Platforms

  • FR3 uses front and wrist cameras, while xArm6 uses side and wrist cameras for real-world visual observations.Source: Figure 4 · p. 5

Experiment Protocol

  • GSWorld is evaluated on four FR3 tasks and three xArm6 tasks, with the bimanual R1 used for virtual simulation teleoperation.The study also uses ACT and Pi0 to demonstrate policy-agnostic evaluation and varies training settings and data sizes for benchmarking.Source: §IV · p. 5, Figure 5 · p. 5, Figure 6 · p. 5
  • Policies are trained with varying settings and data sizes to compare stronger and weaker policies for visual benchmarking.Source: §IV · p. 5

Manipulation Tasks Design

  • The evaluation includes four FR3 tasks and three xArm tasks, with task visualizations documenting the corresponding real-world setups.Source: Figure 5 · p. 5, Figure 6 · p. 5, §IV · p. 5
  • The task suite covers distinct tabletop manipulation skills involving bottles, boxes, sauce, bread, cans, racks, bananas, spoons, and cutting boards.Source: §IV · p. 5

A. Zero-shot Sim2real Imitation Learning

The zero-shot imitation experiment trains policies only on simulation data and evaluates whether GSWorld supports direct transfer to real manipulation tasks.

Zero-shot Sim2real Imitation Learning

  • GSWorld enables zero-shot sim-to-real policy transfer, with promising success rates after training on simulation data only.The experiment collects 100 trajectories per iteration and trains on the accumulated generated data.Source: A. Zero-shot Sim2real Imitation Learning · p. 5

B. Closed-loop DAgger for Continuous Policy Improvement

GSWorld uses closed-loop DAgger to improve manipulation policies with corrective data collected in simulation, including adaptation of deployed real-world policies. Its visual benchmarking shows that simulation performance tracks real-world outcomes across policies and tasks.

B. Closed-loop DAgger for Continuous Policy Improvement

C. Benchmarking

D. Virtual Teleoperation

GSWorld supports scalable simulation data collection through mouse-and-keyboard teleoperation and provides a parallel visual-RL environment. Its demonstrations include virtual teleoperation, SAC training, and improved real-world success over ManiSkill on two tasks.

D. Virtual Teleoperation

E. Visual Reinforcement Learning

V. CONCLUSION

  • GSWorld delivers highly correlated simulation and real-world performance metrics while enabling corrective-data collection at scale.Source: §V · p. 7

APPENDIX

The appendix describes Gaussian Splatting representations, view-dependent color, feature rendering, camera projection, and covariance transformation for neural rendering.

A. Neural Rendering Background

B. More Experimental Details

The appendix specifies evaluation protocols, policy implementations, Gaussian-training settings, and data-collection procedures for simulation and real-world experiments.

B. More Experimental Details

C. Gaussian Splatting Training

  • Gaussian training uses the official implementation without modification, with approximately 100 object images and 300 robot images, initialized from Colmap and saved at iterations 7,000 and 30,000.Source: C. Gaussian Splatting Training · p. 10

D. GSWorld Data Collection

  • Simulation data collection primarily uses MPlib motion planning over decomposed subtasks, while keyboard-and-mouse teleoperation supports keyframes and planner-assisted poses.Source: D. GSWorld Data Collection · p. 10
  • Real-world demonstrations use VR-based 6-DOF hand tracking mapped to end-effector poses, with ACE-F used for the xArm6 platform.Source: D. GSWorld Data Collection · p. 10

E. GSWorld Usage example

GSWorld can be integrated into existing Gym environments with a lightweight wrapper, after which it is used through the standard Gym interface for pixel-based interaction.

Loading 2510.20813v1…