PaperlayerView exact revision on arXiv

Source-linked AI summary

GaussGym: An open-source real-to-sim framework for learning locomotion from pixels

Alejandro Escontrela, Justin Kerr, Arthur Allshire, Jonas Frey, Rocky Duan, Carmelo Sferrazza, Pieter Abbeel

arXiv:2510.15352v1cs.ROcs.AIcs.GR

TL;DR

Visual locomotion and navigation simulation is limited by the speed and fidelity of existing visual pipelines, often forcing reliance on depth or geometric inputs. GaussGym integrates 3D Gaussian Splatting with vectorized physics and diverse real or generated environments to train policies directly from RGB. It supports high-throughput photorealistic simulation, semantic navigation, and initial sim-to-real transfer, while remaining limited by imperfect world-model outputs and restricted transfer generalization.

  • Problem

    Existing simulators often render visual information too slowly or inaccurately, restricting perceptive locomotion policies to depth, LiDAR, or geometric inputs.

    Source: §2.1 · p. 3

  • Method

    GaussGym integrates photorealistic 3D Gaussian-Splatting rendering with vectorized physics and accepts real-world scans, datasets, and generative video environments for RGB policy training.

    Source: §1 · p. 2

  • Results

    GaussGym trains visual locomotion and navigation policies, improves stair climbing with auxiliary geometry reconstruction, avoids undesirable regions, and shows initial zero-shot real-world stair-climbing transfer.

    Source: §1 · pp. 2–3, §4.2.2 · p. 8

  • Takeaways & Limitations

    GaussGym provides an open baseline for scalable visual locomotion and navigation research using diverse photorealistic training worlds.

    Source: §1 · p. 2, §6 · p. 9

    Video-model world generation can be inconsistent and lacks sufficient camera control, while the current pipeline cannot simulate dynamic scenes, fluids, or deformable assets.

    Source: §5 · p. 9

Abstract

from arXiv · show

We present a novel approach for photorealistic robot simulation that integrates 3D Gaussian Splatting as a drop-in renderer within vectorized physics simulators such as IsaacGym. This enables unprecedented speed -- exceeding 100,000 steps per second on consumer GPUs -- while maintaining high visual fidelity, which we showcase across diverse tasks. We additionally demonstrate its applicability in a sim-to-real robotics setting. Beyond depth-based sensing, our results highlight how rich visual semantics improve navigation and decision-making, such as avoiding undesirable regions. We further showcase the ease of incorporating thousands of environments from iPhone scans, large-scale scene datasets (e.g., GrandTour, ARKit), and outputs from generative video models like Veo, enabling rapid creation of realistic training worlds. This work bridges high-throughput simulation and high-fidelity perception, advancing scalable and generalizable robot learning. All code and data will be open-sourced for the community to build upon. Videos, code, and data available at https://escontrela.me/gauss_gym/.

1 INTRODUCTION

GaussGym addresses the limited visual fidelity and speed of existing simulators by combining photorealistic rendering with high-throughput physics for RGB-based locomotion and navigation. It supports diverse environments and reports improved visual-policy learning, semantic navigation, and initial sim-to-real transfer.

  • Existing simulators often make visual simulation too slow or inaccurate, leading perceptive locomotion policies to rely mainly on LiDAR, depth, or geometric inputs.Source: §1 · p. 2
  • GaussGym digitizes real-world and video-model environments for photorealistic physics simulation and RGB-based locomotion and navigation learning.It accepts smartphone scans, SLAM captures, 3D datasets, handheld videos, and generative video outputs.Source: §1 · p. 2
  • The framework trains locomotion and navigation policies for humanoid and quadrupedal robots and demonstrates initial zero-shot transfer of visual locomotion to real-world stair climbing.Source: §1 · p. 2
  • GaussGym provides a fast open-source simulator with 2,500 scenes from manual scans, open-source datasets, and generative video models.Source: §1 · p. 2
  • Geometry reconstruction as an auxiliary task significantly improves stair-climbing performance, while RGB navigation policies avoid undesirable regions invisible to depth-only policies.Source: §1 · pp. 2–3

2 RELATED WORK

Prior work established scalable physics simulation and radiance-field representations, but visual locomotion remains constrained by throughput, scene diversity, and the sim-to-real gap. GaussGym builds on these directions by integrating 3D Gaussian Splatting with vectorized physics and broad scene sources.

2.1 SIM-TO-REAL RL FOR LOCOMOTION

  • GPU-accelerated simulators democratized reinforcement-learning training for locomotion and navigation through consumer-grade parallel simulation.Source: §2.1 · p. 3
  • Most deployed locomotion policies still use geometric and proprioceptive inputs because visual sim-to-real transfer, diverse assets, and required training throughput remain challenging.Source: §2.1 · p. 3

2.2 SCENE GENERATION

  • Procedural terrains and textured-mesh pipelines support geometric locomotion but generally lack meaningful visual appearance or high visual fidelity.Source: §2.2 · p. 3
  • NeRF-based real-to-sim approaches improve visual fidelity but can be computationally expensive, while prior 3DGS systems and LucidSim provide narrower or less scalable integrations.Source: §2.2 · p. 3, §2.3 · p. 5

2.3 RADIANCE FIELDS IN ROBOTICS

  • Radiance fields offer high-quality scene representations for navigation and RGB-based control, and 3D Gaussian Splatting mitigates neural-field slowness through rapid GPU rasterization.Source: §2.3 · p. 5

3 GAUSSGYM

GaussGym converts diverse real and synthetic visual data into aligned Gaussian-splat renderings and collision meshes inside vectorized physics simulation. Its rendering and scheduling optimizations support photorealistic RGB and depth at high throughput across many environments.

3 GAUSSGYM

  • GaussGym standardizes posed datasets, smartphone scans, and video-model sequences with VGGT, then produces meshes and Gaussian splats in a shared global frame.VGGT estimates camera parameters, dense point clouds, and normals; NKSR generates meshes while splats initialize from VGGT points.Source: §3 · pp. 5–6, §3.1 · p. 6

3.2 3D GAUSSIAN SPLATTING AS A DROP-IN RENDERER

  • Gaussian Splatting serves as a vectorized drop-in renderer that produces photorealistic visuals with minimal overhead and synchronized physics-based collision handling.Source: §3 · p. 6, §3.2 · p. 6

3.1 DATA COLLECTION AND PROCESSING

  • The pipeline supports ARKitScenes, GrandTour, smartphone captures, and Veo-generated sequences, enabling photorealistic environments from diverse data sources.Source: Figure 4 · p. 5, §3.1 · p. 6

3.3 OPTIMIZATIONS FOR HIGH-THROUGHPUT AND REALISM

  • Rendering at the camera frame rate instead of the control rate improves speed, while alpha-blending velocity-offset frames simulates motion blur for more realistic observations.Source: §3.3 · pp. 6–7

3.2 3D GAUSSIAN SPLATTING AS A DROP-IN RENDERER

  • Rendering RGB and depth together adds no rendering time because depth is a by-product of Gaussian-Splatting rasterization.Source: §3.2 · p. 6, Figure 5 · p. 7

3.3 OPTIMIZATIONS FOR HIGH-THROUGHPUT AND REALISM

  • 100,000 simulator steps per second are achieved on one RTX 4090 across 4,096 environments and 128 scenes, with near-linear multi-GPU scaling.Source: §3.3 · p. 7

4 RESULTS

GaussGym combines photorealistic environments, recurrent RGB-proprioceptive policies, and auxiliary geometric prediction for visual locomotion and navigation. It supports generated worlds, sim-to-real transfer, semantic avoidance, and large-scale training across scenes.

4.1 TRAINING ENVIRONMENTS BEYOND REALITY

  • GaussGym generates entirely new photorealistic worlds from video models, including environments difficult or impossible to capture in reality.Veo’s multi-view consistency and VGGT’s camera estimation and dense point-cloud generation enable these scenes.Source: §4.1 · p. 7

4.2 VISUAL LOCOMOTION AND NAVIGATION

  • The framework trains visual locomotion and navigation policies end-to-end from RGB in diverse, visually complex terrains using asymmetric actor-critic learning.The evaluation focuses on visual stair climbing and sparse-goal navigation without student-teacher distillation or multi-stage training.Source: §4.2 · p. 7

4.2.1 NEURAL ARCHITECTURE

  • An LSTM fuses proprioception with DinoV2 RGB embeddings, then feeds occupancy-terrain reconstruction and Gaussian-action policy heads.The voxel head predicts dense occupancy and terrain heights, while a second LSTM outputs joint-position offset distributions.Source: §4.2.1 · pp. 7–8

4.2.2 VISUAL LOCOMOTION RESULTS

  • RGB-trained policies precisely place quadruped feet on stairs, avoid riser collisions, match commanded velocities, and transfer to real hardware without fine-tuning.A Booster T1 policy trained with a head-mounted camera also learns to navigate slopes.Source: Figure 6 · p. 7, §4.2.2 · p. 8

4.2.3 VISUAL NAVIGATION RESULTS

  • RGB policies avoid penalty regions invisible to depth-only policies, demonstrating semantic navigation cues beyond geometric depth.In sparse-goal obstacle fields, the RGB policy avoids a yellow floor patch whereas the depth-only policy walks through it.Source: §4.2.3 · p. 8, Figure 8 · p. 9
  • Ablations show that voxel regression, a pretrained DINO encoder, and training on many scenes each improve performance across four terrain scenarios.Using all scenes outperforms training with 10% or 50% of the scenes.Source: §4.2.3 · p. 8

5 LIMITATIONS

GaussGym remains limited by sim-to-real generalization, reward specification, physical-material modeling, and the capabilities of its vision and world-generation models.

5 LIMITATIONS

  • Visual transfer remains difficult: the walking policy was not tested on unseen staircases, and precise foot placement declined after real-world transfer.Real hardware also introduces image latency and egocentric-observation challenges.Source: §5 · p. 9
  • GaussGym lacks automated cost or reward generation for visually critical tasks and therefore relies on hand-crafted cost terms.The paper identifies social-norm tasks such as sidewalks and crosswalks as examples.Source: §5 · p. 9
  • Uniform physical parameters prevent accurate simulation of surfaces such as ice, mud, and sand, weakening the link between appearance and physical feel.Source: §5 · p. 9
  • Video-model world generation inherits inconsistent outputs and limited text-only camera control, and cannot yet represent dynamic scenes, fluids, or deformable assets.The current pipeline remains limited to simple rigid-body physics.Source: §5 · p. 9

6 CONCLUSION

GaussGym is presented as a fast, open-source photorealistic simulator for training RGB-based visual locomotion and navigation. Its policies show vision-based behavior in simulation and partial real-world transfer across diverse scene sources.

6 CONCLUSION

  • GaussGym provides an open baseline for training visual navigation and locomotion policies directly from RGB using scenes from deployments, scans, video-generation models, and datasets.The authors position it as infrastructure intended to accelerate research in vision-based locomotion and navigation.Source: §6 · p. 9

A.1 ADDITIONAL SCENES AND MOTION BLUR

GaussGym extends its scene library with large photorealistic scans and simulates motion blur by blending frames along camera motion.

A.1 ADDITIONAL SCENES AND MOTION BLUR

  • GrandTour supplies large photorealistic scenes, including a 20 m2 environment with mesh and robot-view renders.Source: Figure 9 · p. 15
  • The motion-blur renderer alpha-blends frames sampled along the camera-velocity direction using shutter speed and camera velocity.The effect is especially pronounced during jerky motion, such as foot contact with stairs.Source: Figure 10 · p. 15

A.2 POLICY LEARNING

Policies are trained with shared locomotion reward terms and task-specific rewards for velocity and goal tracking. The visual locomotion policy also learns stair negotiation by placing the A1’s feet securely from vision.

Goal tracking

  • Goal tracking results are reported for both A1 and T1, with bold numbers marking the best performance in each scenario.Source: Table 2 · p. 16

Visual locomotion

  • The visual locomotion policy places the A1’s front and hind feet without stumbling on a stair edge, leading with a large front-foot step onto the middle of the second step.The trajectory indicates that safe footholds can be inferred directly from vision.Source: Figure 11 · p. 16

Shared rewards

  • Training uses general reward terms covering angular velocity, body orientation, commanded actions, joint angles, foot positions, contacts, gait phase, and foot contact force.The associated mathematical expressions and weights are specified for locomotion-policy training.Source: Table 3 · p. 16

Velocity tracking

  • Velocity tracking rewards compare current and desired base velocities and yaw rates.Source: Table 4 · p. 17

Goal tracking

  • Goal tracking rewards compare current and desired base positions and yaws while accounting for the remaining time to reach the goal.The reward design is based on Hoeller et al. (2024).Source: Table 5 · p. 17

Observations

  • The policy-learning setup specifies joint positions and velocities alongside observations used across all tasks.Source: §A.2 · p. 17, Table 6
Loading 2510.15352v1…