PaperlayerView exact revision on arXiv

Source-linked AI summary

GWM: Towards Scalable Gaussian World Models for Robotic Manipulation

Guanxing Lu, Baoxiong Jia, Puhao Li, Yixin Chen, Ziwei Wang, Yansong Tang, Siyuan Huang

arXiv:2508.17600v2cs.ROcs.AIcs.CVcs.LG

TL;DR

Image-based world models can lack the 3D geometric and spatial understanding needed for robotic manipulation, while offline 3D reconstruction poses scalability challenges. GWM predicts future scenes by modeling Gaussian primitives under robot actions, using a DiT and 3D-aware VAE. It improves reported simulation and real-world policy results over prior baselines.

  • Problem

    Image-based world models lack robust 3D geometric and spatial understanding, and offline per-scene 3D reconstruction is computationally challenging to scale for robotic manipulation.

    Source: §1 · p. 2

  • Method

    GWM combines a Diffusion Transformer with a 3D Gaussian VAE to predict future Gaussian scene states conditioned on observations and robot actions.

    Source: §1 · p. 2, §5 · p. 8

  • Results

    16.25% improvement over previous state-of-the-art baselines in two simulation environments; GWM also improves a typical diffusion policy by 30% with 20 trials in real-world scenarios.

    Source: §1 · p. 2

  • Takeaways & Limitations

    GWM serves both as a visual representation for imitation learning and as a neural simulator for model-based reinforcement learning.

    Source: §1 · p. 2, §5 · p. 8

Abstract

from arXiv · show

Training robot policies within a learned world model is trending due to the inefficiency of real-world interactions. The established image-based world models and policies have shown prior success, but lack robust geometric information that requires consistent spatial and physical understanding of the three-dimensional world, even pre-trained on internet-scale video sources. To this end, we propose a novel branch of world model named Gaussian World Model (GWM) for robotic manipulation, which reconstructs the future state by inferring the propagation of Gaussian primitives under the effect of robot actions. At its core is a latent Diffusion Transformer (DiT) combined with a 3D variational autoencoder, enabling fine-grained scene-level future state reconstruction with Gaussian Splatting. GWM can not only enhance the visual representation for imitation learning agent by self-supervised future prediction training, but can serve as a neural simulator that supports model-based reinforcement learning. Both simulated and real-world experiments depict that GWM can precisely predict future scenes conditioned on diverse robot actions, and can be further utilized to train policies that outperform the state-of-the-art by impressive margins, showcasing the initial data scaling potential of 3D world model.

1. Introduction

Robotic manipulation needs world models that combine fine visual detail with robust 3D spatial information, while existing 3D reconstruction methods pose scalability challenges. GWM combines Gaussian Splatting with generative models to predict future scenes and support policy learning.

Motivation

  • Image-based world models lack 3D geometric and spatial understanding, while offline per-scene reconstruction makes 3D representations computationally challenging to scale for manipulation.The introduction identifies the need to retain fine visual details alongside 3D spatial information.Source: §1 · p. 2

Approach

  • GWM combines 3D Gaussian Splatting with high-capacity generative models to reconstruct future scenes conditioned on current observations and robot actions.A 3D Gaussian VAE provides compact latent representations for efficient diffusion-model training and inference.Source: §1 · p. 2

Evaluation

  • The evaluation covers 31 robotic tasks across 3 domains, including a real-world Franka PnP suite with 20 in-domain and out-of-domain variations.The ablations assess both perceptual metrics and success rates.Source: §1 · p. 2

Main contributions

  • GWM improves previous state-of-the-art baselines by 16.25% in two simulation environments and improves a typical diffusion policy by 30% with 20 trials in real-world scenarios.The paper reports these results alongside extensive experiments across simulation and real-world manipulation.Source: §1 · p. 2

2. Related Work

Prior world models span latent prediction and direct pixel-space generation, while Gaussian Splatting offers explicit, spatially meaningful scene representations. Related work also highlights scalability challenges in reconstruction and in transferring visual manipulation policies to real-world settings.

World Models

Gaussian Splatting

  • 3D Gaussian Splatting [35] offers efficient, spatially meaningful scene representations, but offline per-scene reconstruction and reliance on known camera poses constrain scalability.Feed-forward novel-view synthesis from unposed images [8] [37] [71] [80] is a related direction; this work develops a scalable Gaussian world model from unposed images.Source: §2 · p. 3

Visual Manipulation

  • Visual imitation learning uses representations including point clouds [7] [15], voxels [44], NeRFs [11] [32] [41] [69], and 3D-GS [49], but can struggle in unseen real-world scenarios [53] [54].RL refinement requires costly real-world rollouts, while sim-to-real approaches face dependence on predefined assets [58] [79] or labor-intensive object conversion [38] [48] [63].Source: §2 · p. 3

3. Gaussian World Model

GWM encodes observed scenes as 3D Gaussian representations, models future states with action-conditioned diffusion in a compact latent space, and integrates the model into imitation learning and model-based reinforcement learning.

3.1. World State Encoding

  • GWM uses feed-forward reconstruction to obtain 3D Gaussian scene states from images, avoiding vanilla 3D-GS’s time-consuming per-scene offline optimization.Splatt3R uses Mast3R point maps and an additional prediction head to predict Gaussian parameters.Source: §3.1 · p. 3
  • A Gaussian VAE maps variable-sized scene representations to a fixed-length latent set and decodes them back into reconstructed Gaussians.Farthest Point Sampling selects a fixed number of Gaussians; cross-attention aggregates scene information, and the VAE uses Chamfer and rendering losses.Source: §3.1 · pp. 3–4

3.2. Diffusion-based Dynamics Modeling

  • The dynamics model generates a future latent state from noise, conditioned on history states and actions, using a diffusion process and reverse-time sampling.The forward perturbation adds Gaussian noise to the ground-truth future state, and a learned denoiser estimates the clean state.Source: §3.2 · p. 4
  • EDM preconditioning lets the denoising network receive noise levels as conditioning and adaptively mixes signal and noise across the noise schedule.The method uses a preconditioned network Fθ rather than directly learning the denoiser Dθ.Source: §3.2 · p. 4

3.3. GWM for Policy Learning

  • For model-based reinforcement learning, GWM models state and reward dynamics from policy rollouts, adding a reward-prediction head to the world model.The policy is optimized using the predictive model.Source: §3.3 · p. 5, §3.2 · p. 5
  • For imitation learning, GWM supplies policy features from the first diffusion denoising step, with actions predicted in sequential chunks.The first denoising step is used for its representative spatial information under severe noise.Source: §3.3 · p. 5

4. Experiments

Experiments evaluate GWM across scene prediction, imitation learning, reinforcement learning, and real-world manipulation in synthetic and physical settings. Results cover prediction quality, policy performance, and component ablations.

4. Experiments

  • The evaluation spans Meta-World and RoboCasa synthetic environments and the real-world FRANKA-PNP pick-and-place setting, across four task types.The four types are action-conditioned scene prediction, GWM-based imitation learning, GWM-based reinforcement learning, and real-world deployment.Source: §4 · p. 5

4.1. Action-conditioned Scene Prediction

  • GWM outperforms iVideoGPT on future-state prediction across synthetic and real-world environments, with qualitative gains in gripper-motion details.Prediction quality is tested on unseen validation-set action trajectories; Table 1 reports Meta-World and FRANKA PNP results.Source: §4.1 · p. 6, Table 1 · p. 6, Figure 3 · p. 6

4.2. GWM-based Imitation Learning

  • Across 24 RoboCasa kitchen tasks, GWM improves average success rates by 10.5% with H-50 and 7.6% with G-3000 demonstrations.The results are evaluated over 50 episodes with different floor plans and styles.Source: §4.2 · p. 6, Table 2 · p. 7

4.3. GWM-based Reinforcement Learning

  • On all six Meta-World tasks, GWM outperforms iVideoGPT, converging approximately 2× faster and reaching higher asymptotic performance on complex tasks.The comparison uses the same context length, horizon, and maximum training steps, without pretrained initialization.Source: §4.3 · p. 7, Figure 5 · p. 7

4.4. Real-world Deployment

  • GWM achieves a 65% success rate versus Diffusion Policy’s 35% across 20 real-world trials, with a wider gap on novel distractors.The task is picking a colored cup and placing it on a plate; performance is attributed to task-relevant dynamics and robustness to visual differences.Source: §4.4 · p. 8, Table 3 · p. 8

4.5. Ablation Analysis

  • On RoboCasa, adding Gaussian Splatting raises success rate from 4% to 18%, and adding the 3D Gaussian VAE raises it further to 24%.The VAE addition also improves all reported metrics, including PSNR; Gaussian Splatting improves SSIM and LPIPS despite a slight PSNR decrease.Source: §4.5 · p. 8, Table 4 · p. 8

5. Conclusion

GWM models robotic-manipulation futures by propagating Gaussian primitives under robot actions, combining a DiT with a 3D-aware VAE for Gaussian Splatting reconstruction. The paper reports effective future-scene prediction and policy training in simulated and real-world settings.

5. Conclusion

  • GWM introduces a 3D world model that addresses image-based world-model limitations by incorporating geometric information.It reconstructs future states by modeling Gaussian-primitive propagation under robot actions.Source: §5 · p. 8
  • A DiT and 3D-aware variational autoencoder enable scene-level future-state reconstruction through Gaussian Splatting.A scalable processing pipeline supports test-time updates in model-based reinforcement learning by extracting aligned Gaussian splats from unposed images.Source: §5 · p. 8
  • Simulated and real-world experiments demonstrate future-scene prediction and training of superior policies.Source: §5 · p. 8

A. Datasets and Benchmarks

The described benchmarks include RoboCasa kitchen manipulation data and MetaWorld simulation tasks. The section also points to a table of model-based reinforcement-learning hyperparameters.

A. Datasets and Benchmarks

  • RoboCasa provides kitchen-manipulation data extracted from MuJoCo with a Franka Emika Panda, including the Human-50 and Generated-3000 datasets based on human demonstrations.The benchmark includes 24 atomic tasks.Source: A. Datasets and Benchmarks · p. 13
  • MetaWorld comprises 50 simulated manipulation tasks with a Sawyer robot, 64 × 64 RGB observations, and a 4-dimensional continuous action vector.Source: A. Datasets and Benchmarks · p. 13
  • Table A1 lists hyperparameters for the model-based reinforcement-learning experiments.Source: Table A1 · p. 13

B.1. EDM Preconditioning

The appendix lists EDM preconditioners intended to improve network training and specifies the noise-parameter sampling distribution. It also points to a separate table of imitation-learning hyperparameters.

B.1. EDM Preconditioning

  • The listed preconditioners are designed to improve network training [33], with factors involving σ(τ), σ_data, and the denominator √(σ(τ)^2 + σ_data^2).Source: §B.1 · p. 13
  • Table A2 lists hyperparameters for the imitation-learning experiments.Source: Table A2 · p. 13
  • The noise parameter σ(τ) is sampled to maximize training effectiveness, with σ_data = 0.5 and log σ(τ) distributed as N(Pmean, Pstd²).Source: §B.1 · p. 13

B.2. Architectural Design

The architecture encodes point clouds with a transformer VAE and predicts denoised next states with an action- and timestep-conditioned DiT. A separate reward model combines convolutional and sequential processing.

B.2. Architectural Design

  • The VAE downsamples 2,048 input points to 512 latent points using farthest-point sampling before self- and cross-attention blocks.Source: §B.2 · p. 13
  • In its probabilistic variant, the VAE encoder outputs mean and log-variance parameters for reparameterized latent sampling, with optional KL-divergence regularization.Source: §B.2 · p. 13
  • The DiT processes current and noisy next observations with timestep and action embeddings, then predicts the denoised next state using the EDM formulation.Source: §B.2 · p. 13
  • The reward model encodes observation pairs conditioned on actions with convolutional ResBlocks, then uses an LSTM and MLP head for reward prediction.Before inference, its LSTM hidden states are initialized through a burn-in procedure with conditioning frames.Source: §B.2 · pp. 13–14

B.3. Hyper-parameters

Robocasa and MetaWorld hyperparameters are listed in Tables A2 and A1, respectively.

B.3. Hyper-parameters

  • Robocasa and MetaWorld hyperparameters are listed in Tables A2 and A1, respectively.Source: §B.3 · p. 14

Related papers

Loading 2508.17600v2…