Source-linked AI summary
GaussianDream++: Efficient 3D Gaussian World Modeling for Robotic Manipulation
Yuqing Jiang, Zijian Zhang, Weitao Zhou, Jiawei Wang, Junjie He, Lei Yang, Haifang Qing, Si Liu, Ding Zhao, Ping Luo, Haibao Yu
TL;DR
VLA action imitation weakly supervises metric 3D structure and short-horizon evolution. GaussianDream++ addresses this with compact world tokens and training-only Gaussian reconstruction and prediction, achieving strong benchmark and real-robot results while removing runtime world decoding and rollout.
Problem
VLA action-imitation objectives provide weak supervision for metric 3D structure and short-horizon physical evolution required in manipulation.
Method
GaussianDream++ embeds 20 World State and World Prediction Tokens in the VLA backbone and decodes them during training into coupled current and future Gaussian worlds.
Results
98.6% on LIBERO and 87.8% on LIBERO-Plus, with real-robot success improving from 29.2% to 52.5% over reproduced π0.5.
Takeaways & Limitations
Gaussian world supervision can be internalized in a compact VLA representation while retaining efficient deployment without online Gaussian decoding or future rollout.
Takeaways & Limitations
Performance under Robot shift remains below reproduced π0.5, leaving embodiment-level generalization as a separate challenge.
Abstract
from arXiv · showhide
Vision-Language-Action (VLA) policies have advanced language-conditioned robotic manipulation, yet action-imitation objectives provide only weak supervision for metric 3D structure and short-horizon physical evolution. Geometry-enhanced policies mainly improve current-scene grounding, whereas predictive policies often model future dynamics in RGB or latent spaces and may incur substantial deployment cost. GaussianDream demonstrates that training-time current Gaussian reconstruction and future Gaussian prediction provide effective 3D supervision, but its dense VGGT/TGE-based prefix jointly carries state, dynamics, and action-conditioning information. We present \textbf{\methodname}, a compact, policy-native extension that inserts \textbf{World State Tokens} and \textbf{World Prediction Tokens} directly into the VLA backbone. A training-only \textbf{World Representation Head} decodes these tokens into a Current World and coupled Future Prediction over shared Gaussian primitives, while static--dynamic factorization preserves persistent structure and focuses residual motion on interaction-relevant regions. At inference, the head, renderer, auxiliary objectives, and VGGT/TGE pathway are removed, leaving only 20 world tokens without online Gaussian decoding or rollout. \method achieves \textbf{98.6\%} on LIBERO and \textbf{87.8\%} on LIBERO-Plus, with clear gains under Camera and Layout shifts. Real-robot experiments further improve average success from 29.2\% to 52.5\% over reproduced $π_{0.5}$ while maintaining efficient closed-loop control.
1 Introduction
VLA policies remain weakly constrained in metric 3D structure and short-horizon physical evolution, while prior geometry and predictive approaches address these needs incompletely. GaussianDream++ embeds compact, role-structured world tokens in the VLA backbone, uses Gaussian supervision during training, and removes decoding overhead at deployment while improving benchmark and real-robot performance.
- Motivation: Action imitation indirectly constrains metric 3D structure, object-relative geometry, and short-horizon physical evolution needed for manipulation.The introduction identifies implicit geometry, unused trajectory evidence, and entangled state–future representations as central challenges.
- Prior paradigms: Geometry-enhanced policies strengthen current-scene grounding, but predictive and world-model policies add temporal supervision through future modeling.Prior approaches therefore address different parts of the physical-structure problem rather than providing the compact integrated representation proposed here.
- Deployment: Gaussian decoding, rendering, and auxiliary supervision are confined to training, leaving only 20 world tokens at deployment without runtime VGGT/TGE processing or future rollout.This asymmetric design preserves training-time physical supervision while reducing deployment requirements.
- Results: 98.6% success on LIBERO and 87.8% on LIBERO-Plus accompany gains under Camera and Layout shifts, while real-robot success rises from 29.2% to 52.5% over reproduced π0.5.The matched GaussianDream-family protocol reports 0.8-point overall improvement over GaussianDream and further real-robot gains over the reproduced baseline.
- Representation: GaussianDream++ replaces GaussianDream’s dense VGGT/TGE-based prefix with 20 World State and World Prediction Tokens embedded directly in the VLA backbone.The tokens remain visible to the Action Expert through the native attention pathway, aligning world-supervised representations with action generation.
- Representation: World State Tokens reconstruct the Current World, while World Prediction Tokens model short-horizon evolution over shared Gaussian primitives.Static–dynamic factorization preserves persistent structure and focuses residual prediction on interaction-induced changes.
2 Related Work
Related work improves robotic policies through spatial or temporal supervision, but often separates current geometry from future evolution or incurs deployment cost. GaussianDream++ retains training-time Gaussian supervision while moving a compact, role-structured representation directly into the native VLA backbone.
- VLA policies: VLA systems combine pretrained vision-language representations with robot demonstrations, but action imitation leaves metric scene geometry indirectly constrained.This motivates additional spatial and temporal supervision beyond one-step behavior cloning.
- Geometry-enhanced policies: Geometry-enhanced VLAs add depth, stereo, point clouds, spatial encodings, 4D features, or pretrained 3D representations to strengthen spatial grounding.These methods primarily target representation of the current scene and calibrated geometric information.
- Predictive policies: Predictive policies and World-Action Models use future observations, trajectories, videos, or latent dynamics as supervision beyond behavior cloning.Their temporal priors support action learning and planning, while future states are often modeled in RGB or video-latent spaces.
- Gaussian representations: Explicit 3D Gaussian primitives associate geometry, visibility, appearance, and motion with physical scene elements while supporting differentiable rendering.This representation provides a natural interface for robotic reconstruction and manipulation.
- GaussianDream: GaussianDream uses training-only reconstruction and prediction heads for current and future Gaussian states, converting trajectories into dense RGB, depth, and pseudo-3D scene-flow supervision.Its auxiliary heads are removed at deployment, establishing asymmetric training and inference as a useful design principle.
- GaussianDream++: GaussianDream++ moves supervised world representation from a dedicated temporal-geometry pathway into the native VLA backbone using compact role-structured tokens.Figure 2 shows these tokens contextualized with visual-language inputs and retained after training-only world-modeling branches are removed.
3 Method
GaussianDream++ embeds compact, role-specific world tokens in the VLA backbone and uses a training-only representation head for current reconstruction and future prediction. Shared Gaussian primitives and static–dynamic factorization preserve persistent structure while concentrating prediction on interaction-relevant motion, with all dense world decoding removed at deployment.
- Coupled world representation: A lightweight World Representation Head decodes shared token states into a renderable Current World and horizon-dependent Future Prediction.Future observations provide training targets but never enter the policy forward pass.
- Policy-native world representation: GaussianDream++ replaces GaussianDream’s dense external prefix with 20 World State and World Prediction Tokens embedded directly in PaliGemma.The tokens are compact latent summaries, with 16 state tokens and 4 prediction tokens in the main configuration.
- Current World reconstruction: Current World reconstruction uses metric depth, calibrated multi-view unprojection, differentiable Gaussian splatting, RGB, depth, and coverage supervision.Geometry and appearance are decoded separately, with detached geometry features preventing photometric shortcuts from altering metric scene structure.
- Policy-native world representation: World State Tokens encode the current physical scene, while World Prediction Tokens encode short-horizon evolution over shared Gaussian primitives.The Action Expert directly conditions on the complete world-token-augmented prefix through the native attention pathway.
- Coupled world representation: Static–dynamic factorization preserves persistent primitives and concentrates residual motion on the robot, manipulated objects, and interaction regions.Static-consistency supervision penalizes residual motion in persistent background and stationary-object regions.
- Efficient deployment: At deployment, the representation head, renderer, target builders, auxiliary objectives, and VGGT/TGE pathway are removed, leaving only compact world-token processing in the VLA backbone.Inference requires neither online Gaussian reconstruction and rendering nor future rollout.
4 Experiments
Experiments evaluate GaussianDream++ on standard manipulation, distribution shifts, real-robot transfer, qualitative world representations, latency, and ablations. Results show strong benchmark and real-world performance, robustness gains for geometry-sensitive shifts, compact deployment, and contributions from the proposed representation and supervision components.
- Main benchmark performance: GaussianDream++ achieves 98.6% average success on LIBERO and 87.8% Overall on LIBERO-Plus.It improves Overall by 0.8 points over GaussianDream in the matched family block and obtains the highest Overall score in Table 1.
- Distribution-shift robustness: Camera success rises from 73.2% to 80.1% and Layout success from 87.7% to 90.0% over reproduced π0.5.Relative to GaussianDream, the corresponding gains are 2.8 and 1.6 points.
- Distribution-shift robustness: Performance under Robot shift remains below reproduced π0.5, while Noise improves from 89.9% to 94.2%.The reported gains are strongest for viewpoint, layout, and visual perturbations; embodiment-level generalization remains a separate challenge.
- Real-world manipulation: Real-robot pooled success increases from 29.2% to 52.5% over reproduced π0.5 across Bowl-Proximity and Eggplant-to-Pink-Plate.Bowl-Proximity rises from 25.0% to 46.7%, while Eggplant-to-Pink-Plate rises from 33.3% to 58.3%.
- Real-world manipulation: Layout variation doubles pooled real-robot success from 25.0% to 50.0%, while Camera variation increases it from 22.5% to 42.5%.These conditions change object-relative positions or global image projection while retaining the task goal.
- Qualitative analysis: Current Reconstructions preserve dominant spatial organization, while Future Prediction maintains coarse scene geometry and short-horizon changes.The coupled formulation uses the Current World as a persistent Gaussian template and predicts residual evolution rather than regenerating the full scene.
- Inference efficiency: Action-chunk latency increases from 286 ms to 330 ms, with no Gaussian construction, rendering, or future rollout executed online.The overhead comes from contextualizing 20 additional world tokens inside PaliGemma.
- Ablation studies: The ablation progression reaches 87.8% Overall as Current World supervision, Future Prediction, coupling, and static consistency are added.Token capacity alone reaches 86.3%, while Current World supervision, Future Prediction, coupled decoding, and static consistency raise Overall to 86.9%, 87.2%, 87.5%, and 87.8%.
5 Conclusion
GaussianDream++ is a compact, policy-native successor that embeds role-separated world tokens in the VLA backbone and uses training-only Gaussian supervision. It removes dedicated world-modeling components at deployment while achieving strong benchmark and real-robot performance.
- GaussianDream++ replaces the dense VGGT/TGE-based prefix with 20 World State and World Prediction Tokens embedded in the PaliGemma backbone.
- The training-only World Representation Head decodes these tokens into a Current World and coupled Future Prediction over shared Gaussian primitives.Static–dynamic factorization preserves persistent scene structure and concentrates residual motion on interaction-relevant regions.
- At deployment, GaussianDream++ removes the representation head, renderer, auxiliary branches, and VGGT/TGE pathway, leaving the Action Expert to use compact world-token representations directly.The deployed policy performs neither online Gaussian decoding nor future rollout.
- 98.6% success on LIBERO and 87.8% on LIBERO-Plus are achieved under the matched GaussianDream-family protocol, with clearest gains under Camera and Layout shifts.
- Real-robot success improves from 29.2% to 52.5% over reproduced π0.5 while maintaining efficient closed-loop control.