Source-linked AI summary
GaussianWAM: Distilling Geometry and Semantics from 3D Gaussian Fields into World-Action Models
Zijian Zhang, Yuqing Jiang, Weitao Zhou, Minglei Li, Jinhao Zhang, Yao Mu, Xiaofan Li, Hao Zhao, Haibao Yu
TL;DR
WAM video latents are not explicitly encouraged to preserve cross-view geometry or spatially localized object semantics, despite their importance for manipulation. GaussianWAM binds geometric and semantic teacher signals in a shared 3D Gaussian field and distills rendered supervision during training. It improves FastWAM and Cosmos Policy on LIBERO-Plus while removing all Gaussian-related components at inference.
Problem
WAM video latents are primarily optimized for visual prediction and lack explicit cross-view geometric and localized object-semantic constraints relevant to manipulation.
Method
GaussianWAM binds geometric and visual-semantic knowledge to shared 3D Gaussian primitives, renders semantic, depth, and coverage signals, and distills them into current-observation WAM representations.
Results
GaussianWAM improves FastWAM from 52.05% to 71.29% and Cosmos Policy from 71.52% to 77.30% overall on LIBERO-Plus.
Takeaways & Limitations
Training-time Gaussian distillation strengthens geometric and visual-semantic grounding across two WAM architectures without changing deployment architecture or inference cost.
Takeaways & Limitations
The Gaussian teacher takes only visual observations as input; language instructions and proprioceptive states remain policy inputs.
Abstract
from arXiv · showhide
World-Action Models (WAMs) jointly learn future visual prediction and action generation, using video dynamics as a representation-learning signal for robotic manipulation. However, their video latents are primarily optimized for visual prediction and are not explicitly encouraged to preserve cross-view geometric structure or spatially localized, object-relevant semantics. We propose \textbf{GaussianWAM}, a training-time representation-enhancement framework that organizes geometric and semantic supervision through a 3D Gaussian field. Given synchronized multi-view observations, frozen geometry and vision foundation models provide depth, camera parameters, and dense semantic features. GaussianWAM binds these heterogeneous signals to shared Gaussian primitives and renders spatially aligned semantic, depth, and coverage targets, which are distilled into the current-observation representations of the WAM. All teacher models, Gaussian components, and auxiliary prediction heads are removed after training, leaving the original WAM inference path without additional modules or forward computation. On LIBERO-Plus, GaussianWAM improves FastWAM from 52.05\% to 71.29\% and Cosmos Policy from 71.52\% to 77.30\%. Direct CLIP and VGGT distillation already establishes a strong FastWAM baseline of 69.37\%, while Gaussian-field unification further improves it to 71.29\%, supporting the benefit of spatially organizing heterogeneous teacher signals. GaussianWAM also improves performance on standard LIBERO and shows positive transfer trends on RoboTwin and real-world manipulation. These results suggest that training-time Gaussian distillation provides a practical way to inject geometry- and semantics-related supervision into WAM representations without changing their deployment architecture.
1 Introduction
WAMs use video dynamics for robotic control, but their representations lack explicit cross-view geometry and localized object semantics. GaussianWAM organizes heterogeneous supervision in a shared 3D Gaussian field and distills it during training without changing deployment.
- WAMs jointly model future visual dynamics and actions, moving beyond purely reactive action prediction for robotic manipulation.
- RGB- and video-prediction objectives do not explicitly preserve cross-view geometry, although precise manipulation requires 3D scene relationships.
- WAM latents also lack explicit spatially localized, object-relevant semantics despite language conditioning and available foundation-model priors.
- Independent teacher distillation leaves geometry and semantics in disconnected spaces, while explicit 3D/4D models require additional modeling components.
- GaussianWAM binds geometric and semantic signals to shared Gaussian primitives, then renders aligned semantic, depth, and coverage supervision for WAM representations.
- Training-only Gaussian enhancement preserves the original WAM inference pipeline and computational cost while supporting validation across FastWAM and Cosmos Policy.
2 Related Work
Related work spans predictive world-action modeling, geometry- and semantic-aware WAM enhancements, and Gaussian fields that carry spatial or foundation-model features. GaussianWAM applies these ideas as a training-time spatial distillation framework.
- World-action models and video-action models: WAMs augment robot policies with predictive world modeling, jointly learning future visual dynamics and action generation.
- Geometry- and semantic-aware world-action learning: Geometry- and semantic-aware WAMs explore explicit 3D/4D representations, geometric foresight, spatial features, and auxiliary prediction.
- 3D Gaussian fields for representation enhancement: 3D Gaussian fields represent scenes with explicit primitives that can carry spatial attributes and high-dimensional foundation-model features.
- GaussianWAM positioning: GaussianWAM constructs an offline Gaussian teacher whose signals are rendered onto WAM-aligned grids and distilled into action-relevant visual representations.
3 Method
GaussianWAM builds an offline 3D Gaussian teacher from synchronized multi-view observations, combining geometric and visual-semantic signals in a shared spatial representation. Rendered semantic, depth, coverage, and validity targets are distilled into WAM visual representations during training, while Gaussian-related modules are removed at inference.
- Gaussian teacher construction: GaussianWAM uses an offline 3D Gaussian teacher to combine geometry, visual-semantic features, and coverage information in a unified spatial representation.The teacher is constructed from current multi-view observations and its signals are rendered onto grids aligned with WAM visual representations.
- Teacher construction: Frozen VGGT-Omega provides depth, confidence, and camera parameters, while frozen CLIP produces dense 64-dimensional visual-semantic features on a shared teacher grid.Invalid or low-confidence geometry is filtered before valid pixels are back-projected into 3D; CLIP patch features are resized to the same grid.
- Gaussian field initialization: Each Gaussian primitive binds a 3D center and appearance-related parameters to a visual-semantic feature, spatially aligning heterogeneous supervision in one reconstructed coordinate system.The shared primitives carry geometry- and semantics-related attributes rather than treating the two signals as independent 2D feature losses.
- Multi-view Gaussian fitting: The differentiable renderer outputs visual-semantic feature maps, depth maps, and accumulated coverage maps through depth-aware, opacity-weighted soft splatting.Coverage is derived from accumulated Gaussian weights, while projected contributions are aggregated into rendered feature and depth maps.
- Gaussian distillation: Gaussian distillation trains auxiliary heads to predict cached semantic, depth, and coverage targets from current-observation WAM tokens using validity-masked losses alongside the original WAM objective.The targets are cached offline after Gaussian fitting, and supervision is applied only at valid token-grid locations.
- Deployment: At inference, Gaussian teachers, construction and rendering modules, and auxiliary heads are removed, preserving the original WAM forward path and inference cost.Gaussian supervision still influences action learning through the enhanced visual representations shared with the action model in both FastWAM-style and Cosmos-Policy-style architectures.
4 Experiments
GaussianWAM is evaluated across simulated, randomized, and real-world manipulation settings, consistently improving WAM performance and representation quality. Ablations and probing indicate benefits from unified geometry-semantic supervision, complete supervision components, and final-layer distillation.
- LIBERO and LIBERO-Plus: GaussianWAM improves FastWAM’s LIBERO-Plus overall success from 52.05% to 71.29% and Cosmos Policy’s from 71.52% to 77.30%.The gains occur across two representative WAM architectures.
- RoboTwin and real-world evaluation: GaussianWAM preserves or improves RoboTwin performance, including FastWAM Random success from 0.70% to 1.60% and Cosmos Policy Clean success from 34.40% to 68.90%.Cosmos Policy Random success also rises from 7.10% to 14.40%.
- Real-world manipulation: GaussianWAM raises FastWAM’s real-world average success rate from 30.00% to 40.00% across standard, layout, camera, and visual-noise conditions.Each task uses 100 demonstrations and 20 evaluation trials per setting.
- Ablation studies: Direct CLIP and VGGT distillation reaches 69.37% on LIBERO-Plus, while Gaussian-field unification reaches 70.99% without alpha supervision and 71.29% with it.The comparison evaluates spatial organization beyond independent 2D teacher distillation.
- Ablation studies: Removing semantic, depth, or alpha supervision reduces overall success to 68.06%, 66.28%, or 70.99%, respectively, showing complementary supervision effects.Removing depth produces the largest drop from the full 71.29% result.
- Ablation studies: Distilling at the final Transformer layer achieves 71.29% overall, compared with 67.86% and 68.44% at layers 10 and 20.The final layer is layer 30 in the evaluated FastWAM configuration.
- Representation analysis: Frozen-backbone probing shows clearer scene structure and more coherent depth responses for GaussianWAM representations than for base WAM representations.The depth probe uses VGGT-Omega pseudo-depth supervision.
- Representation analysis: t-SNE visualizations show more compact local clusters and clearer separation among semantic groups for GaussianWAM representations.Final-layer visual features are projected using the same t-SNE configuration for base and GaussianWAM models.
5 Conclusion
GaussianWAM enhances World-Action Models by distilling geometric and visual-semantic knowledge through a unified 3D Gaussian field, without changing inference. Experiments across simulated and real-world manipulation show effectiveness across architectures and challenging distribution shifts.
- GaussianWAM uses a 3D Gaussian field as a unified spatial teacher that binds geometric and visual-semantic knowledge.Rendered semantic, depth, and alpha signals are distilled into action-relevant WAM representations, with validity masks restricting supervision to reliable regions.
- All Gaussian-related teachers and prediction modules are removed at inference, preserving the original WAM deployment pipeline and computational cost.
- GaussianWAM improves performance across LIBERO, LIBERO-Plus, RoboTwin, and real-world robotic manipulation on FastWAM and Cosmos Policy.The evaluation spans two distinct WAM architectures and multiple simulated and real-world settings.
- Improvements are particularly pronounced under challenging distribution shifts, indicating stronger geometric and visual-semantic grounding of the learned representations.
- Ablations demonstrate complementary contributions from semantic, depth, and alpha supervision.