PaperlayerView exact revision on arXiv

Source-linked AI summary

ManiGaussian: Dynamic Gaussian Splatting for Multi-task Robotic Manipulation

Guanxing Lu, Shiyi Zhang, Ziwei Wang, Changliu Liu, Jiwen Lu, Yansong Tang

arXiv:2403.08321v2cs.ROcs.CV

TL;DR

Language-conditioned manipulation methods often overlook scene-level spatiotemporal dynamics needed for complex object interactions. ManiGaussian models these dynamics with dynamic Gaussian Splatting and future scene reconstruction, achieving higher success than state-of-the-art methods with less computation across 10 RLBench tasks.

  • Problem

    Existing manipulation methods use semantic representations but ignore spatiotemporal dynamics that depict physical interactions among objects.

  • Method

    ManiGaussian uses dynamic Gaussian Splatting and a Gaussian world model to propagate semantic features and reconstruct future scenes for action prediction.

  • Results

    Across 10 RLBench tasks, ManiGaussian achieves higher success than state-of-the-art methods with less computation.

  • Takeaways & Limitations

    Encoding scene-level spatiotemporal dynamics provides a basis for predicting manipulation actions involving physical object interactions.

  • Takeaways & Limitations

    Additional details and experiments are placed in supplementary material because of space limitations.

Abstract

from arXiv · show

Performing language-conditioned robotic manipulation tasks in unstructured environments is highly demanded for general intelligent robots. Conventional robotic manipulation methods usually learn semantic representation of the observation for action prediction, which ignores the scene-level spatiotemporal dynamics for human goal completion. In this paper, we propose a dynamic Gaussian Splatting method named ManiGaussian for multi-task robotic manipulation, which mines scene dynamics via future scene reconstruction. Specifically, we first formulate the dynamic Gaussian Splatting framework that infers the semantics propagation in the Gaussian embedding space, where the semantic representation is leveraged to predict the optimal robot action. Then, we build a Gaussian world model to parameterize the distribution in our dynamic Gaussian Splatting framework, which provides informative supervision in the interactive environment via future scene reconstruction. We evaluate our ManiGaussian on 10 RLBench tasks with 166 variations, and the results demonstrate our framework can outperform the state-of-the-art methods by 13.1\% in average success rate. Project page: https://guanxinglu.github.io/ManiGaussian/.

1 Introduction

ManiGaussian addresses failures of semantic-only manipulation policies by modeling scene-level spatiotemporal dynamics for language-conditioned action prediction. It combines dynamic Gaussian Splatting with a Gaussian world model that reconstructs future scenes for supervision.

  • 1 Introduction: Fig. 1 shows ManiGaussian stacking two rose blocks on a green block, whereas GNFactor incorrectly attempts to move the fixed green base.The example attributes the difference to explicit scene-dynamics encoding through future reconstruction.
  • 1 Introduction: ManiGaussian models scene-level spatiotemporal dynamics so robots can predict actions for language-conditioned manipulation in unstructured environments.The framework propagates semantic features in Gaussian embedding space and uses the resulting representation for action prediction.
  • 1 Introduction: The method builds a Gaussian world model that reconstructs future scenes from the current scene and robot actions, constraining consistency with realistic futures.This reconstruction supplies supervision for mining dynamics in interactive environments.
  • 1 Introduction: The proposed framework is designed to achieve higher success rates with less computation than state-of-the-art methods.The contribution summary states both performance and computational advantages.

2 Related Work

Prior manipulation methods emphasize visual semantics or latent future prediction, while conventional generative representations often overlook object interactions. ManiGaussian introduces dynamic Gaussian Splatting to model future states and scene dynamics for manipulation.

  • 2 Related Work: Prior generative manipulation methods can ignore scene-level spatiotemporal dynamics, causing incorrect object interactions and failed human goals.GNFactor is described as improving reconstruction and behavior cloning while still belonging to this broader limitation.
  • 2 Related Work: World models predict future states from current states and actions, but latent future prediction requires substantial data and is limited to simple tasks.This limitation motivates a different representation for interactive manipulation dynamics.
  • 2 Related Work: Gaussian Splatting represents scenes with projected 3D Gaussians and offers fast inference, high fidelity, and editability compared with implicit NeRF representations.Related variants add generalization, semantic information, or deformable-scene reconstruction.
  • 2 Related Work: ManiGaussian formulates dynamic Gaussian Splatting to model object-interaction dynamics and enhance physical reasoning across robotic manipulation tasks.The framework extends Gaussian scene representations with temporal modeling for future-state prediction.

3 Approach

ManiGaussian models scene-level spatiotemporal dynamics for language-conditioned robotic manipulation by propagating semantic features in Gaussian space and reconstructing future scenes with a Gaussian world model.

  • 3 Approach: ManiGaussian combines dynamic Gaussian Splatting with a Gaussian world model to model semantic propagation and reconstruct future scenes for manipulation.The framework uses future-scene supervision to mine scene-level dynamics from interactive environments.
  • 3.3 Dynamic Gaussian Splatting for Robotic Manipulation: Dynamic Gaussian Splatting propagates Gaussian scene representations over time so physical interactions can inform optimal robot-action prediction.The representation includes geometric, semantic, and dynamic information, while Gaussian positions and rotations change during manipulation.
  • 3.3 Dynamic Gaussian Splatting for Robotic Manipulation: The method augments Gaussian primitives with time-dependent positions, colors, rotations, scales, opacities, and semantic features distilled from RGB observations.Positions and rotations vary with manipulation, while colors, scales, opacities, and semantic features are treated as time-independent for rigid objects.
  • 3.3 Dynamic Gaussian Splatting for Robotic Manipulation: A Gaussian world model predicts future scenes from current observations and robot actions by propagating Gaussian distribution parameters and rendering the resulting scene.The model constrains reconstructed scenes to remain consistent with realistic future scenes, providing supervision for dynamics learning.
  • 3.4 Learning Objectives: The training objectives enforce current-scene geometric consistency, semantic-feature consistency, and action prediction from the learned Gaussian representation.The geometric objective compares rendered and ground-truth observations, while the semantic objective aligns Gaussian features with pretrained visual features.

4 Experiments

Experiments evaluate ManiGaussian on diverse RLBench manipulation tasks, comparing its success rate, efficiency, component contributions, and qualitative behavior against prior methods. The results show stronger benchmark performance, faster training, and improved handling of scene dynamics.

  • 4.2 Comparison with the State-of-the-Art Methods: 44.8% average success rate makes ManiGaussian state-of-the-art on 10 RLBench tasks, outperforming GNFactor by 41.3% relative improvement.The evaluation covers 25 episodes per task across 10 tasks and 166 variations; Table 1 reports the final multi-task comparison.
  • 4.3 Ablation Study: 23.6% to 44.8% performance gain after combining all techniques supports the contribution of dynamic Gaussian Splatting and the Gaussian world model.The ablation progressively adds Gaussian regression, semantic features, consistency losses, and future-scene deformation prediction.
  • 4.3 Ablation Study: The Gaussian regressor improves performance by 15.6%, while semantic features add 2.4% and the deformation predictor adds 4.4%.The deformation predictor improves completion on 4 of 6 task types, particularly long-horizon tasks, while dynamic-loss balancing may slightly affect short-term results.
  • 4.3 Ablation Study: 1.18× better performance and 2.29× faster training than GNFactor demonstrate ManiGaussian’s accuracy and efficiency advantage.Both methods converge within 100k training steps, with checkpoints tested every 10k parameter updates.
  • 4.4 Qualitative Analysis: Qualitative trajectories show ManiGaussian completing block-sliding and tap-turning tasks where GNFactor misinterprets object contact or directional language.The method returns to the red square before sliding it and selects the left tap while executing the operation successfully.
  • 4.4 Qualitative Analysis: Novel-view synthesis reconstructs current scenes and predicts future gripper and object states under physical interaction.The visualization removes the action loss and highlights future cube displacement caused by the gripper.

5 Conclusion

The conclusion presents ManiGaussian as a language-conditioned manipulation agent that encodes scene-level spatiotemporal dynamics for action prediction. Its dynamic Gaussian Splatting framework and Gaussian world model support this representation through future-scene modeling.

  • 5 Conclusion: ManiGaussian encodes scene-level spatiotemporal dynamics and uses the resulting latent representation to predict robot actions.The framework models feature propagation in Gaussian embedding space for language-conditioned manipulation.
  • 5 Conclusion: The Gaussian world model parameterizes the dynamic framework to mine scene dynamics through future-scene reconstruction.The conclusion identifies this world-model component as the mechanism supporting the dynamic representation.

Supplementary Material In this supplementary material, we provide additional details and experiments not included in the main paper due to limitations in space.

The supplementary material provides additional dataset, training, implementation, quantitative, and qualitative details beyond the main paper.

  • Supplementary Material: Appendix A covers the RLBench dataset and training pipeline used in the experiments.
  • Supplementary Material: Appendix B provides additional ManiGaussian implementation details, while Appendices C and D provide quantitative and qualitative analyses.

A Details of RLBench

The RLBench evaluation uses selected tasks with diverse object properties, placements, and manipulation challenges. The training pipeline samples expert input-action pairs across task variations.

  • RLBench Dataset: RLBench includes 10 selected tasks with variations in object colors, sizes, counts, categories, and tabletop placements.The color palette contains 20 shades, object sizes are short or tall, and counts range from 1 to 3.
  • RLBench Dataset: The selected tasks are grouped by challenges involving planning, long horizons, tool use, precise motion, gripper rotation, and occlusion.The groups cover multiple subtasks, more than 10 keyframes, object-tool interaction, predefined-motion precision, screw-like gripper rotation, and severe viewpoint occlusion.
  • Training Pipeline: Training samples are formed by uniformly selecting expert episodes across task variations and randomly choosing an input-action pair for each task.The pipeline allows alternative sampling strategies such as Auto-λ.

B Additional Implementation Details

The Gaussian world model encodes voxelized visual inputs and predicts Gaussian scene parameters with lightweight specialized heads. Training emphasizes action prediction through selected geometry, semantic, and dynamics loss weights.

  • Representation model: A shallow 3D UNet encodes voxelized RGB features, coordinates, indices, and occupancy into 128-dimensional visual features.The input voxel representation has dimensions R100^3×10, and the encoded features have dimensions R100^3×128.
  • Gaussian regressor: A lightweight multi-head Gaussian regressor predicts position offsets, spherical-harmonic colors, rotations, scales, and opacity from current visual features.The heads respectively output 3D offsets, 12 color coefficients, 4D rotation quaternions, 3D scales, and opacity predictions.
  • Hyperparameters: ManiGaussian uses λGeo = 0.01, λSem = 0.0001, and λDyna = 0.001 to emphasize action prediction during training.Other hyperparameters follow previous works for fair comparison.

C Additional Quantitative Analysis

The ablation study finds that balancing the loss components affects overall manipulation performance. Table 5 evaluates different balance hyperparameter choices.

  • Impact of Balance Hyperparameters: The balance of each loss item is important for learning an optimal manipulation policy.The study examines how different balance hyperparameters affect overall performance.

D Additional Qualitative Analysis

Qualitative episodes show ManiGaussian handling long-term ordering, tool interactions, semantic instance selection, and precise control across diverse manipulation tasks. The baseline often confuses object instances in tasks requiring semantic understanding and precise control.

  • Long-term tasks: ManiGaussian completes long-term tasks in the correct order by using high-level scene dynamics mined by its Gaussian world model.Examples include stacking blocks, placing an item in a drawer, and taking steak off a grill.
  • Tool-use tasks: ManiGaussian solves tool-use tasks by understanding low-level object-contact dynamics.The examples involve sweeping dirt and dragging a cube with a stick onto a target.
  • Semantic and precise-control tasks: In semantic and precise-control tasks, ManiGaussian selects the correct object instance, whereas the baseline often confuses different instances.Examples include sliding a block, turning a tap, closing a jar, and opening a drawer.
Loading 2403.08321v2…