PaperlayerView exact revision on arXiv

Source-linked AI summary

ParticleFormer: A 3D Point Cloud World Model for Multi-Object, Multi-Material Robotic Manipulation

Suning Huang, Qianzhong Chen, Xiaohan Zhang, Jiankai Sun, Mac Schwager

arXiv:2506.23126v4cs.RO

TL;DR

Existing robotic manipulation approaches often map observations directly to actions, coupling environmental dynamics with task objectives and hindering generalization across task-environment combinations. ParticleFormer combines a Transformer-based point cloud dynamics model with Chamfer and Hausdorff supervision applied directly to sensor streams. ParticleFormer consistently outperforms competitive baselines across six simulation tasks and three real-world experiments in dynamics prediction and downstream visuomotor control.

  • Problem

    Existing robotic manipulation approaches often map observations directly to actions, coupling environmental dynamics with task objectives and hindering generalization across task-environment combinations.

    Source: §1 · p. 1

  • Method

    ParticleFormer combines a Transformer-based point cloud dynamics model with Chamfer and Hausdorff supervision applied directly to sensor streams.

    Source: §1 · pp. 2–3, §3.2 · p. 4

  • Results

    ParticleFormer consistently outperforms competitive baselines across six simulation tasks and three real-world experiments in dynamics prediction and downstream visuomotor control.

    Source: §1 · pp. 2–3, §5 · p. 8

  • Takeaways & Limitations

    The framework supports fine-grained multi-material, multi-object dynamics modeling and direct model-predictive control without explicit 3D scene reconstruction.

    Source: §1 · pp. 2–3, §3.2 · p. 4

    ParticleFormer is currently trained per scene and has not yet been shown to generalize across a wide variety of environments and robots.

    Source: §5 · p. 9

Abstract

from arXiv · show

3D world models (i.e., learning-based 3D dynamics models) offer a promising approach to generalizable robotic manipulation by capturing the underlying physics of environment evolution conditioned on robot actions. However, existing 3D world models are primarily limited to single-material dynamics using a particle-based Graph Neural Network model, and often require time-consuming 3D scene reconstruction to obtain 3D particle tracks for training. In this work, we present ParticleFormer, a Transformer-based point cloud world model trained with a hybrid point cloud reconstruction loss, supervising both global and local dynamics features in multi-material, multi-object robot interactions. ParticleFormer captures fine-grained multi-object interactions between rigid, deformable, and flexible materials, trained directly from real-world robot perception data without an elaborate scene reconstruction. We demonstrate the model's effectiveness both in 3D scene forecasting tasks, and in downstream manipulation tasks using a Model Predictive Control (MPC) policy. In addition, we extend existing dynamics learning benchmarks to include diverse multi-material, multi-object interaction scenarios. We validate our method on six simulation and three real-world experiments, where it consistently outperforms leading baselines by achieving superior dynamics prediction accuracy and less rollout error in downstream visuomotor tasks. Experimental videos are available at https://suninghuang19.github.io/particleformer_page/.

1 Introduction

ParticleFormer addresses the difficulty of modeling heterogeneous multi-object interactions by combining a Transformer-based point cloud world model with hybrid supervision. It avoids costly 3D reconstruction, extends evaluation to diverse multi-material scenarios, and supports dynamics prediction and model-based control.

Motivation

  • Existing 3D world models struggle with heterogeneous multi-material interactions, graph-topology sensitivity, and costly reconstruction of 3D particle tracks for real-world training.2D alternatives avoid reconstruction but can miss 3D structure and fine-grained dynamics in compositional scenes.Source: §1 · pp. 1–2

Method

  • ParticleFormer uses a Transformer-based dynamics backbone to infer detailed particle interactions without manually specified graph adjacency hyperparameters.Its attention mechanism models interactions across the full particle set rather than relying on predefined distance-based graphs.Source: §1 · pp. 2–3

Control

  • The model enables direct use of model predictive control for goal-directed robotic manipulation from its predicted dynamics.The paper frames this capability as a downstream consequence of accurate world-model predictions.Source: §1 · p. 2

Method

  • Its hybrid Chamfer-and-Hausdorff loss jointly captures local geometric accuracy and global structural consistency directly from RGB sensor streams without dense correspondence or explicit 3D reconstruction.This design provides fine-grained supervision for multi-material, multi-object dynamics modeling.Source: §1 · pp. 2–3

Evaluation

  • ParticleFormer consistently outperforms competitive baselines across six simulation tasks and three real-world experiments in dynamics prediction and downstream visuomotor control.The benchmark is extended to multi-material, multi-object robot interactions.Source: §1 · pp. 2–3

2 Related work

Prior world-model approaches either model local object interactions with topology-sensitive graphs or compress scenes into latent vectors that can lose structural detail. ParticleFormer retains object-centric structure while addressing graph-based limitations and targets multi-material interactions that prior work often overlooks.

Object-centric Modeling

  • Object-centric methods represent objects as particles or meshes and model local interactions with GNNs, but depend heavily on adjacency sizes and connection thresholds.These hyperparameter dependencies limit scalability to complex scenes.Source: §2 · p. 3

Scene-centric Modeling

  • Scene-centric methods avoid explicit topology by predicting latent dynamics from pixels, but single-vector scene representations can overlook structural details in compositional or fine-grained settings.This can impair generalization when detailed scene structure matters.Source: §2 · p. 3

ParticleFormer

  • ParticleFormer learns object-centric dynamics while preserving structural details and reducing reliance on manual interaction-graph tuning.The paper presents this combination as improving prediction accuracy, robustness, and generalizability relative to prior approaches.Source: §2 · p. 3

Multi-Material Robot-Object Interactions

  • Prior learned world models commonly focus on single-material robot-object scenarios, whereas ParticleFormer explicitly benchmarks heterogeneous multi-material interactions.The extended benchmark covers diverse material compositions and evaluates unified handling of differing physical properties.Source: §2 · p. 3

3 Method

ParticleFormer learns multi-material particle dynamics from partially observed point clouds using Transformer-based interaction modeling, hybrid geometric supervision, and model-based control. Its method spans problem formulation, point-cloud embedding and prediction, hybrid loss training, and predictive action planning.

3.1 Problem Formulation

  • ParticleFormer models partially observed object and end-effector point clouds, learning a world model that minimizes accumulated prediction error over time.The state includes downsampled object particles and end-effector points, with material encodings distinguishing particle types.Source: §3.1 · pp. 3–4

3.2 Transformer-Based Neural Dynamics Learning

  • Stereo reconstruction and segmentation produce task-relevant point clouds whose particle position, material, and motion features are embedded for dynamics prediction.FoundationStereo reconstructs dense point clouds, while GroundingDINO and Segment Anything isolate the manipulated object.Source: §3.2 · p. 4, Figure 2 · p. 4
  • A three-layer multi-head self-attention transition lets every particle attend to all others, implicitly learning interactions without predefined graphs or distance thresholds.A shared decoder then predicts each particle's 3D displacement and recovers the next position from the current position plus that displacement.Source: §3.2 · pp. 4–5

3.3 Hybrid Supervision

  • The hybrid supervision combines Chamfer Distance for local nearest-neighbor accuracy with Hausdorff Distance for worst-case deviations, covering dominant and passive motions without correspondences.Training autoregressively unrolls the model for k = 5 future steps using its own predictions as inputs.Source: §3.3 · p. 5

3.4 Model-Based Planning and Control

  • Model Predictive Path Integral control samples action sequences, rolls them out through the learned world model, and selects actions minimizing target distance, infeasibility, and collision costs.The planner operates over a predefined horizon H and replans at each control step.Source: §3.4 · p. 5

4 Experiment

ParticleFormer is evaluated across multi-material, multi-object dynamics prediction and downstream model-based control, using six simulation and three real-world tasks. It consistently captures richer interactions, achieves stronger prediction accuracy, and produces lower-error MPC rollouts than the evaluated baselines.

4.1 Experiment Setup and Task Design

  • The evaluation spans six simulation tasks and three real-world tasks, covering single-material and multi-material interactions and testing both prediction and visuomotor control.The experimental questions address physical plausibility, baseline comparison, and downstream manipulation performance.Source: §4.1 · pp. 5–6, §4 · p. 5

4.2 Qualitative Evaluation on Dynamics Prediction

  • ParticleFormer produces more accurate one-step predictions across box pushing, cloth gathering, and rope sweeping, including indirect motion and multi-material interactions.Compared with GBND and the no-hybrid ablation, it better captures rigid motion, force propagation, rope deformation, and induced granular displacement.Source: §4.2 · pp. 6–7, Figure 3 · p. 6
  • ParticleFormer’s attention heatmaps reveal structured intra- and inter-material interactions, including rope connectivity, granular clusters, force transfer, and asymmetric end-effector attention.These patterns indicate that the model learns fine-grained interaction structure without predefined topologies.Source: Figure 3 · p. 6, §4.2 · p. 7

4.3 Quantitative Evaluation on Dynamics Prediction

  • ParticleFormer consistently achieves the best quantitative dynamics-prediction performance across metrics, particularly MSE and CD+HD, indicating stronger local and global accuracy.Table 1 reports results on three multi-material simulation tasks; DINO-WM is evaluated with point-cloud metrics because it lacks particle correspondences.Source: §4.3 · p. 7, Table 1 · p. 7
  • Increasing GBND’s TopK from 5 to 15 lowers prediction error but rapidly raises computational cost, whereas ParticleFormer achieves lower error without manual graph-hyperparameter tuning.ParticleFormer uses soft attention rather than a manually specified neighbor limit, improving performance and scalability.Source: Figure 4 · p. 7, §4.3 · p. 8

4.4 Model-Based Planning and Control

  • ParticleFormer enables more accurate MPC toward novel target configurations, achieving lower final-state errors and finer manipulation behaviors than baselines across three real-world tasks.The controller uses MPPI with normalized CD+HD as its planning cost and operates in closed loop.Source: §4.4 · p. 8, Figure 5 · p. 8

5 Conclusion

The paper concludes that ParticleFormer improves robotic dynamics modeling through Transformer-based soft interaction modeling and hybrid point-cloud supervision. It reports effectiveness across synthetic and real-world multi-material settings while identifying scene-specific training and mask dependence as current boundaries.

5 Conclusion

  • ParticleFormer combines Transformer-based particle parametrization with hybrid point-cloud supervision to model soft interactions, global structure, and local motion details.The conclusion presents these as the two key designs behind improved dynamics modeling accuracy and robustness.Source: §5 · p. 8
  • The extended benchmarks cover multi-material scenarios, and experiments demonstrate effectiveness across both synthetic and real-world robotic manipulation settings.The conclusion characterizes the framework as capable and easy to use for robotic world-model applications.Source: §5 · p. 8
  • ParticleFormer is trained per scene, has not been shown to generalize widely across environments and robots, and depends on externally generated object masks whose failures can propagate to prediction.The authors suggest broader scene-agnostic training and integrating semantic codes without explicit masking as future directions.Source: §5 · p. 9

A Experiment Implementation Details

The implementation evaluates ParticleFormer in simulation and real-world manipulation using xArm-6 robots, stereo RGB sensing, and task-specific end-effectors. Simulation uses six benchmark tasks with 1,000 training episodes each, while real-world tasks use recorded random-interaction episodes.

A.1 Setup

  • Simulation extends a FleX-based benchmark to six tasks, including three multi-material tasks, using xArm-6 robots and 1,000 training episodes per task.End-effectors include cylindrical and planar pushers and a parallel gripper.Source: §A.1 · p. 13, Figure 6 · p. 13
  • Real-world experiments use one or two six-degree-of-freedom xArm-6 arms, task-specific pushers or grippers, and a front-facing ZED-2i stereo camera.The camera records 1280×720 stereo RGB images at 10 Hz in an approximately 70 cm × 55 cm workspace.Source: §A.1 · p. 13, Figure 7 · p. 14

A.2 Task Design

The task suite spans deformable, granular, rigid, and multi-material interactions, evaluating both dynamics prediction and downstream control. It includes six simulation tasks and three real-world tasks with increasingly indirect force transmission and coordinated manipulation.

A.2 Task Design

  • Rope, Granular Object, and Cloth isolate deformable or granular dynamics, testing rope deformation, disjoint-particle motion, and fabric manipulation.These single-material tasks focus on direct robot-object interaction.Source: §A.2 · p. 14
  • Box Pushing tests rigid-body translation and rotation, while Cloth Gathering combines cloth deformation with passive granular flow caused by lifting.The latter requires prediction of granular motion induced by localized fabric manipulation.Source: §A.2 · p. 14
  • Rope Maneuver and Rope Sweeping require indirect force transmission between rope, rigid or granular objects, and coordinated robot arms.Rope Maneuver avoids tipping a box, while Rope Sweeping surrounds and moves granular particles through rope deformation and friction.Source: §A.2 · p. 15
  • The benchmark includes Rope, Granular Object, Cloth, Cloth Gathering, Rope Maneuver, and Rope Sweeping in simulation, plus Box Pushing, Cloth Gathering, and Rope Sweeping in the real world.These tasks jointly assess dynamics modeling and downstream visuomotor control under complex scenarios.Source: §A.2 · pp. 14–15

B Additional Results for Dynamics Prediction

This section reports additional simulation results for dynamics modeling comparison.

B.1 Dynamics Prediction Comparison

ParticleFormer outperforms the compared methods on three single-material simulation tasks using both MSE and CD+HD metrics. The table evaluates prediction errors using ground-truth object states, with point-cloud metrics used for DINO-WM.

B.1 Dynamics Prediction Comparison

  • ParticleFormer achieves superior MSE and CD+HD performance across three single-material simulation tasks.Source: §B.1 · p. 15
  • The comparison evaluates dynamics prediction errors where ground-truth object-state information is available.Source: Table 2 · p. 15
  • DINO-WM is evaluated only with point-cloud metrics because it lacks point-to-point particle correspondences.Source: Table 2 · p. 15

B.2 Hyperparameter Analysis

The analysis shows that GBND is sensitive to graph-construction hyperparameters, whereas ParticleFormer avoids these dependencies through soft attention. Increasing TopK raises memory use, and MaxDist changes GBND accuracy substantially.

TopK Analysis

  • Increasing GBND’s TopK substantially increases GPU memory usage, exposing a scalability bottleneck in graph-based dynamics modeling.Source: §B.2 · p. 15, Figure 8 · p. 16
  • ParticleFormer avoids explicit neighbor selection and its associated memory-growth issue through soft attention.Source: Figure 8 · p. 16

MaxDist Analysis

  • GBND dynamics accuracy varies significantly across all-particle, MaxDist = 0.3, and MaxDist = 0.7 edge-construction settings on Cloth.Source: §B.2 · p. 16, Figure 9 · p. 16
  • ParticleFormer uses Transformer soft attention to bypass edge-construction heuristics and provide more stable, accurate performance.Source: §B.2 · p. 16, Figure 9 · p. 16

C Additional Results for Model-Based Planning and Control

ParticleFormer supports model-based visuomotor control in three multi-object, multi-material simulation tasks. Compared with baselines, it produces more accurate planning, finer control behavior, and lower final-state mismatch on novel targets.

C Additional Results for Model-Based Planning and Control

Loading 2506.23126v4…