PaperlayerView exact revision on arXiv

Source-linked AI summary

Particle-Grid Neural Dynamics for Learning Deformable Object Models from RGB-D Videos

Kaifeng Zhang, Baoyu Li, Kris Hauser, Yunzhu Li

arXiv:2506.15680v2cs.ROcs.CVcs.LG

TL;DR

Deformable-object dynamics are difficult to learn from limited visual observations because objects vary in shape and physical properties. The paper combines particles and spatial grids in a neural dynamics model trained from RGB-D robot interactions, and reports strong performance across diverse objects, sparse views, rendering, and manipulation planning. The method remains limited by fixed particle counts, implicit physical-property inference, and dependence on perception and reconstruction accuracy.

  • Problem

    Accurate and generalizable deformable-object dynamics are difficult to learn when physical properties vary and visual observations are limited.

    Source: §I · p. 1

  • Method

    A hybrid particle-grid neural model learns dense particle dynamics from RGB-D videos, integrates with Gaussian Splatting, and supports model-based planning.

    Source: §I · p. 2, 4) The Particle-Grid Model · p. 3, C. Data Collection and Training · p. 5

  • Results

    The model simulates diverse deformable objects, outperforms prior graph-based dynamics models in prediction accuracy and modeling density, and supports sparse-view video prediction and manipulation.

    Source: §I · p. 2, §VI · p. 10

  • Takeaways & Limitations

    The framework provides a learning-based digital twin that jointly models deformable-object geometry, appearance changes, and action-conditioned motion.

    Source: §VI · p. 10

    The framework assumes a fixed number of particles and depends on accurate perception and reconstruction models, whose failures can reduce performance.

    Source: §V · p. 10

Abstract

from arXiv · show

Modeling the dynamics of deformable objects is challenging due to their diverse physical properties and the difficulty of estimating states from limited visual information. We address these challenges with a neural dynamics framework that combines object particles and spatial grids in a hybrid representation. Our particle-grid model captures global shape and motion information while predicting dense particle movements, enabling the modeling of objects with varied shapes and materials. Particles represent object shapes, while the spatial grid discretizes the 3D space to ensure spatial continuity and enhance learning efficiency. Coupled with Gaussian Splattings for visual rendering, our framework achieves a fully learning-based digital twin of deformable objects and generates 3D action-conditioned videos. Through experiments, we demonstrate that our model learns the dynamics of diverse objects -- such as ropes, cloths, stuffed animals, and paper bags -- from sparse-view RGB-D recordings of robot-object interactions, while also generalizing at the category level to unseen instances. Our approach outperforms state-of-the-art learning-based and physics-based simulators, particularly in scenarios with limited camera views. Furthermore, we showcase the utility of our learned models in model-based planning, enabling goal-conditioned object manipulation across a range of tasks. The project page is available at https://kywind.github.io/pgnd .

I. INTRODUCTION

The paper introduces Particle-Grid Neural Dynamics to model diverse deformable objects from partial RGB-D observations, combining particles, spatial grids, and learned motion prediction. The framework supports dense dynamics prediction, realistic 3D video rendering, and model-based manipulation.

A. Physics-Based Deformable Modeling.

B. Learning-Based Deformable Modeling.

I. INTRODUCTION

  • Particle-Grid Neural Dynamics combines object particles with fixed spatial grids to predict dense motion while remaining robust to incomplete observations and diverse materials.The particle representation captures object geometry, while the grid enforces spatial continuity and reduces computational costs for large particle sets.Source: §I · p. 2
  • The framework learns from RGB-D robot-object videos using 3D fusion and tracking to produce persistent, dense particle trajectories for training.Source: §I · p. 2

III. METHODS

  • The model integrates with 3D Gaussian Splatting for action-conditioned video prediction and with MPC for deformable-object manipulation.Source: §I · p. 2, Figure 2 · p. 3

1) State Representation:

2) Action Representation:

The dynamics function represents robot actions and historical object states, then uses a hybrid particle-grid model to predict spatially continuous motion. Grid-based components connect learned particle features, velocity fields, action effects, and particle updates.

2) Action Representation:

  • Robot actions are represented using an action-type label, end-effector pose, pose derivative, and gripper opening distance.The action label distinguishes grasped from nonprehensile interactions.Source: 2) Action Representation · p. 3

3) Dynamics Function:

  • The dynamics function uses historical particle states because positions alone omit information such as internal stress and contact mode.A neural network approximates the dynamics function and uses forward Euler integration to derive the next particle positions.Source: 3) Dynamics Function · p. 3

4) The Particle-Grid Model:

  • The particle-grid model uses a fixed spatial grid whose limits and resolution are controlled by l_x, l_y, l_z, and δ.The grid is translated with the particles and end effector to enforce translational invariance during prediction.Source: 4) The Particle-Grid Model · pp. 3–4
  • The hybrid grid representation injects spatial-continuity and local-information inductive biases to stabilize neural dynamics predictions affected by accumulated rollout error.Source: 4) The Particle-Grid Model · p. 3
  • The model maps particle features to a neural velocity field, edits grid velocities for collision and gripper effects, and integrates the field back to particle velocities.Source: 4) The Particle-Grid Model · p. 4

B. Model Components

The model encodes particle states globally, predicts velocities on a spatial grid, transfers them back to particles, and supports both grasped and nonprehensile interactions.

1) Point Encoder:

  • The PointNet encoder extracts global shape and historical-motion features from particle positions and velocities, supporting occlusion-robust dynamics prediction.These features implicitly capture physical properties and dynamic state under incomplete observations.Source: 1) Point Encoder · p. 4

2) Neural Velocity Field:

  • The neural velocity field uses grid locations and neighborhood-pooled particle features to predict local per-grid velocity vectors.A radius parameter controls which particle features each grid point attends to, encouraging locality-aware predictions.Source: 2) Neural Velocity Field · p. 4
  • Continuous B-spline interpolation transfers predicted grid velocities to particle velocities, producing the dynamics output for time integration.Closer grid-particle pairs receive larger interpolation weights, yielding smooth spatial transfer.Source: 2) Neural Velocity Field · p. 4

4) Controlling Deformation:

  • Grid Velocity Editing modifies grid velocities to enforce ground contact and rigid grasp constraints during external interactions.The method projects velocity from contact surfaces with friction terms and constrains grid points near the grasp center to gripper motion.Source: 4) Controlling Deformation · p. 4
  • Robot Particles inject gripper action information into the point cloud without prescribing particle velocities, enabling nonprehensile manipulation.The implementation samples gripper-shape points and computes their velocities from end-effector transformations.Source: 4) Controlling Deformation · p. 4

C. Data Collection and Training

The pipeline learns from automatically tracked multi-view RGB-D interactions, renders predicted motion with 3D Gaussian Splatting, and supports model-predictive planning.

C. Data Collection and Training

  • Foundation-model segmentation, 2D tracking, and depth-based inverse projection produce fused 3D point clouds with persistent particle trajectories.Segment Anything extracts object masks, CoTracker supplies trajectories, and multi-view fusion maps them into 3D.Source: C. Data Collection and Training · pp. 4–5
  • Training rolls out the differentiable dynamics model for K=5 steps over tracked trajectories using MSE between predicted and actual particle positions.The history window is h=2, and network parameters are optimized by gradient descent.Source: C. Data Collection and Training · p. 5

D. Rendering and Action-Conditioned Video Prediction

  • 3D Gaussian Splatting renders predicted object motion by transforming Gaussian centers and rotations through particle-motion interpolation.Linear Blend Skinning uses predicted point motions as control points while color, scale, and opacity remain constant.Source: D. Rendering and Action-Conditioned… · p. 5

E. Planning

  • Model Predictive Control rolls out the learned dynamics under sampled actions and uses Chamfer Distance to optimize toward a target state.MPPI performs trajectory optimization, enabling online iterative closed-loop planning.Source: E. Planning · p. 5

IV. EXPERIMENTS

Experiments evaluate deformable-object prediction across object types, camera-view availability, category-level generalization, and downstream video and planning applications.

IV. EXPERIMENTS

  • The experiments test dynamics learning, sparse-view robustness, category-level generalization, and downstream action-conditioned video prediction and planning.These questions define the evaluation scope across prediction and applications.Source: §IV · p. 5

A. Experiment Setup

  • The method is evaluated on cloth, rope, plush toys, bags, boxes, and bread, with comparisons against physics-based, graph-based, and particle-only baselines.The setup uses bimanual seven-degree-of-freedom xArm robots and diverse object interactions.Source: §IV · p. 5, A. Experiment Setup · pp. 6–7

IV. EXPERIMENTS

  • The method is reported to outperform previous state-of-the-art approaches in dynamics prediction accuracy while remaining robust to incomplete camera views.The evaluation covers a diverse set of challenging deformable objects.Source: §IV · p. 5
  • Qualitatively, the model aligns more closely with ground truth than GBND, producing denser particle predictions and fewer artifacts.The comparison overlays predictions with ground-truth final-state images and marks gripper pose with red spheres.Source: Figure 3 · p. 6
  • The model also supports category-level training and downstream applications including 3D action-conditioned video prediction and model-based planning.These capabilities are validated as additional experimental outcomes.Source: §IV · p. 6

B. Dynamics Learning and Prediction

The model achieves strong deformable-object dynamics prediction across sparse-view inputs and unseen category instances, outperforming the GBND baseline in the supplied evaluations.

B. Dynamics Learning and Prediction

  • The experiments evaluate held-out 3-second robot-object interaction clips using Mean Distance Error, Chamfer Distance, and Earth Mover’s Distance.For boxes, MPM is omitted because the Robot Particles control representation is not directly compatible with it.Source: B. Dynamics Learning and Prediction · p. 7
  • Our method outperforms all baselines in dynamics rollout accuracy across the evaluated deformable-object categories.Predictions also align more closely with ground-truth images and have higher resolution, while GBND produces inadequate motions or rope artifacts.Source: B. Dynamics Learning and Prediction · p. 7

C. Sparse-View Dynamics Prediction

  • Our model outperforms GBND in dynamics prediction accuracy regardless of whether evaluation uses 1 to 4 camera views.Its performance degrades less as views decrease; for cloth, it maintains low error with one view while the baseline drops substantially.Source: C. Sparse-View Dynamics Prediction · pp. 7–8

D. Category-Level Model

  • Across ropes and cloths, our model achieves lower prediction errors than GBND on both seen and unseen instances.The advantage is especially pronounced for unseen cloth instances, where the baseline’s performance drops significantly but our model retains relatively low error.Source: D. Category-Level Model · pp. 8–9

E. Action-Conditioned Video Prediction

The learned dynamics model drives Gaussian-Splatting video prediction and model-based manipulation, while the paper identifies fixed-particle rollouts and perception dependence as limitations.

E. Action-Conditioned Video Prediction

  • Our approach achieves the best overall action-conditioned video prediction performance using Gaussian Splatting and predicted particle trajectories.Spiky reconstructions can reduce mask performance for large, strongly deforming objects, while higher baseline mask scores largely reflect inadequate particle motion.Source: E. Action-Conditioned Video Prediction · pp. 9–10
  • Gaussian Splatting combined with particle-grid dynamics generates higher-quality rendered videos that align better with ground truth and supports high-fidelity simulation from phone-scanned scenes.The pipeline reconstructs Gaussians from four views and deforms them using predicted particle trajectories.Source: E. Action-Conditioned Video Prediction · p. 10

F. Planning

  • Across four manipulation tasks, our method achieves lower terminal error and a higher error reduction rate than GBND.It also produces visually closer targets; examples include successful two-sided box lifting and accurate rope bending where the baseline struggles.Source: F. Planning · p. 10

V. LIMITATIONS

  • The framework assumes a fixed number of particles during iterative rollout, so it is inapplicable to particle appearance or disappearance without future corrections.The authors suggest correcting particle sets with new observations or modeling per-frame particle visibility.Source: §V · p. 10
  • The method depends on accurate segmentation, tracking, and Gaussian-Splatting predictions, so failures in these perception and reconstruction models can reduce performance.Source: §V · p. 10

VI. CONCLUSION

  • The paper presents Particle-Grid Neural Dynamics as a sparse-view RGB-D framework that models diverse deformable objects and improves manipulation efficiency and success rate.Integration with 3D Gaussian Splatting provides a learning-based digital twin capturing geometry and appearance changes.Source: §VI · p. 10

APPENDIX

The appendix describes a foundation-model-based pipeline for extracting persistent 3D motion tracks and transferring particle motion to Gaussian kernels for video rendering.

A.1 Data Collection

  • The training-data pipeline uses foundational vision models for object detection and segmentation, 2D tracking with 3D velocity computation, and 3D tracking extraction.Grounded-SAM-2 [38] [18] extracts object masks, while CoTracker [15] predicts 2D trajectories from cropped video clips.Source: §A.1 · p. 14
  • Per-pixel 3D velocities are obtained by inverse-projecting tracked pixel positions with depth and camera transformations, then fused across camera views.Velocity prediction is restricted to the first 15 frames of each overlapping clip so every frame receives estimates.Source: §A.1 · p. 14
  • The segmentation and tracking pipeline avoids differentiable rendering and optimization, improving computational efficiency while handling textureless deformable objects such as cloth and bread.Source: §A.1 · p. 14

A.2 Video Prediction

  • Gaussian-kernel transformations are interpolated from nearby particle rotations and translations using Linear Blend Skinning.The implementation uses eight nearest particles for both rotation estimation and LBS interpolation, with larger weights assigned to spatially closer particles.Source: §A.2 · pp. 14–15
  • The experiment setup uses four calibrated RGB-D cameras and evaluates materials including paper bags, stuffed animals, bread, boxes, cloth, and ropes.Source: Figure 10 · p. 15

A.3 Model-Based Planning

The planning appendix specifies task-dependent action spaces and samples candidate trajectories for MPPI evaluation with the learned dynamics model.

A.3 Model-Based Planning

  • Single-arm rope and plush-toy tasks use 3-DoF end-effector positions, while bimanual cloth and box tasks use 12-DoF translations and rotations.Source: §A.3 · p. 15
  • MPPI samples delta actions from a normal distribution and adds them to the current optimal action sequence to construct candidate trajectories.Source: §A.3 · p. 15
  • Candidate action trajectories are evaluated with the learned dynamics model using Chamfer Distance, with an additional gripper-separation penalty for cloth lifting.The penalty applies when the gripper distance exceeds L = 0.6 m.Source: §A.3 · p. 15

APPENDIX B IMPLEMENTATION DETAILS

The appendix describes the experimental setup and implementation of physics-based, graph-based, and particle-only baselines. It also explains the particle-only ablation’s limitations relative to the full particle-grid model.

B.1 Experiment Setup

  • The experiments use six deformable-object categories, with cloth and rope evaluated in dynamics prediction and the remaining categories used for category-level model experiments.The setup includes a paper bag, plush toy, bread, boxes, cloth pieces, and ropes with varied physical and visual properties.Source: §B.1 · p. 16

B.21 MPM:

  • The MPM baseline assumes hyperelastic materials with fixed-corotated elasticity, identity plasticity, shared Young’s modulus and friction parameters, and zero initial strain.Its parameters are optimized using the same training setting and loss framework as the particle-grid model.Source: §B.21 · p. 16

B.22 GBND:

  • The GBND baseline represents object and gripper particles as a dynamically constructed nearest-neighbor graph and supervises four-step predictions with position and edge-length losses.Each node uses a two-step position history, and the graph limits each node to five edges.Source: §B.22 · p. 16

B.23 Particle:

  • The Particle baseline removes the fixed grid and directly queries the velocity field at particle positions while retaining the remaining model and training procedure.Particle velocities are edited for grasping actions and table collisions using locally averaged particle features.Source: §B.23 · p. 16
  • The particle-only ablation must predict the entire particle set, making its velocity field harder to learn and leaving spatial smoothness unregularized under high-frequency positional encoding.The appendix reports that this simplification reduces dynamics prediction accuracy.Source: §B.23 · p. 16

B.3 Method Implementation

The implementation uses GPU-accelerated particle-grid dynamics with object-specific scaling and grid sizes, and evaluates partial-view prediction, unseen-instance generalization, and qualitative baseline comparisons. These experiments show accurate predictions from incomplete observations, generalization to unseen objects, and better qualitative alignment than the compared baselines.

B.3 Method Implementation

  • The model uses NVIDIA Warp for differentiable grid-velocity editing and G2P, with 0.1 s timesteps and object-specific scaling and grid resolutions.Bread uses s = 3.0 and δ = 1 cm, cloth uses s = 0.8 and δ = 2 cm, and other categories use s = 1.0 and δ = 2 cm.Source: §B.3 · pp. 16–17

C.1 Qualitative Comparisons on Partial Views

  • Single-view partial observations still produce predictions that closely match ground-truth future particle positions and frames.The evaluation uses highly incomplete particle data from one depth view.Source: Figure 11 · p. 17, §C.1 · p. 17

C.3 Qualitative Comparisons on Dynamics Prediction

  • Compared with MPM, GBND, and Particle baselines, the particle-grid model aligns best with ground truth and exhibits fewer artifacts across qualitative dynamics predictions.Baseline failures include drifting or scattered MPM motions and inadequate or gripper-adjacent Particle motions.Source: §C.3 · pp. 17–18

C.2 Qualitative Comparisons on Generalization

  • On unseen cloth and rope instances, the model generalizes to novel objects and produces realistic rendered predictions compared with ground truth.The qualitative visualizations use Gaussian Splatting for video prediction.Source: §C.2 · p. 17, Figure 12 · p. 18

C.4 Additional Quantitative Experiments

Additional experiments assess runtime, deformation-control choices, grid and encoder configurations, and comparison with MPM. The results support fast inference, action-dependent control choices, and advantages of the particle-grid representation under partial observations.

C.41 Runtime Analysis:

  • 4.8 ms per forward pass at batch size 1 and 11.9 ms at batch size 50 enable real-time, batched inference for planning.Runtime is measured on the rope test set.Source: §C.41 · p. 18

C.42 Ablation Studies:

  • Grid Velocity Editing performs better for prehensile tasks such as cloth, whereas Robot Particles are more effective for nonprehensile cases such as boxes.This ablation supports using different deformation-control mechanisms for different action types.Source: §C.42 · p. 18
  • A 1.25 cm grid and PTv3 encoder achieve performance comparable to the default design but require higher computational cost.The comparison evaluates grid sizes of 1.25 cm and 4 cm and shared MLP, PTv3, and PointNet encoders.Source: §C.42 · p. 18, Table V · p. 18

C.43 Further Comparison with MPM:

  • MPM underperforms the particle-grid model across all rope tasks because it cannot handle partial observations from depth inputs.The comparison is reported in Table VI and concerns a model-based simulator with explicit material parameters.Source: §C.43 · p. 18, Table VI · p. 18
  • Qualitative comparisons show fewer artifacts for the particle-grid model than MPM, GBND, and Particle across ropes, paper bags, cloth, plush toys, boxes, and bread.The baselines exhibit mismatched motions, unwanted breakage, or insufficient particle movement depending on the object.Source: Figure 13 · p. 19, §C.43 · p. 19, Figure 14 · p. 20

Related papers

Loading 2506.15680v2…