Source-linked AI summary
Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluids
Yunzhu Li, Jiajun Wu, Russ Tedrake, Joshua B. Tenenbaum, Antonio Torralba
TL;DR
Complex materials challenge rigid-body simulators, while approximate particle simulators can depart from real-world physics over long horizons. The paper learns DPI-Nets, a differentiable particle-based simulator with dynamic and hierarchical interactions, and combines it with perception and gradient-based control. Experiments show manipulation of deformable objects and fluids, rapid adaptation to unknown physical parameters, and real-world demonstrations.
Problem
Different materials exhibit distinct dynamics, challenging rigid-body simulators, while approximate particle simulations can deviate from real-world physics over long horizons.
Method
The paper learns DPI-Nets, a differentiable particle-based simulator combining dynamic interaction graphs, hierarchical particle structure, multi-step propagation, perception, and gradient-based control.
Results
DPI-Nets support manipulation of deformable objects and fluids, outperform several baselines, adapt to unknown physical parameters online, and generalize to real-world experiments.
Takeaways & Limitations
Particle-based learned dynamics can approximate interactions across rigid, deformable, and fluid objects while supporting complex robot manipulation tasks.
Takeaways & Limitations
Interaction networks using only local information at each time step cannot handle instantaneous force propagation common in rigid-body dynamics.
Abstract
from arXiv · showhide
Real-life control tasks involve matters of various substances---rigid or soft bodies, liquid, gas---each with distinct physical behaviors. This poses challenges to traditional rigid-body physics engines. Particle-based simulators have been developed to model the dynamics of these complex scenes; however, relying on approximation techniques, their simulation often deviates from real-world physics, especially in the long term. In this paper, we propose to learn a particle-based simulator for complex control tasks. Combining learning with particle-based systems brings in two major benefits: first, the learned simulator, just like other particle-based systems, acts widely on objects of different materials; second, the particle-based representation poses strong inductive bias for learning: particles of the same type have the same dynamics within. This enables the model to quickly adapt to new environments of unknown dynamics within a few observations. We demonstrate robots achieving complex manipulation tasks using the learned simulator, such as manipulating fluids and deformable foam, with experiments both in simulation and in the real world. Our study helps lay the foundation for robot learning of dynamic scenes with particle-based representations.
1 INTRODUCTION
The paper learns a differentiable particle-based simulator for controlling rigid bodies, deformable objects, and fluids. DPI-Nets combine dynamic, hierarchical, and long-range particle interactions with perception and gradient-based control, supporting adaptation and manipulation in simulation and the real world.
- Traditional rigid-body simulators struggle with objects whose dynamics differ, while approximate particle simulators can diverge from real-world physics over long horizons.
- DPI-Nets learn dynamic particle interactions, hierarchical structure, and long-range propagation for differentiable simulation and control of deformable objects and fluids.
- The learned simulator captures deformable objects, fluids, and rigid bodies, enabling manipulation tasks such as molding plasticine to a target shape.
- Particles of the same type share dynamics within the representation, helping DPI-Nets adapt quickly to unknown physical parameters through online system identification.
- Experiments report strong baseline performance and successful downstream manipulation of deformable objects and fluids, including scenarios requiring online parameter identification and real-world generalization.
2 RELATED WORK
Related work covers differentiable simulators, learned interaction graphs, model-predictive control, and propagation networks. The paper distinguishes DPI-Nets by modeling deformable particle dynamics with state-specific representations and dynamic graphs, then directly optimizing control trajectories.
- Prior differentiable-simulator research includes analytical-gradient engines and neural models that learn object interactions or propagate signals along interaction graphs.
- Differentiable simulation for deformable objects has received less study, with SPNets addressing position-based fluids and HRN approximating particle dynamics of deformable shapes.
- Recent work also learns interaction graphs and augments learned dynamics with perception modules for raw visual input.
- Model-predictive-control methods often learn abstract state transitions to facilitate policy training rather than explicitly representing the environment.
- Unlike interaction-network planning approaches based on policy rollouts, DPI-Nets learn a dynamics simulator and directly optimize trajectories for continuous control.
3 APPROACH
The approach extends interaction networks into a learnable particle-based physical engine for modeling and controlling rigid bodies, deformable objects, and fluids. It combines multi-step propagation, hierarchical structure, and dynamically built interaction graphs to capture short- and long-range particle interactions.
- 3.1 PRELIMINARIES: Interaction networks represent physical systems as directed graphs whose vertices encode object states and attributes, while edges encode directed relations and their attributes.Object states include position and velocity; attributes can include mass and radius, while relation attributes can encode collision or spring connections.
- 3.1 PRELIMINARIES: Propagation networks address instantaneous force propagation by performing multi-step message passing over object and relation representations.The model reuses encoded information across propagation steps, with relation and object propagators updating interaction influences and object states.
- 3.2 DYNAMIC PARTICLE INTERACTION NETWORKS: DPI-Nets extend object-level interaction models to particle-level deformation using dynamic particle graphs and material-specific propagation rules.Particles form graph vertices, and edges are generated over time, commonly by connecting particles within a predefined distance.
- 3.2 DYNAMIC PARTICLE INTERACTION NETWORKS: Hierarchical modeling propagates information from particles to roots, among roots, and back to particles to represent long-range dependence efficiently.The multi-stage sequence is leaf-to-leaf, leaf-to-root, root-to-root, and root-to-leaf propagation, with leaf signals used for final prediction.
- 3.2 DYNAMIC PARTICLE INTERACTION NETWORKS: Material-specific models handle rigid transformations, deformable-object resting positions, and local fluid neighborhoods with dynamic edges.Rigid bodies use averaged particle signals for rotation and translation; elastic or plastic objects predict current and resting-position motion; fluids use local neighbors to enforce density and incompressibility.
- 3.3 CONTROL ON THE LEARNED DYNAMICS: The learned differentiable dynamics can be rolled out for trajectory optimization and online system identification when physical parameters are unknown.Control inputs are optimized by minimizing the distance between the simulated outcome and a target configuration, while identification minimizes prediction error against reality.
4 EXPERIMENTS
The experiments evaluate DPI-Net across four simulated environments, forward-simulation variants, ablations, and control tasks in simulation and the real world. Results show advantages from dynamic graphs, hierarchy, state-specific predictors, and online adaptation.
- Experimental setup: DPI-Net is evaluated in FluidFall, BoxBath, FluidShake, and RiceGrip, covering fluids, rigid bodies, and deformable objects.The evaluation uses rollout MSE for simulation and Chamfer distance for control.
- Forward simulation: The forward-simulation comparison uses fixed hyperparameters across all four environments and compares DPI-Net with Interaction Networks, HRN, and a no-hierarchy variant.FluidFall and FluidShake use no hierarchy, so DPI-Net matches its no-hierarchy variant in those environments.
- Forward simulation: DPI-Net outperforms HRN on fluid environments BoxBath and FluidShake, where dynamic graphs handle changing particle neighborhoods.Without dynamic graphs, fluid modeling becomes difficult because neighboring particles change constantly.
- Ablation studies: Ablations show that fewer hierarchy roots improve RiceGrip motion capture, longer propagation is not consistently better, and larger neighborhoods improve results but slow computation.RiceGrip forward-step times are 30ms for d = 0.04, 33ms for d = 0.08, and 40ms for d = 0.12 on one TITAN Xp.
- Ablation studies: State-specific motion predictors outperform a unified predictor in BoxBath while remaining nearly as efficient, with a time difference smaller than 3ms per forward step.The comparison reflects the distinct physical behaviors of solids, liquids, and soft bodies.
- Control: Control experiments show that RS + TO outperforms random search and model-free PPO in FluidShake, while RiceGrip performs best with online system identification and transfers to real-world foam manipulation.FluidShake requires 600K time steps for the proposed method versus more than 10 million for RL; RiceGrip uses 150K versus 10M for RL.
5 CONCLUSION
The paper shows that learned particle dynamics can model interactions among diverse objects and support complex manipulation of deformable objects, with potential deployment in household and manufacturing environments.
- A learned particle dynamics model approximates interactions among diverse objects and helps solve complex manipulation tasks involving deformable objects.
A CONTROL ALGORITHM
The control algorithm jointly updates estimated physical attributes and control inputs while repeatedly simulating learned dynamics toward a goal state.
- At each time step, the algorithm updates physical-attribute estimates by descending the state loss gradient.
- It forward-simulates the current dynamics graph and stores predicted graphs in a state-history buffer.
- The algorithm predicts subsequent graphs through the time horizon using repeated forward simulation with the learned dynamics function.
- It updates the control sequence by descending the goal loss gradient, then returns the controls and predicted next graph.
B GENERALIZATION ON EXTRAPOLATION
The model is evaluated on fluids, rigid bodies, and deformable objects with more particles than seen during training, and the reported results show that it scales to larger objects.
- The model’s performance is evaluated on fluids, rigid bodies, and deformable objects containing more particles than the training set.
- The experiments report qualitative and quantitative results for scaling the model to larger objects.
C DATA GENERATION
The experiments use data generated with NVIDIA FleX across fluid, rigid-body, and deformable-object environments, with randomized initial conditions and a held-out validation split.
- The data-generation pipeline uses NVIDIA FleX and a Python interface for creating and interacting with different environments.
- FluidFall contains 3,000 rollouts over 120 time steps with two fluid drops containing 64 and 125 particles.
- BoxBath contains 3,000 rollouts over 150 time steps, combining 960 fluid particles with a rigid cube of 27 to 150 particles.
- FluidShake contains 2,000 rollouts over 300 time steps, with randomized fluid dimensions and initial box position.
- RiceGrip randomizes the initial rice-cuboid size, material properties, and gripper configuration for generated rollouts.
- Figure 6 evaluates extrapolation using MSE (×10−2), contrasting training-range particle counts with cases containing two times more particles.
- Ninety percent of generated rollouts are used for training and 10% for validation.
D TRAINING DETAILS
Training uses environment-specific particle propagation and motion-prediction architectures, with hierarchical models receiving multiple propagation networks where needed. The environments differ in training duration and output representation.
- Optimization: Across the implementation, models use Adam with learning rate 0.0001, batch size 1, and weight updates once every two forward rounds.
- FluidFall: FluidFall predicts three-dimensional particle velocities, which are multiplied by ∆t and added to current positions for rollouts.
- BoxBath: BoxBath uses separate fluid and rigid-body motion predictors, outputting particle velocities for fluids and rigid transformations for the rigid body.
- BoxBath: BoxBath uses four propagation networks, eight rigid-cube roots, and five training epochs.
- FluidShake: FluidShake uses one propagation network and is trained for five epochs.
- RiceGrip: RiceGrip uses four propagation networks with 30 roots for the rice and is trained for 20 epochs.
E CONTROL DETAILS
Control selects candidate sequences using the learned model, then refines inputs with RMSprop before applying them through model-predictive control. The procedures vary by environment and include online physical-parameter estimation in the real-world setting.
- Candidate selection: For three cases, 20 random control sequences are sampled and the best is selected using the learned model and Chamfer distance to the target configuration.
- FluidShake: FluidShake optimizes the selected control inputs with RMSprop for 10 iterations at learning rate 0.003 before model-predictive control.
- RiceGrip: RiceGrip optimizes grip configurations with RMSprop for 20 iterations at learning rate 0.003 before model-predictive control.
- RiceGrip in Real World: In real-world RiceGrip, physical parameters are estimated online, then the refined control sequence is applied through model-predictive control.