Source-linked AI summary
EMPM: Embodied MPM for Modeling and Simulation of Deformable Objects
Yunuo Chen, Yafei Hu, Lingfeng Sun, Tushar Kusnur, Laura Herlant, Chenfanfu Jiang
TL;DR
Modeling deformable objects from vision remains challenging because existing methods can oversimplify their dynamics or require large training sets. EMPM combines multi-view RGB-D reconstruction with differentiable MPM simulation and online parameter adaptation. Experiments report that it outperforms spring-mass baselines.
Problem
Modeling deformable objects with plausible geometry, appearance, physical properties, and dynamics from visual information remains challenging across 3D vision, graphics, and robotics.
Source: §1 · p. 1Method
EMPM reconstructs geometry and appearance from multi-view RGB-D video, fits differentiable MPM material parameters to observed rollouts, and adapts them online from sensory observations.
Source: §1 · p. 2, §3.3 · p. 4, §3.4 · pp. 4–5Results
Takeaways & Limitations
The framework offers a real-to-sim-to-real basis for physics-aware deformable-object modeling and potential downstream robotic control.
Source: §1 · pp. 1–2Reliable point tracking degrades under occlusion and large deformations, and first-frame-conditioned tracking can yield no reliably tracked points after a few seconds.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Modeling deformable objects - especially continuum materials - in a way that is physically plausible, generalizable, and data-efficient remains challenging across 3D vision, graphics, and robotic manipulation. Many existing methods oversimplify the rich dynamics of deformable objects or require large training sets, which often limits generalization. We introduce embodied MPM (EMPM), a deformable object modeling and simulation framework built on a differentiable Material Point Method (MPM) simulator that captures the dynamics of challenging materials. From multi-view RGB-D videos, our approach reconstructs geometry and appearance, then uses an MPM physics engine to simulate object behavior by minimizing the mismatch between predicted and observed visual data. We further optimize MPM parameters online using sensory feedback, enabling adaptive, robust, and physics-aware object representations that open new possibilities for robotic manipulation of complex deformables. Experiments show that EMPM outperforms spring-mass baseline models. Project website: https://embodied-mpm.github.io.
1. Introduction
Modeling deformable objects from visual information remains challenging, while existing physics models can oversimplify their dynamics and learning-based models often need extensive data and generalize poorly. EMPM combines differentiable MPM simulation with geometry and appearance reconstruction, and experiments report better performance than spring-mass baselines.
Motivation
- Existing particle and spring-mass models can oversimplify deformable dynamics, while learning-based models often require large datasets and generalize poorly to novel objects.These limitations constrain their applicability to real-world objects.Source: §1 · pp. 1–2
Approach
- EMPM jointly models and simulates elastic and elastoplastic deformables from multi-view RGB-D video, combining MPM physics with reconstructed geometry and 3D Gaussian Splatting appearance.It optimizes physical parameters against discrepancies between real observations and simulated rollouts, offline and through online sensory feedback.Source: §1 · p. 2
Contributions and results
- EMPM outperforms spring-mass baselines in experiments on a range of deformable objects, and its online parameter adaptation is proposed for potential downstream robotic control.The paper identifies differentiable optimization grounded in real sensory observations and online adaptation as contributions.Source: §1 · p. 2
2. Related Work
Related work spans dynamic-scene reconstruction, physics models for deformables, and differentiable MPM. EMPM follows this reconstruction and simulation research while using Warp for its differentiable MPM engine.
2.1. 4D visual reconstruction
- The reconstruction pipeline lifts point clouds, tracks 3D points, and models appearance with Gaussian Splatting, following established dynamic-scene reconstruction approaches.The cited background describes 4D reconstruction as modeling dynamic 3D geometry over time to capture motion.Source: §2.1 · p. 2
2.2. Physics simulation for deformable objects
- Physics approaches progressed from particle and spring-mass models to MPM, which is suited to continuum materials and large deformations, fracture, and tearing.Spring-mass and learning-based models may fail to capture more complex deformable dynamics and interactive motions.Source: §2.2 · p. 2
2.3. Differentiable MPM simulation
- Differentiable MPM enables physical-parameter estimation from observations; EMPM uses Warp to differentiate through simulator outputs.Prior work also reports reliable estimation capabilities and GPU acceleration for differentiable MPM systems.Source: §2.3 · pp. 2–3
3. Method
EMPM couples action-conditioned MPM dynamics with visual reconstruction and fits material parameters to observed geometry and motion. It supports offline identification, online correction during interaction, and predictive simulation for manipulation planning.
3.1. Overview
- An action-conditioned MPM model predicts the next particle state from the current state and controller actions, while RGB-D inputs are lifted into point clouds and tracked over time.The material parameters include Young’s modulus, Poisson’s ratio, density, and plastic yield stress.Source: §3.1 · p. 3
3.2. MPM and Continuum Mechanics
- MPM transfers particle mass and momentum to a grid, updates grid velocities under stress, gravity, and controller boundary conditions, then transfers velocities back to particles.For elastoplastic materials, the plastic deformation stores the new rest shape while the elastic part computes stress; EMPM uses Fixed Corotated stress and von Mises return mapping.Source: §3.2 · pp. 3–4
3.3. Offline Simulation and Optimization
- Offline identification fits MPM rollouts to RGB-D observations and reconstructed end-effector motion, minimizing geometric and tracked-particle discrepancies with differentiable optimization.Occlusion masks filter invalid particles; Warp supplies gradients, while CMA-ES is an optional zero-order alternative.Source: §3.3 · p. 4
3.4. Online Adaptive Identification
- Online adaptation uses streaming 3D observations and gripper positions to drive synchronized simulation, then corrects parameters using geometry and mask losses at quasi-static states.The quasi-static restriction assumes the object is close to stable equilibrium; online keypoint tracking is omitted because occlusion can make tracks unreliable over time.Source: §3.4 · pp. 4–5
3.5. Applications
- Action-conditioned rollouts let users inspect object responses to teleoperation or hypothetical controller inputs and compare candidate end-effector trajectories for manipulation planning.The simulator interface could support gradient-free sampling or trajectory optimization.Source: §3.5 · p. 5
4. Experiments
Across elastic and elastoplastic objects, EMPM aligns simulations with observations better than the compared baselines, and online correction improves accuracy during manipulation.
4.1. Experiment Setup
- The experiments cover elastic and elastoplastic objects under varied deformations, using three RGB-D cameras and human-hand demonstrations for offline data collection.The objects include rope, cloth, an elastic toy, plasticine, bread dough, and pita bread.Source: §4.1 · p. 5, Figure 3 · p. 5
4.2. Offline Simulation and Optimization
- EMPM models elastic and elastoplastic materials, including permanent shape change and fracture that spring-mass systems cannot capture.The pita bread example illustrates fracture modeling, while PhysTwin fails on fracture and over-bends plasticine during squeezing.Source: §4.2 · p. 6, Figure 4 · p. 6
4.3. Online Optimization
- Online correction improves simulation accuracy for rope and bread dough as the simulated objects align more closely with real observations over time.Optimization occurs every five streaming steps when the object is relatively steady; Table 2 reports 2D mask and 3D distance errors.Source: §4.3 · p. 6, Table 2 · p. 7
4.2. Offline Simulation and Optimization
- EMPM outperforms PhysTwin and PGND across reconstruction and visual metrics, with a particularly notable advantage on elastoplastic objects.The comparison covers distance and tracking, plus IoU, PSNR, SSIM, and LPIPS.Source: §4.2 · p. 6, Table 1 · p. 7
4.3. Online Optimization
- EMPM training is the most efficient among the compared methods, while PGND has the lowest inference time but much longer training time.The timing analysis compares EMPM and baselines using the same training data.Source: §4.3 · pp. 6–7, Table 3 · p. 7
4.4. Applications
- The paper demonstrates a proof-of-concept deformable-object manipulation application with real robotic hardware.The authors identify model-based motion planning and control as potential uses of the dynamics model.Source: §4.4 · p. 7
5. Conclusions & Discussions
EMPM models geometry, appearance, and dynamics for complex deformable materials, including cases that challenge spring–mass baselines. Online optimization improves alignment with observations, but tracking reliability constrains optimization under occlusion and large deformations.
- EMPM models geometry, appearance, and dynamics of complex deformables, including elastic and soft/hard elastoplastic materials challenging for spring–mass baselines.Source: §5 · p. 7
- Online optimization progressively improves simulation-observation alignment, although bread-dough updates are visually subtle under small motion and gripper occlusion.Source: Figure 6 · p. 8
- The paper demonstrates model rollout for potential manipulation planning and control, with the robot tracking simulated changes.Source: Figure 7 · p. 8
- Future work targets improved Gaussian Splatting appearance modeling and integration of model-based control for autonomous deformable manipulation.Source: §5 · p. 8
- Reliable point tracking degrades under occlusion and large deformations; first-frame-conditioned tracking can lose all reliable points after seconds, hindering online optimization.Source: §5 · p. 8