PaperlayerView exact revision on arXiv

Source-linked AI summary

PhysReal: Learning Real-World Deformable Object Physics via Hybrid Constitutive Modeling

Yinan Deng, Jianqiao Song, Yisi Zhang, Yuhan Wang, Jiahui Wang, Yufeng Yue

arXiv:2609.07532v1cs.RO

TL;DR

Learning physically grounded dynamics for deformable objects is challenging because their material responses can vary spatially. PhysReal learns these dynamics with hybrid constitutive modeling and progressive fitting, and reports improved reconstruction and future prediction over existing approaches.

  • Problem

    Learning the underlying dynamics of real deformable objects is difficult because their deformation mechanisms and material properties are complex and diverse.

    Source: §I · p. 1

  • Method

    PhysReal combines interpretable analytical constitutive models with neural residuals and spatially distributed patches in a differentiable MPM framework.

    Source: §I · p. 2, C. Differentiable Physics Simulation · p. 4

  • Results

    PhysReal achieves more accurate reconstruction and future prediction than existing approaches, with best CD, IoU, and PSNR on the PhysTwin dataset in reconstruction and future prediction.

    Source: B. Comparison with Baselines · p. 6, §V · p. 8

  • Takeaways & Limitations

    The recovered physical models support downstream applications, including potential robotic applications.

    Source: §V · p. 8

Abstract

from arXiv · show

Learning physically plausible dynamics from visual observations is essential for interactive world models and embodied agents. However, modeling real-world deformable objects remains challenging because their dynamics often arise from complex, spatially heterogeneous material responses. To address this challenge, we propose PhysReal, a video-driven framework for learning and simulating the underlying physics of real deformable objects. PhysReal integrates a spatially varying hybrid expert-neural constitutive model with a differentiable MPM simulator and 3DGS renderer. Analytical expert models provide interpretable physical priors, while neural constitutive residuals capture material responses beyond predefined formulations. Spatially distributed patches parameterize the constitutive field, enabling a continuous representation of local material variations. To organize the identification of this model from sparse visual observations, we adopt a progressive curriculum that sequentially optimizes global material properties, spatially varying local parameters, and neural constitutive residuals, together with complementary motion and mask supervision. Extensive experiments on diverse deformable-object interactions demonstrate that PhysReal achieves superior performance in dynamic reconstruction and future-state prediction, while showing strong potential for downstream robotic applications.

I. INTRODUCTION

PhysReal addresses the challenge of learning heterogeneous deformable-object dynamics by combining interpretable constitutive priors with neural residuals and progressively fitting them from visual observations.

I. INTRODUCTION

  • Real deformable objects exhibit complex, diverse material responses that make their underlying physical dynamics difficult to recover.Source: §I · p. 1

I. Stably modeling complex spatially varying constitutive

  • PhysReal combines analytical constitutive priors with neural residuals and patch-level latent codes to represent material responses beyond predefined formulations and spatially varying properties.Source: §I · p. 2
  • A progressive curriculum introduces global properties, local variations, and residual behaviors, with complementary motion and mask supervision for learning from sparse visual observations.Source: §I · p. 2
  • The paper frames PhysReal as a video-driven framework for recovering real-world deformable dynamics with complex, heterogeneous material behavior.Source: §I · p. 2
  • PhysReal is reported to outperform existing approaches in reconstruction and future prediction, with strong potential for downstream robotic applications.Source: §I · p. 2

A. Learning-based Deformable Dynamics Modeling

Prior work models deformable dynamics either by learning state transitions or by using explicit mechanical formulations, each with limitations for representing complex material behavior. PhysReal combines spatially varying constitutive fields with progressive optimization and complementary visual supervision.

A. Learning-based Deformable Dynamics Modeling

  • Learning-based methods include particle, graph, mesh, and object-conditioned approaches, but often learn implicit transitions that are less interpretable and may generalize poorly to unseen interactions or configurations.Source: A. Learning-based Deformable Dynamics… · p. 2

B. Physics-based Deformable Simulation

  • Physics-based methods offer explicit mechanics, but spring-mass models have limited intrinsic-material correspondence, while continuum methods often use global parameters and predefined constitutive formulations.Source: B. Physics-based Deformable Simulation · p. 2

A. Framework Overview

B. Interactive Video Parsing

Interactive video parsing turns single-view observations into temporally consistent object masks and an aligned 3D object representation. These outputs provide geometric inputs for later optimization.

B. Interactive Video Parsing

  • Tracked points connect framewise segmentation candidates across time, and the candidate mask with the most valid tracked points is selected for each frame.The resulting mask sequence supplies geometric supervision for later optimization.Source: B. Interactive Video Parsing · p. 3
  • A generative model reconstructs an initial 3D Gaussian representation and mesh, whose transform is optimized against observed RGB, depth, and mask images.Differential Evolution provides a coarse estimate before refinement with differentiable 3D Gaussian rendering.Source: B. Interactive Video Parsing · pp. 3–4
  • The generated object's transform into camera coordinates is parameterized by rotation, translation, and scale.The aligned mesh initializes volumetric particles, while the Gaussian representation supplies the appearance model for differentiable rendering.Source: B. Interactive Video Parsing · pp. 3–4

C. Differentiable Physics Simulation

PhysReal simulates deformable objects with a differentiable MPM pipeline, combining spatially varying material representations, analytical constitutive experts, and neural residuals. Interaction trajectories drive the simulation, whose particle states are linked to rendered observations.

C. Differentiable Physics Simulation

  • PhysReal samples material particles inside the reconstructed mesh and initializes them with zero velocity and an identity deformation gradient.The simulation coordinate system uses the estimated supporting-surface normal as its vertical axis.Source: C. Differentiable Physics Simulation · p. 4
  • Each Gaussian primitive is bound to nearby MPM particles, whose weighted motion deforms it before rendering into image space.The rendered outputs include RGB, depth, and alpha masks.Source: C. Differentiable Physics Simulation · p. 4
  • Representative patches carry local constitutive parameters and latent codes, which are interpolated to particles for smooth material-response variation with fewer learnable parameters.The patch representation avoids assigning independent constitutive parameters to every MPM particle.Source: C. Differentiable Physics Simulation · p. 4
  • Nearby material particles receive compliant forces that transfer the observed hand or robot trajectory into the object rather than directly prescribing particle motion.Particle binding weights and interaction stiffness determine the force.Source: C. Differentiable Physics Simulation · p. 4
  • PhysReal combines expert formulations and neural residual corrections for both elastic and plastic responses, representing behaviors beyond predefined constitutive models.The neural elastic network is conditioned on local latent codes, and plastic corrections are added to the expert response.Source: C. Differentiable Physics Simulation · pp. 4–5
  • A global friction coefficient models object-ground interactions, while bounding neural correction magnitudes relative to expert responses preserves numerical stability.Source: C. Differentiable Physics Simulation · p. 5
  • Differentiable MPM evolves particle states between observations, incorporating interaction, constitutive, gravity, and contact effects through multiple simulation substeps.The simulator uses particle-to-grid transfer, grid evolution, and grid-to-particle transfer.Source: C. Differentiable Physics Simulation · p. 5

D. Training

PhysReal addresses the coupled estimation of material properties and neural residuals with a progressive curriculum. It combines staged constitutive optimization with motion and mask supervision from interactive video.

D. Training

  • A three-stage curriculum addresses the coupled optimization of constitutive parameters and neural residuals.The stages introduce global material properties, local variations, and residual behaviors in sequence.Source: D. Training · p. 5
  • Stage 1 fits a homogeneous expert model using global Young’s modulus, Poisson’s ratio, and friction coefficient.These parameters capture the object’s dominant deformation characteristics.Source: D. Training · p. 5
  • Stage 2 propagates global parameters to patches and optimizes local material properties to capture heterogeneous distributions.Each local region adapts its constitutive parameters.Source: D. Training · p. 5
  • Stage 3 fixes expert parameters and activates zero-initialized neural residuals to learn constitutive discrepancies beyond analytical models.The neural corrections initially vanish, so optimization begins from the expert model.Source: D. Training · p. 5
  • Tracked points constrain simulated particle trajectories in 3D and 2D, while rendered alpha masks provide dense geometric supervision.The 3D constraint supervises particle evolution, and the 2D constraint improves robustness to depth errors; motion and mask losses are jointly weighted.Source: D. Training · p. 5

A. Experimental Setup

The evaluation uses public and self-collected interaction sequences, comparing methods under a shared single-view setting. Observed frames support model identification and reconstruction, while remaining frames are held out for prediction.

A. Experimental Setup

  • The evaluation covers 22 PhysTwin sequences and six self-collected sequences involving two stuffed toys and one cloth.PhysTwin includes deformable linear, planar, and volumetric objects.Source: A. Experimental Setup · pp. 5–6
  • Observed intervals are used for model identification and reconstruction, with remaining frames reserved for future-state prediction.Figure 3 illustrates this training/testing split on representative sequences.Source: A. Experimental Setup · p. 6, Figure 3 · p. 6
  • PhysReal is compared with GS-Dynamics, Spring-Gaus, PhysFlow, and PhysTwin under the same single-view observation setting.The baselines are identified as representative methods.Source: A. Experimental Setup · p. 6
  • Four metrics assess 3D geometry, deformation trajectories, rendered masks, and appearance: CD, Track, IoU, and PSNR.CD and Track are measured in meters; IoU is a percentage and PSNR is in decibels.Source: A. Experimental Setup · p. 6

B. Comparison with Baselines

PhysReal compares favorably with four baselines across reconstruction and future prediction. Its advantage is strongest on the self-collected dataset, while performance varies by object geometry.

B. Comparison with Baselines

  • On PhysTwin, PhysReal achieves the best CD, IoU, and PSNR in reconstruction and future prediction, with Track Error comparable to PhysTwin; on the self-collected dataset, it leads across all metrics.Tables I–II report the quantitative comparisons.Source: B. Comparison with Baselines · p. 6, Table I · p. 7, Table II · p. 7
  • Spring-Gaus and PhysTwin lack explicit correspondence to intrinsic material properties, while PhysFlow’s global parameters make heterogeneous responses difficult to represent.PhysReal instead models spatially varying hybrid constitutive fields, supporting more faithful local deformation and future-dynamics prediction.Source: B. Comparison with Baselines · p. 7
  • Across geometries, PhysReal substantially improves prediction on linear and volumetric objects; for planar objects it improves IoU and PSNR but has slightly higher Track Error than PhysTwin.The authors relate the planar gap likely to thin structures’ membrane and bending dynamics, which volumetric MPM challenges to capture; visual fidelity remains higher across all three categories.Source: B. Comparison with Baselines · p. 7

C. Ablation Studies

Ablations indicate incremental benefits from increasing constitutive-model capacity and combining motion with mask supervision. The effectiveness of each supervision signal depends on the object’s deformation.

C. Ablation Studies

  • Stage 1 captures dominant deformation but misses local material differences; adding patch-level parameters in Stage 2 improves future prediction, and Stage 3 further improves geometry and appearance with neural residuals.Table IV and Figure 4 analyze the progressive stages.Source: C. Ablation Studies · p. 7, Table IV · p. 7, Figure 4 · p. 7
  • Later optimization stages capture increasingly detailed local material variations and deformation responses.Figure 4 qualitatively compares the progressive constitutive stages.Source: Figure 4 · p. 7, C. Ablation Studies · p. 7
  • Motion supervision is especially important for cloth’s internal folding correspondences, whereas mask supervision is effective for rope when its silhouette characterizes deformation.Combining both signals generally improves robustness by constraining temporal correspondence and global geometry.Source: C. Ablation Studies · p. 7, Table V · p. 7

D. Generalization to Unseen Interactions

PhysReal remains stable under interactions that differ substantially from the identification sequence, preserving object-specific characteristics across interactions.

D. Generalization to Unseen Interactions

  • PhysReal remains stable under substantially different interactions and preserves object-specific characteristics across them.For different initial configurations, patch locations are aligned with observed geometry while learned material representations are preserved.Source: D. Generalization to Unseen Interactions · p. 7

E. Downstream Applications

PhysReal supports interactive simulation, embodied data generation, digital-twin monitoring, and model-based manipulation. The conclusion reports more accurate reconstruction and future prediction than existing approaches and identifies learning from multiple interaction videos as future work.

E. Downstream Applications

  • Users can manipulate the recovered object under novel trajectories and observe predicted deformation without retraining or additional optimization.Source: E. Downstream Applications · pp. 7–8
  • A model identified from one interaction can be rolled out under different manipulation trajectories without further optimization.The figure illustrates the observed training action and novel test actions.Source: Figure 5 · p. 8
  • Privileged particle states identify grasping and interaction targets across object configurations, supporting paired visual-observation and robot-action generation.Realistic 3DGS rendering supports scalable generation of these pairs.Source: E. Downstream Applications · p. 8
  • The coupled 3DGS-MPM representation synchronizes visual and physical states for real-time monitoring and simulation of workspace interactions.PhysReal jointly reconstructs the environment, robot, and deformable objects to create a physics-grounded digital twin.Source: E. Downstream Applications · p. 8
  • For model-based manipulation, MPC evaluates candidate actions through simulation and optimizes the robot trajectory toward a target object configuration.Source: E. Downstream Applications · p. 8

V. CONCLUSION

  • Experiments show more accurate reconstruction and future prediction than existing approaches, while recovered physical models support downstream applications.Future work will explore jointly learning from multiple interaction videos, whose diverse manipulation trajectories can provide complementary observations of physical properties.Source: §V · p. 8

Related papers

Loading 2609.07532v1…