Source-linked AI summary
PhysTwin: Physics-Informed Reconstruction and Simulation of Deformable Objects from Videos
Hanxiao Jiang, Hao-Yu Hsu, Kaifeng Zhang, Hsin-Ni Yu, Shenlong Wang, Yunzhu Li
TL;DR
PhysTwin addresses the challenge of constructing physically realistic digital twins for deformable objects from sparse videos. It combines spring-mass simulation, generative shape priors, Gaussian rendering, and staged inverse optimization, with results supporting reconstruction, prediction, and unseen-interaction simulation. The paper also reports real-time simulation and robotic motion-planning applications.
Problem
Constructing digital twins that capture deformable-object geometry, appearance, and physical properties from sparse observations remains challenging.
Method
PhysTwin combines a spring-mass representation, generative shape priors, Gaussian splats, and multi-stage inverse optimization driven by video observations and tracking cues.
Results
PhysTwin consistently outperforms augmented baselines across reconstruction and resimulation, future prediction, and unseen-interaction generalization tasks.
Takeaways & Limitations
PhysTwin supports real-time interactive simulation and model-based robotic motion planning for deformable objects.
Takeaways & Limitations
In practice, the first-frame object state is used as the canonical state for the spring-mass system.
Abstract
from arXiv · showhide
Creating a physical digital twin of a real-world object has immense potential in robotics, content creation, and XR. In this paper, we present PhysTwin, a novel framework that uses sparse videos of dynamic objects under interaction to produce a photo- and physically realistic, real-time interactive virtual replica. Our approach centers on two key components: (1) a physics-informed representation that combines spring-mass models for realistic physical simulation, generative shape models for geometry, and Gaussian splats for rendering; and (2) a novel multi-stage, optimization-based inverse modeling framework that reconstructs complete geometry, infers dense physical properties, and replicates realistic appearance from videos. Our method integrates an inverse physics framework with visual perception cues, enabling high-fidelity reconstruction even from partial, occluded, and limited viewpoints. PhysTwin supports modeling various deformable objects, including ropes, stuffed animals, cloth, and delivery packages. Experiments show that PhysTwin outperforms competing methods in reconstruction, rendering, future prediction, and simulation under novel interactions. We further demonstrate its applications in interactive real-time simulation and model-based robotic motion planning.
1. Introduction
PhysTwin addresses the challenge of constructing physically realistic digital twins for deformable objects from sparse observations. It combines physics-based modeling with visual and generative priors to reconstruct and simulate objects under interaction.
- Dynamic 3D methods capture observed motion, appearance, and geometry but omit underlying physics needed for unseen-interaction simulation.
- PhysTwin targets sparse-viewpoint RGB-D videos to capture deformable-object geometry, non-rigid dynamics, and appearance.
- A spring-mass representation enables efficient simulation across ropes, stuffed animals, cloth, and delivery packages.
- The inverse modeling framework reconstructs interactive PhysTwins and evaluates reconstruction, resimulation, future prediction, and unseen-interaction generalization.
- Quantitative and qualitative results show accurate observation alignment, precise future prediction, realistic unseen-interaction simulation, and real-time dynamics and rendering.
2. Related Works
Prior work reconstructs dynamic appearance and geometry or learns deformable-object dynamics, but existing approaches often lack physically grounded generalization. PhysTwin combines these directions with an explicit physics-informed representation.
- Dynamic Scene Reconstruction: Dynamic NeRF and Gaussian-splatting methods achieve high-fidelity synthesis but primarily reconstruct appearance and geometry without real-world dynamics.
- Physics-Based Simulation of Deformable Objects: Physics-based reconstruction methods perform system identification, but some rely on pre-scanned objects, clean point clouds, or manually specified physical parameters.
- Learning-Based Simulation of Deformable Objects: Neural simulators learn dynamics for varied deformable materials, addressing the complexity of analytically modeling high-dimensional and variable physical systems.
- PhysTwin Framework: PhysTwin represents geometry, topology, physical parameters, and Gaussian kernels, optimizing rendering alongside simulated-observed geometry and motion discrepancies.
3. Preliminary: Spring-Mass Model
The spring-mass model represents a deformable object as mass nodes connected by topology-defined springs. Newtonian state updates combine elastic, damping, external, collision, and interaction effects for efficient simulation.
- A deformable object is represented as a graph of mass nodes and springs, with node positions and velocities evolving over time.
- Spring connections follow a predefined topology that defines the object’s elastic structure.
- The force on each node combines effects from adjacent spring-connected nodes.
- Spring stiffness, rest length, dashpot damping, and external forces govern elasticity, energy dissipation, gravity, collisions, and user interactions.
- The model updates state through Xt+1 = fα,G0(Xt, at) using explicit Euler integration, with α covering physical and control-interaction parameters.
4. Method
PhysTwin formulates deformable-object reconstruction as physics-constrained optimization from RGB-D videos, separating physics and geometry optimization from appearance optimization. Generative shape and tracking priors address sparse observations, while Gaussian rendering supports real-time interaction.
- Optimization Strategy: Optimization proceeds in two stages: geometry and physical parameters first, followed by appearance-related parameters.
- Problem Formulation: The framework constructs a PhysTwin from three RGB-D videos by optimizing geometry, appearance, and physical parameters to match observations.
- Problem Formulation: Its objective combines geometry, motion, and rendering costs that compare predicted observations with 3D geometry, tracking, and 2D color evidence.
- Physics and Geometry Optimization: Generative shape initialization recovers complete geometry from partial observations, then sparse-to-dense optimization refines topology and physical parameters.
- Physics and Geometry Optimization: Tracking priors are lifted from 2D vision-foundation-model predictions into 3D pseudo-ground-truth data for the motion cost.
- Appearance Optimization: Appearance optimization fits static Gaussian kernels with rendering loss, uses isotropic shapes to prevent spiky deformation artifacts, and updates Gaussians with LBS.
- Applications: PhysTwin supports real-time photorealistic simulation and interactive exploration, including control-point-driven motions and model-based robotic motion planning.
5. Experiments
Experiments evaluate PhysTwin on reconstruction and resimulation, future prediction, unseen-interaction generalization, and downstream applications using deformable-object RGB-D videos. Across the evaluated tasks, PhysTwin consistently outperforms the augmented baselines and supports interactive simulation and model-based robotic planning.
- Experiment Settings: The evaluation covers reconstruction and resimulation, future prediction, and generalization to unseen interactions, alongside downstream-utility demonstrations.Unseen-interaction tests use different motions, hand configurations, and interaction types on the same object.
- Experiment Settings: The dataset contains RGB-D videos of human interactions with ropes, stuffed animals, cloth, and delivery packages, recorded across 22 scenarios.Videos capture 1–10-second interactions such as lifting, stretching, pushing, and squeezing using three RealSense-D455 cameras.
- Experiment Settings: PhysTwin is compared with augmented Spring-Gaus and GS-Dynamics baselines across the three task settings using 3D and 2D metrics.Evaluation includes Chamfer Distance, tracking error, PSNR, SSIM, LPIPS, and IoU.
- Results: PhysTwin consistently outperforms the baselines across tasks, improving reconstruction and resimulation alignment, rendering quality, and tracking under sparse observations.The physics-based representation also yields tracking better than the original CoTracker3 predictions after global optimization.
- Results: PhysTwin achieves accurate future-frame prediction and robust generalization from observed interactions such as single lifts to unseen actions such as double stretches.GS-Dynamics tends to remain static or struggles with environmental changes and diverse interactions, while Spring-Gaus frequently crashes in qualitative comparisons.
- Application: Efficient forward simulation enables real-time keyboard or robot-teleoperation interaction and model-based planning for tasks such as lifting a rope into a target configuration.The Spring-Mass simulator is implemented using Warp and supports future-state prediction during robotic-arm manipulation.
6. Conclusion
PhysTwin constructs physical digital twins from sparse videos for deformable-object reconstruction and resimulation. It supports future prediction, unseen-action simulation, and downstream robotic manipulation.
- PhysTwin constructs physical digital twins from sparse videos, enabling reconstruction and resimulation of deformable objects.
- The framework predicts future states and simulates object interactions that generalize to unseen actions.
- PhysTwin outperforms prior work across object types, control configurations, and task settings.
- PhysTwin enables downstream tasks requiring high-speed simulation and accurate future prediction, including robotic manipulation.
Supplement Index
The supplement adds framework details, qualitative results, and further method analysis, with videos available online for showcased instances, interactions, and tasks.
- The supplement provides additional details about the PhysTwin framework.
- It includes qualitative results across different tasks and further analysis of the methods.
- Videos showcase results across various instances, interactions, and tasks on the project website.
A. Additional Details for the Shape Prior
The shape-prior supplement addresses the challenge of registering a generated full mesh with a real observation by using 2D matching for geometric alignment.
- Shape Prior: TRELLIS generates a full mesh from a single RGB observation, but potential non-rigid registration remains challenging.
- Shape Prior: The registration module uses 2D matching to handle scale estimation, rigid registration, and non-rigid deformation.
- Shape Prior: A coarse-to-fine rotation strategy renders uniformly distributed virtual-camera views and selects the view with the most SuperGlue correspondences.
- Shape Prior: After rotation estimation, matched-point distances are optimized to resolve scale and translation ambiguities.
B. Additional Details for 3D Gaussian Update
The Gaussian update transfers predicted deformable-object motion to Gaussian kernels by estimating node transformations and interpolating neighboring-node transformations with Linear Blend Skinning.
- 3D Gaussian Update: Each mass receives a 6-DoF transformation from the previous state and predicted next state.
- 3D Gaussian Update: Node translations are obtained from predicted node translations, while local rotations are estimated from neighboring-vertex motions.
- 3D Gaussian Update: Gaussian kernels are transformed using Linear Blend Skinning by locally interpolating neighboring-node transformations.
- 3D Gaussian Update: The interpolation uses K-nearest vertices around each Gaussian center and weights associated with corresponding vertices.
- 3D Gaussian Update: Weights are derived inversely proportional to 3D distance so spatially closer pairs receive larger weights.
- 3D Gaussian Update: Updated Gaussian parameters enable rendering at timestep t + 1 with transformed 3D Gaussians.
C. Additional Experimental Details
The additional experiments evaluate PhysTwin on reconstruction and resimulation, future prediction, and generalization to unseen interactions, with comparisons against Spring-Gaus and GS-Dynamics. Results include qualitative superiority, robust viewpoint handling, improved tracking, and stronger data efficiency, while ablations support the sparse-to-dense optimization strategy.
- Baselines: Spring-Gaus is limited to relatively simple gravity-driven deformations, while GS-Dynamics is compared across all three tasks with Gaussian blending for rendering.Spring-Gaus is evaluated only on reconstruction and resimulation and future prediction because its Gaussians do not generalize to different initial conditions.
- Tasks: PhysTwin is evaluated on reconstruction and resimulation, future prediction, and generalization to unseen interactions.The generalization evaluation transfers a twin trained on one interaction to a different interaction on the same object.
- Qualitative Results: Qualitative results report superior performance over prior work across reconstruction and resimulation, future prediction, and generalization to unseen interactions.Additional figures cover different object instances and interactions across all three tasks.
- Different Viewpoints: Visualizations demonstrate robustness to rendering from different viewpoints.Figure 9 presents rendering results from different viewpoints.
- Ablation Study on Hierarchical Optimization: The complete sparse-to-dense optimization pipeline achieves the best performance across the ablation variants using only zero-order or only first-order optimization.The ablation compares these variants on reconstruction and resimulation and future prediction; the table attributes the strongest physical-parameter accuracy to the complete strategy.
- Tracking Results: PhysTwin produces better and smoother tracking than pseudo-GT tracking during reconstruction and resimulation and future prediction.The comparison uses noisy GT tracking during optimization and visualizes both PhysTwin and CoTracker3 pseudo-GT results.
- Data Efficiency Experiment: With 30 training data points versus PhysTwin's 1, GS-Dynamics shows no performance boost and still struggles to capture precise dynamics as effectively.The experiment characterizes GS-Dynamics as data-hungry and PhysTwin as significantly more data-efficient for learning a useful dynamics model.
D. Future Work
Future work targets broader observation settings and richer interaction data for deformable-object digital twins. The authors also identify robotics applications enabled by efficient simulation and real-time rendering.
- Sparser Observations: Although the current setup uses three RGBD views, appropriate priors could extend PhysTwin to a single RGB video for more scalable in-the-wild use.The paper presents single-video operation as a promising alternative rather than a demonstrated capability.
- Broader Interactions: Optimizing physical parameters from multiple action modalities could reveal richer intrinsic physical characteristics and improve robustness.The current framework optimizes physical parameters based on a single interaction type.
- Robotics Applications: The structured digital twin could simplify real-to-sim transfer, reduce reliance on domain randomization for reinforcement learning, and support model-based robotic planning.High-speed simulation and real-time rendering are identified as foundations for more effective downstream robotics applications.