Source-linked AI summary
ReconPhys: Reconstruct Appearance and Physical Attributes from Single Video
Boyuan Wang, Xiaofeng Wang, Yongkang Li, Zheng Zhu, Yifan Chang, Angen Ye, Guosheng Zhao, Chaojun Ni, Guan Huang, Yijie Ren, Yueqi Duan, Xingang Wang
TL;DR
Recovering non-rigid objects with physical plausibility is difficult because existing methods often rely on expensive per-scene optimization or supervision. ReconPhys jointly predicts physical attributes and reconstructs 3DGS from a single monocular video using a dual-branch, self-supervised framework. It reports superior future prediction and geometric accuracy while supporting rapid simulation-ready asset generation.
Problem
Existing approaches for physically plausible non-rigid reconstruction often require expensive per-scene optimization, manual annotation, or ground-truth physics labels, limiting practicality and generalizability.
Method
ReconPhys jointly reconstructs 3DGS geometry and appearance and estimates physical attributes from a single monocular video using a dual-branch, self-supervised pipeline.
Results
21.64 PSNR versus 13.27 for Spring-Gaus is reported in future prediction, while Chamfer Distance decreases from 0.593 to 0.001 versus 4DGS.
Takeaways & Limitations
ReconPhys enables fast generation of simulation-ready 3DGS assets for robotics and simulation without per-scene optimization.
Takeaways & Limitations
Prior differentiable-physics and dynamic-reconstruction methods remain constrained by computational cost, material assumptions, multi-view inputs, or per-scene optimization.
Abstract
from arXiv · showhide
Reconstructing non-rigid objects with physical plausibility remains a significant challenge. Existing approaches leverage differentiable rendering for per-scene optimization, recovering geometry and dynamics but requiring expensive tuning or manual annotation, which limits practicality and generalizability. To address this, we propose ReconPhys, the first feedforward framework that jointly learns physical attribute estimation and 3D Gaussian Splatting reconstruction from a single monocular video. Our method employs a dual-branch architecture trained via a self-supervised strategy, eliminating the need for ground-truth physics labels. Given a video sequence, ReconPhys simultaneously infers geometry, appearance, and physical attributes. Experiments on a large-scale synthetic dataset demonstrate superior performance: our method achieves 21.64 PSNR in future prediction compared to 13.27 by state-of-the-art optimization baselines, while reducing Chamfer Distance from 0.349 to 0.004. Crucially, ReconPhys enables fast inference (<1 second) versus hours required by existing methods, facilitating rapid generation of simulation-ready assets for robotics and graphics.
1. Introduction
ReconPhys addresses the challenge of recovering non-rigid objects’ geometry, appearance, and physical behavior from a single monocular video. It replaces costly scene-specific optimization with a feedforward, self-supervised framework and reports stronger reconstruction and future prediction results.
- Motivation: Non-rigid reconstruction is an ill-posed inverse problem because objects deform under external forces or internal dynamics.Recovering geometry, appearance, and physical behavior is important for realistic interaction and simulation.
- Results: 21.64 PSNR is achieved by ReconPhys versus 13.27 by Spring-Gaus in future prediction.The method also reports state-of-the-art performance in dynamic reconstruction and future-state prediction.
- ReconPhys: ReconPhys jointly reconstructs non-rigid geometry, appearance, and physical attributes from a single video without per-scene optimization.The framework produces simulation-ready 3DGS assets within one second.
- ReconPhys: ReconPhys combines a dual-branch architecture with self-supervised physics training and an automated dataset pipeline for deformable objects.The dataset contains diverse physical attributes and dynamics for training.
- Results: 0.001 Chamfer Distance is achieved by ReconPhys versus 0.593 for 4DGS in 3D geometric accuracy.The comparison is reported for dynamic reconstruction.
2. Related Works
Prior work enables dynamic visual reconstruction and differentiable physics, but often remains computationally expensive or requires supervision, specialized data, multi-view capture, or scene-specific optimization. ReconPhys targets feedforward monocular physical attribute inference to improve scalability and deployment.
- Dynamic 3DGS Reconstruction: NeRF-based dynamic methods model temporal changes but can be computationally expensive and slow at render time.These costs limit interactive usage.
- Dynamic 3DGS Reconstruction: 3DGS provides an explicit representation with real-time differentiable rendering for dynamic scene reconstruction.Existing dynamic 3DGS methods use deformation-based or explicit modeling strategies.
- Physics-Aware Reconstruction: Differentiable physics methods connect image-space losses to physical parameters, but PAC-NeRF assumes known material families and coarse global parameters.These assumptions limit adaptability to heterogeneous real objects.
- Physics-Aware Reconstruction: Learning-based physical modeling often requires explicit state supervision, expensive per-scene optimization, or specialized simulation data.These requirements constrain practical deployment and generalization.
- Physics-Aware Reconstruction: Spring-Gaus requires multi-view capture and per-scene optimization, making it impractical for general monocular deployment and large-scale usage.ReconPhys instead predicts physical parameters directly from video using feedforward monocular inference.
3. Method
ReconPhys combines a canonical 3D Gaussian Splatting representation with a differentiable spring–mass system whose physical attributes are inferred from monocular videos. Differentiable simulation, Gaussian binding, rendering, and self-supervised reconstruction jointly train the physical predictor.
- Model representation: The model represents object geometry and appearance with canonical 3DGS while modeling deformation through a bound spring–mass system.Gaussian centers are determined by simulated mass-point states, while orientation, color, scale, and opacity characterize each kernel.
- Physical dynamics: Anchor points are sampled throughout the object volume and connected to K nearest neighbors to form a sparse spring–mass system.Volume sampling distributes anchors through the object rather than only on its surface, and each spring has a rest length, stiffness, and damping coefficient.
- 3DGS binding: Gaussian centers follow simulated anchor motion through inverse-distance-weighted interpolation, preserving local deformation patterns with sparse anchors.The interpolation uses nearest anchors and a distance-falloff parameter while keeping the simulation efficient.
- Dual-branch prediction: ReconPhys jointly predicts a canonical Gaussian representation and compact physical attributes from a monocular video.The physical predictor estimates mass, stiffness, damping, and friction, while the 3DGS predictor supplies the canonical appearance and geometry.
- Self-supervised training: The differentiable simulator recursively predicts anchor states, propagates them to Gaussian centers, and renders deformed frames for reconstruction supervision.Gradients flow from rendered-frame errors through the renderer, binding, and unrolled dynamics to the predicted physical attributes.
- Self-supervised training: Self Forcing advances each rollout from the model’s previous predicted state rather than recovered or proxy ground-truth states, reducing training–inference mismatch.Truncated backpropagation detaches each rollout step’s input state to limit unstable gradients while rendered frames continue supervising physical-parameter prediction.
4. Experiment
ReconPhys is evaluated on synthetic and real-world non-rigid assets for cross-object generalization, future prediction, physical disentanglement, and practical manipulation. It maintains reconstruction fidelity while improving physical prediction and reducing inference time.
- Synthetic Data Pipeline: The pipeline samples continuous mass, stiffness, damping, and friction values and simulates 30-frame free-fall trajectories for 3DGS assets.The sampled physical configuration is p = (m, k, d, f).
- Synthetic Data Pipeline: The dataset uses 496 synthesized objects, with 450 for training and 46 for testing cross-object generalization.Each training object has 10 physical samples, while each test object has 2 reserved for evaluation.
- Experimental Setup: ReconPhys uses single-view 20-frame sequences, whereas baselines use per-scene optimization with four-view inputs.Evaluation includes PSNR, SSIM, LPIPS, Chamfer Distance, and Earth Mover’s Distance.
- Cross-Object Generalization: +7.6 dB PSNR over Spring-Gaus is achieved in prediction tasks, while processing time falls from over 1 hour to under 1 second.The method also maintains competitive reconstruction quality and enables predictive simulation on unseen geometries.
- Physical Disentanglement: ReconPhys preserves reconstruction fidelity while distinguishing physical states assigned to identical geometry.Different predicted physical attributes produce divergent future trajectories without explicit physics supervision.
- Real-World Applications: Real-world demonstrations reconstruct deformable assets and support manipulation scenarios including stretching, squeezing, squashing, and stretching.The pipeline processes segmented videos into 3DGS representations and spring-mass systems for simulation.
5. Conclusion
ReconPhys jointly infers 3D Gaussian Splatting representations and physical attributes from monocular free-fall videos using self-supervised learning. Its physics-aware reconstruction improves future prediction and physical disentanglement without ground-truth physics labels.
- ReconPhys jointly infers 3D Gaussian Splatting representations and physical attributes from monocular free-fall videos.
- Its dual-branch architecture uses differentiable links between visual reconstruction loss and physical simulation, eliminating ground-truth physics labels.
- 8.37dB higher PSNR in future prediction than appearance-only baselines is reported, alongside trajectory divergence for identical geometries with different materials.