Source-linked AI summary
PAC-NeRF: Physics Augmented Continuum Neural Radiance Fields for Geometry-Agnostic System Identification
Xuan Li, Yi-Ling Qiao, Peter Yichen Chen, Krishna Murthy Jatavallabhula, Ming Lin, Chenfanfu Jiang, Chuang Gan
TL;DR
Existing video-based system identification assumes known object geometry, limiting use in scenes with complex or unknown shapes. PAC-NeRF jointly estimates geometry and physical properties by combining a conservation-law-constrained NeRF with hybrid Eulerian-Lagrangian dynamics. It outperforms prior approaches by up to two orders of magnitude on a broad range of material-estimation tasks, while requiring synchronized calibrated cameras and continuum-mechanics-compatible scenes.
Problem
Video-based system identification commonly assumes known object geometry, but PAC-NeRF targets geometry- and topology-agnostic estimation from multi-view videos.
Method
PAC-NeRF combines a conservation-law-constrained neural radiance field with hybrid Eulerian-Lagrangian representation and differentiable MPM simulation.
Results
PAC-NeRF outperforms state-of-the-art approaches by up to two orders of magnitude across geometry and physical-parameter estimation tasks for diverse materials.
Takeaways & Limitations
The framework unifies physical simulation and rendering to estimate geometry and physical properties from image-space signals across highly dynamic continuum materials.
Takeaways & Limitations
PAC-NeRF assumes synchronized, accurately calibrated cameras, suitable foreground masks, and phenomena governed by continuum mechanics.
Abstract
from arXiv · showhide
Existing approaches to system identification (estimating the physical parameters of an object) from videos assume known object geometries. This precludes their applicability in a vast majority of scenes where object geometries are complex or unknown. In this work, we aim to identify parameters characterizing a physical system from a set of multi-view videos without any assumption on object geometry or topology. To this end, we propose "Physics Augmented Continuum Neural Radiance Fields" (PAC-NeRF), to estimate both the unknown geometry and physical parameters of highly dynamic objects from multi-view videos. We design PAC-NeRF to only ever produce physically plausible states by enforcing the neural radiance field to follow the conservation laws of continuum mechanics. For this, we design a hybrid Eulerian-Lagrangian representation of the neural radiance field, i.e., we use the Eulerian grid representation for NeRF density and color fields, while advecting the neural radiance fields via Lagrangian particles. This hybrid Eulerian-Lagrangian representation seamlessly blends efficient neural rendering with the material point method (MPM) for robust differentiable physics simulation. We validate the effectiveness of our proposed framework on geometry and physical parameter estimation over a vast range of materials, including elastic bodies, plasticine, sand, Newtonian and non-Newtonian fluids, and demonstrate significant performance gain on most tasks.
1 INTRODUCTION
PAC-NeRF addresses system identification from multi-view videos without assuming known object geometry. It combines neural radiance fields with differentiable continuum physics to estimate geometry and physical properties across diverse materials.
- PAC-NeRF asks whether geometric structure and physical properties can be recovered from multi-view video without known geometry or rendering configurations.
- Dynamic neural representations otherwise learn object dynamics from scratch and remain uninterpretable, whereas differentiable physics offers a more prescriptive and data-efficient dynamics model.
- PAC-NeRF augments a NeRF with differentiable continuum dynamics to estimate object geometry and physical properties in one unified framework.
- The method couples density and color fields with a velocity field satisfying the continuum conservation law, then advects geometry and appearance across video frames.
- Up to two orders of magnitude improvement is reported across estimation tasks spanning elastic materials, plasticine, sand, and Newtonian/non-Newtonian fluids.
2 RELATED WORK
Prior work represents scenes with neural fields and addresses soft-body identification with differentiable physics, but typically relies on known geometry. PAC-NeRF combines voxel NeRFs with MPM to handle broader material behavior and topology changes.
- Voxel NeRFs encode dynamic scene appearance and geometry efficiently in an Eulerian world-frame representation suitable for MPM.
- Dynamic-scene neural fields model motion through frame consistency, canonical representations, deformation maps, or density advection, but do not address all boundary conditions.
- Soft-body system identification is difficult because of high dimensionality and large deformations, while black-box methods struggle to achieve high accuracy.
- Differentiable physics methods generally assume watertight geometric meshes, whereas PAC-NeRF does not require known object geometries.
- MPM supports topology changes, frictional contacts, and materials ranging from elastic objects and sand to fluids and foam.
3 METHOD
PAC-NeRF jointly recovers object geometry and physical properties from posed multi-view videos without requiring known object geometries. It combines neural radiance fields with differentiable continuum physics through particle–grid conversions, rendering losses, and geometry regularization.
- PAC-NeRF recovers explicit geometry and physical properties from posed multi-view videos without requiring known object geometries.
- Continuum neural radiance fields: Dynamic NeRF represents time-dependent density and color fields, whose rendered pixel colors are trained to match video observations.
- Continuum neural radiance fields: The method constrains density and appearance with conservation laws and evolves continuum momentum using a differentiable Material Point Method.
- Particle-grid interconversion: A hybrid Eulerian–Lagrangian representation advects particle fields with MPM, maps them back to voxels for collisions and rendering, and renders at least 100× faster than the referenced alternative.
- Geometry-agnostic system identification: The pipeline preprocesses videos with matting, seeds geometry through coarse-to-fine optimization, then estimates physical parameters by rendering future frames and minimizing video error.
- Geometry-agnostic system identification: A surface regularizer minimizes total surface area, producing more compact reconstructed point clouds that fit segmentation boundaries more closely.
4 IMPLEMENTATION DETAILS
The implementation uses a voxelized NeRF with a shallow view-dependent color network and a differentiable MPM simulator. The evaluation dataset spans diverse materials and photorealistic falling-object collision scenarios, while rendering and simulation are substantially faster than a cited prior system.
- NeRF and simulator implementation: The voxelized NeRF stores density and color features in 160^3 voxels and uses a 2-layer MLP with hidden dimension 128 for view-dependent colors.
- Evaluation dataset: The photorealistic dataset covers Newtonian and non-Newtonian fluids, granular media, deformable solids, and plasticine.
- Evaluation dataset: Objects in the dataset freely fall under gravity and undergo collisions under complex environmental lighting conditions.
- Efficiency: Simulation and rendering of one frame takes approximately 1s, compared with approximately 10min for Chen et al. (2022).
5 EXPERIMENTS
PAC-NeRF is evaluated on synthetic and real multiview data spanning diverse continuum materials and system-identification settings. It reconstructs dynamic objects and estimates physical parameters, outperforming comparison methods in most reported categories while revealing limitations from sparse views and unstable baselines.
- Experiment setup: Experiments cover deformable objects, plastics, granular media, Newtonian fluids, and non-Newtonian fluids.The physical-parameter implementation supports elasticity, plasticine, sand, Newtonian fluids, and non-Newtonian fluids.
- Synthetic data: PAC-NeRF estimates physical parameters closely agreeing with ground truth across 9 synthetic problem instances.Table 1 reports initial guesses, optimized values, and ground truth for each instance.
- Real data: PAC-NeRF reconstructs real-world multiview video scenes qualitatively, although four available views produce slight surface-geometry errors.The real-data evaluation uses synchronized RGB cameras without recorded depth data.
- System-identification results: PAC-NeRF achieves the best results in 14 of 17 physical-property categories compared with the evaluated approaches.Performance is measured using mean absolute error and standard deviation over evaluated instances.
- Baseline analysis: The D-NeRF plus differentiable-simulation comparison produces noisy deformation and is evaluated on only one scenario per material because of instability.The comparison cannot guarantee physical correctness because it lacks PAC-NeRF’s conservation-law constraints.
- Baseline analysis: PAC-NeRF’s Eulerian-Lagrangian representation is more robust under large deformations and permits larger time steps than the FEM-based NeRF+∇Sim baseline.NeRF+∇Sim supports only elastic materials and requires tens of thousands of substeps for the full sequence.
6 CONCLUSION
PAC-NeRF unifies physical simulation and neural rendering to estimate geometry and physical properties from image-space signals without geometry priors. Its hybrid representation combines efficient voxel-based NeRF with flexible MPM, while requiring synchronized, calibrated cameras and suitable foreground masks.
- PAC-NeRF enables differentiable simulators to estimate both geometry and physical properties from image-space signals without geometry priors.
- The implementation combines an efficient Eulerian voxel-based NeRF with the flexibility of MPM through a hybrid representation.
- The method assumes synchronized, accurately calibrated cameras and scenes amenable to video matting or available foreground masks.
- PAC-NeRF assumes continuum mechanics and cannot automatically distinguish between different materials.
A.1 PHYSICAL MODELS
The physical models use MPM deformation gradients and constitutive laws tailored to elastic, plastic, fluid, granular, and non-Newtonian materials. Plasticity is handled through yield criteria and return mappings that constrain deformation to elastic regions.
- MPM computes internal forces from Cauchy stress and tracks deformation gradients on particles to measure distortion from the initial state.
- Elasticity: Neo-Hookean elasticity models elastic objects, with Lamé parameters related to Young’s modulus and Poisson’s ratio.
- Fluids: J-based fluid dynamics with a viscosity term models Newtonian fluids, while viscoplastic dynamics model non-Newtonian fluids with yield stress.
- Plasticine: St.Venant-Kirchhoff elasticity with von-Mises plastic return mapping models plasticine using Hencky strain from the deformation-gradient SVD.
- Sand: Drucker-Prager yield criteria model sand, with return mapping determined partly by the friction angle and volumetric strain.
B COMPLEX BOUNDARY CONDITIONS
PAC-NeRF handles elastic ropes falling onto rigid cylinders by incorporating pre-known boundary conditions into MPM simulations. Figure 6 compares ground-truth video, PAC-NeRF renderings, and reconstructed MPM particles, with physical parameters reported quantitatively in Table 3.
- Pre-known boundary conditions, such as the ground, can be added directly to MPM simulations.
- Figure 6 shows elastic ropes falling onto two rigid cylinders and compares the ground-truth video with PAC-NeRF renderings and reconstructed MPM particles.
- Table 3 lists the initial guesses, optimized values, and ground-truth values of the physical parameters for the elastic-rope example.
C RECONSTRUCTION QUALITY COMPARISON WITH D-NERF
Under sudden, large deformations, D-NeRF develops artifacts that can break apart or partially remove the object. PAC-NeRF avoids this unphysical behavior by constraining forward deformations with continuum conservation laws.
- D-NeRF is sensitive to sudden, large deformations and develops artifacts as simulation progresses.
- D-NeRF’s artifacts can make the object break apart and cause parts of it to partially disappear in rendered frames.
- PAC-NeRF avoids this unphysical result by constraining forward deformations to follow the physical conservation law.
D QUALITATIVE COMPARISON WITH ∇SIM ON REAL-WORLD DATA
The paper qualitatively compares PAC-NeRF with ∇Sim on real-world data using reconstructed geometry and manually configured rendering. It also highlights numerical difficulties associated with ∇Sim’s explicit FEM simulation.
- Figure 8 presents a qualitative comparison between PAC-NeRF and ∇Sim on real-world data.
- For the baseline comparison, the authors extract a surface mesh from the voxel-NeRF reconstructed point cloud and generate a tetrahedral mesh.
- The comparison uses a manually selected camera and an approximate rendering configuration.
- ∇Sim’s explicit FEM suffers from tiny time steps, leading to significant numerical errors in backpropagation.