Source-linked AI summary
NVFi: Neural Velocity Fields for 3D Physics Learning from Dynamic Videos
Jinxi Li, Ziyang Song, Bo Yang
TL;DR
Dynamic-scene radiance fields often prioritize interpolation and do not explicitly learn physical velocity, while existing velocity learning is challenging without ground-truth labels and can be under-constrained. NVFi jointly learns scene geometry, appearance, and velocity from multi-view video using keyframe and interframe fields with photometric and PDE losses. The authors report superior performance over baselines, particularly in future-frame extrapolation and semantic decomposition.
Problem
Dynamic-scene methods commonly fit observed sequences without explicitly learning physical velocity, while learning whole-scene velocity is difficult without 3D labels and can be under-constrained.
Source: §1 · p. 1Method
NVFi jointly optimizes keyframe dynamic radiance and interframe velocity fields using photometric and governing-PDE losses.
Source: §1 · p. 2Results
NVFi outperforms baselines across the reported applications, particularly in future-frame extrapolation and semantic scene decomposition.
Source: §1 · p. 2, §4.1 · p. 7, §4.2 · p. 8Takeaways & Limitations
The authors report that learned velocity supports future-frame extrapolation, semantic scene decomposition, and motion transfer across scenes.
Source: §1 · p. 2, §5 · p. 9The canonical-frame strategy assumes corresponding point appearances remain unchanged across timestamps, so dramatically changing scenes may yield inferior geometry, appearance, and learned physics.
Source: §3.2 · p. 4
Abstract
from arXiv · showhide
In this paper, we aim to model 3D scene dynamics from multi-view videos. Unlike the majority of existing works which usually focus on the common task of novel view synthesis within the training time period, we propose to simultaneously learn the geometry, appearance, and physical velocity of 3D scenes only from video frames, such that multiple desirable applications can be supported, including future frame extrapolation, unsupervised 3D semantic scene decomposition, and dynamic motion transfer. Our method consists of three major components, 1) the keyframe dynamic radiance field, 2) the interframe velocity field, and 3) a joint keyframe and interframe optimization module which is the core of our framework to effectively train both networks. To validate our method, we further introduce two dynamic 3D datasets: 1) Dynamic Object dataset, and 2) Dynamic Indoor Scene dataset. We conduct extensive experiments on multiple datasets, demonstrating the superior performance of our method over all baselines, particularly in the critical tasks of future frame extrapolation and unsupervised 3D semantic scene decomposition.
1 Introduction
Dynamic NeRF methods often fit observed sequences without explicitly learning velocity, limiting their ability to predict future motion; existing physics-based approaches also face disentanglement challenges. NVFi jointly learns scene geometry, appearance, and velocity from multi-view video, using joint optimization to support future-frame extrapolation, semantic decomposition, and motion transfer.
1 Introduction
- Existing dynamic-scene methods generally fit training sequences without explicitly learning physical velocity, limiting future-motion prediction.Prior physics-informed approaches also couple learned properties to target objects or require foreground masks, limiting transferability.Source: §1 · p. 1
- Velocity is the paper’s chosen physical property because it governs scene movement and is proposed to support future-frame prediction, motion transfer, and semantic decomposition without extra annotations.The authors describe these applications as expected uses of learned velocity fields.Source: §1 · p. 2
- Learning whole-scene velocity is difficult without 3D velocity labels, known object types or materials, or dense visual trajectories; separate velocity learning is under-constrained even with PINN.Source: §1 · p. 2
- NVFi learns geometry, appearance, and disentangled velocity from multi-view video using a keyframe radiance field, an interframe velocity field, and joint optimization.The optimization combines keyframe and interframe photometric losses with governing PDE losses.Source: §1 · p. 2
- NVFi is reported to outperform baselines in future-frame extrapolation, semantic decomposition, and velocity transfer across two new datasets and a real-world dataset.Source: §1 · p. 2
2 Related Works
Related work spans implicit representations, dynamic radiance fields, and physics-informed learning. NVFi targets the gap between fitting dynamic video and learning transferable scene velocity.
Static 3D Representations
- Implicit neural representations regress point properties such as occupancy, distance, color, or semantics, enabling continuous shape and appearance modeling with low memory footprint.The passage contrasts them with discretized representations including voxels, point clouds, octrees, meshes, and primitives.Source: §2 · pp. 2–3
Dynamic 3D Representations
- Dynamic NeRF methods commonly add time and optimize photometric loss, but most focus on interpolation rather than predicting future frames.The surveyed approaches include deformation-based, flow-based, and direct space-time methods, as well as domain-specific models.Source: §2 · p. 3
Physics Informed Deep Learning
- Physics-informed neural networks encode tensor fields with neural networks and PDE-based losses, but commonly used PINN constraints are insufficient for accurate 3D velocity fields in this framework.The authors address this by proposing a keyframe-based velocity propagation module.Source: §2 · p. 3
3 NVFi
NVFi represents dynamic scenes with a keyframe radiance field and an interframe velocity field, then trains them using photometric and physics-based constraints. Its strategy combines sparse keyframe fitting, velocity-based transport at interframes, and joint optimization.
3.1 Overview
- The keyframe field maps 3D position, viewing angle, and time to density and color, while the interframe field maps position and time to 3D velocity.The method uses HexPlane for the radiance field and MLPs for velocity, though other architectures may be used.Source: §3.1 · p. 3
- The central challenge is optimizing both networks so the learned velocity is precise and disentangled enough to support extrapolation, motion transfer, and semantic decomposition.Source: §3.1 · p. 4
3.2 Optimization of Keyframe Dynamic Radiance Fields
- Keyframe optimization uniformly samples K of T timestamps; unlike dense-frame fitting, it lets remaining interframes train velocity, while avoiding the canonical strategy’s unchanged-appearance assumption.Dense fitting can make geometry and appearance inefficient to learn and entangle physical change; canonical-frame optimization can degrade on dramatically changing scenes.Source: §3.2 · p. 4
- At keyframes, the radiance field is trained by comparing rendered ray colors with ground-truth image colors.Source: §3.2 · p. 4
3.3 Optimization of Interframe Velocity Fields
- The velocity field is regularized by divergence-free and momentum PDE losses, but these constraints alone leave infinitely many possible solutions.The losses sample scene positions and times up to the maximum extrapolation time; acceleration is learned by another MLP.Source: §3.3 · pp. 4–5
- For interframes, the method transports points to a nearest keyframe using velocity, retrieves density and appearance features there, and renders the resulting colors for photometric supervision.It avoids directly querying the radiance field at untrained interframe timestamps, so interframes constrain both networks through their photometric loss.Source: §3.3 · p. 5
3.4 Joint Keyframe and Interframe Optimization
- Joint optimization sends keyframe and interframe photometric losses to the radiance field, and interframe photometric plus PDE losses to the velocity field.The authors report that propagating interframe errors back to the radiance field improves overall performance compared with training the fields separately.Source: §3.4 · p. 6
4 Experiments
The experiments evaluate NVFi on new synthetic datasets and a real-world dataset, across future-frame extrapolation, semantic scene decomposition, motion transfer, and ablations. Results highlight gains in extrapolation and decomposition, while identifying limits tied to motion-transfer scene reconstruction and sparse views.
4 Experiments
- The evaluation uses two synthetic datasets and NVIDIA Dynamic Scene, whose selected clips have predictable motions and train/test splits for interpolation and future-frame extrapolation.The new datasets target meaningful physical motion rather than only fitting frames; the indoor dataset also supplies ground-truth segmentation masks.Source: §4 · p. 6
4.1 Evaluation of Future Frame Extrapolation
- NVFi significantly outperforms all baselines on both synthetic datasets, especially in future-frame extrapolation; on NVIDIA Dynamic Scene, it is best in extrapolation and comparable in interpolation.The reported comparison attributes extrapolation performance without an interpolation drop to accurate motion prediction.Source: Table 1 · p. 7, Table 2 · p. 7, §4.1 · p. 7
4.2 Evaluation of 3D Semantic Scene Decomposition
- At t=0, a pointwise object code is optimized using transported-point motions, rigid-consistency, and smoothness losses, enabling tracking and mask rendering; NVFi beats D-NeRF and Mask2Former on all metrics.The evaluation renders 7,200 masks across 30 viewing angles and 60 frames per scene.Source: §4.2 · p. 8, Table 3 · p. 8
4.3 Evaluation of Motion Transfer
- Motion transfer preserves the new static object's geometry and appearance better than D-NeRF, but the quantitative advantage is limited by artifacts where the hidden ground plane lacks supervision.The comparison transfers Gnome's learned velocity field to a separately trained static TensoRF model of Gnome-new.Source: §4.3 · pp. 8–9, Table 4 · p. 8
4.4 Ablation Study
- Ablations indicate that joint optimization is critical, larger keyframe counts reduce extrapolation, and extremely sparse camera views provide insufficient visual information for motion learning.The study varies joint optimization, physics constraints, keyframe count, and camera count.Source: §4.4 · p. 9, Table 5 · p. 9, Table 6 · p. 9
5 Conclusion
NVFi represents dynamic 3D scenes from multi-view video by jointly learning radiance and disentangled velocity fields. Experiments across three datasets demonstrate accurate velocity learning and applications in extrapolation, semantic decomposition, and motion transfer.
5 Conclusion
- NVFi jointly optimizes a keyframe dynamic radiance field and an interframe velocity field to disentangle physical velocity from scene geometry and appearance.Source: §5 · p. 9
- Experiments on three dynamic datasets demonstrate that the framework learns accurate velocity.Source: §5 · p. 9
- The learned velocity supports future-frame extrapolation, semantic scene decomposition, and motion transfer.Source: §5 · p. 9
A Appendix
The appendix lists supplementary implementation, dataset, and experiment details, and includes a demo video visualizing results across tasks.
A Appendix
- Supplementary materials cover the rendering equation, field and optimization implementation, physics priors, datasets, and semantic decomposition and motion transfer.Source: A Appendix · p. 15
- Additional ablation results and future-frame extrapolation results are included in the appendix.Source: A Appendix · p. 15
- A demo video visualizes qualitative results across multiple tasks.Source: A Appendix · p. 15
A.1 Rendering Equation
The rendering equation maps points along a camera ray to a pixel by accumulating their colors and densities. The ray runs from the camera center toward the pixel, between predefined near and far bounds.
A.1 Rendering Equation
- A camera ray runs from the camera center toward a pixel, and points along it are sampled between predefined near and far bounds.The ray is parameterized by its origin, direction, and distance from the camera.Source: §A.1 · p. 15
- The expected pixel color is obtained by sampling points along the ray and applying classical volume rendering.Source: §A.1 · p. 15
- Equation (11) connects 3D points to image pixels by accumulating sample colors and densities; δi is the distance between adjacent samples, and N is the number of samples.Source: §A.1 · p. 15
A.2 Implementation Details of Keyframe Dynamic Radiance Field
The keyframe dynamic radiance field adapts HexPlane to represent volumetric keyframes, combining spatially interpolated grid features to predict density and appearance. Time is selected by keyframe index, and a two-layer MLP regresses color from appearance features and view direction.
A.2 Implementation Details of Keyframe Dynamic Radiance Field
- The field adapts HexPlane, a direct extension of static TensoRF to a dynamic radiance field, to parameterize volumetric keyframes.Source: §A.2 · p. 15
- Rather than concatenating features from different planes as in TensoRF and HexPlane, the method uses Hadamard products, which are reported to represent complex geometries more strongly.Source: §A.2 · p. 15
- At keyframe time τk, spatial grid interpolation predicts density σ and appearance feature A; time is selected by keyframe index rather than interpolated.Density features are summed, while appearance features are linearly weighted; a two-layer MLP uses the appearance feature and view direction to regress RGB color.Source: §A.2 · pp. 15–16
- Experiments use Mσ=24 and Mapp=48; keyframe counts are 16 for object extrapolation and 4 for indoor extrapolation and segmentation.The color decoder MLP has two hidden layers with 128 nodes each.Source: §A.2 · p. 16
A.3 Implementation Details of Velocity Field
The velocity field represents 3D velocity as a function of spatial position and time, using a weight MLP and a velocity basis field. The authors report that this decomposition can make velocity convergence faster.
A.3 Implementation Details of Velocity Field
- The velocity field maps position and time, (x,y,z,t), to a 3D velocity through a weight MLP multiplied by a velocity basis field.Source: §A.3 · p. 16
- The authors report that decomposing velocity into a weight MLP and basis field can make velocity convergence faster.Source: §A.3 · p. 16
- The weight MLP has four hidden layers of 128 nodes and outputs six weights interpreted as linear velocity in x, y, z and angular velocity around those axes.It uses a positional encoder of dimension 3, as in [37].Source: §A.3 · p. 16
A.4 Additional Details of Physics Prior
The physics prior imposes conservation laws on appearance and volume-density features using a divergence-free implicit velocity field. It assumes local rigidity and incompressible motion, and also derives a momentum-conservation constraint under uniform mass density.
A.4 Additional Details of Physics Prior
- Appearance and volume-density features are constrained to obey conservation laws characterized by the divergence-free implicit velocity field.The material derivative is identified as Dfi/Dt.Source: §A.4 · p. 16
- The prior assumes local rigidity and incompressible motion, yielding the divergence-free condition ∇·v = 0.Source: §A.4 · p. 16
- Momentum conservation is derived assuming uniform mass density, with Newton's second law stating that momentum is conserved when no forces act on the mechanical system.Source: §A.4 · p. 16
A.5 Implementation Details of Joint Optimization
The implementation details specify a progressively refined keyframe grid, optimization settings, and sampling and physics-loss configurations for training.
A.5 Implementation Details of Joint Optimization
- The keyframe grid grows from 64^3 to 200^3, with resolution increases at 2k-iteration intervals through 10k iterations.Feature-plane learning rate is 0.02; the color decoder and velocity field use 0.001.Source: §A.5 · p. 16
- Adam uses β1 = 0.9 and β2 = 0.99, with learning rates decayed to one tenth by iteration 30k.Total variation loss is applied to all feature planes with spatial-axis weight λ = 0.001.Source: §A.5 · p. 17
- Each iteration samples 262,144 points for dynamic objects or 1,310,672 for indoor scenes, applying physics losses only in occupied regions.Divergence-free and momentum-conservation weights are 5 and 0.1, respectively; each scene takes 1.5 hours on one NVIDIA RTX 3090.Source: §A.5 · p. 17
A.6 Additional Details of Datasets
These appendices detail the datasets, scene-decomposition and motion-transfer procedures, and additional quantitative and qualitative evaluations.
A.6 Additional Details of Datasets
- The Dynamic Object dataset has six rigid or deformable objects, while the four-scene indoor dataset includes rigid objects and ground-truth segmentation masks.Both use 60-frame views, training on the first 45 frames; the object dataset uses 12 training and 3 test views, and the indoor dataset uses 25 and 5.Source: §A.6 · p. 17
A.7 Implementation Details about Semantic Scene Decomposition
- Scene decomposition filters low-density points at time 0, predicts per-point object codes, and transports points with the learned velocity field to compute motion vectors.Dynamic rigid consistency and spatial smoothness losses distinguish motion groups and discourage over-segmentation; the decomposition network is a four-layer MLP with at most eight objects.Source: §A.7 · p. 18
- For Mask2Former comparison, the ground plane and wall are manually merged into one static background segment for fair comparison.The implementation uses a pretrained Swin-L backbone.Source: §A.7 · p. 18
A.8 Additional Implementation Details about Motion Transfer
- Motion transfer reuses the velocity field trained for future-frame extrapolation and trains a target-scene static representation by setting NVFi's keyframe count to one.The pretrained velocity-field parameters are loaded into the target-scene model.Source: §A.8 · p. 19
A.9 Additional Quantitative Results for Ablation Study
- The additional ablations report interpolation and extrapolation results, varying keyframe number and camera number across the listed datasets.Tables 7–9 cover the Dynamic Object dataset, Dynamic Indoor Scene dataset, and Dynamic Objects dataset, respectively.Source: §A.9 · p. 19, Table 7 · p. 19, Table 8 · p. 19
A.10 Additional Quantitative Results for Future Frame Extrapolation
- Across individual Dynamic Object and Dynamic Indoor Scene cases, NVFi significantly outperforms all baselines on future-frame extrapolation and frequently leads on interpolation.The per-scene results are reported in Tables 10 and 11.Source: §A.10 · p. 19, Table 10 · p. 20, Table 11 · p. 21
A.11 Additional Quantitative Results for Semantic Scene Decomposition
- With an integral time step of 0.06, semantic-decomposition results are significantly boosted and surpass both baselines by large margins.Tables 12 and 13 report overall and per-scene results.Source: §A.11 · p. 19, Table 12 · p. 21, Table 13 · p. 21
A.12 Additional Qualitative results for Future Frame Extrapolation; A.13 Additional Qualitative results for semantic scene decomposition
- Additional qualitative panels show interpolation within the observed interval and future frames at time 0.83 or 1 for indoor decomposition, plus part decomposition for dynamic objects.Other panels show interpolation at times 0 and 0.5 and future frames at time 1, including a seen viewing angle at a future timestamp.Source: §A.13 · p. 19, §A.12 · p. 19