Source-linked AI summary
D-NeRF: Neural Radiance Fields for Dynamic Scenes
Albert Pumarola, Enric Corona, Gerard Pons-Moll, Francesc Moreno-Noguer
TL;DR
Neural radiance fields are highly effective for static-scene novel-view synthesis but do not support moving or deforming scenes. D-NeRF addresses this gap by learning canonical geometry and time-dependent deformation from sparse monocular images, producing high-quality novel views across varied motions.
Problem
Existing neural radiance-field approaches assume static scenes, whereas dynamic-scene synthesis from sparse monocular observations remains the target problem.
Method
D-NeRF jointly learns a canonical scene representation and a time-dependent displacement field using neural networks.
Results
D-NeRF synthesizes high-quality novel views for articulated objects and humans performing complex body postures.
Takeaways & Limitations
The method controls both camera viewpoint and time while learning from sparse images captured by a moving monocular camera.
Takeaways & Limitations
The evaluation setting assumes a single image per time instance and does not observe a specific deformation state from multiple viewpoints.
Abstract
from arXiv · showhide
Neural rendering techniques combining machine learning with geometric reasoning have arisen as one of the most promising approaches for synthesizing novel views of a scene from a sparse set of images. Among these, stands out the Neural radiance fields (NeRF), which trains a deep network to map 5D input coordinates (representing spatial location and viewing direction) into a volume density and view-dependent emitted radiance. However, despite achieving an unprecedented level of photorealism on the generated images, NeRF is only applicable to static scenes, where the same spatial location can be queried from different images. In this paper we introduce D-NeRF, a method that extends neural radiance fields to a dynamic domain, allowing to reconstruct and render novel images of objects under rigid and non-rigid motions from a \emph{single} camera moving around the scene. For this purpose we consider time as an additional input to the system, and split the learning process in two main stages: one that encodes the scene into a canonical space and another that maps this canonical representation into the deformed scene at a particular time. Both mappings are simultaneously learned using fully-connected networks. Once the networks are trained, D-NeRF can render novel images, controlling both the camera view and the time variable, and thus, the object movement. We demonstrate the effectiveness of our approach on scenes with objects under rigid, articulated and non-rigid motions. Code, model weights and the dynamic scenes dataset will be released.
1. Introduction
D-NeRF extends neural radiance fields to dynamic scenes, addressing the static-scene assumption with a canonical representation and a learned deformation mapping. It enables novel-view synthesis under varied object motion from sparse monocular observations.
- Neural radiance fields map spatial locations and camera views to emitted radiance and volume density for photorealistic free-viewpoint rendering.
- Existing neural rendering approaches assume static scenes without moving or deforming objects.
- D-NeRF requires only a single moving camera and learns end-to-end without precomputed 3D reconstruction.
- D-NeRF represents dynamic scenes with a time-aware spatial mapping to a canonical configuration and a second module that models scene radiance.
- The method renders high-quality images while controlling both camera view and time across articulated and complex non-rigid motions.
2. Related work
Prior neural and geometric methods either target static scenes or require stronger supervision and capture setups for dynamic, non-rigid scenes. D-NeRF instead learns a time-varying neural volume from monocular data and supports arbitrary-view rendering.
- Neural implicit representations encode properties of 3D points with neural networks, enabling continuous surface reconstruction at low memory cost.
- Earlier implicit geometry methods often required 3D ground-truth geometry, while later approaches used differentiable rendering for 2D supervision.
- Existing neural radiance-field methods achieve strong results on rigid scenes but do not handle dynamic and deformable scenes.
- Several dynamic-scene approaches require full 3D supervision, multi-view capture, or more than one view, limiting monocular use.
- D-NeRF learns a neural implicit representation for non-rigid, time-varying scenes from monocular data without 3D ground-truth supervision or multi-view capture.
- Unlike view interpolation, D-NeRF learns time-varying volume density and radiance, making novel-view synthesis a ray-casting process for arbitrary viewpoints.
3. Problem Formulation
The problem is to learn a time-conditioned neural representation that synthesizes novel views of a dynamic scene from sparse monocular images. D-NeRF decomposes this mapping through a shared canonical configuration and time-dependent deformation.
- The goal is to synthesize novel views at arbitrary times from a sparse set of images captured by a monocular camera.
- The desired mapping takes a 3D point, viewing direction, and time as inputs and outputs emitted color and volume density.
- Rather than learn the full time-conditioned mapping directly, D-NeRF splits it into canonical-scene and deformation mappings.
- The deformation mapping transforms points at each time into a common canonical configuration, where the canonical mapping predicts color and density.
- Training assumes one image per time instance, so the same deformation state is not observed from different viewpoints.
4. Method
D-NeRF extends NeRF to dynamic scenes by learning a canonical representation and a deformation field from sparse images, including a single view per time instant. Volume rendering maps deformed ray samples into canonical space, where color and density are predicted and optimized against observed pixels.
- Model Architecture: D-NeRF uses two neural modules: a canonical network predicts color and density, while a deformation network maps time-specific points into canonical space.The canonical network Ψx(x, d) outputs emitted color c and volume density σ; the deformation network Ψt(x, t) outputs displacement Δx.
- Canonical Network: The canonical configuration aggregates corresponding information across images, allowing missing viewpoint information to be retrieved from a shared scene representation.The canonical scene serves as an anchor interconnecting images.
- Volume Rendering: For each camera ray, D-NeRF transforms sampled points at time t into canonical space, then uses the canonical network to predict density and view-dependent color.The expected pixel color is computed with adapted NeRF volume-rendering equations using accumulated transmittance and sampled ray points.
- Volume Rendering: Stratified sampling uniformly draws quadrature points from evenly spaced bins along each ray, with δn representing the distance between adjacent samples.The sampled points support numerical approximation of the volume-rendering integrals.
- Learning the Model: The canonical and deformation networks are simultaneously learned by minimizing mean squared error between rendered and real RGB pixels from the training images.Each time instant is acquired by a single camera, and training batches sample pixels corresponding to camera rays.
5. Implementation Details
The implementation uses simple fully connected multilayer perceptrons, fixes the canonical state at t = 0, and trains with a specified ray-sampling and optimization setup.
- Network Configuration: Both networks are 8-layer MLPs with ReLU activations; the canonical network applies a final sigmoid to color and density, while deformation outputs remain unconstrained by a nonlinearity.The deformation network directly predicts Δx.
- Training Schedule: The canonical configuration is fixed at t = 0, and images are introduced in increasing timestamp order through curriculum learning.The curriculum incrementally adds images with higher time stamps to improve convergence.
- Optimization: Training uses 400×400 images for 800k iterations, with batches of 4096 rays and 64 samples per ray, optimized by Adam on one Nvidia GTX 1080 for two days.The learning rate starts at 5e−4 and decays exponentially to 5e−5.
6. Experiments
D-NeRF is evaluated through component analyses, quantitative comparisons, and novel-view synthesis across diverse dynamic scenes. The experiments show canonical and deformation modeling, shading handling, and time-view conditioning for complex motions.
- 6.1. Dissecting the Model: D-NeRF’s canonical network models fine geometric and appearance details from sparse images captured under different deformations.The canonical RGB image, occupancy mesh, and depth map are obtained by evaluating the canonical network and applying marching cubes to its learned density.
- 6.1. Dissecting the Model: The deformation network estimates consistent mappings from the canonical scene to configurations at t = 0.5 and t = 1.Consistent colors across corresponding canonical and deformed points indicate that the displacement field is correctly estimated.
- 6.1. Dissecting the Model: D-NeRF synthesizes time-varying shading effects for points corresponding between canonical space and scenes at different times.The analysis uses plastic, translucent glass, and metal balls to examine appearance changes under deformation.
- 6.2. Quantitative Comparison: Compared with NeRF and T-NeRF, D-NeRF retains high-frequency details in novel views of dynamic scenes observed from a single viewpoint per deformation state.NeRF converges to a blurry mean representation, while T-NeRF captures dynamics but misses high-frequency details such as shoulder-pad spikes, stones, and cracks.
- 6.3. Additional Results: D-NeRF renders scenes from two novel viewpoints across time for articulated, human, and asynchronous motions.The evaluated scenes include Tractor, Jumping Jacks, Warrior, and Bouncing Balls; the canonical representation is also displayed.
- 6.3. Additional Results: The canonical representation is generally sharp, although the Jumping Jacks arms are blurry without harming rendered-image quality.This indicates that the network can warp the canonical configuration to maximize rendering quality and encode shading.
7. Conclusion
D-NeRF models dynamic scenes from sparse images captured by a moving camera, without precomputed 3D priors or repeated viewpoints of the same configuration. Its canonical-space and displacement-field modules synthesize high-quality novel views across articulated objects and complex human postures.
- D-NeRF is trained end-to-end from sparse images acquired with a moving camera.
- The method does not require precomputed 3D priors or observing the same scene configuration from different viewpoints.
- D-NeRF represents time-varying deformations with a canonical-configuration module and a displacement-field module defined relative to canonical space.
- Evaluation shows high-quality novel-view synthesis for articulated objects and human bodies performing complex postures.