Source-linked AI summary
Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video
Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, Christian Theobalt
TL;DR
General non-rigid dynamic scenes are difficult to reconstruct and render from a single monocular video because the problem is severely under-constrained. NR-NeRF separates a canonical radiance field from deformation using ray bending and unsupervised rigidity estimation, producing high-fidelity novel-view results, correspondences, and editing capabilities.
Problem
Single-camera monocular video makes general dynamic-scene reconstruction severely under-constrained, while existing methods often restrict scene categories or omit appearance.
Method
NR-NeRF jointly learns a canonical neural radiance field, a ray-bending deformation model, an unsupervised rigidity network, and deformation regularizers from monocular observations.
Results
NR-NeRF achieves high-fidelity reconstruction and novel-view synthesis for general non-rigid scenes, while providing unsupervised rigidity scores and dense 3D correspondences.
Takeaways & Limitations
A monocular video can support free-viewpoint rendering of dynamic scenes, including novel camera trajectories, dense correspondences, and simple scene editing.
Takeaways & Limitations
The method does not model appearance changes from deformation or lighting, cannot obtain exact cross-time correspondences, and depends on a static dominant background for camera estimation.
Abstract
from arXiv · showhide
We present Non-Rigid Neural Radiance Fields (NR-NeRF), a reconstruction and novel view synthesis approach for general non-rigid dynamic scenes. Our approach takes RGB images of a dynamic scene as input (e.g., from a monocular video recording), and creates a high-quality space-time geometry and appearance representation. We show that a single handheld consumer-grade camera is sufficient to synthesize sophisticated renderings of a dynamic scene from novel virtual camera views, e.g. a `bullet-time' video effect. NR-NeRF disentangles the dynamic scene into a canonical volume and its deformation. Scene deformation is implemented as ray bending, where straight rays are deformed non-rigidly. We also propose a novel rigidity network to better constrain rigid regions of the scene, leading to more stable results. The ray bending and rigidity network are trained without explicit supervision. Our formulation enables dense correspondence estimation across views and time, and compelling video editing applications such as motion exaggeration. Our code will be open sourced.
1. Introduction
NR-NeRF reconstructs general non-rigid dynamic scenes from monocular video by separating a canonical radiance field from scene deformation. It uses ray bending, rigidity estimation, and regularization to support novel spatio-temporal views and scene editing.
- Monocular video of a general dynamic scene is used to reconstruct geometry and appearance for novel spatio-temporal viewpoints.
- NR-NeRF represents the scene with a static canonical neural radiance field and a deformation field implemented as non-rigid ray bending.Both components are jointly trained from monocular observations, and the deformation model is independent of camera parameters for novel-view rendering.
- A learned rigidity score preserves static regions while separating non-rigid foreground from rigid background without direct supervision.
- Regularizers encourage sparse, small, volume-preserving deformations and preserve local shape, constraining the underdetermined reconstruction.
- The approach is evaluated against several neural novel-view rendering methods, with high-fidelity reconstruction and novel-view synthesis reported across non-rigid scenes.
- The method supports free-viewpoint rendering, dense correspondences between arbitrary frames, and editing operations including foreground removal and deformation exaggeration.
2. Related Work
Prior dynamic-scene rendering methods use multi-view, depth, geometric, or object-specific representations, while monocular deformable-scene methods often rely on correspondences or limited object settings. NR-NeRF instead targets general deformable scenes with appearance and arbitrary backgrounds without templates, explicit 3D reconstruction, or multi-view setups.
- Early dynamic-scene rendering used light fields, multi-view stereo, layered depth images, point clouds, meshes, and surfels.
- Fusion methods and active depth cameras improved dynamic capture using few-camera or single-camera configurations, but remained tied to depth or short-term temporal integration.
- Monocular approaches reconstruct deformable geometry using dense point tracks factorized into camera poses and per-view non-rigid 3D states.
- NR-NeRF avoids explicit image-based 3D reconstruction and supports arbitrary backgrounds, multiple objects, complex deformations, and appearance rendering.
- Related neural methods address humans, body reenactment, explicit dynamic reconstruction, neural rerendering, texture warping, or single-image human novel views.
3. Method
NR-NeRF models a dynamic scene with a canonical neural radiance volume and learned ray bending conditioned on per-image latent codes. Rigidity-aware and geometric regularization constrain deformations for stable novel-view rendering.
- Deformation model: NR-NeRF learns one canonical neural radiance volume whose geometry and appearance are deformed into each image by ray bending.The canonical volume is represented by an MLP, while deformation is modeled separately.
- Deformation model: Each image receives a latent code that conditions the ray-bending network, while the canonical volume remains unconditioned on that code.The network weights and latent codes are shared between coarse and fine volumes.
- Rigidity network: The rigidity network assigns each canonical point a rigidity score, masking deformations in rigid regions while allowing non-rigid regions to deform.Rigidity is jointly learned without direct supervision.
- Losses: The data loss compares rendered ray colors with ground-truth pixel colors, while opacity- and transmittance-weighted offset losses regularize visible deformations.The opacity weighting avoids penalizing visually unoccupied space, and transmittance weighting addresses artifacts from occluded points.
- Losses: Applying the offset loss to masked offsets destabilizes novel-view backgrounds, so the method separately regularizes the rigidity mask and raw offsets.The authors attribute the issue to multiplicative ambiguity between unmasked offsets and the rigidity mask.
- Losses: A divergence loss regularizes hidden regions by encouraging local shape preservation after deformation.The implementation estimates divergence with FFJORD’s fast, unbiased estimator, which is three times less expensive than exact computation.
4. Results
NR-NeRF produces faithful reconstructions and novel-view renderings while estimating rigidity and dense correspondences. Ablations and comparisons show that its regularization and rigidity design choices improve foreground deformation stability, background stability, and perceptual rendering quality.
- Qualitative Results: NR-NeRF renders deformed states from a given time into novel views while exposing rigidity scores and correspondences as additional outputs.These outputs are visualized from the novel viewpoint.
- Qualitative Results: NR-NeRF reconstructs inputs faithfully and synthesizes high-quality views from freely specified nearby camera positions and time steps.The method also estimates rigidity scores and dense 3D correspondences across camera views and time steps.
- Ablation Study: The divergence loss is crucial for stable deformations of foreground non-rigid objects.Removing it destabilizes the non-rigid component in novel-view renderings.
- Ablation Study: The remaining design choices jointly stabilize rigid backgrounds, with NR-NeRF producing the least background instability in fixed novel-view renderings.Background stability is measured from each pixel’s color standard deviation across time; the full method is the most stable configuration.
- Comparisons: Compared with Neural Volumes and rigid NeRF variants, NR-NeRF yields more plausible and temporally stable novel-view results.The rigid NeRF variants are blurry and static, while Neural Volumes can produce implausible results and naïve NR-NeRF shows temporal inconsistencies on different spatio-temporal trajectories.
- Comparisons: NR-NeRF achieves the best SSIM and LPIPS scores and the second-best PSNR, after naïve NR-NeRF.PSNR and SSIM favor higher values, whereas LPIPS favors lower values; results are averaged across scenes.
- Scene Editing: The learned representation supports foreground removal by segmenting canonical rigid and non-rigid regions with rigidity scores and making the non-rigid part transparent.This editing operation assumes non-rigid foreground objects receive higher rigidity-network scores than rigid background objects.
5. Limitations
NR-NeRF achieves sharp novel-view synthesis and stable backgrounds, but remains bounded by modeling, data, and regularization assumptions. Its runtime also increases by about 20% from ray bending.
- Ray bending increases runtime by about 20%, although fewer sampled rays and points allow training in about six hours.
- Only NR-NeRF synthesizes sharp novel views, while competing methods produce implausible, blurry, or static results.The comparison covers input reconstruction in the first row and novel-view synthesis in the second row.
- NR-NeRF shows the least background instability when evaluated across all test-time steps.
- Foreground removal can fail when a foreground part is entirely static, and changing shadows or deformation-driven appearance are not modeled.
- Realistic rendering is limited for scene regions barely observed during training, motion-blurred inputs, or scenes without a static, dominant background for structure-from-motion.
6. Conclusion
The paper concludes that NR-NeRF enables free-viewpoint rendering of dynamic scenes from monocular video, with high-quality reconstructions, novel views, and unsupervised scene understanding. It identifies multi-view and view-dependent extensions as future directions.
- NR-NeRF performs free-viewpoint rendering of dynamic scenes using only monocular video input.
- The method produces high-quality reconstruction and novel-view synthesis results, unsupervised rigidity scores, and dense 3D correspondences.
- Learning a rigidity mask mitigates background instability in an unsupervised fashion.
- Extensions to multi-view data and view dependence are proposed for future work in more constrained settings targeting higher quality.
S.1. Loss Illustration
The supplemental illustration shows that Ldivergence encourages the offsets field to preserve local volume during deformation rather than lose it.
- Ldivergence encourages the offsets field to preserve local volume while deforming.Without this behavior, deformation can lose local volume.
S.2. Training Details
Training uses auto-decoded latent codes and NeRF-style networks, with dedicated MLPs for ray bending and rigidity. Optimization uses ADAM, learning-rate scheduling, and scene-specific loss weights.
- Latent codes li are auto-decoded and optimized directly as free variables alongside network weights.The latent codes are initialized to zero vectors.
- The radiance field uses the NeRF architecture, while ray bending and rigidity use 5-layer and 3-layer MLPs respectively.Both ray-bending and rigidity final layers are initialized with zero weights.
- ADAM training exponentially decays the learning rate to 10% of its initial 5 · 10^-4 over 250k iterations.Dark scenes additionally use a 1000-iteration warm-up from 1/20 of the original learning rate.
- Training takes between six and seven hours on a single Quadro RTX 8000.
- Scene-specific loss weights are used because the method handles varied non-rigid objects and deformations.Typical values are ωrigidity = 0.003, ωoffsets = 600, and ωdivergence = 3 or 10.
S.3. Implementation Details
The implementation ports NeRF to PyTorch, uses FFJORD for Eq. 5, and estimates missing camera parameters with COLMAP. It also applies positional encoding selectively for smooth deformations.
- The implementation is based on a faithful PyTorch port of the official TensorFlow NeRF code.
- FFJORD is used to estimate Eq. 5 from the main paper.
- When camera intrinsics or extrinsics are unavailable, COLMAP estimates them using Structure-from-Motion.
S.4. Data
The experiments use recordings from three camera types, with frames sampled at 5 fps. Kinect Azure images are undistorted before processing because of strong radial distortion.
- Scenes were recorded with Kinect Azure, Blackmagic, and phone cameras.
- Frames were extracted at 5 fps, producing usually 80 to 300 images per scene.
- Kinect Azure RGB images were undistorted using manufacturer-provided intrinsics and distortion parameters.
S.5.1. Visualizations
The supplemental material visualizes NR-NeRF’s rigidity, correspondences, canonical volume, novel-view rendering, and editing capabilities, while examining stability, view dependence, and limitations.
- S.5.1. Visualizations: Rigidity visualization assigns each ray the rigidity of the point nearest the median accumulated rendering weight.
- S.5.1. Visualizations: Correspondences are visualized by encoding canonical-space xyz coordinates as RGB colors on a 100^3 voxel grid.
- S.5.1. Visualizations: Novel-view outputs include rigidity scores, correspondences, and the canonical volume, which is shown as a plausible state without baked-in deformations.
- S.5.1. Visualizations: Ablations evaluate novel-view stability after removing divergence regularization, all regularizers, or none, with divergence regularization important for foreground stability.
- S.8.2. View Dependence: Multi-view data can extend NR-NeRF and provide an upper-quality-bound setting, while view-direction conditioning reduces smoke-like artifacts around specular regions.
- S.8.2. View Dependence: In monocular sequences with significant motion and strongly different novel views, view-dependent effects can produce artifacts because the setting is underconstrained.