PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural 3D Video Synthesis from Multi-view Video

Tianye Li, Mira Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, Zhaoyang Lv

arXiv:2103.02597v2cs.CVcs.GR

TL;DR

Dynamic real-world scenes are difficult to represent and render efficiently, especially at video scale. The paper introduces DyNeRF, a compact time-conditioned neural radiance field with latent codes and importance sampling, achieving tenfold faster training and a model 40 times smaller than a naïve per-frame NeRF baseline.

  • Problem

    Dynamic scenes challenge classical representations, while ray-based neural rendering makes large multiview video reconstruction prohibitively costly to train.

  • Method

    DyNeRF represents multiview dynamic scenes with a continuous latent-conditioned neural radiance field and trains it using hierarchical spatiotemporal importance sampling.

  • Results

    One order of magnitude acceleration in training speed and a model 40 times smaller than a naïve per-frame NeRF baseline are achieved for 10 seconds of 30 FPS 3D video.

  • Takeaways & Limitations

    The representation supports compact, high-quality novel-view rendering and smooth motion interpolation for complex dynamic multiview videos.

  • Takeaways & Limitations

    Highly dynamic scenes with large and fast motions remain challenging and may produce blur in moving regions.

Abstract

from arXiv · show

We propose a novel approach for 3D video synthesis that is able to represent multi-view video recordings of a dynamic real-world scene in a compact, yet expressive representation that enables high-quality view synthesis and motion interpolation. Our approach takes the high quality and compactness of static neural radiance fields in a new direction: to a model-free, dynamic setting. At the core of our approach is a novel time-conditioned neural radiance field that represents scene dynamics using a set of compact latent codes. We are able to significantly boost the training speed and perceptual quality of the generated imagery by a novel hierarchical training scheme in combination with ray importance sampling. Our learned representation is highly compact and able to represent a 10 second 30 FPS multiview video recording by 18 cameras with a model size of only 28MB. We demonstrate that our method can render high-fidelity wide-angle novel views at over 1K resolution, even for complex and dynamic scenes. We perform an extensive qualitative and quantitative evaluation that shows that our approach outperforms the state of the art. Project website: https://neural-3d-video.github.io/.

1. Introduction

Dynamic real-world scenes are difficult to represent and render because of complex geometry, appearance, and motion, while per-frame neural radiance fields are prohibitively costly. The paper addresses this with compact temporal latent codes and importance sampling for high-quality view synthesis and motion interpolation.

  • Motivation: Dynamic real-world scenes challenge classical mesh representations because they contain thin structures, semi-transparent objects, specular surfaces, and evolving topology.These difficulties arise from complex motion involving multiple scene elements.
  • Problem: 15 000 GPU hours and about 1 GB in storage would be required for a per-frame NeRF representing 10 seconds of 30 FPS video from 18 cameras.Per-frame neural radiance fields scale linearly with time in both training cost and storage.
  • Approach: The proposed approach represents complex dynamic scenes compactly while enabling high-quality view synthesis and motion interpolation.It exploits time-invariant scene content and locally correlated changes in geometry and appearance.
  • Temporal Representation: Compact latent codes parameterize scene motion and appearance changes instead of directly adding time as an input to the neural radiance field.The codes capture vivid moving geometry and texture details and support smooth temporal interpolation.
  • Results: One order of magnitude acceleration in training speed and a model 40 times smaller are achieved for 10 seconds of 30 FPS 3D video versus a naïve per-frame NeRF.The result uses combined temporal and spatial importance sampling.

2. Related Work

The work relates to static-scene novel view synthesis, dynamic-scene 3D video synthesis, image-based rendering, and neural rendering. Prior methods support static view interpolation or dynamic capture and replay, whereas this work seeks continuous viewpoint-and-time interpolation in a compact unified representation.

  • Research domains: Related research spans static-scene novel view synthesis, dynamic-scene 3D video synthesis, image-based rendering, and neural rendering.Surveys [55] and [56] provide broader discussions of neural rendering applications and neural scene representations.
  • Novel View Synthesis for Static Scenes: Static-scene methods interpolate views effectively, but extending them to dynamic scenes remains unclear.One approach combines a dense voxel grid of features with a screen space network for view synthesis.
  • 3D Video Synthesis for Dynamic Scenes: Dynamic-scene techniques support view synthesis and potentially time interpolation through explicit geometry and textures, temporal layered representations, or human-focused reconstruction and animation.Human reconstruction and animation are usually model-based and/or only partially applicable to the broader dynamic-scene setting.
  • 3D Video Synthesis for Dynamic Scenes: The proposed direction is a unified space-time representation for continuous viewpoint and time interpolation of 10 seconds of multi-view video in as little as 28MB.The contrasted approach uses 46 cameras, requires a large training dataset for a strong scene prior, and operates at hundreds of Mb/s.

3. DyNeRF: Dynamic Neural Radiance Fields

DyNeRF reconstructs dynamic 3D scenes from time-synchronized multiview videos using a continuous space-time neural radiance field with jointly optimized temporal latent embeddings. Hierarchical training and temporal-variance-aware ray importance sampling address the scale and redundancy of video data to accelerate training and improve rendering quality.

  • Representation: DyNeRF represents dynamic scenes as a continuous 6D space-time neural radiance field controlled by jointly optimized temporal latent embeddings.The representation is queried continuously in space and time and compresses multiview videos into a compact representation.
  • Representation: Temporal latent codes compactly encode scene state while handling deformation, topological changes, and radiance changes.The codes are independently randomly initialized across frames, and positional encoding is applied to input positions but not to the latent codes.
  • Efficient Training: 7.4 billion ray samples occur in one epoch for a 10-second, 30 FPS, 1 MP sequence from 18 cameras, requiring about half a week on 8 NVIDIA Volta-class GPUs.The training burden arises because iterations scale linearly with the total number of input multiview-video pixels.
  • Efficient Training: Hierarchical training and ray importance sampling focus optimization on keyframes and regions with higher temporal variance, yielding faster training and improved rendering quality.The model is first trained on equidistant keyframes, then initialized and trained at the full video temporal resolution.
  • Efficient Training: DyNeRF-IS⋆ combines sharp-detail recovery from DyNeRF-ISG with temporal smoothing from DyNeRF-IST to balance jitter and blur.High learning rates make DyNeRF-ISG recover dynamic detail quickly but introduce jitter, whereas low-rate DyNeRF-IST produces smoother yet somewhat blurry sequences.

4. Experiments

Experiments evaluate DyNeRF on challenging multi-view and immersive video data using quantitative and qualitative comparisons against established baselines. The method achieves high-quality 1K novel-view rendering, while fast motion and computational cost remain limitations.

  • Datasets and Setup: The capture system uses 21 synchronized GoPro cameras recording 2.7K video at 30 FPS, with 18 training views and one evaluation view for most datasets.One multiple-people sequence uses 14 training views.
  • Datasets and Setup: The datasets include specular, translucent, transparent, topologically changing, shadowed, volumetric, view-dependent, variably lit, and multi-person scenes.These conditions include poured liquid, fire, moving shadows, and nighttime or spotlight illumination.
  • Novel-View Results: DyNeRF renders near-photorealistic continuous novel views at 1K resolution for multi-view videos up to 10 seconds at 30 FPS.Sub-frame latent-code interpolation enables effects such as slow motion and bullet time.
  • Quantitative Comparison: Compared with MVS, NeuralVolumes, LLFF, NeRF-T, and DyNeRF†, DyNeRF variants using hierarchical training and importance sampling perform significantly better across all reported metrics.The evaluation uses PSNR, mean square error, DSSIM, FLIP, and JOD, with all neural radiance field baselines trained for the same number of iterations.
  • Qualitative Comparison: Qualitative comparisons show that DyNeRF preserves photorealistic appearance, view-dependent effects, and moving-region details that competing methods often blur, omit, or reconstruct incompletely.MVS particularly suffers from incomplete reconstruction near occlusion boundaries, while NeRF-T loses moving-region appearance details.
  • Limitations: Fast, highly dynamic motion can cause blur, and achieving sharper results requires longer training, making computation a current challenge.The reported implementation trains for about a week using eight NVIDIA V100 GPUs.

5. Conclusion

The paper proposes a compact, expressive neural 3D video representation for dynamic real-world multi-view recordings, enabling high-quality view synthesis and motion interpolation.

  • Conclusion: The approach represents a 10-second multi-view recording from 18 cameras in under 28MB.Its model-free representation supports both high-quality view synthesis and motion interpolation.

B. Datasets

The dataset uses a synchronized mobile multiview system with 21 GoPro cameras capturing 2.7K, 30 FPS video, supplemented by diverse indoor, outdoor, and publicly available immersive sequences.

  • Capture setup: 21 GoPro Hero 7 cameras capture synchronized multiview videos at 2028 × 2704 resolution and 30 FPS.Camera parameters are obtained with COLMAP and remain fixed throughout capture.
  • Dataset characteristics: The captured data targets challenging dynamic scenes and view-dependent effects, including specularity, translucency, transparency, scene changes, and changing topology.The collection is intended to support high-quality 4D reconstruction in natural indoor environments.
  • Training sequences: Training uses a 60-second flame-salmon sequence, five 10-second cooking videos, a 25-second indoor video, and additional 5-second outdoor chunks.The sequences span different times, motions, lighting conditions, and keyframe densities.
  • Evaluation protocol: The top-center camera is held out for testing, while other cameras provide training views; cameras with synchronization failures are removed.The experiments use a pinhole camera model because the GoPro linear field of view sufficiently compensates for fisheye effects.
  • Public datasets: The method is additionally validated on public 5-second clips from “Welder,” “Flames,” and “Alexa Meade Face Paint.”These videos use capture setups differing from the authors’ linear-camera system.

C. Importance Sampling Schemes

The paper introduces global-median and temporal-difference importance sampling schemes for dynamic video radiance fields, then combines them to recover sharp detail while smoothing temporal motion. The median-based scheme improves handling of high-frequency moving regions and accelerates convergence toward sharp results.

  • Global Median Maps (DyNeRF-ISG): DyNeRF-ISG samples rays using each ray’s global median value across all timestamps and robust residuals against the resulting median image.The Geman-McLure function uses γ to adjust sensitivity to variance.
  • Global Median Maps (DyNeRF-ISG): A larger γ increases sampling of time-variant regions, while γ approaching zero approximates uniform sampling.The robust function is applied element-wise to residuals.
  • Global Median Maps (DyNeRF-ISG): Using a median image rather than alternatives such as a mean better handles high-frequency moving-region signals and reaches sharp results faster during training.The median image serves as a representative image across time.
  • Temporal Difference (DyNeRF-IST): DyNeRF-IST compares nearby frames within |ti −tj| ≤25, emphasizing pixels with the largest temporal differences while clamping weights with lower-bound α.Small α favors highly dynamic regions, whereas large α gives rays similar importance.
  • Combined Method (DyNeRF-IS⋆): DyNeRF-IS⋆ first uses DyNeRF-ISG for sharp detail recovery and then DyNeRF-IST to smooth the resulting temporal motion.The combination addresses jitter from high-learning-rate ISG and blur from low-learning-rate IST.

D. More Results

DyNeRF outperforms established baselines in quantitative and visual quality, while importance sampling improves convergence and sharp motion recovery. The representation is highly compact, and latent-code length presents a quality–efficiency trade-off with temporal extrapolation remaining difficult.

  • Baseline comparisons: DyNeRF produces more photo-realistic images than MVS, NeuralVolumes, and LLFF across all reported quantitative measures.All neural radiance field baselines and DyNeRF are trained for the same number of iterations.
  • Latent-code size: A latent code length of 8 is insufficient for sharp details, whereas 8,192 is too slow to process; 1,024 balances quality and convergence speed.The ablation uses 60 consecutive frames, and the 8,192-dimensional setting cannot fit the same number of GPU samples.
  • Importance sampling: Importance-sampling variants recover moving flame-gun details in two times fewer iterations than DyNeRF-noIS.Without importance sampling, DyNeRF-noIS remains blurrier and does not reach acceptable visual quality even with extended training.
  • Importance sampling: DyNeRF-ISG, DyNeRF-IST, and DyNeRF-IS⋆ recover sharp details in moving regions, including the torch gun and flames.DyNeRF-ISG and DyNeRF-IST differ in early convergence, while DyNeRF-IS⋆ achieves superior photorealism.
  • Model compression: DyNeRF is more than two orders of magnitude smaller than PNG or JPEG video storage and 50 times smaller than HEVC.The comparison evaluates storage size against raw image formats and the GoPro camera’s default HEVC codec.
  • Limitations: Temporal extrapolation of latent codes cannot directly produce high-quality extrapolated views.The authors identify this as a failure case in manipulating the learned latent codes.

DyNeRF-noIS · DyNeRF-ISG DyNeRF-IST DyNeRF-noIS final

The section analyzes latent-code and network learnability, importance-sampling strategies, view-dependent effects, and latent-code-based 3D video editing in DyNeRF. It also identifies color inconsistency in dark indoor scenes as a limitation requiring calibration.

  • DyNeRF-ISG DyNeRF-IST DyNeRF-noIS final: Figure 11 compares importance sampling strategies over training iterations.
  • DyNeRF-noIS: Learning both keyframe and remaining-frame latent codes together with the network is necessary for sharp, high-quality DyNeRF renderings.Freezing keyframe latent codes produces blurrier results than the all-learnable case.
  • DyNeRF-ISG DyNeRF-IST DyNeRF-noIS final: DyNeRF represents view-dependent effects and motion within one continuous representation, including in dark indoor scenes.
  • DyNeRF-ISG DyNeRF-IST DyNeRF-noIS final: Slight appearance differences between input camera streams can be modeled as view-dependent effects when DyNeRF generates novel views.
  • DyNeRF-ISG DyNeRF-IST DyNeRF-noIS final: Color inconsistency from wide-angle input streams creates artifacts in dark indoor scenes.More careful color calibration or learned color calibration may address this problem, which the authors leave for future work.
  • DyNeRF-ISG DyNeRF-IST DyNeRF-noIS final: DyNeRF’s continuous spatial-temporal dynamic-scene representation supports rendering any view within the inter-The passage introduces 3D video editing via manipulating the latent codes.

NeRF-T DyNeRF† DyNeRF-ISG DyNeRF-IST DyNeRF-IS*

DyNeRF’s hierarchical variants improve qualitative rendering of dynamic details, while the method supports smooth temporal interpolation and bullet-time view manipulation. Rendering remains on par with NeRF, requiring 45 seconds for one 1080p frame in the current implementation.

  • Qualitative comparison: The three hierarchical DyNeRF variants outperform the baseline methods qualitatively on the moving flame gun.DyNeRF-ISG produces sharper details than DyNeRF-IST, while DyNeRF-IST recovers more of the flame.
  • Temporal interpolation: DyNeRF enables smooth interpolation from 30fps to 60fps or 150fps by interpolating latent codes at sub-frame times.This supports rendering slow-motion 3D video at a specified frame rate.
  • Temporal interpolation: DyNeRF can create a bullet-time effect by freezing the latent code at any arbitrary time and manipulating camera views in space.The method can render this effect at arbitrary times.
  • Rendering time: 45 seconds is required to render one 1080p frame using two V-100 GPUs with 16 GB memory in the current, not fully optimized implementation.The rendering time is on par with NeRF because the approaches are structurally similar.
Loading 2103.02597v2…