Source-linked AI summary
Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes
Zhengqi Li, Simon Niklaus, Noah Snavely, Oliver Wang
TL;DR
Dynamic-scene view synthesis from one monocular video remains challenging because moving objects require dense 3D motion and the problem is ill-posed. The paper introduces Neural Scene Flow Fields, which model time-varying appearance, geometry, and motion, and reports substantially better results than the evaluated baselines while retaining clear scope limitations.
Problem
Monocular dynamic-scene view synthesis is ill-posed, and moving-object multiview constraints require dense 3D motion information.
Method
Neural Scene Flow Fields represent dynamic scenes as continuous space-time functions of reflectance, density, and 3D motion, optimized with a scene-flow warping loss.
Results
The method significantly outperforms the evaluated baselines for novel view synthesis and novel view-and-time synthesis in dynamic regions and across the entire scene.
Takeaways & Limitations
The representation enables simultaneous synthesis and interpolation across viewpoint and time for complex dynamic scenes captured by a monocular camera.
Takeaways & Limitations
Training and rendering are slow, each scene requires reconstruction from scratch, unseen content cannot be extrapolated, and quality degrades with longer sequences or extreme motion.
Abstract
from arXiv · showhide
We present a method to perform novel view and time synthesis of dynamic scenes, requiring only a monocular video with known camera poses as input. To do this, we introduce Neural Scene Flow Fields, a new representation that models the dynamic scene as a time-variant continuous function of appearance, geometry, and 3D scene motion. Our representation is optimized through a neural network to fit the observed input views. We show that our representation can be used for complex dynamic scenes, including thin structures, view-dependent effects, and natural degrees of motion. We conduct a number of experiments that demonstrate our approach significantly outperforms recent monocular view synthesis methods, and show qualitative results of space-time view synthesis on a variety of real-world videos.
1 Cornell Tech 2 Adobe Research
Figure 1 demonstrates space-time novel-view synthesis from a single monocular video of a dynamic scene, with results across different view/time configurations and recovered scene geometry.
- The method synthesizes novel views across both spatial viewpoints and time from a single monocular dynamic-scene video.
- The figure shows configurations that fix or interpolate the view and time dimensions.
- The visualization includes recovered scene geometry alongside the video synthesis results.
1. Introduction
The paper addresses novel view and time synthesis for dynamic scenes from monocular video, where prior static-scene and synchronized-multiview assumptions do not hold. It introduces Neural Scene Flow Fields to model time-varying appearance, geometry, and 3D motion while optimizing against monocular observations.
- Most prior novel-view methods assume static scenes or multiple synchronized views, unlike Internet videos containing dynamic content recorded by one camera.
- Monocular dynamic-scene synthesis is ill-posed because multiple scene configurations can produce the same observed image sequences.
- Moving-object multiview constraints require dense 3D motion information for all scene points, known as scene flow.
- Neural Scene Flow Fields represent dynamic scenes as continuous space-time functions outputting reflectance, density, and 3D scene motion.
- A scene-flow warping loss enforces temporal consistency with input views, while dense 3D flow represents sharp image-space motion discontinuities.
- The method analyzes motion-disocclusion ambiguity, uses data-driven priors to avoid local minima, and combines static and dynamic representations for higher-quality rendering.
2. Related Work
Prior work spans explicit and implicit novel-view representations, monocular image-based synthesis, 2D frame interpolation, space-time rendering, and dynamic-scene reconstruction. These approaches variously rely on static-scene assumptions, multiple synchronized views, RGBD input, sparse geometry, or restricted motion.
- Novel view synthesis: Novel-view synthesis methods use explicit geometry, light fields, multiplane images, learned features, voxel representations, or neural radiance fields.
- Novel view synthesis: Single-image novel-view methods predict depth, learned features, or layered representations but operate per frame and cannot leverage repeated observations across views.
- Novel time synthesis: Most video-time interpolation methods operate in 2D by blending images or warping frames and therefore cannot synthesize novel spatial views.
- Space-time view synthesis: Methods addressing time-varying appearance from illumination generally assume static scenes and treat moving objects as outliers.
- Space-time view synthesis: Prior 3D-motion methods commonly require synchronized multiview videos and have limited ability to model complicated scene geometry.
- Dynamic scene reconstruction: Dynamic-scene reconstruction systems often require RGBD input, recover only sparse geometry, or rely on handcrafted priors for piecewise-rigid decomposition.
3. Approach
The approach extends neural radiance fields into a time-dependent representation that predicts appearance, density, dense 3D scene flow, and disocclusion weights. It optimizes temporal consistency on monocular video while combining dynamic and static representations to improve rendering quality.
- 3.1. Neural scene flow fields for dynamic scenes: Neural scene flow fields extend static radiance fields with time, forward and backward dense 3D motion, and disocclusion weights.The model assumes motion between observed time instances is linear.
- 3.2. Optimization: Temporal photometric consistency warps sampled 3D points between neighboring frames before volume rendering and compares the warped view with the target image.The loss uses scene flow to undo motion between times and minimizes the difference between warped and ground-truth views.
- 3.2. Optimization: Disocclusion weights modulate temporal and cycle-consistency losses because scene flow is ambiguous where motion reveals or occludes 3D regions.An ℓ1 regularizer encourages weights near one, and the resulting weighting improves rendering near motion boundaries.
- 3.2. Optimization: Data-driven geometric and single-view depth priors help avoid poor local minima, but their influence is linearly decayed to zero because the predictions are noisy.The geometric prior uses reprojection error against derived 2D optical flow to improve correspondence association.
- 3.3. Integrating a static scene representation: A jointly optimized static representation is blended with the dynamic representation so static regions can use higher-fidelity rendering while moving regions rely more on the dynamic model.The blending weight is learned as an unsupervised 3D field.
- 3.4. Space-time view synthesis: Direct time-index interpolation cannot represent intermediate time-varying geometry and produces linearly blended frames instead of motion-consistent interpolation.The limitation is illustrated by blending artifacts in the novel-time synthesis comparison.
4. Experiments
The experiments evaluate monocular dynamic-scene view synthesis against single-view and multi-view baselines using quantitative and qualitative comparisons. The method performs strongly for novel views and joint novel view-time synthesis, while its components and handling of complex structures are examined through ablations and visual results.
- Experimental setup: Rendering quality is measured with SSIM, PSNR, and LPIPS over the full scene and dynamic regions only.The comparisons include single-view methods such as MPIs and SinSyn, plus multi-view baselines.
- Experimental setup: The Dynamic Scenes evaluation uses 8 scenes recorded by 12 synchronized cameras, with 24 training frames and 11 held-out images per time instance.Camera poses are estimated with COLMAP and fixed during optimization.
- Qualitative evaluation: Qualitative results show fewer artifacts and better reconstruction of disocclusions, thin structures, and complex scene regions than competing methods.The comparisons include Dynamic Scenes and in-the-wild monocular videos with jumping, running, dancing, and occlusions.
- Quantitative evaluation: The method without its static representation already outperforms single-view and multi-view baselines for both dynamic regions and the entire scene.NeRF has the second-best whole-scene performance but cannot model scene dynamics.
- Quantitative evaluation: Adding the static scene representation improves overall rendering quality by more than 30%.The static component leverages global multi-view information from rigid regions.
- Quantitative evaluation: For novel view and time synthesis, the method significantly outperforms all baselines in both dynamic regions and the entire scene.Baselines combine view synthesis with 2D frame interpolation, while NeRF is evaluated directly at novel space-time views.
- Ablation study: The full system performs best in ablations that remove its losses, priors, disocclusion weight fields, or static representation.The ablation isolates temporal, depth, geometry, scene-flow, disocclusion, and static components.
5. Discussion
The discussion identifies computational, data, motion, and ambiguity limitations of monocular space-time view synthesis. The method cannot extrapolate unseen content and degrades with longer sequences, extreme motion, or degenerate motion configurations.
- Limitations: Training and rendering times remain high even at limited resolutions, and each scene must be reconstructed from scratch.These constraints are described as limitations similar to NeRF.
- Limitations: The method cannot extrapolate content unseen in the training views.This limitation is illustrated in Figure 10(a).
- Limitations: Rendering quality degrades as sequence length increases with the default parameter count or when object motion is extreme.Most sequences were trained for 1–2 seconds, and extreme-motion cases are illustrated in Figure 10(b–c).
- Limitations: Object and camera motions near degenerate configurations, such as colinearity, can lead optimization to an incorrect local minimum.The limitation is attributed to ambiguity in the motion configuration.
- Conclusion: The conclusion reports compelling space-time synthesis for natural in-the-wild motion while targeting future high-resolution views, larger scenes, and larger viewpoint changes.The stated future directions extend beyond the demonstrated scope.