Source-linked AI summary
Dynamic View Synthesis from Dynamic Monocular Video
Chen Gao, Ayush Saraf, Johannes Kopf, Jia-Bin Huang
TL;DR
Single monocular videos provide only one 2D observation per time step, making dynamic-scene reconstruction highly ill-posed and allowing infinitely many matching solutions. The paper combines static and dynamic NeRFs with unsupervised blending and scene-flow-based regularization, obtaining favorable results against state-of-the-art methods while retaining acknowledged failure cases.
Problem
A single 2D observation at each time step leaves infinitely many possible dynamic-scene reconstructions, while directly adding time to NeRF produces low visual quality.
Method
The method jointly trains a static NeRF excluding dynamic pixels and a dynamic NeRF that predicts scene flow, warped radiance fields, blending weights, and regularization for temporal consistency.
Results
The proposed method produces favorable visual results and compares favorably against state-of-the-art dynamic view-synthesis algorithms.
Takeaways & Limitations
Scene-flow-based temporal consistency and rigidity regularization are identified as key design choices for better visual results in monocular dynamic view synthesis.
Takeaways & Limitations
Dynamic view synthesis remains challenging, with several failure cases reported by the authors.
Abstract
from arXiv · showhide
We present an algorithm for generating novel views at arbitrary viewpoints and any input time step given a monocular video of a dynamic scene. Our work builds upon recent advances in neural implicit representation and uses continuous and differentiable functions for modeling the time-varying structure and the appearance of the scene. We jointly train a time-invariant static NeRF and a time-varying dynamic NeRF, and learn how to blend the results in an unsupervised manner. However, learning this implicit function from a single video is highly ill-posed (with infinitely many solutions that match the input video). To resolve the ambiguity, we introduce regularization losses to encourage a more physically plausible solution. We show extensive quantitative and qualitative results of dynamic view synthesis from casually captured videos.
1. Introduction
Dynamic view synthesis seeks photorealistic views of dynamic scenes from arbitrary viewpoints and times, but single-video reconstruction is difficult. This paper addresses the challenge with separate static and dynamic models, regularization losses, and unsupervised blending.
- Dynamic view synthesis enables arbitrary-viewpoint and time-based viewing of dynamic scenes for applications including sports replay, cinematic effects, virtual reality, and teleportation.
- Existing systems often require expensive multi-camera capture, while monocular approaches remain limited in scope or rely on estimated depth for image warping and blending.
- The method jointly trains a time-invariant static model and a time-varying dynamic model, learning to blend their results without supervision.
- Regularization losses are designed to resolve ambiguities in learning dynamic radiance fields from monocular video.
- The model produces favorable results compared with state-of-the-art algorithms on the Dynamic Scenes Dataset.
2. Related Work
Prior view-synthesis methods use image-based, geometric, or implicit representations, but dynamic video introduces substantially greater capture, reconstruction, compression, and rendering challenges. Recent dynamic methods extend neural representations through scene decomposition, motion fields, or spatiotemporal radiance fields.
- Light fields, Lumigraphs, explicit geometric proxies, local warps, and learning-based methods address novel-view synthesis from relatively few or many posed images.
- Single-image view-synthesis methods use representations such as depth, multiplane images, or layered depth images, whereas this work targets dynamic scenes from one video.
- Dynamic video view synthesis is harder than image view synthesis, and existing systems often target specific domains or require substantial capture infrastructure.
- Implicit neural representations use continuous, differentiable MLPs to map coordinates to quantities such as occupancy, signed distance, volume density, or color.
- Directly adding time to MLP inputs can entangle shape and motion, motivating separate static and dynamic models for dynamic-scene representation.
- Recent dynamic NeRF methods either deform canonical templates across frames or directly estimate 4D spatiotemporal radiance fields; this work adopts the latter approach.
3. Method
The method represents static and dynamic scene components with separate NeRFs, blends them using learned weights, and regularizes dynamic geometry to address monocular-video ambiguity. Scene flow provides temporal multi-view constraints, while motion and smoothness priors support novel-view rendering.
- The model takes a monocular video and foreground masks, then targets free-viewpoint rendering at arbitrary views and input times.
- Directly adding time to NeRF is ill-posed because one 2D observation per time step admits infinitely many solutions and produces low visual quality.
- Static NeRF: Static NeRF excludes dynamic pixels during training, allowing the background’s structure and appearance to be reconstructed without moving-object conflicts.
- Dynamic NeRF: Dynamic NeRF predicts time-dependent density, color, blending weights, and forward/backward scene flow, then warps neighboring radiance fields into the current time.
- Dynamic NeRF: Warping neighboring-time radiance fields supplies up to three 2D observations for each 3D point, creating a multi-view constraint despite monocular input.
- Regularization Losses for Dynamic NeRF: Regularization combines optical-flow motion matching with slow, temporally smooth scene flow to reduce, but not fully eliminate, ambiguity.
4. Experimental Results
The method is evaluated on monocular videos synthesized from a multi-camera Dynamic Scene Dataset, with quantitative, qualitative, comparative, ablation, and failure-case analyses. Regularization losses improve geometric plausibility, motion handling, and static-background reconstruction, while non-rigid deformation and inaccurate optical flow remain challenging.
- Evaluation: The Dynamic Scene Dataset contains 9 sequences captured simultaneously by 12 cameras, with monocular inputs formed by sampling different cameras at successive time steps.Each input consists of twelve frames, and camera poses and scene bounds are estimated with COLMAP.
- Quantitative evaluation: The method compares favorably against state-of-the-art algorithms using PSNR and LPIPS on synthesized views rendered from a fixed viewpoint across time.The evaluation uses official implementations for competing methods, although one baseline requires per-sequence hyper-parameter tuning.
- Qualitative evaluation: The learned neural implicit representation synthesizes novel views that were never observed during training.Qualitative results are presented in Figure 8 and supplemented with video comparisons.
- Comparisons: Time-varying blending weights and rigidity regularization improve separation of static regions and produce cleaner backgrounds on large-motion sequences.Rigidity regularization encourages zero scene flow in rigid regions, strengthening the multi-view constraint.
- Failure cases: The method does not handle non-rigid deformation well and produces artifacts when optical-flow estimates are inaccurate.These failure cases are illustrated in Figure 11.
- Ablation study: Depth order loss resolves incorrect relative-depth solutions that can fit input images but produce artifacts at novel views.With L_depth, the dynamic NeRF renders more plausible content.
- Ablation study: Motion regularization produces plausible results by constraining scene flow to be temporally and spatially smooth and consistent, avoiding blurry synthesis.The regularization also helps handle videos with large object motion.
5. Conclusions
The paper presents a single-monocular-video dynamic view-synthesis algorithm whose scene-flow regularization addresses ambiguity in modeling dynamic scenes from one observation at each time. The proposed temporal-consistency and rigidity losses produce better visual results and compare favorably with state-of-the-art methods.
- Conclusions: The algorithm models dynamic scenes from a single monocular video using scene-flow-based regularization to enforce temporal consistency and reduce ambiguity.The contribution targets the one-observation-per-time setting.
- Conclusions: Scene-flow-based 3D temporal consistency and rigidity regularization are identified as key design choices for better visual results.The paper validates these choices and reports favorable comparisons with state-of-the-art methods.