Source-linked AI summary
Novel View Synthesis of Dynamic Scenes with Globally Coherent Depths from a Monocular Camera
Jae Shin Yoon, Kihwan Kim, Orazio Gallo, Hyun Soo Park, Jan Kautz
TL;DR
Dynamic-scene view synthesis from a moving monocular camera is difficult because local motion breaks standard multiview geometry. The paper fuses complete but view-variant single-view depth with incomplete but view-invariant multiview depth, then uses DeepBlender to render missing regions and refine novel views. Evaluations on real-world dynamic scenes report effective synthesis and performance exceeding existing approaches, while limitations remain for cluttered scenes, failed calibration, and failed foreground masks.
Problem
Dynamic contents violate the geometric assumptions used by many view-synthesis methods, while reconstructing their 3D geometry from a moving monocular camera is fundamentally ill-posed.
Method
DFNet learns scale correction and fuses DSV with DMV to produce complete, view-invariant depth, while DeepBlender combines foreground, background, and missing regions for photorealistic rendering.
Results
The method effectively generates unseen images, and its rendering network outperforms existing synthesis approaches quantitatively and qualitatively across real-world dynamic scenes.
Takeaways & Limitations
Combining single-view and multiview depth supports coherent dynamic-scene geometry and novel-view synthesis from monocular-camera imagery.
Takeaways & Limitations
Performance is limited by cluttered depth discontinuities, scenes where camera calibration fails, and completely failed foreground masks that produce artifacts.
Abstract
from arXiv · showhide
This paper presents a new method to synthesize an image from arbitrary views and times given a collection of images of a dynamic scene. A key challenge for the novel view synthesis arises from dynamic scene reconstruction where epipolar geometry does not apply to the local motion of dynamic contents. To address this challenge, we propose to combine the depth from single view (DSV) and the depth from multi-view stereo (DMV), where DSV is complete, i.e., a depth is assigned to every pixel, yet view-variant in its scale, while DMV is view-invariant yet incomplete. Our insight is that although its scale and quality are inconsistent with other views, the depth estimation from a single view can be used to reason about the globally coherent geometry of dynamic contents. We cast this problem as learning to correct the scale of DSV, and to refine each depth with locally consistent motions between views to form a coherent depth estimation. We integrate these tasks into a depth fusion network in a self-supervised fashion. Given the fused depth maps, we synthesize a photorealistic virtual view in a specific location and time with our deep blending network that completes the scene and renders the virtual view. We evaluate our method of depth estimation and view synthesis on diverse real-world dynamic scenes and show the outstanding performance over existing methods.
1. Introduction
The paper targets novel view synthesis for dynamic scenes from a moving monocular camera, where standard geometric assumptions fail for moving content. It combines complementary depth cues and learns coherent geometry before refining the rendered view.
- Motivation: Existing view-synthesis methods largely target static scenes because dynamic people, pets, and vehicles violate their geometric assumptions.Dynamic content is treated as outliers despite its importance in videography.
- Motivation: Multi-view stereo provides incomplete but view-invariant static geometry, while single-view depth provides complete but view-variant dynamic geometry.These complementary cues support consistent static reconstruction and locally consistent foreground scene flow.
- Approach: The method learns a nonlinear scale correction that upgrades single-view geometries into a coherent 4D reconstruction with simple, locally consistent foreground motion.The motion description is chosen to be slow and smooth, minimizing stereoscopic disparity from novel views.
- Approach: DFNet fuses images, DSV, and DMV into complete, view-invariant depth using self-supervised static alignment, relative-depth consistency, and minimal scene-flow signals.The rendering network then uses geometrically consistent depths to synthesize a photorealistic image despite missing data.
- Results: The method generates unseen images effectively, while its rendering network blends foreground and background and outperforms existing synthesis approaches quantitatively and qualitatively.The paper also contributes a real-world dynamic-scene dataset and benchmark examples.
2. Related Work
Related work connects novel view synthesis to multiview 3D reconstruction and reviews depth and scene-reconstruction strategies for handling incomplete or dynamic geometry.
- Novel View Synthesis: Novel view synthesis transports pixels across views through scene geometry; static scenes commonly use calibrated-camera correspondences and triangulation.The broader review covers view synthesis, depth estimation, and scene reconstruction.
- Depth Fusion: DFNet’s self-supervised formulation enforces background consistency with DMV, relative consistency with DSV, 3D scene flow, and spatial regularity.The network fuses DSV and DMV with the image to predict complete, view-invariant depth.
- Scene Reconstruction: Prior work completes reconstruction through independent image-region models, spatial and temporal priors, dense correspondences, and human-specific constraints.These approaches include shape bases, local patches, complete 4D reconstruction, and monocular markerless motion capture.
3. Approach
The approach reconstructs dynamic scenes by fusing complementary depth cues into globally coherent geometry, then warps and blends foreground and background content into a virtual view.
- Dynamic scene view synthesis: The method casts synthesis as image warping from source views to a virtual view using an underlying 4D reconstruction.The rendering function then refines the warped images to complete the synthesis.
- Dynamic scene view synthesis: Dynamic-scene warping incorporates time-varying geometry, so a source image cannot generally be synthesized from another source image through ordinary static-scene warping.For a moving monocular camera, the view itself is a function of time.
- Globally coherent depth: A nonlinear scale correction function upgrades single-view depths into coherent geometry by matching static regions to DMV and preserving relative depth in dynamic regions.Multi-scale neighboring pixels constrain local and global relative gradients, while slow and smooth 3D motion provides an additional constraint.
- Globally coherent depth: The depth fusion network combines image inputs, complete but view-variant DSV, and incomplete but view-invariant DMV to predict complete, view-invariant depth.Its self-supervision aligns static DSV regions with DMV, preserves dynamic relative depth, and encourages minimal, locally consistent scene flow.
- Dynamic scene view synthesis: The rendering stage constructs a global background from warped static views and combines it with warped dynamic content using DeepBlender.DeepBlender predicts a blending residual that fills unseen regions and refines artifacts, using reconstruction and adversarial losses for photorealistic synthesis.
4. Implementation Details
The implementation pre-trains DFNet and DeepBlender on synthetic data, then uses consistency checks and depth refinement during image warping.
- Pre-training: DFNet is pre-trained on synthetic data with ground-truth optical flow, depth, and foreground masks, while simulated depth removal and noise approximate real data.Depth noise is added with 5% tolerance of the variance at every training iteration.
- Pre-training: DeepBlender is pre-trained using synthetic residuals created from seams, holes, and image-boundary removals in a video object segmentation dataset.The synthetic residuals target missing content around foreground regions and image boundaries.
- Warping refinement: Bidirectional warping consistency is checked to prevent pixel holes, and bilateral weighted median filters refine depth after each image warp.
5. Experiments
The experiments evaluate depth estimation and novel view synthesis on diverse real-world dynamic scenes, using quantitative metrics, qualitative comparisons, and ablations. Results show that depth fusion and refinement improve geometric consistency and visual plausibility, while mask failures and challenging scene configurations remain limitations.
- Experimental setup: The evaluation covers diverse dynamic scenes and includes depth estimation, view synthesis, qualitative comparisons, and ablation studies.Depth estimation uses metric-scale RMSE; view synthesis uses optical-flow magnitude and perceptual similarity against ground truth.
- Depth estimation: Depth fusion is essential because MonoDepth has biased depth ranges, while Sparse2Dense and RMVSNet fail to recover reliable dynamic foreground geometry.The experiments support combining single-view and stereo-based geometry to obtain complete, globally coherent depth maps.
- Ablation study: The MVS-alignment loss Lg is the most critical self-supervision signal because accurate static depth provides a reference for predicting missing regions.Single-view depth further improves depth accuracy around dynamic contents when guided by accurate relative depths.
- Limitations: Small mask-detection mistakes have little effect, but completely failed foreground masks produce object fragmentation and afterimages.The reported artifacts are illustrated in Figure 7 and define a practical boundary for the rendering pipeline.
- Novel view synthesis: The proposed method achieves the smallest average perceptual distance, 0.15, for novel view synthesis with DFNet+B3W.B3W produces flying-pixel noise with incorrect depths, whereas ASAPW produces image distortion; accurate depth favors pixel-wise warping.
- Novel view synthesis: The proposed method also obtains the best flow evaluation at 5.3 pixels, compared with 6.8 pixels for MVS+B3W and 10.8 pixels for MonoDepth+B3W.DFNet+B3W-Deepblender has higher flow error because holes near image boundaries and dynamic contents confuse the flow estimator.
6. Conclusion
The paper addresses ill-posed monocular dynamic-scene view synthesis by learning to correct single-view depth and produce geometrically coherent novel views. It then uses the computed depth to synthesize images with foreground, background, and missing regions combined by DeepBlender.
- The central challenge is reconstructing dynamic contents for geometrically coherent view synthesis from a monocular camera.
- The method learns a scale correction function that matches single-view depth to multi-view depth for static contents while producing locally consistent scene motion.
- DeepBlender combines foreground, background, and missing regions to synthesize a novel view from the computed depth.
- Evaluations show that the proposed method applies to daily scenarios captured from a monocular camera.