PaperlayerView exact revision on arXiv

Source-linked AI summary

Fast View Synthesis of Casual Videos with Soup-of-Planes

Yao-Chih Lee, Zhoutong Zhang, Kevin Blackburn-Matzen, Simon Niklaus, Jianming Zhang, Jia-Bin Huang, Feng Liu

arXiv:2312.02135v2cs.CV

TL;DR

Novel view synthesis from monocular in-the-wild video must handle scene dynamics and weak parallax, while NeRF-based methods can be slow to train and render. The paper uses a hybrid explicit representation for static and dynamic content, reporting comparable quality to NeRF-based methods with over 100× faster training and rendering.

  • Problem

    NeRF-based dynamic view synthesis can take one or more days to train and remains challenging to render in real time, while monocular videos provide limited parallax for 3D Gaussian methods.

    Source: §1 · p. 2

  • Method

    The method combines a global soup of oriented planes, augmented with spherical harmonics and displacements, for static content with per-frame point clouds for dynamic content.

    Source: §1 · p. 2, §5 · p. 14

  • Results

    The method achieves rendering quality comparable to NeRF-based dynamic synthesis algorithms while being over 100× faster for both training and rendering.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The explicit hybrid representation supports real-time rendering of novel views from casual videos.

    Source: §1 · p. 3, §5 · p. 14

    The approach may fail when preprocessed video depth and pose are severely inaccurate, producing poor plane initialization and blurry static reconstruction.

    Source: §4.6 · p. 13

Abstract

from arXiv · show

Novel view synthesis from an in-the-wild video is difficult due to challenges like scene dynamics and lack of parallax. While existing methods have shown promising results with implicit neural radiance fields, they are slow to train and render. This paper revisits explicit video representations to synthesize high-quality novel views from a monocular video efficiently. We treat static and dynamic video content separately. Specifically, we build a global static scene model using an extended plane-based scene representation to synthesize temporally coherent novel video. Our plane-based scene representation is augmented with spherical harmonics and displacement maps to capture view-dependent effects and model non-planar complex surface geometry. We opt to represent the dynamic content as per-frame point clouds for efficiency. While such representations are inconsistency-prone, minor temporal inconsistencies are perceptually masked due to motion. We develop a method to quickly estimate such a hybrid video representation and render novel views in real time. Our experiments show that our method can render high-quality novel views from an in-the-wild video with comparable quality to state-of-the-art methods while being 100x faster in training and enabling real-time rendering.

1 Introduction

Monocular video view synthesis must handle scene dynamics and weak parallax, while existing NeRF-based methods can be slow to train and render. The paper proposes a hybrid explicit representation and reports comparable quality to NeRF-based methods with over 100× faster training and rendering.

1 Introduction

  • NeRF-based dynamic view synthesis can require one or more days of training and remains challenging to render in real time.Source: §1 · p. 2
  • 3D Gaussian Splatting can reveal floaters and artifacts in novel views where monocular video provides weak parallax.Source: §1 · p. 2
  • A hybrid representation uses global oriented planes for static content and per-frame point clouds for dynamic content, extending the planes with spherical harmonics and displacement maps.The planes capture view-dependent effects and complex non-planar geometry; neighboring dynamic frames are blended to support temporal coherence.Source: §1 · p. 2
  • Small temporal inconsistencies in the dynamic representation are described as usually imperceptible because of motion, though the approach remains prone to temporal issues.Source: §1 · p. 3
  • The method achieves rendering quality comparable to NeRF-based dynamic synthesis algorithms while being over 100× faster for both training and rendering.Per-video optimization takes 15 minutes on a single GPU.Source: §1 · p. 3
  • The paper develops an optimization method and loss functions to estimate the hybrid video representation from a monocular video.Its listed contributions include the representation and per-video optimization algorithm.Source: §1 · p. 3

2 Related Work

Prior dynamic-scene synthesis often relies on simultaneous multi-camera input, while monocular-video approaches face depth inconsistency and computational costs. Explicit representations offer faster alternatives, but existing point- and plane-based approaches have limitations that motivate the paper’s oriented-plane representation.

2.1 Dynamic-scene view synthesis

  • Monocular-video methods face globally inconsistent video depth, while many dynamic-scene methods reconstruct 4D scenes from simultaneous multi-camera captures that are impractical for casual applications.Neural rendering methods can achieve decent quality but have high computational costs for per-scene training and rendering.Source: §2.1 · pp. 3–4

2.2 View synthesis with explicit representations

  • Explicit point-based methods can fail with little parallax, while fronto-parallel MPIs restrict viewpoint changes and oriented feature planes require time-consuming optimization.The paper instead fits non-neural RGBA soup-of-oriented-planes to scene surfaces; Fig. 3 also shows per-frame point clouds for dynamic content.Source: §2.2 · p. 4, Figure 3 · p. 5

3 Method

The method combines a global static scene representation with per-frame dynamic content, then optimizes and composites them to render novel views. Its static planes model view-dependent appearance and non-planar surfaces, while dynamic content is represented and warped frame by frame.

3 Method

  • A global static soup of oriented planes and per-frame dynamic point clouds handle background and moving content separately for novel-view rendering.The input video is preprocessed for depth, camera poses, and dynamic masks before the representations are rendered and composited.Source: §3 · pp. 5–6, §3.1 · p. 6

3.2 Extended Soup of Planes for Static Content

  • Plane textures use spherical harmonics for view-specific color and displacement, shifting color and transparency to better represent non-planar surfaces.The planes are fitted to a static scene point cloud using point-to-plane distance, normal alignment, and a compact-area penalty.Source: §3.2 · pp. 6–7

3.3 Consistent Dynamic Content Synthesis

  • Dynamic appearance and masks are forward-splatted to the target view, then blended over static content using a soft mask that considers depth order.Neighboring-frame colors and masks are blended to address temporal inconsistencies from noisy precomputed masks.Source: §3.3 · p. 8

3.4 Optimization

  • The hybrid representation is jointly optimized with reconstruction objectives and regularizers, including photometric, perceptual, mask, and depth-alignment losses.Depth smoothness and plane-transparency smoothness are also used; scene-flow regularization is disabled by default because it can compromise quality under its motion assumption and slows training.Source: §3.4 · pp. 8–9
  • Dynamic-mask optimization combines sparsity penalties with binary cross-entropy, while depth alignment keeps rendered depth consistent with the precomputed geometry prior.The method also applies multi-scale depth smoothness and plane-transparency smoothness regularization.Source: §3.4 · p. 9

4 Experimental Results

Across the evaluated videos, the method combines competitive novel-view quality with fast training and rendering, while ablations and in-the-wild examples illustrate its strengths and limitations.

4.3 Speed Comparison

  • The method reaches the second-best LPIPS score on the NVIDIA dataset while providing real-time rendering, and is over 100× faster than several NeRF baselines in training and rendering.The speed comparison focuses on representation training and inference rendering, not the potentially lengthy preprocessing stage.Source: §4.1 · p. 10, Table 1 · p. 12, §4.3 · p. 12

4.5 Novel view results on in-the-wild videos

  • In diverse in-the-wild examples, scene optimization mitigates imperfect depth in complex structures, producing a more coherent novel view than naïve depth warping.The comparison includes strong parallax and complex backgrounds, including a fence and a building behind it.Source: §4.5 · p. 13, Figure 9 · p. 14

4.6 Limitations

  • The method may fail with severely inaccurate depth or pose, subtle motion that defeats segmentation, long volumetric 360 scenes, or incomplete foreground dynamics in distant views.Per-frame dynamic representations also cannot synthesize sub-timesteps.Source: §4.6 · pp. 13–14, Figure 10 · p. 14

5 Conclusions

The paper presents an efficient casual-video view-synthesis method using a hybrid representation: soup of planes for static scenes and per-frame point clouds for dynamic content. It reports novel-view quality comparable to state-of-the-art NeRF-based approaches with faster training and rendering.

5 Conclusions

  • The method generates high-quality novel views with quality comparable to state-of-the-art NeRF-based approaches while being faster to train and render.It uses a hybrid static-dynamic representation optimized from an in-the-wild video.Source: §5 · p. 14
  • A soup of planes represents the global static scene, augmented with spherical harmonics and displacements for view-dependent effects and complex non-planar geometry.Source: §5 · p. 14
  • Per-frame point clouds represent dynamic content for efficiency, and a tailored optimization method with loss functions fits the hybrid representation from an in-the-wild video.Source: §5 · p. 14

Related papers

Loading 2312.02135v2…