PaperlayerView exact revision on arXiv

Source-linked AI summary

Stable View Synthesis

Gernot Riegler, Vladlen Koltun

arXiv:2011.07233v2cs.CV

TL;DR

Photorealistic view synthesis seeks realistic images from freely chosen viewpoints, but requires a representation that supports stable target-view rendering. SVS aggregates source-image features on a geometric scaffold with differentiable, end-to-end modules, and outperforms state-of-the-art methods across three real-world datasets. Its scope remains limited to static scenes.

  • Problem

    Photorealistic view synthesis must produce realistic images while allowing exploration from any physically realizable viewpoint.

  • Method

    SVS aggregates deep source-image features adaptively on a geometric scaffold through a differentiable set network and renders target views end-to-end.

  • Results

    SVS convincingly outperforms state-of-the-art methods on three diverse real-world datasets, including up to 10 absolute percentage-point lower LPIPS error on Tanks and Temples.

  • Takeaways & Limitations

    SVS sets a new state of the art for photorealistic view synthesis on large-scale real-world scenes.

  • Takeaways & Limitations

    Like most recent view-synthesis work, SVS handles only static scenes and does not support interactive manipulation.

Abstract

from arXiv · show

We present Stable View Synthesis (SVS). Given a set of source images depicting a scene from freely distributed viewpoints, SVS synthesizes new views of the scene. The method operates on a geometric scaffold computed via structure-from-motion and multi-view stereo. Each point on this 3D scaffold is associated with view rays and corresponding feature vectors that encode the appearance of this point in the input images. The core of SVS is view-dependent on-surface feature aggregation, in which directional feature vectors at each 3D point are processed to produce a new feature vector for a ray that maps this point into the new target view. The target view is then rendered by a convolutional network from a tensor of features synthesized in this way for all pixels. The method is composed of differentiable modules and is trained end-to-end. It supports spatially-varying view-dependent importance weighting and feature transformation of source images at each point; spatial and temporal stability due to the smooth dependence of on-surface feature aggregation on the target view; and synthesis of view-dependent effects such as specular reflection. Experimental results demonstrate that SVS outperforms state-of-the-art view synthesis methods both quantitatively and qualitatively on three diverse real-world datasets, achieving unprecedented levels of realism in free-viewpoint video of challenging large-scale scenes. Code is available at https://github.com/intel-isl/StableViewSynthesis

1. Introduction

Stable View Synthesis targets photorealistic rendering from freely chosen physically realizable viewpoints. It builds a geometric scene representation, aggregates source-image features for each target ray, and reports state-of-the-art results across three real-world datasets.

  • Photorealistic view synthesis must support both images indistinguishable from reality and movement through physically realizable viewpoints.
  • SVS constructs a 3D geometric scaffold from source images using structure-from-motion, multi-view stereo, and meshing, then maps deep image features onto it.
  • For each target pixel, SVS maps the pixel onto the scaffold, aggregates visible source-ray features with a differentiable geometry-aware module, and renders the resulting tensor with a convolutional network.
  • End-to-end differentiable aggregation uses all input rays without heuristic selection and supports temporal stability for smoothly moving viewpoints.
  • On FVS and DTU, SVS outperforms prior methods across reported metrics, with DTU LPIPS errors of 4.5% in extrapolation and 1.6% in interpolation.

2. Related Work

SVS builds on geometric-scaffold and neural view-synthesis methods while replacing heuristic source-image selection with differentiable aggregation of available information.

  • Earlier image-based rendering methods established foundational approaches, while deep learning later increased flexibility and realism.
  • Related geometric-scaffold methods encode source images into features and warp or blend them for target-view synthesis.
  • Unlike FVS, SVS does not heuristically select relevant source images before synthesizing the target feature tensor.
  • SVS instead processes all available information with permutation-invariant set operators in an entirely differentiable, end-to-end trainable pipeline.
  • Other view-synthesis families use planesweep volumes, multi-plane images, or volumetric representations for rendering novel views.

3. Overview

SVS uses source images and camera information to construct a geometric scaffold, attach learned features to visible surface points, and synthesize target-view features for convolutional rendering.

  • Source images and camera parameters erect a geometric scaffold Γ that supports SVS’s on-surface feature representation and target-view synthesis.
  • Preprocessing: Preprocessing estimates camera intrinsics and poses with structure-from-motion, computes depthmaps with multi-view stereo, and fuses them into a point cloud.
  • Preprocessing: Surface reconstruction converts the fused point cloud into a 3D mesh Γ; experiments use COLMAP, although other SfM and MVS pipelines are compatible.
  • Feature encoding: A convolutional network encodes each source image I_n into a feature tensor F_n containing one feature vector per image pixel.
  • View synthesis: For each target pixel, SVS back-projects onto Γ, gathers features from images where the point is visible, and applies a differentiable set network conditioned on target direction u.
  • Relation to prior representations: SVS differs from neural point-feature and mesh-texture methods by aggregating features from source images rather than fitting per-scene features from random initialization.

4. Feature Processing and Aggregation

SVS encodes source images into feature tensors, projects visible features onto a geometric scaffold, and aggregates them into target-view features before convolutional rendering. Its aggregation is differentiable, order-invariant, direction-conditioned, and designed to handle incomplete geometry and missing source correspondences.

  • Image encoding: A U-Net-based encoder converts each source image I_n into a feature tensor F_n.The encoder uses an ImageNet-pretrained ResNet18 in its encoder and nearest-neighbor upsampling with skip connections in its decoder.
  • On-surface aggregation: For each scaffold point x, SVS samples feature vectors f_k and viewing directions v_k from source images in which x is visible.Features are obtained by projecting x into each source image and bilinearly interpolating the corresponding encoded feature tensor.
  • On-surface aggregation: The aggregation function φ_aggr produces a target feature g(x, u) from source tuples {(v_k, f_k(x))} and target direction u.It must be differentiable and process any number of input features in any order; the paper explores differentiable set operators.
  • Aggregation designs: SVS can use direction-aligned weighted averaging, PointNet-style MLP aggregation, or graph attention over source-view features.The PointNet formulation concatenates source and target directions with each feature before per-feature transformation and aggregation.
  • Rendering: Target-ray features are computed for every output pixel and assembled into a feature tensor G, which a convolutional renderer maps to the output image.The renderer uses residual U-Nets to regularize the feature map, address scale and exposure differences, and inpaint missing regions.

5. Training

SVS is trained with perceptual supervision using sampled scenes and source images, then can be adapted to new scenes through network or scene fine-tuning. Scene fine-tuning additionally optimizes mutable source-image parameters to harmonize cross-image inconsistencies while preserving the ground-truth image.

  • Scene-agnostic training: Training samples a scene, designates one source image as ground truth, and uses a subset of remaining images as inputs.The three networks φ_enc, φ_aggr, and φ_render are trained end-to-end with a perceptual loss.
  • Network fine-tuning: Scene-agnostic training produces a general network that can be applied to new scenes without retraining or fine-tuning.The paper notes that target scenes may differ substantially from training scenes, motivating additional adaptation strategies.
  • Scene fine-tuning: Scene fine-tuning optimizes both network parameters and source-image parameters initialized from the actual source images.The source images become mutable inputs, while the ground-truth image remains unaltered in the loss.
  • Scene fine-tuning: Optimizing source-image parameters can harmonize inconsistencies from autoexposure, motion blur, and other image-specific aberrations.This procedure changes the encoder input from image content to an index into a pool of trainable source-image parameters.
  • Scene-agnostic training: The perceptual loss compares the rendered output with immutable ground-truth images, preventing a trivial uniform-color solution.The optimization can modify perceived source inputs while the target images used in the loss remain unchanged.

6. Evaluation

SVS is evaluated through controlled architectural experiments and comparisons with prior view-synthesis methods on Tanks and Temples, FVS, and DTU. It achieves strong quantitative and qualitative results, including state-of-the-art LPIPS performance on DTU.

  • Evaluation setup: The evaluation uses 15 Tanks and Temples training scenes, 2 validation scenes, and 4 test scenes, with comparisons against recent view-synthesis methods.The experiments include controlled architectural studies and evaluations on Tanks and Temples, FVS, and DTU.
  • Architectural choices: MLP Mean aggregation provides a slight edge, particularly for LPIPS, so it is adopted in subsequent experiments.The aggregation study compares weighted mean, MLP, graph attention, and related alternatives.
  • Architectural choices: Increasing residual U-Net refinement stages lowers LPIPS, while PSNR and SSIM show no significant difference; the method therefore uses L = 9.The number of residual rendering stages is varied in the controlled experiments.
  • Architectural choices: Scene-specific fine-tuning clearly improves LPIPS, although it does not significantly alter PSNR or SSIM, and is used for the remaining experiments.Fine-tuning uses only source images; target views used for evaluation are excluded from training and fine-tuning.
  • Tanks and Temples: On Tanks and Temples, the full SVS method achieves the best results on all scenes, while SVS without fine-tuning already outperforms prior work for most scenes.The qualitative results are described as sharper, more complete, more accurate, and more temporally stable than prior work.
  • FVS and DTU: SVS improves over prior work on all FVS scenes and metrics, reducing LPIPS by at least 5 absolute percentage points per scene.On DTU, SVS sets the state of the art for LPIPS, attaining 4.5% error in extrapolation and 1.6% in view interpolation.

7. Discussion

SVS uses differentiable on-surface feature processing to aggregate source-image features on a geometric scaffold and achieves state-of-the-art photorealistic view synthesis on large-scale real-world scenes.

  • SVS adaptively aggregates deep features from source images on a scene’s geometric scaffold using a differentiable set network.
  • The end-to-end pipeline learns from all source images, removing heuristic selection of relevant source images.
  • The method is limited to static scenes, preventing interactive manipulation while maintaining photorealism.

8. Appendix

The appendix describes SVS’s two-stage setup-and-synthesis pipeline, implementation details, qualitative comparisons, and runtime trade-offs across datasets and competing methods.

  • Pipeline: SVS first constructs and encodes a scene, then synthesizes an image from a user-specified target camera and viewpoint.The scene setup includes a geometric scaffold and encoded source images.
  • Pipeline: Target-view synthesis backprojects one 3D point per pixel, projects points into source images, interpolates valid features, and aggregates them in 3D.
  • Training: The method’s scene-agnostic model uses 600,000 training iterations, while scene-specific training uses 256 · N iterations from that model.N is the number of source images for the scene.
  • Qualitative results: SVS produces higher-fidelity FVS results, sometimes appearing sharper than ground truth because some inputs contain motion blur.
  • Qualitative results: DTU extrapolation results can look better than ground truth because camera-setup shadow artifacts are absent from synthesized images.
  • Runtimes: Scaffold construction takes under 97 minutes, while novel-view synthesis takes under 1 second, or under 0.2 seconds with encoded images already in GPU memory.
Loading 2011.07233v2…