PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepStereo: Learning to Predict New Views from the World's Imagery

John Flynn, Ivan Neulander, James Philbin, Noah Snavely

arXiv:1506.06825v1cs.CV

TL;DR

New view synthesis asks how to create a scene view by combining nearby posed images. DeepStereo regresses output pixels end-to-end from posed views, and the authors report results competitive with existing image-based rendering methods despite considerably different training and test imagery.

  • Problem

    The paper addresses how to synthesize a new scene view by warping and combining nearby posed images, a task with applications including virtual reality and cinematography.

    Source: §1 · p. 1

  • Method

    The system end-to-end regresses output pixel colors from posed inputs, combining depth-specific color layers with per-pixel weights from selection maps.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    The authors report results competitive with existing image-based rendering methods despite training data considerably different from the test sets.

    Source: §5 · p. 7

  • Takeaways & Limitations

    The authors conclude novel view synthesis can be trained end-to-end using only sets of posed imagery.

    Source: §5 · p. 7

    Rendering a 512×512 image takes about 12 minutes on a multi-core workstation in the current implementation, which does not fully exploit the model’s convolutional nature.

    Source: §4 · p. 6

Abstract

from arXiv · show

Deep networks have recently enjoyed enormous success when applied to recognition and classification problems in computer vision, but their use in graphics problems has been limited. In this work, we present a novel deep architecture that performs new view synthesis directly from pixels, trained from a large number of posed image sets. In contrast to traditional approaches which consist of multiple complex stages of processing, each of which require careful tuning and can fail in unexpected ways, our system is trained end-to-end. The pixels from neighboring views of a scene are presented to the network which then directly produces the pixels of the unseen view. The benefits of our approach include generality (we only require posed image sets and can easily apply our method to different domains), and high quality results on traditionally difficult scenes. We believe this is due to the end-to-end nature of our system which is able to plausibly generate pixels according to color, depth, and texture priors learnt automatically from the training data. To verify our method we show that it can convincingly reproduce known test views from nearby imagery. Additionally we show images rendered from novel viewpoints. To our knowledge, our work is the first to apply deep learning to the problem of new view synthesis from sets of real-world, natural imagery.

1. Introduction

DeepStereo uses a deep network to synthesize unseen views directly from posed input images, learning pixel predictions from large collections of posed imagery. It targets the artifacts of traditional view synthesis while aiming to remain plausible under uncertainty and scene variation.

1. Introduction

  • The authors frame new-view synthesis as image-based rendering for applications including cinematography, virtual reality, teleconferencing, image stabilization, and 3-dimensionalizing monocular film footage.The task synthesizes a view by warping and combining nearby posed images.Source: §1 · p. 1
  • New-view synthesis is underconstrained because occlusions hide geometry and texture-poor surfaces have ambiguous geometry, so IBR needs priors for uncertain or occluded pixels.An exact solution would require full 3D knowledge of visible geometry in the unseen view, which is generally unavailable.Source: §1 · pp. 1–2
  • Traditional stereo and warping methods model stereo, color, and occlusion, but can produce tearing, lost fine structure, and aliasing, especially around complex self-occluding objects and wide baselines.When uncertain, DeepStereo prefers blur over tearing or repetition; it assumes scenes are largely static and within a finite depth range, yet can degrade gracefully when those assumptions fail.Source: §1 · p. 2
  • Despite deep networks’ success in image understanding [29], they had seen little use in graphics; prior denoising, super-resolution, and rendering work challenged concerns about direct pixel generation [21].The paper also cites evidence that state-of-the-art networks can have parameters in the low millions, reducing overfitting potential.Source: §1 · p. 2
  • The system regresses output pixel colors from posed inputs, interpolates wide-baseline views, and degrades gracefully with motion or specularities.The authors attribute this resilience to end-to-end training and learned nonlinear functions of the inputs.Source: §1 · p. 2
  • Training leaves out an image from each posed set and predicts it from the others, using Google Street View imagery to support generalization across indoor and outdoor scenes.The paper also reports comparisons with captured street and indoor images and qualitative comparisons to existing state-of-the-art IBR methods.Source: §1 · p. 2

2. Related Work

DeepStereo distinguishes itself from prior depth, stereo, and image-based rendering methods by learning to synthesize views directly from posed images, optimizing output-image prediction without requiring depth labels. The paper also places this approach among early deep-learning work on image synthesis, claiming it is the first to target novel natural imagery from real-world inputs.

Learning depth from images

  • Single-view depth and shape prediction includes methods using aligned photos and laser scans, manually annotated geometric classes, Kinect data, and deep learning [9] [33].The paper notes that single-view prediction remains challenging and that gathering sufficient training data is difficult and time-consuming.Source: §2 · p. 2
  • Unlike prior stereo-learning work, DeepStereo learns to synthesize new views directly with a deep architecture, without requiring known depth or disparity as training data.Earlier stereo-learning methods estimated traditional model parameters or learned low-level correlation filters for disparity estimation.Source: §2 · p. 2

View interpolation

  • Prior image-based rendering used light fields, correspondence and warping, shape and appearance estimation, or 3D shape with warping and blending; these methods are largely hand-built and do not leverage training data.DeepStereo instead learns to predict viewpoints by minimizing prediction error on its training set.Source: §2 · p. 2
  • Inspired by Fitzgibbon et al. and Szeliski, the method optimizes faithful prediction of the output image rather than depth or other intermediate representations.It directly minimizes image prediction error.Source: §2 · pp. 2–3
  • The authors believe theirs is the first deep-learning method to synthesize novel natural imagery from posed real-world input images.They contrast this with networks for generating rendered 3D-chair images and parsing and rerendering imagery such as faces [22].Source: §2 · p. 3

3. Approach

DeepStereo reprojects posed source images into target-camera plane-sweep volumes, then learns depth-wise selection and color prediction to synthesize the target view. The network is trained on posed street-scene imagery captured by a moving vehicle.

3. Approach

  • Plane-sweep stereo reprojects each source image into target camera C across candidate depths, making poses implicit and aligning corresponding pixels by depth to avoid dense long-range connections.Reprojection can be performed with basic texture mapping on a GPU.Source: §3 · pp. 3–4
  • A selection tower predicts per-pixel depth probabilities, while a color tower predicts a color at each depth; their weighted sum forms the output and blurs uncertain regions rather than distorting source pixels.The towers are learned end-to-end, and training compares the synthesized image with the target using per-pixel L1 loss.Source: §3 · pp. 4–5, Figure 4 · p. 4
  • The selection tower shares 2D convolution weights across depth planes, models interactions across planes, and applies a depth-wise softmax; the color tower uses shared 2D convolutions without across-depth interaction.A tanh penultimate activation made selection-tower training more stable than a linear layer in the experiments.Source: §3 · pp. 4–5
  • The system predicts output patches using four resolutions of successively larger input areas, processed independently before nearest-neighbor upsampling and concatenation to provide more context.Rather than predicting a full image at once, the network predicts patch-by-patch.Source: §3 · p. 5

3.1. Training

  • Training used approximately 100K posed street-scene image sets from a moving vehicle, with targets selected from one rosette and source rosettes chosen nearby, up to 3 m away.Images were posed using odometry and structure-from-motion techniques.Source: §3.1 · pp. 5–6

4. Results

The model produced plausible interpolations of Street View imagery and transferred reasonably to a distinct handheld-DSLR dataset, while showing identifiable artifacts and limits.

4. Results

  • Street View interpolation was evaluated against withheld images at matching viewpoints, with outdoor and indoor examples reported.The outdoor example is shown in Figure 8; the indoor example is shown in Figure 9.Source: §4 · p. 6, Figure 8 · p. 7
  • It reasonably reproduced and interpolated handheld-DSLR imagery from Chaurasia, et al., despite not being trained directly for that task.The paper also describes results on that dataset as competitive despite differences in training and test imagery and cameras.Source: §4 · pp. 6–7
  • Rendering a 512×512 pixel image took about 12 minutes on a multi-core workstation; images were rendered in small patches to avoid the prohibitive RAM cost of rendering an entire image.The implementation did not fully exploit the model’s convolutional nature, and the authors suggested a GPU implementation could reduce rendering times to minutes or even seconds.Source: §4 · p. 6
  • The model produced plausible outputs difficult to distinguish from original imagery, handled trees and glass, and degraded gracefully around specular surfaces.The network did not attempt to model specular surfaces.Source: §4 · pp. 6–7
  • Artifacts included slight resolution loss, disappearance of thin foreground structures, blur around partially occluded objects, and inability to render surfaces absent from all inputs.Source: §4 · p. 7
  • Moving objects were handled gracefully with motion-blur-like appearances, but violating the maximum camera-motion assumption significantly degraded interpolation quality.Pedestrians in Figure 8 illustrate the blurred appearance.Source: §4 · p. 7, Figure 8 · p. 7

5. Discussion

DeepStereo synthesizes novel views end-to-end from posed imagery and reports competitive results despite differences between its training data and test sets. The discussion identifies speed and fixed input-view count as limitations and outlines possible improvements and applications.

Discussion

  • The end-to-end method uses posed image sets and produces results competitive with existing image-based rendering methods despite different training and test data.The discussion presents comparisons between real and synthesized views as evidence of the method’s generality.Source: §5 · p. 7
  • The method is not optimized for speed, and its 96 depth planes limit output resolution while increasing training time, RAM use, and runtime as resolution grows.Unlike other volumetric stereo methods, it requires reprojected images for every rendered frame rather than only once when creating the scene.Source: §5 · pp. 7–8
  • Recurrent processing of reprojected depth images, precomputed network components, and warping before final layers are proposed as ways to improve speed, potentially enabling real-time GPU performance.Processing one depth image at a time could avoid connections across depth and likely make inference faster.Source: §5 · p. 8
  • Training uses five input views per target, and the number cannot currently change after training, limiting use of denser camera sets.Selecting input views per pixel risks discontinuities at view transitions; a more complex recurrent model might accept arbitrary view counts but complicate training. Intermediate layers may also reveal a pixel-similarity measure useful to traditional stereo.Source: §5 · p. 8
  • With appropriate training data, a similar network could synthesize intermediate video frames or regress to a depth map.Source: §5 · p. 8

Related papers

Loading 1506.06825v1…