Source-linked AI summary
DeepStereo: Learning to Predict New Views from the World's Imagery
John Flynn, Ivan Neulander, James Philbin, Noah Snavely
TL;DR
New view synthesis is challenging because occlusions and ambiguous geometry make unseen pixels difficult to reconstruct, while deep learning had seen limited use for graphics. DeepStereo addresses this with an end-to-end architecture that predicts output pixels from posed images, achieving competitive results with existing image-based rendering methods despite different training and test data.
Problem
New view synthesis must infer unseen pixels despite occluders and ambiguous geometry, and deep learning had been little used for generating views from real imagery.
Method
DeepStereo uses an end-to-end deep architecture whose selection and color towers combine depth-conditioned color layers into output images from posed inputs.
Results
The method produces competitive results with existing image-based rendering methods even though its training data differs considerably from the test sets.
Takeaways & Limitations
The system requires only sets of posed imagery and generalizes across indoor, outdoor, and prior-work image collections.
Takeaways & Limitations
Rendering a 512×512 pixel image takes about 12 minutes on a multi-core workstation because the implementation renders small patches and does not fully exploit convolution.
Abstract
from arXiv · showhide
Deep networks have recently enjoyed enormous success when applied to recognition and classification problems in computer vision, but their use in graphics problems has been limited. In this work, we present a novel deep architecture that performs new view synthesis directly from pixels, trained from a large number of posed image sets. In contrast to traditional approaches which consist of multiple complex stages of processing, each of which require careful tuning and can fail in unexpected ways, our system is trained end-to-end. The pixels from neighboring views of a scene are presented to the network which then directly produces the pixels of the unseen view. The benefits of our approach include generality (we only require posed image sets and can easily apply our method to different domains), and high quality results on traditionally difficult scenes. We believe this is due to the end-to-end nature of our system which is able to plausibly generate pixels according to color, depth, and texture priors learnt automatically from the training data. To verify our method we show that it can convincingly reproduce known test views from nearby imagery. Additionally we show images rendered from novel viewpoints. To our knowledge, our work is the first to apply deep learning to the problem of new view synthesis from sets of real-world, natural imagery.
1. Introduction
The paper frames new view synthesis as a difficult, underconstrained rendering problem and proposes an end-to-end deep network that predicts output pixels from posed imagery. It trains on abundant Street View data to generalize across scene types and compares synthesized views with captured imagery and existing IBR methods.
- New view synthesis synthesizes a scene view by warping and combining nearby posed images for applications including cinematography, virtual reality, and teleconferencing.
- Occlusions and textureless surfaces make the unseen geometry and target colors ambiguous, so image-based rendering requires strong priors.
- Traditional stereo and warping pipelines often produce tearing around occluders, remove fine structures, and introduce aliasing, especially for self-occluding objects such as trees.
- The proposed approach regresses output pixel colors directly from posed inputs using an end-to-end deep network with minimal scene assumptions.
- Large-scale posed imagery mined from Google Street View supplies training data, supporting robustness across indoor and outdoor scenes and prior-work image collections.
- The evaluation reproduces withheld Street View and indoor views and qualitatively compares synthesized results with state-of-the-art image-based rendering methods.
2. Related Work
Related work spans learned depth and stereo, hand-built image-based rendering, and early deep image synthesis. DeepStereo differs by learning natural-image view synthesis directly without requiring depth or disparity labels.
- Earlier shape-prediction methods learned depth or 3D structure from single images using aligned photos, scans, annotations, or Kinect data.
- Machine learning has also been used to estimate parameters of traditional stereo models or learn correlation filters for disparity estimation.
- Unlike prior learned stereo work, DeepStereo synthesizes new views directly with a deep architecture and does not require known depth or disparity as training data.
- Image-based rendering research includes light fields, correspondence and warping, and explicit shape-and-appearance estimation, often combining 3D shape with warping and blending.
- The paper learns to minimize prediction error on training images rather than reconstructing depth or another intermediate representation.
- The authors position their work as the first deep-learning method for synthesizing novel natural imagery from posed real-world inputs.
3. Approach
The approach converts posed input images into plane-sweep volumes, then uses separate selection and color towers to synthesize the target view end-to-end. Selection probabilities choose among depth-specific colors, whose weighted sum produces each output pixel.
- Using plane-sweep volumes: Plane-sweep volumes reproject each posed input image into the target camera across multiple depths, making pose and epipolar correspondence implicit network inputs.Each reprojected image is formed at depths d1 through dD, and corresponding pixels occupy corresponding columns.
- Our model: The network uses separate selection and color towers for depth selection and per-depth color prediction.The selection tower estimates the likelihood that each output pixel lies at each depth, while the color tower predicts the best color for each depth.
- Our model: Selection probabilities weight the depth-specific color outputs, and summing these products produces the final output image.The weighted combination also reduces distortion under uncertainty by blurring uncertain regions rather than copying warped pixels.
- The selection tower: The selection tower shares early 2D convolutional weights across planes, adds across-depth interactions for occlusion, and normalizes depth scores with a per-pixel softmax.The softmax encourages selecting one depth plane while keeping the probabilities across planes summed to 1.
- The color tower: The color tower uses shared 2D convolutional layers and a linear reconstruction layer to produce RGB values independently for each depth plane.Unlike the selection tower, it does not require across-depth interaction for occlusion effects.
- Training: Training compares the synthesized image with the known target using a per-pixel L1 loss, while inference predicts output patches rather than full images at once.The architecture multiplies selection and color outputs before applying the loss.
4. Results
The model produces plausible interpolated views across outdoor, indoor, and different-camera imagery, while retaining some limitations and sensitivity to camera motion.
- The model was evaluated by reconstructing withheld Street View views and interpolating imagery from a prior handheld-DSLR dataset.The evaluation included outdoor and indoor scenes, plus data substantially different from the training imagery.
- 12 minutes are required to render a 512×512 pixel image on a multi-core workstation.The implementation renders small patches because rendering an entire image would be prohibitively expensive in RAM.
- The model produced plausible outputs that were difficult to immediately distinguish from the original images.
- The outputs show slight resolution loss, disappearing thin foreground structures, and overblurred partially occluded objects.The model cannot render surfaces that appear in none of the input images.
- Moving objects are handled gracefully by appearing blurred in a manner evocative of motion blur, but violating the maximum camera-motion assumption significantly degrades quality.
5. Discussion
The method demonstrates end-to-end novel view synthesis from posed imagery and achieves competitive results despite training and test data differences. Its main limitations are computational cost, fixed input-view count, and dependence on reprojecting images for every rendered frame.
- End-to-end training successfully performs novel view synthesis from posed imagery and requires only sets of posed images.
- The results are competitive with existing image-based rendering methods even though the training data differs considerably from the test sets.
- The method is not optimized for speed and is likely far from real-time performance; its use of 96 depth planes also limits output resolution.
- Unlike volumetric stereo methods that reproject inputs once during scene creation, this method requires reprojected images for every rendered frame.
- Future directions include recurrent architectures for faster inference and applying similar networks to intermediate video-frame synthesis or depth-map regression.
- The network is trained with 5 input views per target and cannot currently change that number after training, limiting use of denser camera sets.