Source-linked AI summary
Stereo Magnification: Learning View Synthesis using Multiplane Images
Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, Noah Snavely
TL;DR
The paper addresses view extrapolation from narrow-baseline stereo imagery, a setting relevant to dual-lens phones and VR cameras. It predicts a reusable Multiplane Image from the input pair using a deep network trained with online-video data, and reports favorable view-synthesis performance and effective stereo magnification. The method remains limited by depth-placement errors and artifacts when extrapolating beyond the MPI representation.
Problem
The paper addresses synthesizing views significantly beyond two narrow-baseline stereo inputs, including handling reflections, transparency, and occluded pixels.
Method
A deep network predicts a reusable Multiplane Image from an input stereo pair, using a layered RGBA representation and online videos as training data.
Results
The method achieves better numerical performance on a held-out test set, produces more spatially stable outputs, generalizes without retraining, and magnifies cellphone and stereo-camera baselines.
Takeaways & Limitations
The learned MPI system handles diverse indoor and outdoor scenes, including partially reflective or transparent surfaces, and supports stereo magnification.
Takeaways & Limitations
Fine detail against complex backgrounds can be placed at incorrect depths, and large inter-plane disparities can produce duplicated edges or a “stack of cards” effect.
Abstract
from arXiv · showhide
The view synthesis problem--generating novel views of a scene from known imagery--has garnered recent attention due in part to compelling applications in virtual and augmented reality. In this paper, we explore an intriguing scenario for view synthesis: extrapolating views from imagery captured by narrow-baseline stereo cameras, including VR cameras and now-widespread dual-lens camera phones. We call this problem stereo magnification, and propose a learning framework that leverages a new layered representation that we call multiplane images (MPIs). Our method also uses a massive new data source for learning view extrapolation: online videos on YouTube. Using data mined from such videos, we train a deep network that predicts an MPI from an input stereo image pair. This inferred MPI can then be used to synthesize a range of novel views of the scene, including views that extrapolate significantly beyond the input baseline. We show that our method compares favorably with several recent view synthesis methods, and demonstrate applications in magnifying narrow-baseline stereo images.
1 INTRODUCTION
The paper introduces stereo magnification: learning to extrapolate substantially beyond narrow-baseline stereo inputs using a reusable scene representation and large-scale online-video data.
- Problem: Stereo magnification extrapolates novel views from narrow-baseline stereo pairs, including baseline expansion from about 1 cm to 6.3 cm or up to about 8×.The target applications include creating stronger stereo effects from cellphone images and enabling parallax for VR head motion.
- Challenges: The problem requires handling only two input views, reflections, transparency, and pixels occluded in both inputs.The method therefore seeks a representation predicted once from the pair and reused for many output views.
- Results: The method achieves better numerical performance on a held-out test set and produces more spatially stable output imagery.The shared inferred scene representation supports synthesizing all target views.
- Contributions: The framework combines a learning approach for stereo magnification with Multiplane Images, a new scene representation for view synthesis.The contributions specifically identify both the learning framework and the MPI representation.
- Training data: Online video provides a large-scale source of data for learning view extrapolation.The paper presents online video as a new use for learning this task.
2 RELATED WORK
Prior work includes classical and learning-based view synthesis, but this paper targets extrapolation from narrow-baseline stereo pairs with a reusable layered representation.
- Classical approaches: Classical methods generally interpolate rays or reconstruct geometry, often requiring multiple input views and compositing corresponding input pixels or rays.These approaches can produce high-quality novel views but typically work well only with more than two inputs.
- Learning-based synthesis: Learning-based methods train networks to predict missing views or scene representations from captured imagery and withheld ground-truth views.Examples include volumetric prediction for street scenes and target-view color prediction from light-field photos.
- Extrapolation settings: Other work synthesizes stereo pairs, handles large camera motion, or generates light fields from a single image, while this paper focuses on narrow-baseline stereo pairs.The cited single-view light-field method uses homogeneous flower datasets and extrapolates only to a small Lytro baseline.
- Related applications: View synthesis has also been used as indirect supervision for depth, shape, and optical-flow prediction rather than as the explicit task.This distinguishes those computer-vision applications from direct novel-view generation.
- Scene representations: The MPI representation combines multiple layers, soft visibility handling, and suitability for prediction with deep networks.These properties support reuse of the representation for rendering multiple views.
3 APPROACH
The approach predicts a global multiplane image from stereo inputs, then renders novel views by warping and compositing its layered RGBA planes. The representation and differentiable pipeline support view extrapolation while reusing one inferred scene representation across target views.
- 3.1 Multiplane image representation: A Multiplane Image (MPI) represents a scene as fronto-parallel RGBA planes at fixed depths in a reference camera frame.Each plane stores an RGB color image and an alpha/transparency map; layers are composited back-to-front.
- 3.1 Multiplane image representation: MPIs model occluded, reflective, and transparent content through layered color and alpha, while additional planes expand depth and camera-movement coverage.The paper motivates MPIs by their ability to represent hidden elements, partial transparency, soft edges, and a wider depth range.
- 3.2 Learning from stereo pairs: Given stereo images and camera parameters, a neural network predicts an MPI in the first camera’s reference frame for rendering target views.Training uses tuples containing two input images, their camera parameters, a target image, and target camera parameters.
- 3.2 Learning from stereo pairs: The network uses a plane sweep volume that reprojects the second image onto fixed depth planes, enabling comparisons with the first image for geometry reasoning.The reprojected stack is concatenated with the first image to form the network input tensor.
- 3.2 Learning from stereo pairs: Instead of directly predicting every plane’s color, the network predicts alpha maps, one background image, and per-plane blending weights.Each plane’s color is formed as a weighted combination of the reference image and predicted background, reducing the output parameterization.
- 3.3 Differentiable view synthesis using MPIs: Novel views are synthesized by inverse-warping each MPI plane with a planar transformation and alpha-compositing the warped layers from back to front.Both operations are differentiable, allowing the renderer to be incorporated into training.
4 DATA
The paper builds a large multi-view training dataset from YouTube videos by tracking camera motion, refining poses, filtering clips, and sampling training tuples.
- Dataset source: YouTube videos provide large-scale multi-view training data by supplying frames of the same scenes captured from varied camera baselines.The dataset focuses on real-estate footage because it commonly contains smooth camera motion and little scene movement.
- Processing pipeline: The mining pipeline identifies candidate videos, tracks cameras to segment shots, performs bundle adjustment, and filters unsuitable sequences.ORB-SLAM2 supplies initial poses and clip boundaries; higher-resolution structure from motion then estimates adjusted poses, field of view, and a sparse point cloud.
- Processing pipeline: The pipeline assumes a 90-degree field of view for initial SLAM processing because arbitrary online videos lack known camera intrinsics.This assumption was used to identify good clips, while later processing estimates the field of view for each sequence.
- Processing pipeline: Scale-normalization resolves the arbitrary per-clip scale of reconstructed camera poses so scenes can be represented with MPI layers at fixed depths.The sequences are scaled using the estimated 3D point cloud, with the MPI near plane set to 1m.
- Filtering and clipping: Sequences are filtered by discarding cross-fade boundary frames, retaining the longest smooth camera-track subsequence, and removing sequences shorter than 30 frames.A frame is smooth when its position is sufficiently close to the average of its adjacent positions under the stated threshold.
- Training tuple construction: From approximately 1,500 videos, the pipeline produces approximately 7,000 sequences containing approximately 750K frames.Training tuples are sampled from random ten-frame subsequences by selecting two input frames and a third target frame, allowing interpolation or extrapolation up to nine times the input distance.
5 EXPERIMENTS AND RESULTS
Experiments show that the inferred MPI captures scene structure for novel-view rendering, outperforms the stronger Kalantari variant, and supports coherent extrapolation across datasets and applications. Ablations identify perceptual loss, color representation, and depth-plane count as important design factors.
- 5.1 Visualizing the multiplane images: The inferred MPI captures scene appearance layer by layer while respecting scene geometry, enabling realistic rendering of novel views.
- 5.2 Comparison with Kalantari et al.: Using the central two frames of registered videos as input, the inferred MPI synthesizes both previous and future frames in the same sequence.
- 5.2 Comparison with Kalantari et al.: Our model outperforms the stronger Kalantari variant using VGG loss and the authors’ deeper network, while the deeper architecture and perceptual loss each improve performance.
- 5.2 Comparison with Kalantari et al.: A shared scene-level MPI produces more spatially coherent continuous-view sequences and fewer frame-to-frame artifacts than Kalantari.
- 5.3 Comparison with extrapolation methods: On the HCI light-field dataset, the model generalizes without fine-tuning and introduces fewer depth-boundary distortions than Zhang et al. during extrapolation.
- 5.4 Ablation studies: Perceptual loss better preserves object structure and texture than RGB-space L1 loss, while more depth planes improve accuracy, especially at larger view offsets.
6 DISCUSSION
The learned MPI system handles varied indoor and outdoor scenes, including partially reflective or transparent surfaces, but has depth-placement and representation-range limitations. The paper concludes by proposing extensions to additional input configurations and multidimensional lightfields.
- The system handles both indoor and outdoor scenes and generalizes to scenes differing substantially from its training data.Its learned MPIs also represent partially reflective or transparent surfaces, demonstrated with stereo-magnified anaglyphs.
- Fine detail against complex backgrounds can be placed at incorrect depths, as illustrated by overhead cables jumping between depths.The authors suggest this may reflect overly local depth decisions.
- When adjacent MPI layers differ by more than one pixel in disparity, duplicated edges can create a “stack of cards” effect.
- The framework could be extended to extrapolate from more than two input images or from one image and to generate lightfields with multidimensional view movement.