Source-linked AI summary
Learning-Based View Synthesis for Light Field Cameras
Nima Khademi Kalantari, Ting-Chun Wang, Ravi Ramamoorthi
TL;DR
Consumer light field cameras sparsely sample angular or spatial dimensions because of an inherent resolution trade-off. This paper uses two sequential CNNs to estimate disparity and color for novel-view synthesis from sparse inputs, achieving high-quality results that outperform state-of-the-art approaches and may support reduced angular resolution.
Problem
Consumer light field cameras face an inherent trade-off between angular and spatial resolution, causing sparse sampling in one domain.
Method
The method decomposes view synthesis into disparity and color estimation using two sequential CNNs trained by minimizing synthesized-image error against ground truth.
Results
Using four corner sub-aperture views from Lytro Illum light fields, the method synthesizes high-quality images and outperforms state-of-the-art approaches on challenging scenes.
Takeaways & Limitations
The method could potentially reduce the angular resolution required by consumer light field cameras, allowing their spatial resolution to increase.
Takeaways & Limitations
The system requires retraining for unstructured light fields with larger disparities and cannot produce high-quality results when warped inputs contain no valid information.
Abstract
from arXiv · showhide
With the introduction of consumer light field cameras, light field imaging has recently become widespread. However, there is an inherent trade-off between the angular and spatial resolution, and thus, these cameras often sparsely sample in either spatial or angular domain. In this paper, we use machine learning to mitigate this trade-off. Specifically, we propose a novel learning-based approach to synthesize new views from a sparse set of input views. We build upon existing view synthesis techniques and break down the process into disparity and color estimation components. We use two sequential convolutional neural networks to model these two components and train both networks simultaneously by minimizing the error between the synthesized and ground truth images. We show the performance of our approach using only four corner sub-aperture views from the light fields captured by the Lytro Illum camera. Experimental results show that our approach synthesizes high-quality images that are superior to the state-of-the-art techniques on a variety of challenging real-world scenes. We believe our method could potentially decrease the required angular resolution of consumer light field cameras, which allows their spatial resolution to increase.
1 Introduction
Consumer light field cameras trade angular resolution against spatial resolution, motivating learning-based synthesis of novel views from sparse inputs. The proposed system decomposes synthesis into disparity and color estimation, jointly trains two sequential CNNs, and demonstrates strong results from four corner views.
- 1 Introduction: Consumer light field cameras sparsely sample either angular or spatial dimensions because sensor resolution creates an inherent trade-off.The introduction gives Pelican’s 2 × 2 camera array as an example.
- 1 Introduction: The method synthesizes novel views from sparse consumer-camera inputs using machine learning.It is motivated by the limited resolution of consumer light field cameras.
- 1 Introduction: A single end-to-end CNN is difficult to train for this task and produces blurry novel views.The paper therefore builds on depth-based view synthesis and separates disparity from color estimation.
- 1 Introduction: The system uses two sequential CNNs for disparity and color estimation, training both by minimizing synthesized-image error against ground truth.The disparity estimator is trained through the final synthesis objective rather than requiring ground-truth disparities.
- 1 Introduction: Training on consumer-camera light fields lets the method model camera noise and inaccuracies, producing disparities suited to view synthesis.The paper describes this application-specific disparity estimator as the first of its kind to the authors’ knowledge.
- 1 Introduction: Four corner sub-aperture views from 8 × 8 Lytro Illum light fields suffice to demonstrate synthesis that outperforms state-of-the-art schemes.The system synthesizes a 541 × 376 image in 12.3 seconds, reported as two orders of magnitude faster than DeepStereo.
2 Related Work
Related work reconstructs novel views through light-field super-resolution, geometry-based warping, direct image synthesis, and deep learning. These approaches remain limited by sampling patterns, noise, arbitrary-view synthesis, artifacts, or computational cost.
- 2 Related Work: Light-field super-resolution methods reconstruct dense fields from sparse samples but often require specific input patterns and cannot synthesize arbitrary positions.The reviewed methods exploit dimensionality priors, Fourier sparsity, or multidimensional patches.
- 2 Related Work: Patch- and phase-based reconstruction methods face robustness, baseline, or speed limitations on commercial light-field imagery.The cited approaches struggle with noise, target micro-baseline stereo pairs, or use iterative optimization.
- 2 Related Work: CNN-based light-field super-resolution increases resolution by only a factor of two and cannot synthesize views at arbitrary locations.Layered patch-based synthesis is also discussed for light-field editing applications.
- 2 Related Work: For sparse inputs, Wanner and Goldluecke’s optimization often produces tearing, ghosting, and other artifacts.The review attributes this behavior to sparse views and limitations in the preceding disparity estimation.
- 2 Related Work: Geometry-based scene view synthesis estimates depth, warps input images, and combines the warped images into a novel view.This two-step structure underlies several classical approaches.
- 2 Related Work: Non-geometric methods reconstruct novel views from gradients or patches but use only two input images and therefore cannot exploit all light-field information.The limitation is stated for gradient-moving and patch-based optimization approaches.
- 2 Related Work: DeepStereo projects inputs onto depth planes, predicts colors and weights, and averages them to synthesize the final pixel color.It represents a deep-learning alternative for view synthesis from wide-baseline image sequences.
3 Proposed Learning-Based Algorithm
The method synthesizes novel views by decomposing view synthesis into disparity estimation and color prediction, modeled with two sequential CNNs trained jointly from image reconstruction error. It uses warped sparse inputs and learned features to produce the final image.
- The system estimates novel-view disparity and final pixel colors with two sequential CNNs, replacing a single difficult end-to-end mapping.The disparity output guides backward warping of input views before the color predictor generates the image.
- 3.1 Disparity Estimator: For disparity estimation, input views are backward-warped across predefined disparity levels, then summarized by per-level mean and standard deviation features.The implementation uses 100 disparity levels spanning [-21, 21] pixels and concatenates the resulting features into 200 channels.
- 3.1 Disparity Estimator: The disparity CNN learns the disparity map by minimizing synthesized-image error rather than disparity error, avoiding the need for ground-truth disparities.This objective is application-specific because inaccurate disparity can still reconstruct constant-color regions correctly.
- 3.2 Color Predictor: The color predictor uses warped images, estimated disparity, and novel-view position to learn how to combine valid information, especially near occlusion boundaries.The input feature vector is H = {warped input images, D_q, q}, and the network outputs an RGB image.
- 3.3 Training: Both networks are trained simultaneously by minimizing the L2 distance between synthesized and ground-truth novel-view images, with gradients propagated through both CNNs.The optimization differentiates the error with respect to the disparity-network and color-network weights.
4 Results
Experiments show that the two-network approach produces plausible novel views from four corner views, outperforming competing methods especially around occlusion boundaries, while remaining limited when warped inputs lack valid information.
- Architecture comparison: The two-network architecture avoids the blurry results produced by a single CNN that directly maps input images to the novel view.The single-network relationship is difficult to learn because it requires connecting distant pixels.
- Comparison against other approaches: Across 30 test scenes, the approach produces results significantly better than other methods, including challenging foliage and flower scenes.The training and test scenes were completely separate, and the examples show plausible reconstructions near complex boundaries.
- Comparison against other approaches: The method reconstructs occlusion-boundary details more reliably than competing approaches, which produce tearing, ghosting, discoloration, and other artifacts.Examples include the truck roof, highlights, thin branch details, and areas between flower stems and petals.
- Disparity estimation: The disparity estimator produces reasonable estimates in regions important for view synthesis, although its inaccuracies remain in less important regions.Other disparity methods can have higher overall quality but are often inaccurate around occlusion boundaries.
- Efficiency: 12.3 seconds is required to synthesize a 541 × 376 novel view from four input images on the reported hardware.Feature extraction takes 5.5 seconds, disparity estimation 5.1 seconds, warping 0.2 seconds, and color prediction 1.5 seconds.
- Color prediction: The color predictor detects artifacts in warped inputs and combines appropriate information to produce a high-quality final image.This improves results when occlusion causes individual warped images to contain artifacts.
- Denoising effect: Using all input views produces results that are generally less noisy than the ground-truth images, with potential usefulness in low-light capture.The paper attributes this effect to combining information across all input views.
- Limitations: When no warped input contains valid information, the color predictor cannot produce a high-quality result and may create tearing artifacts.The LEAVES example fails to synthesize a post between leaves, although the result remains better than other approaches.
5 Conclusions and Future Work
The paper presents a two-CNN system for synthesizing views from four corner sub-aperture images captured by a consumer light field camera. It reports better performance than state-of-the-art approaches and identifies extensions toward broader light-field applications and faster processing.
- The system models disparity and color estimation with two sequential convolutional neural networks for sparse-view synthesis.
- Using four corner sub-aperture images from a Lytro Illum camera, the method outperforms state-of-the-art approaches across a variety of scenes.
- Future work includes high-dynamic-range light fields, arbitrary numbers of input views, interactive or real-time speed, and integration with light-field compression.