PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Synthesize a 4D RGBD Light Field from a Single Image

Pratul P. Srinivasan, Tongzhou Wang, Ashwin Sreelal, Ravi Ramamoorthi, Ren Ng

arXiv:1708.03292v1cs.CVcs.GR

TL;DR

Recovering a 4D RGBD light field from one 2D image is ill-posed, motivating learned priors and a large scene-specific dataset. The paper combines ray-depth estimation, Lambertian rendering, and prediction of occluded and non-Lambertian rays, demonstrating light-field and depth synthesis from single images. Its broader scope remains limited because generalization to the full diversity of everyday scenes is open.

  • Problem

    Reconstructing a 4D light field from a single 2D image is severely ill-posed, while estimating single-image geometry lacks convenient ground-truth depth for real scenes.

  • Method

    The paper trains an end-to-end CNN that predicts per-ray depths, renders a Lambertian light field, and refines it with occlusion and non-Lambertian predictions.

  • Results

    The method synthesizes high-quality 4D light fields and ray depths from a single image without ground-truth depths during training.

  • Takeaways & Limitations

    Light-field synthesis provides a strategy for unsupervised single-image geometry estimation and supports progress toward dense VR/AR content from sparsely sampled images.

  • Takeaways & Limitations

    The approach is trained on flowers and plants with camerascale baselines, and generalization to the full diversity of everyday scenes remains open.

Abstract

from arXiv · show

We present a machine learning algorithm that takes as input a 2D RGB image and synthesizes a 4D RGBD light field (color and depth of the scene in each ray direction). For training, we introduce the largest public light field dataset, consisting of over 3300 plenoptic camera light fields of scenes containing flowers and plants. Our synthesis pipeline consists of a convolutional neural network (CNN) that estimates scene geometry, a stage that renders a Lambertian light field using that geometry, and a second CNN that predicts occluded rays and non-Lambertian effects. Our algorithm builds on recent view synthesis methods, but is unique in predicting RGBD for each light field ray and improving unsupervised single image depth estimation by enforcing consistency of ray depths that should intersect the same scene point. Please see our supplementary video at https://youtu.be/yLCvWoQLnms

1. Introduction

The paper frames local light field synthesis as promoting one photograph into a dense plenoptic light field, addressing an ill-posed reconstruction problem with learned scene geometry and light-field priors. It contributes a large flowers-and-plants dataset, ray-depth consistency regularization, and an end-to-end CNN pipeline that synthesizes convincing light fields and depths.

  • Motivation: Local light field synthesis promotes a single photograph to a plenoptic camera light field and can provide synthetic apertures, refocusing, and denser sampling for VR/AR capture.The task expands one view into a dense 2D patch of views, potentially filling gaps between discrete viewpoints.
  • Problem: Reconstructing a 4D light field from a single 2D image is severely ill-posed, so the method uses machine learning to exploit natural-light-field priors.Flowers and plants provide complex occlusions and varied relative depths for this study.
  • Contributions: 3343 flower and plant light fields form the largest available light field dataset introduced for training this synthesis problem.The data were captured with the Lytro Illum camera.
  • Ray Depths and Regularization: The method predicts a separate depth map for every viewpoint, equivalent to estimating depth for each ray in the 4D light field, with consistency regularization across viewpoints.The regularizer is physically based and targets depth inconsistencies among rays that should intersect the same scene point.
  • CNN Framework: The end-to-end CNN estimates ray depths, renders a Lambertian approximation, and predicts occluded rays and non-Lambertian effects to refine the light field.Training minimizes reconstruction errors for the Lambertian and predicted light fields together with physically based depth regularization.
  • Results: The algorithm synthesizes high-quality light fields and ray depths from a single image without access to ground-truth depths during training.The paper presents this as evidence that single-image depth estimation through light-field synthesis is feasible.

2. Related Work

Prior work renders or reconstructs light fields from multiple views, synthesizes individual novel views, or estimates geometry before warping views. This paper instead synthesizes an entire 4D light field and all corresponding depths from one image at once, using geometry estimation to improve depth estimates.

  • Light Fields: A 4D light field represents the total spatioangular distribution of light rays through a region of free space.Prior applications include novel-view rendering, post-capture refocusing, lens-aberration correction, and scene-flow estimation.
  • View Synthesis from Light Fields: Classical light-field rendering captures densely sampled 4D rays and extracts new viewpoints as 2D slices, while Lumigraph methods use approximate geometry to refine those slices.Unstructured Lumigraph rendering extends this idea to unstructured 2D light-field slices.
  • View Synthesis without Geometry Estimation: Geometry-free approaches reconstruct light fields from carefully constructed 2D views using Fourier sparsity or phase-based methods.The cited phase-based methods use pairs of 2D slices, whereas the Fourier approach uses a 2D collection of views.
  • CNN View Synthesis: CNN view-synthesis methods regress novel images or predict flow fields from one or more input views, often using synthetic images or structured multi-view settings.Examples include single-view camera-conditioned regression, sharper flow-based synthesis, four-corner light-field interpolation, and stereo-pair prediction.
  • View Synthesis by Geometry Estimation: Geometry-based view interpolation estimates scene geometry from input light-field slices and warps views to reconstruct new viewpoints.Earlier approaches include wider-baseline unstructured stereo pairs processed with multi-view stereo algorithms.
  • This Work: Unlike previous methods, this paper synthesizes the entire 4D light field and all corresponding depths simultaneously from a single image, leveraging joint prediction for better depth estimation.The approach is inspired by geometry-based view synthesis but expands the output from individual novel views to the full light field.

3. Light Field Synthesis

Light field synthesis reconstructs a densely sampled 4D light field from one central view by exploiting scene geometry and light-field redundancies. The proposed factorization estimates ray depths, renders a Lambertian approximation, and predicts occluded or non-Lambertian rays.

  • Problem: A single central 2D slice is used to synthesize views on a densely sampled grid, equivalent to reconstructing a 4D light field.Light-field redundancies and scene statistics make this ill-posed reconstruction tractable.
  • CNN Framework: The synthesis function is factorized into depth estimation, Lambertian rendering from the central view, and prediction of occluded rays and non-Lambertian effects.Depth and occlusion functions are CNNs, while rendering is physically based.
  • Unsupervised Geometry: The factorization enables unsupervised estimation of scene depths from a single image by learning to reduce the discrepancy between rendered and ground-truth light fields.Predicted ray depths determine which pixels in the input view are sampled for rendering.
  • Training: End-to-end training minimizes Lambertian and predicted-light-field reconstruction errors together with consistency and total-variation regularization losses.Including both reconstruction errors prevents the occlusion network from learning the entire prediction task independently of depth estimation.

4. Light Field Dataset

The paper introduces a large light-field dataset of flowers and plants captured with a Lytro Illum camera. Its scenes vary in species, locations, times of day, lighting, and complex occlusion structure, supporting future light-field and geometry research.

  • Dataset Construction: The dataset contains 3343 flower and plant light fields, split into 3243 training examples and 100 testing examples.Each light field has 376x541 spatial samples and 14x14 angular samples.
  • Scene Diversity: The collection spans roses, poppies, thistles, orchids, lilies, irises, and other plants with complex occlusions.Epipolar slices visualize the dataset’s geometric complexity.
  • Scene Diversity: Captures were collected across locations and times of day under different natural lighting conditions.This variation contributes to the dataset’s diversity.
  • Research Uses: The authors intend the dataset to support future work on light-field synthesis, single-view synthesis, and unsupervised geometry learning.

5. Synthesizing 4D Ray Depths

The method predicts a depth for every light-field ray from a single image and regularizes depths along rays that should intersect the same scene point. This addresses depth ambiguities in texture-less and occluded regions without ground-truth depths.

  • Depth estimation: Depth is learned by minimizing Lambertian light-field reconstruction error together with novel depth regularization.The depth function predicts ray depths from the input view using a CNN.
  • Depth ambiguities: Texture-less regions can have incorrect depths because multiple depths produce the same synthesized view.View-reconstruction loss therefore provides no incentive to recover the correct depth in these regions.
  • Depth ambiguities: Occluded scene points can also receive incorrect depths because the correct depth would sample pixels from the input-view occluder.This failure occurs when depth is learned only through novel-view reconstruction.
  • Depth consistency: Ray-depth consistency requires depths from rays intersecting the same scene point to agree along lines with the same slope.The constraint is expressed for any continuous k, with finite-difference regularization implemented using k = 1.
  • Depth consistency: The regularization improves consistency in texture-less areas and for rays occluded from the input view.Incorrect depths create larger gradients and new edges along the sheared lines used by the regularizer.
  • Dataset and network: The dataset contains 3343 flower and plant light fields captured with a Lytro Illum camera, spanning complex occlusions and wide relative-depth ranges.The dataset has no ground-truth depths, and disparities are summarized using predictions from the algorithm.

6. Synthesizing the 4D Light Field

The predicted depths first render a differentiable Lambertian approximation by sampling the central view. A residual 3D CNN then predicts occluded rays and non-Lambertian effects to produce the synthesized light field.

  • Lambertian rendering: Predicted ray depths render the 4D light field by sampling the input central-view image.The Lambertian rendering uses a physically based warp and differentiable bilinear sampling.
  • Occlusions and non-Lambertian effects: Depth-based Lambertian rendering cannot correctly synthesize rays occluded from the input view or accurately represent non-Lambertian effects.A separate prediction stage is therefore needed after the approximate rendering.
  • Occlusions and non-Lambertian effects: A residual block predicts occluded rays and non-Lambertian effects from the approximate light field and estimated depths.The residual is added to the Lambertian approximation, focusing loss reductions on effects absent from that approximation.
  • Occlusions and non-Lambertian effects: The occlusion network is a 3D CNN whose filters access every stacked 2D view of the light field.The implementation uses five 3D convolution layers with 3x3x3 filters.
  • Training: Training uses randomly cropped and downsampled 4D light fields with fully differentiable bilinear sampling and end-to-end Adam optimization.Examples are cropped at 192x192x8x8 and downsampled to 96x96x8x8.

7. Results

The method synthesizes convincing 4D light fields and ray depths from single images, including complex occlusions, while outperforming appearance flow quantitatively and supporting photographic effects and fast inference. It also generalizes to cell-phone images and toys, although toy results are perceptually weaker despite similar mean error.

  • Our method produces compelling 4D ray depths and light fields containing complex occlusions and relative depths.
  • Predicted and Lambertian light fields have lower mean L1 errors than appearance flow, with consistent superiority across view positions.
  • 0.026 versus 0.031 mean L1 error across all views shows our model outperforming a directly regressed input-to-output light-field CNN.
  • 0.0176 versus 0.0145 mean L1 error shows single-view synthesis performs only slightly worse than interpolation using four corner views.
  • Synthetic defocus increases the aperture from f/28 to f/3.5, and the synthesized field supports refocusing from flowers to the background.
  • Inference takes under 1 second to synthesize a 187x270x8x8 light field and ray depths on a single Titan X GPU.
  • The method generalizes to iPhone 5s photographs and toys, but toy results are perceptually less impressive despite similar mean L1 errors.The reported mean L1 error is 0.027 for toys and 0.026 for flowers.

8. Conclusion

The work shows that consumer light-field cameras can support datasets for synthesizing local light fields from single photographs. It identifies broader opportunities in unsupervised geometry estimation and immersive VR/AR content, while generalization to everyday scenes remains open.

  • Consumer light-field cameras enable practical capture of datasets large enough to train single-photograph local light-field synthesis for specific scenes.
  • The approach can extend to other niches such as toys, but generalizing across the full diversity of everyday scenes remains an open problem.
  • Consistent geometry for rays intersecting the same scene point could extend beyond ray depths to voxels, point clouds, and meshes.
  • Synthesizing dense light fields is presented as important for capturing VR/AR content and progressing toward immersive content from sparsely sampled images.
Loading 1708.03292v1…