PaperlayerView exact revision on arXiv

Source-linked AI summary

Light Field Reconstruction Using Convolutional Network on EPI and Extended Applications

Gaochang Wu, Yebin Liu, Lu Fang, Qionghai Dai, Tianyou Chai

arXiv:2103.13043v1eess.IVcs.CV

TL;DR

Sparse light-field capture creates a spatial-angular resolution trade-off, and direct EPI angular super-resolution can suffer from information asymmetry. The paper addresses this with a blur-restoration-deblur framework that restores angular detail while recovering spatial frequencies, achieving better performance across diverse scenes and enabling extended rendering applications.

  • Problem

    Limited sensor resolution forces a trade-off between spatial and angular sampling, while sparse angular input damages angular detail and complicates light-field reconstruction.

  • Method

    The framework blurs each EPI spatially, uses a CNN to restore undersampled angular detail, and applies non-blind deblurring to recover suppressed spatial detail.

  • Results

    Experiments on synthetic, real-world, and microscope light fields show that the framework outperforms state-of-the-art approaches in occluded and transparent regions and on non-Lambertian surfaces.

  • Takeaways & Limitations

    The framework supports depth enhancement, unstructured-input interpolation, and depth-assisted rendering for large-disparity data.

  • Takeaways & Limitations

    The current implementation requires at least three views per angular dimension, cannot handle extrapolation, and supports maximum disparity of 5 pixels without depth-assisted extensions.

Abstract

from arXiv · show

In this paper, a novel convolutional neural network (CNN)-based framework is developed for light field reconstruction from a sparse set of views. We indicate that the reconstruction can be efficiently modeled as angular restoration on an epipolar plane image (EPI). The main problem in direct reconstruction on the EPI involves an information asymmetry between the spatial and angular dimensions, where the detailed portion in the angular dimensions is damaged by undersampling. Directly upsampling or super-resolving the light field in the angular dimensions causes ghosting effects. To suppress these ghosting effects, we contribute a novel "blur-restoration-deblur" framework. First, the "blur" step is applied to extract the low-frequency components of the light field in the spatial dimensions by convolving each EPI slice with a selected blur kernel. Then, the "restoration" step is implemented by a CNN, which is trained to restore the angular details of the EPI. Finally, we use a non-blind "deblur" operation to recover the spatial high frequencies suppressed by the EPI blur. We evaluate our approach on several datasets, including synthetic scenes, real-world scenes and challenging microscope light field data. We demonstrate the high performance and robustness of the proposed framework compared with state-of-the-art algorithms. We further show extended applications, including depth enhancement and interpolation for unstructured input. More importantly, a novel rendering approach is presented by combining the proposed framework and depth information to handle large disparities.

1 INTRODUCTION

Light field cameras trade spatial resolution against angular resolution, making sparse-view reconstruction difficult. This paper reconstructs angular detail on EPIs with a blur-restoration-deblur framework and demonstrates strong performance across challenging scenes and applications.

  • Limited sensor resolution forces light field cameras to trade spatial sampling for angular sampling, leaving one dimension sparsely captured.
  • The paper models sparse-view reconstruction as angular detail restoration on EPIs, whose structures are shared across macroscopic and microscope light fields.
  • Information asymmetry damages high-frequency angular content, so direct angular super-resolution produces aliasing and ghosting in reconstructed light fields.
  • The blur-restoration-deblur framework extracts spatial low frequencies, uses a CNN to restore angular detail, and applies deblurring to recover suppressed spatial frequencies.The blur step uses a selected kernel on each EPI slice, while restoration requires at least three views in each angular dimension.
  • The framework outperforms state-of-the-art approaches especially on complex occlusions, non-Lambertian surfaces, and challenging microscope scenes.On the Neurons 20× microscope case, the competing method produces blur in occluded regions while the proposed approach produces reasonable results.
  • Extended applications include depth enhancement, interpolation for unstructured light fields, and depth-assisted rendering for large disparities.The full paper also analyzes Fourier-domain behavior and adds a rendering scheme combining the framework with depth information.

2 RELATED WORK

Related work addresses light field reconstruction through super-resolution, novel-view synthesis, plenoptic-function reconstruction, geometric estimation, and learning-based methods. These approaches differ in sampling assumptions, input patterns, and robustness to occlusions and challenging scenes.

  • Light field methods commonly target spatial or angular super-resolution because limited sensor resolution creates a spatial-angular trade-off.
  • Sampling-based methods support direct ray interpolation for densely sampled fields, while Fourier-domain methods reconstruct sparsely sampled fields under specialized priors.Direct ray interpolation assumes neighboring-view disparity does not exceed 1 pixel.
  • EPI-based geometric methods estimate disparity with structure tensors and optimize or regularize depth, but prior angular super-resolution work reverted to sub-aperture images.
  • Learning-based methods include sparse-coding, deep spatial-angular super-resolution, and CNN novel-view synthesis, but can underuse full light-field information or depend on fixed inputs.
  • Prior approaches exhibit limitations including artifacts in occluded regions, restricted micro-baseline assumptions, and failures on challenging cases.Kalantari et al.'s method uses four corner views and can produce ghosting in occluded regions.

3 PROBLEM ANALYSIS AND FORMULATION

The paper formulates sparse light-field reconstruction as inverse angular restoration on EPIs, where angular undersampling damages detail and causes ghosting during direct super-resolution. Its blur-restoration-deblur formulation extracts spatial low frequencies before restoring angular detail and deblurring spatial detail.

  • Problem formulation: An EPI is a 2D light-field slice containing spatial and angular information, and the reconstruction task seeks an inverse operation from low- to high-angular-resolution EPIs.The low-angular-resolution EPI is obtained by downsampling the high-angular-resolution EPI.
  • Information asymmetry: When neighboring-view disparity exceeds 1 pixel, angular undersampling destroys high-frequency angular detail while preserving spatial information, creating information asymmetry.Dense sampling satisfies the Nyquist criterion when neighboring-view disparity does not exceed 1 pixel.
  • Information asymmetry: Direct angular super-resolution cannot reconstruct continuous EPI structures in undersampled cases, and the resulting aliasing produces ghosting effects.The paper illustrates this with a continuous ground-truth line that remains discontinuous after direct super-resolution.
  • Proposed formulation: The proposed framework first extracts spatial low-frequency information with a 1D EPI blur, then restores angular detail and finally recovers suppressed spatial high frequencies with non-blind deblurring.The blur step is designed to avoid information asymmetry while retaining the original spatial resolution.

4 PROPOSED FRAMEWORK

The framework reconstructs full light fields by restoring angular EPI detail after spatial low-frequency extraction, then deblurring the result. It uses hierarchical view generation, a residual CNN, and a Gaussian blur kernel selected for low error without ringing artifacts.

  • 4.1 Overview: The blur-restoration-deblur pipeline extracts spatial low frequencies, upsamples angularly, restores angular detail with a CNN, and deblurs the spatial frequencies suppressed by blur.The EPI structure allows sparse reconstruction to be modeled as learning-based angular information restoration.
  • 4.1 Overview: A two-step hierarchical strategy reconstructs the full light field by generating horizontal- and vertical-view novel images first, then using generated views to produce the remaining views.Step 1 uses horizontal EPIs for green views and vertical EPIs for blue views; Step 2 produces yellow views.
  • 4.2 Low-frequency extraction based on EPI blur: The method uses a 1D spatial blur kernel because EPI blur extracts low frequencies only along the spatial dimension; Gaussian blur is selected after comparison with sinc and Butterworth kernels.On the Stanford Light Field Archive, the reported errors are 0.153, 0.089, and 0.061 for sinc, Butterworth, and Gaussian kernels, respectively.
  • 4.3.1 CNN architecture: The detail-restoration CNN has three convolution layers, with ReLU after the first two, and adds the predicted residual detail to the input EPI.The residual design focuses learning on angular details damaged by undersampling rather than relearning low-frequency content.
  • 4.3.1 CNN architecture: The residual network uses 64 filters of size 1 × 9 × 9, 32 filters of size 64 × 5 × 5, and one 32 × 5 × 5 output filter for detail reconstruction.The first layer extracts features, the second performs nonlinear mapping, and the last reconstructs detail.
  • 4.3.2 Training details: Training uses Stanford Light Field Archive data with augmentation, factors of 2 and 4, cascading for factor-4 upsampling, more than 8e6 examples, and mini-batches of 64.The network processes only the luminance channel after transforming EPIs to YCbCr space.
  • 4.3.2 Training details: Angular undersampling creates Fourier-spectrum copies, and direct CNN super-resolution leaks high-frequency content from those copies; band-limited filtering is depth-limited.The analysis motivates separating low-frequency extraction from angular detail restoration.

5 FOURIER ANALYSIS

The Fourier analysis explains how angular undersampling creates aliasing copies and motivates blur-restoration-deblur as a way to restore angular detail without retaining those copies.

  • 5 FOURIER ANALYSIS: Angular undersampling destroys high-frequency angular details for large disparities and produces Fourier-spectrum copies that cause aliasing and ghosting during direct super-resolution.The effect is weaker for small disparities but severe when disparity exceeds one pixel.
  • 5 FOURIER ANALYSIS: A depth-dependent band-limited filter can remove leakage, but its shape cannot reconstruct light fields across all scene depths.This limitation motivates balancing spatial and angular information before learning-based restoration.
  • 5 FOURIER ANALYSIS: The blur step suppresses spatial high-frequency components and efficiently removes high-frequency Fourier copies from the undersampled EPI.A Gaussian kernel is selected according to the highest disparity of the light field.
  • 5 FOURIER ANALYSIS: The restoration CNN recovers high-frequency angular components while leaving aliasing copies unchanged, so it restores detail rather than hallucinating aliased frequencies.The framework uses at least three views in each angular dimension for restoration.
  • 5 FOURIER ANALYSIS: The final deblur operation recovers spatial high frequencies, and the complete framework produces a super-resolved EPI without aliasing.The deblur step is the inverse operation of the selected blur kernel.

6 EVALUATION

The evaluation compares the framework with learning-based and depth-based alternatives across real-world, microscope, and synthetic light fields, including ablations of its three stages.

  • 6.1 Real-world scenes: On real-world scenes, the proposed framework is significantly better than other approaches, while single-CNN variants produce lower-quality light fields than the complete framework.The comparison covers 30 scenes and additional reflective and occlusion cases.
  • 6.1 Real-world scenes: The proposed method reduces blurring, ghosting, discontinuities, and occlusion-region artifacts in challenging real-world scenes involving complex structures and reflective surfaces.The cited examples include Leaves, Reflective 29, and Occlusion 16.
  • 6.2 Microscope light field datasets: On microscope light fields, competing methods show blur, depth failures, tearing, or artifacts on complicated occlusions and translucency, whereas the proposed method achieves higher visual coherency.The evaluation uses microscope datasets with reconstructed angular resolutions from sparse inputs.
  • 6.3 Synthetic scenes: Synthetic HCI experiments show that the framework handles different input sparsity levels, and learning-based restoration is especially beneficial when the input is sparser.The outputs are evaluated at 9 × 9 angular resolution using 3×3 and 5×5 inputs.
  • 6.3 Synthetic scenes: The synthetic evaluation omits quantitative results for the Worm and Cells cases because all ground-truth views are used as input.This is an evaluation-scope boundary rather than a reported performance result.

7 EXTENDED APPLICATIONS

The framework is extended to depth enhancement, unstructured light-field interpolation, and depth-assisted rendering, including large-disparity cases handled with disparity-guided shearing.

  • 7.1 Depth enhancement: Reconstructed high-angular-resolution light fields produce more accurate depth maps with better edge preservation than competing reconstructions on synthetic and real-world scenes.Examples include reflective surfaces and branches in occluded regions.
  • 7.2 Interpolation for unstructured light field: For unstructured light fields, the depth-free EPI framework avoids geometry calibration and depth estimation, enabling angular super-resolution from irregular camera inputs.The method implicitly reconstructs the light field by restoring angular details on the EPI.
  • 7.2 Interpolation for unstructured light field: On the Basket dataset, the proposed interpolation reaches an averaged PSNR of 41.60, versus 40.43 for Meyer et al. and 33.82 for Sun et al.The experiment uses 25 of 49 views as input.
  • 7.3 Depth-assisted rendering: Depth-assisted rendering handles disparities up to 40 pixels and produces high-quality views in occluded and non-Lambertian regions, where a DIBR baseline shows ghosting at occlusion boundaries.Table 5 reports disparity range, RMSE, PSNR, and MS-SSIM results across stereo and outdoor light-field datasets.
  • 7.3 Depth-assisted rendering: For large disparities, the rendering method shears EPIs by discretized disparity values, applies blur-restoration-deblur, inversely shears the results, and blends the best regions.Regions matched to their corresponding disparity are retained while mismatched regions can contain aliasing.

8 DISCUSSION AND CONCLUSION

The discussion explains why EPI-based reconstruction generalizes across varied scenes and reports the framework’s performance, applications, and practical limits.

  • 8 DISCUSSION AND CONCLUSION: EPI structures remain similar across different scenes, helping reduce the data dependence that limits networks trained directly on sub-aperture images.This similarity supports reconstruction across real-world and microscopy light fields with different appearances.
  • 8 DISCUSSION AND CONCLUSION: The blur-restoration-deblur framework extracts spatial low frequencies, restores angular details with a CNN, and recovers suppressed spatial details through non-blind deblurring.The blur and deblur operations address the information asymmetry that otherwise causes angular ghosting.
  • 8 DISCUSSION AND CONCLUSION: The framework outperforms state-of-the-art approaches in occluded and transparent regions and on non-Lambertian surfaces across evaluated datasets.Experiments include synthetic, real-world, and challenging microscope light field data.
  • 8 DISCUSSION AND CONCLUSION: Extended applications use reconstructed high-angular-resolution light fields for depth enhancement, unstructured-input interpolation, and depth-assisted rendering.Depth-assisted rendering extends handling to large-disparity data such as multi-view stereo inputs.
  • 8 DISCUSSION AND CONCLUSION: The framework requires at least three views per angular dimension, cannot extrapolate, and handles disparities up to 5 pixels without depth assistance.Unstructured reconstruction also assumes vertical disparities below one pixel because blur and deblur operate in one direction.
  • 8 DISCUSSION AND CONCLUSION: The method fails when texture frequency exceeds the spatial sampling rate, although its CNN can suppress noise and produce smoother outputs.The reported denoising effect appears in the Neurons 20× and Neurons 40× cases.
Loading 2103.13043v1…