PaperlayerView exact revision on arXiv

Source-linked AI summary

Light Field Image Super-Resolution Using Deformable Convolution

Yingqian Wang, Jungang Yang, Longguang Wang, Xinyi Ying, Tianhao Wu, Wei An, Yulan Guo

arXiv:2007.03535v4eess.IVcs.CV

TL;DR

LF image SR must exploit angular information without being disrupted by disparities among views. The paper proposes LF-DFnet with deformable feature alignment and a baseline-adjustable dataset, reporting state-of-the-art reconstruction accuracy and greater robustness to disparity variations.

  • Problem

    Disparities among LF images make angular information difficult to incorporate, while low-resolution sub-aperture images require high-resolution reconstruction.

  • Method

    LF-DFnet uses an angular deformable alignment module with collect-and-distribute bidirectional alignment to encode angular information into each view, alongside the baseline-adjustable NUDT dataset.

  • Results

    LF-DFnet achieves state-of-the-art reconstruction accuracy, with performance evenly distributed across angular views and greater robustness to disparity variations.

  • Takeaways & Limitations

    The method generates high-quality, angular-consistent high-resolution light fields that can benefit downstream tasks such as LF depth estimation.

Abstract

from arXiv · show

Light field (LF) cameras can record scenes from multiple perspectives, and thus introduce beneficial angular information for image super-resolution (SR). However, it is challenging to incorporate angular information due to disparities among LF images. In this paper, we propose a deformable convolution network (i.e., LF-DFnet) to handle the disparity problem for LF image SR. Specifically, we design an angular deformable alignment module (ADAM) for feature-level alignment. Based on ADAM, we further propose a collect-and-distribute approach to perform bidirectional alignment between the center-view feature and each side-view feature. Using our approach, angular information can be well incorporated and encoded into features of each view, which benefits the SR reconstruction of all LF images. Moreover, we develop a baseline-adjustable LF dataset to evaluate SR performance under different disparity variations. Experiments on both public and our self-developed datasets have demonstrated the superiority of our method. Our LF-DFnet can generate high-resolution images with more faithful details and achieve state-of-the-art reconstruction accuracy. Besides, our LF-DFnet is more robust to disparity variations, which has not been well addressed in literature.

I. INTRODUCTION

LF image SR must recover high-resolution sub-aperture images while exploiting angular information despite disparity among views. LF-DFnet addresses this with deformable feature alignment, and NUDT evaluates robustness across adjustable disparity variations.

  • LF cameras provide angular information, but low-resolution sub-aperture images hinder LF imaging and motivate reconstruction of high-resolution views.
  • Existing CNN-based LF SR methods use cascaded convolutions to cover disparities, but misalignment impedes angular-information incorporation and degrades performance.
  • LF-DFnet uses an angular deformable alignment module and collect-and-distribute approach for bidirectional feature-level alignment between center and side views.Collected side-view features are fused and distributed back to corresponding views through alignment with their original features.
  • The proposed method targets state-of-the-art LF image SR performance by explicitly addressing the disparity problem.
  • The NUDT dataset renders synthetic scenes with adjustable camera baselines to evaluate robustness under disparity variations.

A. Single Image SR

Single-image SR methods improve spatial reconstruction but discard angular information when applied independently to LF images. LF SR methods therefore seek to combine spatial and angular cues while handling disparity-induced misalignment.

  • Single-image SR generates a clear high-resolution image from a blurry low-resolution counterpart, an ill-posed mapping because multiple high-resolution outputs may correspond to one input.
  • Deep SISR networks increasingly improve spatial-information exploitation through greater depth and complexity.
  • Applying SISR independently to LF images is straightforward but discards angular information, limiting performance.
  • LF SR methods combine spatial and angular information using traditional models or deep CNNs, including SAI-wise super-resolution and EPI-based processing.
  • Existing LF SR methods lack effective disparity handling, so performance degrades in scenes with large disparities.
  • Deformable convolution learns offsets that let kernels process features far from their local neighborhoods, supporting adaptive alignment.
  • Existing deformable-convolution video SR methods use unidirectional alignment, whereas LF SR requires bidirectional alignment across views to avoid repetitive computation.

III. NETWORK ARCHITECTURE

LF-DFnet extracts shared per-view features, aligns side views with the center view through deformable convolution, and redistributes fused angular information back to each view. Cascaded ADAM modules repeatedly perform this collect-and-distribute process before reconstruction and upsampling.

  • Overall architecture: LF-DFnet converts light-field inputs into a 4D tensor and processes LR SAIs through feature extraction, angular alignment, reconstruction, and upsampling.The network super-resolves the Y channel while bicubicly upscaling Cb and Cr channels.
  • Feature extraction: Shared residual ASPP and residual blocks extract hierarchical features independently from each view using dense, multi-scale sampling.The residual ASPP module combines parallel 3 × 3 dilated convolutions with dilation rates 1, 2, and 4.
  • Angular deformable alignment: ADAM first aligns every side-view feature to the center view, fuses the collected features with a 1 × 1 convolution, and then distributes fused information back to corresponding views.Feature collection and distribution both use deformable convolution, with shared weights for the two operations.
  • Angular deformable alignment: Deformable convolution adds learnable offsets to a 3 × 3 sampling neighborhood, allowing spatially variant kernels to process information beyond local positions.Fractional offsets are handled through bilinear interpolation.
  • Angular deformable alignment: The offset-generation branch concatenates side-view and center-view features, reduces feature depth, enlarges the receptive field with residual ASPP, and predicts offsets.This design targets complicated and large motions between side and center views.
  • Angular deformable alignment: Three cascaded ADAMs repeatedly incorporate angular information into the center view and propagate it to all side views, producing notable performance improvements.After distribution, both center-view and side-view features are available for subsequent processing.

C. Reconstruction & Upsampling Module

The reconstruction module fuses angular-aligned features with information multi-distillation blocks, then upsamples each reconstructed feature to generate super-resolved SAIs.

  • Reconstruction: The reconstruction module concatenates outputs from feature extraction and each ADAM, then processes them with stacked information multi-distillation blocks.The blocks progressively extract and process hierarchical features using a distillation mechanism.
  • Upsampling: The upsampling module expands feature depth, applies pixel shuffle to reach resolution αH × αW, and uses a final 1 × 1 convolution to produce one-channel SR SAIs.The preceding reconstruction features are first expanded to α^2C channels, where α is the upsampling factor.

IV. THE NUDT DATASET

The NUDT dataset is introduced to study LF algorithms under varying camera baselines, addressing the fixed-baseline limitation of existing datasets. It uses concentric camera configurations to align arrays at center views while changing baseline lengths.

  • The concentric configuration aligns camera arrays at their center views while supporting multiple baseline settings.
  • Existing LF datasets use fixed baselines, limiting evaluation under baseline variations.
  • The NUDT dataset introduces adjustable baselines for studying LF algorithms under different disparity conditions.

A. Technical Details

NUDT contains rendered LF scenes captured with controlled camera geometry and adjustable baselines. Its design provides diverse spatial and angular data for LF analysis.

  • NUDT contains 32 synthetic scenes with 9 × 9 angular resolution and 1024 × 1024 spatial resolution.The scenes are rendered using 3dsMax, and groundtruth depth maps are available.
  • Virtual cameras share identical internal parameters, lie on a common plane, and have parallel optical axes.
  • A concentric configuration captures LF images with different baselines while aligning camera arrays at the center views.

B. Comparison to Existing Datasets

The paper compares NUDT with popular LF datasets using spatial-resolution and no-reference perceptual-quality criteria. The comparison covers both individual-image and LF-level quality.

  • NUDT is compared with several popular LF datasets using spatial resolution and perceptual-quality measures.
  • The evaluation uses BRISQUE, NIQE, CEIQ, and ENIQA for center-view image quality.
  • NRLFQA is used to assess the no-reference quality of complete light fields.
  • Five public LF datasets are used for training and testing experiments.

V. EXPERIMENTS

Experiments evaluate LF-DFnet against single-image and LF image SR methods on public datasets using standard training settings and quantitative and visual comparisons. LF-DFnet achieves the strongest reconstruction accuracy across nearly all reported dataset and scale conditions, while remaining robust to larger disparities.

  • Experimental Setup: The experiments retrain competing deep-learning methods on the same datasets and include bicubic interpolation as a baseline.
  • Quantitative Results: LF-DFnet achieves the highest PSNR and SSIM on all five datasets for 4×SR and on four of five datasets for 2×SR.
  • Quantitative Results: For 2×SR, LF-DFnet attains state-of-the-art PI, while ESRGAN and resLF are slightly better on PI for 4×SR.PI evaluates perceptual quality but does not measure faithfulness of reconstructed images.
  • Quantitative Results: LF-DFnet maintains promising performance on small-baseline LFs and handles the larger disparity variations of STFgantry through deformable-convolution angular alignment.
  • Qualitative Results: The method produces more faithful details and fewer artifacts than compared SISR and LF image SR methods for 2× and 4×SR.For 2×SR, it faithfully recovers the stairway in INRIA Sculpture and horizontal stripes in HCInew Origami.

2) Qualitative Results:

LF-DFnet achieves strong reconstruction quality while remaining computationally efficient, and its deformable alignment is especially valuable under large disparity variations. It also maintains relatively balanced quality across angular perspectives.

  • Qualitative Results: LF-DFnet recovers missing details by using complementary information among different views, avoiding the unfaithful textures produced by some perceptual single-image SR methods.The authors attribute the benefit to the ill-posed nature of 4×SR, where a single image lacks sufficient spatial information.
  • Computational Efficiency: LF-DFnet achieves the highest reconstruction accuracy with fewer parameters and FLOPs than EDSR and RCAN.Its computational cost is slightly higher than resLF and LF-InterNet because its more complex feature extraction and reconstruction modules improve performance.
  • Disparity Variations: LF-DFnet remains relatively insensitive to disparity variations and matches RCAN when Kd = 4, while LFSSR and resLF degrade as disparity increases.Replacing deformable convolutions with regular convolutions causes notable degradation, especially for Kd > 2.
  • Perspectives: LF-DFnet achieves improved reconstruction accuracy with a relatively balanced PSNR distribution across different angular perspectives.The comparison includes four state-of-the-art methods and variants without deformable convolution, feature distribution, or ADAM.

6) Performance on Real-World LF images:

On real-world LF images, LF-DFnet produces finer details and fewer artifacts than the compared methods. Its angular-consistent outputs also support depth estimation close to that obtained from original high-resolution LFs.

  • Real-World Visual Results: LF-DFnet recovers finer details than RCAN and produces fewer artifacts than ESRGAN on real-world LF images.The comparison uses EPFL images without ground-truth high-resolution references, so performance is assessed visually.
  • Depth Estimation: LF-DFnet improves depth estimation by generating high-resolution and angular-consistent LF images.Single-image SR methods can achieve strong SR metrics yet provide comparable or worse depth estimation because they process each SAI independently.
  • Depth Estimation: Depth estimation from LF-DFnet outputs is very close to that obtained from original high-resolution LFs.This indicates strong spatial reconstruction quality and angular consistency for the evaluated HCIold setting.

C. Ablation Study

The ablation study shows that deformable convolution, ADAM, and bidirectional feature distribution each materially contribute to LF-DFnet’s reconstruction performance. Angular information and its propagation to side views are particularly important.

  • Deformable Convolution: Replacing deformable convolutions with regular 3 × 3 convolutions decreases average PSNR by 0.57 dB.The degradation is more significant for datasets with wide baselines, supporting the role of deformable alignment under larger disparities.
  • ADAM: Removing all ADAMs reduces average PSNR to 35.92 dB versus 38.42 dB for LF-DFnet.The no-ADAM variant behaves like an SISR model that uses only spatial information within each view.
  • Overall Ablation Findings: The ablations demonstrate that angular information is important for LF image SR and that bidirectional interaction improves reconstruction.The study evaluates variants without deformable convolution, ADAM, or feature distribution to isolate these design choices.
  • Feature Distribution: Removing feature distribution decreases average PSNR by 1.85 dB compared with LF-DFnet.Feature collection alone propagates angular information to the center view but does not distribute the incorporated features back to side views.

2) Residual ASPP Module:

The study evaluates residual ASPP modules and the number of ADAMs as architectural choices. Reconstruction accuracy improves with more ADAMs but saturates after three, while removing residual ASPP from feature extraction reduces PSNR.

  • Residual ASPP Module: Removing the residual ASPP module from feature extraction decreases average PSNR by 0.21 dB.The comparison replaces residual ASPP blocks with residual blocks in the feature extraction module.
  • Number of ADAMs: Reconstruction accuracy consistently improves as the number of ADAMs increases, but gains tend to saturate from 3 to 4 ADAMs.The authors select K = 3 as a tradeoff between reconstruction accuracy and the linearly increasing parameter count.
  • Conclusion: LF-DFnet achieves state-of-the-art SR performance with a small computational cost and more even accuracy across angular views.The method is also reported to be more robust to disparity variations and effective on public and self-developed datasets.
  • Conclusion: LF-DFnet works on real-world LF images and generates high-resolution, angular-consistent LFs that benefit downstream tasks such as depth estimation.The paper identifies post-capture refocusing and depth sensing as potential applications.
  • Limitations: The visual superiority of LF-DFnet remains minor because training uses only L1 loss and the input LR images can be very low quality.The authors propose GAN-based LF SR as future work for better perception-distortion tradeoffs and larger scaling factors.
Loading 2007.03535v4…