PaperlayerView exact revision on arXiv

Source-linked AI summary

Light Field Image Super-Resolution with Transformers

Zhengyu Liang, Yingqian Wang, Longguang Wang, Jungang Yang, Shilin Zhou

arXiv:2108.07597v2cs.CV

TL;DR

LF image SR must recover high-resolution light fields while exploiting angular and spatial dependencies that CNN-based methods do not fully model. LFT addresses this with angular and spatial Transformers, achieving strong performance across five datasets with a small model and low computational cost.

  • Problem

    Existing CNN-based LF SR methods cannot fully exploit angular information across views or long-range spatial dependencies within sub-aperture images.

  • Method

    LFT uses an angular Transformer to incorporate complementary information among views and a spatial Transformer to capture local and long-range context within each sub-aperture image.

  • Results

    LFT achieves competitive PSNR and SSIM on five datasets for 2× and 4× SR, including a 0.65 dB advantage over the second top-performing method on STFgantry.

  • Takeaways & Limitations

    The angular and spatial Transformer design provides superior SR performance with smaller model size and lower computational cost than compared methods.

Abstract

from arXiv · show

Light field (LF) image super-resolution (SR) aims at reconstructing high-resolution LF images from their low-resolution counterparts. Although CNN-based methods have achieved remarkable performance in LF image SR, these methods cannot fully model the non-local properties of the 4D LF data. In this paper, we propose a simple but effective Transformer-based method for LF image SR. In our method, an angular Transformer is designed to incorporate complementary information among different views, and a spatial Transformer is developed to capture both local and long-range dependencies within each sub-aperture image. With the proposed angular and spatial Transformers, the beneficial information in an LF can be fully exploited and the SR performance is boosted. We validate the effectiveness of our angular and spatial Transformers through extensive ablation studies, and compare our method to recent state-of-the-art methods on five public LF datasets. Our method achieves superior SR performance with a small model size and low computational cost. Code is available at https://github.com/ZhengyuLiang24/LFT.

I. INTRODUCTION

LF image SR needs to exploit complementary information across views while reconstructing high-resolution light fields. The paper proposes LFT, using angular and spatial Transformers to address limitations of CNN-based methods.

  • LF cameras capture intensity and light-ray directions, enabling applications that require high-resolution light-field images.
  • CNN-based LF SR methods either underuse angular information or rely on local convolutions that miss long-range spatial dependencies.
  • LFT uses an angular Transformer for relationships among views and a spatial Transformer for local and non-local context within each sub-aperture image.
  • The proposed paradigm is validated through extensive ablations and achieves superior SR performance with small model size and low computational cost.

II. RELATED WORK

Prior LF image SR work is dominated by CNN-based architectures that exploit epipolar geometry, sub-aperture patterns, or macro-pixel patterns. The paper frames Transformer-based processing as an alternative for modeling global relationships in LF data.

  • CNN-based LF SR methods have used independent SAI processing, epipolar geometry, and feature rearrangements across sub-aperture and macro-pixel patterns.
  • Vision Transformers represent images as token sequences and use self-attention to model relationships among all tokens rather than only local neighborhoods.
  • Transformer models have been applied to classification, detection, segmentation, depth estimation, and super-resolution.
  • The LF is formulated as a 4D tensor with angular dimensions U and V and spatial dimensions H and W, processed through feature extraction, Transformer-based incorporation, and up-sampling.

A. Angular Transformer

The angular Transformer converts extracted LF features into angular-token sequences, adds positional information, and applies multi-head self-attention to model relationships among views. Its output is reshaped and passed to the spatial Transformer.

  • A. Angular Transformer: Cascaded 3×3 convolutions produce initial LF features, which the angular Transformer processes to exploit complementary information across input views.
  • A. Angular Transformer: Features are reshaped into angular tokens with sequence length N_A=UV and embedding dimension d_A before angular positional encoding.
  • A. Angular Transformer: Angular positional encoding represents view positions using sinusoids whose wavelengths form a geometric progression, with α set to 10000 in the experiments.
  • A. Angular Transformer: Position-enriched tokens generate queries and keys through layer normalization, while values remain the original angular tokens for multi-head self-attention.
  • A. Angular Transformer: The attention output is further processed by a feed-forward network containing layer normalization and a multi-layer perceptron.
  • A. Angular Transformer: After angular processing, the tokens are reshaped into LF features and passed to the spatial Transformer for spatial-context incorporation.

B. Spatial Transformer

The spatial Transformer embeds local neighborhoods into overlapping tokens, combines local context with 2D positional information, and models both local and long-range dependencies within each SAI.

  • B. Spatial Transformer: The input feature is unfolded over 3×3 neighborhoods and embedded with an MLP before forming overlapping spatial tokens.This construction integrates local context into the spatial-token representation.
  • B. Spatial Transformer: Overlapping spatial tokens enable the Transformer to model both local and non-local spatial dependencies within each SAI.Feature unfolding and overlapped cropping preserve local context while supporting broader token relationships.
  • B. Spatial Transformer: Two-dimensional positional encoding supplies spatial-position information to the spatial tokens before attention processing.The encoding uses spatial coordinates and an embedding-dimension index, with α set to 10000.
  • B. Spatial Transformer: The spatial Transformer uses multi-head self-attention and a feed-forward network, then reshapes the output for subsequent angular processing and final up-sampling.After all angular and spatial Transformers, pixel shuffling produces the super-resolved LF image.

A. Implementation Details

The evaluation uses five public LF datasets with 5×5 angular resolution, PSNR and SSIM metrics, and training settings tailored to 2× and 4× SR.

  • A. Implementation Details: Five public LF datasets with 5×5 angular resolution are used, with 64×64/128×128 HR patches for 2×/4× SR and bicubic LR generation.The corresponding LR patches are 32×32.
  • A. Implementation Details: Performance is evaluated using PSNR and SSIM averaged across all SAIs and scenes in each dataset.For each dataset, scores average the metrics over M×A^2 SAI-level measurements.
  • A. Implementation Details: Training uses PyTorch with Adam optimization, Xavier initialization, batch sizes of 4/8, and an 80-epoch schedule for 2×/4× SR.The learning rate starts at 2×10^-4 and is halved every 15 epochs.
  • A. Implementation Details: Table I reports PSNR/SSIM values for different methods at both 2× and 4× SR, with best results marked in bold.The supplied table entries include multiple comparison methods and their PSNR/SSIM pairs.
  • A. Implementation Details: Figure 2 presents visual results for 2× SR from different methods.The figure caption identifies the comparison setting but does not report a numerical outcome.

B. Comparison to state-of-the-art methods

LFT is compared with retrained single-image and LF SR methods on common training datasets, achieving competitive results across five datasets and both scale factors.

  • B. Comparison to state-of-the-art methods: The comparison includes 3 single-image SR methods and 8 LF image SR methods, all retrained on the same datasets as LFT.This establishes a common retraining setup for the reported comparison.
  • B. Comparison to state-of-the-art methods: LFT achieves competitive PSNR and SSIM results on all five datasets for both 2× and 4× SR.The paper reports the comparison in Table I.
  • B. Comparison to state-of-the-art methods: 0.65 dB higher PSNR than the second top-performing method is reported for LFT on STFgantry at both 2× and 4× SR.The paper attributes this stronger advantage to the dataset’s complex structures and larger disparity variations.
  • B. Comparison to state-of-the-art methods: LFT maintains state-of-the-art performance on other datasets while handling the complex scenes in STFgantry.The authors connect this behavior to the proposed angular and spatial Transformers.

2) Qualitative Results:

Qualitative comparisons examine visual results for 4× SR alongside method metrics and model-complexity reporting, while LFT is described as preserving textures and details.

  • 2) Qualitative Results:: LFT preserves textures and details in super-resolved images and achieves competitive visual performance in the qualitative comparisons.The paper also provides a demo video for comparing angular consistency.
  • 2) Qualitative Results:: The reported method comparisons include LF-DFnet, LF-ATO, and LF-IINet with listed PSNR/SSIM values.The supplied entries include 30.59/0.903, 30.29/0.898, and 30.72/0.905, respectively.
  • 2) Qualitative Results:: Figure 3 shows visual results achieved by different methods for 4× SR.The supplied caption identifies the scale factor but does not state a specific visual winner.
  • 2) Qualitative Results:: Table II reports parameter counts, FLOPs, and average PSNR/SSIM scores for state-of-the-art methods at 2× and 4× SR.FLOPs are computed using an input LF of size 5×5×32×32, with best results marked in bold.

3) Efficiency:

LFT combines angular and spatial Transformers to exploit complementary information across views and long-range spatial dependencies, while achieving higher accuracy with a smaller model and lower computational cost. Ablation results show measurable gains from both Transformer components and angular positional encoding.

  • Efficiency: LFT achieves higher accuracy with smaller model size and lower computational cost than competing LF image SR methods.The comparison includes parameter counts and FLOPs against several competitive methods.
  • Ablation setup: The ablation table reports PSNR on EPFL, HCIold, and INRIA for 4×SR variants using angular and spatial Transformers and positional encodings.AngTr and SpaTr denote the Transformer components, while AngPos and SpaPos denote their positional encodings.
  • Angular Transformer: Adding the angular Transformer improves PSNR by 0.2∼0.3 dB over the baseline model.Angular positional encoding adds a further 0.1 dB improvement on the EPFL and HCIold datasets.
  • Angular Transformer: Removing the angular Transformer and angular positional encoding causes an approximately 0.2 dB PSNR drop.This ablation supports the contribution of angular modeling to SR performance.
  • Angular Transformer: The angular Transformer produces local angular similarity maps that compare each current view with all views at corresponding spatial locations.The attention maps visualize similarities among angular tokens across a 5×5 light field.
  • Angular Transformer: Compared with LF-InterNet, LFT's angular attention is contrasted with the CNN model's mainly adjacent-view correlations.LF-InterNet's view-wise correlation is described as not obvious beyond neighboring views.

2) Spatial Transformer:

The spatial Transformer improves LF image SR by modeling spatial information with a global receptive field. Its ablations show higher PSNR than the baseline and degradation when the module is removed.

  • Spatial Transformer: The spatial Transformer improves PSNR by 0.3 dB over the baseline model.This result comes from comparing model-4 with model-1.
  • Spatial Transformer: Removing the spatial Transformer reduces PSNR by 0.08∼0.27 dB.The comparison removes the spatial Transformer from model-3.
  • Spatial Transformer: Compared with cascaded convolutions, the spatial Transformer uses a global receptive field to exploit long-range context and beneficial spatial information.The paper attributes the ablation gains to this broader spatial modeling capability.
  • Spatial Transformer: LFT incorporates long-range spatial dependencies within each sub-aperture image through its spatial Transformer.The conclusion places spatial dependency modeling alongside complementary angular information from the angular Transformer.
Loading 2108.07597v2…