Detail-Preserving Transformer for Light Field Image Super-Resolution

Introduction

• Light-field cameras trade angular sampling against spatial resolution, limiting practical applications and motivating spatial super-resolution of low-resolution views.The work focuses on increasing spatial resolution rather than synthesizing new angular views.Source: Introduction · p. 1
• Prior light-field super-resolution methods exploit complementary information across views, but their convolutional architectures lack strong global relation modeling.Independent single-image super-resolution can neglect angular redundancy and produce angularly inconsistent reconstructions.Source: Introduction · p. 1
• DPT reformulates light-field super-resolution as sequence-to-sequence learning over sub-aperture images captured along horizontal or vertical directions.Self-attention models the intrinsic geometric structure of each sequence.Source: Introduction · pp. 1–2
• SA-LSA first strengthens locality with convolutions, then promotes non-local spatial-angular dependencies within each sub-aperture image sequence.This design addresses vanilla Transformers’ risk of damaging image structures with patch tokens and their neglect of spatial locality.Source: Introduction · p. 2
• DPT uses content and gradient Transformers to learn from light-field images and their gradients, then fuses their features with cross-attention for reconstruction.The two branches provide content and gradient representations for the high-resolution light field.Source: Introduction · p. 2
• DPT reports promising performance on multiple benchmarks while maintaining parameter counts and computational cost similar to convolution-based networks.The architecture is designed to mine non-local context across views while preserving details within each view.Source: Introduction · p. 2

Related Work

• Early deep learning methods often used separate subnetworks for spatial super-resolution of each sub-aperture image and angular view interpolation.Later approaches sought to capture spatial and angular information within unified frameworks.Source: Related Work · p. 2
• Unified light-field methods include recurrent networks, multi-branch models, deformable feature alignment, and spatial-angular interaction networks.The passage identifies long-range dependencies among multiple sub-aperture images as a major drawback of these approaches.Source: Related Work · p. 2
• Image-super-resolution Transformers address either intra-frame dependencies for single images or inter-frame context for multiple images, while this work proposes the first Transformer-based architecture for light-field super-resolution.The cited multiple-image examples include reference-based and video super-resolution.Source: Related Work · p. 2

Our Approach

• DPT treats the light field and its gradient field as horizontal and vertical angular sequences, processes them separately, then fuses their features for reconstruction.Processing sequences independently avoids expensive computation and redundant interactions among irrelevant sub-aperture images.Source: Our Approach · pp. 2–3, Detail-Preserving Transformer (DPT) · p. 4

Detail-Preserving Transformer (DPT)

• Content and gradient Transformers use spatial-angular attention blocks to enrich convolutional features with non-local context across sub-aperture images.The content Transformer uses K blocks to aggregate contextual knowledge among sub-aperture images; the gradient Transformer operates similarly on gradient features.Source: Detail-Preserving Transformer (DPT) · p. 3

Spatial-Angular Locally-Enhanced Self-Attention

• SA-LSA combines convolutional tokenization, self-attention, and de-tokenization to capture local spatial context and global spatial-angular context.Convolutional tokenization is intended to capture local context from low-level features to higher-level representations.Source: Spatial-Angular Locally-Enhanced… · pp. 4–5

Detailed Network Architecture

• The detailed network uses two CNN feature extractors, two-block content and gradient Transformers by default, and a cascaded reconstruction module.Each feature extractor has two residual blocks and two residual atrous spatial pyramid pooling blocks; reconstruction uses five information multi-distillation blocks followed by pixel-shuffle upsampling.Source: Detailed Network Architecture · p. 5

Datasets and Evaluation Metrics

• Evaluation covers EPFL, HCInew, HCIold, INRIA, and STFgantry, using PSNR and SSIM averaged across scenes and angular views; source light fields have 9×9 angular resolution.The final metric averages values over T scenes and U × V sub-aperture images.Source: Datasets and Evaluation Metrics · p. 5

Implementation Details

• For ×2/×4 SR, experiments use 5×5 angular resolution, derive gradients from each SAI’s Y channel, and bicubic-upsample Cb and Cr.The model super-resolves only the Y channel.Source: Implementation Details · p. 5

Comparisons with State-of-the-Art

• At ×4, DPT outperforms single-image and CNN-based methods and surpasses LF-DFNet on EPFL, HCIold, and INRIA; visual examples preserve structures distorted by competing methods.Table 1 compares methods for ×2 and ×4 SR; Figure 3 shows Bicycle and Sculpture scenes.Source: Comparisons with State-of-the-Art · p. 5, Table 1 · p. 6, Figure 3 · p. 7
• DPT has fewer parameters and better accuracy than LF-DFNet for ×4 SR on EPFL, yielding the reported accuracy-efficiency trade-off.The computational comparison reports parameters, FLOPs, and reconstruction accuracy.Source: Comparisons with State-of-the-Art · p. 6, Table 2 · p. 6

Ablation Study

• The content Transformer alone has fewer parameters than the LF-DFNet baseline at similar performance, while adding the gradient Transformer improves PSNR by 0.09 dB and SSIM by 0.0011.Dual-Transformer models also outperform the content-only model; replacing the Transformers with residual blocks causes a 0.21 dB PSNR drop.Source: Ablation Study · p. 6
• Cross-attention fusion improves PSNR and SSIM over element-wise summation and single-view non-local fusion, attributed to complementary non-local content and gradient information..Source: Ablation Study · p. 6
• SA-LSA performs better with fewer parameters than vanilla fully connected self-attention, and DPT achieves its best results with K = 2 attention blocks.The authors select K = 2 as the default.Source: Ablation Study · p. 7

• DPT is presented as the first Transformer application to LFSR, using SA-LSA to promote non-local spatial-angular dependencies in each sub-aperture image sequence.Source: Conclusion · p. 7
• Across five commonly used benchmarks, DPT achieves favorable performance against other competitors.Source: Conclusion · p. 7
• Content and gradient Transformers learn spatial-angular representations, which a cross-attention fusion Transformer aggregates for high-resolution light-field reconstruction.Source: Conclusion · p. 7