Source-linked AI summary
Spatial-Angular Interaction for Light Field Image Super-Resolution
Yingqian Wang, Longguang Wang, Jungang Yang, Wei An, Jingyi Yu, Yulan Guo
TL;DR
LF image SR must recover high-resolution sub-aperture images while exploiting both spatial information within views and angular information across views, which existing methods may not fully use. LF-InterNet decouples spatial and angular features and progressively interacts them before fusion for super-resolution. The paper reports superior reconstruction performance over state-of-the-art methods, with ablations showing contributions from both feature types and their interaction.
Problem
LF image SR seeks to reconstruct high-resolution sub-aperture images despite methods that may not fully exploit coupled spatial and angular information.
Method
LF-InterNet extracts and decouples spatial and angular features from MacPI-represented light fields, then progressively interacts them for reconstruction.
Results
LF-InterNet achieves superior performance over state-of-the-art SISR and LF image SR methods, while ablations show benefits from spatial information, angular information, and feature interaction.
Takeaways & Limitations
Spatial information plays a major role in LF image SR, angular information is complementary, and their interaction enables efficient use of the 4D light-field structure.
Abstract
from arXiv · showhide
Light field (LF) cameras record both intensity and directions of light rays, and capture scenes from a number of viewpoints. Both information within each perspective (i.e., spatial information) and among different perspectives (i.e., angular information) is beneficial to image super-resolution (SR). In this paper, we propose a spatial-angular interactive network (namely, LF-InterNet) for LF image SR. Specifically, spatial and angular features are first separately extracted from input LFs, and then repetitively interacted to progressively incorporate spatial and angular information. Finally, the interacted features are fused to superresolve each sub-aperture image. Experimental results demonstrate the superiority of LF-InterNet over the state-of-the-art methods, i.e., our method can achieve high PSNR and SSIM scores with low computational cost, and recover faithful details in the reconstructed images.
1 Introduction
LF image SR must exploit both within-view spatial information and cross-view angular information, but existing methods often use them incompletely. LF-InterNet decouples and progressively interacts these features to improve reconstruction efficiently.
- LF cameras trade off spatial resolution against angular sampling density, motivating LF image SR to reconstruct high-resolution sub-aperture images.
- Spatial neighborhood context helps single-image SR recover missing details, while complementary information across LF views can further improve LF image SR.
- Existing LF image SR methods may ignore angular information during upsampling, discard views, or fail to fully use spatial information.
- LF-InterNet separately extracts spatial and angular features, then repeatedly interacts them to incorporate both information progressively.
- The authors report extensive ablations and comparisons showing LF-InterNet's superiority over state-of-the-art SISR and LF image SR methods.
2 Related Works
Related work spans deep single-image SR methods and traditional or deep-learning approaches for LF image SR. Deep LF methods commonly super-resolve sub-aperture images separately before later angular refinement.
- Deep SISR progressed from shallow SRCNN to deeper VDSR, EDSR, RDN, RCAN, and SAN architectures with increasingly powerful spatial feature exploitation.
- Traditional LF image SR methods use depth, disparity, subspace, filtering, or graph optimization formulations to reconstruct high-resolution sub-aperture images.
- Deep LF image SR methods such as LFCNN and LF-DCNN super-resolve sub-aperture images separately, then fine-tune results to incorporate angular information.
- LFCNN and LF-DCNN use two stages and therefore do not use angular information during their first-stage super-resolution.
3 Method
LF-InterNet represents light fields as macro-pixels to decouple spatial and angular information, then progressively interacts these features to reconstruct high-resolution sub-aperture images. Its receptive field and reconstruction design address disparities across views while preserving information from multiple interaction stages.
- Spatial-Angular Feature Decoupling: A light field can be organized either as an angular collection of sub-aperture images or as a spatial collection of macro-pixels.The SAI representation indexes views by angular coordinates, whereas MacPI groups pixels sharing spatial coordinates across views.
- Spatial-Angular Feature Decoupling: LF-InterNet uses the MacPI representation because angular information is implicit and difficult to extract in an SAI array.The network therefore applies separate spatial and angular feature extractors to the MacPI input.
- Spatial-Angular Feature Decoupling: AFE extracts angular features with A×A convolution and stride A, while SFE extracts spatial features with 3×3 convolution, stride 1, and dilation A.These choices isolate each macro-pixel for angular extraction and each SAI for spatial extraction, decoupling the two information types.
- Spatial-Angular Feature Interaction: Multiple SFEs and AFEs are cascaded with interaction to enlarge the receptive field because scene depth causes view-dependent disparities.Grad-CAM visualizations show contributive regions whose EPI receptive field covers output EPI slopes, indicating disparity handling.
- Spatial-Angular Feature Interaction: Each Inter-Block upsamples angular features toward the spatial representation using a 1×1 convolution and pixel shuffling to learn discontinuities from edges and occlusions.The interaction block takes paired angular and spatial features as inputs for spatial-angular feature interaction.
Spatial-Angular Feature Interaction
LF-InterNet repeatedly exchanges spatial and angular features through cascaded interaction blocks, then fuses the resulting features to reconstruct HR SAIs.
- Spatial-Angular Feature Interaction: The interaction module processes spatial and angular feature pairs within cascaded Inter-Blocks.The output of each Inter-Block becomes the input to the subsequent block in an Inter-Group.
- Feature Fusion & Reconstruction: The bottleneck compresses concatenated angular features, upsamples them, and concatenates them with spatial features for fusion.A 1×1 convolution and ReLU produce a squeezed angular feature before its concatenation with spatial features.
- Feature Fusion & Reconstruction: After fusion, LF reshape converts MacPI-organized features into an SAI array representation for pixel shuffling.Pixels sharing angular coordinates are extracted and reorganized according to spatial coordinates before reconstruction.
- Feature Fusion & Reconstruction: A final 1×1 convolution reduces the feature channels to one for HR SAI reconstruction.This reconstruction follows bottleneck fusion, LF reshape, and pixel shuffling.
4 Experiments
Experiments evaluate LF-InterNet through dataset-based testing, ablations, comparisons with state-of-the-art methods, and tests under unseen and real-world conditions. Results show that combining spatial-angular features and their interaction improves reconstruction quality while preserving details and reducing computational cost.
- Datasets and Implementation Details: Experiments use six public LF datasets with 9×9 angular resolution and bicubic downsampling at factors of 2 and 4.Training uses 64×64 SAI patches reorganized into MacPI inputs, with PSNR and SSIM evaluated on the Y channel.
- Ablation Study: 29.98 PSNR and 0.897 SSIM are achieved by the spatial-only variant, significantly below LF-InterNet, demonstrating the value of angular information.The spatial-only network is equivalent to a SISR network and uses only information within each SAI.
- Ablation Study: Spatial information is essential: the angular-only variant performs worse than bicubic interpolation, while angular features serve as a complement rather than a standalone substitute.Removing spatial context by changing SFE kernels from 3×3 to 1×1 prevents convolutional integration of neighborhood information.
- Ablation Study: 29.84 PSNR and 0.894 SSIM are obtained without feature interaction, while increasing interaction count steadily improves reconstruction accuracy.Without repeated interaction, spatial and angular features can only be combined by the bottleneck block.
- Ablation Study: Pixel shuffling outperforms nearest upsampling, while bilinear upsampling performs worse because it introduces aliasing among macro-pixels.Pixel shuffling additionally models discontinuities within unevenly distributed macro-pixels caused by edges and occlusions.
- Comparison to the State-of-the-arts: 1.54 dB at 2×SR and 1.00 dB at 4×SR are the PSNR improvements over RCAN across six test datasets; LF-InterNet also matches LF-ATO comparably.The method preserves textures and details, generalizes to the unseen UCSD dataset, and produces faithful details with fewer artifacts under real-world degradation.
5 Conclusion and Future Work
LF-InterNet is a deep convolutional network for LF image SR that decouples spatial and angular features before interacting them. Experiments demonstrate superiority over state-of-the-art methods, while the framework is positioned for future angular and joint spatial-angular SR.
- LF-InterNet extracts and decouples spatial and angular features, then uses feature interaction to incorporate both types of information.
- Experimental results demonstrate LF-InterNet’s superiority over state-of-the-art methods.
- The spatial-angular interaction mechanism is described as a generic framework that processes light fields efficiently.
- Future work will apply LF-InterNet to LF angular SR and joint spatial-angular SR.