PaperlayerView exact revision on arXiv

Source-linked AI summary

Disentangling Light Fields for Super-Resolution and Disparity Estimation

Yingqian Wang, Longguang Wang, Gaochang Wu, Jungang Yang, Wei An, Jingyi Yu, Yulan Guo

arXiv:2202.10603v5eess.IVcs.CV

TL;DR

CNNs have difficulty processing LF images because spatial and angular information are intertwined under varying disparities. The paper introduces domain-specific convolutions and task-specific modules to disentangle LF features, and reports state-of-the-art performance across spatial SR, angular SR, and disparity estimation.

  • Problem

    CNNs face difficulty exploiting informative LF cues because spatial and angular information are highly intertwined with varying disparities.

  • Method

    The paper organizes LFs into disentangled subspaces with domain-specific feature extractors, then uses task-specific modules and networks for three LF processing tasks.

  • Results

    The proposed DistgSSR, DistgASR, and DistgDisp achieve state-of-the-art performance on spatial SR, angular SR, and disparity estimation.

  • Takeaways & Limitations

    The disentangling mechanism is presented as compact, effective, efficient, and applicable to multiple LF image processing tasks.

  • Takeaways & Limitations

    For LFEPICNN comparisons, the training code was unavailable, so the authors used its released model directly for inference.

Abstract

from arXiv · show

Light field (LF) cameras record both intensity and directions of light rays, and encode 3D scenes into 4D LF images. Recently, many convolutional neural networks (CNNs) have been proposed for various LF image processing tasks. However, it is challenging for CNNs to effectively process LF images since the spatial and angular information are highly inter-twined with varying disparities. In this paper, we propose a generic mechanism to disentangle these coupled information for LF image processing. Specifically, we first design a class of domain-specific convolutions to disentangle LFs from different dimensions, and then leverage these disentangled features by designing task-specific modules. Our disentangling mechanism can well incorporate the LF structure prior and effectively handle 4D LF data. Based on the proposed mechanism, we develop three networks (i.e., DistgSSR, DistgASR and DistgDisp) for spatial super-resolution, angular super-resolution and disparity estimation. Experimental results show that our networks achieve state-of-the-art performance on all these three tasks, which demonstrates the effectiveness, efficiency, and generality of our disentangling mechanism. Project page: https://yingqianwang.github.io/DistgLF/.

1 INTRODUCTION

The paper proposes a generic disentangling mechanism that incorporates LF structure priors to address CNNs’ difficulty processing intertwined spatial and angular information. It applies this mechanism across spatial SR, angular SR, and disparity estimation.

  • Motivation: CNNs struggle to exploit informative LF cues because spatial and angular information are intertwined under varying disparities.Existing methods reduce LF dimensionality using neighboring views, EPIs, or 3D sub-LFs, but rich angular information remains under-exploited.
  • Proposed mechanism: The method organizes LFs as macro-pixel images and uses domain-specific convolutions to extract disentangled features.Task-specific modules then fuse these features for different LF image processing tasks.
  • Properties: The mechanism incorporates all angular views, simplifies representation learning by processing separate 2D subspaces, and applies across LF tasks.These properties are presented as advantages over existing LF image processing frameworks.
  • Extensions: The work generalizes the prior spatial-angular interaction mechanism into a generic disentangling mechanism for LF data.The authors describe the proposed mechanism as more effective than LF-InterNet for handling LF data.
  • Networks: DistgSSR applies the mechanism to LF spatial super-resolution and is reported to outperform LF-InterNet with a smaller model size.The paper additionally develops DistgASR and DistgDisp for angular SR and disparity estimation.

2 RELATED WORK

Related work covers CNN and traditional approaches for LF spatial SR, angular SR, disparity estimation, and general LF processing. Existing architectures exploit different subsets of spatial or angular structure, motivating a mechanism that uses the full LF structure prior.

  • Spatial super-resolution: LF spatial SR must exploit both within-view spatial information and cross-view angular information, which independent SISR does not capture.Traditional methods include Gaussian mixture, linear subspace, BM3D-based, and graph-optimization approaches.
  • Spatial super-resolution: CNN-based spatial SR methods include independent or staged SAI processing, EPI enhancement, all-to-one designs, recurrent networks, multi-stream networks, and MacPI reshaping.These approaches progressively incorporate more LF structure through pairwise, directional, EPI, or reshaped processing.
  • Angular super-resolution: LF angular SR reconstructs densely sampled views from sparse inputs, with methods divided into depth-dependent and depth-independent categories.Depth-dependent methods warp and blend SAIs using estimated disparities, while depth-independent methods avoid explicit depth estimation.
  • Disparity estimation: Disparity estimation methods range from geometry, consistency, Fourier, and EPI-slope analysis to CNNs using 4D inputs, 3D convolutions, or multi-stream structures.EPI line slopes are related to disparity values and have motivated structure-tensor and parallelogram-based methods.
  • CNN architectures: Neighbor-view methods discard rich angular information, whereas EPI methods omit spatial context and directional coverage beyond horizontal or vertical views.Multi-stream networks use more angular views, but the paper positions its mechanism as incorporating all angular views and the LF structure prior.

3 THE LF DISENTANGLING MECHANISM

The disentangling mechanism reorganizes 4D LFs into MacPI form and applies spatial, angular, and EPI feature extractors to separate spatial-angular subspaces. Processing these 2D subspaces incorporates LF structure while reducing representation-learning complexity.

  • LF representation: A 4D LF is represented as L(u, v, h, w) with angular dimensions U,V and spatial dimensions H,W.SAI arrays expose spatial information, while angular information is distributed implicitly among views.
  • Feature extractors: Horizontal and vertical EPI extractors process slices whose line patterns encode spatial-angular correlations related to disparity.The horizontal extractor uses an asymmetric 1×A^2 kernel and stride A to preserve LF structure during convolution.
  • MacPI representation: A MacPI groups pixels sharing spatial coordinates across views into macro-pixels, then arranges those macro-pixels by spatial coordinates.The conversion from SAI to MacPI therefore makes angularly corresponding pixels explicit in the representation.
  • Mechanism: The extractors disentangle LFs into different 2D subspaces, reducing the complexity of 4D feature processing and easing LF representation learning.They are combined into task-specific modules and can be stacked to deepen networks.
  • Feature extractors: The spatial feature extractor convolves pixels from the same view, while the angular feature extractor convolves pixels within the same macro-pixel.Their kernels and strides isolate spatial and angular subspaces in the MacPI pattern.

4 DISTGSSR: DISENTANGLING MECHANISM FOR SPATIAL SUPER-RESOLUTION

The paper applies its disentangling mechanism to LF spatial super-resolution through a network named DistgSSR. The section presents the network design and experimental results.

  • 4 DISTGSSR: DISENTANGLING MECHANISM FOR SPATIAL SUPER-RESOLUTION: The disentangling mechanism is applied to LF spatial super-resolution.
  • 4 DISTGSSR: DISENTANGLING MECHANISM FOR SPATIAL SUPER-RESOLUTION: The resulting network is named DistgSSR.
  • 4 DISTGSSR: DISENTANGLING MECHANISM FOR SPATIAL SUPER-RESOLUTION: The section covers DistgSSR’s network design and experimental results.

4.1 Network Design

DistgSSR converts low-resolution SAI inputs into MacPI features, processes them with disentangled spatial, angular, and EPI branches, and reconstructs super-resolved SAIs. Its design addresses disparity across views and performs spatial upsampling through reshaping, convolution, and pixel shuffling.

  • DistgSSR converts an LR SAI array into an LR MacPI and processes it with spatial, angular, and EPI convolutions.
  • Four residual disentangling groups, each containing four residual disentangling blocks, form the network’s residual-in-residual design.The design follows RCAN for improved super-resolution performance.
  • Distg-Block for SSR: Each Distg-Block uses parallel angular, spatial, and two orthogonal EPI branches to disentangle and incorporate complementary LF features.The spatial branch applies two sequential spatial feature extractors, while EPI branches target the “U-H” and “V-W” subspaces.
  • Distg-Block for SSR: EPI branches address disparity by extracting features across orthogonal subspaces, while angular-to-spatial upsampling uses 1×1 convolution and 2D pixel shuffling.The design targets discontinuities caused by uneven pixel distribution from edges and occlusions.
  • Spatial Upsampling: After branch fusion and residual learning, DistgSSR reshapes MacPI features to SAI format, expands channels to α^2C, and applies 2D pixel shuffling.A final 1×1 convolution squeezes the channels to one and produces super-resolved SAIs at αAH × αAW resolution.

4.2 Experiments

Experiments evaluate DistgSSR through datasets, implementation details, ablations, and comparisons with state-of-the-art methods. Results show that angular and EPI information improve reconstruction, while disentangling achieves strong quality and efficiency.

  • Datasets and Implementation Details: DistgSSR was trained and evaluated on five public 9×9 LF datasets using consistent splits, augmentation, PSNR/SSIM metrics, and Adam optimization.Training used cropped 32 × 32 LF patches, eightfold geometric augmentation, L1 loss, and 50 epochs.
  • Ablation Study: 2.79dB lower PSNR resulted when only spatial features were used, with model-1 reaching 36.13 versus DistgSSR’s 38.92.The comparison demonstrates the importance of angular information for LF spatial super-resolution.
  • Ablation Study: 0.30dB lower PSNR resulted without EPI branches, increasing to 0.41dB on STFgantry, where disparity variations are larger.EPI feature extraction improves robustness because it models spatial-angular correlations under varying disparities.
  • Ablation Study: A 22.3M-parameter 4D-convolution variant reached 38.61dB for 2×SR, 0.31dB below DistgSSR, without improving performance.The authors attribute the advantage to disentangling LF data into subspaces for specialized feature extraction.
  • Ablation Study: Performance improved as angular resolution increased, but gains tended to saturate from 7×7 to 9×9 because 7×7 views already provide sufficient angular information.Additional views provide richer angular information, while further increases yield only minor improvements.
  • Comparisons with State-of-the-art Methods: DistgSSR produced more faithful details, fewer artifacts, straighter EPI lines, and the highest PSNR with relatively few parameters and reasonable FLOPs.It also remained better than LF-ATO and LF-InterNet when its feature depth was halved to 32.

5 DISTGASR: DISENTANGLING MECHANISM FOR ANGULAR SUPER-RESOLUTION

The paper applies its disentangling mechanism to LF angular super-resolution through a network named DistgASR. The section presents the network design and experimental results.

  • 5 DISTGASR: DISENTANGLING MECHANISM FOR ANGULAR SUPER-RESOLUTION: DistgASR applies the proposed disentangling mechanism to LF angular super-resolution.The network is introduced as a dedicated model for reconstructing angular views.

5.1 Network Design

DistgASR converts sparse LF inputs into dense angular outputs using MacPI processing, disentangling blocks, multi-stage fusion, and a non-integer-compatible upsampling design.

  • Network Overview: The input SAI array is converted into a MacPI for separate spatial, angular, and EPI feature extraction.The network contains four Distg-Groups, each with four Distg-Blocks, followed by 1×1-convolution fusion.
  • Distg-Block: Each Distg-Block separately processes angular and EPI information, concatenates them with the input, and then incorporates spatial information through a residual block with two SFEs.The ASR block emphasizes angular and EPI information because novel views must be reconstructed from sparse inputs.
  • Angular Upsampling: To support non-integer angular upsampling, DistgASR first downsamples angular features, expands channels, and applies 2D pixel shuffling for the target resolution.The approach divides β× upsampling into A× downsampling and βA× upsampling, where βA is an integer.
  • Angular Upsampling: After pixel shuffling, each feature pixel corresponds to a βA × βA × C macro-pixel, which is converted back into SAIs after spatial feature extraction.This final MacPI-to-SAI conversion produces the output angular array.

5.2 Experiments

The angular SR experiments evaluate DistgASR on synthetic and real-world datasets, including ablations of its spatial, angular, and EPI branches. Results show that combining these branches improves reconstruction quality, angular consistency, and detail preservation.

  • Experimental Setup: The experiments use HCInew, HCIold, 30scenes, and STFlytro datasets, with 2×2 corner views reconstructing the remaining views in a 7×7 angular grid.Training used 20 synthetic and 100 real-world scenes; testing covered scenes from all four datasets.
  • Ablation Study: Removing the angular and EPI branches decreases average PSNR by 3.16dB compared with DistgASR.The spatial-only variant cannot incorporate angular information.
  • Ablation Study: Removing the spatial and EPI branches decreases average PSNR by 2.33dB compared with DistgASR.The angular-only variant cannot incorporate spatial information.
  • Ablation Study: Removing the EPI branches decreases average PSNR by 0.86dB because sparse inputs create larger disparities between adjacent views.EPI feature extractors further disentangle LF features into EPI subspaces, allowing disparity-aware convolution.
  • Ablation Study: Removing weight sharing between horizontal and vertical EPI branches slightly reduces performance, supporting weight sharing as a regularizer for LF parallax structure.The comparison is made against the main DistgASR model.
  • Comparison Results: DistgASR achieves higher PSNR than competing methods on 4 of 5 datasets, while preserving delicate structures, textures, and linear parallax patterns.It also improves disparity estimation when its reconstructed LFs are processed by SPO.

6 DISTGDISP: DISENTANGLING MECHANISM FOR DISPARITY ESTIMATION

This section applies the proposed disentangling mechanism to LF disparity estimation and introduces the DistgDisp network.

  • Section Overview: DistgDisp is a network that applies the disentangling mechanism to LF disparity estimation.The following subsections describe its design and experiments.

6.1 Network Design

DistgDisp reorganizes 9×9 LF views into a MacPI, extracts spatial and disparity-selective angular features, constructs cost volumes, and regresses disparity after 3D aggregation.

  • Network Design: DistgDisp takes a 9×9-angular-resolution MacPI through spatial feature extraction, cost volume construction, cost aggregation, and disparity regression.The overview describes the complete processing sequence.
  • Spatial Feature Extraction: Spatial residual blocks model relationships between spatially adjacent pixels, which is important for specular and textureless regions.Each block uses SFE, batch normalization, LeakyReLU, a second SFE and batch normalization, followed by a local residual connection.
  • Cost Volume Construction: Disparity-selective angular feature extractors convolve pixels corresponding to specific disparities to construct cost volumes without repetitive shifting.This directly addresses the computational inefficiency of shift-and-concat cost-volume construction.
  • Cost Volume Construction: The method uses nine preset disparity levels from -4 to 4 and concatenates the resulting features into a cost-volume tensor.For zero disparity, DS-AFE reduces to an angular feature extractor operating within a macro-pixel.
  • Cost Aggregation and Disparity Regression: Eight cascaded 3D convolutions aggregate the matching costs into Ffinal ∈ R^D×H×W, and softmax-weighted disparity levels produce the final estimate.The preset disparity set is D = {-4, -3, · · ·, 3, 4}.

6.2 Experiments

DistgDisp is evaluated on the 4D LF benchmark and real-world light fields. It achieves strong disparity accuracy, fast inference, and visually sharper estimates with fewer artifacts.

  • Experimental Setup: The 4D LF benchmark uses 9×9 angular views and 512×512 spatial resolution, with separate training, validation, and test scene splits.Evaluation uses MSE and BadPix thresholds of 0.07, 0.03, and 0.01.
  • Quantitative Results: DistgDisp ranks first on 10 of 12 scenes for both MSE and BadPix0.01.These results are reported across the benchmark validation and test scenes.
  • Quantitative Results: Among 81 submissions, DistgDisp ranks in the top four across five major error metrics and first in average running time.The five metrics are BadPix0.07, BadPix0.03, BadPix0.01, MSE, and Q25.
  • Visual Comparison: Visual comparisons show sharper disparity boundaries and smoother surfaces, with larger improvements in heavily occluded or noisy scenes.The reported examples include nested structures in Boxes and noisy dots in Dots.
  • Efficiency: DistgDisp runs in 4 milliseconds per scene and is faster than other methods by an order of magnitude.The efficiency is attributed to DS-AFE cost-volume construction without repetitive feature shifting.
  • Real-World Evaluation: On real-world LFs without ground-truth disparities, the synthetic-trained model produces sharper edges and fewer artifacts than Epinet-fcn.The authors report that this demonstrates generalization to real light fields.

7 CONCLUSION

The proposed disentangling mechanism supports three light-field processing tasks through DistgSSR, DistgASR, and DistgDisp, with experiments demonstrating its effectiveness and efficiency. Sequential use of these networks reconstructs a high-resolution dense light field from a low-resolution sparse one while estimating scene disparities for refocusing.

  • 7 CONCLUSION: The mechanism uses spatial, angular, and epipolar feature extractors to disentangle light fields into different subspaces.It is designed to learn inherent light-field structure characteristics efficiently.
  • 7 CONCLUSION: Three networks—DistgSSR, DistgASR, and DistgDisp—address spatial super-resolution, angular super-resolution, and disparity estimation, respectively.
  • 7 CONCLUSION: Extensive experiments demonstrate the effectiveness and efficiency of the proposed disentangling mechanism across these light-field processing tasks.
  • 7 CONCLUSION: Sequentially applying the three networks reconstructs an HR dense light field from an LR sparse light field and estimates scene disparities for refocusing.Estimated disparities enable refocusing light fields to arbitrary regions.
Loading 2202.10603v5…