Source-linked AI summary
Light Field Spatial Super-resolution via Deep Combinatorial Geometry Embedding and Structural Consistency Regularization
Jing Jin, Junhui Hou, Jie Chen, Sam Kwong
TL;DR
LF spatial SR must recover spatial detail despite limited sensor resolution while preserving complex cross-view geometry, and existing methods do not fully exploit view complementarity or maintain structural consistency. The paper combines All-to-One combinatorial geometry embedding with structural consistency regularization, improving reconstruction quality and parallax preservation over prior methods, including an average PSNR gain of more than 1.0 dB at lower computational cost.
Problem
LF spatial SR is challenging because limited sampling reduces spatial resolution, while existing methods insufficiently exploit cross-view complementarity and preserve parallax structure.
Method
The framework super-resolves each view using All-to-One combinatorial geometry embedding, then applies structural consistency regularization to the intermediate LF reconstruction.
Results
The method improves average PSNR by more than 1.0 dB, preserves more accurate parallax details, and uses lower computational cost than state-of-the-art methods.
Takeaways & Limitations
Combining complete cross-view information fusion with structural regularization yields super-resolved LF images with higher PSNR/SSIM and better LF structure.
Takeaways & Limitations
The authors identify adversarial and perceptual losses for realistic textures as future directions for LF processing.
Abstract
from arXiv · showhide
Light field (LF) images acquired by hand-held devices usually suffer from low spatial resolution as the limited sampling resources have to be shared with the angular dimension. LF spatial super-resolution (SR) thus becomes an indispensable part of the LF camera processing pipeline. The high-dimensionality characteristic and complex geometrical structure of LF images make the problem more challenging than traditional single-image SR. The performance of existing methods is still limited as they fail to thoroughly explore the coherence among LF views and are insufficient in accurately preserving the parallax structure of the scene. In this paper, we propose a novel learning-based LF spatial SR framework, in which each view of an LF image is first individually super-resolved by exploring the complementary information among views with combinatorial geometry embedding. For accurate preservation of the parallax structure among the reconstructed views, a regularization network trained over a structure-aware loss function is subsequently appended to enforce correct parallax relationships over the intermediate estimation. Our proposed approach is evaluated over datasets with a large number of testing images including both synthetic and real-world scenes. Experimental results demonstrate the advantage of our approach over state-of-the-art methods, i.e., our method not only improves the average PSNR by more than 1.0 dB but also preserves more accurate parallax details, at a lower computational cost.
1. Introduction
LF images encode directional light information but suffer from low spatial resolution because sensor resources are shared with angular sampling. Existing LF spatial SR methods leave cross-view complementarity and structural consistency insufficiently addressed, motivating a two-module learning-based approach.
- 4D LF images record both light intensity and ray direction, enabling applications including 3D reconstruction, refocusing, and virtual reality.
- Limited sensor resolution causes recorded LF images to have low spatial resolution, making LF spatial SR necessary for downstream applications.
- Existing learning-based methods improve LF spatial SR but do not fully exploit complementary information across views or adequately preserve reconstruction structure.
- The proposed framework addresses these gaps through All-to-One SR via combinatorial geometry embedding and a structural consistency regularization module.The first module super-resolves individual views using information from all views, while the second enforces geometry coherence in the high-resolution result.
2. Related Work
LF spatial SR uses the multi-view structure of light fields to constrain reconstruction, with approaches ranging from disparity-based optimization to data-driven learning. Light-field geometry and epipolar structure provide the basis for modeling cross-view relationships and scene depth.
- A 4D LF is represented by spatial and angular coordinates, and each fixed angular position defines a 2D view from a slightly different viewpoint.
- Under the Lambertian assumption, corresponding scene points retain the same intensity across views, producing a structured LF parallax relationship.
- Epipolar-plane images are 2D LF slices whose straight-line slopes correspond to scene points at different depths.
- LF spatial SR exploits cross-view correlations and complementary information to constrain reconstruction from low-resolution to high-resolution images.
- Optimization-based methods estimate disparities and use physical or geometric priors, including variational, Gaussian-mixture, and graph-based formulations.
3. Motivation
High-quality LF spatial SR requires both comprehensive use of complementary information across views and preservation of view-level parallax structure. The motivation contrasts existing fusion strategies with the proposed All-to-One design.
- LF spatial SR reconstructs a high-resolution LF from a low-resolution LF while preserving its spatial and angular dimensions.
- Two central requirements are thorough cross-view complementarity exploration and strict regularization of LF structural parallax.
- Occlusions and non-Lambertian reflections make information asymmetric across views, so information missing from one view may be available in another.
- Existing fusion methods use limited view subsets or directional structures, leaving some potentially useful views unused.
- All-to-All fusion stacks all views and predicts details simultaneously, but average error optimization can compromise view-specific details.
- The proposed All-to-One strategy super-resolves each individual view by combining information from all views.
- Parallax preservation is treated as a central LF property, with prior methods enforcing corresponding pixels to maintain similar intensities.
4. The Proposed Method
The proposed framework first super-resolves each reference view by combining complementary information from auxiliary views, then regularizes the intermediate LF to preserve parallax consistency. Its design combines combinatorial geometry embedding, efficient spatial-angular processing, and EPI-based structural supervision.
- All-to-One SR via Combinatorial Geometry Embedding: The All-to-One SR module super-resolves each view by fusing combinatorial embeddings from the other LF views.It separately extracts per-view features, learns pairwise correlations, and combines aligned auxiliary-view features for reconstruction.
- All-to-One SR via Combinatorial Geometry Embedding: The module extracts shared per-view features with a network whose parameters are shared across all views.Feature extraction uses a convolutional layer, ReLU activations, and residual blocks.
- All-to-One SR via Combinatorial Geometry Embedding: Aligned auxiliary-view features are fused channel-wise before deeper feature extraction and upsampling.Feature learning and fusion remain in low-resolution space; sub-pixel convolution and bicubic interpolation are used during reconstruction.
- Structural Consistency Regularization: Structural consistency regularization models cross-view correlations with alternate spatial-angular convolutions applied to the intermediate LF.The network repeatedly reshapes spatial and angular feature stacks so 2D convolutions process both dimensions efficiently.
- Structural Consistency Regularization: The regularization module uses LF reconstruction and EPI-gradient losses to enforce consistency with the ground-truth LF and its parallax structure.EPI gradients are computed along spatial and angular dimensions on horizontal and vertical EPIs.
5. Experimental Results
Experiments on real-world and synthetic LF scenes show that the method improves reconstruction quality, preserves parallax structure, and benefits from its two proposed modules.
- Quantitative comparisons: The method outperforms ResLF by around 1–2 dB in average PSNR for both 2× and 4× LF spatial SR.PSNR and SSIM are used for quantitative evaluation across testing datasets.
- Quantitative comparisons: Using all views reduces the PSNR gap between central and corner views compared with ResLF.The authors attribute ResLF’s corner-view degradation to using fewer views for directional stacks.
- Qualitative comparisons: The reconstructions produce sharper textures than competing methods, which often lose high-frequency details or introduce blur.Examples include characters on a pen, branches on the ground, and digits on a clock.
- Parallax structure: The method preserves clearer and more consistent straight lines in EPIs than other methods.Straight EPI lines directly represent LF parallax structure.
- Parallax structure: Its LF edge parallax PR curves lie closer to the top-right corner, indicating stronger structural consistency.The PR-curve comparison provides a quantitative assessment of parallax preservation.
- Ablation study: All-to-One fusion improves PSNR by more than 0.6 dB on real-world data and 1.0 dB on synthetic data over All-to-All fusion.The comparison uses networks with the same number of parameters.
- Ablation study: Structural consistency regularization improves average reconstruction quality by around 0.2–0.3 dB across datasets.Some scenes, including ’Occlusion 43 eslf’ and ’Antiques dense’, improve by more than 0.5 dB.
6. Conclusion and Future Work
The method addresses complementary-view fusion and parallax preservation through two subnetworks, producing super-resolved LF images with higher PSNR/SSIM and better structure than state-of-the-art methods. Future work could explore adversarial and perceptual losses for more realistic textures in LF processing.
- Conclusion: The method models complementary-view fusion and parallax preservation with All-to-One SR and structural consistency regularization subnetworks.These subnetworks efficiently generate super-resolved LF images.
- Conclusion: The method produces higher PSNR/SSIM and better LF structure than state-of-the-art methods.
- Future Work: Future work could exploit adversarial and perceptual losses, along with extensions to high-dimensional data, for LF processing.The paper notes that these losses have promoted realistic textures in single-image super-resolution.