PaperlayerView exact revision on arXiv

Source-linked AI summary

High-dimensional Dense Residual Convolutional Neural Network for Light Field Reconstruction

Nan Meng, Hayden K. -H. So, Xing Sun, Edmund Y. Lam

arXiv:1910.01426v4eess.IVcs.CV

TL;DR

Light-field super-resolution must recover spatial and angular detail, while existing approaches may require disparity clues or restore those details separately. The paper formulates the task as tensor restoration and uses two-stage 4D convolution. It reports geometry-sensitive features near occlusions and says M-HDDRNet improves reconstruction of subpixel information in challenging scenes.

  • Problem

    Light-field super-resolution methods may require disparity clues or restore spatial and angular details separately, with difficulties on non-Lambertian surfaces and around occlusions.

    Source: §1 · p. 1

  • Method

    HDDRNet treats light-field super-resolution as tensor restoration and uses high-dimensional convolution in a progressive, two-stage network to recover angular information and spatial detail.

    Source: §1 · pp. 1–2, §3.2 · p. 3

  • Results

    The authors report that M-HDDRNet improves reconstruction results and is effective at recovering subpixel information in challenging scenes.

    Source: §7 · pp. 11–12

  • Takeaways & Limitations

    The extracted geometric features are sensitive to object borders and indicate scene geometric structure.

    Source: §7 · p. 11

    For the same output size, 4D convolution theoretically requires additional computation compared with conventional 2D convolution.

    Source: §6.4 · p. 10

Abstract

from arXiv · show

We consider the problem of high-dimensional light field reconstruction and develop a learning-based framework for spatial and angular super-resolution. Many current approaches either require disparity clues or restore the spatial and angular details separately. Such methods have difficulties with non-Lambertian surfaces or occlusions. In contrast, we formulate light field super-resolution (LFSR) as tensor restoration and develop a learning framework based on a two-stage restoration with 4-dimensional (4D) convolution. This allows our model to learn the features capturing the geometry information encoded in multiple adjacent views. Such geometric features vary near the occlusion regions and indicate the foreground object border. To train a feasible network, we propose a novel normalization operation based on a group of views in the feature maps, design a stage-wise loss function, and develop the multi-range training strategy to further improve the performance. Evaluations are conducted on a number of light field datasets including real-world scenes, synthetic data, and microscope light fields. The proposed method achieves superior performance and less execution time comparing with other state-of-the-art schemes.

1 INTRODUCTION

The paper addresses limits in light-field super-resolution by developing HDDRNet, a 4D-convolution network that progressively restores angular information and spatial detail. It also introduces geometric features and multi-range training to improve reconstruction.

1 INTRODUCTION

  • Many depth-based approaches rely on the Lambertian assumption and work poorly on glossy surfaces, while occlusions can introduce reconstruction artifacts.Light-field cameras trade spatial resolution against angular sampling density.Source: §1 · p. 1
  • Existing CNN methods often process neighboring views or epipolar-plane images as 2D data, leaving the light field’s inherent structure incompletely represented.Reconstructing individual sub-aperture or EPI images also makes these algorithms inefficient.Source: §1 · p. 1
  • HDDRNet uses densely connected high-dimensional residual blocks to restore light-field geometry, then refines spatial detail while preserving angular correlations.The two-stage network is trained end-to-end; the refinement stage uses aperture-wise perceptual loss rather than applying ℓ2 loss to the whole network.Source: §1 · p. 1
  • High-order convolution learns geometry-informed representations from the full light field, supporting reconstruction at multiple spatial and angular scales.The authors report that these representations can enhance novel-view synthesis.Source: §1 · p. 1
  • The resulting geometric features vary near occlusions and indicate foreground-object borders.The paper names these geometry-informed representations “geometric features.”Source: §1 · pp. 1–2
  • Angular-loss training first learns the light distribution, while perceptual-loss training in the second stage produces more realistic spatial reconstruction.The second stage preserves the light-distribution properties learned by the first.Source: §1 · p. 2
  • M-HDDRNet uses spatial inter-scale correlations and multiple angular baseline ranges to improve reconstruction accuracy.The authors introduce this multi-range training strategy to train the network more effectively.Source: §1 · p. 2

2 RELATED WORK

Related work is organized around spatial and angular super-resolution. The reviewed approaches include disparity-based methods, CNNs, and EPI-based reconstruction; the paper positions its own whole-light-field throughput as an alternative to EPI-wise, pixel-wise, and aperture-wise schemes.

2 RELATED WORK

  • The review treats spatial and angular resolution enhancement as separate strands of light-field super-resolution work.Source: §2 · p. 2

2.1 Spatial super-resolution

  • Disparity-based spatial methods use scene geometry or neighboring-view correlations, but can produce tearing and ghosting at occlusions or on non-Lambertian surfaces.CNN approaches avoid disparity estimation, yet methods using separate or sequential views do not fully represent the light field’s 2D angular relationships.Source: §2.1 · p. 2

2.2 Angular super-resolution

  • Angular synthesis methods include depth-based warping and plenoptic-function reconstruction; reported drawbacks include artifacts in synthesized views, edge artifacts, or inefficient EPI-wise restoration.The reviewed examples include depth-dependent ghosting in occluded regions and lattice artifacts near edges.Source: §2.2 · p. 2
  • The paper contrasts its whole-light-field throughput with EPI-wise, pixel-wise, and aperture-wise reconstruction schemes, presenting it as a way to improve practical reconstruction efficiency.Source: §2.2 · p. 3

3 PROBLEM ANALYSIS AND FORMULATION

This section formulates light-field super-resolution as 4D tensor restoration and explains how 4D convolutions and view-group normalization model spatial-angular structure. It also relates learned feature behavior near occlusions to partially occluded EPI regions.

3.1 Light field representation

  • A light field is represented as a 4D function mapping spatial and viewpoint coordinates to ray intensity.The coordinates are (x, y) on the image plane and (s, t) on the focal plane.Source: §3.1 · p. 3

3.2 Problem formulation

  • LFSR is formulated as restoration from a low-resolution 4D light-field tensor to a high-resolution tensor, with separate spatial and angular scaling factors.The proposed network learns the LR-to-HR mapping and reconstructs the whole light field in one feedforward propagation.Source: §3.2 · p. 3

3.3 4D convolutional neural networks

  • 4D convolution connects adjacent views across spatial and angular dimensions, capturing spatio-angular information that 2D processing of views or EPIs underuses.The passage associates 2D approaches with artifacts in complex-light-distribution regions, including occlusions and non-Lambertian surfaces.Source: §3.3 · p. 3

3.4 Geometric features

  • Learned 4D features preserve spatial structure and adjacent-view relationships, with border features sensitive to occlusions while texture features retain clear edges.The feature EPIs resemble light-field EPIs; object-border feature edges are smoother than corresponding reconstruction edges.Source: §3.4 · p. 4
  • In partially occluded EPI regions, foreground pixels shift farther across views than background pixels, and the layer's weighted multi-view features consequently smooth near object borders.The passage attributes the smoothing to scene geometry rather than random occurrence.Source: §3.4 · pp. 4–5

3.5 Aperture group batch normalization

  • Aperture group batch normalization applies the normalization transform over a group of sub-aperture images in each feature channel, rather than whitening every view separately.The design is motivated by the high coherence among adjacent views in geometric features.Source: §3.5 · p. 5

4 METHOD

HDDRNet restores light fields in two stages: 4D convolution first recovers spatio-angular information, then a refinement network reconstructs spatial details. Its training combines angular and perceptual spatial losses with multi-range sampling.

4.1 High-dimensional dense residual CNNs

  • The 4D-convolution network progressively restores an input low-resolution light field through spatio-angular restoration and details refinement.It takes the LR light field directly rather than an upscaled version.Source: §4.1 · p. 5, Figure 5 · p. 6

4.1.1 Spatio-angular restoration

  • The first stage learns light distribution by minimizing angular mean-squared error between predicted and ground-truth high-resolution light fields.Upsampling combines angular interpolation with a channel-to-space shuffle for spatial resolution.Source: §4.1.1 · p. 5, Figure 6 · p. 7

4.1.2 Details refinement

  • The refinement network uses sub-aperture perceptual loss to recover spatial high-frequency details that per-pixel loss may smooth.The paper reports that this loss can drive recovery of spatial details with sub-pixel accuracy.Source: §4.1.2 · p. 5

4.2 Loss Function

  • The loss combines angular MSE, which preserves relationships between adjacent views, with spatial perceptual loss for high-frequency detail.The spatial term uses aperture-wise differences between high-level VGG feature representations.Source: §4.2 · p. 6

4.2.1 Network Settings

  • HDDRNet uses 64-filter 4D convolution layers with 3×3 spatial and 5×5 angular kernels, residual blocks, batch normalization, and LeakyReLU.The illustrated upscaling example applies 4D convolution, angular interpolation, and spatial shuffle.Source: §4.2.1 · p. 6, Figure 6 · p. 7

4.2.2 Multi-range training

  • Multi-range training samples spatial scales from 0.8 to 1.0 and five angular directions at varied ranges to model light distribution near occluders.The paper reports greater robustness to complex light distributions and different spatial-detail scaling.Source: §4.2.2 · p. 7

4.2.3 Implementation and training details

  • Training uses 96×96 HR patches with 5×5 angular views, randomly cropped from Lytro Archive and Fraunhofer data, and stochastic gradient descent.The learning rate starts at 10^-5 and is reduced by a factor of 0.1 every 10 epochs.Source: §4.2.3 · p. 7

5 EXPERIMENTS

The experiments examine training-data generalization and model-design choices, including angular filters, residual connections, losses, and multi-range training. The reported analyses favor dense skip connections and find gains from multi-range training and the defined loss.

5.1 Training data and analysis

  • Training on Lytro Archive and Fraunhofer data generalized to real-world, synthetic, and microscopy light fields, including datasets with complex occlusions and translucency.The authors interpret this generalization as evidence that the geometric features represent multiple situations.Source: §5.1 · pp. 7–8

5.2 Model design

  • The model-design analysis varies filter size, local residual connections, and multi-range training using a five-block restoration stage and a three-block refinement stage.These are the model settings used for the reported analysis.Source: §5.2 · p. 8

5.2.1 Angular filter size of 4D convolution

  • Among tested angular kernels, 5×5, 3×3, and increasing-size kernels showed competitive capacity to learn angular correlations.The tested settings included fixed 1×1, 3×3, and 5×5 kernels and increasing or decreasing kernel sizes across layers.Source: §5.2.1 · p. 8, Figure 8 · p. 8

5.2.2 Varying residual connections

5.2.3 Reconstruction strategy

  • Quantitative comparisons report improvement from multi-range training and the defined loss, while spatial loss promotes recovery of high-frequency details.The visual comparison examines spatial-loss contribution to details such as roof-tile texture and window frames.Source: §5.2.3 · pp. 8–9, Figure 11 · p. 9, Table 1 · p. 9

6 RESULTS

Across spatial and angular super-resolution comparisons, M-HDDRNet performs favorably against existing methods and is reported to be at least 40× faster than two reconstruction approaches. The comparisons also examine fine-texture recovery, angular correlations, and view synthesis.

6.1.2 Comparison with 3D SR methods

  • M-HDDRNet gives more realistic spatial results while preserving 2D angular correlations, unlike 3D SR methods that lose 1D angular correlation.The 3D comparison treats sub-aperture images as an image sequence.Source: §6.1.2 · p. 9
  • M-HDDRNet obtains higher average PSNR than Kalantari et al. and Yeung et al. in view-synthesis comparisons.The authors report better reconstruction quality and better preservation of geometric structure around complex occlusions than Kalantari et al..Source: §6.1.2 · p. 9, Table 3 · p. 10

6.3 Comparison with spatio-angular SR methods

  • For simultaneous spatial and angular SR, the compared pixel-level approach can produce lattice artifacts in bright regions, while Yoon et al. yields smooth spatial results.The paper illustrates lattice artifacts in the wall region.Source: §6.3 · p. 9

6.1 Overall Comparisons

  • M-HDDRNet performs favorably against existing 2D and 3D SR methods across seven public datasets and spatial 2×, 3×, and 4× tasks.The comparison reports average PSNR and SSIM; the paper also evaluates angular 2× and 3× SR.Source: §6.1 · p. 9, Table 2 · p. 10

6.4 Computational analysis

  • At least 40× faster, M-HDDRNet outperforms aperture-wise and EPI-wise methods in runtime for 3×3→9×9 angular SR.The comparison measures runtime versus PSNR; the paper notes that 4D convolutions require additional computation.Source: §6.4 · p. 10

6.1.1 Comparison with 2D SR methods

  • The model restores the fine “whisker” texture that is nearly lost in the LR input, while MSLapSRN and RDN produce blurry results despite competitive PSNR and SSIM.The visual comparison is on a real-world light-field scene at 4× SR.Source: §6.1.1 · p. 9, Figure 13 · p. 11

7 CONCLUSIONS

The conclusion presents HDDRNet as a dense high-dimensional network for progressively recovering light-field information, and reports that M-HDDRNet is effective for recovering subpixel information in challenging scenes.

7 CONCLUSIONS

  • Visual comparisons cover 4× spatial SR against 3D image SR, angular SR from 2×2 to 8×8, and combined 2× spatial and angular enhancement.The captions specify real-world scenes for the spatial and combined spatio-angular comparisons.Source: Figure 14 · p. 11, Figure 15 · p. 12, Figure 16 · p. 12
  • HDDRNet progressively recovers spatio-angular information and high-frequency spatial details using MSE-based angular loss and content spatial loss.The conclusion describes a high-dimensional deep convolutional network with dense connections for light-field super-resolution.Source: §7 · p. 11
  • Its high-dimensional convolution layers reconstruct multiple spatial or angular scales, while geometric features indicate scene structure and object borders.The conclusion describes those features as sensitive to object borders.Source: §7 · p. 11
  • A group-based normalization operation and multi-range training are proposed to ease training and improve reconstruction results.The normalization is based on groups of sub-aperture images in each feature map.Source: §7 · pp. 11–12
  • M-HDDRNet is shown to be effective for recovering subpixel information in challenging scenes.The model is the improved version of HDDRNet, developed with multi-range training to further improve reconstruction results.Source: §7 · pp. 11–12
Loading 1910.01426v4…