Source-linked AI summary
Neighbor-Aware View Synthesis for Restoring Missing Views in Light-Field Camera Arrays
Sakshi Goel, Ayush Goyal, K S Venkatesh, Koteswar Rao Jerripothula
TL;DR
Hardware failures can remove light-field views and degrade downstream reconstruction. The paper uses an adaptive neighbor-aware cGAN with positional encoding and a 3D-2D fusion generator to synthesize missing views, reporting visually plausible, photometrically accurate, and geometrically consistent results.
Problem
Hardware malfunctions create missing sub-aperture images that degrade light-field reconstruction and downstream processing.
Method
The framework adaptively selects four neighboring views and conditions a cGAN on them, positional encoding, and a 3D-2D fusion generator.
Results
The method produces visually plausible, photometrically accurate, and geometrically consistent missing-view reconstructions in extensive experiments.
Takeaways & Limitations
The framework offers a robust and efficient approach for fault-tolerant light-field image acquisition.
Abstract
from arXiv · showhide
In light-field (LF) imaging systems, dense spatial sampling from a camera array enables powerful post-capture capabilities such as refocusing and depth estimation. However, real-world LF capture is often affected by hardware malfunctions, where one or more cameras in the array fail, leading to missing sub-aperture images and degraded reconstruction quality. This paper addresses the problem of defective or missing view restoration in light-field camera arrays. We propose a novel generative framework that synthesizes the absent views by exploiting information from a carefully selected subset of neighboring cameras. These selected images, along with a positional encoding map indicating both their locations and the desired target view, are fed into a conditional Generative Adversarial Network (cGAN) trained to generate the missing viewpoint in a geometrically consistent manner. Extensive experiments on synthetic and real-world LF datasets demonstrate that our method produces visually plausible and photometrically accurate reconstructions, outperforming baselines for view interpolation both quantitatively and qualitatively. The proposed framework thus offers a robust and efficient solution for fault-tolerant light-field image acquisition.
I. INTRODUCTION
Missing camera-array views create gaps in light-field data and can degrade downstream reconstruction. The paper proposes a neighbor-aware generative framework with adaptive inputs, positional encoding, and a new benchmark.
- Hardware failures can remove sub-aperture images, causing light-field gaps, incorrect depth maps, and visible rendering discontinuities.
- Existing view-synthesis methods may struggle with occluded and non-Lambertian surfaces, while some restrict inputs to four corner views.
- The proposed framework adaptively selects four neighboring views along cardinal or diagonal directions instead of requiring fixed corner inputs.
- The paper introduces a synthetic benchmark containing 121 light-field scenes covering varied objects.
- A positional encoding map supplies an explicit geometric prior to improve view-synthesis consistency.
- The framework uses a novel 3D-2D fusion generator to integrate multi-view information for high-fidelity reconstruction.
II. METHODOLOGY
The restoration pipeline combines adaptive neighbor selection and positional encoding with a conditional GAN and robust training losses.
- The method restores defective light-field views using a cGAN that exploits geometric and photometric redundancy.
- Its pipeline includes adaptive neighbor selection with positional encoding, a tailored conditional GAN architecture, and a robust loss function.
B. Adaptive Neighbor Selection and Positional Encoding
The method selects four neighboring views around a missing target using either cardinal or diagonal configurations. Positional encoding records their geometry and supports distance-aware reconstruction weighting.
- For target coordinates c_t = (u_t, v_t), the method adaptively selects four available neighboring sub-aperture images within a neighborhood.
- The plus configuration samples neighbors along right, left, up, and down axes at random distances from 1 to 10.
- The x configuration samples diagonal neighbors at random distances from 1 to 10.
- The four selected views are stacked into a 4D input tensor X.
- The positional encoding map records selected-view and target coordinates, while Manhattan distance to the farthest neighbor weights reconstruction loss.
1) Generator G: UNet with 3D-2D Fusion:
The generator fuses four views into a 2D feature representation, injects positional information during encoding, and reconstructs the target through a U-Net. A PatchGAN discriminator evaluates local realism using the generated image and neighboring-view context.
- 1) Generator G: UNet with 3D-2D Fusion:: The modified U-Net generator fuses multi-view information and geometric constraints before producing the synthesized target view.
- 1) Generator G: UNet with 3D-2D Fusion:: A 3D convolution compresses four input views into a fused 2D feature map while retaining multi-view spatial correlations.
- 1) Generator G: UNet with 3D-2D Fusion:: The positional map is upsampled, processed by 2D convolutions, and concatenated with encoder features at an intermediate stage.
- 1) Generator G: UNet with 3D-2D Fusion:: U-Net skip connections preserve detailed high-frequency information in the synthesized target view.
- 2) Discriminator D: PatchGAN:: PatchGAN receives the restored or ground-truth image with the mean of four neighboring views and outputs local-patch realness probabilities.
D. Training and Loss Functions
The cGAN is trained end-to-end with separate generator and discriminator objectives. The generator uses a composite loss combining adversarial, reconstruction, structural, and perceptual terms.
- The cGAN is trained end-to-end by minimizing generator loss LG and discriminator loss LD.
- The discriminator uses a standard Binary Cross-Entropy adversarial loss.
- The generator loss combines adversarial, reconstruction, structural, and perceptual terms.
2) Generator Loss:
The generator loss combines adversarial, weighted reconstruction, perceptual, and edge terms to improve realism, photometric accuracy, perceptual quality, and boundary sharpness.
- Adversarial Loss: Adversarial loss encourages the synthesized image to fool the discriminator.
- Weighted Reconstruction Loss: Weighted L1 and SSIM losses enforce photometric accuracy while accounting for target-view synthesis difficulty.The weighting uses the Manhattan distance of the target view, and SSIM is calculated on images normalized to [0, 1].
- Edge Loss: Edge loss was introduced to mitigate recurring vertical blur and produce sharper object boundaries.It uses TEED edge maps and a Double Loss combining edge completeness and quality components.
- The training weights are λL1 = 10.0, λSSIM = 5.0, λP erc = 1.0, and λdist = 1.0.
A. Datasets and Implementaion Details
The dataset is captured with a precision ball screw–based 2D actuation platform and a single moving camera. Training uses an 80/20 train-test split and standardized optimization settings.
- Datasets: The capture system records 51×51 light-field views using 2 mm horizontal and 4 mm vertical camera steps.The platform uses Arduino and MATLAB automation and supports static-scene capture.
- Implementation Details: Models are trained for 200 epochs with Adam at a 2×10−4 learning rate and betas (β1, β2) = (0.5, 0.999).The data are divided into 80% training and 20% testing.
B. Evaluation Metrics and Baselines
Evaluation compares synthesized views with ground truth using PSNR, SSIM, and LPIPS, alongside a pixelwise-average baseline from four neighboring inputs.
- Evaluation Metrics: PSNR, SSIM, and LPIPS quantify synthesized-view quality against ground-truth images.
- Baselines: The baseline averages the pixels of four selected neighboring sub-aperture images to represent trivial interpolation.It measures improvement over a simple non-learning approach.
C. Ablation Study and Quantitative Analysis
The ablation study progressively adds the discriminator, positional encoding, and composite loss to a 3D-2D fusion U-Net. Results show incremental component benefits, with positional encoding especially improving geometric and perceptual consistency.
- Ablation design: The ablation builds from a standalone U-Net with weighted L1 loss by adding PatchGAN, positional encoding, and the full composite loss.The basic U-Net measures raw synthesis capability before successive components are incorporated.
- Quantitative analysis: The baseline averaging method performs poorly, while the basic 3D U-Net provides a reasonable starting point.
- Quantitative analysis: Adding the PatchGAN discriminator significantly boosts PSNR over the basic 3D U-Net.
- Quantitative analysis: Introducing positional encoding produces the largest gains in SSIM and LPIPS, confirming its critical role in geometric consistency.Qualitative comparisons also show structures aligning more faithfully with ground truth after positional encoding is included.
- Quantitative analysis: The final model preserves fine spatial details, sharper object boundaries, and consistent illumination across synthesized views.These properties support photometrically and structurally coherent reconstruction from multi-view and geometric cues.