Source-linked AI summary
Light Field Reconstruction Using Shearlet Transform
Suren Vagharshakyan, Robert Bregovic, Atanas Gotchev
TL;DR
The paper addresses reconstruction of densely sampled light fields from a small number of wide-baseline, rectified views, where decimation produces aliased EPIs. It uses sparse shearlet-domain EPI representation with iterative regularized reconstruction, achieving favorable comparisons with reference DIBR and state-of-the-art IBR methods while remaining limited to Lambertian scenes.
Problem
Dense light fields are useful but often cannot be captured directly, and decimated EPIs can contain overlapping aliasing that makes classical band-limited reconstruction infeasible.
Method
The method reconstructs EPIs using a shearlet frame designed for EPI structure, sparse representation, iterative regularization, and adaptive convergence tuning.
Results
The method compares favorably with reference DIBR and state-of-the-art IBR methods; on Teddy, PSNR is 33.33 dB versus 33.25 dB for the comparison method.
Takeaways & Limitations
The reconstructed light field has at most 1 pixel disparity between adjacent views even when input views have relatively high disparity.
Takeaways & Limitations
The reported implementation is limited to Lambertian scenes; extending it to non-Lambertian scenes would require modified reconstruction bases.
Abstract
from arXiv · showhide
In this article we develop an image based rendering technique based on light field reconstruction from a limited set of perspective views acquired by cameras. Our approach utilizes sparse representation of epipolar-plane images in a directionally sensitive transform domain, obtained by an adapted discrete shearlet transform. The used iterative thresholding algorithm provides high-quality reconstruction results for relatively big disparities between neighboring views. The generated densely sampled light field of a given 3D scene is thus suitable for all applications which requires light field reconstruction. The proposed algorithm is compared favorably against state of the art depth image based rendering techniques.
1 INTRODUCTION
Image-based rendering must reconstruct dense intermediate views from sparse camera captures. The paper proposes shearlet-domain EPI inpainting to recover densely sampled light fields without relying directly on estimated depth.
- Motivation: Intermediate views are needed for immersive applications, but real scenes are often captured by only a coarse set of camera views.The motivation includes free viewpoint television and virtual reality, where continuous parallax is desired.
- Existing approaches: Depth-based view synthesis is content-dependent because small depth errors can create visually annoying artifacts.Light-field methods instead treat pixels from the captured views as samples of a multidimensional function.
- Sampling challenge: Dense light-field sampling requires neighboring-view disparity below one pixel and sufficient spatial sampling of the scene texture.The required camera spacing depends on minimum scene depth and camera resolution.
- Applications: Densely sampled light fields support refocusing, dense depth estimation, novel-view generation, and holographic stereography, but are often impractical to capture directly.The paper therefore addresses reconstruction from a sparse set of images.
- Proposed approach: The proposed method represents EPIs sparsely in an adapted shearlet transform and performs EPI inpainting using depth-layer structure.The approach extends earlier LF sparsification and depth-layering ideas.
- Paper scope: The paper extends preliminary shearlet-based novel-view synthesis with transform construction, a detailed reconstruction algorithm, and evaluation on various datasets.The evaluation includes comparison with state-of-the-art methods.
2 LIGHT FIELD FORMALIZATION
A light field parameterizes captured light rays, while an EPI is formed by fixing one image coordinate across camera positions. EPI slopes encode scene depth, and sparse camera sampling creates aliasing that motivates depth-aware reconstruction.
- 2.1 Light field representation: The plenoptic function can be reduced to a 4D light field describing static, monochromatic rays using two-plane or plane-and-direction parameterizations.The two-plane form is written as L(u, v, s, t).
- 2.1 Light field representation: Camera motion along the t-axis connects light-field coordinates with discrete camera positions and image-plane sampling.The setup fixes the vertical camera coordinate and varies the camera position along one axis.
- 2.2 EPI representation: An EPI is obtained by collecting image rows at fixed u across camera positions, producing the slice E(v, t) = L(u0, v, s0, t).The EPI provides a structured representation of multiview data.
- 2.2 EPI representation: Scene points appear as lines in an EPI, with line slopes determined by their depths.The same geometric relation is visible in the camera projections and the EPI.
- 2.2 Sampling requirements: Camera sampling must satisfy the Nyquist criterion for the highest scene texture frequency while keeping nearby-view disparity at most one pixel.The required camera step is related to scene depth and horizontal image sampling.
- 2.3 Motivation: The reconstruction problem is to recover densely sampled EPIs from decimated, aliased versions created by a larger camera step.When aliased replicas overlap the baseband, classical band-limited filtering cannot reconstruct the EPI.
- 2.3 Motivation: Because real scenes may occupy a small number of depths, the method seeks sparse depth layers through frequency-domain analysis and depth-guided filtering.This corresponds to a frequency-plane tiling that separates depth layers.
3 EPIPOLAR-PLANE IMAGE IN TRANSFORM DO-
The section motivates directionally sensitive sparse representations for piecewise-smooth EPIs and develops compactly supported, modified shearlet transforms suited to their line singularities.
- Directional sensitivity: Wavelets achieve an O(N^-1) approximation rate for cartoon-like functions, whereas adaptive triangle approximation achieves O(N^-2).This motivates transforms with directional sensitivity for efficiently representing singularities.
- Directional sensitivity: Shearlet frames provide optimal sparse approximation for cartoon-like functions by retaining the largest frame coefficients.Both tight and compactly supported shearlet constructions are reported to have this property.
- Cone-adapted shearlets: The cone-adapted shearlet system separates low-frequency and cone-like frequency regions through scaling and shearlet generators.Its transform maps functions to coefficient sequences indexed by scale, shear, and translation parameters.
- Compactly supported shearlets: Separable shearlet generators create overlapping frequency supports and poor directional selectivity, while nonseparable generators improve frequency coverage and directionality.The nonseparable construction is also used to obtain compactly supported digital shearlets.
- Modified shearlet transform for EPI: For EPIs, the parabolic scaling choice is modified because their singularities lie along straight lines rather than parabolic curves.The modified construction supports scaling along one axis and a desirable number of shears at each scale.
- Digital implementation: The digital transform uses filter-based shear and convolution operations, with low-pass filtering enabling implementation on a refined grid.The dual frame guarantees stable reconstruction when its frequency response remains between finite positive bounds; for J = 2, the bounds are 0.03 and 1.03.
4 RECONSTRUCTION ALGORITHM
The algorithm reconstructs densely sampled EPIs from sparse view samples by posing the task as shearlet-domain sparse inpainting and solving it iteratively. It uses adaptive acceleration and directionally selected shearlet atoms to improve convergence and reconstruction quality.
- Problem formulation: The method treats reconstruction of the unknown densely sampled EPI as an inpainting problem constrained by sparsity in the shearlet transform domain.The measurements retain only selected rows, while the reconstruction seeks a shearlet-sparse solution.
- Iterative reconstruction: The iterative thresholding algorithm updates the estimate using the sampling residual, shearlet analysis and synthesis, coefficient thresholding, and an acceleration parameter.The threshold decreases linearly from λmax to λmin, and the iterates approach the target after sufficient iterations.
- Convergence control: Adaptive acceleration provides high convergence speed and stable reconstruction, whereas excessively large fixed acceleration can cause divergence.Increasing fixed αn accelerates convergence only up to a limit; the figure illustrates divergence at α = 20.
- Shearlet construction: The transform uses only shearlet atoms associated with valid EPI directions, with dyadic scales and disparity-related shears selected according to dmax.The construction uses J = ⌈log2 dmax⌉ scales and 2^j+1 + 1 shears at scale j.
- Evaluation: With every fourth EPI row available, the proposed transform achieved 41.57 dB, exceeding FFST at 39.65 dB, shearlab at 33 dB, and Haar at 24 dB.The comparison uses a densely sampled synthetic-scene ground truth EPI.
5 EVALUATION
The evaluation tests reconstruction across multiview datasets, compares the method with DERS+VSRS and another depth-layering approach, and examines full-parallax light fields and refocusing. Results generally favor the proposed method, including competitive or better PSNR under several sampling configurations.
- Multiview evaluation: The evaluation uses leave-N-out tests, reconstructing dropped views from downsampled multiview inputs and assessing quality with PSNR.DERS+VSRS serves as a reference algorithm for comparison.
- Comparison with DIBR: The proposed method performs better than DERS+VSRS in all tested cases except the Coach dataset.For the Couch dataset, the comparison uses disparity maps estimated from the full image set, representing an idealized DIBR case; the proposed method remains competitive.
- Comparison with depth layering: 33.33 dB versus 33.25 dB: the proposed method and the depth-layering method show equal performance on the Teddy dataset.PSNR is averaged over four reconstructed views.
- Full-parallax reconstruction: On the Bunny dataset, the proposed method uses fewer input views while performing similarly to the full-parallax method; on Truck, it achieves significantly better average PSNR.The comparison uses 17 × 17-view Bunny and Truck datasets and reports PSNR over reconstructed views.
- Refocusing application: Refocusing with only a 5 × 5 subset produces strong artifacts, whereas the reconstructed 49 × 49-view light field yields smooth blurring from small inter-view disparities.The reconstructed result is compared with refocusing from the original dataset.
6 CONCLUSIONS
The paper reconstructs densely sampled light fields from relatively few wide-baseline views by exploiting sparse EPI representations in a tailored shearlet domain. Experiments favor the method against reference DIBR and state-of-the-art IBR, while the reported implementation remains limited to Lambertian scenes.
- Contribution: The method reconstructs densely sampled light fields from a small number of rectified, wide-baseline multiview images.Adjacent reconstructed views have disparity of at most one pixel, despite potentially high disparity between input views.
- Method: Sparse EPI representation in a shearlet domain underlies the iterative regularized reconstruction algorithm.The paper also constructs a shearlet frame for EPIs and adaptively tunes the convergence parameter.
- Evaluation: Experiments compare the method favorably with reference DIBR software and state-of-the-art IBR across various datasets.The method reconstructs all views for applications requiring dense sampling.
- Applications: The reconstructed light field supports applications including refocusing, wide-field-of-view light-field displays, and digital holographic printing.These applications require densely sampled views.
- Limitation: The reported implementation is limited to Lambertian scenes; extending it to non-Lambertian scenes is identified as future research.The proposed extension would modify the reconstruction bases to cover different frequency-domain regions.
Truck Bunny
Figure 10 evaluates light-field reconstruction on the Truck and Bunny datasets using different input-view sampling patterns and compares the proposed method with SFFT.
- Datasets: Figure 10 evaluates reconstruction algorithms on the Truck and Bunny datasets.The comparison concerns light-field reconstruction quality across the two datasets.