PaperlayerView exact revision on arXiv

Source-linked AI summary

PhoCoLens: Photorealistic and Consistent Reconstruction in Lensless Imaging

Xin Cai, Zhiyuan You, Hailong Zhang, Wentao Liu, Jinwei Gu, Tianfan Xue

arXiv:2409.17996v2eess.IVcs.CVcs.LG

TL;DR

Lensless reconstruction must recover high-quality scenes from blurry, ambiguous measurements, while existing models can miss spatial PSF variation. PhoCoLens combines spatially varying deconvolution with conditioned diffusion, and reports a better balance of fidelity and visual quality than existing methods on two lensless camera systems.

  • Problem

    Blurry, ambiguous lensless measurements and spatially varying PSFs complicate reconstruction that must preserve scene consistency while recovering realistic detail.

    Source: §1 · p. 2

  • Method

    PhoCoLens first uses spatially varying deconvolution to recover range-space low frequencies, then conditions a pre-trained diffusion model on that output to restore null-space detail.

    Source: §1 · p. 2

  • Results

    PhoCoLens reports a balance of data consistency and visual quality surpassing existing methods on PhlatCam and DiffuserCam.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The two-stage design separates measurement-consistent range content from added high-frequency detail while aiming to preserve both fidelity and photorealism.

    Source: §1 · p. 2

    The two-stage design and diffusion sampling time hinder real-time applicability.

    Source: §6 · p. 10

Abstract

from arXiv · show

Lensless cameras offer significant advantages in size, weight, and cost compared to traditional lens-based systems. Without a focusing lens, lensless cameras rely on computational algorithms to recover the scenes from multiplexed measurements. However, current algorithms struggle with inaccurate forward imaging models and insufficient priors to reconstruct high-quality images. To overcome these limitations, we introduce a novel two-stage approach for consistent and photorealistic lensless image reconstruction. The first stage of our approach ensures data consistency by focusing on accurately reconstructing the low-frequency content with a spatially varying deconvolution method that adjusts to changes in the Point Spread Function (PSF) across the camera's field of view. The second stage enhances photorealism by incorporating a generative prior from pre-trained diffusion models. By conditioning on the low-frequency content retrieved in the first stage, the diffusion model effectively reconstructs the high-frequency details that are typically lost in the lensless imaging process, while also maintaining image fidelity. Our method achieves a superior balance between data fidelity and visual quality compared to existing methods, as demonstrated with two popular lensless systems, PhlatCam and DiffuserCam. Project website: https://phocolens.github.io/.

1 Introduction

Lensless reconstruction must balance photorealism with consistency, despite blurry measurements, lost high-frequency detail, and spatially varying PSFs. PhoCoLens combines spatially varying deconvolution for measurement-consistent low-frequency content with diffusion-based recovery of realistic detail, and reports a balance of fidelity and visual quality beyond existing methods.

Problem

  • Lensless measurements are blurry and ambiguous because the large PSF acts as a low-pass filter that encodes scenes into diffraction patterns.A single measurement can admit multiple possible recoveries.Source: §1 · p. 2
  • The reconstruction challenge is to retain scene consistency while recovering rich details, since generative priors can alter content or insert nonexistent objects.The introduction contrasts ground-truth-aligned but degraded WienerDeconv results, weak high-frequency recovery by FlatNet-gen, and possible content changes from DiffBIR [21].Source: §1 · p. 2

Problem and Method

  • Shift-invariant PSF assumptions are inaccurate because real PSFs vary spatially, with reconstruction similarity especially affected at the field periphery.The second stage uses a pre-trained diffusion prior conditioned on first-stage low-frequency content to recover lost high-frequency detail.Source: §1 · pp. 2–3

Method

  • PhoCoLens decomposes reconstruction into measurement-derived low-frequency range content and lost high-frequency null-space detail.The first stage prioritizes consistency; the second adds detail while maintaining the consistency established in the first stage.Source: §1 · p. 2
  • Spatially varying deconvolution reconstructs range-space content while adapting to PSF changes across the camera’s field of view.Unlike single-kernel methods, it is designed to model the forward imaging process more accurately and improve low-frequency structural integrity.Source: §1 · p. 2

Results

  • PhoCoLens reports a balance between data consistency and visual quality that surpasses existing methods on PhlatCam and DiffuserCam.The paper describes qualitative improvements on PhlatCam and comparisons using metrics for fidelity and visual quality.Source: §1 · p. 3

2 Related Work

Prior lensless reconstruction work spans regularized convolutional models, learned deconvolution, spatially varying methods, and diffusion-based inverse imaging. PhoCoLens positions its approach as combining learned spatial variation with supervised diffusion and range-null space decomposition.

Lensless imaging reconstruction

  • Traditional lensless methods solve regularized least-squares problems, while deep approaches learn deconvolution or image enhancement but generally assume a constant PSF.The cited approaches include deep unrolled techniques [17] [23], image-space deconvolution, and feature-space deconvolution.Source: §2 · p. 3

Spatially varying deconvolution

  • Existing spatially varying deconvolution methods can be slow and computationally intensive, while MultiWienerNet requires multi-location PSF calibration.PhoCoLens instead uses one initial PSF and learns spatial variations automatically.Source: §2 · p. 3

Diffusion inverse imaging

  • Diffusion inverse-imaging methods include supervised conditional models [7] [11] [10] [28] [44] and zero-shot guidance [33], while PhoCoLens combines supervised fine-tuning with range-null space decomposition.The related-work discussion also notes mathematical decomposition approaches including DDRM [13] and DDNM [37].Source: §2 · p. 3

3 Preliminary

The preliminary section establishes range-null space decomposition and explains why a spatially invariant convolution is an imperfect lensless imaging model. It relates off-axis PSF variation to peripheral reconstruction artifacts and previews the two-stage pipeline.

3.1 Range-Null Space Decomposition

  • Any scene can be decomposed into a range-space projection A†Ax and a null-space component (I − A†A)x.The range component is measurement-consistent, while the null component lies in the null space of A.Source: §3.1 · pp. 3–4

3.2 Mismatch in Convolutional Lensless Imaging Model

  • Figure 4 compares PSFs at 0°, 15°, and 30° and shows greater reconstruction degradation at the periphery when using the 0° PSF.The caption describes angle-dependent PSF shifts and compares on-axis and off-axis PSF similarity across field positions.Source: Figure 4 · p. 4
  • Lensless measurements are often modeled as y = h ∗x because the full transfer matrix is too large to compute, but this assumes a spatially invariant PSF.The Huygens-Fresnel formulation motivates the model; the Fresnel approximation requires d ≫ (x − ξ)^2 + (y − η)^2, which is violated when masks sit close to the sensor, typically at d = 2mm.Source: §3.2 · p. 4
  • 1.0 to 0.7: PSF similarity between the center and −30° drops as incidence angle changes.The simulated PSFs differ across angles, and using the center PSF for deconvolution produces more artifacts at the boundary than at the center.Source: §3.2 · p. 4
  • Figure 5 previews a two-stage pipeline: spatially varying deconvolution reconstructs range space, then conditional diffusion recovers null-space details.The second stage conditions on the first-stage output to form the final reconstruction.Source: Figure 5 · p. 5

4 Method

PhoCoLens decomposes reconstruction into a measurement-consistent range-space component and a photorealistic null-space component, recovered in two stages. SVDeconv estimates the former, and conditional null-space diffusion supplies the latter while keeping the residual in the camera operator’s null space.

4 Method

  • PhoCoLens reconstructs range-space content from the measurement, then adds null-space content to improve photorealism while preserving consistency.The decomposition separates the component that maintains consistency from the component that supports photorealism.Source: §4 · p. 5

4.1 Spatially-varying Deconvolution Network

  • SVDeconv applies learnable PSFs on a K × K image grid, distance-weights the resulting deconvolutions into one image, then refines it with a U-Net.The interpolation weights each deconvolved image by its inverse Euclidean distance from the corresponding PSF center.Source: §4.1 · pp. 5–6
  • SVDeconv requires only one calibrated PSF to initialize its kernels, unlike multi-kernel methods that require multiple calibrated PSFs.The model learns PSF variations automatically after the center PSF is defined.Source: §4.1 · p. 6

4.2 Null Space Content Recovery

  • Conditioned on range-space content, null-space diffusion uses frozen pretrained diffusion weights and learned conditioning modules to add realistic detail.The training process progressively noises images and trains a network to predict the added noise.Source: §4.2 · pp. 6–7
  • The generated image’s residual relative to range-space content is constrained to lie in the camera operator’s null space, preserving measurement consistency.The stated condition is A(ẋ − A†Ax) = 0.Source: §4.2 · p. 6

5 Experiments

PhoCoLens is evaluated on PhlatCam and DiffuserCam using fidelity and visual-quality metrics, with comparisons and ablations examining its two-stage design. The reported results favor the method’s balance of consistency and visual quality across the datasets.

5.3 Comparison with Other Approaches

  • The evaluation uses real-world PhlatCam and DiffuserCam datasets and compares PhoCoLens with traditional, learning-based, and diffusion-based approaches.PhlatCam has 10,000 images across 1,000 classes, with 990 classes for training and 10 for testing; DiffuserCam has 25,000 paired images, split into 24,000 training and 1,000 testing images.Source: §5 · p. 7, §5.3 · p. 7, §5.1 · p. 7

5.1 Dataset and Metrics

  • Full-reference PSNR, SSIM, and LPIPS assess consistency, while non-reference ManIQA, ClipIQA, and MUSIQ assess visual quality.The two metric sets evaluate different aspects of reconstruction rather than a single combined score.Source: §5.1 · p. 7

5.2 Implementation Details

  • SVDeconv uses 3 × 3 PSF kernels initialized from a calibrated PSF, and null-space diffusion is conditioned on its reconstructed range-space content.Training uses 100 epochs for SVDeconv and 200 epochs for null-space diffusion; DiffuserCam measurements receive replicate padding because of sensor cropping.Source: §5.2 · p. 7

5.3 Comparison with Other Approaches

  • On PhlatCam, PhoCoLens achieves the best performance on both fidelity and visual-quality metrics, with reconstructions closest to ground truth.The DiffuserCam comparison favors PhoCoLens on SSIM and LPIPS; UPDN has slightly higher PSNR, while FlatNet+DiffBIR scores higher on non-reference metrics but loses consistency.Source: §5.3 · p. 8, Table 1 · p. 8
  • On DiffuserCam, PhoCoLens has superior SSIM and LPIPS, while UPDN has slightly higher PSNR and FlatNet+DiffBIR higher non-reference scores.FlatNet+DiffBIR’s higher visual-quality metrics come with low full-reference metrics and sometimes distorted image content; the authors report PhoCoLens has the best trade-off between consistency and visual quality.Source: §5.3 · p. 8, Table 1 · p. 8

5.4 Analysis and Discussion

  • SVDeconv improves faithful reconstruction of range-space and original content, especially fine details at the field-of-view periphery.The comparison uses the same U-Net and training loss across methods; Tab. 2 and Fig. 8 report the comparison.Source: §5.4 · p. 9, Table 2 · p. 9, Figure 8 · p. 9
  • Null-space diffusion performs best in consistency and visual quality among the compared recovery methods.The comparison includes DiffBIR, StableSR, and DDNM, with results reported in Tab. 3 and Fig. 9.Source: §5.4 · p. 9, Table 3 · p. 9, Figure 9 · p. 9
  • Using PhoCoLens’s reconstructed range content as the diffusion condition improves consistency over the tested alternative conditions.The authors attribute artifacts from alternatives such as FlatNet-gen to hindering recovery of the original image; PhoCoLens outperforms the alternatives on all fidelity metrics in Tab. 4 and Fig. 10.Source: Table 4 · p. 9, Figure 10 · p. 9, §5.4 · p. 10

6 Conclusion and Limitation

PhoCoLens combines spatially varying deconvolution for consistency with null-space diffusion for photorealism. Experiments on two lensless cameras demonstrate its effectiveness, while its two-stage process and diffusion sampling time limit real-time applicability.

6 Conclusion and Limitation

  • PhoCoLens combines spatially varying deconvolution for consistency with null-space diffusion for photorealism, and experiments on two cameras demonstrate realistic reconstructions that preserve fidelity.The two-stage process and diffusion sampling time hinder real-time applicability; generated high-frequency details may also deviate from the original, particularly for smooth scenes lacking detail.Source: §6 · p. 10

A PhlatCam Simluation

The PhlatCam simulation constructs a phase mask to produce a desired PSF and uses Fresnel propagation to simulate lensless imaging. Visual examples show the target PSF, optimized mask, and simulated PSF.

A PhlatCam Simluation

  • The simulation optimizes a phase mask to produce a target PSF, using near-field phase retrieval with forward and backward Fresnel propagation.The setup uses a 1 mm mask-to-sensor distance and 532 nm light; Fig. A1 shows the target PSF, optimized mask, and resulting simulated PSF.Source: A PhlatCam Simluation · p. 13, Figure A1 · p. 13

B Range Space Content Approximation

Because directly obtaining the lensless transfer matrix is computationally infeasible, the appendix approximates the forward process and its pseudoinverse. With a single calibrated PSF, it uses convolution for simulation and replicate padding plus Wiener deconvolution for range-space content.

B Range Space Content Approximation

  • Directly obtaining the transfer matrix is computationally infeasible, so the method approximates the forward process and its pseudoinverse.With one calibrated PSF, the forward approximation convolves the image with the PSF and applies cropping; the inverse approximation uses replicate padding and Wiener deconvolution.Source: B Range Space Content Approximation · p. 13
  • The simulation includes cropping and sensor and quantization noise; when multiple calibrated PSFs are available, a low-rank model can better approximate spatially varying convolution.The reported setups use one calibrated PSF, and the resulting simulated range-space content remains an approximation of its real counterpart.Source: B Range Space Content Approximation · pp. 13–14
  • Simulated range-space content retains low-frequency scene structure while losing some high-frequency detail.Examples show SVDeconv estimates preserve low-frequency content and align closely with ground truth, while FlatNet alters scene content and introduces incorrect high-frequency details.Source: B Range Space Content Approximation · pp. 14–15, Figure A2 · p. 14, Figure A3 · p. 14

C Training Target for the Null-space Diffusion Model

The appendix connects the consistency goal to the diffusion-training objective and describes a noise-prediction formulation and a practical approximation.

C Training Target for the Null-space Diffusion Model

  • The appendix establishes that Eqn. (7)'s consistency goal corresponds to the diffusion-training objective in Eqn. (8).The text says a detailed derivation supports this equivalence.Source: C Training Target for the Null-space… · p. 15
  • The diffusion output is conditioned on noisy input x_t and range-space content c, with an objective penalizing measurement inconsistency against x_0.The stated constraint is A(ẋ − x_0) = 0.Source: C Training Target for the Null-space… · p. 15
  • The reconstruction objective is reparameterized as a denoising objective that trains predicted noise to match injected noise.The appendix identifies this as Eqn. (8), then gives an approximate null-space objective because simulating transfer matrix A is computationally costly.Source: C Training Target for the Null-space… · p. 15

D Qualitative Results

The appendix presents qualitative comparisons of PhoCoLens with other methods on PhlatCam and DiffuserCam.

D Qualitative Results

Related papers

Loading 2409.17996v2…