Source-linked AI summary
Unrolled Primal-Dual Networks for Lensless Cameras
Oliver Kingshott, Nick Antipa, Emrah Bostan, Kaan Akşit
TL;DR
Lensless reconstruction models based on a single calibrated PSF may not represent optical aberrations or scenes with depth variation. The paper embeds learnable forward and adjoint models in a supervised primal-dual reconstruction framework. It reports up to 50% lower reconstruction error with a fraction of the parameters of previous works, and evaluates the model on an open dataset and a prototype camera.
Problem
Single-PSF reconstruction models may fail to represent optical aberrations or scenes with depth variation, limiting their fidelity to lensless image formation.
Source: §1 · p. 2Method
The method embeds jointly optimized learnable forward-adjoint models in a learned primal-dual reconstruction framework for lensless cameras.
Source: §1 · p. 2, §3.2.1 · p. 4Results
Up to 50% reduction in reconstruction error is promised while using a fraction of the parameters of previous works.
Source: §1 · p. 2Takeaways & Limitations
The authors report on-par image reconstruction quality without extensive training resources for neural networks with embedded physical priors.
Source: §7 · p. 7On the in-house camera, reconstruction quality is lower than on state-of-the-art datasets, which the authors attribute to the off-the-shelf diffuser.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Conventional image reconstruction models for lensless cameras often assume that each measurement results from convolving a given scene with a single experimentally measured point-spread function. These image reconstruction models fall short in simulating lensless cameras truthfully as these models are not sophisticated enough to account for optical aberrations or scenes with depth variations. Our work shows that learning a supervised primal-dual reconstruction method results in image quality matching state of the art in the literature without demanding a large network capacity. This improvement stems from our primary finding that embedding learnable forward and adjoint models in a learned primal-dual optimization framework can even improve the quality of reconstructed images (+5dB PSNR) compared to works that do not correct for the model error. In addition, we built a proof-of-concept lensless camera prototype that uses a pseudo-random phase mask to demonstrate our point. Finally, we share the extensive evaluation of our learned model based on an open dataset and a dataset from our proof-of-concept lensless camera prototype.
1 INTRODUCTION
Lensless reconstruction models can mismatch real light transport when a single calibrated PSF does not capture optical aberrations or depth variation. The paper proposes a learned primal-dual method with learnable physical models, evaluates it on benchmark and prototype-camera data, and reports improved reconstruction quality.
1 INTRODUCTION
- Single-PSF models may not capture aberrations or depth variation, so model mismatch can lead to hallucinated features or overly smooth reconstructions.The paper argues for correcting model error without increased computational complexity or extensive calibration.Source: §1 · p. 2
- Up to 50% reduction in reconstruction error is promised by the learned forward-adjoint model, using a fraction of the parameters of previous works.The model is jointly optimized with the rest of the network using paired training examples.Source: §1 · p. 2
- The paper evaluates its learned model on an open dataset and data from a proof-of-concept camera using a pseudo-random phase mask.An automatic calibration routine trains the model without an additional conventional camera.Source: §1 · p. 2
- On the in-house camera, reconstructions are lower quality than those on state-of-the-art datasets, which the authors attribute to their off-the-shelf diffuser.The authors say the diffuser does not fully resemble the motivating case.Source: §1 · p. 2
2 RELATED WORK
The related work surveys lensless camera designs and reconstruction approaches, positioning this paper's prototype and learned model among methods that combine physical models with data-driven reconstruction.
2 RELATED WORK
- The paper introduces its reconstruction method alongside a brief survey of lensless cameras and learned and unsupervised reconstruction approaches.Source: §2 · p. 2
2.1 Lensless cameras
- Lensless cameras use masks instead of conventional lenses, with designs spanning coded apertures, amplitude and phase masks, and metalenses.Phase-mask depth-varying PSFs can add near-field 3D imaging to 2D sensors.Source: §2.1 · p. 2
- The authors built a lensless-camera prototype for experimental validation, similar to Antipa et al. [2018] but differing in implementation details.Source: §2.1 · p. 2
2.2 Unsupervised Lensless Image Reconstruction Methods
- Cropped-convolution models support convex reconstruction with ADMM, while field-varying aberrations can require a 10x or greater increase in computational cost.The sensor crop arises because the PSFs extend beyond the limited imaging sensor.Source: §2.2 · pp. 2–3
2.3 Learned Lensless Image Reconstruction Methods
- Learned reconstruction methods trade off speed, model-error correction, and portability; this paper embeds multiple large kernels in an unrolled iterative model to compensate for aberrations.The paper separates model reconstruction from denoising to produce accurate intermediate reconstructions.Source: §2.3 · p. 3
- The authors report state-of-the-art-parity speed and image quality with greater parameter efficiency, by directing supervision toward the physical model of light transport.They contrast this with learned methods that rely on accurate PSF calibration and additional training data for an image prior.Source: §2.3 · p. 3
3 METHOD
The method formulates lensless reconstruction as an inverse problem and trains a modified learned primal-dual network with learnable camera operators. It also evaluates a mixed-channel variant that trades subjective color accuracy for signal-to-noise performance.
3.1 Problem Formulation
- The reconstruction model approximates the camera operator as cropped convolution with an on-axis PSF, avoiding storage of a PSF for every scene point.The crop accounts for lateral shifts of the large PSF beyond the sensor bounds.Source: §3.1 · p. 3
- The method embeds learnable convolution kernels the size of the PSF within a learned optimization scheme.Source: §3.1 · p. 3
3.2 Learning Large Kernels with Physically Informed Networks
- Paired examples train a reconstruction network to minimize mean-squared distance from ground-truth reconstructions, without requiring knowledge of the true linear system.Source: §3.2 · p. 3
3.2.1 Learned Primal Dual with a Physical Model.
- The learned primal-dual model uses cropped convolution, optimizes PSF-initialized kernels, and assigns each primal and dual variable its own kernel across ten iterations.Source: §3.2.1 · p. 4
3.3 Per-channel & Mixed-channel models
- The mixed-channel model increases signal-to-noise performance but decreases subjective color accuracy relative to the per-channel model.It replaces k learned RGB kernels with 3 × k single-channel kernels, enabling cross-channel communication across feature maps.Source: §3.3 · p. 5
4 IMPLEMENTATION
The implementation uses a custom housing to position a diffuser near a Raspberry Pi camera sensor, then captures paired training and test images from a display.
Camera Design
- A 0.5-degree engineered diffuser sits about 10 mm from the sensor in a custom 3D-printed housing for a Raspberry Pi High-Quality camera.The housing also has space for an optional infrared filter.Source: §4 · p. 5
Data Capture
- The camera captures an on-axis PSF from a 5×5 illuminated pixel grid, then uses FISTA reconstruction to estimate a homography for matching ground-truth perspective.Source: §4 · p. 5
- Automated software presents DIV2K images on a 5.5-inch OLED display, yielding 8,000 training images and 1,000 test images.Source: §4 · p. 5
5 EVALUATION
Evaluation compares the method with DiffuserCam work, tests component ablations, and includes reconstructions from the lab prototype. The reported qualitative results show improved image quality, while ablations identify contributions from the reconstruction network and multiple learned models.
5 EVALUATION
- The evaluation compares against two state-of-the-art works using the DiffuserCam dataset, conducts ablations, and verifies the method with the hardware prototype.Source: §5 · p. 5
5.1 DiffuserCam results
- The method improves reconstructed-image quality on DiffuserCam measurements, with features appearing more faithful to ground truth.Source: §5.1 · p. 5
5.2 Ablation Studies
- Without the U-Net denoiser, intermediate reconstructions are noisy but remain faithful to ground truth, indicating the model-based network performs most of the reconstruction work.Source: §5.2 · p. 5
- Reducing the number of learned models from five to one decreases post-processed image quality by about 2 dB.Source: §5.2 · p. 5
5.3 Prototype results
- The evaluation also presents sample reconstructions from the lab-built prototype camera.Source: §5.3 · p. 5
6 DISCUSSION
The discussion reports gains from learned kernels and contrasts color accuracy with recovered detail, while noting unresolved questions about model interpretation, spatial aberrations, and prototype image quality.
Discussion
- The learned components are difficult to map individually onto classical methods, and future work could test learned physical models within classical variational methods.An independently learned forward model could assess reconstructed-image data fidelity against lensless measurements.Source: §6 · p. 5
- Against fixed-PSF learned methods, the reconstructions more closely resemble lensed-camera images, but the exact mechanism behind the improvement remains unclear.The authors suggest field-varying aberrations may be involved, although the method has no explicit way to assign learned models to spatial regions.Source: §6 · p. 5
- With U-Net denoising disabled, intermediate reconstructions are noisy but faithful to ground truth, and the model-based network performs most of the reconstruction work.The figure comparison is against converged unsupervised ADMM; the authors reason that physical-model consistency yields fewer artifacts.Source: Figure 4 · p. 6
- ∼2dB PSNR from increasing learned kernels with U-Net denoising, with cross-channel denoising adding another ∼2dB.The learned kernels occupy most of the parameter space; adding a small U-Net also improves reconstruction quality.Source: Table 1 · p. 7
- The in-house prototype’s 0.76mm thick holographic diffuser has bulk scattering that degrades reconstruction quality.Source: Figure 5 · p. 7
- Measured or simulated field-varying PSFs are proposed for robust models that explicitly correct field-varying aberrations without manual calibration.Source: §6 · p. 7
- Per-channel reconstructions subjectively improve color accuracy, while mixed-channel reconstructions recover higher-frequency content.The authors suspect the per-channel model is vulnerable to color fringing from the chosen phase masks.Source: Figure 3 · p. 6, §6 · p. 7
7 CONCLUSION
Thin-mask cameras offer freedom from traditional optical constraints, but reconstruction speed and quality remain drawbacks. The paper argues that neural networks with embedded physical priors can provide on-par reconstruction quality without extensive training resources.
Conclusion
- Thin masks offer freedom from the constraints of traditional optics, while reconstruction speed and image quality remain significant drawbacks.Source: §7 · p. 7
- Neural networks with embedded physical priors can help counter these drawbacks and provide on-par reconstruction quality without extensive training resources.Source: §7 · p. 7
- The authors hope this work will support further development of performant and interpretable lensless image reconstruction methods.Source: §7 · p. 7