PaperlayerView exact revision on arXiv

Source-linked AI summary

Learned reconstructions for practical mask-based lensless imaging

Kristina Monakhova, Joshua Yurtsever, Grace Kuo, Nick Antipa, Kyrollos Yanny, Laura Waller

arXiv:1908.11502v1eess.IVcs.CV

TL;DR

Iterative lensless reconstruction can be slow and artifact-prone, limiting interactive previewing. The paper unrolls a physical model-based algorithm into trainable networks and reports faster reconstruction with improved perceptual quality, including tests on natural images captured in the wild.

  • Problem

    Iterative lensless reconstruction can produce artifacts and require computation too complex for interactive previewing.

    Source: §1 · p. 1

  • Method

    The authors unroll lensless-specific ADMM into bounded-compute trainable networks and optimize their parameters using experimentally captured image pairs.

    Source: §1 · p. 3

  • Results

    Le-ADMM-U has 3× better image quality than standard algorithms on the LPIPS perceptual similarity scale, while the fastest networks are 20× faster than converged ADMM.

    Source: §6.1 · p. 9, §8 · p. 12

  • Takeaways & Limitations

    The number of learned parameters can be varied by application, with the networks performing comparably to classic algorithms at a fraction of the speed.

    Source: §7 · p. 10

    For photography, deeper methods may produce more visually appealing images at the expense of possibly hallucinating details not present in the scene.

    Source: §7 · p. 11

Abstract

from arXiv · show

Mask-based lensless imagers are smaller and lighter than traditional lensed cameras. In these imagers, the sensor does not directly record an image of the scene; rather, a computational algorithm reconstructs it. Typically, mask-based lensless imagers use a model-based reconstruction approach that suffers from long compute times and a heavy reliance on both system calibration and heuristically chosen denoisers. In this work, we address these limitations using a bounded-compute, trainable neural network to reconstruct the image. We leverage our knowledge of the physical system by unrolling a traditional model-based optimization algorithm, whose parameters we optimize using experimentally gathered ground-truth data. Optionally, images produced by the unrolled network are then fed into a jointly-trained denoiser. As compared to traditional methods, our architecture achieves better perceptual image quality and runs 20x faster, enabling interactive previewing of the scene. We explore a spectrum between model-based and deep learning methods, showing the benefits of using an intermediate approach. Finally, we test our network on images taken in the wild with a prototype mask-based camera, demonstrating that our network generalizes to natural images.

1. Introduction

Lensless cameras encode scene information indirectly, so reconstruction methods must balance image quality and computational cost. This work unrolls a physical model-based algorithm into a trainable, bounded-compute network and evaluates its speed, perceptual quality, and generalization.

1. Introduction

  • Lensless cameras can be small, cheap, and lightweight; their multiplexing can support higher-dimensional signals such as 3D volumes or video from a single 2D measurement.Reported applications include fluorescence microscopy, thermal imaging, and refocusable photography.Source: §1 · p. 1
  • Iterative reconstructions allow priors and generally produce better results, but are slow, can contain artifacts, and can require bulky compute hardware that precludes interactive previewing.Single-step methods can be fast but often require custom masks and make it difficult to incorporate priors or compressed sensing.Source: §1 · p. 1
  • Model-based iterative methods can take hundreds to thousands of iterations and remain sensitive to model mismatch, calibration errors, hand-tuned parameters, and hand-picked regularizers.Deep reconstruction networks can learn scene statistics but do not incorporate prior knowledge of image formation and are difficult to interpret.Source: §1 · p. 2
  • The proposed networks unroll lensless-specific ADMM, learn parameters from experimentally captured data, and span classic-to-deep variants with bounded compute and a data-fidelity–perceptual-quality trade-off.The training pipeline compares network outputs with lensed-camera ground-truth labels; the network variants differ in their number of trainable parameters.Source: Figure 1 · p. 2, §1 · p. 3
  • The prototype network generalizes from monitor-trained images to natural objects captured in the wild.The work uses DiffuserCam as its prototype lensless camera.Source: §1 · p. 3
  • 20× speedup and 3× better perceptual similarity are reported for lensless reconstructions, alongside a dataset of 25,000 aligned lensed and lensless image pairs.The paper also reports generalization to images taken in the wild on a prototype lensless camera.Source: §1 · p. 3

2. Lensless imaging forward model

The forward model describes how a scene becomes a sensor measurement in DiffuserCam. It uses a shift-invariant point-spread function and crops the convolution to the physical sensor dimensions.

2. Lensless imaging forward model

  • DiffuserCam models light from each scene point as a shift-invariant point-spread function, with the sensor measurement formed from incoherent point sources of varying color and intensity.The diffuser refracts light into a high-contrast caustic pattern on the sensor; a lateral source shift translates the pattern in the opposite direction.Source: §2 · p. 4
  • The forward model convolves the scene with the system PSF and crops the result to the physical sensor size; in matrix notation, crop and convolution are C and H.The relation is computed separately for each color channel.Source: §2 · p. 4
  • Reconstruction seeks the scene from the measurement, assuming the PSF is known; traditional methods use a regularized objective with a sparsifying transform Ψ and tuning parameter τ.The PSF can be measured experimentally with an LED point source; τ adjusts the sparsity level.Source: §2 · p. 4

3. Model-based inverse algorithm

The inverse solver uses ADMM with a variable split that exploits the problem structure to produce closed-form updates. Its performance still depends on hand-chosen parameters, and convergence can take hundreds of iterations.

3. Model-based inverse algorithm

  • ADMM reformulates the physics-based inverse problem with a variable split that enables closed-form updates at each iteration.The solver minimizes the difference between actual and predicted measurements while satisfying additional constraints.Source: §3 · p. 4
  • The architecture feeds the measurement and calibration PSF through unrolled Le-ADMM layers, with an optional denoiser and training against the lensed image.Red arrows indicate backpropagation through network parameters.Source: Figure 3 · p. 5
  • The updates include sparsifying soft-thresholding, a least-squares step, and enforcement of non-negativity.The update parameters include dual variables and scalar penalty parameters.Source: §3 · p. 5
  • Traditional ADMM depends heavily on hand-chosen sparsifying transforms and parameters, and may require hundreds of iterations while producing reconstruction artifacts.It performs well when the transform and parameters are chosen correctly.Source: §3 · p. 5

4. Learned reconstruction networks

The learned reconstruction networks unroll ADMM so the physical model remains part of the architecture while parameters are learned from data. The variants add trainable regularization or denoising, trading additional parameters for learned image priors and correction of model mismatch.

4. Learned reconstruction networks

  • The network variants unroll ADMM iterations as layers, retaining the physical model while learning tunable parameters from training data.All variants use five iterations, targeting reconstruction in 75 ms rather than 1.5 s.Source: §4 · pp. 5–6

4.1. Learned AMMM (Le-ADMM)

  • Le-ADMM learns ADMM tuning and hyper-parameters, with 20 learned parameters across five unrolled layers.Its parameters are updated by backpropagation against the ground-truth lensed images.Source: §4.1 · p. 6

4.2. Le-ADMM*, with learned regularizer

  • Le-ADMM* adds a trainable CNN regularizer, using 32,135 learned parameters to learn a data prior and account for forward-model mismatch.The added parameters require more training data.Source: §4.2 · pp. 6–7

4.3. Le-ADMM-U

  • Le-ADMM-U jointly trains an unrolled Le-ADMM reconstruction and a final U-Net denoiser, with 10,605,927 learned parameters.The Le-ADMM portion performs most deconvolution using the forward model; the U-Net denoises and can correct model mismatch.Source: §4.3 · p. 7

4.4. U-Net

  • The comparison also includes a standalone U-Net with 10,605,907 learned parameters and no system-physics knowledge in reconstruction.Source: §4.4 · p. 7

4.5. Loss functions

  • Learned methods train with combined MSE and LPIPS losses, while classical methods use data fidelity with total variation regularization.MSE tends to favor low frequencies and produce blurry reconstructions, motivating the use of perceptual LPIPS features alongside it.Source: §4.5 · p. 7, Table 1 · p. 8

5. Implementation

The implementation pairs lensless captures with lensed ground truth using a synchronized optical setup. Training uses a down-sampled MirFlickr dataset and a combined pixelwise and perceptual loss.

5. Implementation

  • A lensed camera, DiffuserCam, beamsplitter, and computer monitor are triggered together to collect paired training images overnight.Source: §5 · p. 8
  • The prototype uses an off-the-shelf diffuser about 9 mm from a CMOS sensor, and a calibration PSF from an LED point source at the screen distance.The paired camera views are optically aligned and then co-aligned with a homography computed from displayed points.Source: §5 · p. 8
  • The dataset has 25,000 MirFlickr images, split into 24,000 training and 1,000 test images, down-sampled to 480×270 to reduce screen moiré.Training combines MSE and LPIPS, with greater MSE weight early and greater LPIPS weight later.Source: §5 · p. 8

6. Results

On the test set, the fastest learned networks match or exceed converged ADMM at the runtime of bounded ADMM, and Le-ADMM-U has the strongest reported reconstruction performance. The learned networks also produce visually strong reconstructions of natural objects under variable lighting.

6. Results

  • The comparison uses 100-iteration converged ADMM as a classic-method reference, five-iteration bounded ADMM as a similar-runtime baseline, and a U-Net trained on raw measurements.The U-Net uses raw DiffuserCam measurements and ground-truth labels.Source: §6 · p. 8

6.1. Test set results

  • Moving from Le-ADMM to Le-ADMM-U improves MSE and LPIPS but worsens data fidelity, as added learned parameters correct artifacts from model mismatch.Across network layers, MSE and LPIPS tend to decrease while data fidelity increases; for Le-ADMM-U, the U-Net improves MSE and LPIPS at the cost of data fidelity.Source: §6.1 · pp. 9–10
  • 20× faster than converged ADMM, Le-ADMM and Le-ADMM-U achieve comparable or better average MSE and LPIPS on 1,000 test images.The fastest learned networks run in the same amount of time as five-iteration bounded ADMM.Source: §6 · p. 8, §6.1 · p. 9, Table 2 · p. 10
  • Le-ADMM-U has 3× better image quality than converged ADMM as measured by LPIPS, and the sample reconstructions are visually similar to the ground truth.Figure 4 shows Le-ADMM* and Le-ADMM-U with noticeably better visual quality than Le-ADMM.Source: §6.1 · pp. 9–10, Figure 4 · p. 9

6.2. Generalization to images in the wild

  • On natural objects imaged in the wild, the learned networks produce images of similar or higher visual quality than converged ADMM, with Le-ADMM-U particularly appealing.These reconstructions generalize beyond monitor images to 3D objects with variable lighting conditions.Source: §6 · p. 9, §6.2 · p. 10

7. Discussion

The learned reconstructions combine model-based and deep methods, with test-set and in-the-wild results showing strong image quality. The discussion also notes trade-offs involving data fidelity, model size, and the training data’s limits.

7. Discussion

  • The authors favor combining classic and deep methods, reporting comparable performance to classic algorithms at a fraction of the speed and improved image quality with more learned parameters.The parameter count can be varied by application; deeper methods may produce more appealing images but risk hallucinating details.Source: §7 · pp. 10–11
  • On the test set, Le-ADMM-U has the best LPIPS and MSE scores and outperforms converged ADMM, while Le-ADMM matches it with fewer training pairs.Learned methods have higher data-fidelity error, indicating less consistency with the image-formation model.Source: Figure 5 · p. 11
  • Reconstruction quality and resolution are bounded by the training dataset, including imperfections in the physical system.Lensed-camera or beamsplitter aberrations can affect learned reconstructions; distortion is corrected before training, and chromatic aberration and field curvature are negligible at the reconstruction grid size.Source: §7 · p. 11
  • In wild-object captures, learned networks match or exceed converged ADMM’s visual quality, with Le-ADMM-U producing the most visually appealing images.Examples include details in a sideways plant and a toy duck’s eye.Source: Figure 6 · p. 12

8. Conclusion

The paper presents unrolled model-based networks for lensless imaging, combining imaging physics with learned parameters. They offer image quality comparable to or better than standard algorithms, with faster reconstruction and a training step before camera operation.

8. Conclusion

  • The networks produce similar or better image quality than standard algorithms, with the fastest offering a 20× improvement.The deeper Le-ADMM-U achieves 3× better image quality than standard algorithms on the LPIPS perceptual similarity scale.Source: §8 · p. 12
  • Le-ADMM-U achieves 3× better image quality than standard algorithms on the LPIPS perceptual similarity scale.The conclusion also reports that learned reconstruction is fast enough for interactive scene previewing.Source: §8 · p. 12
  • The networks jointly incorporate imaging physics and learned parameters, and their use requires a training step before camera operation.The authors suggest this approach could improve lensless imaging speed and quality.Source: §8 · p. 12

Funding

The work acknowledges support from the NSF Graduate Research Fellowship, STROBE, and the Gordon and Betty Moore Foundation’s Data-Driven Discovery Initiative.

Funding

  • The work was supported by the National Science Foundation Graduate Research Fellowship (DGE 1752814).Source: Funding · p. 13
  • STROBE, a National Science Foundation Science & Technology Center, supported the work (DMR 1548924).Source: Funding · p. 13
  • The Gordon and Betty Moore Foundation’s Data-Driven Discovery Initiative (GBMF4562) supported the work.Source: Funding · p. 13

Network architecture

The architecture section describes U-Net components used in the learned networks and compares performance as training-set size varies. Le-ADMM-U performs better than U-Net with fewer training images.

Network architecture

Effect of training size

  • Le-ADMM-U performs better than U-Net with fewer training images, while Le-ADMM does not improve as training-set size increases.The authors attribute Le-ADMM-U’s advantage to incorporating knowledge of the imaging system, and explain Le-ADMM’s lack of improvement by its few parameters.Source: Effect of training size · p. 14, Figure 7 · p. 14
Loading 1908.11502v1…