Source-linked AI summary
Lensless computational imaging through deep learning
Ayan Sinha, Justin Lee, Shuai Li, George Barbastathis
TL;DR
The paper asks whether deep neural networks can solve inverse problems in computational imaging, where raw intensity measurements must be used to recover an object. It trains a DNN on experimental diffraction data from a lensless phase-imaging system and finds accurate reconstruction across object classes and modest distance changes. The authors also suggest that the network learns a generalizable model of the imaging system rather than merely matching patterns.
Problem
The paper addresses whether a neural network can recover object estimates from raw intensity images in a computational-imaging inverse problem.
Method
The authors train convolutional residual DNNs using experimentally generated diffraction patterns and corresponding phase objects from ImageNet or Faces-LFW.
Results
The trained networks accurately reconstruct objects outside their training classes and remain accurate when testing object-to-sensor distances slightly differ from training distances.
Takeaways & Limitations
The results suggest that DNNs can learn a generalizable inverse mapping and aspects of the imaging system without requiring a prior forward model.
Takeaways & Limitations
The study deliberately controls training and testing with an SLM, leaving application-specific training with physical phase objects beyond its scope.
Abstract
from arXiv · showhide
Deep learning has been proven to yield reliably generalizable answers to numerous classification and decision tasks. Here, we demonstrate for the first time, to our knowledge, that deep neural networks (DNNs) can be trained to solve inverse problems in computational imaging. We experimentally demonstrate a lens-less imaging system where a DNN was trained to recover a phase object given a raw intensity image recorded some distance away.
1. INTRODUCTION
The paper tests whether a neural network can recover phase objects from raw intensity measurements in a lensless diffraction-imaging system. Results indicate that DNNs learn an inverse mapping that generalizes across object classes and modest changes in object-to-sensor distance.
- 1. INTRODUCTION: The study tests whether a neural network can recover object estimates from raw intensity images, thereby solving an inverse computational-imaging problem.The experiment uses a lensless setup in which diffraction patterns from pure phase objects are captured under coherent illumination.
- 1. INTRODUCTION: The proposed approach learns the inverse mapping directly from experimental data rather than relying on a prior model of the imaging system.The network is suggested to learn the system’s forward operator and deviations from idealized assumptions.
- 1. INTRODUCTION: Training examples are synthesized from ImageNet, with a separate network trained on the narrower Faces-LFW image class.Sampling all possible phase objects is considered unrealistic, so the paper uses natural and facial image databases as representative phase objects.
- 1. INTRODUCTION: Both networks reconstruct objects outside their training classes, including faces, natural objects, handwritten digits, multilingual characters, and images from a disjoint natural-image dataset.The result extends beyond the expected within-class reconstruction performance.
- 1. INTRODUCTION: Both trained networks remain accurate when the object-to-sensor distance in testing slightly differs from the training distance, suggesting a generalizable system model rather than simple pattern matching.This observation supports the paper’s claim that the network approximates the underlying imaging system.
2. EXPERIMENT
The experiment uses a phase-only spatial light modulator to generate diffraction patterns, which a residual convolutional network reconstructs as phase objects. Multiple training runs and cross-dataset testing evaluate reconstruction behavior.
- 2. EXPERIMENT: A He-Ne laser, spatial filter, collimating lens, mirror, polarizers, beam splitter, SLM, and CMOS detector form the lensless experimental arrangement.The CMOS records diffraction patterns after free-space propagation from the modulated phase object.
- 2. EXPERIMENT: Polarization settings make the SLM’s amplitude modulation nearly independent of its 8-bit input while retaining monotonic, approximately linear phase modulation.The incident polarization is set to 45° and the analyzer to 340° relative to the vertical direction.
- 2. EXPERIMENT: The CMOS detector records diffraction patterns at object-to-sensor distances ranging from approximately 37.5–97.5 cm during training and testing.Training uses randomly selected Faces-LFW or ImageNet samples displayed on the SLM.
- 2. EXPERIMENT: The DNN uses a convolutional ResNet architecture with seven convolution-plus-downsampling blocks, six deconvolution-plus-upsampling blocks, and two final residual blocks.Dilated convolutions aggregate diffraction effects across scales, while skip connections preserve high-frequency information.
- 2. EXPERIMENT: The network is trained by comparing its final phase estimate with ground-truth phase values using the defined output dimensions and phase target.Six separate DNNs are trained from six experiment runs spanning two databases and three distances.
- 2. EXPERIMENT: Testing samples from the training databases and other datasets are displayed on the SLM, measured by the camera, reconstructed by the DNN, and compared with ground truth.The described external datasets include MNIST, CIFAR, Faces-ATT, and other image collections.
3. RESULTS AND NETWORK ANALYSIS
The trained networks reconstructed phase objects from raw diffraction intensities and generalized across datasets, while remaining moderately robust to sensor displacement. Network analysis suggests inversion relies on low-level diffraction-related textures.
- Quantitative performance: ImageNet-trained networks converged after approximately 10 epochs and showed low test error on non-trained datasets, indicating cross-domain generalization.Lower MAE occurred for characters, digits, and Null images because their sparse diffraction patterns were easier to invert.
- Cross-domain generalization: ImageNet-trained networks accurately reconstructed faces, handwritten digits, and multilingual characters despite training exclusively on ImageNet images.The result suggests learning of the imaging system’s underlying physics or a generalizable mapping of low-level textures.
- Qualitative reconstruction: The networks reconstructed images from raw intensity measurements across three object-to-sensor distances and multiple test datasets.Figure 4 compares ground truth, calibrated phase, raw intensity, and DNN reconstructions for Faces-LFW- and ImageNet-trained networks.
- Cross-domain generalization: Faces-ATT-trained networks accurately reconstructed ImageNet, handwritten digits, and character images, unlike the cited face-trained SVM that failed on other object classes.This qualitative comparison appears in Figure 4 columns (iv), (vii), and (x).
- Robustness: The network remained robust to moderate lateral and axial sensor displacement and was somewhat shift- and rotation-invariant, but failed under significantly larger displacement.The displacement analyses are reported in Figures 6–9, with qualitative examples for networks trained at 37.5 cm.
- Network analysis: Maximally activated patterns indicated that the phase-retrieval network used fine, low-level textures associated with localized diffraction patterns.These patterns were compared with those from an ImageNet classifier and a similarly structured deblurring network.
4. CONCLUSIONS AND DISCUSSION
The study used a deliberately controlled SLM-based architecture for precise analysis, while leaving practical physical-object imaging and complex-object reconstruction for future work.
- Conclusions and discussion: The controlled architecture used an SLM to create phase-object inputs during both training and testing, enabling quantitative analysis of learning behavior.Application-specific training with physical phase objects was considered beyond the scope of this work.
- Conclusions and discussion: Future extensions include adding microscope optics and reconstructing complex objects that combine attenuation and phase delay.The authors anticipate that a neural network could learn the optics automatically for complex-object reconstruction.
FUNDING INFORMATION
The research received funding from Singapore’s National Research Foundation through SMART, iARPA through RAVEN, and the U.S. Department of Energy’s CSGF for Justin Lee.
- Singapore’s National Research Foundation funded the research through the SMART program.SMART is identified as the Singapore-MIT Alliance for Research and Technology.
- iARPA supported the research through its RAVEN Program.
- Justin Lee acknowledges support from the U.S. Department of Energy Computational Science Graduate Fellowship.The fellowship grant is identified as DE-FG02-97ER25308.