Source-linked AI summary
Imaging through glass diffusers using densely connected convolutional networks
Shuai Li, Mo Deng, Justin Lee, Ayan Sinha, George Barbastathis
TL;DR
Imaging through scattering normally requires explicit characterization of the medium and object priors, but these representations can be incomplete or infeasible. The paper introduces IDiffNet, a densely connected convolutional network trained to learn the inverse mapping from speckle patterns. It finds that reconstruction depends on scattering strength, training-data sparsity, and loss function, with NPCC particularly useful for sparse objects under strong scattering.
Problem
Explicitly determining scattering forward operators and object regularizers can be incomplete, error-prone, or infeasible for ill-posed imaging through scatter.
Method
IDiffNet uses a densely connected convolutional architecture trained on examples to learn the inverse mapping from detector speckle patterns to object reconstructions.
Results
Reconstruction performance depends on diffuser scattering strength, training-dataset sparsity, and optimization loss; NPCC is preferable for sparse objects under strong scattering.
Takeaways & Limitations
IDiffNet can learn scattering properties and object priors automatically, with the effective training database and loss function depending on the imaging conditions.
Takeaways & Limitations
The diffuser is modeled as a thin mask and its transmittance as a pure-phase object.
Abstract
from arXiv · showhide
Computational imaging through scatter generally is accomplished by first characterizing the scattering medium so that its forward operator is obtained; and then imposing additional priors in the form of regularizers on the reconstruction functional so as to improve the condition of the originally ill-posed inverse problem. In the functional, the forward operator and regularizer must be entered explicitly or parametrically (e.g. scattering matrices and dictionaries, respectively.) However, the process of determining these representations is often incomplete, prone to errors, or infeasible. Recently, deep learning architectures have been proposed to instead learn both the forward operator and regularizer through examples. Here, we propose for the first time, to our knowledge, a convolutional neural network architecture called "IDiffNet" for the problem of imaging through diffuse media and demonstrate that IDiffNet has superior generalization capability through extensive tests with well-calibrated diffusers. We found that the Negative Pearson Correlation Coefficient loss function for training is more appropriate for spatially sparse objects and strong scattering conditions. Our results show that the convolutional architecture is robust to the choice of prior, as demonstrated by the use of multiple training and testing object databases, and capable of achieving higher space-bandwidth product reconstructions than previously reported.
1. INTRODUCTION
Imaging through scattering is difficult because propagation loses and scrambles information, while conventional inverse methods require explicit medium models and object priors. The paper introduces IDiffNet, a convolutional approach that learns these elements from examples and examines its generalization and loss-function behavior.
- Scattering loses out-of-aperture light and scrambles transmitted spatial-frequency phases, making information extraction difficult.
- When the medium is unknown or incompletely characterized, deconvolution remains difficult and can produce noise-induced artifacts even with a known convolution kernel.
- Tikhonov-Wiener reconstruction combines a measured forward operator with a prior-based regularizer to constrain the ill-posed inverse problem.
- Existing approaches characterize random media using transmission-matrix measurements, angular-memory speckle correlations, or sparsity priors.
- IDiffNet learns the scattering response and object prior simultaneously from examples, using a convolutional architecture and Negative Pearson Correlation Coefficient loss.
- The study evaluates generalization across object databases and scattering conditions, including sparse classes, and compares NPCC with MAE training.
2. COMPUTATIONAL IMAGING SYSTEM ARCHITECTURE
The experimental system uses an intensity-modulating SLM, calibrated glass diffusers, telescopic imaging, and a CMOS camera, while IDiffNet learns the inverse mapping from speckle patterns. Diffuser grit controls the simulated PSF and scattering strength, and the network uses dense, dilated, downsampling, upsampling, and skip-connected convolutional blocks.
- A He-Ne laser illuminates a reflective SLM acting as a pixel-wise intensity object before polarization analysis, glass diffusion, telescopic imaging, and CMOS detection.
- The telescope magnifies the object by 0.6, matching CMOS and SLM pixel sizes; processing uses the central 512 × 512 CMOS crop.
- The diffuser is approximated as a thin mask with amplitude transmittance, and its forward model relates object-plane and detector-plane optical fields.
- Diffuser surface heights are generated by smoothing normally distributed random values, with parameters determined by glass-diffuser grit size.
- The 600-grit diffuser has a sharper PSF, whereas the more widely spread 220-grit PSF indicates stronger scattering.
- The optical intensity forward operator maps object intensity to detector intensity through propagation, the diffuser, and modulus-square and square-root operations.
- IDiffNet receives a CMOS speckle pattern and reconstructs the object through densely connected convolutional blocks with dilated convolutions and skip connections.
3. RESULTS AND NETWORK ANALYSIS
IDiffNet was evaluated across diffuser strengths, training databases, and loss functions. NPCC improved reconstruction of sparse objects under strong scattering, while dataset choice governed generalization and overfitting.
- Experimental setup: 10,000 object-speckle pairs were collected for each of six IDiffNet training configurations using Faces-LFW, ImageNet, or MNIST with two glass diffusers.Inputs and outputs were matched at 128 × 128 dimensions after cropping, downsampling, and reference-speckle subtraction.
- MAE results: 600-grit diffusion produced better MAE reconstructions than 220-grit diffusion, with ImageNet training giving the strongest result under 220-grit scattering.Under 220-grit scattering, reconstructions generally retained salient shapes but lost high-frequency features.
- MAE results: MNIST-trained IDiffNet produced featureless uniform estimates under 220-grit scattering with MAE because sparse local minima trapped the network.The sparse training objects encouraged sparse estimates, but the resulting minima lacked recognizable features.
- NPCC results: NPCC enabled high-quality 220-grit reconstructions for Characters and MNIST test images when trained on MNIST, unlike the MAE-trained network.NPCC helps the network learn ground-truth sparsity and use it as a strong inverse-problem prior.
- NPCC results: NPCC-trained ImageNet reconstructions retained general object shapes under 220-grit scattering but were visually slightly worse than MAE-trained reconstructions.This shows a trade-off between sparse-object reconstruction under severe scattering and reconstruction quality for generic objects.
- Network analysis: Generic training datasets generalized best for weak scattering, whereas constrained sparse datasets helped overcome strong scattering at the cost of greater overfitting.ImageNet training generalized well across databases and diffuser strengths, while MNIST and Faces-LFW showed stronger database-specific behavior.
4. COMPARISON WITH DENOISING NEURAL NETWORKS
IDiffNet reconstructs diffuse speckle images rather than merely denoising them, producing better results than a denoising network on the same diffuse inputs. Maximally activated patterns further indicate that the networks learn different internal representations.
- Comparison with denoising neural networks: A denoising network performs well on Poisson-noisy inputs but produces highly blurred outputs for diffuse speckle images, whereas IDiffNet reconstructs them better.The comparison used ImageNet training data and matched the simulated noise and captured speckle images by visual PSNR.
- Comparison with denoising neural networks: The diffuse-image result shows that IDiffNet is not simply performing denoising despite the visual similarity between speckle and noisy images.
- Comparison with denoising neural networks: The denoising network learns spatial structure primarily as an object-class prior because Poisson noise acts pixel-wise.
- Network analysis: IDiffNet and denoising-network maximally activated patterns are qualitatively different in both shallow and deep convolutional layers.MAPs are inputs optimized to maximize the average response of network filters.
5. CONCLUSIONS
The study demonstrates IDiffNet as an end-to-end approach for imaging through scattering media. Reconstruction depends on diffuser strength, training-dataset sparsity, and the optimization loss, while the network learns scattering properties and object priors from training data.
- IDiffNet provides an end-to-end approach for imaging through scattering media using a densely connected convolutional architecture.
- Reconstruction performance depends on diffuser scattering strength, training-dataset type and sparsity, and the loss function used for optimization.
- IDiffNets appear to learn both scattering-media properties and object priors automatically from the training data.The learned priors restrict the objects on which the network is expected to perform well.
FUNDING INFORMATION
The research was funded by the Singapore National Research Foundation through SMART and by IARPA through the RAVEN Program.
- The research received funding from Singapore’s National Research Foundation through the SMART program and from IARPA through the RAVEN Program.
- Justin Lee received support from the U.S. Department of Energy Computational Science Graduate Fellowship.