Source-linked AI summary
Phase recovery and holographic image reconstruction using deep learning in neural networks
Yair Rivenson, Yibo Zhang, Harun Gunaydin, Da Teng, Aydogan Ozcan
TL;DR
Intensity-only measurements lose the phase information needed for coherent imaging and holographic reconstruction. The paper trains a neural network to recover phase and reconstruct holographic images from one hologram, reducing measurements and computation while suppressing spatial artifacts.
Problem
Intensity-only sensors cannot directly record phase, motivating phase recovery for reconstructing specimen amplitude and phase images without twin-image artifacts.
Method
A trained convolutional neural network performs phase recovery and holographic reconstruction from a single intensity-only hologram.
Results
The method reduces required holograms by 2-3 times and improves reconstruction computation by more than 3- and 4-fold versus multi-height recovery using 2 and 3 holograms, respectively.
Takeaways & Limitations
The framework reconstructs improved phase and amplitude images while suppressing twin-image, self-interference, and out-of-focus interference artifacts in biological samples.
Takeaways & Limitations
The evaluation assumes weakly scattering samples, and RBC measurements use effective refractive volume because refractive index is difficult to estimate.
Abstract
from arXiv · showhide
Phase recovery from intensity-only measurements forms the heart of coherent imaging techniques and holography. Here we demonstrate that a neural network can learn to perform phase recovery and holographic image reconstruction after appropriate training. This deep learning-based approach provides an entirely new framework to conduct holographic imaging by rapidly eliminating twin-image and self-interference related spatial artifacts. Compared to existing approaches, this neural network based method is significantly faster to compute, and reconstructs improved phase and amplitude images of the objects using only one hologram, i.e., requires less number of measurements in addition to being computationally faster. We validated this method by reconstructing phase and amplitude images of various samples, including blood and Pap smears, and tissue sections. These results are broadly applicable to any phase recovery problem, and highlight that through machine learning challenging problems in imaging science can be overcome, providing new avenues to design powerful computational imaging systems.
Introduction
The paper introduces a deep-learning framework that recovers phase and reconstructs holographic images from a single intensity-only hologram. It targets artifacts and measurement burdens in coherent imaging, demonstrating faster reconstruction on blood, Pap smear, and tissue samples.
- Motivation: Opto-electronic imagers record light intensity but not object or diffracted-wave phase, motivating phase recovery from intensity-only measurements.Phase recovery enables reconstruction of phase and amplitude images.
- Method: A convolutional neural network trained through deep learning performs phase recovery and holographic image reconstruction using a single intensity-only hologram.The method uses a multi-layered artificial neural network for data modeling, analysis, and decision making.
- Experimental demonstration: The framework was demonstrated with lensfree digital in-line holography of transmissive human tissue sections, blood, and Pap smears.Three convolutional neural networks were separately trained for the three sample types using an identical architecture.
- Training: Training maps a single-hologram back-propagation image to the same object’s multi-height phase-retrieval reconstruction, treated as the gold standard.The multi-height reference uses 8 hologram intensities acquired at different heights.
- Results: The neural-network reconstruction reduces required holograms by 2-3 times and improves computation time by more than 3- and 4-fold versus multi-height retrieval with Nholo = 2 and Nholo = 3, respectively.Single-hologram performance closely matches SSIM for Nholo = 2 in Pap smear and breast tissue and Nholo = 3 in blood smear samples.
- Artifact suppression: The framework significantly suppresses out-of-focus interference artifacts caused by dust particles and imperfections in imaging surfaces or optical components.Artifacts highlighted in holographic images were cleaned in corresponding network outputs.
Figures and Tables
The figures and tables show that a trained deep neural network reconstructs artifact-free phase and amplitude images from one hologram intensity across multiple biological samples. They also compare this approach with multi-height recovery, evaluate defocus tolerance and network generality, and report reconstruction runtimes.
- Network operation: After training, the network outputs artifact-free phase and amplitude images from one hologram intensity using convolutional, residual, and upsampling blocks.It processes complex-valued inputs in parallel across multiple scales.
- Sample reconstruction: Single-hologram free-space backpropagation produces twin-image and self-interference artifacts in Pap smear and breast tissue reconstructions.The figure compares these artifact-contaminated results with neural-network holographic reconstructions.
- Sample reconstruction: The neural-network red-blood-cell volume estimates show good agreement with multi-height phase recovery using Nholo = 8.The multi-height results exhibit an out-of-focus fringe artifact at the center of the cell.
- Robustness evaluation: The defocus-tolerance evaluation uses the SSIM index to compare neural-network outputs against multi-height phase recovery with Nholo = 8.Inputs are intentionally deviated from the optimal training focus.
- Network generality: Sample-type-specific and universal networks are compared using single hologram intensities across blood smears, Papanicolaou smears, and breast tissue sections.The comparison also includes multi-height reconstruction with Nholo = 8.
- Runtime comparison: For a field of view of ~1 mm2, single-hologram reconstruction requires 6.45 sec with the sample type specific network and 7.85 sec with the universal neural network.The corresponding deep neural network processing times are 3.11 sec and 4.51 sec, respectively.
Network architecture
The network processes a back-propagated hologram intensity through separate real and imaginary input channels and four convolutional layers. Training minimizes mean-squared error against real and imaginary ground-truth object images.
- Inputs: The real and imaginary parts of the back-propagated hologram intensity form two M×N input channels, exemplified by M = 1392 and N = 1392.The two channels are supplied simultaneously to the network.
- Convolutional architecture: Four convolutional layers each produce 16 M×N feature maps, using 3×3 convolutional kernels throughout the implementation.The 3×3 kernel specification appears in the architecture description.
- Training objective: The loss function compares the network’s real and imaginary outputs with corresponding real and imaginary ground-truth object images using mean-squared error.Ground-truth images are obtained using multi-height phase retrieval with 8 holograms.
- Training objective: The loss is calculated over a mini-batch of K input image patches.The network parameters include kernels, biases, and weights, collectively represented by Θ.
Pixel super resolution (PSR)
Pixel super resolution mitigated spatial undersampling from the sensor’s relatively large pixel pitch by synthesizing a higher-resolution hologram from multiple subpixel-shifted measurements.
- Pixel super resolution (PSR): A 6-by-6 rectangular x-y shift grid with 0.37 μm increments, approximately one-third of the sensor pixel size, synthesized a pixel-super-resolved hologram.The sensor was mechanically shifted to acquire the subpixel-shifted holograms.
Calculation of red blood cell (RBC) phase integral and effective refractive volume
The section defines RBC phase integrals by integrating background-relative optical phase over each cell’s area and introduces effective refractive volume when stained-cell refractive index is difficult to estimate.
- Phase delay: Relative optical phase delay depends on RBC thickness and refractive-index difference from the background, scaled by the illumination wavelength.Here, Δn(x,y) = n(x,y) − n0, with λ denoting the illumination wavelength in air.
- Phase integral: The phase integral integrates each RBC’s relative phase over its defined area, yielding units of rad·µm2.The integration is performed over the area Si of each RBC.
- Phase integral: Using the average refractive-index difference within a cell, the phase integral is related to the cell volume Vi through the wavelength-scaled refractive-index contrast.The passage identifies Vi as the volume of the ith cell.
- Effective refractive volume: Because the average refractive index of fixed, stained RBCs is difficult to determine, the section defines an effective refractive volume instead.This definition is intended for blood-smear samples containing fixed and stained RBCs.
- Effective refractive volume: The effective refractive volume has units of volume, such as femtoliters (fL).The passage explicitly gives femtoliters as an example unit.
Structural similarity (SSIM) index calculation
The SSIM index compares two images using their means, variances, and cross-covariance, with stabilization constants preventing division by small denominators. Its value ranges from 0 to 1, reaching 1 for identical images.
- Structural similarity (SSIM) index calculation: SSIM compares images I1 and I2 using their means, variances, and cross-covariance.µ1 and µ2 denote image averages; σ1^2 and σ2^2 denote variances; σ1,2 denotes cross-covariance.
- Structural similarity (SSIM) index calculation: Stabilization constants c1 and c2 prevent division by a small denominator.The constants are selected as c1 = K1^2L^2 and c2 = K2^2L^2, where L is the image dynamic range and K1, K2 are much smaller than 1.
- Structural similarity (SSIM) index calculation: The SSIM index ranges from 0 to 1, with 1 corresponding to identical images.This bounded range provides the interpretation of the index across image comparisons.
Evaluation of scattering strength of the samples
The evaluation tested the weakly scattering condition by analyzing regions reconstructed from eight-height multi-height phase recovery. The RMS scattered-to-reference wave ratio was 0.28 for breast tissue, 0.34 for Pap smear, and 0.37 for blood smear samples.
- Evaluation of scattering strength of the samples: Regions of interest for each sample were reconstructed using multi-height phase recovery based on 8 hologram heights.The evaluation was performed after the phase recovery step.
- Evaluation of scattering strength of the samples: The complex image was normalized by estimating the reference amplitude A from a background region without a sample.The normalized image is expressed as u/A = 1 + a(x,y)/A.
- Evaluation of scattering strength of the samples: R is the RMS modulus of the scattered wave |a(x,y)| divided by the reference wave modulus |A|.This ratio evaluates the validity of the weakly scattering condition.
- Evaluation of scattering strength of the samples: 0.28, 0.34, and 0.37 were obtained for breast tissue, Pap smear, and blood smear samples, respectively.These values were reported for the imaged samples.
Calculation of the sample-to-sensor distance
The method estimates relative separation between successive sensor heights for TIE and multi-height phase recovery, while the absolute sample-to-sensor distance enables final back-propagation to the sample plane.
- Calculation of the sample-to-sensor distance: Relative sensor-height separation is estimated for applying TIE and multi-height phase-recovery algorithms.The estimation uses an autofocusing algorithm based on the axial magnitude differential.
- Calculation of the sample-to-sensor distance: The absolute z2 sample-to-sensor distance is required to back-propagate the recovered complex wave onto the sample plane.This distance is identified as the absolute z2 distance and is referenced in Supplementary Fig. 4.