PaperlayerView exact revision on arXiv

Source-linked AI summary

Reconstructing undersampled photoacoustic microscopy images using deep learning

Anthony DiSpirito III, Daiwei Li, Tri Vu, Maomao Chen, Dong Zhang, Jianwen Luo, Roarke Horstmeyer, Junjie Yao

arXiv:2006.00251v1eess.IVcs.CVcs.LG

TL;DR

PAM faces a trade-off between imaging speed and spatial resolution, and larger scan steps can reduce image quality. This paper applies deep learning to reconstruct undersampled PAM images from fully sampled training examples; at about 2% of the original pixels, FD U-net still outperformed bicubic interpolation on the listed metrics, though close vessels could blur.

  • Problem

    PAM trades spatial resolution against imaging speed: larger scanning steps can speed acquisition but reduce resolution and image quality.

    Source: §I · p. 1

  • Method

    The paper uses deep learning as a software-only approach to map undersampled PAM images toward fully sampled counterparts.

    Source: §I · pp. 1–2

  • Results

    At about 2% of the original pixels, FD U-net outperformed bicubic interpolation on all listed metrics, but began to blur vessels that were close together or overlapping.

    Source: E. Downsampling Ratio: [10, 5] – 2%… · p. 7

  • Takeaways & Limitations

    Using 2% to 20% of the original pixels, FD U-net could potentially accelerate traditional PAM imaging with larger scanning steps.

    Source: §V · p. 7

    The method improves images without exceeding the baseline resolution achievable by the imaging system.

    Source: §V · p. 8

Abstract

from arXiv · show

One primary technical challenge in photoacoustic microscopy (PAM) is the necessary compromise between spatial resolution and imaging speed. In this study, we propose a novel application of deep learning principles to reconstruct undersampled PAM images and transcend the trade-off between spatial resolution and imaging speed. We compared various convolutional neural network (CNN) architectures, and selected a fully dense U-net (FD U-net) model that produced the best results. To mimic various undersampling conditions in practice, we artificially downsampled fully-sampled PAM images of mouse brain vasculature at different ratios. This allowed us to not only definitively establish the ground truth, but also train and test our deep learning model at various imaging conditions. Our results and numerical analysis have collectively demonstrated the robust performance of our model to reconstruct PAM images with as few as 2% of the original pixels, which may effectively shorten the imaging time without substantially sacrificing the image quality.

I. INTRODUCTION

PAM combines optical excitation with ultrasonic detection and uses raster scanning to form high-resolution 3D images. The paper proposes deep learning to improve undersampled PAM images, addressing a speed–resolution trade-off that existing approaches do not resolve for traditional systems.

I. INTRODUCTION

  • PAM’s raster scanning supports high-resolution 3D imaging, but large-field imaging is slow; larger scan steps can improve speed at the cost of spatial resolution and image quality.PAM scans the optical excitation beam and resultant ultrasound waves in an aqueous environment.Source: §I · p. 1
  • Earlier speed-up approaches use advanced scanners or compressive sensing, but often require hardware modifications unavailable to traditional PAM systems.Source: §I · p. 1
  • Deep learning has been used in PACT for artifact removal and contrast improvement, but its PAM application has been scarce because PAM directly forms images without inverse reconstruction.A prior PAM example used deep learning for motion correction; dictionary learning had also been reported for undersampled PAM reconstruction.Source: §I · p. 1
  • Deep learning is proposed to improve undersampled PAM images using as few as 2% of the original pixels, with a software-only approach to the resolution–speed trade-off.The model approximates the mapping from undersampled images to fully sampled counterparts and is trained using fully sampled PAM images as ground truth.Source: §I · pp. 1–2

II. METHODS

The framework models reconstruction as a learned mapping from an undersampled PAM image to its fully sampled counterpart. Training combines pixel-wise and Fourier-domain errors in a weighted loss.

A. Deep Learning Framework

  • The model learns G(θ, X) to approximate the mapping from an undersampled image X to its fully sampled counterpart Y through supervised loss minimization.Source: A. Deep Learning Framework · p. 2

B. Loss Function

  • The primary loss is mean absolute error (MAE) between the ground-truth and reconstructed images, representing pixel-wise error.Source: B. Loss Function · p. 2
  • Fourier loss (FMAE) compares the magnitudes of the 2D Fourier transforms of the ground-truth and reconstructed images, and is combined with MAE by a weighted sum.The weights are λ1 = 1.0 for MAE and λ2 = 0.01 for FMAE; the small FMAE weight limits its impact because it can destabilize early training.Source: B. Loss Function · p. 2

C. Fully Dense U-net

The FD U-net uses dense blocks in both contracting and expanding paths, with each layer drawing on feature maps from preceding layers. The implementation changes the activation and downsampling operations from the original architecture.

C. Fully Dense U-net

  • Dense blocks in both U-net paths concatenate each layer’s input with outputs from all previous layers, allowing layers to refine or diversify features and supporting deeper networks without vanishing gradients.Source: C. Fully Dense U-net · p. 2
  • The architecture figure labels compression depth as L1 through Li and gives the image size as N × N at each depth.Source: Figure 1 · p. 2
  • The implementation replaces ReLU with ELU and max pooling with stride-2 1 × 1 and 3 × 3 convolution blocks, enabling learned rather than rigid downsampling.Convolutional blocks use batch normalization after convolution and activation.Source: C. Fully Dense U-net · p. 2
  • The dense-block figure identifies X as input, Y as output, and shows the filter count doubling from f_in to f_out; with f_in = 32, f_out = 64 and k = 8.Source: Figure 2 · p. 2

III. EXPERIMENTAL PROCEDURES

The study uses in vivo mouse brain microvasculature images split into training, validation, and testing sets. Training crops and geometric or noise augmentation expand the training examples, while validation and testing data are not augmented.

A. Data Preparation

  • The dataset contains 292 in vivo mouse brain microvasculature images acquired at 532 nm, divided into 233 training, 30 validation, and 29 testing images.Images were thresholded, median-filtered, contrast-enhanced, and intensity-rescaled; the system’s lateral and axial resolutions were 5 μm and 15 μm.Source: A. Data Preparation · p. 3

B. Data Augmentation

  • Images were zero-padded and training used randomly located 128 × 128 crops, with 10 crops per iteration to reduce the chance of batches containing too few vessels.Source: B. Data Augmentation · p. 3
  • Training augmentation included rotations, shifts, shear, and Gaussian noise; validation and testing used no augmentation, apart from repeated validation crops for stabilizing the saving metric.The geometric transformations were capped at 20 degrees, 20% shifts, and a 0.2 shear factor.Source: B. Data Augmentation · p. 3

C. Downsampling Procedure

Fully sampled PAM images were synthetically downsampled at specified x- and y-axis ratios to mimic undersampling, and zero-fill was selected to restore missing pixels. Network training used Adam, while checkpoints were chosen using a metric that weighted SSIM more heavily than PSNR.

C. Downsampling Procedure

  • Fully sampled images were synthetically downsampled along the x- and y-axes to mimic different undersampling ratios.The ratio format is [Sx, Sy], where each value specifies the downsampling factor for its corresponding image axis.Source: C. Downsampling Procedure · p. 3
  • At a [7,3] downsampling ratio, zero-fill outperformed bicubic resizing by approximately 19% in PSNR and 18% in SSIM, so it was used in all experiments.The ratio denotes downsampling along the image x- and y-axes, respectively.Source: C. Downsampling Procedure · p. 3

D. Network Training

  • Training used Adam with mini-batches of 16 for 500 epochs, and checkpoints used a metric weighting SSIM more heavily than PSNR.The training passage says gains after about 350 epochs were moderate in some cases; 40.0 was used as an empirical PSNR limit in the saving metric.Source: D. Network Training · p. 3
  • The patchwork illustration shows later cleaning passes removing edge distortions left by the first pass.The caption identifies horizontal dark lines as the visible edge distortions and marks the second and third passes in red and blue.Source: Figure 3 · p. 4

E. Model Patchwork Algorithm

The patchwork algorithm handles images larger than the model’s 128 × 128-pixel input by dividing them into patches, processing them, and stitching outputs back into the original image dimensions. Two offset cleaning passes replace edge regions to address artifacts from the initial pass.

E. Model Patchwork Algorithm

  • Images larger than 128 × 128 pixels are zero-padded, divided into non-overlapping 128 × 128 patches, and processed for reconstruction.The patches are stitched back at their original locations to form the full-size image.Source: E. Model Patchwork Algorithm · p. 4
  • A second pass reprocesses offset patches and replaces a 20-pixel-deep buffer along first-pass patch edges to remove artifacts.The offset places patch centers on the edges of first-pass patches.Source: E. Model Patchwork Algorithm · p. 4
  • A third pass refines horizontal-vertical intersections, addressing edge artifacts remaining from the first pass or introduced in the second.Together, the three passes cover all possible edge-aberration regions.Source: E. Model Patchwork Algorithm · p. 4

A. Model Architecture comparison

The architecture comparison selected FD U-net for the main experiments because it performed strongly across metrics and showed greater robustness to high-intensity response aberrations. Plain U-net required more contrast rescaling and achieved lower SSIM under the shared enhancement setting.

A. Model Architecture comparison

  • FD U-net ranked best in PSNR and MSE and second in SSIM in the architecture comparison, and was selected for the main experiments.The table reports statistical metrics as mean ± SD for comparing model architectures.Source: A. Model Architecture comparison · p. 4, Table I · p. 4
  • FD U-net showed the highest robustness, while other architectures often needed greater contrast rescaling because of single-pixel high-intensity response aberrations.With the same contrast enhancement, U-net achieved an SSIM of ~85.6.Source: A. Model Architecture comparison · p. 4
  • All compared architectures were trained for 500 epochs on data downsampled 5:1 in the x-direction, using 20% of the original pixels.The passage states this downsampling could reduce effective imaging time by 80%.Source: A. Model Architecture comparison · p. 4
  • The comparison clipped pixel values below the 0.05 percentile and above the 99.95 percentile, then rescaled the remaining values to 0–1.These thresholds were chosen to bring plain U-net’s performance closer to bicubic interpolation; the relative performance of FD U-net, Res U-net, and ResICL U-net remained relatively consistent across tested thresholds.Source: A. Model Architecture comparison · pp. 4–5

B. FD U-net performance at different downsampling ratios

Across the tested downsampling ratios, FD U-net outperformed bicubic interpolation, with the advantage becoming more pronounced at 4.76% of the original pixels. The comparisons cover 29 test images and assess both quantitative metrics and reconstructed vessel appearance.

B. FD U-net performance at different downsampling ratios

C. Downsampling Ratio: [5, 1] – 20% effective pixels

D. Downsampling Ratio: [7, 3] – 4.76% effective pixels

  • At, using 4.76% of the original pixels, FD U-net outperformed bicubic interpolation in comparative metrics and reconstructed smoother vessels.Its performance at this ratio exceeded bicubic interpolation at; the authors describe the image quality as acceptable and expect a 95.24% reduction in acquisition time.Source: D. Downsampling Ratio: [7, 3] – 4.76%… · pp. 6–7, Figure 6 · p. 7

E. Downsampling Ratio: [10, 5] – 2% effective pixels

At, FD U-net outperformed bicubic interpolation across the listed metrics using about 2% of the original pixels, although nearby vessels began to blur. The discussion frames ratio choice as an application-specific balance between image quality and imaging speed.

E. Downsampling Ratio: [10, 5] – 2% effective pixels

  • At, using about 2% of the original pixels, FD U-net outperformed bicubic interpolation across all listed metrics, but blurred nearby or overlapping vessels.The authors considered bulk microvascular physiology acceptable at this ratio and projected a potential 50-fold imaging-speed improvement.Source: E. Downsampling Ratio: [10, 5] – 2%… · p. 7

V. DISCUSSION

  • The authors say performance deteriorates as sparsity increases, so the downsampling ratio can be tuned to balance desired resolution and imaging speed.They describe the approach as circumventing expensive hardware advances and suggest performance may improve as the dataset grows through retraining and refinement.Source: §V · p. 7
  • The method improves image quality through pixel-wise subsampling but does not exceed the baseline resolution achievable by the imaging system.The authors also identify tailored models for specific PAM systems and additional data as future development directions.Source: §V · p. 8
  • The authors identify acquiring large amounts of fully sampled in vivo PAM data as a prominent challenge for implementing deep learning in PAM.Their discussion describes sharing and growing a PAM vascular-image database for training and retraining.Source: §V · p. 8

VI. CONCLUSION

The conclusion reports that FD U-net was the best-performing tested architecture and outperformed bicubic interpolation at all representative downsampling ratios. The authors also made their mouse brain microvasculature dataset and source code freely available.

VI. CONCLUSION

  • FD U-net had the best performance among the tested architectures and outperformed bicubic interpolation at every representative downsampling ratio.The conclusion presents the model as addressing the speed–resolution trade-off in undersampled PAM.Source: §VI · p. 8
  • The authors made their mouse brain microvasculature dataset and model source code freely available to the research community.Source: §VI · p. 8

SUPPLEMENTAL FIGURES:

The supplemental figures identify U-net, Res U-net, and ResICL U-net architectures, with notation describing compression depth and image size at each depth.

Related papers

Loading 2006.00251v1…