Deep Learning in Photoacoustic Tomography: Current approaches and future directions

Andreas Hauptmann; Ben Cox

Source-linked AI summary

Deep Learning in Photoacoustic Tomography: Current approaches and future directions

Andreas Hauptmann, Ben Cox

arXiv:2009.07608v1Sep 16, 2020eess.IVcs.CVcs.LGeess.SPphysics.med-ph

TL;DR

PAT reconstruction must handle sparse or imperfect measurements, difficult forward models, and the need for rapid imaging, while accurate priors and ground-truth data remain challenging. The paper reviews deep-learning reconstruction methods, relates them to classical and Bayesian frameworks, and presents tutorial examples. The review identifies learned approaches as promising for practical and in vivo PAT, while emphasizing limits from training data, acquisition assumptions, and computational design.

Problem
PAT reconstruction must address incomplete or imperfect data, rapid imaging demands, difficult priors, and limited ground-truth data in clinical and in vivo settings.
Method
The paper reviews deep-learning methods for PAT reconstruction, connects them with classical and model-based approaches, and discusses hybrid, learned iterative, and alternative training strategies.
Results
Learned iterative reconstruction was demonstrated on realistic 3D limited-view volumes, while an approximate model achieved an 8× speed-up over a previous learned approach and 32× over iterative total-variation reconstruction in in vivo measurements.
Takeaways & Limitations
Deep learning is positioned as a useful complement to physical reconstruction models for sparse, fast, and clinically constrained PAT imaging.
Takeaways & Limitations
Fully learned approaches are practically limited by large dense parameterizations, fixed acquisition geometry and acoustic properties, and dependence on scarce ground-truth data.

Abstract

from arXiv · show

Biomedical photoacoustic tomography, which can provide high resolution 3D soft tissue images based on the optical absorption, has advanced to the stage at which translation from the laboratory to clinical settings is becoming possible. The need for rapid image formation and the practical restrictions on data acquisition that arise from the constraints of a clinical workflow are presenting new image reconstruction challenges. There are many classical approaches to image reconstruction, but ameliorating the effects of incomplete or imperfect data through the incorporation of accurate priors is challenging and leads to slow algorithms. Recently, the application of Deep Learning, or deep neural networks, to this problem has received a great deal of attention. This paper reviews the literature on learned image reconstruction, summarising the current trends, and explains how these new approaches fit within, and to some extent have arisen from, a framework that encompasses classical reconstruction methods. In particular, it shows how these new techniques can be understood from a Bayesian perspective, providing useful insights. The paper also provides a concise tutorial demonstration of three prototypical approaches to learned image reconstruction. The code and data sets for these demonstrations are available to researchers. It is anticipated that it is in in vivo applications - where data may be sparse, fast imaging critical and priors difficult to construct by hand - that Deep Learning will have the most impact. With this in mind, the paper concludes with some indications of possible future research directions.

1 Introduction

PAT offers high-resolution, quantitatively informative imaging, but practical acquisition and reconstruction constraints complicate clinical translation. This review surveys learned reconstruction methods, connects them to classical approaches, and focuses on PAT tomography applications of deep learning.

Motivation: PAT can provide high-resolution images based on optical absorption and quantitatively estimate molecular contrast such as blood oxygenation.These capabilities motivate growing clinical and pre-clinical interest.
Motivation: Practical PAT reconstruction involves acoustic and optical challenges, especially with incomplete data, nonlinear optical inversion, and computationally demanding model-based methods.The acoustic problem becomes difficult with heterogeneous sound speed or incomplete, bandlimited measurements, while optical inversion adds nonlinear complexity.
Scope and contribution: The paper reviews recent learned reconstruction work, relates it to classical analytical reconstruction, and provides a tutorial introduction with publicly available code and training/test data.The review covers relevant papers and preprints appearing through the end of June 2020.
Motivation: PAT spans data-intensive 3D imaging and high-frame-rate 2D imaging, creating challenges involving limited data, expensive forward operators, uncertain parameters, and scarce in vivo training data.The field is moving from proof-of-concept studies toward practical applications.
Scope and contribution: The review examines deep learning for PAT tomographic reconstruction, including networks combined with classical methods and pre- or post-processing intended to improve reconstruction quality.It excludes PAT microscopy and deep learning applied only after final reconstruction.

2 Forward and Inverse Problems in Photoacoustic Tomography

PAT combines optical excitation, acoustic propagation, and spectroscopic inference to recover tissue properties from measured acoustic signals. The inverse problems become difficult because measurements and forward-model inputs are often incomplete, filtered, noisy, or inaccurate.

Forward model: Photoacoustic imaging converts absorbed optical energy into acoustic pressure through rapid heating of tissue chromophores.The initial pressure distribution f is the acoustic quantity generated by optical absorption.
Forward model: The acoustic wave equation models pressure propagation, while optical fluence depends on absorption and scattering, making the optical relation nonlinear in absorption.The optical model links wavelength, absorption coefficient, fluence, and the Grüneisen parameter.
Measurements: PAT measurements are acquired on a surface using detector filtering and spatial sampling, with additive noise and geometries ranging from points and rings to linear arrays.Finite detector size restricts frequency and spatial-wavenumber response, while low signal-to-noise systems may require averaging.
Inverse problems: The acoustic inverse is well-posed only with sufficient coverage, sampling, bandwidth, and known sound speed; limited-view linear arrays can produce severe ill-posedness and artifacts.Ideal coverage requires detectors arranged so all rays through the object reach at least one detector.
Model mismatch: Forward models can be inaccurate because they use simplifying approximations or depend on poorly known sound-speed, illumination, and scattering parameters.Treating these parameters as unknowns makes the inversions less well-posed and usually requires additional data or constraints.

3 Classical Approaches to PAT Image Reconstruction

Classical PAT reconstruction includes analytical backprojection, Fourier, and time-reversal methods, as well as iterative variational optimization. These approaches offer physical flexibility and prior incorporation but must address incomplete data, ill-posedness, and modeling limitations.

Analytical methods: Classical PAT methods map acoustic measurements to the initial pressure distribution using reconstruction operators such as backprojection, adjoints, and filtered backprojection.Backprojection places data on spherical shells centered at detector locations, while filtering may be applied before or after projection.
Analytical methods: Linear-array PAT commonly uses delay-and-sum or beamforming, but short, bandlimited, plane-focused arrays make the reconstruction highly ill-posed.A typical array may contain only 128 detection elements.
Analytical methods: Fourier-based analytical solutions exploit plane-wave decompositions and can be computed efficiently with the Fast Fourier Transform.Variants used with linear arrays are often called Fourier beamforming and can appear inside learned iterative reconstructions.
Time reversal: Time reversal reconstructs the initial pressure by reproducing measured pressure data on the measurement surface in reverse time.For enclosing measurements in a homogeneous three-dimensional medium, the reconstructed field can recover f after sufficient propagation time.
Variational methods: Variational reconstruction minimizes data inconsistency plus a regularizer encoding prior information, with the regularization parameter balancing these terms.The framework supports gradient or proximal-gradient iterations and corresponds to a maximum a posteriori estimate when the regularizer represents a negative log prior.
Spectroscopic inversion: Direct spectroscopic inversion by linear unmixing ignores fluence dependence on absorption and therefore has questionable accuracy because of spectral coloring.Fluence approximations or complementary measurements can improve the approach, but accuracy must be demonstrated case by case.

4.1 What Role could Deep Learning Play?

Deep Learning can support PAT reconstruction at several pipeline stages, especially where data are incomplete or imperfect. Its speed may benefit real-time imaging, although practical reconstruction time also depends on network and forward-operator usage.

Potential roles: Deep Learning can correct corrupted measurements, reconstruct from incomplete data, approximate forward or inverse operators, and remove artifacts or noise.The review focuses on reconstruction-related uses and excludes segmentation and classification from its scope.
Speed: Small networks can process inputs quickly, which may support real-time or dynamic imaging requiring short reconstruction times.Evaluation speed depends on network size and input-data size.
Speed: Final reconstruction speed also depends on how the forward operator is used within the processing pipeline.
Practical adoption: Open-source tools and algorithms make Deep Learning methods easier to implement and accelerate method development.The paper cites TensorFlow and PyTorch as examples of accessible tools.

4.2 Brief Introduction to Deep Learning

Deep neural networks learn parameterized nonlinear mappings through layered operations. Fully connected layers connect all inputs to outputs, whereas convolutional layers preserve local spatial structure and are therefore natural for image-to-image mappings.

Deep neural networks: The review formulates PAT reconstruction as learning a parameterized mapping from measurement data g ∈Y to initial acoustic pressure f ∈Xf.The reconstruction operator may combine model-based components with learned components and is intended to improve speed or accuracy over classical approaches.
Deep neural networks: A deep neural network composes multiple layers, each combining a learnable affine map with a nonlinear function.The network is denoted by the nonlinear operator Λθ.
Fully connected layers: In a fully connected layer, every input is related to every output through a dense matrix, with biases and nonlinearities forming the layer operation.The network parameters are the layer matrices and bias vectors.
Fully connected layers: Fully connected imaging networks can reduce layer dimensions to extract low-dimensional features, while autoencoders encode and then decode representations of the input.Autoencoders use contracting and expanding paths and typically produce outputs with dimensions similar to the input.
Convolutional neural networks: CNNs apply learned filters across multichannel images, preserving spatial structure and producing output channels through convolution, biases, and nonlinearities.Each output channel uses a specific filter connecting it to the input channels.
Convolutional neural networks: CNN layers map between multichannel images at the same or similar resolution, making them natural for data-to-data and image-to-image mappings.Pooling layers can reduce image resolution to address memory constraints, create sparser representations, or widen receptive fields.
Convolutional neural networks: CNN translation equivariance means translating the input and then applying the function is equivalent to applying the function before translating the output.
Training: In supervised training, network parameters are learned by minimizing a distance between the network output and known ground truth for a specified task.The task may involve reconstructing or denoising an image, among other applications.

4.3 Architectures for Learned Reconstruction

Learned PAT reconstruction methods range from fully learned networks to model-based iterative schemes, differing in how often the physical forward operator is used. These architectures trade reconstruction flexibility, computational cost, data consistency, and sensitivity to acquisition geometry.

Architectural taxonomy: Learned reconstructions can be classified by whether the forward operator is used never, once, or multiple times.Four common strategies connect directly to classical reconstruction schemes.
The fully-learned approach: Fully learned reconstruction replaces the analytical inverse with one network, enabling low-latency inference but requiring large, geometry-specific learned mappings.A dense data-to-image layer can require approximately 500GB in a 64 × 64 × 64 voxel example, limiting practical use largely to two-dimensional problems.
The fully-learned approach: The AUTOMAP-like architecture combines three dense ELU layers with a small CNN and a final one-channel ReLU output that enforces non-negativity.The fully connected component maps vectorized measurements toward image space before convolutional post-processing.
Reconstruction and post-processing: Analytical reconstruction followed by learned post-processing uses a CNN to remove artifacts while reducing the memory burden and geometry inflexibility of fully connected layers.The network operates from image space to image space and can exploit multi-scale U-Net structure and residual corrections.
Reconstruction and post-processing: Residual U-Nets can handle strong local dependencies such as limited-view problems, but scarce training data can cause overfitting and the output is not guaranteed to remain optimally data-consistent.The three-scale architecture uses convolutional layers, downsampling, and concatenating skip connections.
Model-based learned iterative reconstruction: Learned iterative schemes interlace neural updates with repeated forward and adjoint-operator evaluations, improving data consistency and reconstruction quality at higher computational cost.The learned update can use the current reconstruction and data-consistency gradient, with smaller networks compensated by more informative inputs.

4.4 Generating Training Data

Training data for learned PAT reconstruction can be synthetic, experimentally measured, or combined through transfer training. The data-generation strategy determines the image prior represented by the network and helps address limited access to suitable measurements.

Training-data choices: The training set defines the image distribution, functioning as the Bayesian prior that the network learns for structures of interest.This is especially useful when informative biomedical priors, such as blood-vessel structure, are difficult to handcraft.
Synthetic training data: Synthetic training pairs are created by selecting ground-truth images, simulating corresponding noisy measurements, and pairing each measurement with its known image.The experiments used vessel structures derived from lung CT and retinal scans, with example datasets contrasting piece-wise or linear features against smoother features.
Experimental training data: Experimental training uses measured data together with reference reconstructions, ideally obtained from complete measurements and a high-quality reconstruction method.The resulting measurement–reference pairs can train a network for reconstruction.
Transfer training: Transfer training first pre-trains on abundant synthetic data and then fine-tunes on limited experimental measurements to adapt network parameters to experimental characteristics such as noise.Fine-tuning may use a reduced learning rate and update only selected layers.
Training-data choices: Self-supervised training may help transition toward experimental measurement data, but it had not yet been widely explored in this context.

4.5 Comparison of Learned Image Reconstruction Approaches

The experiments compare fully learned, post-processing, and learned iterative reconstruction in limited-view PAT under matched, mismatched, and mixed training priors. Model-based approaches are generally more robust, while the fully learned method is strongly affected by limited training data and prior mismatch.

Experimental design: The study evaluates three learned reconstruction approaches in a 2D limited-view geometry using synthetic forward-model data.The approaches are fully learned, post-processing, and learned iterative reconstruction.
Consistent priors: In the consistent-prior case, the fully learned approach overfits strongly, whereas model-based approaches reconstruct images closer to ground truth.Reconstructions remain less accurate near boundaries where limited-view artifacts are stronger.
Inconsistent priors: With mismatched priors, all methods deteriorate and reproduce the piece-wise prior learned from the training data.The fully learned method produces little resemblance to the ground truth, while model-based methods retain some features near the detector.
Combined priors: With combined training data, model-based methods reconstruct the target reasonably well, while the fully learned method improves but remains unsatisfactory.The larger training set reduces overfitting, although mixed priors still influence the reconstructions.
Quantitative comparison: Quantitative results show that model-based methods deteriorate under prior mismatch, while the fully learned method needs substantially more data to work well.PSNR improves for the fully learned approach with larger training data, whereas SSIM slightly deteriorates.

5 Deep Learning in PAT - Literature Review

The PAT literature increasingly uses deep learning for acoustic reconstruction, especially post-processing approaches that combine classical reconstructions with neural networks. Reported results favor architectures that incorporate useful representations, but robustness depends on consistent training and test conditions.

Scope: Most reviewed PAT deep-learning studies address acoustic reconstruction, with fewer studies targeting optical reconstruction.The review therefore focuses primarily on acoustic methods and treats optical reconstruction separately.
Post-processing approaches: Early post-processing methods apply a U-Net to an initial classical reconstruction, including sparse and limited-view settings.Variants use filtered backprojection or an iterative reconstruction as the network input.
Architecture and robustness: U-Net-based methods generally outperform simpler or alternative architectures, particularly in robustness.However, a classical ℓ1-regularized method outperformed network post-processing when undersampling artifacts varied through random system-matrix sampling.
In vivo applications: U-Net post-processing improved in vivo imaging over backprojection when measurements were undersampled, limited-view, or acquired with low-fluence sources.These results extend reported benefits beyond synthetic reconstruction examples.
Physics and uncertainty: Post-processing depends on training-data information rather than directly incorporating the acquisition physics, motivating nullspace projections and other data-consistency mechanisms.Nullspace projection adds only components that do not change the data-consistency term.
Temporal processing: Recurrent convolutional networks exploit temporal dependencies in noisy measurements and were reported to improve over single-frame post-processing.This direction is motivated by motion-sensitive, low-power LED-based acquisition requiring many averages.

5.2 Pre-Processing

Pre-processing networks improve PAT by transforming degraded measurements before classical inversion, including denoising, artifact removal, and data enhancement. Their usefulness is strongest in relatively simple measurement settings because reconstruction quality remains bounded by the processed data.

Approach: Pre-processing shifts learning from image inversion to measurement denoising and artifact removal before applying a classical reconstruction algorithm.The network operates on the data side, while the final image is obtained through classical inversion.
Task-specific processing: Task-specific pre-processing can separate true signals from noise and artifacts when the application requires locating a point-like source.This formulation uses object detection and classification for tracking-oriented applications.
Sampling and bandwidth enhancement: Awasthi et al. enhanced sparse, limited-bandwidth measurements by upsampling detectors, denoising, and increasing bandwidth before filtered backprojection.They compared a seven-layer CNN with a U-Net-based architecture.
Results: The U-Net generally performed better, but the simple CNN was highly competitive at low noise and both methods transferred successfully to in vivo measurements without retraining.The comparison demonstrates that architecture advantages depend on the noise regime.
Scope boundary: Pre-processing is considered viable mainly for fairly simple measurement scenarios because its reconstruction quality is limited by the quality of the processed data.The approach can be interpreted as learning a representation of the likelihood π(g|f).

5.3 Fully-Learned

Fully learned methods map PAT measurements directly into image space, often using large multi-scale convolutional networks to handle spatially nonlocal relationships. They can be competitive, but their lack of explicit physics and high data requirements constrain robustness and deployment.

Direct mapping: Fully learned reconstruction maps measurements from the spatio-temporal measurement space directly into the image space.A two-step variant first learns a fully connected measurement-to-image mapping and then applies a fixed-weight U-Net.
Convolutional approaches: Although direct CNN mappings lack clear theoretical justification for spatially invariant operations, multi-scale networks can enlarge receptive fields and encode broader locality.An asymmetric U-Net was feasible for limited-view reconstruction and competitive in mean reconstruction error, but had higher error variance than post-processing.
Task-specific outputs: Some applications simplify fully learned reconstruction to source localization using feature detection or image-to-coordinate networks.These methods target the task of finding a source location rather than reconstructing the full image.
Limitations: Fully learned methods do not explicitly include acquisition geometry or sound speed, so robustness to experimental changes requires suitable training data.Their generality may avoid explicitly determining instrument characteristics, but this shifts requirements into the training set.
Limitations: Fully learned approaches require substantially more training data than methods that involve the forward operator explicitly.This requirement is identified as a major limitation for transition to experimental measurements.

5.4 Learned Iterative Reconstructions

Learned iterative reconstructions embed known forward and adjoint models within trainable updates, improving robustness and reconstruction quality while imposing substantial computational costs. Greedy training and faster approximate models extend these methods toward feasible 3D and in vivo reconstruction.

Model-based learned updates: Learned iterative schemes use known forward and adjoint models within learned updates, and can be trained end-to-end from paired data.The training task optimizes network parameters using paired measurements and target images.
Computational constraints: Training and evaluating end-to-end learned iterative schemes is computationally burdensome because back-propagation requires storing and repeatedly evaluating forward and adjoint operators.These requirements have largely limited demonstrations to 2D imaging and can exceed single-GPU memory configurations.
Learned primal-dual reconstruction: LPD learns updates in both data and image spaces, with robustness varying across noise, structural, background, and contrast changes.The method is based on the primal-dual hybrid gradient algorithm and uses separate networks for data and image spaces.
Greedy training for 3D: Greedy training makes 3D learned iterative reconstruction feasible by optimizing each unrolled iterate separately rather than jointly optimizing the full end-to-end network.The approach was demonstrated on 240 × 240 × 80 volumes in limited-view geometry, with improved reconstructions over U-Net post-processing and total-variation reconstruction.
Faster reconstruction: An approximate fast forward model produced an 8× speed-up over the previous learned approach and 32× over iterative total-variation reconstruction, with results shown in vivo.The approximate gradient introduced artifacts that the learned networks reduced through their updates.
Overall assessment: Learned iterative approaches can improve reconstruction quality, but repeated application of forward and adjoint models remains a major limitation on reconstruction speed.A recurrent inference-machine extension improved 2D in silico results by 2dB in PSNR.

5.5 Hybrid Approaches

Hybrid approaches combine learned components with analytical reconstruction or explicit model-based data consistency. These designs seek to retain model dependence and convergence properties, but iterative forward and adjoint evaluations remain costly.

Augmented end-to-end approaches: Hybrid methods augment fully learned reconstructions with reconstructed images to address their missing model dependence.The reconstructed image may enter the network directly or through a separate processing branch.
Learned regularisation: Learned regularisers can be inserted into variational formulations, producing an explicit objective that is minimized iteratively.This approach is represented by the NETT framework and has been applied to PAT.
Trade-offs: Model-based data consistency and convergence guarantees under certain conditions are strengths of learned variational approaches, but explicit forward and adjoint models keep minimization time-consuming.The computational burden remains similar to that of learned gradient schemes.

5.6 Optical Inversions

Deep learning for PAT optical inversions estimates absorption, chromophore concentrations, or blood oxygenation after acoustic inversion, using spatial, spectral, or hybrid image information. Results are promising in simulation, but generalization to in vivo settings remains constrained by fluence effects, training-data realism, and uncertainty estimation.

Scope of optical inversions: Most learned optical-inversion studies assume acoustic inversion is already solved and target absorption coefficients, chromophore concentrations, or blood oxygen saturation.The principal inverse problems are F^-1(f) and (FL)^-1(f).
U-Net-based approaches: ResU-Net and DR2U-Net estimate blood oxygen saturation and contrast agents from 2D multiwavelength PAT images, outperforming linear unmixing in simple in silico tests.Their residual blocks replace or extend standard U-Net convolutional stages.
U-Net-based approaches: A U-Net recovered 2D optical absorption from single-wavelength PAT data, but geometric simplicity and train-test similarity leave its general applicability unclear.The study used simulated examples and experimentally measured data, with rotation-based augmentation of the measured training set.
U-Net-based approaches: Learned optical inversion has been demonstrated through segmentation-linked O-Nets, fluence-estimation pipelines, and direct absorption estimation with varied U-Net architectures.These approaches exploit spatial information about fluence present in PA images.
U-Net-based approaches: EDA-Net performed slightly better than ResU-Net and U-Net++ and much better than linear unmixing on realistic simulated data based on a 3D digital breast phantom.The setup used a 3D light model and acoustically processed 2D slices to mimic limited-view linear-array measurements.
Uncertainty estimation: A meta-network estimated uncertainty in absorption maps, and its uncertainty correlated well with actual image error in an in silico study.The authors motivate confidence estimation by the difficulty of translating quantitative PAT algorithms to in vivo cases.
Pixelwise spectral approaches: Pixelwise autoencoder spectral unmixing may avoid ground-truth training data, but its ability to account for fluence effects remains unclear.The limitation is identified in a preliminary simulation study.
Translation to in vivo imaging: Classical quantitative PAT methods have been difficult to translate convincingly to in vivo measurements because auxiliary input parameters are hard to obtain accurately under experimental conditions.Deep learning is presented as a possible way to address this challenge by learning from data, within the scope of the reviewed work.

6 Conclusions and Future Directions

Learned PAT reconstruction is moving beyond proof-of-concept methods toward approaches that combine neural networks, physical models, and new training regimes. Key open issues include data consistency, limited in vivo training data, computational cost, model uncertainty, and application-specific trade-offs.

Data consistency: Data consistency remains important because realistic-looking reconstructions may lack an assessable connection to the measured data.Null-space methods and learned iterative reconstructions provide possible ways to incorporate data consistency.
Training data: In vivo applications are constrained by the difficulty of matching simulated training-data distributions to experimental targets.Fully supervised methods are primarily limited by the availability of ground-truth data, motivating semi-supervised and cycle-consistent training regimes.
Physics-informed learning: Physics-informed neural networks incorporate partial-differential-equation models into the loss function so networks learn less of the physical operator directly from data.This offers a training direction when ground-truth data are scarce.
3D reconstruction: Three-dimensional PAT remains computationally demanding because learned models are time-consuming and require substantial memory for data and intermediate states.Proposed scaling strategies include invertible networks and multiresolution forward-model computations.
Model augmentation and correction: Model-based learned schemes can be slow and depend on uncertain parameters, especially sound speed and optical scattering.Potential alternatives include embedding model discretizations into network architectures or learning corrections to faster approximate models.
Trade-offs and choices: The best network placement in the reconstruction pipeline depends on application-specific trade-offs rather than a single universally preferred design.Designers must weigh the available reconstruction options against the nature of the imaging application.