Source-linked AI summary
Deep learning for biomedical photoacoustic imaging: A review
Janek Gröhl, Melanie Schellenberg, Kris Dreher, Lena Maier-Hein
TL;DR
PAI offers deep imaging of optical tissue properties, but extracting clinically relevant parameters requires difficult inverse problems and reliable validation data. This review synthesizes deep-learning applications across major PAI challenges and evaluates progress toward clinical applicability. It finds advantages including rapid inference, while highlighting limited validation on in vivo and human data and weak generalization across data distributions.
Problem
Extracting optical tissue parameters from PAI measurements requires difficult acoustic and optical inverse problems, while reliable annotated or ground-truth data remain limited.
Method
The review analyzes deep-learning applications in PAI across acoustic reconstruction, image post-processing, optical inverse problems, and semantic image annotation.
Results
Deep learning enables millisecond-scale evaluation of entire high-resolution 2D and 3D images, but only approximately 20% of reviewed papers validated methods on in vivo data.
Takeaways & Limitations
Deep learning has potential to facilitate PAI clinical translation, with additional information improving generalizability compared with direct reconstruction alone.
Takeaways & Limitations
High-quality clinician-annotated reference data are rare, and cross-modality and inter-institutional performance has not yet been examined to the authors’ knowledge.
Abstract
from arXiv · showhide
Photoacoustic imaging (PAI) is a promising emerging imaging modality that enables spatially resolved imaging of optical tissue properties up to several centimeters deep in tissue, creating the potential for numerous exciting clinical applications. However, extraction of relevant tissue parameters from the raw data requires the solving of inverse image reconstruction problems, which have proven extremely difficult to solve. The application of deep learning methods has recently exploded in popularity, leading to impressive successes in the context of medical imaging and also finding first use in the field of PAI. Deep learning methods possess unique advantages that can facilitate the clinical translation of PAI, such as extremely fast computation times and the fact that they can be adapted to any given problem. In this review, we examine the current state of the art regarding deep learning in PAI and identify potential directions of research that will help to reach the goal of clinical applicability
1 Introduction
PAI uses acoustic signals to image optical tissue properties several centimeters deep, but clinically relevant parameter extraction requires difficult acoustic and optical inverse problems. Deep learning is being applied to these challenges because it can support fast inference and clinical image interpretation.
- Photoacoustic imaging: PAI measures optical tissue properties by detecting acoustic signals generated from light-absorbing chromophores several centimeters deep in tissue.Acoustic scattering is orders of magnitude smaller than optical scattering, enabling greater penetration than other optical imaging modalities.
- Clinical potential: Clinical applications include cancer research, inflammatory-joint imaging, Crohn’s disease staging, multimodal ultrasound imaging, brain imaging, and needle tracking.These applications use endogenous chromophore contrast and changes in tissue hemodynamics or blood oxygenation.
- Open problems: The acoustic inverse problem reconstructs an image from recorded pressure time series by estimating the initial pressure distribution.Common approaches include universal back-projection, delay-and-sum, time reversal, and iterative reconstruction schemes.
- Open problems: The optical inverse problem estimates tissue optical properties, especially absorption, after reconstructing the initial pressure distribution.It is ill-posed and has not yet been successfully applied to in vivo data.
- Open problems: Image post-processing targets artifacts and noise because ill-posed reconstruction settings can reduce PA image quality below its theoretical contrast and resolution.Deep learning also enters PAI through medical-imaging tasks such as segmentation and classification.
- Review scope: The review summarizes deep learning development in PAI since 2017 and evaluates progress across defined task categories.It covers literature-search methods, topical foci, data acquisition, simulation frameworks, network architectures, and four principal categories.
2 Methods of literature research
The review used a systematic search across multiple scientific databases for literature published from January 2017 through September 2020. Candidate papers were supplemented through additional sources and filtered by deduplication and abstract screening.
- Review design: The review divided deep-learning applications in PAI into four categories: acoustic inverse problem, image post-processing, optical inverse problem, and semantic image annotation.Papers were assigned to the most suitable category during the systematic review.
- Search strategy: The search covered January 2017 to September 2020 and used Google Scholar, IEEE Xplore, PubMed, Microsoft Academic Search Engine, and arXiv.The supplied passage identifies the review period and named search engines.
- Screening process: Potential candidates were identified by title and search-engine overviews, then supplemented with papers from conferences, key-author websites, and PA-device-vendor websites.Non-relevant papers were removed through duplicate removal and abstract scanning.
- Review yield: 66 relevant papers published since 2017 remained after excluding duplicates and related out-of-scope work.This is the final count reported for the search algorithm.
3 General findings
The reviewed PAI literature focuses on reconstruction, post-processing, optical-property estimation, and semantic annotation, with U-Net-based CNNs especially prevalent. Training relies heavily on synthetic data, while validation across differing or in vivo data remains limited.
- Topical foci: Deep-learning research in PAI accelerated over the three years reviewed and covered four main topical areas.These areas include acoustic reconstruction, image-quality improvement, chromophore-concentration estimation, and tissue or functional-parameter annotation.
- Training data: Training-set sizes ranged from 32 to 296.300 samples, with a median of 2.400.The review identifies insufficient reliable experimental training data as a core bottleneck.
- Training data: Nearly 75% of papers relied exclusively on simulated data for neural-network training.Synthetic data were generated through random, model-based, reference-based, or combined approaches, including transfer learning.
- Validation: About 50% of papers tested methods on multiple data sets with distributions significantly different from training data, and about 25% tested on in vivo data.Nearly all papers using multiple data sets tested experimental data.
- Simulation frameworks: The review identified five simulation frameworks used to create synthetic PAI data, including k-Wave, mcxyz, MCX, NIRFAST, and Toast++.These frameworks model acoustic fields, photon transport, near-infrared propagation, or light propagation in scattering media.
- Network architectures: U-Net architectures appeared in over 70% of reviewed papers, while residual and fully connected networks each appeared in less than 10%.Slightly modified U-Nets were reported to yield the best performance for target applications.
4 Acoustic Inverse Problem
The acoustic inverse problem reconstructs the initial pressure distribution from measured time-series data. Deep learning supports model-enhanced and direct reconstruction, point-source localization, and speed-of-sound estimation, but generalization and validation remain limited.
- Main approaches: 40% of papers enhanced model-based reconstruction, while 35% performed direct image reconstruction from time-series pressure data.
- Speed of sound estimation: 10% of papers estimated the medium’s speed of sound, while automatic integration of this estimate was identified as a direction that can enhance image quality.
- Limitations: All 20 acoustic-inverse-problem papers used simulated training data, and only approximately 40% tested methods on experimental data.
- Direct image reconstruction: Direct reconstruction either replaces model-based methods or incorporates reference reconstructions and hand-crafted features as additional information.
- Point source localization: Point-source localization estimates source coordinates or a probability heat map from time-series data, but its clinical scope is questionable for heterogeneous in vivo chromophore distributions.
5 Image Post-Processing
Image post-processing addresses artifacts and quality loss after reconstruction. Deep learning is applied to recover information from sparse or limited-bandwidth data, remove artifacts, improve signal-to-noise ratio, and enhance image quality.
- Overview: Post-processing targets artifact elimination and image-quality enhancement after reconstruction.
- Artifact removal: Sparse-data reconstruction can accelerate imaging but introduces under-sampling artifacts, many of which deep learning methods have recovered.
- Artifact removal: Deep learning methods address reflection and motion artifacts by distinguishing artifacts from true signals and eliminating motion-induced image artifacts.
- Limitations: Artifact-removal models are comparatively difficult to train on in vivo data because artifact sources vary and depend on the specific processing pipeline.
- Image-quality enhancement: Full-bandwidth recovery relies on simulated pairs of full-bandwidth and limited-bandwidth data because experimental systems are always band-limited.
- Image-quality enhancement: CNNs improve low-energy image quality either by enhancing individual reconstructions or by fusing multiple reconstructions into a higher-quality image.
- Key insight: Common computer-vision tasks can be translated to PA images relatively easily because such algorithms are generally straightforward to train and validate.
6 Optical Inverse Problem
The optical inverse problem estimates optical tissue properties, especially absorption, from the reconstructed initial pressure distribution. Deep learning and other data-driven approaches reduce reliance on explicit model assumptions, but in vivo application remains unresolved.
- Problem definition: The optical inverse problem estimates optical tissue properties, primarily the absorption coefficient, from the initial pressure distribution.
- Approaches: Reviewed work includes iterative reconstruction, classical machine learning, and deep learning approaches for optical-property estimation.
- Problem definition: Identified approaches aim to estimate optical absorption coefficients and subsequently absolute chromophore concentrations.
- Key insight: Model-based methods require explicit assumptions that typically do not hold in complex scenarios, whereas data-driven methods encode many assumptions implicitly in the data distribution.
- Limitations: Obtaining ground-truth optical tissue properties in vivo is considered impossible and remains exceptionally involved and error-prone even in vitro.
- Limitations: No reviewed optical-inverse-problem approach had yet been applied to in vivo data, leaving this area among the most challenging in PAI.
7 Semantic Image Annotation
Multispectral PAI supports estimation of functional tissue properties and semantic tissue annotations, including oxygenation, classification, and segmentation. Deep learning approaches show feasibility, but clinical validation remains limited by data and domain-gap challenges.
- Functional property estimation: Multispectral signal changes enable estimation of functional tissue properties, especially blood oxygenation, and classification or segmentation of tissue types.These tasks use information across multiple wavelengths rather than single-wavelength measurements.
- Functional property estimation: sO2 estimation separates oxyhemoglobin HbO2 and deoxyhemoglobin Hb contributions using measurements at least two wavelengths.Wavelengths around the isosbestic point near 800 nm are commonly selected.
- Functional property estimation: Neural networks estimated oxygenation from single-pixel spectra and produced plausible results on experimental in vitro and in vivo data.Later work incorporated fluence eigenspectra and larger tissue patches as regularization sources.
- Functional property estimation: CNNs estimated oxygenation from entire 2D images and demonstrated feasibility on multispectral 3D images, but successful in vitro or in vivo application has not been shown.The reported limitation is attributed most probably to the domain gap between simulated and experimental PA images.
- Tissue classification and segmentation: Semantic annotation algorithms classify and segment multispectral PA images into tissue types and estimate clinically relevant parameters such as blood oxygenation.Applications include skin-structure differentiation, cancer classification, disease detection, vessel segmentation, and needle-tip segmentation.
- Tissue classification and segmentation: Semantic image annotation can enable intuitive and fast interpretation, but scarce expert-annotated reference data and unexamined cross-device or inter-institutional performance hinder in vivo validation.Image-quality dependence on acoustic and optical inverse-problem solutions also complicates manual annotation and clinical integration.
8 Discussion
Deep learning has broad potential for PAI clinical translation through fast, adaptable, and potentially uncertainty-aware analysis. However, generalizability, validation, comparability, and workflow integration remain constrained by simulated-data reliance and limited clinical evidence.
- Clinical translation: Clinical translation of deep learning in PAI remains in its infancy despite applications to classical and PA-specific image-processing tasks.The review identifies persistent limitations that must be addressed before clinical use.
- Generalizability: High-quality experimental training data are difficult to obtain because PAI is young, annotations are costly, and reliable ground truth is limited.These constraints arise from signal variability, few clinically approved devices, and elaborate reference measurements or manual annotation.
- Generalizability: Approximately 75% of models were trained on simulated PA data, yet purely simulation-trained methods have shown poor performance on experimental data.Systematic differences between computational forward-model outputs and experimental PA images create a substantial domain gap.
- Reliability: Uncertainty estimation can provide confidence intervals or posterior distributions, but its potential in deep learning-based PAI remains largely untapped.Out-of-distribution detection is also needed because uncertainty metrics may not indicate estimate quality for OOD samples.
- Validation: Only approximately 20% of reviewed papers validated methods on in vivo data, and even fewer used human measurements.No prospective deep learning studies in PAI had been conducted, motivating substantially more preclinical work.
- Standardization: Reported results are difficult to compare because standardized metrics, common datasets, and openly accessible test data are lacking.The review notes that ongoing PA standardization efforts aim to address image-quality assessment, multicentric phantom studies, and data exchange.
- Computational efficiency: Deep learning inference can be exceptionally fast on GPUs, potentially enabling real-time application of complex PAI algorithms in time-critical settings.This combines PAI’s real-time imaging capability with massive GPU parallelization.
- Clinical workflow integration: Clinical workflow integration depends on demonstrable clinical impact and methods that are intuitive and impose no significant time burden.The review frames accurate, reliable, uncertainty-aware, and explainable biomarker estimation as a potential route toward integration.
Author contributions statement
The author contributions statement assigns conceptualization, investigation, and writing roles across the authors, with methodology led by J.G.
- Author contributions: J.G., M.S., K.D., and L.M.-H. contributed to conceptualization, while J.G., M.S., and K.D. conducted the investigation.J.G. led methodology and wrote the original draft; review and editing were shared by J.G., M.S., K.D., and L.M.-H.