Source-linked AI summary
Model based learning for accelerated, limited-view 3D photoacoustic tomography
Andreas Hauptmann, Felix Lucka, Marta Betcke, Nam Huynh, Jonas Adler, Ben Cox, Paul Beard, Sebastien Ourselin, Simon Arridge
TL;DR
Restricted and limited-view photoacoustic measurements make high-resolution 3D reconstruction difficult because direct methods produce artefacts and iterative methods are computationally expensive. The paper embeds data-fit gradients in an iterative neural network, learns vessel-structure priors, and separates training from gradient computation. The method improves reconstruction quality, reduces iterations, and transfers to in-vivo measurements, but depends on training-data quality and update training.
Problem
Restricted spatial or temporal sampling in PAT creates limited-view reconstruction challenges, while existing iterative methods have high computational complexity and difficult-to-design priors.
Method
The paper learns an iterative convolutional reconstruction scheme that incorporates the physical data-fit gradient and vessel-structure priors, with layer-by-layer training excluding the photoacoustic operator.
Results
The learned scheme produces competitive reconstruction results with fewer iterations, reduces limited-view artefacts, and remains robust when applied to real measurement data.
Takeaways & Limitations
The approach enables detailed reconstructions from sub-sampled in-vivo photoacoustic data compared with TV minimisation of the same data.
Takeaways & Limitations
Performance is constrained by training-data quality and the possibility of performing update training.
Abstract
from arXiv · showhide
Recent advances in deep learning for tomographic reconstructions have shown great potential to create accurate and high quality images with a considerable speed-up. In this work we present a deep neural network that is specifically designed to provide high resolution 3D images from restricted photoacoustic measurements. The network is designed to represent an iterative scheme and incorporates gradient information of the data fit to compensate for limited view artefacts. Due to the high complexity of the photoacoustic forward operator, we separate training and computation of the gradient information. A suitable prior for the desired image structures is learned as part of the training. The resulting network is trained and tested on a set of segmented vessels from lung CT scans and then applied to in-vivo photoacoustic measurement data.
I. INTRODUCTION
Photoacoustic tomography can produce high-resolution 3D images, but restricted measurements create reconstruction challenges. This paper uses model-based learning to combine the physical forward model with learned image priors.
- Restricted spatial or temporal sampling can arise from geometric limitations such as limited view or from compressed sensing choices.
- Variational reconstructions can improve restricted-data results, but they require computationally expensive iterations and difficult-to-design structural priors.
- The study assumes vessel-rich targets and learns suitable prior constraints from segmented vessel structures.
- Deep neural networks offer potentially efficient reconstruction because their forward passes have low latency.
- The proposed model-based learning approach incorporates imaging forward and adjoint operators while learning multiscale regularisation parameters from data.
B. Compressed Sensing and Limited View PAT
Compressed sensing and limited-view acquisition reduce PAT measurements but make reconstruction dependent on an appropriate physical model and regularisation strategy. PAT maps initial pressure to measured ultrasound time series through a large linear forward operator.
- Compressed sensing seeks faster acquisition or reduced ionising-radiation dose, and its use in PAT requires an appropriate regularisation strategy.
- Limited-view PAT occurs when geometric restrictions permit access to the ultrasound field from only one side of the tissue.
- Photoacoustic signals arise when laser-induced pressure increases propagate through tissue as ultrasound waves.
- The sensing process is modeled by a linear operator acting on the pressure field at the computational-domain boundary over a finite time window.
- The forward map sends initial pressure x to measured pressure time series y, while reconstruction solves the corresponding inverse problem.
- For the paper’s examples, the operator has dimensions around 7M by 4.6M, making a fully dense single-precision representation require about 123TB of memory.
B. Direct methods for PAT Image Reconstruction
Direct PAT reconstruction is computationally efficient but performs poorly on limited-view and sub-sampled data. Iterative variational methods improve artefact suppression, at the cost of substantially greater computation and regularisation choices.
- Computing the adjoint solution A*y provides a direct reconstruction with the cost of one wave-equation solution.
- Direct reconstructions suffer severe circular artefacts and systematic contrast loss with depth, worsened by sub-sampling.
- Variational methods minimise a data-fit term plus a regulariser encoding prior knowledge about image structures.
- Choosing a suitable regulariser and balancing parameter is difficult, while convergence may require many iterations.
- Iterative NNLS and TV reconstructions reduce limited-view artefacts and contrast loss, although deeper regions converge more slowly and edges remain blurry.
- Twenty NNLS or TV iterations require 40 additional wave-equation solutions, compared with one for linear backprojection.
D. Model Based Learning
The paper replaces fixed vessel priors with learned update functions that combine data-fit gradients and image processing in an iterative reconstruction scheme. Layer-wise training excludes the costly photoacoustic operator while retaining model-based measurement information.
- Learned iterative reconstruction: Each DGD iteration receives measurement information through ∇d(y, Axk) and combines it with an image-processing step.The update is therefore driven by both the current iterate and data-fit gradient information at every step.
- Learned iterative reconstruction: Deep gradient descent (DGD) learns an update function for each iteration by combining the current image with the gradient of the data fit.The learned functions Gθk are CNNs with iteration-specific parameters and a shared architecture designed to mimic proximal gradient updates.
- Learned iterative reconstruction: DGD learns the effect of the regularising term from training data instead of imposing handcrafted priors on vessel structures.This addresses the limitation that fixed regularisers such as TV may not provide optimal priors for complex vessel networks.
- Training strategy: Layer-by-layer training makes it possible to exclude the photoacoustic operator from training because its complexity is O(N^4 log(N)) for an N × N × N volume.The paper contrasts this with CT and MRI operators, whose stated complexity is O(N^3 log(N)).
III. IMPLEMENTATION
The implementation trains separate CNN updates sequentially, using current iterates, data-fit gradients, and ground truths. The architecture expands and then contracts feature channels, adds the learned residual to the iterate, and enforces positivity.
- Network architecture: After combining the pipelines, the network reduces features to 16 channels and then one channel, adds the result to the current iterate, and projects it to nonnegative values.The final ReLU implements the positivity projection associated with the NNLS proximal step.
- Network architecture: The architecture increases channels during analysis and decreases them during reassembly to refine features and produce an output image.The paper states that the specific neural-network architecture is not its main contribution.
- Training procedure: The sequential strategy trains θ0, θ1, and later parameters independently using the preceding trained iterate as input.This splits the objective into kmax independent optimisation problems over disjoint parameter subsets.
- Training procedure: The alternative joint strategy is more flexible after kmax iterations, whereas sequential training decouples gradient computation from training and provides an upper bound on training error.The paper uses the sequential approach because 3D acoustic-gradient computation is prohibitively expensive during training.
- Training procedure: Each CNN is trained on the current iterate xk, the data-fit gradient ∇d(y, Axk), and the ground truth xtrue.The gradient is computed from the forward and adjoint operators before the corresponding update network is trained.
B. Evaluation of the deep gradient descent
In evaluation, each learned iteration applies its CNN to the current iterate and the gradient of the data fit, repeating the same model-based update without retraining.
- Evaluation procedure: The next iterate is computed as xk+1 = Gθk(∇d(y, Axk), xk), using the current image and data-fit gradient as inputs.The evaluated reconstruction repeats this update at every iteration.
IV. EXPERIMENTS
The experiments train DGD on simulated limited-view, sub-sampled measurements generated from segmented lung-vessel CT data, then evaluate reconstructions on held-out vessel volumes. The training uses five sequential iterates and separates expensive gradient precomputation from CNN optimisation.
- Training data: The supervised training set uses segmented human vessel structures because measured in-vivo data lack ground-truth targets.The vessel structures are extracted from lung CT scans and used to create simulated measurements.
- Training data: The data comprise about 1200 segmented vessel volumes from 50 whole-lung CT scans, with 1024 volumes selected as training ground truths.Volumes are scaled to 80 × 240 × 240 before limited-view, sub-sampled data are simulated.
- Training data: The simulations use measurements from one of two outer planes, 486 pressure samples, a sub-sampling factor of 4, and added normally distributed noise.The voxel size is dx = 84.75µm and the temporal sampling interval is dt = 16.6ns.
- Training and computation: Each CNN requires about 10 hours to precompute gradient information, while training each CNN takes about 1 day on the GPU.Five iterates are trained, for a total training time of 7 days.
- Reconstruction results: Figure 3 compares backprojection initialization, DGD with 5 iterations, TV reconstruction with 50 iterations, and the generating phantom using top-down maximum intensity projections.The displayed test image comes from segmented CT data and is not included in training.
B. Post-processing by Deep Learning
The study complements model-based reconstruction with a modified U-Net that post-processes an initial direct reconstruction to remove artefacts and noise.
- A modified 3D U-Net learns updates from initial direct reconstructions to remove artefacts and noise.The initial reconstruction is obtained as x0 = A∗y, and the network is trained on pairs of initial and true images.
- The method represents a post-processing approach rather than incorporating the forward and adjoint operators into each reconstruction update.It starts from a direct reconstruction and learns to improve it afterward.
- Figure 4 compares the DGD output on real human-palm data with a TV reconstruction from fully sampled data using top-down maximum intensity projections.The DGD was trained on images without added background.
- The U-Net training combines loss(x) and lossadd(x) and uses Adam for 75 epochs at a learning rate of 10^-4.Training one pair at a time took 3 days because of memory restrictions.
C. Application to in-vivo data
The trained networks are applied to sub-sampled in-vivo human-palm data, where synthetic background augmentation and transfer training are used to address mismatch with real measurements.
- In-vivo reconstruction uses the same volume dimensions, temporal sampling, and sub-sampling pattern as the simulated experiments.The measurement setup and procedure are described elsewhere in the paper.
- Clean-background training is insufficient for real data because low-absorption regions contain noise-like artefacts that the network does not effectively threshold.The network interprets structures in the data as important and enhances them equally.
- Background augmentation adds a clipped, scaled Gaussian random field with maximal value 0.1 to low-intensity regions of segmented vessel volumes.The clean volumes remain the training references, while measurements are generated from the modified volumes with noise.
- Measurement rescaling matches the standard deviation of real sub-sampled data to simulated targets before five-iteration DGD reconstruction.The resulting reconstruction is shown in Figure 5.
- Transfer training updates the DGD using 20 fully sampled finger, wrist, and palm measurements with weakly regularised TV references.It uses 10 additional epochs at a learning rate of 10^-5 and takes 90 minutes for the entire DGD.
- The corresponding U-Net update uses the same 10-epoch, 10^-5 schedule and takes 20 minutes.Its updated result is included in Figure 5.
V. DISCUSSION OF RESULTS
The DGD progressively reduces reconstruction error, with the largest improvement occurring after the first iteration, while achieving competitive results with substantially fewer iterations than classical methods and remaining robust on real measurements.
- The DGD produces competitive reconstruction results while considerably reducing the iterations needed.The formulation represents gradient descent as a CNN update in each iteration.
- 0.8492, 0.6171, and 0.4124 are the maximal difference values at initialization, after one iteration, and after five iterations, respectively.The values are reported for a test image from segmented CT data.
- The iterative network is robust when transferred to real measurement data.The paper identifies this robustness as important for inverse problems and image reconstruction.
- After one DGD iteration, the background is cleared and contrast is mostly restored, while five iterations further reduce limited-view artefacts.The remaining errors after one iteration include noisy patches and insufficient recovery near the boundary.
- The computation-time comparison includes communication overhead, so the reported timings indicate efficiency rather than provide an absolute comparison.Applying A or A∗ takes about 12 seconds on the Titan Xp GPU.
- Similar results after 2 DGD iterations and 50 TV iterations imply a prospective 20-fold speed-up, including initialization.The estimate uses two forward or adjoint applications per iterative update and includes the initial reconstruction.
B. Comparison to post-processing by Deep Learning
Compared with direct reconstruction followed by U-Net post-processing, DGD achieves lower simulated-data error and can recover information through iterative model-based updates, while transfer training is important for real measurements and background suppression.
- 0.4081 versus 0.6012 is the maximal error for DGD and U-Net, respectively, on the simulated test image.DGD has the smaller overall error, especially in the center of the domain.
- U-Net removes artefacts effectively but is limited by information contained in the initial reconstruction, leaving some smaller details unrecovered.DGD can improve the reconstruction through iterative updates using the measurement model.
- Adding background structures to DGD training improves real-data reconstructions and enables recovery of fine details after five iterations.A network trained only on clean-background images is insufficient because it enhances all structures in the data equally.
- Transfer training substantially improves reconstruction quality, allowing both learned methods to remove undesired background structures in vivo.The quantitative comparison uses a reference TV reconstruction from fully sampled limited-view data.
- A change in sampling pattern deteriorates err by 0.5% for DGD and 5% for U-Net on the simulated test data.The iterative network is reported as more robust to small perturbations than the one-step U-Net approach.
- With five iterations, DGD achieves an error similar to TV after 20 iterations on a tumor phantom with structural differences from training vessels.The authors associate this behavior with prior-like filters learned by the network.
VI. CONCLUSIONS
Learned iterative reconstruction incorporates the physical model through data-fit gradients and achieves better quality with substantial speed gains than classical iterative reconstruction. It also outperforms TV on sub-sampled in-vivo wrist data, while post-processing remains faster but limited by the initial reconstruction.
- Learned iterative reconstruction was superior to learned post-processing in image quality and robustness, while remaining considerably faster than iterative total variation.Post-processing was fastest, but at the cost of inferior image quality and flexibility.
- The method incorporates the physical model as the data-fit gradient and learns an iterative algorithm from convolutional neural networks.This design avoids handcrafted vessel priors by learning them from data.
- CNN post-processing produced fast, good initial reconstructions but remained limited by the quality of direct reconstructions when features were absent.It is promising when low latency matters more than best reconstruction quality.
- The proposed learning method is constrained by the quality of its training data and the possibility of performing update training.