Source-linked AI summary
Label-driven weakly-supervised learning for multimodal deformable image registration
Yipeng Hu, Marc Modat, Eli Gibson, Nooshin Ghavami, Ester Bonmati, Caroline M. Moore, Mark Emberton, J. Alison Noble, Dean C. Barratt, Tom Vercauteren
TL;DR
Multimodal MR–TRUS registration is difficult because robust intensity similarity measures are unavailable for intraoperative applications. The paper learns dense displacement fields from anatomical label correspondence during training and jointly models global and local deformation, achieving 4.2 mm median landmark TRE and 0.88 median prostate Dice.
Problem
Robust image-similarity measures for fast, robust multimodal registration are unavailable in the intraoperative MR–TRUS setting.
Method
A weakly supervised network learns a dense displacement field by aligning corresponding anatomical labels during training, then predicts global affine and local non-rigid deformation from image pairs alone.
Results
4.2 mm median TRE on landmark centroids and 0.88 median Dice on prostate glands were achieved on 111 paired MR–TRUS registrations.
Takeaways & Limitations
The framework enables fully automatic multimodal deformable registration using only the input image pair during inference and supports more than 4 3D registrations per second on the same GPU.
Abstract
from arXiv · showhide
Spatially aligning medical images from different modalities remains a challenging task, especially for intraoperative applications that require fast and robust algorithms. We propose a weakly-supervised, label-driven formulation for learning 3D voxel correspondence from higher-level label correspondence, thereby bypassing classical intensity-based image similarity measures. During training, a convolutional neural network is optimised by outputting a dense displacement field (DDF) that warps a set of available anatomical labels from the moving image to match their corresponding counterparts in the fixed image. These label pairs, including solid organs, ducts, vessels, point landmarks and other ad hoc structures, are only required at training time and can be spatially aligned by minimising a cross-entropy function of the warped moving label and the fixed label. During inference, the trained network takes a new image pair to predict an optimal DDF, resulting in a fully-automatic, label-free, real-time and deformable registration. For interventional applications where large global transformation prevails, we also propose a neural network architecture to jointly optimise the global- and local displacements. Experiment results are presented based on cross-validating registrations of 111 pairs of T2-weighted magnetic resonance images and 3D transrectal ultrasound images from prostate cancer patients with a total of over 4000 anatomical labels, yielding a median target registration error of 4.2 mm on landmark centroids and a median Dice of 0.88 on prostate glands.
1. INTRODUCTION
Multimodal registration is needed to combine diagnostically informative MR with real-time TRUS during prostate interventions. The challenge is the lack of a robust image-similarity measure for anatomically corresponding but modality-specific structures.
- TRUS guides many targeted prostate biopsies and focal therapies but generally provides poor contrast between healthy and cancerous regions.
- MR imaging is clinically important for detecting and localising prostate cancer but is usually acquired outside interventions.
- Multimodal registration aims to fuse preoperative tumour locations with real-time intraoperative imaging to improve patient care and disease risk stratification.
- A robust image-similarity measure has not been demonstrated for this application because corresponding anatomical structures can differ across TRUS and MR.
- The paper describes a network that predicts composite global and local deformations for registering MR and TRUS during prostate cancer interventions.
2. METHOD
The method learns dense voxel correspondence from higher-level anatomical label correspondence, while using a two-stage network to handle global and local interventional deformation. Labels supervise training but are not required as network inputs during inference.
- 2.1. Label-Driven Correspondence Learning: The framework trains a neural network to compute a dense displacement field from corresponding anatomical structures rather than scarce voxel-level ground-truth correspondences.
- 2.1. Label-Driven Correspondence Learning: Training minimises cross-entropy between warped moving labels and fixed labels, while the images serve as network inputs without directly contributing to the loss.
- 2.1. Label-Driven Correspondence Learning: One-sided label smoothing preserves one-hot foreground probabilities while producing spatially smoothed background probabilities and normalising each label map's total weight.
- 2.1. Label-Driven Correspondence Learning: The smoothed, normalised maps provide a larger capture range and more balanced weighting among labels from the same image than simple Gaussian smoothing.
- 2.2. Learning Interventional Deformation: The global-net predicts a 12-degree-of-freedom affine transformation, then the local-net predicts a bending-energy-regularised non-rigid DDF from the preliminarily aligned images.
- 2.2. Learning Interventional Deformation: The affine transformation and local DDF are composed into the final displacement field, with preliminary warping designed to improve learning and practical convergence speed.
- 2.2. Learning Interventional Deformation: The local-net uses four down-sampling and four up-sampling blocks with shortcut summations, while the global-net shares the down-sampling architecture and projects linearly to an affine matrix.
3. EXPERIMENT
The experiment evaluates the method on paired T2-weighted MR and TRUS prostate images acquired from patients in SmartTarget clinical trials, using standardised 3D voxel data and anatomical labels.
- 111 pairs of T2-weighted MR and TRUS images were acquired from 76 patients during SmartTarget clinical trials.
- Each case contained 35–112 parasagittal TRUS frames reconstructed into 3D Cartesian volumes covering most of the prostate gland.
- MR and TRUS images were resampled to 1.0 mm^3 isotropic voxels and normalised to zero-mean, unit-variance intensities.
- The dataset included 2351 pairs of anatomical labels for training and validation.
4. RESULTS
The proposed joint global-local registration network achieved accurate multimodal alignment and real-time throughput on prostate MR–TRUS cases. It outperformed either global-only or local-only alternative on landmark TRE.
- 4.2 mm median TRE on landmark centroids and 0.88 median Dice on prostate glands were achieved by the proposed network.The reported 5th–95th percentile ranges were 1.0–14.9 mm for TRE and 0.80–0.94 for Dice.
- More than 4 3D registrations per second can be performed on the same GPU.
- The proposed network's TRE was significantly better than either the global-net or local-net alone, with both paired-test p-values < 0.001.
- The global-net alone achieved median TRE 9.4 mm and median Dice 0.73, while the local-net alone achieved 8.5 mm and 0.86, respectively.
- Example slices compare registered moving MR images with fixed TRUS images and visually closest slices from original MR images.
5. CONCLUSION
The paper presents a flexible, fully automatic multimodal registration framework using input image pairs and a network that learns global and local interventional deformations. It reports rigorous validation on 111 pairs of labelled intraoperative prostate image data and identifies cross-centre and broader application generalisation as future work.
- The framework supports different image resamplers, deformation regularisers, and anatomical label types while requiring only the input image pair during registration.
- The proposed network architecture learns global and local deformations for multimodal registration during prostate cancer interventions.
- 111 pairs of labelled intraoperative prostate image data were used for rigorous validation.
- Future work will examine generalisation to data from different centres and a wider range of applications.