Source-linked AI summary
Label-driven weakly-supervised learning for multimodal deformable image registration
Yipeng Hu, Marc Modat, Eli Gibson, Nooshin Ghavami, Ester Bonmati, Caroline M. Moore, Mark Emberton, J. Alison Noble, Dean C. Barratt, Tom Vercauteren
TL;DR
Multimodal registration must align preoperative diagnostic information with real-time, often low-quality intraoperative imaging, where robust similarity measures have not been demonstrated. This paper learns dense deformations from training-time anatomical label correspondences and reports 4.2 mm median landmark-centroid TRE and 0.88 median prostate-gland Dice on prostate MR–TRUS pairs.
Problem
Multimodal registration must fuse preoperative diagnostic information with real-time, often low-quality intraoperative imaging, yet robust similarity measures are lacking and voxel-level correspondence labels are generally unavailable.
Source: §1 · p. 1, §2.1 · p. 2Method
A network learns voxel correspondence from anatomical label pairs via warped-label cross-entropy, then registers image pairs without labels at inference, with affine global and regularized local displacements.
Source: §2.1 · p. 2, §2.2 · p. 3Results
Takeaways & Limitations
The study presents a flexible multimodal registration approach, validated on 111 pairs of labelled intraoperative prostate images.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Spatially aligning medical images from different modalities remains a challenging task, especially for intraoperative applications that require fast and robust algorithms. We propose a weakly-supervised, label-driven formulation for learning 3D voxel correspondence from higher-level label correspondence, thereby bypassing classical intensity-based image similarity measures. During training, a convolutional neural network is optimised by outputting a dense displacement field (DDF) that warps a set of available anatomical labels from the moving image to match their corresponding counterparts in the fixed image. These label pairs, including solid organs, ducts, vessels, point landmarks and other ad hoc structures, are only required at training time and can be spatially aligned by minimising a cross-entropy function of the warped moving label and the fixed label. During inference, the trained network takes a new image pair to predict an optimal DDF, resulting in a fully-automatic, label-free, real-time and deformable registration. For interventional applications where large global transformation prevails, we also propose a neural network architecture to jointly optimise the global- and local displacements. Experiment results are presented based on cross-validating registrations of 111 pairs of T2-weighted magnetic resonance images and 3D transrectal ultrasound images from prostate cancer patients with a total of over 4000 anatomical labels, yielding a median target registration error of 4.2 mm on landmark centroids and a median Dice of 0.88 on prostate glands.
1. INTRODUCTION
The introduction frames MR–TRUS registration as a way to fuse preoperative cancer-localisation information with low-quality intraoperative ultrasound, while noting the lack of robust image similarity measures. It proposes learning dense voxel correspondences from anatomical labels and predicting global and local deformation.
1. INTRODUCTION
- MR provides clinically important cancer-localisation information, while intraoperative TRUS guides most targeted biopsies and focal therapies but has poor contrast between healthy and cancerous regions.Multimodal registration aims to fuse preoperative planning information with real-time intraoperative imaging.Source: §1 · p. 1
- The proposed neural network learns dense voxel correspondences from identifiable anatomical labels and predicts composite global and local deformation for MR–TRUS registration.The label-driven approach is motivated by applications where no robust image similarity measure is known.Source: §1 · pp. 1–2
- Intensity-based methods can falsely align statistically correlated but anatomically different structures, while manual cross-modal landmarks or segmentations are time-consuming and often infeasible intraoperatively.Prior learning-based methods still rely on image similarity measures or potentially unrealistic ground-truth transformations from simulations, existing registration methods, or manual rigid alignment.Source: §1 · p. 1
2. METHOD
The method learns deformable voxel correspondence from paired anatomical labels during training, then registers new image pairs using images alone. For interventional registration, it combines global affine and local non-rigid deformation prediction.
2.1. Label-Driven Correspondence Learning
- Training labels act as weak voxel-correspondence supervision: a DDF warps moving labels toward fixed labels under cross-entropy, while inference uses images alone without an explicit image-similarity loss.Different label pairs are sampled across iterations; one pair per image pair is used at each iteration, yielding an unbiased minibatch gradient estimate. Higher-confidence labels are sampled more frequently to reduce the effect of labeling uncertainty.Source: §2.1 · p. 2
- One-sided smoothing preserves foreground one-hot probabilities while smoothing background probabilities and normalising each label map to the same total weight.The approach is described as providing a larger capture range and more balanced weighting between labels from the same image than simple Gaussian smoothing.Source: §2.1 · pp. 2–3
2.2. Learning Interventional Deformation
- Direct DDF prediction can produce implausible deformations and large TREs outside the gland when labels provide weak support, motivating an architectural constraint for global transformation learning.The problem is attributed to diminishing or perturbing gradients in under-supported regions and sensitivity to the balance between cross-entropy and regularisation.Source: §2.2 · p. 3
- A global-net predicts a 12-degree-of-freedom affine transform, warps the resized moving image, and feeds it with the fixed image to a local-net whose regularised DDF composes with the affine transform.The local DDF is regularised by weighted bending energy; the preliminary warp is designed to improve local-deformation learning and is reported to converge faster in practice.Source: §2.2 · p. 3
- The local-net uses four down-sampling and four up-sampling blocks with residual shortcuts, while the global-net shares its down-sampling architecture and projects features to affine parameters.The local-net has 32 initial channels and the global-net has 4; output initialization is centered near zero for local displacement and near identity with zero translation for the affine transform.Source: §2.2 · p. 3
3. EXPERIMENT
The experiment evaluates label-driven MR–TRUS registration on a patient-level cross-validation cohort with expert-provided anatomical correspondences. It reports landmark-centroid TRE and prostate-gland Dice on held-out patients.
3. EXPERIMENT
- 111 T2-weighted MR–TRUS image pairs from 76 patients formed the dataset, with images resampled to 1.0 mm3 isotropic voxels.Each TRUS volume was reconstructed from 35–112 parasagittal frames acquired across most of the prostate gland.Source: §3 · p. 4
- 2351 corresponding landmark pairs and prostate-gland segmentations were labelled; 330 landmark pairs were confirmed by a second observer as high confidence.Landmarks included anatomical structures and patient-specific points; high-confidence landmarks were sampled twice as frequently during training.Source: §3 · p. 4
- Networks were trained for 12 hours with Adam at a learning rate of 10^-5, using minibatches of 10 and a 24GB NVIDIA Quadro P6000.The global-net was trained first for 1000 iterations; bending-energy weight and weight decay were 10^-2 and 10^-6, respectively.Source: §3 · p. 4
- Separate global-net and local-net comparison models minimized the same loss, with applicable hyperparameters held equal.For local-net-only training, the affineresampled moving image was linearly resized before concatenation with the fixed image.Source: §3 · p. 4
- Ten-fold patient-level cross-validation held out test data from 7–8 patients per fold and evaluated landmark-centroid TRE and prostate-gland Dice on test data.The two independently calculated measures relate to avoiding surrounding structures and locating regions of interest in registration-enabled guidance.Source: §3 · p. 4
4. RESULTS
The jointly optimized registration achieved a median landmark-centroid TRE of 4.2 mm and prostate-gland Dice of 0.88, with more than four 3D registrations per second on the same GPU.
4. RESULTS
- 4.2 (1.0 - 14.9) mm median TRE on landmark centroids and 0.88 (0.80 - 0.94) median Dice on prostate glands were achieved.The reported percentiles are 5th–95th.Source: §4 · p. 4
- The proposed network significantly outperformed the global-net (9.4 (3.0 - 20.4) mm TRE; 0.73 (0.46 - 0.87) Dice) and local-net (8.5 (1.5 - 22.7) mm TRE; 0.86 (0.46 - 0.91) Dice) on TRE (both p < 0.001).The local-net's competitive Dice may reflect over-fitting around prostate glands and was more susceptible to visually non-realistic deformations.Source: §4 · p. 4
- More than 4 3D registrations per second can be performed on the same GPU.Figure 5 shows example slices from the registered MR images and corresponding fixed TRUS images.Source: §4 · p. 4, Figure 5 · p. 4
5. CONCLUSION
The paper presents a flexible framework for fully automatic multimodal registration from image pairs, with a network that learns global and local interventional deformations. It reports validation on 111 labelled intraoperative prostate image pairs and identifies cross-centre and broader application generalisation as future work.
5. CONCLUSION
- The framework supports different image resamplers, deformation regularisers, and anatomical labels to register multimodal images using only an input image pair.The method is described as fully automatic.Source: §5 · p. 4
- The network architecture learns both global and local interventional deformations.Source: §5 · p. 4
- Registration results were validated on 111 pairs of labelled intraoperative prostate image data; future work will examine generalisation across centres and applications.Source: §5 · p. 4