Weakly-Supervised Convolutional Neural Networks for Multimodal Image Registration

Yipeng Hu; Marc Modat; Eli Gibson; Wenqi Li; Nooshin Ghavami; Ester Bonmati; Guotai Wang; Steven Bandula; Caroline M. Moore; Mark Emberton; Sébastien Ourselin; J. Alison Noble; Dean C. Barratt; Tom Vercauteren

Source-linked AI summary

Weakly-Supervised Convolutional Neural Networks for Multimodal Image Registration

Yipeng Hu, Marc Modat, Eli Gibson, Wenqi Li, Nooshin Ghavami, Ester Bonmati, Guotai Wang, Steven Bandula, Caroline M. Moore, Mark Emberton, Sébastien Ourselin, J. Alison Noble, Dean C. Barratt, Tom Vercauteren

arXiv:1807.03361v1Jul 9, 2018cs.CVcs.AIcs.LGcs.NE

TL;DR

Multimodal registration lacks reliable voxel-level ground truth, motivating a weakly supervised approach based on higher-level anatomical correspondence. The paper trains convolutional networks with anatomical labels and image intensities, then performs deformable registration from unlabelled image pairs; its comparisons report improved performance over affine-only and prior architectures, while validation remains limited by the absence of a fully unseen development dataset.

Problem
Voxel-level ground truth for learning multimodal spatial correspondence is scarce, while intensity and manually selected feature approaches face robustness, cost, and feasibility challenges.
Method
The framework trains convolutional registration networks using higher-level anatomical labels, full image intensities, multiscale Dice similarity, and deformation regularisation, while inference uses only image pairs.
Results
The proposed deformable networks significantly outperform the affine-only “Global-Net” and improve generalisation over the “Composite-Net” architecture in target registration error and Dice score.
Takeaways & Limitations
The trained framework supports fast, fully automatic multimodal registration while accommodating varied anatomical labels that need not be consistently available across training pairs.
Takeaways & Limitations
Validation lacked a sizable dataset completely unseen during methodology development, so generalisation ability was not tested conclusively.

Abstract

from arXiv · show

One of the fundamental challenges in supervised learning for multimodal image registration is the lack of ground-truth for voxel-level spatial correspondence. This work describes a method to infer voxel-level transformation from higher-level correspondence information contained in anatomical labels. We argue that such labels are more reliable and practical to obtain for reference sets of image pairs than voxel-level correspondence. Typical anatomical labels of interest may include solid organs, vessels, ducts, structure boundaries and other subject-specific ad hoc landmarks. The proposed end-to-end convolutional neural network approach aims to predict displacement fields to align multiple labelled corresponding structures for individual image pairs during the training, while only unlabelled image pairs are used as the network input for inference. We highlight the versatility of the proposed strategy, for training, utilising diverse types of anatomical labels, which need not to be identifiable over all training image pairs. At inference, the resulting 3D deformable image registration algorithm runs in real-time and is fully-automated without requiring any anatomical labels or initialisation. Several network architecture variants are compared for registering T2-weighted magnetic resonance images and 3D transrectal ultrasound images from prostate cancer patients. A median target registration error of 3.6 mm on landmark centroids and a median Dice of 0.87 on prostate glands are achieved from cross-validation experiments, in which 108 pairs of multimodal images from 76 patients were tested with high-quality anatomical labels.

1 Introduction

Multimodal registration is difficult because intensity and manually selected feature correspondences are unreliable or costly, while voxel-level training ground truth is scarce. The paper proposes using higher-level anatomical labels to train a deformable registration network that requires only unlabelled image pairs at inference.

Multimodal registration supports fusing detailed pre-procedural images with constrained intra-procedural imaging for image-guided procedures.
Robust multimodal intensity similarity is difficult because imaging processes can correlate anatomically unrelated structures and intra-procedural imaging varies spatially and temporally.
Manual anatomical feature selection can be robust but is user-dependent, costly, or infeasible during procedures, while assisted segmentation has not demonstrated clinical value in fast-evolving applications.
For prostate MR-to-TRUS registration, correlated but anatomically different boundaries can produce false alignment with established intensity-based similarity measures.
Model-to-image methods require correspondent features in both images and motion priors because consistently available features are sparse, with ad hoc landmarks typically available case by case.
Higher-level anatomical labels provide weak supervision for voxel correspondence, enabling deformable registration trained with labels and intensities but using only unlabelled image pairs during inference.

2 Method

The method trains a neural network to predict dense displacement fields by maximizing multiscale similarity between warped moving labels and fixed labels, while using only image pairs as network inputs at inference. It combines weak anatomical supervision, differentiable multiscale Dice, stochastic sampling, deformation regularization, and a memory-efficient multiscale architecture.

Weakly-Supervised Image Registration Framework: The network predicts dense displacement fields from image pairs and is trained to maximize expected similarity across corresponding anatomical labels.Training uses label correspondence as weak supervision rather than voxel-level ground truth.
Weakly-Supervised Image Registration Framework: Two-stage sampling draws K image pairs uniformly, then one label pair uniformly from each selected image pair to form minibatches with an unbiased gradient estimator.The procedure handles image pairs with variable numbers of associated labels.
Weakly-Supervised Image Registration Framework: Label similarity is computed between warped moving labels and fixed labels without using labels as network inputs during inference.This allows the trained registration model to operate on unlabelled image pairs.
Multiscale Dice for Measuring Label Similarity: The proposed multiscale Dice averages Dice similarity after Gaussian filtering at seven scales with σ ∈ {0, 1, 2, 4, 8, 16, 32} mm.Including σ=0 retains the original unfiltered binary label, while larger scales capture spatial information between labels.
Multiscale Dice for Measuring Label Similarity: The multiscale loss is differentiable and can be evaluated on-the-fly after non-rigid warping and data augmentation.A multiscale negative cross-entropy variant is also considered for comparison.
Network Architecture: A single network predicts displacement summands across resolution levels, providing global information without the substantial memory usage of a separate affine sub-network.The architecture uses dense connections, summation-based residual shortcuts, and displacement-space skip layers across resolutions.

3 Experiments

Experiments evaluated the proposed registration networks on 108 MR–TRUS image pairs from 76 patients using patient-level cross-validation. Comparisons covered loss functions, preprocessing, displacement-output configurations, and previously proposed global/local architectures.

Dataset and evaluation: 108 image pairs from 76 patients were evaluated using 12-fold patient-level cross-validation.Each fold held out test data from 6–7 patients.
Baseline and loss variants: The Baseline network used Adam with a 10^-5 starting learning rate, minibatch size 4, deformation weight α=0.5, and 32 initial feature channels.Final displacement layers were zero-initialised, while other parameters used Xavier initialisation.
Baseline and loss variants: Baseline-msCE replaced multiscale Dice with multiscale cross-entropy, while Baseline-L2 replaced bending energy with average displacement-gradient L2 regularisation.These variants isolate alternative similarity and deformation-regularisation choices.
Baseline and loss variants: Baseline-preFilt used Gaussian-pre-filtered label maps before training, while multiscale Dice was evaluated on resampled maps during training.Pre-filtering was introduced as a potential way to accelerate training.
Architecture variants: Baseline-δ0 summed only the finest-resolution displacement prediction, whereas Baseline-δ1-4 omitted that prediction and summed levels s1–s4.Both variants retained the same down-sampling and up-sampling blocks as the Baseline network.
Previous networks: The previously proposed Global-Net predicted affine transformations, and Composite-Net combined its output displacement fields with those from Local-Net.Global-Net used the same architecture as Local-Net’s four down-sampling blocks, with independently learnable parameters.
Dataset and evaluation: The evaluation used centroid distance error, target registration error, and prostate-gland Dice similarity.TRE was computed as the root-mean-square centroid distance over landmark pairs for each patient.
Dataset and evaluation: The reported network comparisons used paired Wilcoxon signed-rank tests at αH=0.05, with confidence intervals reported when p-values exceeded αH.All metrics were calculated on left-out test data.

4 Results

The proposed Baseline network achieved strong deformable-registration performance and real-time inference, outperforming affine-only and prior network variants. Results also show that regularisation, multiscale loss design, and multiresolution displacement prediction materially affect accuracy and deformation plausibility.

4.1 “Baseline” Performance: 3.6 mm median TRE and 0.87 median DSC were achieved by the Baseline network on landmark centroids and prostate glands, respectively.The corresponding first and third quartiles were 2.3 and 6.5 mm for TRE, and 0.82 and 0.89 for DSC.
4.2 Variants of the “Baseline” Network: Replacing multiscale Dice with cross-entropy worsened TRE but improved binary DSC, while the Baseline-L2 regulariser produced poorer generalisation on both metrics.The cross-entropy comparison had p-values below 0.001 for TRE and 0.046 for DSC; Baseline-L2 differences were significant for TRE and DSC.
4.2 Variants of the “Baseline” Network: Pre-computing label filtering preserved TRE and DSC while achieving an approximately 25% gain in training time in these experiments.The reported effects were not statistically significant for TRE or DSC, with p-values of 0.458 and 0.498, respectively.
4.2 Variants of the “Baseline” Network: Reducing displacement prediction to the finest resolution worsened performance, whereas omitting that level remained statistically competitive; removing trilinear up-sampling increased median TRE to 6.4 mm.The Baseline outperformed Baseline-δ0, while Baseline-δ1-4 showed no significant TRE or DSC difference from Baseline.
4.3 Comparison Results with the Previous Networks of (Hu et al., 2018): The Baseline significantly outperformed Global-Net’s affine-only transformation and improved generalisation over Composite-Net in both TRE and DSC.Both comparisons reported p-values below 0.001; the proposed architecture also spared GPU memory used to train Global-Net.
4.3 Comparison Results with the Previous Networks of (Hu et al., 2018): With regularisation weight α=0.5, no negative Jacobian determinants were found, whereas α=0.01 produced increasing deformation variance and negative determinants.The latter implied that physically implausible deformation may occur without appropriate regularisation.
4.4 Comparison with Pairwise Registration Methods: General-purpose pairwise registrations performed poorly, with all nine median TREs above 24 mm and no median DSC above 0.77.Intensity-based methods produced median TREs from 26.7–35.0 mm, while the pre-registration median TRE was 34.8 mm.

5 Discussion

The framework demonstrates non-iterative dense correspondence prediction from unlabelled image pairs using sparse anatomical annotations. Its flexibility spans label types, network architectures, and regularisers, while validation and topology-preservation limitations remain.

5 Discussion: The method demonstrated non-iterative voxel-correspondence prediction from unlabelled inputs trained with only sparse annotations.
5 Discussion: Sparse landmark supervision cannot independently represent dense correspondence, so the architecture implicitly learns regularised dense displacement fields from latent correspondence samples.
5 Discussion: Multiscale Dice was proposed to balance inter-class gradient differences, while alternative generalised Dice and weighted cross-entropy strategies did not further improve results in this application.
5 Discussion: The predicted displacement fields lacked explicit topology preservation, limiting direct guarantees against physically implausible deformation.Jacobian-based regularisation is suggested for settings with more landmarks and clinically plausible larger deformations.
5 Discussion: A completely unseen sizable dataset was unavailable, so cross-validation could not conclusively test generalisation beyond methodology development.
5 Discussion: The framework supports varied anatomical features, neural architectures, and deformation regularisers, and uses only input image pairs during fully automatic registration.Validation used 108 labelled intraoperative prostate image pairs; future work targets other centres and applications.