Source-linked AI summary
A Deep Metric for Multimodal Registration
Martin Simonovsky, Benjamín Gutiérrez-Becker, Diana Mateus, Nassir Navab, Nikos Komodakis
TL;DR
Multimodal registration needs similarity measures that handle large appearance differences across imaging modalities. The paper learns such a measure with a CNN trained to distinguish aligned from misaligned patches, and evaluates it on deformable registration across separate datasets. The metric generalizes across scanners and demographics, works with few aligned image pairs, and significantly outperforms mutual information in the reported task.
Problem
Multimodal registration lacks a general rule for comparing images because tissue and organ appearance varies greatly across imaging modalities.
Method
The paper models multimodal similarity with a CNN that discriminates between aligned and misaligned patches and incorporates the metric into first-order registration optimization.
Results
The CNN metric significantly outperforms mutual information in intersubject T1-T2 deformable registration while generalizing from the IXI training dataset to ALBERTs images.
Takeaways & Limitations
The metric can be trained from scratch using a few aligned image pairs and remains reliable across different scanners and demographic groups.
Takeaways & Limitations
The paper reports plans to extend the method to additional modalities such as ultrasound.
Abstract
from arXiv · showhide
Multimodal registration is a challenging problem in medical imaging due the high variability of tissue appearance under different imaging modalities. The crucial component here is the choice of the right similarity measure. We make a step towards a general learning-based solution that can be adapted to specific situations and present a metric based on a convolutional neural network. Our network can be trained from scratch even from a few aligned image pairs. The metric is validated on intersubject deformable registration on a dataset different from the one used for training, demonstrating good generalization. In this task, we outperform mutual information by a significant margin.
1 Introduction
Multimodal registration lacks a general image-comparison rule because tissue and organ appearance varies across modalities. The paper proposes a CNN-based similarity metric that learns aligned-versus-misaligned patch discrimination and generalizes across datasets.
- Multimodal registration is difficult because different imaging principles produce highly variable tissue and organ appearances.
- CNNs are motivated as higher-capacity models for learning similarity across modality combinations from aligned examples.The approach differs from prior probabilistic, structured-support-vector, and Adaboost-based methods by learning characteristics directly from training data.
- The paper proposes a CNN-based similarity metric formulated as classification of aligned and misaligned multimodal patches.The authors identify this as the first use of CNNs for multimodal medical image registration.
- The metric achieves reliable registration on neonatal ALBERTs images while training on the separate adult IXI database.The evaluation spans different scanners and demographic groups.
- The method is designed to learn reliable multimodal similarities from a small training set, addressing a common constraint in medical imaging.
2 Method
The method learns a CNN-based multimodal dissimilarity metric from aligned and misaligned image patches, then aggregates local comparisons for registration optimization. Its design supports continuous and discrete optimization, efficient gradient computation, and training from few image pairs.
- Registration formulation: Registration minimizes an alignment metric plus a regularization term over transformation parameters.The metric quantifies alignment cost, while regularization constrains the mapping.
- Learned similarity metric: The proposed network N estimates dissimilarity between same-sized fixed-image and warped moving-image patches.The two cubic patches are treated as a 2-channel 3D input to a feed-forward CNN.
- Metric aggregation: Local patch comparisons are aggregated over densely sampled, significantly overlapping domains to form the image-level metric.Overlapping patches also influence gradient stability because each pixel can belong to multiple patches.
- Continuous optimization: Small patch-grid strides are important for registration quality, whereas subsampling image-domain samples has a minor performance impact.The authors report considerable quality drops when the patch-grid stride is not small.
- Continuous optimization: Registration gradients combine network intensity derivatives, moving-image gradients, and the transformation Jacobian, decoupling network design from transformation type.Each iteration uses moving-image resampling plus one forward and one backward network pass, with GPU-compatible operations.
- Network architecture: The 2-channel CNN jointly processes both modalities, while pseudo-siamese alternatives consistently reduced registration performance.The network uses five layers, 2M parameters, patch size p = 17, and stride s = 4.
- Training: Training uses augmented aligned image pairs, equal-probability positive and negative patch samples, hinge loss, and online sampling.Online sampling and augmentation provide practically unlimited training data, with no observed overfitting even for small training sets.
3 Experiments and Results
The experiments evaluate the CNN metric on neonatal brain registration after training on an independent adult dataset, comparing it with mutual information and examining data efficiency and optimization behavior.
- 3.1 Deformable Registration of Neonatal Brain MRI Images: The CNN metric is trained on approximately 600 aligned adult T1-T2 image pairs from IXI and tested on the structurally different neonatal ALBERTs dataset.This setup evaluates generalization across scanners and demographic differences.
- 3.1 Deformable Registration of Neonatal Brain MRI Images: 68 intersubject registrations on 18 neonatal subjects evaluate average Dice and Jaccard overlap after T1-to-T2 and T2-to-T1 registration.The ALBERTs dataset contains aligned T1-T2 scans with segmentations of 50 anatomical regions; two subjects were reserved for validation.
- 3.1 Deformable Registration of Neonatal Brain MRI Images: About 4 points of statistically significant improvement in both Dice and Jaccard coefficients is attributed to CNN over mutual information.The comparison uses the same transformation model, optimizer, and registration pipeline for both metrics.
- 3.1 Deformable Registration of Neonatal Brain MRI Images: CNN improved alignment in all registration runs, whereas MI+M and especially MI did not consistently improve alignment.Figure 2 plots initial versus final average Dice scores for each run; the dashed line represents identity transformation.
- 3.1 Deformable Registration of Neonatal Brain MRI Images: CNN was only about 2x slower than MI, with most of the difference attributed to moving-image resampling.The highest accuracies of both methods were similar, reaching up to 0.8, and appeared nearly independent of initial misalignment.
- 3.2 Training Set Size and 3.3 Plausibility of Metric and Its Derivatives: Very small training sets remained effective: k = 11 and k = 6 performed very well, while k = 3 no longer significantly outperformed MI+M.The authors also visualize metric values and derivatives under single-parameter perturbations to assess optimization behavior.
4 Conclusion
The paper concludes that a CNN can provide a multimodal 3D registration metric trainable from few aligned pairs and effective across datasets. It also identifies discrete optimization and additional modalities as future directions.
- The proposed multimodal 3D registration metric is based on a convolutional neural network.
- The network can be trained from scratch using only a few aligned image pairs, largely because of the data sampling scheme.
- On intersubject T1-T2 deformable registration, CNN generalized to a dataset different from training and outperformed mutual information by a significant margin.
- The network could be incorporated into a discrete optimization framework, where only its forward pass would be needed for unary alignment costs.
- The authors plan to apply the method to additional modalities, including ultrasound.
A Comparison with MIND
The comparison with MIND finds lower registration scores for MIND than for both MI+M and the CNN approach, while noting that the implementations used different registration pipelines.
- MIND achieved a Dice score of 0.610 ± 0.073 and a Jaccard score of 0.458 ± 0.070 across 68 registrations.The comparison used validated MIND hyperparameters: a six-neighborhood search region, Gaussian weighting σ = 0.5, and regularization α = 0.2.
- MIND results were clearly inferior to both MI+M and the CNN-based approach, although MIND had a much shorter running time.
- The comparison is illustrative because MIND's deformable registration code used different regularization and optimization and omitted the similarity-transformation step.