PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking Intra-Patient 3D Deformable Multimodal Image Registration

Matteo Barbieri, Giammarco La Barbera, Juan Pablo De La Plata, Sabine Sarnacki, Isabelle Bloch, Pietro Gori

arXiv:2609.15669v1cs.CVcs.AI

TL;DR

Multimodal registration remains challenging because corresponding anatomy can have different intensities across modalities. This paper benchmarks intra-patient deformable methods across datasets and settings, finding no consistently superior method and a trade-off between overlap and image-based similarity.

  • Problem

    Multimodal registration lacks direct intensity correspondence, and the benchmark finds no evaluated method consistently superior across datasets and evaluation criteria.

    Source: §1 · p. 3, §6 · p. 28

  • Method

    The paper benchmarks selected deformable intra-patient multimodal methods on synthetic transformation recovery and realistic correspondence estimation across public and private datasets.

    Source: §1 · p. 3

  • Results

    No evaluated method performs consistently best across datasets and criteria; anatomy-guided approaches improve overlap but often worsen MIND-SSC loss versus unsupervised counterparts.

    Source: §6 · pp. 28, 30

  • Takeaways & Limitations

    Deformation recovery and registration accuracy are not equivalent: lower deformation error need not coincide with better overlap-based metrics.

    Source: §6 · p. 30

    Clinical multimodal registration remains limited where sub-millimetric to millimetric accuracy is required, with performance varying across tasks and anatomies.

    Source: §2.3 · p. 7

Abstract

from arXiv · show

Multimodal image registration is a key component of many clinical workflows, yet it remains challenging because corresponding anatomical structures often exhibit substantially different image intensities across modalities. In this work, we present a comprehensive benchmark of intra-patient 3D multimodal deformable registration methods across three datasets covering different anatomical regions and difficulty levels, including both synthetic deformation recovery and real clinical scenarios. We evaluate classical optimization-based approaches and modern learning-based methods, including recent deep learning and foundation models, using complementary metrics: Average Dice similarity coefficient (DSC), average 95th-percentile Hausdorff distance (HD95), and a modality-independent structural similarity measure based on the MIND self-similarity context (MIND-SSC). Results show high variability across datasets, with learning-based methods demonstrating superior performance on large synthetic benchmarks, while only limited improvements are observed in real pelvic registration. A key finding of this study is the consistent disagreement between geometric metrics (DSC, HD95) and image-based similarity metrics (MIND-SSC), highlighting that improved overlap does not necessarily imply better global multimodal correspondence. Furthermore, anatomy-guided approaches achieve the highest overlap scores but exhibit degraded performance outside of segmented regions, revealing a trade-off between label-driven alignment and global structural coherence. Overall, our results indicate that no current method achieves robust performance across anatomies and modalities. We demonstrate that intra-patient 3D multimodal registration requires multi-criteria evaluation, including deformation-based metrics, and remains an open problem.

1. Introduction

Multimodal registration is difficult because anatomical correspondences do not necessarily share image intensities, limiting conventional assessment. This work benchmarks intra-patient deformable multimodal registration across synthetic and real settings of increasing difficulty.

  • Deformable registration estimates spatially varying transformations, often regularized to promote smoothness, topology preservation, or diffeomorphic mappings.Registration generally aligns a moving image to a fixed reference image.Source: §1 · p. 2
  • Classical methods iteratively optimize similarity and regularization, whereas learning-based methods infer transformations from data and enable real-time inference once trained.Classical approaches offer explicit control over transformation properties.Source: §1 · pp. 2–3
  • Registration strategy depends on modality, dimensionality, anatomy, and whether alignment is intra- or inter-subject, shaping similarity measures and models.Source: §1 · p. 3
  • Multimodal registration is challenging because anatomical features across modalities lack direct intensity correspondence, making intensity-based assessment unsuitable.MIND and MIND-SSC have been proposed as better-suited similarity metrics, alongside learned similarity approximations such as REE-UNET.Source: §1 · p. 3
  • The benchmark evaluates recovery of known transformations under modality shifts and correspondence estimation in realistic multimodal settings across two public and one private dataset.It varies modalities and anatomical regions using synthetic and real misaligned data.Source: §1 · p. 3

2. Related work

Registration methods differ in transformation parameterization, similarity measures, and regularization. The related work surveys monomodal and multimodal approaches and identifies accuracy, correspondence, and generalization challenges that persist in clinical settings.

2. Related work

  • Registration methods vary in transformation parameterization, image similarity, and regularization; the review excludes frameworks that modify intensity or image appearance.The energy formulation combines image similarity with a regularity term, balanced by a positive weight.Source: §2 · p. 4

2.1. Monomodal image registration

  • Monomodal registration assumes corresponding voxels have similar intensity distributions, supporting metrics such as SSD or cross-correlation.Classical methods include diffeomorphic and dense non-parametric approaches, while deep methods include unsupervised deformation prediction and weakly supervised anatomical priors.Source: §2.1 · pp. 4–5
  • Transformer-based methods use self-attention for long-range interactions, while hybrid approaches combine learned similarity or features with explicit optimization.Multi-scale representations are used to combine fine detail and global structural alignment.Source: §2.1 · pp. 5–6

2.2. Multimodal image registration

  • Multimodal methods address absent intensity correspondence with information-theoretic metrics, structural descriptors such as MIND-SSC, and learned cross-modal representations.Recent approaches include contrastive and cycle-consistent learning, cross-modal attention, foundation models, and anatomy-aware architectures.Source: §2.2 · pp. 6–7

2.3. Limitations of existing benchmarks and methods

  • Clinical registration remains limited where sub-millimetric to millimetric accuracy is needed, with task-dependent errors and substantial variation across anatomies and modalities.Learn2Reg reports residual TRE of 1–3 mm in controlled settings, with abdominal HD95 reaching up to 20 mm for top-ranked methods.Source: §2.3 · p. 7
  • Multimodal clinical tasks remain especially difficult: prostate MRI–TRUS methods report TRE of 4–11 mm, against a cited biopsy-guidance tolerance of ≤2.5 mm.The cited review reports an average TRE of 4.95 mm for the best method in a manually curated ultrasound-registration dataset.Source: §2.3 · p. 8

3. Evaluated registration methods

The benchmark covers classical optimization, handcrafted-descriptor, and learning-based registration methods, using MIND-SSC as common training similarity for learning methods. The methods use distinct transformation and consistency strategies.

3. Evaluated registration methods

  • Methods span classical optimization, handcrafted multimodal descriptors, and CNN- and foundation-model-based frameworks, selected for use and implementation availability.Learning-based models were trained using MIND-SSC as a common similarity metric.Source: §3 · p. 9
  • ANTs SyN uses symmetric diffeomorphic flows that move the images toward a common midpoint, with regularity and image similarity in its energy.The final transformation maps the moving image to the target.Source: §3 · pp. 9, 11
  • Deforming Demons iteratively estimates a dense displacement using local image gradients and Gaussian smoothing for spatial coherence.Diffeomorphic extensions can represent the transformation as the exponential of a stationary velocity field.Source: §3 · p. 11
  • Elastix represents deformations with basis functions, typically B-splines on a control-point grid, enabling low-dimensional optimization and explicit smoothness control.Its parameters are optimized by minimizing an energy functional.Source: §3 · pp. 11–12
  • VoxelMorph uses a neural network to predict a dense deformation field from image pairs and can train with a smoothness-regularized objective.Diffeomorphic variants predict a stationary velocity field and integrate it to ensure invertibility.Source: §3 · p. 12
  • GradICON learns forward and backward deformations and penalizes violations of inverse consistency alongside image similarity and smoothness.Spatial image gradients are incorporated to stabilize correspondence estimation and improve alignment near edges.Source: §3 · p. 12
  • M2M-GradICON maps images into a shared monomodal representation and estimates four pairwise deformations, regularized for canonical and inverse consistency.The shared network produces pairwise mappings to support transitive consistency and a globally coherent transformation space.Source: §3 · pp. 12–13
  • multiGradICON composes transformations across resolution levels and enforces similarity, smoothness, and consistency constraints at each level.Its hierarchical structure supports coarse-to-fine alignment while maintaining global coherence across multiple images.Source: §3 · pp. 13–14

4. Benchmarking methodology

The methodology benchmarks registration across three datasets and both synthetic deformation recovery and real clinical misalignment. It specifies training, complementary evaluation metrics, and a common comparison protocol.

4. Benchmarking methodology

  • The benchmark spans multiple anatomies, modalities, and registration scenarios, with datasets and synthetic deformation protocol establishing quantitative ground truth.The framework also specifies training, evaluation metrics, and procedures for comparison.Source: §4 · p. 14

4.1. Datasets

  • BRATS and HCP provide pre-registered brain image pairs, while private IMAG2 contains multimodal pelvic MRI from 80 pediatric subjects, including controls, tumors, and malformations.IMAG2 pairs unregistered T2-weighted MRI with diffusion-weighted imaging acquired during the same procedure.Source: §4.1 · p. 14

4.2. Synthetic Deformation Protocol

  • For pre-registered BRATS and HCP pairs, independent random B-spline fields on a 4 × 4 × 4 control grid synthetically deform target images.Maximum displacement amplitude is 15 voxels per spatial dimension, producing smooth fields intended to span small and large displacements.Source: §4.2 · p. 16

4.3. Training protocol

  • Trainable methods use unsupervised training, effective batch size 8, Adam starting at 10−4, validation every five epochs, and early stopping below learning rate 10−7.The learning-rate scheduler uses validation loss with patience 10 and a decrease factor of 0.5.Source: §4.3 · pp. 16–17

4.4. Evaluation Metrics

  • Evaluation reports average DSC for volumetric overlap, average HD95 for boundary accuracy, and MIND-SSC loss as a surrogate for global structural similarity.It also computes folding ratio and, for synthetic datasets with known fields, relative DVF error.Source: §4.4 · pp. 17–18

4.5. Benchmarking Protocol

  • All methods share preprocessing, image pairs, intra-patient settings, and an evaluation pipeline; classical hyperparameters are tuned on 10 validation cases, while learning methods use defaults.Pretrained multiGradICON requires 175 × 175 × 175 voxel inputs, so images are cropped and its performance interpreted in light of that constraint.Source: §4.5 · p. 19

5. Results

Registration performance varied by dataset and metric: learning-based methods generally led on synthetic brain benchmarks, while gains on real pelvic registration were modest. Anatomy-guided methods improved overlap but could compromise multimodal consistency and deformation recovery.

5.1.1. Image-guided registration

  • On synthetic BRATS and HCP, learning-based methods outperformed classical approaches in overlap and image consistency, but deformation-recovery winners differed; real pelvic gains were modest.GradICON and VoxelMorph led image-based measures, while M2M-GradICON and multiGradICON generally had lower deformation errors and folding ratios; GradICON was most consistent across datasets.Source: Table 5 · p. 21, Table 6 · p. 22, §5.1.1 · p. 23

5.1.2. Anatomy-guided registration

  • Anatomy-guided registration improved geometric alignment, especially overlap, but often traded multimodal consistency and synthetic deformation recovery for alignment of labeled structures.ANTs SyN seg. achieved the highest overlap across datasets, while GradICON seg. generally had lower MIND-SSC loss; labels provide no constraint outside segmented regions.Source: §5.1.2 · pp. 23–25

5.2. Qualitative results

  • Qualitative cases show strongest visible overlap from ANTs SyN Seg. on BRATS and the labeled pelvic structures, while significant brain-stem misalignment remains on HCP.Displayed subjects were selected for the largest pre-to-post registration improvement in global Dice, regardless of method.Source: §5.2 · pp. 25–26, Figure 3 · p. 26, Figure 4 · p. 27

5.3. Computational Performance

  • Learning-based methods ran two to three orders of magnitude faster; VoxelMorph had the lowest runtime at 0.028 s per registration pair.Classical methods used CPU resources, while learning-based methods relied on GPU acceleration and required 7.60–24.56 GB of GPU memory.Source: §5.3 · pp. 26–28

6. Discussion

Multimodal registration must infer spatial relationships across different contrast mechanisms, and its performance depends on the transformation, similarity measure, regularization, anatomy, and modality pair. No evaluated method performs consistently across datasets and criteria, while real-world registration with limited training data remains challenging.

  • Learning-based methods were generally more consistent than classical methods on BRATS and HCP, yet even the best methods showed limited-to-no improvement on IMAG2 pelvis.Classical methods varied by dataset: ANTs SyN improved on BRATS but degraded on HCP, while Elastix deteriorated on IMAG2.Source: §6 · p. 29
  • DVF recovery errors exceeded 80% and reached 211%, but deformation recovery is not a direct measure of registration accuracy.Methods with lower DVF errors did not necessarily achieve better overlap or boundary metrics; the two tasks are related but distinct.Source: §6 · pp. 29–30
  • Methods with the strongest overlap and boundary-alignment gains often produced more local topology violations, whereas nearly fold-free methods had lower registration accuracy.The authors argue that deformation regularity should be a primary evaluation criterion because local folding creates non-invertible mappings.Source: §6 · p. 30
  • Anatomy-guided methods can raise overlap without improving global multimodal correspondence, because their optimization prioritizes annotated structures.ANTs SyN seg. achieved the highest DSC, while GradICON seg. had substantially lower MIND-SSC loss and better BRATS deformation recovery; their training objectives differ.Source: §6 · pp. 30–31
  • Zero-shot multiGradICON did not consistently outperform smaller GradICON models, particularly on pelvic IMAG2, so large-scale pretraining alone did not guarantee robust performance.The authors note its evaluation lacked dataset-specific adaptation and its training objective emphasized cross-task transformation consistency.Source: §6 · p. 31
  • Synthetic BRATS and HCP results improved across multiple methods, but virtually no unsupervised method meaningfully improved over the unregistered baseline on pelvic IMAG2.The contrast suggests synthetic benchmarks may overestimate performance on real multimodal correspondence problems.Source: §6 · pp. 31–32
  • Segmentation overlap, deformation recovery, image similarity, and topology each capture only part of registration quality, so no single metric characterizes it.Visual inspection and quantitative scores can diverge, and the practical significance of such differences depends on the application.Source: §6 · pp. 32–33
  • The study is bounded by synthetic brain deformations, dependence on segmentation quality, untuned learning-method hyperparameters, and a small pelvic dataset without dense correspondences.The authors point to larger real-world multimodal datasets with expert annotations and clinically meaningful evaluation as future needs.Source: §6 · p. 33

7. Conclusion

The benchmark concludes that multimodal deformable registration remains open and highly context-dependent. Anatomy-guided methods can align labeled structures well without guaranteeing global multimodal consistency, while other methods also show limits in robustness across settings.

  • Multimodal deformable registration remains an open, highly context-dependent problem, with no method established as robust across anatomical regions and real clinical deformation scenarios.Source: §7 · pp. 33–34
  • Anatomy-guided methods can align labeled structures excellently but may not guarantee global multimodal consistency.Source: §7 · pp. 33–34
  • GradICON and VoxelMorph provide competitive, sometimes more balanced performance, but their robustness across anatomical regions and real clinical deformation scenarios remains limited.Source: §7 · p. 34
  • Classical methods remain useful baselines but show strong dataset dependence, supporting comprehensive multimetric evaluation and methods that jointly address anatomical accuracy, image-based consistency, plausible deformation, and downstream clinical performance.Source: §7 · p. 34

Related papers

Loading 2609.15669v1…