Source-linked AI summary
Learn2Reg: comprehensive multi-task medical image registration challenge, dataset and evaluation in the era of deep learning
Alessa Hering, Lasse Hansen, Tony C. W. Mok, Albert C. S. Chung, Hanna Siebert, Stephanie Häger, Annkristin Lange, Sven Kuckertz, Stefan Heldmann, Wei Shao, Sulaiman Vesal, Mirabela Rusu, Geoffrey Sonn, Théo Estienne, Maria Vakalopoulou, Luyi Han, Yunzhi Huang, Pew-Thian Yap, Mikael Brudfors, Yaël Balbastre, Samuel Joutard, Marc Modat, Gal Lifshitz, Dan Raviv, Jinxin Lv, Qiang Li, Vincent Jaouen, Dimitris Visvikis, Constance Fourcade, Mathieu Rubeaux, Wentao Pan, Zhe Xu, Bailiang Jian, Francesca De Benetti, Marek Wodzinski, Niklas Gunnarsson, Jens Sjölund, Daniel Grzech, Huaqi Qiu, Zeju Li, Alexander Thorley, Jinming Duan, Christoph Großbröhmer, Andrew Hoopes, Ingerid Reinertsen, Yiming Xiao, Bennett Landman, Yuankai Huo, Keelin Murphy, Nikolas Lessmann, Bram van Ginneken, Adrian V. Dalca, Mattias P. Heinrich
TL;DR
Comprehensive evidence comparing medical image registration methods across clinically relevant tasks remains limited. Learn2Reg addresses this gap with a multi-task benchmark, complementary evaluation, and analyses of transferability, supervision, and bias. Across tasks, no single approach was best everywhere, while optimisation-based methods could achieve runtimes comparable to learning-based methods.
Problem
Few studies comprehensively compare registration approaches across clinically relevant tasks, limiting fair benchmarking evidence.
Method
Learn2Reg establishes an accessible multi-task benchmark spanning diverse registration tasks, complementary metrics, and analyses of transferability, supervision, and label bias.
Results
No single approach performed best across all tasks, while GPU-accelerated optimisation-based methods reached registration runtimes of a few seconds.
Takeaways & Limitations
ConvexAdam, LapIRN, and corrField were shown to work robustly across all tasks with only minor hyperparameter adjustments.
Takeaways & Limitations
Offline participant training could introduce bias from inaccessible additional data and prevents using larger private datasets.
Abstract
from arXiv · showhide
Image registration is a fundamental medical image analysis task, and a wide variety of approaches have been proposed. However, only a few studies have comprehensively compared medical image registration approaches on a wide range of clinically relevant tasks. This limits the development of registration methods, the adoption of research advances into practice, and a fair benchmark across competing approaches. The Learn2Reg challenge addresses these limitations by providing a multi-task medical image registration data set for comprehensive characterisation of deformable registration algorithms. A continuous evaluation will be possible at https://learn2reg.grand-challenge.org. Learn2Reg covers a wide range of anatomies (brain, abdomen, and thorax), modalities (ultrasound, CT, MR), availability of annotations, as well as intra- and inter-patient registration evaluation. We established an easily accessible framework for training and validation of 3D registration methods, which enabled the compilation of results of over 65 individual method submissions from more than 20 unique teams. We used a complementary set of metrics, including robustness, accuracy, plausibility, and runtime, enabling unique insight into the current state-of-the-art of medical image registration. This paper describes datasets, tasks, evaluation methods and results of the challenge, as well as results of further analysis of transferability to new datasets, the importance of label supervision, and resulting bias. While no single approach worked best across all tasks, many methodological aspects could be identified that push the performance of medical image registration to new state-of-the-art performance. Furthermore, we demystified the common belief that conventional registration methods have to be much slower than deep-learning-based methods.
I. INTRODUCTION
Learn2Reg addresses the limited comprehensive comparison of medical image registration by establishing a broad benchmark spanning tasks, anatomies, modalities, and supervision conditions.
- Existing comparisons focused on specific tasks or algorithmic aspects rather than comprehensively characterising current registration approaches.
- Learn2Reg evaluates methodological components and supervision strategies across clinically useful 3D registration tasks.
- The challenge spans brain, abdomen, and thorax anatomies; ultrasound, CT, and MR modalities; and intra- and inter-patient registration.
- Its benchmark assesses accuracy, robustness, plausibility, and speed across task-tailored solutions.
- The overview analyses more than 65 entries from over 20 teams and examines cross-dataset transfer, label bias, and supervision level.These analyses covered submissions throughout 2020 and 2021.
- Learn2Reg differs from prior challenges and benchmarks by combining broad participation with multi-task evaluation and generally applicable baselines.
B. Contributions
Learn2Reg combines six clinically relevant registration tasks with accessible participation and standardised evaluation, covering diverse anatomy, modalities, and registration settings.
- The challenge used validation submissions followed by a test phase accepting displacement fields or Docker containers.Docker submissions enabled runtime evaluation, while organisers’ institutes had the same data access but were ineligible for awards.
- Learn2Reg provides six complementary tasks addressing brain, abdominal, and lung registration scenarios.The tasks include CuRIOUS, Hippocampus MR, Abdomen CT-CT, Lung CT, Abdomen MR-CT, and OASIS.
- CuRIOUS evaluates multimodal MR–ultrasound registration for correcting tissue shift during brain tumour resection.The dataset includes annotated anatomical landmarks for registration-accuracy evaluation.
- Hippocampus MR contains 394 scans from 195 adults with manually traced hippocampus heads and tails.
- Abdomen CT-CT evaluates inter-patient abdominal registration using 50 scans and 13 manually labelled anatomical structures.
- The benchmark also includes paired Abdomen MR-CT scans, whole-brain OASIS MR scans, and expiration-to-inspiration Lung CT registration.These tasks support multimodal abdominal registration, quantitative brain analysis, and longitudinal lung-motion applications.
C. Challenge Design
Learn2Reg evaluates registration with complementary metrics covering accuracy, robustness, plausibility, speed, and statistical significance for final task ranks.
- The evaluation measures accuracy, robustness, plausibility, and speed using complementary metrics.Final task ranks additionally consider whether differences between results are statistically significant.
1) Metrics:
The challenge combines task-specific indicators of overlap, landmark error, transformation smoothness, and runtime to evaluate registration comprehensively.
- DSC measures overlap between fixed-scan labels and labels on the warped moving scan.
- DSC30 uses the 30th percentile of DSC scores across anatomical structures and cases to assess robustness.
- DSC9 evaluates nine additional Abdomen MR-CT anatomical labels unavailable during training to assess label bias.
- HD95 uses the 95th-percentile Hausdorff distance between surfaces rather than the maximum distance.
- TRE measures Euclidean landmark distance in millimetres, while TRE30 summarizes the 30th percentile of largest landmark distances.
- SDlogJ captures displacement-field smoothness through the standard deviation of the logarithm of the Jacobian determinant.The Jacobian is calculated using central differencing.
- RT measures test-time runtime on identical CPU and GPU hardware for Docker-submitted methods.Timing runs from loading the first scan to writing the displacement field to disk.
2) Ranking Scheme:
Methods are ranked by statistically significant pairwise comparisons for each metric, then aggregated into task and overall scores.
- For each task metric, methods are compared with Wilcoxon signed-rank tests using p<0.05.
- Methods are ranked by the number of pairwise comparisons they win, with ties allowed through score sharing.
- Metric ranks are mapped to scores from 0.1 to 1, and task scores are computed as geometric means of individual metric scores.
III. CHALLENGE ENTRIES
Learn2Reg includes 65 entries from 21 teams using diverse conventional, learning-based, supervised, unsupervised, and hybrid registration strategies across the challenge tasks.
- Participation: In 2021, 21 teams contributed 65 unique task-wise challenge entries, compared with ten teams in 2020.
- Deep-learning methods: 3Idiots applies a patchwise Voxelmorph model with intensity, statistical, and label-based similarity penalties to OASIS.
- Hybrid methods: ConvexAdam decouples learned semantic feature extraction from convex global optimisation and local Adam-based refinement across all six tasks.
- Conventional and unsupervised methods: CorrField provides a faster non-learning unsupervised baseline using sparse keypoint correspondences and MIND-SSC similarity.
- Deep-learning methods: Driver uses a dual-encoder UNet with deformation-field integration, non-rigid feature fusion, similarity losses, weak label supervision, and regularisation.
- Conventional and unsupervised methods: MEVIS uses coarse-to-fine conventional registration with Gaussian pyramids and quasi-Newton L-BFGS optimisation for all tasks except Hippocampus MR.
IV. ADDITIONAL EXPERIMENTS
Additional experiments examine label bias, unsupervised alternatives, and cross-dataset transfer to assess how registration methods generalise beyond their training and evaluation settings.
- Label Bias: Label-bias analysis tests whether learning-based registration favours anatomies that are labelled during both training and evaluation.
- Unsupervised Registration: Top-performing methods use segmentation labels modularly, but this creates a risk of overfitting to the manually annotated anatomies.
- Unsupervised Registration: Unsupervised counterparts of LapIRN and ConvexAdam were compared, with additional comparisons restricted to abdomen and brain tasks.
- Transferability: Transferability evaluates lung CT-CT submissions on DIRLab 4DCT scan pairs with different acquisition data.
A. Challenge Outcome
Challenge outcomes vary by task, with different methods leading across anatomies and modalities. ConvexAdam ranked first overall, while several methods achieved strong task-specific results and faster runtimes.
- Hippocampus MR: Hippocampus MR methods all achieved median Dice > 0.7, with label-supervised methods generally outperforming unsupervised methods.
- CuRIOUS: In CuRIOUS, corrField, PDD-Net, and ConvexAdam reduced initial TRE from 6.38 mm to 2.84 mm, 3.08 mm, and 3.31 mm, respectively.
- Abdomen CT-CT: ConvexAdam, LapIRN, and Estienne achieved Dice Scores of 0.67-0.69 in Abdomen CT-CT, at least 0.2 above all other participants.
- Lung CT: In Lung CT, six teams achieved TRE below 2mm in 2021, compared with only MEVIS reaching 1.72mm below 2mm in 2020.
- Overall Ranking: ConvexAdam ranked among the top three on every task, won Abdomen CT-CT and Abdomen MR-CT, and ranked first overall.
- Overall Ranking: GPU acceleration reduced ConvexAdam’s optimisation-based 3D registration runtime to a few seconds while preserving high quality scores.
B. Additional Experiments
Additional experiments show that label supervision benefits inter-patient registration more than intra-patient registration, while strong methods transfer effectively to DIRLab without retuning.
- Label Bias and Unsupervised Registration: Label supervision yielded nearly no advantage for intra-patient Abdomen MR-CT registration but improved accuracy for inter-patient Abdomen CT-CT, Hippocampus, and OASIS tasks.Excluding anatomical labels caused clear deterioration in inter-patient registration accuracy, whereas Abdomen MR-CT showed little benefit from including them.
- Label Bias and Unsupervised Registration: 69% DSC with all 13 training labels fell to 55% with 4 labels and 45% without labels for ConvexAdam on Abdomen CT-CT.Partial supervision also improved alignment of excluded nearby structures, including esophagus by 16 percentage points and pancreas by 12 percentage points.
- Transferability: 1.22 mm, 1.31 mm, and 1.34 mm DIRLab TREs were achieved by MEVIS, ConvexAdam, and corrField without further hyperparameter adaptations.These methods were the three best performers in the Learn2Reg lung task and transferred well to DIRLab.
- Transferability: LapIRN achieved 1.98 mm TRE on both Learn2Reg lung CT and DIRLab, indicating similar performance across the two datasets.Learn2Reg methods were slightly worse than state-of-the-art results on original-resolution DIRLab images because of preprocessing and reduced resolutions.
VI. DISCUSSION
The discussion identifies broad methodological and practical patterns across tasks, including the value of hybrid approaches, label supervision for inter-patient registration, and smooth transformations. It also highlights challenge-design boundaries affecting generalisation and deployment.
- Reducing Entry Barriers: Standardised preprocessing and annotations lowered entry barriers, enabling broad participation across research groups worldwide.The challenge used common dimensions and isotropic resolution, while providing anatomical annotations for training data.
- Reducing Entry Barriers: Challenge generalisation is limited because omitted settings include affine or rigid pre-alignment, differing fields of view and voxel resolutions, and very high-resolution scans.The challenge design also lacked computational constraints such as GPU memory and CPU runtime, which may limit practical impact for some applications.
- Task specific results: Learning-based methods performed considerably better on Hippocampus MR, where evaluated structures were available during training.Similar patterns appeared for OASIS and Abdomen CT-CT, while dense displacement-correlation methods handled CuRIOUS field-of-view differences best.
- Comparison of Learning- vs Optimisation-based Registration: 20% points of Abdomen CT-CT Dice overlap improvement over previous work were achieved using Dice losses.The three highest-ranked approaches combined deep learning and optimisation in different ways across tasks.
- Comparison of Learning- vs Optimisation-based Registration: GPU-accelerated optimisation-based registration required a few seconds, with CNN feature extraction often outweighing optimisation time.This result challenges the belief that conventional optimisation-based registration must be much slower than learning-based methods.
- Comparison to Baselines: CorrField achieved the best overall scores for CuRIOUS and Lung CT and was second for Abdomen MR-CT, remaining highly competitive without supervision.NiftyReg remained reasonably accurate but fell behind supervised methods on inter-patient tasks, while original VoxelMorph was weaker on the reported OASIS and lung baselines.
- Plausibility of Transformations: More accurate transformations tended to be smoother, supporting regularisation as a way to avoid overfitting and improve robustness.Lung CT included exceptions: Imperial appeared under-regularised, whereas PDD-Net and PIMed may have exchanged accuracy for overly smooth fields.
- Impact and Clinical Adoption: Segmentation-based overlap and keypoints became common driving signals, while hybrid learning-and-optimisation strategies emerged as a major trend.Top-ranking methods also showed high robustness across multi-centric MR studies and lung CT transfer, with runtimes reduced to a few seconds for near-real-time interaction.
VII. CONCLUSION
Learn2Reg established a standardized, accessible benchmark spanning diverse medical image registration tasks and methods. Its results show that robustness depends strongly on the task, although several methods performed robustly across tasks with limited hyperparameter adjustment.
- The challenge evaluated inter- and intra-patient, mono- and multimodal registration tasks across a wide range of methods.
- Over 65 submissions from more than 20 teams were compiled through a low-entry-barrier framework for 3D registration training and validation.
- ConvexAdam, LapIRN, and corrField worked robustly across all tasks with only minor hyperparameter adjustments.
- MEVIS also worked robustly across all tasks, using a deep-learning-based method for the hippocampal tasks.
- Several teams achieved very good results with solutions tailored to individual tasks.