Source-linked AI summary
Deep Learning in Medical Image Registration: A Survey
Grant Haskins, Uwe Kruger, Pingkun Yan
TL;DR
Medical image registration is clinically important but difficult, motivating deep learning approaches and a comprehensive synthesis of their development. The survey organizes registration methods and research challenges, including similarity learning, transformation estimation, and training without reliable ground truth. It reports successful applications while identifying unresolved limitations and future directions.
Problem
Medical image registration is challenging, and reliable ground-truth transformations are difficult to acquire, limiting some registration frameworks.
Method
The paper comprehensively surveys deep learning based medical image registration methods, challenges, and future research directions.
Results
Deep learning succeeds in challenging registration tasks, while learned similarity metrics may best complement existing metrics for unimodal registration and iterative methods remain difficult to use in real time.
Takeaways & Limitations
The field includes diverse approaches spanning learned similarity metrics, reinforcement learning, supervised one-step estimation, and unsupervised transformation estimation.
Takeaways & Limitations
Reinforcement-learning-based registration faces challenges handling high-resolution deformation fields.
Abstract
from arXiv · showhide
The establishment of image correspondence through robust image registration is critical to many clinical tasks such as image fusion, organ atlas creation, and tumor growth monitoring, and is a very challenging problem. Since the beginning of the recent deep learning renaissance, the medical imaging research community has developed deep learning based approaches and achieved the state-of-the-art in many applications, including image registration. The rapid adoption of deep learning for image registration applications over the past few years necessitates a comprehensive summary and outlook, which is the main scope of this survey. This requires placing a focus on the different research areas as well as highlighting challenges that practitioners face. This survey, therefore, outlines the evolution of deep learning based medical image registration in the context of both research challenges and relevant innovations in the past few years. Further, this survey highlights future research directions to show how this field may be possibly moved forward to the next level.
1 INTRODUCTION
Medical image registration aligns datasets into a shared coordinate system and supports important clinical applications, but manual and automatic registration remain challenging. This survey organizes rapidly expanding deep learning registration approaches, including similarity learning, reinforcement learning, and transformation estimation, while highlighting challenges and future directions.
- Motivation: Image registration transforms different datasets into one coordinate system with matched imaging contents for clinical applications.Registration may be needed for images acquired from different viewpoints, times, or sensors/modalities.
- Survey scope: The survey categorizes deep learning registration methods by approach type and tracks growing publication activity in the field.Figure 1 presents approach categories, while Figure 2 compares publication trends for deep learning registration and medical imaging.
- Motivation: Manual registration quality depends heavily on clinician expertise, motivating automatic approaches and deep learning research.The paper describes manual registration as common until recently and automatic registration as a response to its shortcomings.
- Research evolution: Deep learning registration evolved from augmenting iterative intensity-based methods to reinforcement learning and one-step transformation estimation.Unsupervised one-step frameworks emerged partly because obtaining or generating ground-truth transformations is difficult.
- Survey scope: The article surveys the field comprehensively, emphasizing practitioner challenges and future research directions.Its deep learning overview covers neural network types, architectures, and training paradigms and strategies.
2 Deep Iterative Registration
Deep iterative registration methods use learned similarity metrics within classical optimization frameworks or reinforcement-learning agents to estimate transformations. These approaches show particular promise for multimodal registration, while iterative methods remain difficult to use for real-time applications and reinforcement learning faces challenges with high-resolution deformation fields.
- Deep similarity based registration: Deep similarity methods insert learned metrics into classical registration frameworks with specified interpolation, transformation, and optimization components.Training-only and training/testing data flows are distinguished in the surveyed framework.
- Unimodal registration: In unimodal registration, CNN descriptors may complement rather than replace handcrafted descriptors.For lung CT registration, combined CNN-based and MRF-based descriptors achieved optimal performance, although handcrafted descriptors alone outperformed CNN descriptors.
- Multimodal registration: Deep learning offers clearer advantages for multimodal registration, where handcrafted similarity metrics have had limited success.Learned metrics outperformed MI-based registration for multimodal brain MR and US/MR applications.
- Discussion and assessment: Iterative deep similarity methods successfully assess multimodal image similarity but are difficult to use for real-time registration.Neural networks have been shown to assess similarity in challenging multimodal medical image registration tasks.
- Reinforcement learning based registration: Reinforcement-learning registration replaces a predefined optimizer with a trained agent that maps states to actions using rewards.Surveyed applications include rigid and deformable registration, with reported successes against several conventional and learned baselines.
- Reinforcement learning based registration: A principal reinforcement-learning challenge is handling high-resolution deformation fields, whereas rigid registration does not face this challenge.The surveyed deformable approach restricts action-space dimensionality through a low-resolution deformation model.
3 Supervised Transformation Estimation
Supervised transformation estimation emerged to avoid slow iterative registration by predicting transformations directly in one step. Fully supervised methods improve speed but require ground-truth data, motivating dual- and weakly supervised alternatives.
- Iterative transformation estimation can be slow, especially for deformable registration with high-dimensional solution spaces.
- Fully supervised one-step methods define their loss exclusively with ground-truth data, but this framework has several challenges.
- The survey organizes supervised transformation estimation into fully supervised and dual/weakly supervised approaches.
3.1 Fully Supervised Transformation Estimation
Fully supervised methods predict rigid or deformable transformations directly, often achieving robust and fast registration. Their main constraints are dependence on expert-labeled registrations and the difficulty of generating realistic, diverse training data.
- Overview: Neural networks replace iterative optimization with a forward pass, significantly speeding registration.Fully convolutional networks avoid additional computational constraints from high-dimensional deformation-field solution spaces.
- Rigid Registration: CNNs predicted rigid transformation parameters for 2D/3D X-ray and brain MR registration, outperforming optimization-based baselines in reported accuracy or efficiency.Miao et al. synthesized ground truth by transforming aligned data; AIRNet also outperformed mutual-information optimization in unimodal and multimodal cases.
- Rigid Registration: Residual and correction networks, geodesic losses, and domain adaptation were used for rigid registration across fetal brain MR and X-ray settings.The residual network increased capture range, while domain adaptation addressed discrepancies between synthetic training data and real data.
- Deformable Registration: Fully supervised deformable methods used U-Net-like or convolutional architectures to predict deformation fields for brain, cardiac, abdominal, chest, lung, and other medical images.Approaches included diffeomorphic modeling, patch-based displacement prediction, multi-scale networks, motion compensation, and uncertainty estimation.
- Deformable Registration: Synthetic transformations can reduce the need for manual annotations and allow supervised training with relatively few medical images, but they only modestly expand dataset size and diversity.More sophisticated ground-truth generation, including statistical appearance models, was developed to address these limitations.
- Discussion and Assessment: Registration quality depends on the quality of expert ground-truth labels, which are difficult to obtain because few practitioners have the necessary expertise.Synthetic data can address label scarcity, but it must remain sufficiently similar to clinical data.
3.2 Dual/Weakly Supervised Transformation Estimation
Dual and weak supervision reduce reliance on exclusive ground-truth transformation labels by incorporating image similarity or anatomical label overlap. These approaches still require annotated data, while weak supervision supports similarity assessment in multimodal registration.
- Overview: Dual supervision combines ground-truth data with an image-similarity metric, whereas weak supervision uses overlap between corresponding anatomical segmentations.
- Dual Supervision: Hierarchical dual-supervised learning predicted deformable fields using both transformation similarity and similarity between warped and fixed images.Fan et al. modified U-Net with gap-filling layers and coarse-to-fine guidance.
- Dual Supervision: An adversarial framework trained a generator to estimate rigid transformations while a discriminator learned to distinguish ground-truth-aligned from predicted-aligned images.Euclidean distance to ground truth and adversarial loss jointly formed the training objective.
- Weak Supervision: Weakly supervised MR-TRUS registration used label similarity with separate networks for global affine transformation and local dense deformation.
- Discussion and Assessment: Partially and weakly supervised methods reduce reliance on trustworthy ground-truth transformations but still require manually annotated data such as segmentations.Partial supervision also permits aggregation of methods for assessing predicted-registration quality, supporting growing interest in these approaches.
4 Unsupervised Transformation Estimation
Unsupervised registration addresses the difficulty of acquiring reliable ground truth by training networks with image-similarity metrics or learned feature representations. Spatial transformer networks are a key innovation used to perform registration deformations.
- Reliable ground-truth acquisition remains a significant hindrance, motivating unsupervised registration approaches.
- Unsupervised methods train networks using image-similarity metrics or feature representations of image data.
- Spatial transformer networks are used by several approaches to perform the deformations required for registration.
4.1 Similarity Metric based Unsupervised Transformation Estimation
Similarity-metric-based unsupervised registration trains networks without ground-truth transformations by optimizing image similarity and regularization. These methods report successes across unimodal applications, but multimodal registration remains difficult because manually defined similarity metrics often fail.
- Similarity-metric formulation: Unsupervised methods use image-similarity metrics, such as NCC, NMI, SSD, or MSE, together with regularization to train transformation-predicting networks.The standard setup estimates transformations by maximizing similarity between warped moving and fixed images while constraining deformation behavior.
- Empirical results: Similarity-metric approaches have outperformed established registration tools and competing deep-learning methods in lung, brain, cardiac, neuron, and other unimodal applications.Reported comparisons include ANTs, Elastix, log-demons, Demons, and SIFT flow baselines.
- Transformation estimation: Several methods estimate affine and deformable transformations jointly or sequentially, including diffeomorphic, coarse-to-fine, multistage, and B-Spline formulations.Examples include variational-inference-based velocity integration, affine prediction followed by deformation, and multiscale transformation estimation.
- Advantages: Unsupervised similarity-based registration avoids expert labels, so model performance does not depend on practitioner expertise.This advantage motivates the substantial attention received by these methods.
- Limitations: Multimodal registration remains the principal limitation because image similarity is difficult to quantify across modalities, leaving most unsupervised similarity-based work confined to unimodal cases.The survey identifies multimodal clinical registration as an important direction for future work.
4.2 Feature based Unsupervised Transformation Estimation
Feature-based unsupervised methods learn representations that support transformation estimation without ground-truth data. They cover unimodal and multimodal applications, but multimodal correspondence remains substantially more difficult.
- Feature-based formulation: Feature-based methods learn image representations and use them to estimate transformations without requiring ground-truth registration data.The surveyed approaches include feature extraction, latent-space learning, and transformation regression.
- Unimodal applications: An autoencoder-based spatial transformer registers serial-section electron microscopy images using reconstruction distance and regularization.The method outperforms bUnwarpJ and Elastic registration techniques.
- Multimodal applications: A tensor-based MIND method with PCANet supports both unimodal thoracic CT and multimodal brain MR registration, outperforming MI-, residual-complexity-, and original-MIND-based methods.Validation includes inhale-exhale thoracic CT pairs and multimodal brain MR images.
- Latent-space approaches: Stochastic latent-space learning can bypass spatial regularization by constraining transformation parameters through a prescribed probability distribution.Krebs et al. use a conditional variational autoencoder and latent-distribution regularization for brain and cardiac MR registration.
- Limitations: Unsupervised multimodal registration remains harder than unimodal registration because handcrafted metrics struggle to establish voxel-to-voxel correspondence across modalities.Feature learning is presented as a way to determine transformations, but the underlying multimodal challenge remains significant.
5 Research Trends and Future Directions
The survey identifies increasing interest in unsupervised transformation estimation and anticipates broader use of GANs, reinforcement learning, and reconstruction-integrated registration. These directions target similarity, deformation realism, multimodal appearance differences, and registration efficiency.
- Research trends: The field is rapidly growing, with unsupervised transformation estimation attracting increasing research attention.The survey frames these trends as motivating future research directions.
- Generative adversarial methods: GANs may serve as learned similarity metrics, enforce realistic transformations, or translate multimodal images into a common appearance domain.These roles address similarity measurement, deformation plausibility, and multimodal registration setup.
- Generative adversarial methods: Adversarial registration methods can be limited because discriminators require pre-aligned pairs and assign all misaligned pairs the same label.The survey states that this may restrict performance in multimodal or challenging unimodal applications and hinder modeling of misalignment severity.
- Generative adversarial methods: GAN-like deformation models are proposed to produce realistic deformations without relying solely on L2 penalties that may limit predicted deformation magnitude.The motivation is to avoid unrealistic organ appearances while reducing the restrictive effect of conventional deformation regularization.
- Reinforcement learning: Reinforcement learning is expected to receive more attention because it can mimic physician registration behavior, although deformable registration creates a high-dimensional action space.The survey considers this challenge potentially surmountable.
- Reconstruction-integrated registration: Future pipelines may integrate image reconstruction directly into end-to-end deep-learning registration, including mappings from raw data to reconstructed images.This direction extends registration beyond workflows operating only on reconstructed images.
6 Conclusion
Deep-learning medical image registration requires application-specific framework design because clinical tasks pose distinct challenges. A recurring field-wide challenge is the lack of robust similarity metrics for multimodal images with substantial appearance differences or different fields of view.
- Conclusion: Deep-learning registration frameworks must be carefully designed for each application because applications present unique challenges.The conclusion identifies application-specific design as a central consideration across surveyed methods.
- Conclusion: A recurring challenge is developing robust similarity metrics for multimodal registration when image appearance differs substantially or fields of view differ.This limitation is presented as common across many deep-learning medical image-registration applications.