PaperlayerView exact revision on arXiv

Source-linked AI summary

A survey on deep learning in medical image registration: new technologies, uncertainty, evaluation metrics, and beyond

Junyu Chen, Yihao Liu, Shuwen Wei, Zhangxing Bian, Shalini Subramanian, Aaron Carass, Jerry L. Prince, Yong Du

arXiv:2307.15615v4eess.IVcs.CV

TL;DR

Deep learning registration has advanced beyond early regression and U-Net foundations, but the field needs a coherent synthesis of methods, uncertainty, and evaluation. This survey analyzes over 250 papers covering architectures, loss functions, regularization, uncertainty estimation, metrics, applications, and future challenges. It organizes recent technological progress while identifying limitations in uncertainty validation, computational cost, topology assumptions, and temporal registration.

  • Problem

    Medical image registration requires improved methods spanning similarity, regularization, architectures, uncertainty estimation, and evaluation across diverse imaging applications.

  • Method

    The paper surveys over 250 learning-based registration articles, synthesizing loss functions, network architectures, uncertainty estimation, accuracy and regularity metrics, applications, and challenges.

  • Results

    The survey identifies recent advances including Transformer architectures, registration-specific loss functions, uncertainty methods, and applications across atlas, motion, and multimodal registration tasks.

  • Takeaways & Limitations

    The synthesis provides insights into clinical applications, future perspectives, and challenges to guide research in learning-based medical image registration.

  • Takeaways & Limitations

    Uncertainty estimation lacks ground truth for direct validation and can be computationally complex, limiting reliability and real-time clinical use.

Abstract

from arXiv · show

Deep learning technologies have dramatically reshaped the field of medical image registration over the past decade. The initial developments, such as regression-based and U-Net-based networks, established the foundation for deep learning in image registration. Subsequent progress has been made in various aspects of deep learning-based registration, including similarity measures, deformation regularizations, network architectures, and uncertainty estimation. These advancements have not only enriched the field of image registration but have also facilitated its application in a wide range of tasks, including atlas construction, multi-atlas segmentation, motion estimation, and 2D-3D registration. In this paper, we present a comprehensive overview of the most recent advancements in deep learning-based image registration. We begin with a concise introduction to the core concepts of deep learning-based image registration. Then, we delve into innovative network architectures, loss functions specific to registration, and methods for estimating registration uncertainty. Additionally, this paper explores appropriate evaluation metrics for assessing the performance of deep learning models in registration tasks. Finally, we highlight the practical applications of these novel techniques in medical imaging and discuss the future prospects of deep learning-based image registration.

1. Introduction

Medical image registration estimates transformations aligning fixed and moving images, with rigid/affine and deformable forms serving different applications. Deep learning addresses traditional methods’ computational limitations through globally trained networks applied directly at test time, while recent work expands architectures and learning paradigms.

  • Registration estimates a spatial transformation aligning structures in fixed and moving images, using rigid/affine or spatially varying deformable transformations.
  • Traditional registration solves a separate optimization problem for every image pair, making it computationally intensive and slow.
  • Deep learning trains one network on a dataset and applies fixed weights directly to test image pairs without further optimization.
  • Supervised deformable registration commonly uses encoder-decoder networks with ground-truth deformation fields, whereas rigid/affine methods use encoder-only networks outputting transformation parameters.
  • The survey analyzes over 250 articles, emphasizing recent advances in loss functions, architectures, uncertainty estimation, accuracy metrics, applications, and future challenges.

2. Fundamentals of Learning-based Image Registration

Learning-based registration optimizes image alignment through neural networks, spatial transformation, and task-specific losses, with supervised methods using transformation targets and unsupervised methods using image agreement. Diffeomorphic approaches preserve invertibility but introduce computational and implementation challenges, especially for time-varying models.

  • Registration minimizes an energy combining similarity between fixed and transformed moving images with deformation regularization weighted by λ.
  • Supervised vs. Unsupervised Learning: Supervised methods use ground-truth transformations, whereas unsupervised methods learn without them; landmark correspondences and anatomical labels remain supervised or semi-supervised signals.
  • Supervised vs. Unsupervised Learning: Ground-truth transformations generated by traditional registration are time-consuming, motivating unsupervised methods that update networks from differences between warped moving and fixed images.
  • Rigid/affine networks typically output transformation parameters, while deformable networks use encoder-decoder architectures to produce image-sized deformation fields.
  • Diffeomorphic Image Registration: Diffeomorphic registration supports invertible, topology-preserving transformations, while small-deformation models may not guarantee one-to-one mappings.
  • Diffeomorphic Image Registration: Time-varying diffeomorphic models can capture complex, large deformations but require high memory or face numerical difficulty in geodesic shooting and end-to-end training.
  • Diffeomorphic Image Registration: Scaling-and-squaring integrates a velocity field through repeated composition, but interpolation can prevent guaranteed folding-free transformations under finite-difference Jacobian evaluation.

3. Loss Functions

Deep learning registration losses combine image similarity, deformation regularization, and auxiliary anatomical or landmark supervision, with choices depending on supervision type and imaging modality. Recent methods address asymmetric deformation errors, spatially varying regularity, folding, inverse consistency, and limitations caused by incomplete anatomical labels.

  • Supervised learning: Supervised registration commonly uses MSE, equivalent EPE, or MAE to compare predicted and ground-truth transformations.These losses are defined directly when the ground-truth transformation is available.
  • Supervised learning: Combining phase-based ⊥-loss with ℓ2 loss produces symmetric deformation-direction errors and outperforms ℓ2 alone in registration performance.The method represents x and y deformation components as the real and imaginary parts of a complex image, then measures phase error.
  • Unsupervised learning: Unsupervised registration losses combine image similarity with transformation regularization because ground-truth transformations are unavailable.This formulation parallels the energy functions used in traditional registration methods.
  • Similarity measure: Parzen-window mutual information enables differentiable multi-modal similarity loss but is relatively difficult to implement and sensitive to intensity-bin and window-smoothness choices.Standard histogram counting is not differentiable, motivating the Parzen-window formulation for backpropagation.
  • Deformation regularizer: Diffusion, ℓ1, bending-energy, Jacobian-based, and learned spatially varying regularizers constrain deformation smoothness, folding, or regional variation.Jacobian determinants identify locally non-invertible transformations, while learned regularization adapts penalties across image regions.
  • Auxiliary anatomical information: Anatomy loss and landmark supervision can improve registration, but anatomy-only objectives do not penalize deformations where labels are missing or ambiguous.Detailed labels are therefore desirable, although manual delineation is time-consuming and expensive.

4. Network Architectures

Learning-based registration architectures have expanded beyond dominant U-Net-like ConvNets to adversarial, contrastive, and modality-aware designs. These approaches use discriminators, synthesis networks, or learned representations to guide alignment across image modalities and deployment constraints.

  • U-Net-like ConvNet architectures have been central to learning-based medical image registration since the field’s inception.
  • Adversarial Learning: Adversarial learning supports both similarity measurement and multimodal registration through modality synthesis before alignment.
  • Adversarial Learning: Adversarial registration treats the registration network as a generator that produces a deformation field and warped moving image, while a discriminator evaluates alignment.
  • Adversarial Learning: A dual-channel probabilistic CycleGAN framework transforms CT and MR into target modalities, fuses deformation fields using uncertainty weighting, and improves accuracy and runtime over baselines.
  • Adversarial Learning: Adversarial learning can also distill a computationally expensive registration network into a student with one-tenth as many parameters.
  • Contrastive Learning: Contrastive registration uses positive and negative examples to replace or supplement task-specific image similarity measures.

4.2. Contrastive Learning

Contrastive learning compares corresponding and non-corresponding examples to provide registration guidance, particularly when explicit similarity metrics are difficult to optimize across modalities. Applications include multimodal alignment, synthesis, and feature-based matching.

  • Contrastive learning compares positive pairs of similar inputs with negative pairs of dissimilar inputs.
  • In registration, contrastive learning can replace explicit image similarity metrics whose usefulness varies across tasks such as lung CT and brain MRI registration.
  • Hu et al. used keypoint-centered CT and MR patches with PatchNCE loss to maximize mutual information between pre- and post-synthesis images.
  • Hu et al.’s method outperformed mutual-information-based and other synthesis-based registration methods in multimodal brain MRI registration.
  • Dey et al. extracted modality-specific features with pretrained autoencoders and sampled same-location positives alongside different-location negatives for PatchNCE optimization.
  • Contrastive learning was later integrated into shared-weight encoder architectures for mono-modal brain MRI registration.

4.3. Transformers

Transformers address the limited effective receptive fields of ConvNets by modeling longer-range spatial correspondences in registration. Surveyed designs combine Transformers and ConvNets in encoder-decoder, attention-based, and fully Transformer architectures.

  • ConvNets’ limited effective receptive fields can hinder long-range spatial correspondence, a central requirement of image registration.
  • Transformer registration architectures include Transformer encoders with ConvNet decoders, Transformers between ConvNets, and fully Transformer encoder-decoder designs.
  • Transformer models use self-attention or cross-attention to relate image features before a decoder generates the deformation field.
  • TransMorph provided empirical evidence of larger effective receptive fields than baseline ConvNets and improved registration performance across brain MRI and abdomen applications.
  • Cross-attention: Cross-attention models such as Attention-Reg and XMorpher exchange information between multimodal or parallel image features and reported favorable or superior comparative performance.
  • ConvNets retain advantages including input-size invariance and convolutional inductive bias, motivating hybrid architectures that incorporate Transformer concepts.

4.4. Diffusion Models

Diffusion models and continuous-depth formulations extend registration beyond conventional feed-forward deformation prediction. The surveyed approaches generate continuous deformations or model deformation evolution through time, while related implicit representations optimize coordinate-wise displacement fields.

  • Diffusion Models: Diffusion models generate conditional score functions indexed by an imaginary time step to produce continuous deformations from moving to fixed images.
  • Diffusion Models: The diffusion registration approach showed promising results against ConvNet-based methods on cardiac MRI and human facial-expression datasets.
  • Neural ODEs: Neural ODEs interpret progressively updated network states as discretized time steps, approximating continuous-depth transformations through numerical integration.
  • Neural ODEs: MS-ODENet parameterizes the final ODE state as the deformation field and uses coarse-to-fine resolution solving to reduce numerical-solver cost.
  • Neural ODEs: NODEO optimizes image similarity throughout deformation evolution while regularizing flow energy, velocity smoothness, spatial regularity, and Jacobian determinants.
  • Implicit Neural Representations: Implicit neural representations use an MLP to map image-grid coordinates to displacement vectors, which are iteratively refined through pair-wise registration optimization.

4.6. Implicit Neural Representations

Implicit neural representations parameterize deformation fields as coordinate-based neural functions, enabling continuous representations and integration with optimization-based registration. Hybrid approaches combine learned initial deformations with implicit residual refinement.

  • 4.6. Implicit Neural Representations: Implicit neural representations model registration as a coordinate-to-deformation function constrained by image similarity and deformation regularization.The network ψ maps spatial coordinates to deformation values, with the constraint C comprising a similarity measure and deformation regularizer.
  • 4.6. Implicit Neural Representations: MLP-based deformation fields can represent continuous image grids and support continuous differentiation.These properties motivate implicit deformable registration models and pairwise optimization-based registration.
  • 4.6. Implicit Neural Representations: A cascade framework uses a learning-based DNN for an initial deformation and an MLP-based INR for residual refinement.The two-stage design combines learning-based registration with optimization-based implicit representations.
  • 4.6. Implicit Neural Representations: Aligning INR coordinates with the tangent space of anticipated intestinal motion was reported to simplify the representation and improve registration performance.This approach replaces integer-grid coordinates with a coordinate system aligned to dominant intestinal motion.

4.7. Hyperparameter Conditioning

Hyperparameter conditioning incorporates registration hyperparameters into network architectures so one training process can represent multiple settings and accelerate test-time adjustment. The approach commonly targets the deformation-regularizer weight, which controls deformation smoothness.

  • 4.7. Hyperparameter Conditioning: Hyperparameter-conditioned registration networks capture multiple hyperparameter settings within one training process, avoiding separate training from scratch for each setting.This design is intended to speed hyperparameter tuning.
  • 4.7. Hyperparameter Conditioning: The conditioned hyperparameter typically weights the deformation regularizer, thereby affecting the smoothness of the predicted deformation.The same hyperparameter can be adjusted at test time in the illustrated strategy.
  • 4.7. Hyperparameter Conditioning: HyperMorph combines a hypernetwork with a U-Net-like registration network to generate deformation fields conditioned on a diffusion-regularizer hyperparameter.The hypernetwork estimates U-Net weights from the supplied hyperparameter value.
  • 4.7. Hyperparameter Conditioning: HyperMorph obtains the hyperparameter through gradient descent, whereas other cited methods select the best value through grid search.The comparison concerns how the hyperparameter value is selected.

4.8. Anatomy-aware Networks

Anatomy-aware registration networks incorporate anatomical label maps as design inputs rather than using them only in training losses. Nonuniform control points sampled along anatomical contours are intended to improve deformation precision at structure boundaries.

  • 4.8. Anatomy-aware Networks: Anatomical label maps are incorporated into registration model design to provide prior anatomical information beyond their use as loss-function components.The motivation is to enhance registration accuracy.
  • 4.8. Anatomy-aware Networks: Contour-sampled, nonuniform control points enable more precise deformation of anatomical structures than uniformly distributed image-grid points.The method combines intensity features from a ConvNet encoder with spatial features.

4.9. Correlation Layer

Correlation layers guide registration networks by explicitly comparing local feature patches between moving and fixed images within a bounded search region. Their volumetric search space creates substantial computational challenges, motivating sparse or multi-scale designs.

  • 4.9. Correlation Layer: Correlation layers compute feature-patch similarities between a moving-image location and candidate fixed-image locations within a maximum displacement.The output represents explicit correspondence evidence for network-based registration.
  • 4.9. Correlation Layer: The correlation output has spatial feature-map dimensions plus a displacement-search dimension, with size H × W × D × d in the described formulation.Here H × W × D denotes feature-map size and d represents the search range.
  • 4.9. Correlation Layer: Volumetric medical images make correlation expensive because each voxel may search a growing 3D neighborhood as displacement distance increases.This computational burden delayed adoption of explicit correspondence methods in medical registration.
  • 4.9. Correlation Layer: A 3D displacement search produces a 6D correlation matrix whose final three dimensions describe the search space.Sparse control points can reduce computation relative to processing the full image grid.
  • 4.9. Correlation Layer: One cited model combines probabilistic displacement heatmaps with multi-channel instance optimization using one-hot anatomical label-map embeddings.Feature vectors sampled at keypoints are converted into heatmap patches through convolution and softmax operations.
  • 4.9. Correlation Layer: Progressive and multi-scale registration designs organize deformation estimation across stages or resolution fractions.The illustrated framework uses intermediate deformed moving images and resolution-specific deformation fields.

4.10. Progressive and Pyramid Registration

Progressive and pyramid registration decompose alignment into multiple refinement steps or resolutions. These designs improve performance, handle large displacements, and produce smoother deformation fields than single-pass prediction.

  • Progressive registration: Progressive registration cascades subnetworks or decoder blocks so each stage refines the currently warped moving image.The final deformation can be composed from intermediate fields, and the approach has been shown effective for large displacements.
  • Progressive registration: VTN uses cascade registration networks, passing each warped image to the next subnetwork and composing intermediate deformation fields.This design was reported to handle large displacements effectively.
  • Pyramid registration: Multi-scale registration generates deformation fields at multiple resolutions and refines them from coarse to fine.The strategy follows conventional registration methods that use stages with varying resolutions.
  • Combined architectures: A Transformer-based architecture integrates progressive and multi-scale strategies through dual-stream feature extraction and Swin Transformer blocks at the bottleneck scale.The architecture extracts multi-scale features independently from moving and fixed images and targets large deformations at coarser scales.
  • Overall findings: Across these approaches, decomposing registration into multiple refinement steps consistently improves performance and enforces smoother deformation fields than predicting one field all at once.The reported benefit applies to both progressive and multi-scale designs.

5. Uncertainty in Learning-based Registration

Deep learning registration uncertainty is organized around data-driven and model-driven variability, with methods that estimate uncertainty from stochastic deformation or image predictions. The survey reviews Bayesian approximations, sampling strategies, and registration-specific uncertainty measures.

  • Uncertainty types: Registration uncertainty includes aleatoric uncertainty from data variability and epistemic uncertainty from model structure, parameters, and assumptions.Aleatoric uncertainty may decrease with better data, whereas epistemic uncertainty may decrease with improved models or additional information.
  • Bayesian estimation: Deep learning uncertainty estimation samples model weights from an approximate posterior because direct integration over the posterior is usually impractical.Variational inference approximates the posterior, after which sampled outputs estimate uncertainty.
  • Sampling methods: Monte Carlo dropout, bootstrap, and snapshot techniques provide sampling strategies for estimating uncertainty, with Monte Carlo dropout widely used because it is straightforward to implement.Monte Carlo dropout performs multiple test-time forward inferences while retaining dropout layers.
  • Aleatoric uncertainty: For aleatoric uncertainty, registration networks predict deformation-field variance, while variational inference estimates an approximate posterior when ground-truth deformations are unavailable.The variational approach minimizes the KL divergence between the true and approximate deformation posteriors and maximizes the evidence lower bound.
  • Epistemic uncertainty: Epistemic uncertainty in registration can be separated into transformation uncertainty and appearance uncertainty.Transformation uncertainty concerns deformation-space variability, whereas appearance uncertainty concerns the plausibility of the resulting warped images.

6. Registration Evaluation Metrics

Registration evaluation combines accuracy measures, transformation regularity, and benchmark evidence because no single metric fully establishes successful alignment. The survey highlights limitations of surrogate accuracy measures and central-difference Jacobians.

  • Accuracy measures: Manual landmark correspondences support target registration error, while surface distances and overlap metrics provide alternatives when appropriate annotations are available.Mean minimum distance measures average nearest-point surface discrepancies, and Dice and Jaccard quantify label overlap.
  • Accuracy measures: Artificial deformations provide dense correspondences, but performance on them may not reflect real-world performance because artificial and real deformations differ.This limits direct interpretation of results obtained solely from synthetic deformation benchmarks.
  • Accuracy measures: Surrogate intensity and overlap measures can detect inaccurate registrations but may not provide sufficient positive evidence that alignment is accurate.Voxel reordering based only on intensity rankings can outperform registration algorithms on many surrogate measures without geometrical constraints.
  • Regularity measures: Transformation regularity is commonly assessed alongside accuracy because spatially smooth, non-folding transformations are treated as more physically realistic.Regularity alone is insufficient: the identity transformation is perfectly regularized but does not constitute meaningful registration.
  • Regularity measures: Central-difference Jacobian determinants can exhibit checkerboard and self-intersection problems, yielding positive values while underlying transformations fold space.The survey describes non-diffeomorphic volume measurement as a forward-and-backward difference alternative for quantifying folding.

7. Benchmark Datasets for Medical Image Registration

Benchmark datasets support reproducible evaluation of learning-based registration across anatomies with distinct alignment and annotation challenges. The survey summarizes public datasets and notes that automated segmentation can expand evaluation resources.

  • Benchmark resources: Public benchmark datasets are important for evaluating deep learning registration models and supporting reproducibility across empirical studies.The survey lists datasets spanning brain, lung, heart, abdomen, and pathological images.
  • Anatomy-specific challenges: Brain registration requires both fine and large-scale alignment because anatomy includes small structures and disease-related local or global distortions.Examples include the hippocampus, ventricles, cortical sulci, cortical thinning, atrophy, and tumor-related distortions.
  • Anatomy-specific challenges: Lung registration must handle large respiratory deformations and finer vessel or airway changes, but landmark acquisition is time-consuming and labor-intensive.Evaluation is typically landmark-based because lung label maps are often too coarse for precise accuracy measurement.
  • Anatomy-specific challenges: Cardiac registration combines respiratory and cardiac motion with limited anatomical landmarks, complicating both alignment and evaluation.The heart has fewer distinct features than the lungs, reducing available landmark information.
  • Expanding benchmark resources: Automated segmentation tools can convert open-source and large public imaging datasets into additional registration evaluation resources.The survey identifies platforms and datasets including OpenNeuro, the Cancer Imaging Archive, ADNI, ABIDE, and autoPET.

8. Applications of Medical Image Registration

Deep learning-based registration supports atlas construction, segmentation, uncertainty-aware analysis, motion estimation, and 2D-3D alignment, while addressing speed and image-alignment challenges across applications.

  • 8.1. Atlas Construction: Deep learning methods have accelerated construction of high-quality atlases and enabled approaches that model anatomical variation through diffeomorphic deformation.Atlas methods include jointly optimized atlases, demographic-conditioned affine registration, adversarial sharpening, and continuous occupancy-grid representations.
  • 8.1. Atlas Construction: Most learning-based atlas construction methods derive the atlas during training, limiting atlas construction at test time when the network requires a learned atlas input.Group-wise registration can align image sets to an implicit template, but the cited frameworks still depend on training-time atlas representations.
  • 8.2. Multi-atlas Segmentation: Deep deformable registration can reduce the computational burden of pairwise atlas-to-target alignment in multi-atlas segmentation, although relatively few studies integrate it into the full pipeline.VoteNet used Quicksilver to accelerate pairwise registration and later added a registration-refinement step.
  • 8.3. Uncertainty: Registration uncertainty can identify clinically relevant regions and quantify variation in treatment-related dose accumulation, supporting interpretation and clinical decision-making.Uncertainty increased at neurosurgical resection sites, while anatomical changes produced greater dose degradation than algorithm-choice uncertainty in one proton-therapy study, despite substantial inter-algorithm dose variation.
  • 8.4. Motion Estimation: Deep learning has been applied to 4D-CT motion estimation and 2D-3D registration, where specialized models can reduce ambiguity and operate quickly but remain less versatile than traditional optimization.For 2D-3D registration, CNN-based approaches address narrow optimal parameter ranges and are comparably fast; scene-coordinate regression also performed well with partially visible structures.

9. Challenges and Future Perspectives

The survey identifies persistent challenges in learning-based registration, including limited spatially varying regularization, uncertainty validation and computation, domain shift, topology changes, and underexplored multi-frame motion tracking. It also outlines prospective solutions involving improved validation, more accessible uncertainty estimates, zero-shot learning, and broader applications.

  • Training data: Synthetic or random-shape images can train effective registration networks, indicating that actual medical images are not always required for training.This observation reflects the correspondence-focused nature of registration compared with tasks emphasizing semantic information.
  • Regularization: Most deep learning methods use spatially invariant regularization, while spatially varying regularization can accommodate varying deformations, preserve discontinuities, and facilitate sliding motion.Only a few studies address spatially varying regularization within deep learning frameworks.
  • Registration uncertainty: Uncertainty estimation is constrained by missing ground truth, computational complexity, and difficult clinical interpretation.Existing evaluations therefore often rely on indirect approaches such as sparsification analysis.
  • Registration uncertainty: Direct validation using synthetic or simulated data, computationally efficient uncertainty methods, and intuitive visual or statistical displays are proposed to improve uncertainty estimation.The survey mentions low-dimensional deformation representations and multilevel methods as ways to reduce computational workload.
  • Generalizability: Deep learning registration is susceptible to domain shift when test images come from distributions different from the training data.Classical methods remain immediately available and allow users to select similarity measures and regularization weights.
  • Towards zero-shot registration: Zero-shot learning and foundation models are presented as potential routes to improve generalizability without task-specific training data.The survey notes that these approaches could reduce the time and resources needed to develop registration algorithms for clinical pipelines.
  • Metamorphic image registration: Diffeomorphic registration cannot represent topological changes caused by anomalies such as tumors, while metamorphic methods can accommodate such changes but often require manual anomaly segmentation and costly optimization.These requirements limit practical adoption in clinical settings.
  • Future applications: Learning-based registration remains primarily focused on aligning one image pair, leaving tissue-motion tracking across tagged/cine MRI, 4D-CT, and echocardiography underexplored.These modalities require registration across multiple frames rather than only pairwise alignment.

10. Conclusion

This survey examines recent deep learning advances in medical image registration through a comprehensive analysis of more than 250 papers. It covers technical methods, uncertainty and evaluation, clinical applications, and future challenges to guide research in the field.

  • Scope: The survey analyzes over 250 papers with emphasis on recent technological advancements in learning-based medical image registration.Its coverage includes medical imaging journals and major medical-imaging and registration conferences.
  • Coverage: The review covers registration fundamentals, loss functions, network architectures, uncertainty estimation, accuracy and regularity metrics, clinical applications, and future challenges.It is intended to guide future research in this rapidly evolving field.
Loading 2307.15615v4…