PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning in Medical Image Registration: A Review

Yabo Fu, Yang Lei, Tonghe Wang, Walter J. Curran, Tian Liu, Xiaofeng Yang

arXiv:1912.12318v1eess.IVcs.CVcs.LGphysics.med-phstat.ML

TL;DR

Medical image registration faces modality, anatomical, temporal, and efficiency challenges. This review organizes deep-learning methods into seven categories and reports growing publication interest in supervised and unsupervised transformation prediction, with direct methods reaching accuracy comparable to leading traditional deformable registration methods.

  • Problem

    Medical image registration must address modality differences, anatomical variation, temporal appearance changes, and computational-efficiency demands.

  • Method

    The review synthesizes deep-learning registration developments, classifies methods into seven categories, and analyzes contributions, challenges, trends, and publication statistics.

  • Results

    Publication interest increased most clearly for supervised and unsupervised transformation prediction, while direct methods achieved accuracy comparable to leading traditional deformable registration methods.

  • Takeaways & Limitations

    As direct transformation prediction improves, the advantage of framing registration as sequential classification in reinforcement learning is diminishing.

  • Takeaways & Limitations

    Deep similarity methods face nonsmooth optimization derivatives and difficult training-data requirements, while reinforcement-learning methods are constrained mainly to low-dimensional transformations.

Abstract

from arXiv · show

This paper presents a review of deep learning (DL) based medical image registration methods. We summarized the latest developments and applications of DL-based registration methods in the medical field. These methods were classified into seven categories according to their methods, functions and popularity. A detailed review of each category was presented, highlighting important contributions and identifying specific challenges. A short assessment was presented following the detailed review of each category to summarize its achievements and future potentials. We provided a comprehensive comparison among DL-based methods for lung and brain deformable registration using benchmark datasets. Lastly, we analyzed the statistics of all the cited works from various aspects, revealing the popularity and future trend of development in medical image registration using deep learning.

1 Introduction

Medical image registration aligns images by finding spatial transformations that match anatomical structures and supports diverse clinical applications. This review summarizes deep-learning developments, contributions, challenges, future trends, and publication statistics.

  • Foundations: Medical image registration aligns two or more images by estimating a spatial transformation that best matches their underlying anatomical structures.It is also called image fusion or image matching.
  • Clinical applications: Registration supports clinical applications including image guidance, motion tracking, segmentation, dose accumulation, and image reconstruction.
  • Challenges: Multi-modal, inter-patient, and different-day intra-patient registration remain challenging because of modality differences, anatomical variation, and changing image appearance.
  • Review scope: The review summarizes the latest developments in deep-learning-based medical image registration while highlighting contributions, identifying challenges, and outlining future trends.
  • Review scope: The review provides detailed statistics on recent publications from different perspectives.

2 Deep Learning

Deep learning architectures support distinct learning and signal-processing roles: CNNs extract visual features, autoencoders learn input reconstruction, RNNs model temporal dependencies, reinforcement learning selects actions, and GANs generate realistic data.

  • Convolutional Neural Networks: CNNs replace general matrix multiplication with convolution operations, making them suitable for visual imagery signal processing and image analysis.Their feature extraction ability has made CNNs successful models for image analysis.
  • Autoencoders: Autoencoders unsupervisedly learn to copy inputs to outputs through an encoder latent space and decoder reconstruction.Regularization helps prevent the autoencoder from learning an identity function.
  • Recurrent Neural Networks: RNNs model dynamic temporal behavior by using internal state to memorize processed information, making outputs dependent on input history.Unlike feedforward networks such as CNNs, RNNs are suitable for temporal signals; LSTM is one RNN type.
  • Reinforcement Learning: Reinforcement learning predicts actions from an environment’s current state and trains an agent to maximize cumulative expected rewards.Its training involves an exploration–exploitation tradeoff within a Markov decision process.
  • Generative Adversarial Networks: GANs use competing generator and discriminator networks, with the discriminator penalizing unrealistic predictions to encourage realistic artificial data.The generator approximates a target data distribution from a low-dimensional latent space, while the discriminator distinguishes artificial from actual data.

3 Deep learning in medical image registration

This section organizes deep learning-based medical image registration into seven categories and reviews their components, applications, achievements, challenges, and future potential. It highlights growing interest in transformation prediction and GAN methods, while noting limitations of deep similarity metrics and reinforcement-learning approaches.

  • Overview: DL-based medical image registration methods are classified into seven categories according to architectures, training processes, inference types, input sizes, and output types.Examples include CNN, RL, and GAN architectures; supervised or unsupervised training; iterative or one-shot inference; patch-based or whole-image inputs; and dense, sparse, or parametric transformation outputs.
  • Overview: Publication trends show increasing interest in supervised and unsupervised transformation prediction, while GAN methods gain popularity and RL-based registration declines in 2019.The review uses “DeepSimilarity” for deep learning similarity measures in traditional registration frameworks and “RegValidation” for deep learning-based registration validation.
  • Deep similarity-based methods: Deep similarity metrics can outperform traditional metrics, but smooth derivatives, computational demands, and the difficulty of obtaining well-aligned training pairs remain challenges.Conventional metrics generally work well for unimodal registration but can be affected by noise and artifacts in modalities such as ultrasound and cone beam CT.
  • Reinforcement learning: Reinforcement-learning registration methods reduce iterative registration to sequential decisions but are constrained by low-dimensional transformation models and therefore predominantly use rigid transformations.Non-rigid applications based on low-dimensional statistical deformation models may inadequately represent actual deformations.
  • Reinforcement learning: Direct transformation prediction methods are improving in accuracy and can achieve performance comparable to leading traditional deformable image-registration methods, reducing the relative advantage of RL classification formulations.The review attributes this change to advancing techniques and the increasing availability of direct transformation prediction methods.

A. Random transformation generation

Randomly transformed images were used to train CNNs for rigid fetal-brain registration and deformable lung CT registration. These approaches predicted transformation parameters or deformation fields, but performance could remain inferior to traditional DIR and fail to generalize from simulated to real CT-US pairs.

  • A. Random transformation generation: CNNs trained on randomly rotated and translated 3D images predicted rotation and translation for fetal-brain rigid registration.Both MSE and geodesic distance were used to calculate the loss.
  • A. Random transformation generation: Simulated CT-US registration did not work on real CT-US pairs because of large appearance differences between simulated and real US images.Deleting dropout layers, simplifying the network, parameter regularization, and weight decay did not counteract overfitting.
  • A. Random transformation generation: CNN training with synthetic random transformations produced a deformation vector field for 3D-CT lung deformable image registration.The output was defined on a thin plate spline transform grid, with MSE between predicted and ground-truth DVFs as the loss.
  • A. Random transformation generation: 4.02±3.08 mm TRE was achieved on DIRLAB, compared with 1.36+1.01 mm for a traditional DIR method.The reported result was substantially worse than the traditional DIR baseline.

B. Traditional registration-generated transformations

Traditional registration methods were used to generate ground-truth transformations for training deep networks. This approach aimed to avoid performance degradation caused by randomly generated transformations that differ from true transformations.

  • B. Traditional registration-generated transformations: Traditional registration methods generated ground-truth transformations from image pairs for network training.The rationale was that random transformations might differ substantially from the true transformation and thereby deteriorate network performance.
  • B. Traditional registration-generated transformations: Sentker et al. used displacement vector fields generated by traditional deformable image-registration methods as ground truth.The methods included Plasti-Match, NiftyReg, and VarReg.

C. Model-based transformation generation

Model-based transformation generation uses statistical appearance models to synthesize large, diverse training pairs with known transformations from few images. In one application, FlowNet registered 2D MR cardiac images using these generated datasets.

  • C. Model-based transformation generation: Statistical appearance models generated large and diverse training image pairs with known transformations from a few sample images.The synthesized datasets were intended to provide realistic ground-truth training data.
  • C. Model-based transformation generation: Assuming Gaussian appearance parameters, the method synthesized huge amounts of realistic ground-truth training datasets.
  • C. Model-based transformation generation: FlowNet registered 2D MR cardiac images using the synthesized training data.

A. Weak supervision

Weakly supervised transformation prediction replaces artificially generated transformations with higher-level anatomical correspondence information. Hu et al. used anatomical organ labels to train a CNN for deformable registration, arguing that these labels are more reliable and practical to obtain.

  • A. Weak supervision: Weakly supervised methods use higher-level correspondence information, such as anatomical organ labels, instead of artificially generated transformations.This approach was proposed for network training in deformable registration.
  • A. Weak supervision: Hu et al. trained a CNN using anatomical organ labels for weakly supervised deformable transformation prediction.The passage identifies this as an alternative to direct transformation prediction trained with ground truth transformations.
  • A. Weak supervision: The authors argued that anatomical labels are more reliable and practical to obtain than artificially generated transformations.This rationale motivates using anatomical correspondence information for weak supervision.

B. Dual supervision · 3.4 Unsupervised transformation prediction

The paper describes dual supervision as training with two important loss types and reviews unsupervised transformation prediction as a way to reduce dependence on datasets with known transformations. Unsupervised methods show promising registration accuracy, but challenges remain in loss design, multimodality, and validation of hybrid supervision.

  • B. Dual supervision: Dual supervision typically trains a network with two important loss-function types, such as MR-MR and CT-CT losses.Cao et al. converted multimodality MR-CT registration into unimodality registration using pre-aligned counterpart images.
  • 3.3.2 Assessment: Supervised CNN transformation prediction has attracted increasing research interest and substantially improved registration accuracy, although it has not surpassed traditional DIR methods.Some supervised approaches achieve registration accuracy comparable to traditional methods.
  • 3.3.2 Assessment: Artificially generated training pairs may differ from inference pairs, motivating transformation models, data augmentation, and domain adaptation to reduce this gap.The passage also notes that image registration is ill-posed and ground-truth transformation generation remains a limitation.
  • 3.4 Unsupervised transformation prediction: Unsupervised registration aims to overcome the lack of training datasets with known transformations, but defining a proper loss without ground-truth transformations is difficult.Spatial transformer networks provide differentiable in-network spatial manipulation for unsupervised estimation.
  • 3.4.1 Overview of works: Unsupervised methods combine differentiable spatial transformation with learned similarity, including feature-based similarity and CNN architectures such as VoxelMorph.Feature-based similarity was reported as better than intensity-based similarity for DIR in 2D electron microscopy registration.
  • 3.4.2 Assessment: Unsupervised methods alleviate the shortage of training datasets, use regularization to encourage plausible transformations, and can achieve comparable or better TRE on DIRLAB 3D-CT lung DIR.Several groups reported comparable or better results in terms of TRE on the DIRLAB 3D-CT lung benchmark.
  • 3.4.2 Assessment: Most unsupervised methods focus on unimodality registration, while multimodality research remains limited and hybrid supervised–unsupervised approaches require further validation.The review expects continued interest in unsupervised registration because of its promising results.

3.5 GAN in medical image registration

GAN-based methods in medical image registration are used for learned transformation regularization and cross-domain image mapping. They complement traditional regularization and can recast challenging multi-modal registration as unimodal registration.

  • GAN-based registration generally serves two roles: additional regularization of predicted transformations and cross-domain image mapping.
  • Because registration is ill-posed, GAN-based regularization aims to encourage plausible transformations and prevent unrealistic tissue folding.Traditional constraints include DVF smoothness, anti-folding, and inverse consistency, but their adequacy remains ambiguous.
  • Adversarial losses provide learned network-based regularization that complements traditional handcrafted regularization terms.
  • GAN-based image domain translation casts challenging multi-modal registration as unimodal registration, enabling application of existing unimodal algorithms.Multi-modal direct transformation prediction benefits less from deep learning than unimodal registration because of vast appearance differences between modalities.

3.6 Registration validation using deep learning · 3.7 Other learning-based methods in medical image registration

The paper reviews deep learning methods for validating registration accuracy, emphasizing learned error estimation as an alternative to manual landmarks, and surveys other learning-based registration strategies across multimodal synthesis, optimization, detection, and domain adaptation.

  • 3.6 Registration validation using deep learning: Image similarity metrics assess whole-image alignment, whereas manual landmark pairs provide local accuracy but are time-consuming, subjective, and error-prone.
  • 3.6 Registration validation using deep learning: Eppenhof et al. used synthetic transformations as ground truth to train a network that estimates registration error with sub-voxel accuracy.The ground-truth error map was the L2 difference between ground-truth and predicted transformations.
  • 3.6 Registration validation using deep learning: Galib et al. predicted an overall registration error index based on the ratio between good-alignment sub-volum...
  • 3.6 Registration validation using deep learning: In 2019, deep learning-based registration evaluation papers increased significantly, with most methods treating error prediction as supervised regression.Existing methods focused largely on lung registration because DIRLAB provides 3D CT lung benchmark datasets with manual landmark pairs.
  • 3.7 Other learning-based methods in medical image registration: Jiang et al. inferred expressive sparse multi-grid configurations with a CNN before B-spline coefficient optimization.
  • 3.7 Other learning-based methods in medical image registration: Liu et al. used a ten-layer FCN to synthesize unimodal images from T1-weighted, T2-weighted, and proton density images before Elastix registration.Their Elastix-based registration used SSD and outperformed the MI similarity index on brain phantom and IXI datasets.
  • 3.7 Other learning-based methods in medical image registration: Yu et al. used FasterRCNN for vertebrae bounding-box detection and matched detected boxes to doctor-annotated boxes on X-ray images.
  • 3.7 Other learning-based methods in medical image registration: Zheng et al. introduced domain adaptation to reduce variance between synthetic and real data using a few paired real and synthetic examples.The trained adaptation module could be plugged into the network to transfer real features toward synthetic features.

4 Benchmark

The benchmark section emphasizes comparing registration accuracy and computational time, while noting that published work reports accuracy more often than speed. It focuses on lung and brain registration, summarizing lung accuracies and brain datasets and evaluation metrics.

  • Benchmarking rationale: Benchmarking compares registration accuracy and computational time, but reported studies emphasize accuracy because speed depends heavily on hardware.Hardware differences across groups make computational-speed comparisons less consistent.
  • Benchmark scope: Brain and lung are the two most common registration regions in the cited literature, motivating separate benchmark summaries.The review summarizes brain registration datasets and lung registration accuracies.
  • Lung registration benchmark: DIRLAB provides 300 manually selected landmark pairs across end-exhalation and end-inhalation phases for 4D-CT lung-registration benchmarking.Table 8 lists TRE values for three leading traditional methods and seven deep-learning lung-registration methods.
  • Brain registration benchmark: Brain-registration benchmarking is hindered by inconsistent dataset, training/testing, and validation-metric choices across studies.Table 9 organizes datasets, transformation models, and evaluation metrics; DSC across multiple regions of interest is most common.

5 Statistics

The cited literature predominantly targets direct transformation prediction, while deep similarity-based and GAN methods represent smaller shares. Across other attributes, 3D-3D registration is the most common image-pair configuration.

  • 5 Statistics: More than half of the 150+ cited papers target direct transformation prediction through supervised or unsupervised approaches.The review analyzed the overall distribution after examining each method category.
  • 5 Statistics: Deep similarity-based methods account for 14% of all methods, while GAN methods account for 10%.
  • 5 Statistics: 60% of the cited works address 3D-3D registration problems.The review also examined distributions of image-pair dimensions, transformation models, image domains, patch-based training, deep-learning frameworks, and regions of interest.

6 Discussion

Deep learning-based medical image registration is a rapidly developing field with over 150 reviewed papers, organized into seven non-exclusive categories. The discussion highlights tradeoffs in registration strategies, training data, and image-scale choices, while noting that direct prediction remains fast but has not consistently surpassed traditional deformable registration.

  • 6 Discussion: Over 150 papers were collected, most published in the last 3 to 4 years, and methods were classified into seven non-exclusive categories.Methods may belong to multiple categories, such as GANs combined with supervised or unsupervised transformation prediction.
  • 6 Discussion: For lung registration, the best-performing deep learning methods were only comparable to state-of-the-art traditional deformable registration methods in terms of TRE.Deep similarity metrics reportedly outperform handcrafted intensity-based similarity metrics in iterative methods, whereas non-iterative deep learning methods have yet to outperform traditional DIR methods.
  • 6 Discussion: Whole-image training preserves a large inception field for large deformations but may require down-sampling that causes information loss and limits registration accuracy.Patch-based and whole-image training each have advantages and disadvantages, partly because of limited GPU memory and local-maxima concerns.
  • 6 Discussion: Supervised methods commonly lack datasets with known transformations, while augmentation can introduce unrealistic-transformation bias and image domain shifts.The passage also notes demonstrations of trained-network generality on different datasets, though the supplied text is truncated.
  • 6 Discussion: Statistics show a clear trend toward direct transformation prediction for fast registration, with supervised and unsupervised transformation-prediction methods studied in nearly equal numbers.The authors speculate that future work will increasingly combine supervised and unsupervised methods.
Loading 1912.12318v1…