Source-linked AI summary
Medical Image Registration Using Deep Neural Networks: A Comprehensive Review
Hamid Reza Boveiri, Raouf Khayami, Reza Javidan, Ali Reza MehdiZadeh
TL;DR
Medical image registration is central to image-guided intervention, but conventional optimization is slow and can struggle with multimodal similarity measures. This paper systematically reviews deep-neural-network registration methods, their evidence base, challenges, and directions, finding a strong emphasis on runtime reduction while preserving performance, including a reported registration time below 50 ms for a typical image pair.
Problem
Medical registration must support image-guided intervention despite prolonged conventional runtimes and similarity measures that can become trapped in local minima, especially for multimodal images.
Method
The paper presents a systematic review of deep-neural-network medical image registration literature, categorizing approaches and analyzing datasets, modalities, techniques, metrics, challenges, and future directions.
Results
The reviewed approaches primarily target lower registration runtime while maintaining performance; one reported method registers a typical image pair in less than 50 ms.
Takeaways & Limitations
The review identifies unsupervised and weakly supervised approaches as less dependent on labeled data and highlights multiresolution policies, transfer learning, geometric theory, and spatial transformer networks as promising directions.
Takeaways & Limitations
Medical datasets are often small, and network inference remains difficult to interpret for physicians despite high precision and accuracy.
Abstract
from arXiv · showhide
Image-guided interventions are saving the lives of a large number of patients where the image registration problem should indeed be considered as the most complex and complicated issue to be tackled. On the other hand, the recently huge progress in the field of machine learning made by the possibility of implementing deep neural networks on the contemporary many-core GPUs opened up a promising window to challenge with many medical applications, where the registration is not an exception. In this paper, a comprehensive review on the state-of-the-art literature known as medical image registration using deep neural networks is presented. The review is systematic and encompasses all the related works previously published in the field. Key concepts, statistical analysis from different points of view, confiding challenges, novelties and main contributions, key-enabling techniques, future directions and prospective trends all are discussed and surveyed in details in this comprehensive review. This review allows a deep understanding and insight for the readers active in the field who are investigating the state-of-the-art and seeking to contribute the future literature.
1. Introduction
Medical image registration is essential for aligning heterogeneous clinical images during image-guided interventions, but conventional optimization is slow and vulnerable to local optima. This motivates a systematic review of deep-neural-network approaches and their future directions.
- Image registration aligns images captured at different times, spatial settings, dimensions, or modalities to support clinical decision-making and image-guided interventions.The paper highlights telesurgery, radiotherapy, and precision medicine as applications requiring registration.
- Conventional registration iteratively selects features, evaluates similarity, updates transformation parameters, and reapplies the transformation until termination.Its output may be transformation parameters or a final interpolated fused image.
- Tens of minutes are common for deformable registration, while multimodal similarity measures can cause premature convergence or stagnation near local optima.These limitations conflict with the real-time requirements of clinical operations.
- Learning-based registration gained popularity because deep neural networks offer a potential route around conventional registration’s runtime and similarity-measure limitations.The paper notes that the field remained relatively limited despite promising potential.
- The paper systematically reviews deep-neural-network registration literature, statistical patterns, seminal works, challenges, open problems, and prospective directions.The review covers work from 2013 through 2019 and organizes the field from multiple perspectives.
2. Reference Gathering Methodology
The review gathered literature through database searches, keyword expansion, and validation searches, while addressing the difficulty of finding studies with inconsistent terminology. The authors also faced limited review tooling and could not conduct an experimental benchmark review.
- The primary search used SCOPUS and PubMed with “Medical Image Registration” combined with deep-learning, deep-neural-network, or convolutional terms.Google Scholar and CrossRef searches were used to validate comprehensiveness.
- The authors expanded the search because relevant conference papers and studies used abbreviations or alternative terms such as image correspondence and pose estimation.Examples included papers beginning with CNN, SAEs, or GAN without defining those abbreviations.
- The review faced four determinative challenges in constructing a comprehensive evidence base.The provided passage introduces these challenges, while the later discussion specifies concerns about overlooked work, tooling, and benchmarking.
- The authors developed a desktop application with embedded SQL because no suitable application supported their systematic-review statistical analysis.They also reported being unable to conduct an experimental benchmark review.
- The collected references were organized with techniques, datasets, organs, modalities, and evaluation metrics, as represented in Table 1.The supplied table materials include registration methods, datasets, and metrics across the reviewed studies.
3. Taxonomy of Image Registration
Medical image registration is classified across dimensions, modalities, feature sources, transformation domains, interaction levels, and parameter-search strategies. The taxonomy also distinguishes rigid, affine, and deformable transformations and intrinsic from extrinsic features.
- Registration methods are classified by image dimension, modality, feature source, transformation domain, transformation model, fusion type, user interaction, and parameter investigation.The taxonomy includes 2D, 3D, and 4D images; intrinsic and extrinsic features; and iterative or direct parameter investigation.
- Medical registration commonly involves 3D images, while sequences of interventional images can require 4D transformations.Operational 4D images are often low-quality and noisy, which challenges registration.
- Multimodal registration is more difficult because modalities respond differently to tissue properties, and SSD cannot generally be applied across differing intensity ranges.T1- and T2-weighted MRI illustrate multimodal inputs.
- Intrinsic registration uses anatomical structures, whereas extrinsic registration uses implanted objects that can provide simple, accurate, and fast transformation estimates.Extrinsic methods are especially associated with implanted markers or sensors.
- Rigid, affine, and deformable models differ in the transformations they represent, with deformable registration using control-point meshes to capture local deformation.Rigid models use translation and rotation; affine models additionally include scaling and shearing.
- Manual, semi-automated, and fully automated methods differ in how much expert interaction they require.Semi-automated approaches minimally incorporate expert knowledge.
4. Problem Formulation
Registration seeks a transformation that aligns a moving image with a fixed image by optimizing a similarity measure. Conventional methods optimize iteratively, whereas machine-learning methods can regress transformation parameters from learned image pairs.
- The registration objective is to find a geometric transformation T that aligns the moving image with the fixed image by optimizing a similarity measure.The transformation is parameterized by μ.
- Similarity measures are broadly intensity-based or feature-based, with examples including Mean Square Difference and Mutual Information.Intensity-based measures consider image-wide correspondence, whereas feature-based measures use extracted information.
- Conventional registration optimizes the similarity objective iteratively using methods such as hill-climbing or gradient descent.The process searches for transformation parameters that improve alignment.
- Machine-learning registration represents transformation parameters as a function of the fixed and moving images and regresses them in one shot.The learned model parameters θ are optimized during training through procedures such as back-propagation.
5. Deep Neural Networks
Deep neural networks expanded medical image registration beyond handcrafted features and iterative optimization, with architectures tailored to spatial, temporal, generative, and decision-making tasks.
- Background: Deep neural networks became dominant in medical image analysis and were applied to registration as a major challenge in image-guided intervention.Applications include organ detection, landmark localization, lesion analysis, treatment planning, and follow-up.
- Auto-Encoders: Auto-encoders and stacked auto-encoders learn compact discriminative representations, which registration methods use instead of handcrafted features.The hidden representation is smaller than the input, and denoising improves generalization.
- CNNs: CNNs process images or patches directly through learned kernels and pooling layers, while U-Net architectures can produce an end-to-end registration field.Multi-channel CNNs can also combine heterogeneous representations as separate input channels.
- GANs: GANs combine competing generator and discriminator subnetworks to synthesize samples and distinguish generated data from real data.The generator learns from ground-truth data, while the discriminator produces a binary real-versus-fake decision.
- RNNs and DRL: RNN feedback loops support temporal analysis, whereas DRL agents iteratively select transformation actions using environmental feedback and rewards.In image registration, DRL was applied specifically to rigid or affine transformations with restricted action spaces.
6. Literature Review
The review organizes deep-learning registration into five generations addressing slow iterative optimization and unreliable multimodal similarity measures, from learned metrics to supervised end-to-end systems.
- Motivation: Conventional registration is slow and can become trapped in local minima, motivating learning-based alternatives for runtime and similarity-measure limitations.The review identifies prolonged runtimes and defective multimodal similarity measures as central challenges.
- Registration Roles: Deep networks either approximate image similarity for conventional registration methods or regress transformation parameters directly in one shot.The second role targets increased runtime speed by avoiding iterative parameter estimation.
- Taxonomy: The review proposes five generations: Deep Similarity Metrics, Supervised End-to-End Registration, Deep Reinforcement Learning, Unsupervised End-to-End Registration, and Weakly/Semi-Supervised End-to-End Registration.These generations form the review’s taxonomy of deep-learning approaches.
- Deep Similarity Metrics: Deep similarity metrics learn structural differences from paired annotated images and feed the learned metric into conventional iterative deformable registration.The review describes this as the first generation and identifies it as a potential alternative to conventional multimodal approaches.
- Supervised End-to-End Registration: Supervised end-to-end registration trains networks on ground truth to predict transformation parameters in one shot, with CNNs common for affine and U-Nets for deformable models.Spatial Transformer Networks add localization, grid generation, and sampling components to learn and apply transformations within networks.
7. Literature Review Analysis
The literature analysis shows a rapidly developing but conference-heavy field dominated by brain and deformable-registration studies, with MRI and DSC/TRE among the most prominent choices.
- Publication Trends: The literature began in 2013, had a one-year publication gap in 2014, and reached a publication peak in 2018.The review cautions that the topic was still in its infancy and conclusions could be premature.
- Publication Types: More than 60% of publications appeared in conference proceedings, with journal articles and book chapters following.MICCAI is identified as a valuable forum for the field.
- Keywords: Deep Learning, Convolutional Neural Network, and Image Registration were the most frequent keywords, appearing 22, 16, and 12 times, respectively.Deformable Image Registration and Deformable Registration appeared 8 and 6 times.
- Evaluation: Dice Coefficient and Target Registration Error were the most frequently used evaluation metrics, measuring overlap and landmark-distance error, respectively.DSC ranges from 0 to 1, whereas TRE requires corresponding landmarks and is reported in millimeters.
- Datasets: The most frequently used datasets included Private, ANDI, LONI, IXI, OASIS, and DIRLAB, each appearing in more than five papers.Private denotes datasets unavailable publicly when the associated papers were published.
- Modalities: MRI accounted for about 50% of reported modalities, while CT and X-ray contributed about another 27%; multimodal registration represented 36% of studies versus 64% unimodal.The review counted 29 multimodal and 51 unimodal publications.
- Organs and Transformations: Deformable registration comprised 78% of cases, compared with 22% for rigid registration, while the brain was the leading organ with 77 papers.The review links brain prevalence to relatively rigid anatomy, visible structures, and easier validation.
8. Discussion on Confiding Challenges, Open Problems and Promising Directions
The review identifies recurring data, computational, interpretability, and contextual challenges in deep learning-based medical image registration, alongside proposed mitigation strategies and open problems. It emphasizes reducing runtime while preserving performance and expanding network receptive fields, while noting persistent limits in datasets, 3D computation, and explainability.
- Data limitations: Small medical datasets motivate augmentation, transfer learning, weakly supervised learning, and dropout to increase usable training information and reduce overfitting.The review also notes that annotated labels can be noisy, motivating noise modeling or fuzzy logic.
- Interpretability: Deep learning systems may lack inference rule chains acceptable to physicians, and current visualization-based explanations remain limited.The interpretability problem is described as still open despite promising visualization studies.
- Context integration: Clinical records, genomics, biopsies, and other results could be supplied through separate channels, but too few integrated datasets limit evaluation of their impact.The review presents background context as potentially informative for registration performance.
- Computational constraints: Most current deep neural networks process inherently 3D medical images in 2D or 2.5D because full 3D processing is often computationally unaffordable.The proposed boundary is dependent on future infrastructure progress enabling the required computational intensity.
- Model design priorities: The literature seeks to decrease registration runtime while maintaining performance and to expand receptive fields beyond small patches whose context may be informative.Typical patches range from 13×13×13 to 30×30×30 voxels, while one reported approach registers image pairs in less than 50 ms.
9. Conclusions and Future Trends
The review organizes deep learning approaches for medical image registration into five categories and compares their shared properties, advantages, and limitations. It highlights unsupervised and weakly supervised learning as promising under limited ground-truth availability, while identifying multistage, multiresolution, transfer-learning, geometric, STN, receptive-field, and emerging neural-network directions.
- Taxonomy: The proposed taxonomy contains DSM, SE2ER, DRL, UE2ER, and WSE2ER categories with distinct specifications, philosophies, advantages, and disadvantages.Deep reinforcement learning is described as hindered by the huge state space of deformable registration.
- Learning paradigms: Unsupervised and weakly supervised approaches require less ground truth, whose acquisition is costly in medical imaging, and are expected to receive increasing attention.The review qualifies this expectation by noting that large publicly available annotated datasets could change the situation.
- Key techniques: Multistage rigid-to-deformable and multiresolution low-to-high registration policies are reported as improving registration performance or precision.These policies are presented as recurring patterns across the reviewed literature.
- Key techniques: Transfer learning, geometric theory, Spatial Transformer Networks, and larger receptive fields are identified as practical or promising design directions.The review also cautions against over-engineering networks for real-time applications.
- Future trends: Future contributions may come from computer vision and machine learning, including models beyond CNNs, SAEs, GANs, DRL, and deep RNNs such as GRUs and SNNs.GRUs are suggested for temporal medical imaging, including continuous ultrasound or discrete fluoroscopy.
Appendix 1: Acronyms
The appendix lists acronyms used throughout the review for neural-network models, registration methods, imaging modalities, anatomical targets, and evaluation measures.
- Learning methods: Neural-network and learning acronyms include CNN, DNN, DRL, GAN, SAEs, AEs, DAEs, LSTM, and SNN.The list expands these terms as convolutional, deep, reinforcement, adversarial, auto-encoder, recurrent, and spiking neural-network concepts.
- Evaluation measures: Evaluation acronyms include PCC, SSIM, PSNR, DSC, ASD, ASSD, MI, NMI, FSIM, RMSEc, NCC, SSD, MSE, MAE, TRE, and RMSD.The list covers correlation, similarity, distance, error, registration, and image-quality measures.
- Registration methods: Registration and deformation acronyms include LDDMM, SVF, TPS, DVF, DSM, SE2ER, UE2ER, and WSE2ER.These include a deformation metric mapping, velocity field, spline, displacement field, and taxonomy categories.
- Imaging modalities: Imaging and intervention acronyms include IGRT, CT, MRI, PET, US, OCT, EM, DSA, DSA, DR, DDR, and TEE.The appendix also includes MSI, FA, CFI, and TRUS among modality or imaging abbreviations.
Appendix 2: Other 164 authors active in this field
The appendix provides a compact author-and-publication listing for additional researchers active in medical image registration and related deep learning work.
- Author index: The appendix lists author names alongside publication and citation counts in repeated tabular columns.The supplied entries include Moctezuma, J-L with 1 publication and 10 citations, plus Rastinehad, Ardeshir R.