Source-linked AI summary
End-to-End Unsupervised Deformable Image Registration with a Convolutional Neural Network
Bob D. de Vos, Floris F. Berendsen, Max A. Viergever, Marius Staring, Ivana Išgum
TL;DR
Deformable image registration traditionally lacks an easy source of labeled transformation parameters. DIRNet learns registration end-to-end from image similarity, then registers unseen pairs in one pass, achieving comparable accuracy to conventional deformable registration with shorter execution times.
Problem
Existing deep-learning registration methods either decouple feature learning from registration or require task-specific transformation labels, which are challenging to obtain for medical images.
Method
DIRNet combines a ConvNet regressor, spatial transformer, and resampler, trained unsupervised by optimizing image similarity between fixed and moving images.
Results
DIRNet achieves accuracy comparable to a conventional deformable registration method with substantially shorter execution times across handwritten-digit and cardiac-MRI evaluations.
Takeaways & Limitations
A trained DIRNet performs deformable registration on unseen image pairs non-iteratively and without labeled transformation training data.
Takeaways & Limitations
The method is designed for 2D images and was evaluated using normalized cross correlation as its similarity metric.
Abstract
from arXiv · showhide
In this work we propose a deep learning network for deformable image registration (DIRNet). The DIRNet consists of a convolutional neural network (ConvNet) regressor, a spatial transformer, and a resampler. The ConvNet analyzes a pair of fixed and moving images and outputs parameters for the spatial transformer, which generates the displacement vector field that enables the resampler to warp the moving image to the fixed image. The DIRNet is trained end-to-end by unsupervised optimization of a similarity metric between input image pairs. A trained DIRNet can be applied to perform registration on unseen image pairs in one pass, thus non-iteratively. Evaluation was performed with registration of images of handwritten digits (MNIST) and cardiac cine MR scans (Sunnybrook Cardiac Data). The results demonstrate that registration with DIRNet is as accurate as a conventional deformable image registration method with substantially shorter execution times.
1 Introduction
DIRNet addresses the challenge of learning deformable image registration without labeled transformations by optimizing image similarity end-to-end. It extends prior deep-learning registration approaches toward unsupervised, non-iterative deformable registration.
- Motivation: Earlier approaches either decoupled feature learning from registration or required supervised transformation parameters.The CAE-based method was not trained end-to-end, while rigid-registration methods used task-specific transformation labels.
- Motivation: Spatial transformer networks can align images within larger networks, but their alignment is task-dependent and typically requires many labeled examples.Prior STN work included affine and thin-plate-spline transformations, but did not guarantee alignment independently of the downstream task.
- Contribution: DIRNet learns deformable registration directly from fixed–moving image pairs by unsupervised optimization of an image-similarity metric.Unlike prior methods, it does not use known registration transformations as training targets.
- Contribution: After training, DIRNet registers unseen image pairs in one pass rather than through iterative optimization.The proposed network outputs warped moving images for new fixed–moving pairs non-iteratively.
2 Method
DIRNet combines a ConvNet regressor, spatial transformer, and resampler into an end-to-end unsupervised registration pipeline. The network predicts local deformation parameters and learns them by backpropagating image dissimilarity.
- Architecture: The ConvNet analyzes corresponding fixed and moving patches and predicts local deformation parameters for the spatial transformer.The transformer converts these parameters into a dense displacement vector field.
- Architecture: The spatial transformer generates a dense displacement vector field that allows the resampler to warp the moving image onto the fixed image.This connects predicted local parameters to the final registered image.
- Network design: The ConvNet uses alternating 3 × 3 convolutions and 2×2 average-pooling layers, followed by 1×1 convolutions for fully convolutional prediction.Batch normalization and exponential linear units are used throughout except for the final linear-output layer.
- Optimization: Training minimizes an image similarity loss between fixed and moving images using mini-batch Adam optimization.Normalized cross correlation is used in this work, although other conventional similarity metrics could be used.
3 Data
The evaluation uses both a controlled handwritten-digit dataset and clinical cardiac cine MRI data. These datasets provide separate training, validation, and test subsets for registration experiments.
- Datasets: DIRNet was evaluated on handwritten digits from MNIST and clinical cardiac cine MRI scans from the Sunnybrook Cardiac Data.The two datasets represent distinct image domains used for registration evaluation.
- MNIST: MNIST contains 28×28 grayscale digit images, with 60,000 training images and 10,000 held-out test images.One sixth of the training data was used for validation.
4 Experiments and Results
Experiments assess DIRNet on MNIST and cardiac MRI using qualitative registration, architectural variants, and quantitative comparison with SimpleElastix. Cardiac results show fast registration and competitive performance against conventional iterative registration.
- MNIST experiments: Separate MNIST DIRNets were trained per digit class for 5,000 iterations using 32 randomly paired fixed and moving images per mini-batch.The networks used 16 kernels per convolution layer and a 7 × 7 control-point grid.
- MNIST experiments: On MNIST, trained DIRNets were qualitatively assessed by registering approximately 1,000 test moving images per digit class to one randomly selected fixed image.Figure 2 compares class-average moving images, selected fixed images, and averaged registration results.
- Cardiac MRI experiments: Cardiac training used 69,540 same-scan image pairs for training and 63,840 pairs for validation, sampled across cardiac-cycle timepoints.Pairs were anatomically corresponding slices from different timepoints in the same patient scan.
- Cardiac MRI experiments: The final validation loss was lowest for DIRNet-C1, while DIRNet-B2 converged after approximately 30,000 iterations with a loss greater than the baseline.Figure 3 compares downsampling, spatial-transformer, and receptive-field variants.
- Quantitative comparison: Table 1 compares pre-registration, SimpleElastix, and DIRNet using Dice score, 95thSD, and MAD from fixed-image and warped-annotation comparisons.The table reports means and standard deviations for each metric.
- Quantitative comparison: 0.049 ± 0.0035 s was the DIRNet-C1 registration time, compared with 0.51 ± 0.070 s for SimpleElastix.The conventional method was iterative and ran on an 8-thread CPU, whereas DIRNet-C1 ran on an NVIDIA Titan X Maxwell GPU.
- Qualitative cardiac results: Figure 4 compares fixed ED, moving ES, DIRNet-C1 warped, and SimpleElastix warped images alongside absolute-difference heatmaps.The heatmaps compare the original, DIRNet-warped, and SimpleElastix-warped moving images against the fixed image.
5 Discussion and Conclusion
DIRNet learns deformable registration end-to-end without supervision and achieves conventional-method accuracy with substantially shorter execution times. Its demonstrated scope remains dependent on domain-specific training, 2D inputs, and data that need no rough pre-alignment.
- Discussion and Conclusion: The method was evaluated on handwritten digits and cardiac cine MRI scans.
- Discussion and Conclusion: A single DIRNet architecture worked across different image domains only with domain-specific training.Whether one DIRNet instance can register images across domains remains for future investigation.
- Discussion and Conclusion: Cardiac cine MRI results were comparable to or slightly better than SimpleElastix despite its task-specific tuning.DIRNet was trained using slices from any cardiac-cycle timepoint, while evaluation compared end-systolic and end-diastolic images.
- Discussion and Conclusion: The experiments used 2D images and data that did not require rough pre-alignment; future work targets affine pre-alignment and 3D registration.Experiments used normalized cross correlation, although any differentiable similarity metric could be used.
- Discussion and Conclusion: DIRNet performs unsupervised end-to-end deformable registration and is as accurate as a conventional method with substantially shorter execution times.Registration is performed in one pass, non-iteratively.