PaperlayerView exact revision on arXiv

Source-linked AI summary

A Deep Learning Framework for Unsupervised Affine and Deformable Image Registration

Bob D. de Vos, Floris F. Berendsen, Max A. Viergever, Hessam Sokooti, Marius Staring, Ivana Isgum

arXiv:1809.06130v2cs.CV

TL;DR

The paper addresses the difficulty of obtaining supervised registration examples by proposing DLIR, which trains ConvNets for affine and deformable registration using image similarity. The trained networks perform one-shot registration, and experiments report performance comparable to conventional registration with much shorter execution times.

  • Problem

    Deep-learning registration methods typically require supervised example registrations, but obtaining suitable examples is not trivial.

  • Method

    DLIR trains ConvNets unsupervised by using image similarity between fixed images and warped moving images, with stacked stages supporting coarse-to-fine registration.

  • Results

    Across cardiac cine MRI and chest CT experiments, DLIR produced registration results similar to conventional methods while executing much faster.

  • Takeaways & Limitations

    DLIR enables accurate affine and deformable registration without labeled example registrations and supports one-shot registration of unseen images.

  • Takeaways & Limitations

    End-to-end training of the multi-stage ConvNets was impossible because of hardware and software memory limitations, so hierarchical training fixed preceding-stage weights.

Abstract

from arXiv · show

Image registration, the process of aligning two or more images, is the core technique of many (semi-)automatic medical image analysis tasks. Recent studies have shown that deep learning methods, notably convolutional neural networks (ConvNets), can be used for image registration. Thus far training of ConvNets for registration was supervised using predefined example registrations. However, obtaining example registrations is not trivial. To circumvent the need for predefined examples, and thereby to increase convenience of training ConvNets for image registration, we propose the Deep Learning Image Registration (DLIR) framework for \textit{unsupervised} affine and deformable image registration. In the DLIR framework ConvNets are trained for image registration by exploiting image similarity analogous to conventional intensity-based image registration. After a ConvNet has been trained with the DLIR framework, it can be used to register pairs of unseen images in one shot. We propose flexible ConvNets designs for affine image registration and for deformable image registration. By stacking multiple of these ConvNets into a larger architecture, we are able to perform coarse-to-fine image registration. We show for registration of cardiac cine MRI and registration of chest CT that performance of the DLIR framework is comparable to conventional image registration while being several orders of magnitude faster.

1. Introduction

Medical image registration aligns images for analysis, but existing deep-learning approaches typically require supervised example registrations that can be difficult to obtain. The DLIR framework addresses this gap by training ConvNets unsupervised from image similarity, enabling one-shot registration of unseen image pairs.

  • Motivation: Deep-learning registration methods are attractive for time-critical medical applications because they are highly parallelizable and fast on GPUs.The paper cites HIFU, MR-linac, and MRI-guided proton therapy as examples.
  • Problem: Existing deep-learning registration methods are generally supervised, relying on example registrations or manual segmentations that may be problem-specific to generate.Synthetic transformations and manual annotations are described as common ways to obtain training examples.
  • DLIR framework: The DLIR framework trains ConvNets without predefined registration examples by optimizing image similarity between fixed and moving image pairs.Unlike conventional registration, it optimizes ConvNet parameters indirectly rather than directly optimizing transformation parameters.
  • DLIR framework: After training, a ConvNet predicts transformations for unseen fixed and moving image pairs in one shot rather than iteratively.The framework is designed for non-iterative application after training.
  • Contributions: The paper extends prior work from 2D to 3D and introduces affine, deformable, and multi-stage ConvNet designs for coarse-to-fine registration.The extensions include B-spline registration with transposed convolutions and a bending-energy penalty for smoother displacements.

2. Method

The DLIR framework trains ConvNets by predicting transformation parameters from fixed and moving images while optimizing image similarity, supporting affine, deformable, and hierarchical multi-stage registration. Its architectures use separate global affine processing, patch-based B-spline deformation prediction, and sequentially trained coarse-to-fine stages.

  • DLIR trains ConvNets by predicting transformation parameters from fixed and moving images and minimizing image dissimilarity between fixed and warped moving images.The ConvNet parameters are optimized rather than the transformation parameters directly.
  • Affine Image Registration: The affine-registration ConvNet processes fixed and moving images in separate pipelines, enabling different input sizes before producing 12 affine transformation parameters.Global average pooling allows the pipeline outputs to connect to fixed-size fully connected layers.
  • Deformable Image Registration: The deformable ConvNet takes equal-sized image pairs and predicts B-spline 3D displacement vectors for patches, with grid spacing determining downsampling and network design.The final displacement vector field is generated by B-spline interpolation of predicted control points, implemented efficiently with transposed convolutions.
  • Multi-Stage Image Registration: A multi-stage architecture stacks affine and coarse-to-fine B-spline ConvNets for multi-resolution registration, training each stage sequentially while fixing preceding-stage weights.Transformation parameters are combined to warp the moving image passed to the next stage.
  • Loss Function: The registration loss combines negative normalized cross-correlation with a bending-energy penalty, using α = 0 for affine registration and α = 0.05 for deformable experiments.The bending-energy term penalizes second-order derivatives of local transformations and enforces global smoothness.

3. Data

The study uses cardiac cine MRI and chest CT datasets for intra-patient and inter-patient registration, but notes that available data are insufficient to demonstrate the method’s full potential.

  • The study uses large cardiac cine MRI datasets for intra-patient registration and low-dose chest CT datasets from NLST for inter-patient registration.
  • The available dataset is not sufficiently large to demonstrate the full potential of the proposed method, although it enables straightforward replication.
  • The cardiac cine MRI dataset contains 45 scans across four pathology categories, with 20 timepoints per scan.
  • The chest CT dataset includes 2,060 scans from fourteen CT scanners produced by four vendors.
  • The chest CT data comprise ten timepoints covering a full breathing cycle for each of ten 4D scans.

4. Evaluation

Evaluation measures registration quality through deformation-field topology, propagated anatomical labels, surface distances, and landmark displacement.

  • The DLIR framework is evaluated on both intra-patient and inter-patient registration experiments.
  • Jacobian determinants are computed throughout each deformation vector field to quantify topology and identify anatomically implausible folding.
  • A Jacobian of 1 indicates no volume change, values above 1 indicate expansion, values from 0 to 1 indicate shrinkage, and values at or below 0 indicate folding.
  • Topology quality is summarized using the fraction of foldings per image and the standard deviation of Jacobian determinants.
  • Registration accuracy is assessed by propagating manual segmentations and measuring Dice overlap, Hausdorff distance, and average symmetric surface distance.
  • Landmark registration error is measured as the average 3D Euclidean distance between transformed and reference points.

5. Implementation

The implementation trains ConvNets with Adam-based optimization, specified affine and deformable output constraints, and comparisons against similarly configured SimpleElastix registration.

  • All ConvNets use Glorot uniform initialization and Adam optimization within the DLIR loss framework.
  • Deformable ConvNet outputs are unconstrained, while affine outputs constrain rotations and shears to −π to +π and scaling to 0.5 to 1.5.
  • Training uses linear resampling for moving images, nearest-neighbor resampling for segmentations, and an NVIDIA Titan-X GPU.
  • Conventional registration is performed with SimpleElastix using settings selected to match DLIR as closely as possible.
  • The deformable registration design distinguishes sequential multi-stage DIR-1 and DIR-2 experiments from single-stage experiments using a DIR-2-equivalent stage.
  • SimpleElastix uses 500 optimization iterations per stage and samples 2,000 random points per iteration.

6. Intra-Patient Registration of Cardiac Cine MRI

Cardiac cine MRI experiments compare single-stage and multi-stage DLIR with conventional registration, evaluating training behavior, topology, label propagation, and pairwise agreement.

  • 6.1. Experimental setup: Cardiac cine MRI registration uses 3-fold cross-validation, with 30 images for training and 15 for evaluation per fold.
  • 6.1. Experimental setup: Multi-stage experiments sequentially apply deformable stages, while single-stage experiments use one deformable stage; bending penalties are evaluated separately.
  • 6.2. Results: Training takes 5 hours for single-stage registration and 8 hours for multi-stage registration.
  • 6.2. Results: The bending penalty mitigates folding in DLIR, while multi-stage registration improves label overlap through increased Dice and decreased ASD.
  • 6.2. Results: DLIR and conventional registration results show no correlation across image pairs, with some pairs favoring DLIR and others favoring conventional registration.
  • 6.2. Results: Multi-stage DLIR with bending penalty yields significantly less folding and lower Jacobian standard deviation, while Dice and ASD are as high as conventional registration.
  • 6.2. Results: Figure 5 compares negative NCC learning curves across single-stage and multi-stage ConvNets with or without bending penalties.

7. Inter-Patient registration of Low-Dose Chest CT

Inter-patient chest CT registration used an affine stage followed by three coarse-to-fine deformable stages to handle differing fields of view and anatomy. DLIR progressively improved alignment, with generally comparable registration quality to conventional registration and very short GPU execution time.

  • Architecture: Inter-patient chest CT registration used sequential affine and coarse-to-fine deformable stages because scans differed in field of view and anatomy.The multi-stage ConvNet comprised one affine registration stage followed by deformable stages.
  • Alignment results: The affine stage correctly aligned evaluation images, while subsequent deformable stages gradually improved the alignment.Intermediate results were evaluated across 90 fixed–moving image permutations from ten manually segmented-aorta scans.
  • Deformation regularity: Only the final deformable stage was substantially affected by folding, whereas the first two DLIR stages showed zero to limited folding.Conventional registration showed folding increasing across deformable stages, and stage-wise results differed significantly between methods.
  • Registration quality: Dice and average surface distance were similar after final-stage registration, while DLIR produced lower Hausdorff distance and fewer outliers than conventional registration.The first two deformable stages had slightly lower Dice and higher average surface distance for DLIR.
  • Execution time: 0.43 s per image pair was the approximate DLIR GPU time including image resampling.Execution times were summarized for all stages in the chest-CT experiment.
  • Method comparison: DLIR and conventional registration showed correlated image-pair quality, but some pairs were easier for one method than the other.The scatter plot indicates that registration tasks were not equally difficult for both frameworks.

8. Intra-Patient Registration of 4D Chest CT

The DIR-Lab experiment evaluated deformable DLIR on a small 4D chest CT dataset using leave-one-out training and testing. The final average registration error was 2.64 mm, while large initial deformations were poorly represented in training.

  • Training design: Leave-one-out cross-validation trained deformable ConvNets on nine scans and evaluated them on the remaining scan.The dataset was too small to train an affine registration ConvNet, so only deformable registration was learned.
  • Results: 2.64 mm was the final average registration error, with a standard deviation of 4.32 mm.The error was highly influenced by outliers associated with the limited dataset size.
  • Results: 1.63 mm was the adjusted average registration error after removing 10% of landmarks with the largest initial registration errors.The adjusted standard deviation was 1.67 mm.
  • Execution and limitations: 0.63 s was the average multi-stage registration time including intermediate and final image resampling.Large initial landmark distances were scarce during training and influenced performance.

9. Discussion

The discussion presents DLIR as an unsupervised, one-shot framework that can combine multiple ConvNets for complex registration. Across cardiac MRI, chest CT, and 4D chest CT, results were generally comparable to conventional registration but depended on representative training data and model constraints.

  • Framework: DLIR trains ConvNets from image similarity without labeled example registrations and supports affine, deformable, and multi-stage registration.Multiple ConvNets can be combined for complex tasks such as inter-patient registration.
  • Framework: After training, DLIR performs one-shot registration of unseen image pairs rather than iteratively optimizing each pair during application.The framework is used during training, while the learned ConvNet is applied for registration.
  • Overall findings: Across cardiac cine MRI, inter-patient chest CT, and 4D chest CT, DLIR produced registration results similar to conventional methods with exceptionally short execution times.The discussion identifies this pattern across all three multi-stage ConvNet experiments.
  • Overall findings: Inter-patient DLIR matched conventional performance in early stages, especially affine registration, but was slightly outperformed at later stages.The authors associate this partly with ConvNet design choices imposed by memory limitations.
  • Training dependence: Representative training data and problem complexity jointly influenced performance, with large deformations insufficiently corrected in the DIR-Lab experiment.The authors report that more training data could improve these results.
  • Regularization: A bending-energy penalty mitigated folding during training without increasing execution time, although it increased memory consumption and did not guarantee complete elimination of folding.The penalty limited the number of registration stages to three in these experiments.
  • Limitations: The final DLIR stage showed increased folding, while end-to-end training was precluded by memory limitations and hampered by exploding gradients in preliminary experiments.Hierarchical training with fixed preceding-stage weights reduced memory consumption.
  • Scope: The experiments used coarse-to-fine stages whose maximum deformations remained within the B-spline capture range and receptive fields.Behavior outside that receptive field was identified as a topic for future study.

10. Conclusion

The paper concludes that DLIR enables unsupervised training of ConvNets for accurate affine and deformable registration without example registrations, while retaining very short execution times.

  • Conclusion: DLIR trains ConvNets for accurate affine and deformable image registration without training examples and with very short execution times.The framework is presented as an unsupervised approach using convolutional neural networks.
Loading 1809.06130v2…