PaperlayerView exact revision on arXiv

Source-linked AI summary

Quicksilver: Fast Predictive Image Registration - a Deep Learning Approach

Xiao Yang, Roland Kwitt, Martin Styner, Marc Niethammer

arXiv:1703.10908v4cs.CV

TL;DR

Deformable image registration is accurate but computationally expensive, especially for high-dimensional models and large imaging datasets. Quicksilver predicts LDDMM momentum patch-by-patch from image appearance, adding uncertainty estimation and correction networks. The method reports fast, accurate registration, including multimodal cases, while retaining LDDMM's diffeomorphic properties.

  • Problem

    High-dimensional deformable registration relies on computationally costly numerical optimization, limiting processing of large and high-resolution imaging datasets.

  • Method

    Quicksilver uses deep regression to predict the initial LDDMM momentum from corresponding moving- and target-image patches, with probabilistic and correction-network extensions.

  • Results

    Quicksilver accurately predicts optimization-based registrations, achieves state-of-the-art results on four validation datasets, and predicts accurate deformations for multimodal registration.

  • Takeaways & Limitations

    The approach combines fast patch-wise deformation prediction with LDDMM's theoretical properties and supports uncertainty estimation and multimodal similarity learning.

  • Takeaways & Limitations

    The approach still depends on a chosen regularizer to generate training data, although ideally the regularizer would be learned from data.

Abstract

from arXiv · show

This paper introduces Quicksilver, a fast deformable image registration method. Quicksilver registration for image-pairs works by patch-wise prediction of a deformation model based directly on image appearance. A deep encoder-decoder network is used as the prediction model. While the prediction strategy is general, we focus on predictions for the Large Deformation Diffeomorphic Metric Mapping (LDDMM) model. Specifically, we predict the momentum-parameterization of LDDMM, which facilitates a patch-wise prediction strategy while maintaining the theoretical properties of LDDMM, such as guaranteed diffeomorphic mappings for sufficiently strong regularization. We also provide a probabilistic version of our prediction network which can be sampled during the testing time to calculate uncertainties in the predicted deformations. Finally, we introduce a new correction network which greatly increases the prediction accuracy of an already existing prediction network. We show experimental results for uni-modal atlas-to-image as well as uni- / multi- modal image-to-image registrations. These experiments demonstrate that our method accurately predicts registrations obtained by numerical optimization, is very fast, achieves state-of-the-art registration results on four standard validation datasets, and can jointly learn an image similarity measure. Quicksilver is freely available as an open-source software.

1. Introduction

Quicksilver addresses the computational cost of deformable image registration with fast, patch-wise prediction of LDDMM momentum. It combines diffeomorphic guarantees with speed, uncertainty estimation, correction-based accuracy improvements, and multimodal registration.

  • Motivation: Deformable registration is computationally costly because high-dimensional models require numerical optimization over many parameters.GPU implementations can still require about 10 minutes for a 128 × 128 × 128 LDDMM volume.
  • Motivation: Large-scale and high-resolution imaging studies increase the need for fast, memory-efficient, and parallelizable registration methods.The paper motivates rapid registration for interactive analysis, advanced processing pipelines, and very large imaging studies.
  • Method: Quicksilver predicts initial LDDMM momentum from image patches with a deep regression model and reconstructs the full deformation patch-by-patch.Momentum need not be spatially smooth, enabling patch-wise prediction while LDDMM subsequently produces smooth velocity fields.
  • Method: Momentum parameterization retains LDDMM's desirable diffeomorphic properties while avoiding ambiguities in uniform image regions.With sufficiently strong regularization, LDDMM produces smooth, one-to-one transformations with smooth inverses.
  • Extensions: A Bayesian prediction network estimates uncertainty in registration parameters and deformation fields for visualization and uncertainty-based applications.The paper identifies uncertainty-based smoothing and surgical treatment planning as possible uses.
  • Extensions: The correction network improves predicted registrations to accuracy similar to optimization-based registrations, while the framework also predicts accurate multimodal deformations and learns a multimodal similarity measure.The paper reports validation on four standard datasets with registration accuracies on par with the state of the art.

2. Materials and Methods

Quicksilver predicts LDDMM initial momentum patches with deep networks, using momentum parameterization to support patch-wise prediction while retaining LDDMM-based deformation modeling. A correction stage, probabilistic formulation, and patch-based strategy extend the method toward accurate and efficient registration.

  • LDDMM formulation: Image registration finds a deformation map Φ that makes the transformed moving image similar to the target image.The formulation uses M ◦Φ^-1(x) ≈ T(x), with regularization constraining spatially irregular transformations.
  • LDDMM formulation: LDDMM optimizes transformations through velocity fields, while shooting searches over geodesic paths represented by the initial momentum.The momentum is the dual of velocity and parameterizes the geodesic path generated by the shooting equations.
  • Patch-wise prediction: Patch-wise prediction uses moving and target patches at corresponding locations to predict local deformation parameters, avoiding full-volume network inputs.This strategy addresses GPU-memory limits for large 3D images and relies on the initial momentum because it need not be spatially smooth.
  • Theoretical scope: Sufficient regularization is intended to support diffeomorphic transformations, but the stated 3D regularizer is too weak to guarantee them under the cited sufficient-regularity conditions.The paper notes that those conditions would require L to be at least a differential operator of order 6 in 3D.
  • Deep prediction network: The encoder processes moving and target patches in parallel, concatenates their learned features, and uses three decoders to predict momentum components.The architecture uses 3D convolutions in the encoder and 3D transposed convolutions in the decoder, with one decoder for each spatial dimension.
  • Probabilistic prediction: Dropout converts the deterministic encoder-decoder into a probabilistic network that predicts a distribution over momentum patches.The probabilistic formulation uses dropout as approximate Bayesian inference and supports uncertainty estimation through predictive sampling.
  • Correction network: The correction network predicts a momentum correction after shooting the initial prediction and warping the target image into moving-image space.The final momentum is m = mLP + mC; experiments report that this combined strategy outperforms simply training the prediction network for more epochs.

3.1. Atlas-to-Image registration

The atlas-to-image experiments compare patch-based prediction settings and deformation parameterizations against numerically optimized LDDMM. Initial momentum prediction provides the strongest combination of accuracy and diffeomorphic mappings, while probabilistic predictions also expose spatial uncertainty.

  • Experimental setup: The evaluation compares stride-5 and stride-14 patch prediction, alternative deformation parameterizations, and deterministic or probabilistic networks against numerical LDDMM optimization.Errors are summarized by voxel-wise deformation-error percentiles, with diffeomorphism assessed using the proportion of registrations having detJ > 0.
  • Parameterization comparison: Initial momentum prediction is more accurate than initial velocity or direct displacement prediction for both tested strides.The authors interpret this as evidence that momentum-based LDDMM is better suited to patch-wise deformation prediction.
  • Limitation: The authors retain regular-grid patch sampling because reducing the number of predicted patches was a design goal, leaving random patch sampling unexplored.Randomly sampling patch locations might avoid the stride-related behavior observed for momentum prediction.
  • Diffeomorphic mappings: Initial momentum and initial velocity networks generate diffeomorphic deformations in all tested scenarios, whereas direct displacement prediction cannot guarantee diffeomorphism.Diffeomorphism is evaluated through positive Jacobian determinants; nonpositive determinants indicate folding.
  • Qualitative results: In an atlas-to-image example, the probabilistic network’s mean deformation closely resembles the deformation obtained by LDDMM optimization.The example uses the mean of 50 samples with stride=14 and patch pruning.
  • Uncertainty: The uncertainty map reports deformation variance across x, y, and z directions, with high uncertainty concentrated near ventricles undergoing drastic deformations.The map uses the square root of the summed directional variances and displays high uncertainty in red and low uncertainty in blue.

3.2. Image-to-Image registration

Quicksilver’s prediction and correction networks produce registrations close to numerical LDDMM optimization across four image-to-image datasets, while preserving competitive label overlap. The correction network improves alignment and deformation accuracy, but repeated or mismatched correction steps can create overly extreme and unstable deformations.

  • LDDMM energy: LPC produces LDDMM energies only slightly above LO and closer to LO than LP, indicating momenta nearer the optimized solution.Both prediction methods substantially reduce energy relative to the affine-registered initial state.
  • Label overlap: LPC achieves target-overlap performance nearly matching numerical LDDMM optimization, while LP shows a slightly larger decrease across the four datasets.The comparison covers LPBA40, IBSR18, CUMC12, and MGH10, with LO used as the numerical reference.
  • Label overlap: LPC is competitive with leading registration methods and achieves top-tier label matching at a small fraction of the computational cost, with virtually no outliers.The numerical LO baseline is itself on par with SyN, ART, and SPM5D in the reported comparison.
  • Deformation distributions: LP produces smoother deformations than LO, whereas LPP, LPC2, and LPC3 produce increasingly drastic expansions and contractions.The correction network is trained to correct prediction errors toward LO; repeated or substituted correction steps do not share that explicit objective.
  • Deformation accuracy: LPC generally has the highest deformation prediction accuracy for up to 80% of voxels and remains on-par with other prediction models through 99% of voxels.For the largest 1% of deformations, LPC does not drastically reduce deformation error.
  • Failure cases: An IBSR18 failure case shows LPC generating reasonable deformations when inconsistent cerebellum anatomy causes optimization-based LDDMM to produce an extreme deformation.The inconsistency arose because different cerebellar regions were preserved in the moving and target images.

3.3. Multi-modal image registration

Quicksilver extends to multimodal image-to-image registration, where networks trained on T1w-T2w images substantially improve over affine registration and remain close to T1w-T1w performance. Correction improves prediction accuracy across the tested settings, including training with only 10 images.

  • Multimodal accuracy: Multimodal T1w-T2w networks significantly reduce deformation error relative to affine registration while suffering only a slight accuracy loss versus T1w-T1w networks.The comparison uses deformation error per voxel against T1w-T1w LDDMM optimization.
  • Correction network: The correction network improves prediction accuracy in all tested multimodal cases, including models trained with 10 images.The 10-image setting contains 90 registrations as training cases.
  • Training data: With sufficient training patches, prediction networks can be trained successfully even when image data are limited.The reported multimodal examples produced warped images similar to the LDDMM optimization result.

3.4. Runtime study

Quicksilver replaces costly 3D LDDMM optimization with patch-wise momentum prediction and geodesic shooting. On a TitanX GPU, it reduces registration time from approximately 10.8 minutes to 18.43 seconds, and multi-GPU prediction provides further acceleration.

  • Single-GPU runtime: 18.43 seconds versus approximately 10.8 minutes yields a 35× speedup for 3D registration on a single TitanX GPU.Momentum prediction takes 7.63 seconds and subsequent geodesic shooting takes 8.9 seconds.
  • Correction cost: Using the correction network doubles computation time but remains an order of magnitude faster than direct LDDMM optimization.The speed advantage depends on the large sliding-window stride enabled by initial-momentum prediction.
  • Sliding-window stride: At stride 1, the 3D computational cost increases about 3000-fold, making large sliding-window strides essential for fast prediction.The reported stride-1 runtime is about 5.5 hours, or 11 hours with correction.
  • Multi-GPU runtime: Increasing GPUs reduces initial momentum prediction time from 11.23 seconds on one GPU to 2.41 seconds on multiple GPUs.Prediction time slightly increases to 2.48 seconds at larger GPU counts, attributed to PyTorch’s then-limited multi-GPU performance.

4. Discussion

Quicksilver uses patch-wise initial-momentum prediction for fast LDDMM registration while preserving diffeomorphic transformations under appropriate regularization. The framework is generalizable and supports further applications and extensions.

  • Patch-wise initial-momentum prediction permits large strides without substantially reducing registration accuracy, producing fast and accurate deformation prediction.
  • Quicksilver computes diffeomorphic LDDMM transformations an order of magnitude faster than GPU-based optimization when appropriately regularized.The correction network improves accuracy while reducing computation speed by a factor of 2.
  • Patch-wise prediction enables multi-GPU implementation, further increasing prediction speed.Average initial momentum prediction time is reported for a 229×193×193 3D brain image using varying numbers of GPUs.
  • The framework can directly predict parameters for other non-parametric registration methods and can be adapted to locally controlled parametric methods such as B-splines.
  • The authors identify extensions including multi-atlas segmentation, image geodesic regression, atlas construction, interactive registration refinements, and additional correction strategies.Quicksilver was open-sourced to support variants for different organs, imaging types, and spatial regularizations.
Loading 1703.10908v4…