PaperlayerView exact revision on arXiv

Source-linked AI summary

Predicting Slice-to-Volume Transformation in Presence of Arbitrary Subject Motion

Benjamin Hou, Amir Alansary, Steven McDonagh, Alice Davidson, Mary Rutherford, Jo V. Hajnal, Daniel Rueckert, Ben Glocker, Bernhard Kainz

arXiv:1702.08891v2cs.CV

TL;DR

Intensity-based 2D/3D registration has a limited capture range and often requires good initialization. The paper uses a CNN to predict slice transformations for initializing reconstruction and registration refinement. It reports prediction errors across brain, fetal brain, and thorax datasets, and describes motion-compensation results for highly corrupted scans.

  • Problem

    Intensity-based registration has a limited capture range and often requires good initial alignment, while aligning many 2D slices manually is impractical.

    Source: §1 · pp. 1–2

  • Method

    A CNN predicts arbitrary 2D slice transformations relative to a canonical atlas coordinate system, and its predictions initialize volume reconstruction and registration refinement.

    Source: §1 · p. 2, §2 · p. 5

  • Results

    Average prediction errors were 5.6±1.07 mm for adult brains, 7.7±4.80 mm for fetal brain MRI, and 5.9±2.43 mm for thorax CT.

    Source: §3 · p. 5

  • Takeaways & Limitations

    The method allows motion compensation for highly motion-corrupted scans and can use acquired images without requiring temporal proximity for registration initialization.

    Source: Exp. 3 · p. 8

    SVRNet requires consistent input intensity ranges, spacing, and translation-offset removal, and training for a specific ROI or organ and imaging scenario.

    Source: Exp. 3 · p. 8

Abstract

from arXiv · show

This paper aims to solve a fundamental problem in intensity-based 2D/3D registration, which concerns the limited capture range and need for very good initialization of state-of-the-art image registration methods. We propose a regression approach that learns to predict rotation and translations of arbitrary 2D image slices from 3D volumes, with respect to a learned canonical atlas co-ordinate system. To this end, we utilize Convolutional Neural Networks (CNNs) to learn the highly complex regression function that maps 2D image slices into their correct position and orientation in 3D space. Our approach is attractive in challenging imaging scenarios, where significant subject motion complicates reconstruction performance of 3D volumes from 2D slice data. We extensively evaluate the effectiveness of our approach quantitatively on simulated MRI brain data with extreme random motion. We further demonstrate qualitative results on fetal MRI where our method is integrated into a full reconstruction and motion compensation pipeline. With our CNN regression approach we obtain an average prediction error of 7mm on simulated data, and convincing reconstruction quality of images of very young fetuses where previous methods fail. We further discuss applications to Computed Tomography and X-ray projections. Our approach is a general solution to the 2D/3D initialization problem. It is computationally efficient, with prediction times per slice of a few milliseconds, making it suitable for real-time scenarios.

1 Introduction

Intensity-based registration has a limited capture range and often needs good initialization, while aligning many independently acquired 2D slices is especially difficult. The paper proposes learning slice transformations into a canonical atlas coordinate system to initialize registration automatically.

1 Introduction

  • The limited capture range of intensity-based registration means many registration-dependent analyses require manual initialization.This affects applications including atlas-based segmentation, motion compensation, tracking, and clinical analysis in a standard coordinate system.Source: §1 · p. 1
  • Manual alignment of hundreds of slices is impractical, and fetal imaging slices lack the 3D information needed for robust landmark matching.Landmark-based alignment also depends on detection accuracy, homography robustness, and the descriptive power of the landmark encoding.Source: §1 · pp. 2–3
  • The proposed CNN learns slice transformations from 2D intensity information relative to a canonical atlas coordinate system, initializing subsequent intensity-based registration.The paper evaluates prediction quantitatively on simulated adult brain and thorax MRI data and qualitatively in a fetal MRI reconstruction and motion-compensation pipeline.Source: §1 · p. 2
  • The paper focuses on slice-to-volume registration because manual alignment of many slices is more challenging than manual initialization of volume-to-slice registration.The related-work discussion distinguishes volume-to-slice alignment, such as registering an intra-operative C-arm X-ray to a pre-operative volume, from the slice-to-volume focus here.Source: §1 · p. 2

2 Method

The method predicts each slice’s transformation from its intensity information and uses the predictions to initialize reconstruction and registration refinement. Training samples cover varied slice orientations, while the network regresses three consistent Cartesian landmarks per slice.

2 Method

  • The pipeline combines approximate organ localization, transformation prediction, and 3D reconstruction with intensity-based registration refinement.The ROI may be delineated manually, acquired with organ-focused sequences, or identified automatically; SVRNet must be trained for the target ROI, modality, and use case.Source: §2 · p. 3
  • Training and validation slices are sampled from motion-free volumes resampled to 1 mm isotropic spacing, with planes spaced 4 mm apart along the Z-axis.Low-variance planes are omitted, and the reported K ≈ 0.2 selects the central 80% of the volume.Source: §2 · p. 4
  • Training planes are rotated about the origin while volumes remain static, and Fibonacci sphere sampling distributes orientation samples approximately evenly over the sphere.Sampling only one hemisphere suffices because the opposite hemisphere produces mirrored images under the stated symmetry constraint.Source: §2 · p. 4
  • The network predicts nine Cartesian coordinates for the slice center and two corners, a representation selected because it performed most accurately among the tested options.The landmarks are transformed by a rotation matrix into world coordinates, keeping the loss in Cartesian coordinate space.Source: §2 · p. 4
  • SVRNet predicts slice transformations from 2D image intensities, and those predictions initialize a reconstructed volume for iterative intensity-based registration refinement.The refinement registers slices individually using cross-correlation and gradient descent, with robust statistics excluding mispredicted slices from later iterations.Source: Figure 1 · p. 3, §2 · p. 5

3 Experiments and Results

Across adult brain MRI, fetal brain MRI, and thorax CT, SVRNet predicts slice transformations, with reported average prediction errors of 5.6±1.07 mm, 7.7±4.80 mm, and 5.9±2.43 mm, respectively. Experiments also show reconstruction refinement with SVR and application to X-ray-to-CT registration, while the method requires training-matched image formatting and a scenario-specific model.

3 Experiments and Results

  • Average prediction errors were 5.6±1.07 mm for adult brains, 7.7±4.80 mm for fetal brains, and 5.9±2.43 mm for thorax CT.The evaluation used 85 adult brains, 34 fetal MRI scans, and 60 low-dose thorax CT scans.Source: §3 · p. 5

Exp. 1:

  • After four SVR iterations, reconstruction from ˆTi initialisation yields PSNR 29.5±2.43 on 15 randomly selected test volumes, versus 23.7 ± 1.09 without SVR.Source: Exp. 1 · p. 6

Exp. 2:

  • SVRNet accurately predicted fetal slice transformations in heavily motion-corrupted ssFSE data, and network inference took approximately 12 ms per slice.The experiment used three overlapping orthogonal stacks; slices were treated individually without their scanner-relative stack transformations.Source: Exp. 2 · p. 6

Exp. 3:

  • An example reconstruction gained 28 dB PSNR with additional SVR; DRR transformation errors averaged 106 mm translation and 5.6° plane rotation for healthy patients, versus 130 mm and 7.0° average error for pathological patients.The DRR test included nine healthy and ten randomly selected pathological volumes. The method was trained on healthy volunteer data.Source: Exp. 3 · p. 7
  • The method was demonstrated on fetal brain data with surrounding tissue and on low-dose thorax CT without organ-specific training, including X-ray-to-CT registration.The conclusion describes motion compensation for highly motion-corrupted scans and states temporal proximity is not required for initialization.Source: Exp. 3 · p. 8
  • SVRNet requires matching image intensity ranges, spacing, and translation-offset preprocessing, and training for the target region, organ, and imaging scenario.The training region need not be delineated accurately, and the method is not restricted to one imaging modality or scenario.Source: Exp. 3 · p. 8

Appendices

The appendices show slice-level prediction performance in referenced figures and illustrate three strategies for sampling planes, including their normals relative to the origin.

Appendices

  • Absolute per-slice prediction performance is shown in Figures 10, 15, and 17.Source: Appendices · p. 9

.1 Data generation illustration

.2 Cartesian co-ordinates to rotation and offset

The appendix describes converting predicted Cartesian points into a plane rotation using a selected origin, then using differentiable rotation-distance metrics as network loss layers for angle regression.

.2 Cartesian co-ordinates to rotation and offset

.3 Euler and Quaternion Loss Functions

Euler Back Propagation Function

The appendix gives sign-based derivatives of the angular distance with respect to each of its two angle arguments.

Euler Back Propagation Function

Quaternion Back Propagation Function

The appendix represents q1 and q2 as four-component quaternion vectors.

Quaternion Back Propagation Function

.4 Network details

Table 1 presents the SVRNet network topology, complementing Figure 1 with textual architecture details.

.4 Network details

  • Table 1 presents the SVRNet network topology and gives textual details complementary to Figure 1.The supplied passage does not specify individual layers or other architectural parameters.Source: .4 Network details · p. 10, Table 1 · p. 11

.5 Randomly selected illustrative inference results

The illustrative experiments compare input ground-truth slices with slices sampled at the locations predicted by SVRNet. They show results for test slices from registered volumes, motion-corrupted fetal MRI, and adult thorax data, including CT and digitally reconstructed radiographs.

.5 Randomly selected illustrative inference results

  • The examples compare each ground-truth image with an atlas slice sampled at the location predicted by SVRNet.The predicted transformation parameters reorient the slice into world coordinates; side-by-side views show where the slice is versus where the network predicts it is.Source: .5 Randomly selected illustrative… · p. 11
  • In Exp. 1, test slices from correctly registered and reconstructed volumes are compared with predictions sampled from the same volume.Fig. 9 notes that incorrectly predicted images are mainly at the boundaries of the trained region of interest; Fig. 10 shows prediction-error distributions for three adult-brain validation volumes.Source: .5 Randomly selected illustrative… · p. 11, Figure 9 · p. 12, Figure 10 · p. 12
  • In Exp. 2, motion-corrupted MRI slices are segmented and cropped, and an arbitrary fetal atlas is used to visualize predictions because queried images lack ground truth.Figs. 11 and 12 show examples labeled as correct and incorrect transformations.Source: .5 Randomly selected illustrative… · p. 13, Figure 11 · p. 13, Figure 12 · p. 13
  • In Exp. 3, the approach is tested on adult thorax data without specifically segmented organs, including CT and digitally reconstructed radiographs.The figures show correct and incorrect CT transformations, randomly selected DRR locations, and prediction-error distributions; most incorrect slices are in the sagittal plane.Source: .5 Randomly selected illustrative… · p. 14, Figure 13 · p. 14, Figure 14 · p. 14
Loading 1702.08891v2…