PaperlayerView exact revision on arXiv

Source-linked AI summary

A Deep Cascade of Convolutional Neural Networks for Dynamic MR Image Reconstruction

Jo Schlemper, Jose Caballero, Joseph V. Hajnal, Anthony Price, Daniel Rueckert

arXiv:1704.02422v2cs.CV

TL;DR

The paper addresses reconstruction of dynamic 2D cardiac MR images from aggressively undersampled Cartesian data, where undersampling causes aliasing and the inverse problem is ill-posed. It proposes a deep cascade of CNNs combining learned image-domain de-aliasing with data consistency and data sharing. The method outperforms the compared reconstruction approaches while reconstructing each frame in 23ms and each dynamic sequence in under 10s.

  • Problem

    Cartesian undersampling can accelerate slow MRI acquisition but produces aliasing and an ill-posed reconstruction problem requiring effective image priors.

  • Method

    A deep cascade of CNNs reconstructs undersampled cardiac MR frames using learned de-aliasing with data consistency and data sharing.

  • Results

    The method outperforms the compared compressed-sensing and low-rank approaches in reconstruction error and perceptual quality, with 23ms per frame and under 10s per sequence.

  • Takeaways & Limitations

    CNNs can learn a generic de-aliasing strategy for aggressive Cartesian undersampling while preserving most anatomical structures at very high undersampling rates.

Abstract

from arXiv · show

Inspired by recent advances in deep learning, we propose a framework for reconstructing dynamic sequences of 2D cardiac magnetic resonance (MR) images from undersampled data using a deep cascade of convolutional neural networks (CNNs) to accelerate the data acquisition process. In particular, we address the case where data is acquired using aggressive Cartesian undersampling. Firstly, we show that when each 2D image frame is reconstructed independently, the proposed method outperforms state-of-the-art 2D compressed sensing approaches such as dictionary learning-based MR image reconstruction, in terms of reconstruction error and reconstruction speed. Secondly, when reconstructing the frames of the sequences jointly, we demonstrate that CNNs can learn spatio-temporal correlations efficiently by combining convolution and data sharing approaches. We show that the proposed method consistently outperforms state-of-the-art methods and is capable of preserving anatomical structure more faithfully up to 11-fold undersampling. Moreover, reconstruction is very fast: each complete dynamic sequence can be reconstructed in less than 10s and, for the 2D case, each image frame can be reconstructed in 23ms, enabling real-time applications.

I. INTRODUCTION

MRI acquisition is slow, motivating Cartesian k-space undersampling, but undersampling creates aliasing and an ill-posed reconstruction problem. The paper investigates CNN-based de-aliasing for independently reconstructed and jointly reconstructed dynamic cardiac MR frames, incorporating learned representations, data fidelity, and data sharing.

  • Motivation: MRI acquisition is slow because k-space samples are acquired sequentially under physiological and hardware constraints.Undersampling can accelerate acquisition, but it introduces aliasing artefacts and requires prior knowledge about the image.
  • Existing approaches: Compressed sensing reconstructs sub-Nyquist MR data using compressibility, incoherent sampling, and nonlinear or iterative optimization.Dictionary learning extends fixed sparsifying transforms by learning adaptive representations from the data.
  • Existing approaches: Dynamic imaging can support more aggressive undersampling by exploiting spatio-temporal redundancy and other explicit data redundancy.These redundancies can make the initially underdetermined reconstruction problem determined or overdetermined.
  • Proposed approach: The paper proposes a cascade of CNNs for 2D cardiac MR reconstruction, treating undersampled-image recovery as image-domain de-aliasing.The approach is compared with DLMRI for independent frames and with DLTG, kt-SLR, and L+S for dynamic reconstruction.
  • Results: 23ms per image frame and under 10s per dynamic sequence demonstrate the proposed method's reconstruction speed.The paper reports that CNN reconstruction outperforms the compared methods in reconstruction error and perceptual quality.
  • Proposed approach: The CNN output incorporates k-space data fidelity through a data-consistency operation that preserves sampled coefficients according to acquisition noise.The reconstruction uses the undersampled Fourier model and regularization-inspired formulation to combine learned reconstruction with measured data.

III. DATA CONSISTENCY LAYER

The data consistency layer combines CNN predictions with acquired k-space measurements, balancing reconstruction output and measured data through a noise-weighted update. It is implemented as a differentiable Fourier-domain network layer for end-to-end training.

  • The CNN prediction is Fourier-encoded, and its k-space coefficients are combined with the undersampled measurements through a closed-form data-consistency solution.Unknown coefficients use the CNN prediction, while sampled coefficients combine the prediction and original measurement according to noise-related weighting.
  • The final image reconstruction is obtained by applying the inverse Fourier encoding to the data-consistent k-space representation.
  • The data consistency step can be treated as a trainable network layer because its forward and backward passes are explicitly defined.This permits CNN parameters to be optimized with stochastic gradient descent.
  • The data consistency operation is decomposed into Fourier transform, element-wise k-space correction, and inverse Fourier transform operations.
  • MR images are complex-valued, so the network accommodates them using real-valued operations in a space of twice the dimensionality.

IV. CASCADING NETWORK

The cascading network concatenates CNNs with data consistency operations to iteratively refine reconstructions. This unfolds an optimization-like reconstruction process and enables joint end-to-end training of the subnetworks.

  • A cascade alternates intermediate CNN de-aliasing with data consistency reconstruction instead of relying on a single one-step CNN.The motivation is to support iterative reconstruction while reducing concerns about overfitting and lengthy training for a single deep mapping.
  • The cascading CNN extends the dictionary-learning reconstruction pipeline by making the whole reconstruction process jointly trainable.
  • All subnetworks are trained end-to-end using the forward and backpropagation rules defined for the data consistency layer.
  • Data sharing aggregates missing k-space entries from up to ±nadj neighboring frames, averaging overlapping lines.

V. DATA SHARING LAYER

The data sharing layer exploits temporal redundancy by aggregating neighboring-frame k-space samples and supplying the resulting images as CNN inputs. This reduces apparent aliasing but can introduce inconsistency in dynamically changing regions.

  • 3D convolutions learn spatio-temporal features while neighboring k-space samples provide additional information when adjacent-frame content changes little.
  • For nadj = 2, overlapping lines make the resulting appearance similar to an acceleration factor around 4 rather than the theoretical 12/5 = 2.4.
  • Data sharing aggregates k-space lines from adjacent frames, producing images that resemble zero-filled reconstructions with lower undersampling.
  • Dynamic cardiac content around the heart creates data inconsistency when adjacent-frame samples are combined.The resulting images are therefore used as additional CNN inputs rather than as final reconstructions.
  • Data sharing layers generate multiple images across nadj values, concatenate them along the channel axis, and feed them into the first CNN convolution layer.This reframes reconstruction as joint estimation of aliasing and dynamic motion.

VI. ARCHITECTURE AND IMPLEMENTATION

The architecture uses residual CNN modules interleaved with data consistency layers, optionally augmented by data sharing for sequences. Experiments also define separate 2D and dynamic configurations and use variable-density Cartesian masks.

  • The cascade repeats CNN reconstruction modules and data consistency layers nc times, with each CNN using nd convolution layers and residual connections.The CNN uses 3D convolutions for sequences and projects the learned representation back to two image channels.
  • Data sharing generates images for nadj ∈ {0, 1, . . . , 5} and concatenates them before the first convolution in each subnetwork.
  • The model supports 2D reconstruction by setting Nt = 1 and using 2D convolutions, but data sharing is excluded in that setting.
  • The Cartesian mask fully samples kx while undersampling ky for each frame, always acquiring the 8 central lines.
  • The remaining k-space lines follow a zero-mean Gaussian variable-density distribution whose offset prevents the acquisition probability from reaching zero.

A. Setup

The study evaluates cardiac MR reconstruction using fully sampled cine scans, Cartesian undersampling, augmentation, and fixed-mask MSE assessment. The setup also notes practical trade-offs of Cartesian sampling and alternative trajectories.

  • Dataset: The dataset contains 10 fully sampled short-axis cardiac cine MR scans, each with 30 temporal frames and 32-channel data.Data were zero-filled to 256 × 256 and reconstructed with SENSE before evaluation.
  • Undersampling: Cartesian undersampling fully samples the frequency-encoding direction while randomly undersampling phase encodes, retaining the eight lowest spatial frequencies.Sampling probabilities for other frequencies follow a marginally offset Gaussian variable-density function.
  • Undersampling: Radial and spiral sampling could provide greater aliasing incoherence, but require nonuniform Fourier transforms and gridding that may propagate interpolation errors.
  • Data augmentation: Because the dataset contains only 300 images, training uses rigid transformations and elastic deformations to counter overfitting.
  • Evaluation: For 2D experiments, 150 frames from five subjects formed each training and testing set, and fixed undersampling masks enabled fair MSE comparisons.An initially used 3-2-5 training-validation-testing split did not produce decreasing validation error after three days of training.

B. Reconstruction of 2D Images

The architecture study finds that interleaving data consistency with CNN modules improves generalization, while additional cascading iterations reduce reconstruction error with diminishing returns after three layers.

  • Trade-offs between nd and nc: D5-C2 achieved lower training and test errors with a tighter gap than D11-C1, indicating better generalisability.D11-C1 began overfitting after about 1.2 × 10^5 backpropagations; the authors associate D5-C2’s advantage with two data consistency steps.
  • Experimental setup: The cascading-iteration experiment initialized each network greedily from the previously trained network and performed 10^5 backpropagations per nc.Random initialization could achieve better results because greedy initialization may become stuck in suboptimal local minima.
  • Effect of cascading iterations nc: After nc = 3, increasing cascading iterations continued reducing test error but at a reduced rate.Deeper models also had lower error standard deviation, despite tending to overfit more.
  • Effect of cascading iterations nc: Increasing nc reconstructed more subtle image details and left less noise-like aliasing in the images.
  • Effect of cascading iterations nc: The D5-C5 intermediate outputs generally became progressively sharper as each cascading subnetwork refined the reconstruction.The fourth subnetwork appeared grainier than its predecessor despite monotonically decreasing reconstruction error.

3) Comparison with DLMRI:

For independently reconstructed 2D cardiac MR frames, CNN outperformed DLMRI at 3-fold and 6-fold acceleration while preserving texture better and reconstructing far faster.

  • Experimental setup: The 2D comparison used two-way cross-validation across 10 subjects, with CNN parameters nd = 5 and nc = 5.DLMRI reconstructed each subject using a jointly trained dictionary over all time frames and 400 iterations.
  • Quantitative comparison: CNN consistently outperformed DLMRI at both 3-fold and 6-fold acceleration, with smaller reconstruction-error standard deviations.
  • Visual comparison: At 6-fold undersampling, CNN preserved texture better than DLMRI, whose reconstruction contained block-like artifacts from over-smoothing.Both methods lost significant structure, but CNN predicted some anatomical details unavailable in the DLMRI reconstruction.
  • Reconstruction speed: 23 ± 0.1 milliseconds per slice enabled CNN inference on a GeForce GTX 1080, whereas DLMRI required about 6.1 ± 1.3 hours per subject on CPU.The authors attribute DLMRI’s bottleneck to repeated dictionary-learning and sparse-coding iterations.

1) Effect of Data Sharing:

For dynamic sequences, data sharing improves reconstruction under aggressive undersampling, and the CNN consistently outperforms competing methods while retaining anatomical motion and enabling fast offline reconstruction.

  • Comparison with State-of-the-art: CNN consistently outperformed state-of-the-art methods across all tested undersampling factors, with larger advantages under more aggressive undersampling.The dynamic comparison evaluated 3-, 6-, 9-, and 11-fold acceleration against DLTG, kt-SLR, and L+S.
  • Effect of data sharing: Data sharing removed blocky artifacts present in CNN without data sharing and enabled better performance than DLTG under severe aliasing.Without data sharing, CNN exceeded DLTG at 3-fold acceleration but not under more aggressive undersampling.
  • Temporal reconstruction: Temporal profiles showed that CNN reconstructed the correct motion with smaller errors than the other methods, despite data sharing causing inconsistency in highly dynamic regions.The CNN corrected this inconsistency internally.
  • Reconstruction speed: 8.21s ± 0.02s was the average time to reconstruct each dynamic sequence on a GeForce GTX 1080, compared with 6.6 hours per subject for DLTG on CPU.The temporal axis makes CNN reconstruction slower than 2D reconstruction, but the method remains reasonably fast for offline use.

D. Memory Requirement

The CNN memory footprint depends on parameters and intermediate activations, with substantially lower requirements during testing than training.

  • Each convolution layer has (kxkyktn′f + 1)nf parameters, including one bias term per current feature.The kernel dimensions and incoming and current feature counts determine the layer parameter total.
  • 0.6 million parameters occupy 2.3MB for 2D D5-C5 reconstruction, while 3.4 million parameters occupy about 13.6MB for dynamic D5-C10(S).These estimates assume single-precision floating-point storage.
  • Training requires more than three times the parameter count for gradient computation and storage of intermediate activation maps.For the proposed architecture, activation-map memory is mainly associated with convolution layers.
  • Testing uses much less memory because intermediate activation maps need not be stored for a forward pass.The overhead is limited to the single largest activation map, scaled by implementation-specific factors.

VIII. DISCUSSION AND CONCLUSION

The study finds that interleaved data consistency enables CNN-based reconstruction of undersampled cardiac MR images, including at aggressive undersampling rates. It also identifies scope boundaries involving training data, acquisition settings, sampling patterns, and generalisation to pathology or more variable content.

  • CNN reconstruction outperformed CS and low-rank methods empirically, despite lacking their mathematical signal-recovery guarantee.At very aggressive undersampling, CNNs reconstructed most anatomical structures more accurately based on learned priors.
  • Training on a small dataset was feasible using local subnetworks and intensive data augmentation to reduce overfitting.The authors speculate that more training data could support learning coarse features with dilated or strided convolutions.
  • The experiments used retrospectively undersampled single-coil-equivalent data derived from an original SENSE reconstruction, not full-array parallel-MR data.Applying CNN reconstruction to full-array coil data remains to be investigated, and could improve performance while increasing architectural complexity.
  • Arbitrary Cartesian masks at a fixed sampling rate and pre-training across rates suggest the network learned a generic de-aliasing strategy.Tolerance to radial and spiral trajectories remains to be validated; jointly optimising the undersampling mask is another proposed direction.
  • The approach remains to be evaluated on pathology, more variable content, and larger datasets despite cross-validation on unseen data from the same acquisition protocol.Application-specific priors and multi-task learning are suggested as ways to improve clinical utility when suitable expert knowledge is available.
Loading 1704.02422v2…