PaperlayerView exact revision on arXiv

Source-linked AI summary

Convolutional Recurrent Neural Networks for Dynamic MR Image Reconstruction

Chen Qin, Jo Schlemper, Jose Caballero, Anthony Price, Joseph V. Hajnal, Daniel Rueckert

arXiv:1712.01751v3cs.CV

TL;DR

Accelerated dynamic MRI must reconstruct images from undersampled k-space while resolving aliasing and exploiting temporal correlation. CRNN-MRI embeds iterative reconstruction structure in a convolutional recurrent network with recurrence across iterations and time, and the authors report improved accuracy and speed with fewer parameters than competing approaches.

  • Problem

    Accelerated dynamic MRI reconstruction must resolve aliasing from undersampled k-space while exploiting temporal and spatio-temporal dependencies.

  • Method

    CRNN-MRI combines a learned CRNN proximal operator and data consistency with recurrent connections across optimization iterations and bidirectional recurrence across time.

  • Results

    The proposed methods outperform competing MR reconstruction methods in reconstruction accuracy and speed while using significantly fewer parameters than 3D CNN models.

  • Takeaways & Limitations

    The architecture jointly models temporal dependency and iterative reconstruction, combining benefits of traditional iterative methods and deep learning for highly undersampled dynamic MRI.

Abstract

from arXiv · show

Accelerating the data acquisition of dynamic magnetic resonance imaging (MRI) leads to a challenging ill-posed inverse problem, which has received great interest from both the signal processing and machine learning community over the last decades. The key ingredient to the problem is how to exploit the temporal correlation of the MR sequence to resolve the aliasing artefact. Traditionally, such observation led to a formulation of a non-convex optimisation problem, which were solved using iterative algorithms. Recently, however, deep learning based-approaches have gained significant popularity due to its ability to solve general inversion problems. In this work, we propose a unique, novel convolutional recurrent neural network (CRNN) architecture which reconstructs high quality cardiac MR images from highly undersampled k-space data by jointly exploiting the dependencies of the temporal sequences as well as the iterative nature of the traditional optimisation algorithms. In particular, the proposed architecture embeds the structure of the traditional iterative algorithms, efficiently modelling the recurrence of the iterative reconstruction stages by using recurrent hidden connections over such iterations. In addition, spatiotemporal dependencies are simultaneously learnt by exploiting bidirectional recurrent hidden connections across time sequences. The proposed algorithm is able to learn both the temporal dependency and the iterative reconstruction process effectively with only a very small number of parameters, while outperforming current MR reconstruction methods in terms of computational complexity, reconstruction accuracy and speed.

I. INTRODUCTION

Dynamic MRI acquisition is limited by hardware, physiology, and sampling constraints, motivating undersampled reconstruction methods that exploit spatiotemporal redundancy. The paper proposes CRNN-MRI, which combines iterative reconstruction structure with recurrent modeling across iterations and time.

  • Motivation: Undersampling k-space accelerates acquisition but introduces image-domain aliasing because it violates the Nyquist sampling theorem.Traditional reconstruction regularizes the solution using assumptions such as smoothness, sparsity, or spatio-temporal redundancy.
  • Motivation: Deep learning methods learn prior information and regularization from data, but dynamic MRI reconstruction remains relatively underexplored.The paper identifies the optimal exploitation of spatio-temporal redundancy as a core architectural question.
  • Proposed method: CRNN-MRI models accelerated dynamic MRI reconstruction as a network combining a CRNN proximal-operator block with a data-consistency layer.The architecture is derived from variable splitting and alternate minimization.
  • Proposed method: Recurrent connections share reconstruction information across iterations, while bidirectional recurrent units propagate contextual information across temporal frames.The architecture jointly learns representations evolving over both reconstruction iterations and time sequences.
  • Results: The authors report improved reconstruction accuracy and speed over current dynamic MR methods, including 3D CNN comparisons with fewer parameters.They also report that the proposed methods outperform 3D CNNs at different undersampling rates and speeds.
  • Related work: Earlier work includes compressed-sensing, low-rank, end-to-end CNN, and optimization-unrolling approaches, whereas this paper adds hidden connections across optimization iterations.The comparison is framed as propagating learned representations rather than independently sharing or parameterizing each stage.

III. CONVOLUTIONAL RECURRENT NEURAL NETWORK FOR MRI RECONSTRUCTION

The reconstruction problem represents dynamic complex-valued MR images and undersampled k-space measurements through a non-convex regularized optimization formulation. CRNN-MRI replaces explicit proximal regularization with a recurrent learned operator interleaved with data consistency while propagating information across iterations.

  • Problem formulation: The target sequence x contains D = DxDyT complex-valued image elements, while y contains M << D undersampled k-space measurements.Dx and Dy denote frame width and height, and T denotes the number of frames.
  • Problem formulation: Dynamic MR reconstruction is formulated as an unconstrained optimization problem using undersampled Fourier encoding, regularization R, and a data-fidelity weighting parameter λ.The regularizer can encode sparsity or low-rank structure, depending on the reconstruction method.
  • Iterative optimization: Because the formulation is generally non-convex, variable splitting decouples its terms and alternate minimization produces iterative updates over x and z.The variable z serves as an intermediate state in the optimization process.
  • Data consistency: The data-consistency update uses the acquired k-space samples to enforce fidelity, with x initialized from the zero-filled reconstruction.The zero-filled reconstruction is denoted u_y, and z is the intermediate optimization state.
  • CRNN formulation: CRNN directly learns the proximal operator associated with the prior R instead of explicitly specifying the regularization form.The learned proximal operation is interleaved with data-consistency processing.
  • CRNN formulation: Unlike independently parameterized unrolled stages, one recurrent model propagates information across optimization iterations and adapts to changes between stages.This formulation models each stage as a learned recurrent forward-encoding step.

B. CRNN for MRI reconstruction

The CRNN-MRI network represents iterative MRI reconstruction with recurrent architecture, combining learned reconstruction updates, data consistency, and temporal processing for undersampled dynamic images.

  • CRNN is inspired by iterative optimisation and uses a recurrent network to model the reconstruction process.The network function is applied across optimisation iterations, with recurrent hidden connections propagating representations between stages.
  • Each iteration combines a learnable CRNN block with a data consistency layer corresponding to the classical data fidelity term.The CRNN produces an intermediate reconstruction, while the data consistency layer uses acquired k-space samples.
  • The CRNN block jointly models recurrence across optimisation iterations and bidirectional recurrence across temporal frames.Its components include BCRNN-t-i, CRNN-i, convolutional layers, residual connections, and data consistency layers.
  • Unlike conventional unrolled networks with independently parameterised stages, the proposed architecture propagates learned representations across iterations.Figure 1 contrasts traditional optimisation, unrolled convolutional networks with data consistency, and the proposed architecture modelling optimisation recurrence.

1) CRNN-i:

CRNN-i treats optimisation iterations as recurrent steps, preserving hidden representations across stages while sharing parameters and expanding spatial context.

  • CRNN-i: CRNN-i views each optimisation iteration as a sequential step in a vanilla recurrent neural network.Information is propagated between iterations through hidden states representing the sequence of frames.
  • CRNN-i: Hidden-to-hidden connections pass contextual spatial features from previous iterations to future reconstruction stages.This lets each stage use hidden features as well as the output image and can avoid redundant computation through memorised features.
  • CRNN-i: The effective spatial receptive field expands with iteration number, unlike a CNN that resets its receptive field at each stage.The expanded contextual support is described as improving reconstruction refinement across iterations.
  • CRNN-i: Shared weight parameters across iterations reduce the number of parameters compared with independently unrolled convolutional filters.The paper suggests this parameter sharing may offer better generalization properties.
  • CRNN-i: The implementation uses a vanilla RNN for simplicity, although LSTM and GRU units could provide better memory properties at higher computational cost.The alternative recurrent units are presented as natural generalisations rather than as evaluated components here.

2) BCRNN-t-i:

BCRNN-t-i jointly propagates information through time and optimisation iterations, separating temporal dynamics from spatial refinement while processing dynamic MRI sequences.

  • BCRNN-t-i: BCRNN-t-i uses bidirectional recurrent connections to model temporal and iteration dependencies jointly.The unit processes information from past and future time frames while also receiving representations from the previous optimisation iteration.
  • BCRNN-t-i: At the first layer, the unit uses undersampled input frames initially and intermediate reconstruction frames on later iterations.The hidden states are indexed by layer, time frame, and iteration; boundary hidden states are initialized to zero.
  • BCRNN-t-i: Temporal connections propagate information across all T frames, allowing the network to learn differences and correlations between successive frames.The recurrent responses over time focus on modelling temporal changes across the sequence.
  • BCRNN-t-i: Recurrent convolutions over iterations focus on spatial refinement across consecutive reconstruction steps.The paper distinguishes their role from temporal recurrent convolutions, which model changing dynamics across frames.
  • BCRNN-t-i: A CRNN block maintains hidden states H(i)l,t across its L layers and T time frames.Each hidden state represents feature information for one layer and time frame at iteration i.

C. Network Learning

Network learning minimizes pixel-wise mean squared error between reconstructed dynamic MR images and fully sampled ground truth data.

  • The training objective is the pixel-wise mean squared error between predicted reconstructions and fully sampled ground-truth images.Training uses input-target pairs (xu, xt) from dataset S.
  • The trainable parameters include input, iteration, and temporal convolution filters together with bias terms.The parameter set is θ = {Wl, Wi, Wt, Bl}.
  • The assumed numbers of time sequences T and optimisation iterations N must be specified before reconstruction.The paper treats both quantities as free network parameters.
  • The network uses He initialization, Adam optimization, and gradient clipping to [−5, 5] during training.Gradient clipping is used to mitigate gradient explosion.

A. Dataset and Implementation Details

The study evaluates CRNN-MRI on retrospectively undersampled cardiac cine data using cross-validation and a fixed recurrent architecture.

  • Dataset: The dataset contains 10 fully sampled short-axis cardiac cine MR scans, each with 30 temporal frames and 32-channel raw data.Scans use SSFP acquisition, a 320 × 320 mm field of view, 10 mm thickness, and a 192 × 190 sampling matrix zero-filled to 256 × 256.
  • Architecture: The experimental CRNN block uses one BCRNN-t-i layer, three CRNN-i layers, and one CNN layer per iteration.Proposed-A uses 64 filters and Proposed-B uses 128 filters, with kernel size k = 3 throughout the recurrent units.
  • Training setup: Training uses 256 × 32 × T patches, with Dpatch = 32 extracted along the frequency-encoding direction.The evaluation uses Cartesian undersampling and retrospective gating.
  • Evaluation: Evaluation uses 3-fold cross-validation, training on 7 subjects and testing on 3 in two folds, and training on 6 and testing on 4 in the remaining fold.This split is applied across the 10-subject dataset.

B. Evaluation Method

The evaluation compares CRNN-MRI with compressed-sensing and 3D CNN baselines using complementary image-quality, capacity, and runtime measures.

  • Compared methods: CRNN-MRI is compared with k-t FOCUSS, k-t SLR, 3D CNN-S, and 3D CNN.The 3D CNN baselines replace recurrent and 2D convolutional units with 3D convolutions; 3D CNN-S shares weights across iterations.
  • Metrics: The study reports MSE, PSNR, SSIM, HFEN, network capacity, and reconstruction time.Table I reports mean values with standard deviations across subjects.
  • Metrics: MSE and PSNR measure overall reconstruction accuracy, SSIM emphasizes perceived image quality, and HFEN measures fine features and edges.The metrics are selected for complementary evaluation of reconstruction quality.

C. Results

CRNN-MRI improves reconstruction quality across acceleration rates while using fewer parameters and less computation than the evaluated baselines. Its recurrent design also supports iterative refinement and temporal modeling, though the chosen architecture is not optimal.

  • Quantitative results: Proposed-A and Proposed-B outperform k-t FOCUSS, k-t SLR, 3D CNN-S, and 3D CNN across acceleration rates despite smaller network capacities.Proposed-A has capacity comparable to 3D CNN-S, while Proposed-B has about one third the capacity of 3D CNN.
  • Quantitative results: Increasing the proposed model’s parameter count substantially improves reconstruction results across all acceleration rates and reported metrics.The paper therefore focuses subsequent results on Proposed-B.
  • Spatial and temporal reconstructions: At 9× acceleration, Proposed-B produces more faithful myocardium reconstructions than the compared methods in spatial visualizations and error maps.The authors associate this with recurrent networks using a larger receptive field for aliasing characteristics within the anatomy.
  • Iteration analysis: Figure 3 examines how Proposed-B mean PSNR changes with test-time iteration count under different acceleration factors.AF denotes acceleration factor.
  • Runtime: The proposed RNN reconstruction is faster than the 3D CNN approaches because it avoids redundant, computationally expensive 3D convolutions.Reported runtimes use a GeForce GTX 1080 for the neural networks and CPU measurements for k-t FOCUSS and k-t SLR.
  • Spatial and temporal reconstructions: The visual comparisons include ground truth, undersampled input, Proposed-B, 3D CNN, 3D CNN-S, k-t FOCUSS, and k-t SLR reconstructions with error maps.Figures 5 and 6 compare spatial and temporal dimensions separately.
  • Recurrence analysis: Removing temporal recurrence reduces the problem to single-frame reconstruction, whereas temporal redundancy can be exploited between dynamic frames.This analysis uses undersampling factor 9 and two iterations.
  • Architecture variants: Adding temporal recurrent connections across all layers improves results over Proposed-A but increases computation and especially training time.The selected architecture reflects a trade-off among performance, training time, and hardware constraints.

E. Feature Map Analysis

The feature-map analysis finds that CRNN learns less redundant and more anatomically differentiated representations than the compared 3D CNN architectures.

  • Feature-map similarity: CRNN feature maps are geometrically more orthogonal than those of 3D-CNN and 3D-CNN-S at layers 1 and 4.The analysis compares representations across 10 cascades using cosine similarity between feature maps.
  • Feature-map similarity: Layer 1 shows higher feature redundancy than layer 4 across the examined networks.CNN-S and CRNN also show diagonal similarity patterns associated with parameter sharing across cascades.
  • Anatomical sensitivity: CRNN activations differ substantially from CNN activations and selectively respond to dynamic regions and areas around the ventricles.Representative layer-4 maps were selected from 16 clusters and color-coded by activation strength.

V. DISCUSSION

The discussion attributes CRNN-MRI’s reconstruction behavior to recurrent information propagation across iterations and time, while identifying motion and single-coil evaluation as important boundaries.

  • Reconstruction performance: CRNN-MRI produces faithful reconstructions from highly undersampled data across quantitative metrics and error-map inspection.The paper reports higher accuracy than 3D CNN models despite substantially lower network capacity.
  • Recurrent architecture: CRNN’s hidden connections propagate information across optimization steps, while bidirectional temporal recurrence propagates context across the sequence.The architecture combines iterative reconstruction recurrence with temporal modeling.
  • Reconstruction performance: CRNN improves background denoising relative to 3D CNNs, which are described as exploiting only local information because of their smaller receptive fields.The discussion links CRNN’s larger receptive fields to improved background denoising.
  • Feature representations: CRNN feature maps have smaller pair-wise cosine distances than 3D-CNN features, consistent with less redundant representations.The paper speculates that hidden iteration connections help propagate information and reduce repetitive feature-map computation.
  • Limitations and future work: The analysis reports that most reconstruction errors occur in moving regions, identifying motion preservation as an unresolved challenge.The current analysis also considers only a single-coil setup; multi-coil reconstruction is proposed for future investigation.
  • Overall conclusion: CRNN-MRI outperforms competing methods in reconstruction accuracy and speed across different undersampling rates.The conclusion describes the network as combining iterative denoising with interleaved data-consistency layers.
Loading 1712.01751v3…