PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Modal Transformer for Accelerated MR Imaging

Chun-Mei Feng, Yunlu Yan, Geng Chen, Yong Xu, Ling Shao, Huazhu Fu

arXiv:2106.14248v3eess.IVcs.CV

TL;DR

Accelerated multi-modal MR imaging needs better fusion of complementary modalities and more global context than common CNN-based or simple auxiliary-prior methods provide. MTrans uses a multi-scale, dual-branch transformer with cross attention to exchange modality information, and it outperforms competing methods across reconstruction, super-resolution, and multi-coil evaluations.

  • Problem

    Existing accelerated multi-modal MR methods often treat the auxiliary modality as prior information, while CNNs have limited ability to capture long-range dependencies and prior transformers overlook cross-modal multi-scale correlations.

  • Method

    MTrans uses a transformer architecture with cross attention to fuse multi-scale features between target and auxiliary modalities, combining local target details with broader auxiliary context.

  • Results

    MTrans outperforms competing multi-modal methods across MR reconstruction, super-resolution, and multi-coil reconstruction evaluations.

  • Takeaways & Limitations

    Multi-scale cross-modal transformer fusion provides a unified approach for restoring target MR modalities in accelerated reconstruction and super-resolution settings.

Abstract

from arXiv · show

Accelerated multi-modal magnetic resonance (MR) imaging is a new and effective solution for fast MR imaging, providing superior performance in restoring the target modality from its undersampled counterpart with guidance from an auxiliary modality. However, existing works simply combine the auxiliary modality as prior information, lacking in-depth investigations on the potential mechanisms for fusing different modalities. Further, they usually rely on the convolutional neural networks (CNNs), which is limited by the intrinsic locality in capturing the long-distance dependency. To this end, we propose a multi-modal transformer (MTrans), which is capable of transferring multi-scale features from the target modality to the auxiliary modality, for accelerated MR imaging. To capture deep multi-modal information, our MTrans utilizes an improved multi-head attention mechanism, named cross attention module, which absorbs features from the auxiliary modality that contribute to the target modality. Our framework provides three appealing benefits: (i) Our MTrans use an improved transformers for multi-modal MR imaging, affording more global information compared with existing CNN-based methods. (ii) A new cross attention module is proposed to exploit the useful information in each modality at different scales. The small patch in the target modality aims to keep more fine details, the large patch in the auxiliary modality aims to obtain high-level context features from the larger region and supplement the target modality effectively. (iii) We evaluate MTrans with various accelerated multi-modal MR imaging tasks, e.g., MR image reconstruction and super-resolution, where MTrans outperforms state-of-the-art methods on fastMRI and real-world clinical datasets.

I. INTRODUCTION

Accelerated multi-modal MR imaging uses complementary modalities to recover high-quality images faster, but prior methods often underuse cross-modality relationships and rely on CNNs with limited global context. MTrans addresses these gaps with a multi-scale transformer and cross attention, and is evaluated across reconstruction and super-resolution tasks.

  • Motivation: Accelerated MR imaging reconstructs high-quality images from undersampled measurements to reduce scan-time costs and aliasing artifacts.Reconstruction removes undersampling artifacts, while super-resolution enhances image resolution.
  • Motivation: Different MR modalities provide complementary structural and modality-specific information, enabling faster modalities to guide restoration of slower target modalities.Examples include PDWI and FS-PDWI, which provide complementary structural and tissue-contrast information.
  • Research gap: Existing transformer-based MR methods largely focus on single modalities and overlook cross-modal correlations, especially across different scales.This motivates a dedicated multi-modal representation and fusion strategy.
  • Proposed approach: MTrans introduces a transformer architecture with cross attention to fuse multi-scale features from target and auxiliary modalities while capturing global information beyond CNN locality.The method combines large-region context from auxiliary features with local target details.
  • Evaluation: The framework is evaluated on reconstruction and super-resolution using fastMRI and raw clinical MR datasets, with results reported as superior in qualitative and quantitative comparisons.The evaluation covers two accelerated MR imaging tasks and compares against multi-modal alternatives.

B. Multi-Modal Medical Image Representation

The paper formulates accelerated MR imaging around real k-space degradation and uses complementary modalities to restore a target image. MTrans processes modality-specific multi-scale features through a dual-branch transformer that exchanges information between branches before producing restored images.

  • Multi-modal representation: Multi-modal MR fusion combines complementary modalities rather than treating the auxiliary modality only as a simple concatenated input.Prior methods commonly concatenate modalities at feature or image scale, while MTrans develops a dedicated fusion strategy.
  • Accelerated MR imaging: The accelerated-imaging formulation uses fully sampled complex k-space data to create zero-filled reconstruction inputs and degraded super-resolution inputs.The framework addresses both aliasing-artifact removal and restoration from resolution-degraded images.
  • Accelerated MR imaging: For reconstruction, undersampled k-space is formed with a binary mask and transformed back into a zero-filled image before target restoration.The target is restored using an additional modality with shared structural information.
  • Accelerated MR imaging: For super-resolution, the method truncates outer k-space regions before applying the Fourier transform, better matching MR acquisition and avoiding checkerboard artifacts.This replaces image-domain blur-based degradation with a k-space-based process.
  • MTrans architecture: MTrans uses two branches with modality-specific heads, a multi-modal transformer for cross-branch feature exchange, and target-side tails that map fused features to restored images.The current branch supplies queries while the other branch supplies complementary information.

A. Heads

MTrans uses separate heads to extract modality-specific feature maps, then divides them into differently sized patches for cross-modal transformer processing. Smaller target patches preserve detail, while larger auxiliary patches provide broader context.

  • A. Heads: Separate auxiliary and target heads extract modality-specific feature maps from the input images.Each head consists of three 3×3 convolutional layers.
  • A. Heads: The auxiliary head processes the fully sampled auxiliary image into a feature map Faux ∈ R^C×H×W.C denotes the channel count, while H and W denote feature-map height and width.
  • A. Heads: The target head processes a zero-filled target image for reconstruction or an LR target image for super-resolution.For super-resolution, the target feature map is spatially scaled according to the factor s.
  • Patch representation: MTrans reshapes auxiliary and target feature maps into patch sequences and adds learnable positional encodings before transformer processing.The sequence length is determined by the number of image patches.
  • Patch representation: The target image patches are one quarter the auxiliary patch size, preserving finer target details while auxiliary patches capture broader context.The differently sized patches are designed to produce stronger complementary image features.
  • Cross-modal fusion: Cross-transformer encoders update each modality using information from the other branch, with the target encoder receiving auxiliary features and vice versa.The cascaded encoders perform bidirectional cross-modal fusion.

1) Cross Transformer Encoder:

The cross transformer encoder fuses target and auxiliary modalities through cascaded, cross-updating branches. Its cross attention modules align dimensions and absorb auxiliary features that support target-modality restoration.

  • Auxiliary features are linearly projected to align dimensions before cross-modal fusion.
  • The architecture cascades several cross transformer encoder modules for multi-modal feature fusion.Green arrows indicate information updates, while red arrows indicate information exchange between modalities.
  • Layer normalization and a feed-forward network further process fused features before passing them to the next encoder.The feed-forward network uses two linear transformations with a ReLU activation.
  • The improved cross attention module absorbs auxiliary features that contribute to the target modality.

2) Cross Attention Module:

The cross attention module uses dimensionally aligned features from opposite branches to model correspondence between modalities. Different feature sizes allow the module to combine high-level context with local details, while deeper stages focus increasingly on brain regions.

  • Different feature sizes let cross attention learn both high-level context features and local details.
  • Target-branch queries interact with auxiliary-branch keys and values, while auxiliary-branch queries interact with target-branch keys and values.
  • The module finds cross-modal correspondence through a bilinear attention model.
  • The module outputs cross-attended features for the target and auxiliary branches.These outputs are defined for each branch and continue the cross-modal processing pipeline.
  • Across deeper encoder stages, attention shifts from background areas toward brain or object regions.The four bright spots represent areas focused on by four attention heads.

C. Tails

The transformer outputs are passed to tail modules that predict restored images for the target and auxiliary modalities.

  • The final multi-modal transformer features are fed into two tail modules to predict restored images.
  • Each reconstruction tail contains three convolutional layers.
  • For super-resolution, the target tail adds a sub-pixel convolutional layer to upscale its output.
  • The target output is an image of size H × W.

D. Loss

The model uses an L1 loss over both modalities, with a weighting factor controlling the trade-off between their contributions.

  • MTrans evaluates its results with an L1 loss.
  • The parameter α weights the trade-off between the target and auxiliary modalities.
  • The auxiliary-modality loss helps encode that modality and fully extract its features.

V. EXPERIMENTS

The experiments evaluate MTrans on raw MR datasets across reconstruction and super-resolution, using single- and multi-modal baselines under varied acceleration settings.

  • Datasets: Three raw MR datasets are used to evaluate reconstruction and super-resolution, including fastMRI and uiMRI.PDWI guides FS-PDWI restoration in fastMRI; the datasets include raw MR images.
  • Experimental settings: Reconstruction uses random and equispaced undersampling at 4×, 6×, and 8× accelerations, while super-resolution uses 4× enlargement.Low-resolution images are generated by truncating the outer k-space region before inverse Fourier transformation.
  • Baselines: The comparison includes single-modal methods such as EDSR, UNet, TransMRI, MoDL, and LORAKS, alongside multi-modal methods including MDUNet, HyperDense-Net, Transmed, rsGAN, PRO, and MCSR.Baseline hyperparameters are optimized or specified for the experiments.
  • Evaluation: Figures compare reconstructed or super-resolved images, error maps, and PSNR/SSIM measurements across single-modal and multi-modal methods.Figure 5 covers uiMRI reconstruction, while Figure 6 covers fastMRI super-resolution.
  • Implementation: The model is implemented in PyTorch and trained for 50 epochs with SGD on four NVIDIA Tesla V100 GPUs.Training uses a learning rate of 1e-4, batch size 8, and four cross-attention heads.

B. Results on MR Image Reconstruction

MTrans improves accelerated MR reconstruction and super-resolution by combining complementary modalities, with the strongest reported gains over multi-modal CNN baselines in super-resolution.

  • Quantitative reconstruction: MTrans achieves 29.3 dB and 28.4 dB PSNR on fastMRI and 31.7 dB and 30.8 dB on uiMRI reconstruction settings.Reconstruction is evaluated with SSIM, PSNR, and NMSE against fully sampled ground truth images.
  • Qualitative evaluation: On uiMRI reconstruction, multi-modal methods reduce errors beyond single-modal methods, and MTrans produces the lowest reconstruction error.The qualitative comparison uses reconstructed images and blue error maps.
  • Qualitative evaluation: On fastMRI 4× super-resolution, MTrans produces clearer details, minimal checkerboard effects, and less structural loss than the compared methods.The qualitative comparison separates single-modal and multi-modal results by rows.

2) Qualitative Evaluation:

Additional experiments test multi-coil reconstruction, joint undersampling, fusion strategies, and model components to examine the breadth and sources of MTrans performance.

  • D. Results on Multi-coil Data: MTrans achieves the best PSNR and SSIM among the compared multi-coil reconstruction methods.Its reconstructed images are reported to have the lowest texture error and to be close to ground truth.
  • Statistical analysis: Paired Student’s t-tests are used to assess differences, with P < 0.001 treated as statistically significant in the reported analyses.The experiments include fusion-strategy, cross-attention, transformer, and CNN-attention ablations.
  • 1) Comparison with Joint Reconstruction:: When both modalities are undersampled, MTJoint obtains PSNR = 29.2 dB and SSIM = 0.630 on the target modality.On the auxiliary modality, MTJoint obtains PSNR = 29.3 dB and SSIM = 0.635.
  • 1) Comparison with Joint Reconstruction:: MTrans outperforms MTJoint on the target modality when the auxiliary modality is fully sampled.The passage attributes this difference to the detailed supplementary information available from the fully sampled auxiliary modality.
  • Ablation study: The ablation study compares early fusion in ETransMRI with single-scale fusion in ITransMRI and the proposed cross multi-modal transformer.These variants are used to evaluate key components and fusion strategies.

2) Comparison of Different Fusion Schemes:

MTrans benefits from feature-level cross-modal fusion and transformer-based long-range modeling, producing clearer multi-modal MR reconstruction and super-resolution results than alternative fusion or CNN-attention schemes.

  • Fusion schemes: Feature-level fusion provides richer supplementary information than simple early fusion, with ETransMRI performing worst among compared schemes.The comparison includes MTJoint, whose target and auxiliary inputs are both undersampled.
  • Fusion schemes: MTrans achieves the lowest texture error and results nearly as clear as ground truth in qualitative reconstruction and super-resolution comparisons.The visual comparison uses fastMRI error maps across reconstruction and SR results.
  • Fusion schemes: CNN-based self-attention and channel-spatial attention remain below MTrans in both reconstruction and super-resolution performance.The comparison attributes the advantage to transformer multi-head attention for modeling long-range dependency.
  • Fusion schemes: The best trade-off occurs at α = 0.9, where target-modality influence is increased without eliminating auxiliary-modality influence.Larger α values favor the target modality and reduce the auxiliary modality’s influence.
  • Fusion schemes: Using the same loss across baselines isolates architecture-related differences in the reported comparisons.The accompanying table reports standard deviations and paired Student’s t-test results with P < 0.001.

4) Baseline Comparisons with Same Loss:

The ablations support MTrans’s transformer and cross-attention design, while parameter analysis identifies a substantial memory-cost trade-off and a practical patch-count setting.

  • Baseline comparisons with same loss: Removing the transformer causes dramatic reconstruction and super-resolution performance drops because long-range relationships are not modeled well.The comparison includes a transformer without the cross-attention module and a model without the transformer.
  • Baseline comparisons with same loss: MTrans uses 189.2M parameters, substantially more than the listed CNN baselines, while achieving the best reconstruction and super-resolution performance.The authors identify reducing memory cost as future work.
  • Baseline comparisons with same loss: PSNR and SSIM improve as patch count N increases, but gains become marginal beyond N = 400; therefore, the experiment uses N = 400.N = 1600 would require substantially more parameters than N = 400.
  • Baseline comparisons with same loss: Across reconstruction and super-resolution tasks on fastMRI and clinical datasets, MTrans is reported to outperform state-of-the-art methods while reducing artifacts.The framework is presented as applicable to both MR image reconstruction and super-resolution.
Loading 2106.14248v3…