PaperlayerView exact revision on arXiv

Source-linked AI summary

Unsupervised MRI Reconstruction via Zero-Shot Learned Adversarial Transformers

Yilmaz Korkmaz, Salman UH Dar, Mahmut Yurt, Muzaffer Özbey, Tolga Çukur

arXiv:2105.08059v3eess.IVcs.CVcs.LG

TL;DR

Supervised MRI reconstruction depends on paired data and imaging-operator supervision, while DIP uses randomly initialized CNN priors with limited long-range modeling. SLATER pre-trains an unconditional adversarial MRI prior with cross-attention transformers and adapts it zero-shot to undersampled data, achieving superior performance to the reported unsupervised baselines on brain MRI.

  • Problem

    Supervised reconstruction requires paired acquisitions and imaging-operator supervision, while DIP priors may be weakly matched to MR data and CNNs may miss long-range spatial interactions.

  • Method

    SLATER pre-trains an unconditional adversarial model with cross-attention transformers, then optimizes its prior and imaging consistency during zero-shot reconstruction without extra training samples.

  • Results

    SLATER demonstrated superior performance to state-of-the-art supervised and unsupervised methods in brain MRI experiments, including across-domain tasks where it outperformed GANsup.

  • Takeaways & Limitations

    Decoupling the MRI prior from the imaging operator enables subject-specific adaptation across varying contrasts or acceleration rates without paired training datasets.

Abstract

from arXiv · show

Supervised reconstruction models are characteristically trained on matched pairs of undersampled and fully-sampled data to capture an MRI prior, along with supervision regarding the imaging operator to enforce data consistency. To reduce supervision requirements, the recent deep image prior framework instead conjoins untrained MRI priors with the imaging operator during inference. Yet, canonical convolutional architectures are suboptimal in capturing long-range relationships, and priors based on randomly initialized networks may yield suboptimal performance. To address these limitations, here we introduce a novel unsupervised MRI reconstruction method based on zero-Shot Learned Adversarial TransformERs (SLATER). SLATER embodies a deep adversarial network with cross-attention transformers to map noise and latent variables onto coil-combined MR images. During pre-training, this unconditional network learns a high-quality MRI prior in an unsupervised generative modeling task. During inference, a zero-shot reconstruction is then performed by incorporating the imaging operator and optimizing the prior to maximize consistency to undersampled data. Comprehensive experiments on brain MRI datasets clearly demonstrate the superior performance of SLATER against state-of-the-art unsupervised methods.

I. INTRODUCTION

Accelerated MRI reduces scan time by reconstructing images from undersampled k-space, but conventional supervised and DIP-based approaches retain important supervision or prior limitations. SLATER addresses these issues by separating MRI-prior learning from imaging-operator adaptation and using adversarial cross-attention transformers.

  • Motivation: Accelerated MRI reconstructs images from undersampled k-space to shorten exams, addressing MRI’s limited signal-to-noise ratio and prolonged acquisition times.Longer exams increase economic costs and may be infeasible for uncooperative patients.
  • Limitations of Existing Supervision: Supervised reconstruction learns from paired undersampled and fully-sampled data while also embedding the imaging operator through data consistency.Such models may require retraining when MRI contrast, coil configuration, or sampling density changes.
  • SLATER: SLATER learns an unconditional MRI prior during adversarial pre-training, then combines it with the imaging operator during zero-shot inference without additional training samples.The reconstruction process is explicitly decoupled into prior learning and sample-specific data-consistency adaptation.
  • SLATER: Cross-attention transformer blocks connect low-dimensional latent variables with high-dimensional image features to capture long-range context efficiently.The architecture uses a synthesizer, discriminator, and mapper within an unconditional adversarial model.
  • SLATER: SLATER introduces sample-specific adaptation and cross-sectional weight propagation to improve out-of-domain generalization and inference efficiency.The proposed method is presented as an unsupervised alternative to paired-data reconstruction training.
  • Deep Image Prior: DIP removes pretraining by optimizing an untrained network against corrupted measurements, but its randomly initialized prior may be weakly tuned to MR data.DIP’s conventional CNN architecture also has limited sensitivity to long-range spatial interactions.

B. Deep Image Prior

Deep Image Prior uses an untrained network as an image regularizer, optimizing network weights to fit undersampled MRI data while keeping randomly initialized latents fixed. SLATER replaces this generic CNN prior with a pretrained adversarial architecture using global and local latents, cross-attention, convolutional refinement, and noise injection.

  • Deep Image Prior: DIP uses an untrained CNN whose randomly initialized weights are optimized to enforce consistency with corrupted MRI data.The latent input remains fixed during optimization, while the network weights are updated.
  • Deep Image Prior: Randomly initialized DIP networks may provide a weaker MR-specific prior than trained models, while CNN kernels also inadequately capture long-range spatial interactions.These limitations motivate learned priors and transformer-based architectures.
  • SLATER architecture: SLATER pretrains an unconditional adversarial network to generate high-quality coil-combined MR images from noise and latent variables.The synthesizer uses a global latent together with K local latent variables.
  • Cross-attention: Cross-attention blocks mediate interactions between local latent variables and image feature maps, producing contextual representations while connecting distant image regions.Attention maps are formed from latent variables and vectorized feature maps across network resolutions.
  • SLATER architecture: SLATER combines cross-attention with convolutional feed-forward refinement, style-modulated convolution, a second cross-attention block, and noise injection for fine-detail variability.The mapper separately processes global and local latent streams, including self-attention among local latents.

D. Self- versus Cross-Attention Transformers

Vanilla vision transformers combine self-attention with a feed-forward neural network. Self-attention relates spatial positions to compute contextual image representations, while the feed-forward block extracts hidden features from those representations.

  • Transformer components: Vanilla transformers consist primarily of a self-attention sub-block followed by a feed-forward neural network sub-block.These components provide attention-based contextualization followed by nonlinear feature extraction.

1) Self-attention transformer:

Self-attention transformers relate every image location to every other location, allowing distant pixels to influence contextual representations. This global interaction comes with quadratic complexity in the number of image pixels, limiting use at high MRI resolutions.

  • Self-attention transformer: Convolutional processing is local, so distant pixels with non-overlapping neighborhoods do not directly share long-range relationships.The resulting output dependencies are conditioned on the local neighborhoods surrounding each pixel.
  • Self-attention transformer: Self-attention forms interactions among all spatial locations, making output features at distant pixels statistically dependent.Attention maps aggregate value features across the full image for each queried position.
  • Self-attention transformer: Vanilla transformers use self-attention to compute contextual representations from dependencies across distant pixels.A feed-forward sub-block then processes these attention-derived features.
  • Self-attention transformer: O((h1 × h2)2) is the computational complexity of self-attention with respect to image size, limiting applicability at MRI resolutions.The quadratic cost arises because all possible pixel pairs are examined.

2) Cross-attention transformer:

Cross-attention replaces exhaustive pixel-to-pixel interactions with interactions between image features and a compact set of latent variables. Its lower complexity preserves long-range dependencies while enabling higher-resolution processing.

  • Cross-attention transformer: Cross-attention uses K latent variables to implicitly characterize interactions among high-dimensional image features.Each image position attends to the shared latent set rather than directly attending to every other pixel.
  • Cross-attention transformer: Because distant image features depend on the same latent variables, cross-attention produces statistically dependent contextual representations across image locations.A convolutional feed-forward block carries these dependencies into the output feature maps.
  • Cross-attention transformer: O((h1 × h2) × K) is the cross-attention complexity, while its convolutional feed-forward block costs O((h1 × h2) × r2).Both are lower than exhaustive pixel-pair interactions when K and r2 are much smaller than h1 × h2.

E. Learning Procedures

SLATER uses two learning stages: pre-training learns an MRI prior, then zero-shot reconstruction embeds the imaging operator.

  • SLATER first learns an MRI prior during pre-training and then embeds the imaging operator during zero-shot reconstruction.
  • Pre-training assumes no prior knowledge of undersampling patterns or coil-sensitivity encoding.The adversarial transformer instead learns from coil-combined, complex MR images derived from fully sampled acquisitions.

1) Pre-training of the MRI prior:

SLATER pre-trains an unconditional adversarial model to learn an MRI prior from coil-combined images, using non-saturating logistic adversarial losses and gradient regularization.

  • The synthesizer and mapper minimize a shared non-saturating logistic adversarial loss during MRI-prior pre-training.The loss is selected to prevent discriminator saturation and premature learning cessation.
  • Cross-attention maps are evaluated at 128x128 resolution under no noise and white-noise variances of 0.01 and 0.1.
  • The discriminator uses a non-saturating logistic adversarial loss augmented with a gradient penalty term.The first two terms define adversarial loss, while the penalty enforces limited gradients in real-image samples according to the learned distribution.
  • During inference, data consistency adapts the learned prior to undersampled acquisitions by optimizing synthesizer parameters.

2) Zero-shot reconstruction:

SLATER performs zero-shot reconstruction by optimizing its generative variables against acquired k-space data, using a cross-attention synthesizer and comparisons across reconstruction settings.

  • 2) Zero-shot reconstruction: SLATER is categorized as a zero-shot reconstruction method.
  • 2) Zero-shot reconstruction: Inference jointly optimizes synthesizer noise, latent variables, and weights to minimize data-consistency loss against undersampled k-space.The synthesized coil-combined image is coil-expanded using sensitivity estimates, Fourier transformed, and sampled according to the undersampling pattern.
  • 1) Synthesizer: The synthesizer progressively increases resolution through convolutional upsampling, cross-attention transformer blocks, and skip connections.Separate output channels represent real and imaginary image components.
  • 2) Zero-shot reconstruction: Experiments compare SLATER with Fourier, low-rank, supervised, self-supervised, GAN-prior, and self-attention GAN baselines.Hyperparameters were optimized by cross-validation on three-way subject splits.

C. Datasets

Experiments evaluate SLATER on single-coil IXI and multi-coil fastMRI brain MRI across within-domain, across-domain, and changed-acceleration settings.

  • C. Datasets: Demonstrations use T1- and T2-weighted brain MRI from single-coil IXI and multi-coil fastMRI datasets.IXI uses 25 training, 5 validation, and 10 testing subjects.
  • D. Quantitative Assessments: Reconstruction quality is measured with PSNR and SSIM against Fourier-reconstructed references, reporting mean ± standard deviation across test subjects.Differences between methods are assessed with nonparametric Wilcoxon signed-rank tests.
  • B. Model Invertability: SLATER suggests long-range spatial interactions among distant image pixels in simulated phantom experiments.
  • B. Model Invertability: Compared with the second-best method, SLATER yields 4.4dB higher PSNR and 7.7% higher SSIM for DIP, and 4.1dB higher PSNR and 5.9% higher SSIM for zero-shot reconstruction.These comparisons use representative IXI T1- and T2-weighted acquisitions at R=4.
  • C. Within-Domain Reconstructions: Within-domain, SLATER provides 1.1dB higher PSNR and 1.1% higher SSIM than the second-best method, while generally outperforming GANsup.At R=8, GANsup performs similarly on IXI and has higher SSIM on fastMRI.
  • D. Across-Domain Reconstructions: Across-domain, SLATER offers 2.5dB higher PSNR and 1.5% higher SSIM than the second-best method.Static GANsup and SSDU lose 1.7dB PSNR and 1.7% SSIM on average across domains, whereas adaptive SLATER is more similar across settings.
  • D. Across-Domain Reconstructions: When trained at R=8 and tested at R=4, SLATER improves over the second-best method by 4.0dB PSNR and 1.5% SSIM.The results support improved generalization over fixed-weight supervised and unsupervised models.

E. Ablation Experiments

SLATER’s ablations show that optimizing progressively more parameter sets improves reconstruction, while cross-attention transformers and decoupled prior adaptation support performance across domains. The method achieves strong comparisons with supervised, self-supervised, traditional, and zero-shot baselines, but inference remains computationally demanding and convergence is not fully characterized.

  • Ablation Experiments: 10.8dB and 14.0% average gains in (PSNR, SSIM) occur when latent variables, noise, and network weights are all optimized.Latent-only optimization yields (4.1dB, 7.2%), while latent and noise optimization yields (6.5dB, 11.6%).
  • Ablation Experiments: Zero-shot model adaptation prolongs inference, motivating propagation of model parameters across neighboring cross-sections to improve efficiency.Neighboring cross-sections are expected to have structural correlations and similar reconstructions.
  • Ablation Experiments: Cross-attention transformers efficiently capture contextual image features at higher resolutions than full self-attention, which incurs quadratic complexity.The mechanism operates between low-dimensional latent variables and image features.
  • Ablation Experiments: SLATER matches GANsup within domain and outperforms it across domains, while also outperforming DIP, GANprior, and SAGAN.The reported comparisons attribute SLATER’s advantage over DIP to its MRI prior and over GANprior to transformer blocks.
  • Ablation Experiments: Across-domain priors can sometimes outperform within-domain priors because gradient descent may approach less-preferred minima, despite favorable initialization.The authors report that longer optimization diminished this gap, but use early stopping to balance performance and inference time.
  • Ablation Experiments: SLATER decouples MRI-prior learning from imaging-operator embedding, enabling a prior to be adapted during inference across contrasts or acceleration rates.Pre-training performs unsupervised generative modeling, followed by unsupervised model adaptation on undersampled acquisitions.
  • Ablation Experiments: The method’s potential extension to other anatomies, structural or dynamic MRI, and CT is presented alongside reduced supervision and subject-specific adaptation.The conclusion frames these capabilities as making SLATER a promising candidate for accelerated MRI.

I. SUPPLEMENTARY TEXT

The supplementary generator uses sinusoidal position encoding and cross-attention transformer blocks across progressively larger resolutions, with modulated convolutions in the final layer.

  • I. SUPPLEMENTARY TEXT: Position encoding spans the first two spatial dimensions over [-1, 1], while the channel index forms the third coordinate.The encoding variables are defined for input feature maps X with dimensions h1×h2×u.
  • I. SUPPLEMENTARY TEXT: Layer 7 produces a 256x256 output using a modulated convolution after the preceding upsampling stages.This is the final generator layer described in the supplementary architecture.
  • I. SUPPLEMENTARY TEXT: Each cross-attention transformer block combines cross-attention, noise, and modulated convolution twice before producing its output.The block sequence is cross-attention plus noise, modulated convolution, then cross-attention plus noise.
  • I. SUPPLEMENTARY TEXT: The self-attention block applies self-attention and two fully connected operations with a residual input connection.Its stated sequence is self-attention, fully connected, fully connected plus input, and output.

3) Discriminator:

The discriminator architecture is organized as seven convolutional downsampling layers, reducing spatial resolution from 256x256 to 4x4 while adding a skip-like downsampled input path.

  • 3) Discriminator:: The final discriminator layer operates at 4x4 and retains the convolution-plus-downsampling pattern with the downsampled input branch.The listed sequence ends at layer 7 with 4x4 resolution.

II. SUPPLEMENTARY TABLES

The supplementary material documents reconstruction benchmarks, ablations, training and inference timing, weight propagation, attention behavior, and representative comparisons across domains and methods.

  • II. SUPPLEMENTARY TABLES: Within-domain performance is reported for T1- and T2-weighted IXI acquisitions at acceleration rates R=4 and R=8.The corresponding supplementary table covers both contrasts and both acceleration rates.
  • II. SUPPLEMENTARY TABLES: Across-domain performance is reported for T1- and T2-weighted acquisitions in IXI and fastMRI, with A->B denoting training-to-test acceleration rates.LORAKS, GANprior, SAGAN, and SLATER have across-domain performance equal to target-rate within-domain performance because they do not assume the imaging operator during training.
  • II. SUPPLEMENTARY TABLES: Supplementary tables cover SLATER ablations, per-epoch training times, and reconstruction performance with weight propagation.Weight propagation affects GANprior and SLATER because they optimize weights during inference.
  • II. SUPPLEMENTARY TABLES: Cross-attention maps show segregated spatial distributions that group broadly distributed, sometimes noncontiguous tissue clusters with similar intensity and texture.Maps are displayed at 32x32, 64x64, and 128x128 resolutions for a T2-weighted acquisition.
  • II. SUPPLEMENTARY TABLES: Supplementary figures compare SLATER with Fourier, low-rank, supervised, self-supervised, DIP, GANprior, and SAGAN reconstructions using reference images and error maps.The examples span within-domain and across-domain T1- and T2-weighted acquisitions in IXI and fastMRI.
  • II. SUPPLEMENTARY TABLES: The fastMRI across-domain T1 figure compares ZF, LORAKS, GANsup, SSDU, GANprior, SAGAN, and SLATER against a reference with corresponding error maps.This example is specifically shown at R=4.
Loading 2105.08059v3…