PaperlayerView exact revision on arXiv

Source-linked AI summary

ResViT: Residual vision transformers for multi-modal medical image synthesis

Onat Dalmaz, Mahmut Yurt, Tolga Çukur

arXiv:2106.16031v3eess.IVcs.CV

TL;DR

Medical image synthesis must predict absent target modalities from limited source scans, while CNN-based models have difficulty capturing long-range contextual relationships. ResViT addresses this with an adversarial hybrid CNN-transformer generator using residual ART bottleneck blocks, weight sharing, and unified multi-configuration synthesis. Across ablation studies and multimodal imaging demonstrations, ResViT generally achieves optimal or near-optimal performance and superior synthesis quality relative to competing approaches, while its scope remains bounded by practical data and registration requirements.

  • Problem

    Medical image synthesis is an ill-posed task because target-modality images are absent during inference and CNN-based models have limited ability to capture long-range relationships.

  • Method

    ResViT uses an adversarial encoder-decoder with a residual bottleneck that aggregates CNN and transformer modules, shares weights across transformer blocks, and unifies multiple source-target configurations.

  • Results

    ResViT yields optimal or near-optimal performance across representative ablation tasks and achieves superior synthesis quality to state-of-the-art approaches in multimodal MRI and MRI-CT datasets.

  • Takeaways & Limitations

    The proposed hybrid adversarial design provides a candidate for multimodal medical image synthesis that preserves contextual relations while maintaining localization power.

  • Takeaways & Limitations

    The discussion identifies registration accuracy and training-data requirements as practical constraints, with unregistered or unpaired training requiring additional strategies such as registration blocks or cycle consistency.

Abstract

from arXiv · show

Generative adversarial models with convolutional neural network (CNN) backbones have recently been established as state-of-the-art in numerous medical image synthesis tasks. However, CNNs are designed to perform local processing with compact filters, and this inductive bias compromises learning of contextual features. Here, we propose a novel generative adversarial approach for medical image synthesis, ResViT, that leverages the contextual sensitivity of vision transformers along with the precision of convolution operators and realism of adversarial learning.} ResViT's generator employs a central bottleneck comprising novel aggregated residual transformer (ART) blocks that synergistically combine residual convolutional and transformer modules. Residual connections in ART blocks promote diversity in captured representations, while a channel compression module distills task-relevant information. A weight sharing strategy is introduced among ART blocks to mitigate computational burden. A unified implementation is introduced to avoid the need to rebuild separate synthesis models for varying source-target modality configurations. Comprehensive demonstrations are performed for synthesizing missing sequences in multi-contrast MRI, and CT images from MRI. Our results indicate superiority of ResViT against competing CNN- and transformer-based methods in terms of qualitative observations and quantitative metrics.

I. INTRODUCTION

Medical image synthesis addresses the need to predict unacquired modalities from limited scans, but conventional CNN-based methods have limited access to long-range context. ResViT combines convolutional localization, transformer context, adversarial realism, residual ART blocks, weight sharing, and a unified multi-configuration model.

  • Motivation: Medical image synthesis predicts target-modality images from source images acquired under a limited scan budget, despite absent target data during inference.The task is described as an ill-posed inverse problem involving high-dimensional medical images and nonlinear tissue-contrast differences across modalities.
  • ResViT: ResViT combines vision-transformer sensitivity to global context, CNN localization, and adversarial learning to translate between multimodal imaging data.Its encoder and decoder use CNN blocks, while a central bottleneck contains ART blocks that preserve local and global context.
  • Practical design: ResViT introduces weight sharing among ART blocks to reduce model complexity and a unified synthesis model that supports multiple source-target modality configurations.The unified implementation avoids rebuilding separate synthesis models for varying configurations.
  • Limitations of prior methods: CNN-based synthesis improves medical image translation but can miss broader context and detailed structure because convolutional processing emphasizes local features.CNNs are reported to have limited sensitivity to broader context, while pixel-wise losses can lose detailed structure.
  • Related work: Transformer-based synthesis methods had been relatively limited and often used transformers outside the generator, whereas ResViT places them in a hybrid adversarial generator for multimodal translation.The paper distinguishes ResViT from methods using transformers in discriminators, non-adversarial architectures, or single-modality reconstruction settings.
  • ResViT: ART blocks cascade transformer and CNN modules with skip connections, aggregating contextual and convolutional representations in the generator bottleneck.The residual paths are designed to combine the complementary representations of the two module types.

III. THEORY AND METHODS

ResViT is a unified adversarial synthesis model combining convolutional local precision with transformer-based global context through an ART-block bottleneck. Its availability-conditioned design supports multiple source-target modality configurations in one model.

  • III. THEORY AND METHODS: ResViT uses a generator with convolutional encoder and decoder surrounding an information bottleneck of aggregated residual transformer blocks.The bottleneck combines residual convolutional and transformer branches to preserve local and global representations.
  • III. THEORY AND METHODS: The bottleneck processes downsampled feature maps and uses channel-compression modules to distill task-relevant information.Spatial downsampling emphasizes mid- to high-level information before channel compression.
  • III. THEORY AND METHODS: ResViT receives the full modality set during training, masks unavailable targets, and uses availability conditions to select source-target configurations during inference.The encoder learns shared latent structural representations across consolidated synthesis tasks.
  • III. THEORY AND METHODS: ART blocks aggregate localized convolutional features with contextual transformer representations in the central bottleneck.This design targets both local structural precision and long-range contextual sensitivity.

2) Information Bottleneck:

The information bottleneck combines downsampled transformer processing with convolutional feature extraction, then fuses and compresses their outputs before residual refinement. This pathway is designed to retain contextual and localized information in a compact representation.

  • 2) Information Bottleneck:: Downsampling blocks reduce feature-map resolution before the transformer branch extracts contextual information.The transformer processes non-overlapping patches embedded into a learned representation with positional encoding.
  • 2) Information Bottleneck:: The transformer encoder applies cascaded multi-head self-attention and multilayer perceptrons to patch embeddings.Self-attention combines information across sequence elements using query-key similarities and value projections.
  • 2) Information Bottleneck:: Upsampling restores transformer feature maps to the input resolution before channel-wise concatenation with localized convolutional features.This concatenation fuses global transformer context with convolutional local features.
  • 2) Information Bottleneck:: A channel-compression module distills the concatenated structural and contextual representations, followed by residual CNN processing.The compression module uses two parallel convolutional branches with varying kernel sizes.
  • 2) Information Bottleneck:: The decoder receives bottleneck feature maps and produces all modality outputs, while the unified model supports varying source-target configurations.Synthesized modalities are represented in separate output channels.

3) Decoder:

ResViT generates modality outputs through separate channels and shares transformer parameters across ART blocks. This reduces the burden associated with using multiple independent transformer blocks.

  • 3) Decoder:: The decoder represents synthesized modalities in separate output channels across the ART-block pathway.The output channel index identifies each synthesized modality.
  • 3) Decoder:: Transformer encoder weights are tied across ART blocks to limit memory demand, parameter count, and overfitting risk.The strategy enables multiple ART blocks without independently parameterizing every transformer encoder.

4) Parameter Sharing Transformers:

ResViT uses a conditional PatchGAN discriminator that compares acquired and synthesized modality combinations at the patch level. Its discriminator is guided by modality availability and includes pixel-wise L1 supervision.

  • 4) Parameter Sharing Transformers:: The conditional PatchGAN performs patch-level differentiation between acquired and synthetic images to improve sensitivity to localized details.This emphasizes high-spatial-frequency information.
  • 4) Parameter Sharing Transformers:: The discriminator concatenates source images with either synthesized or acquired target images for conditional comparison.These two concatenations form the synthetic and acquired discriminator inputs.
  • 4) Parameter Sharing Transformers:: The loss includes a pixel-wise L1 term between acquired and synthesized target modalities.This provides direct image-space supervision for target synthesis.

6) Loss Function:

ResViT combines pixel-wise, reconstruction, and adversarial losses into an overall objective, with the reconstruction term supporting unified modality configurations. The experiments span brain MRI and MRI-to-CT synthesis datasets.

  • Loss components: The reconstruction term uses an L1 distance between acquired and reconstructed source modalities.The generator reconstructs source modalities at the output, enforcing pixel-wise consistency.
  • Loss components: An adversarial loss is defined through a conditional discriminator.
  • Overall objective: The pixel-wise, reconstruction, and adversarial losses are linearly combined into the overall objective.Their relative contributions are controlled by λpix, λrec, and λadv.
  • Evaluation scope: ResViT was demonstrated on IXI, BRATS, and a pelvic MRI-CT dataset.These datasets support multi-contrast brain MRI and cross-modality synthesis demonstrations.

1) IXI Dataset:

The IXI and pelvic MRI-CT datasets provide evaluation settings for multi-contrast MRI synthesis and cross-modality image synthesis, using multiple competing convolutional and attention-based methods.

  • Datasets: The BRATS dataset includes T1-weighted, T2-weighted, post-contrast T2-weighted, and FLAIR images from 55 subjects.The split contains 25 training, 10 validation, and 20 testing subjects, with 100 axial cross-sections selected per subject.
  • Datasets: The pelvic MRI-CT dataset contains T2-weighted MR and CT images from 15 male subjects.Nine subjects were used for training, two for validation, and four for testing; 90 axial cross-sections were analyzed per subject.
  • Comparisons: ResViT was compared with convolutional, attention-augmented convolutional, and transformer-based synthesis methods.The listed baselines include pGAN, pix2pix, medSynth, MM-GAN, pGANuni, A-UNet, and SAGAN.
  • Convolutional baselines: pGAN uses a ResNet-backbone convolutional GAN with residual CNN blocks in its bottleneck.
  • Convolutional baselines: pix2pix uses a CNN-based U-Net generator with encoder-decoder structure and skip connections.
  • Unified baselines: MM-GAN trains one convolutional U-Net-based network across varied source-target modality configurations.Its unification strategy matches the strategy used in ResViT.

3) Transformer models:

The transformer comparisons include hybrid, convolution-free, and unified transformer architectures, while ResViT uses transformer modules within a convolutional encoder-decoder bottleneck. Configuration and complexity are selected through validation experiments.

  • Transformer models: TransUNet is a hybrid CNN-transformer generator adapted for synthesis with a convolutional output layer.Its conditional GAN discriminator is identical to ResViT’s discriminator.
  • Transformer models: PTNet is a convolution-free transformer generator adapted to a conditional GAN architecture.It uses the same discriminator as ResViT.
  • Unified transformer models: TransUNetuni consolidates multiple synthesis tasks using the same unification procedure as ResViT.
  • ResViT architecture: ResViT’s bottleneck contains nine ART blocks while its encoder and decoder use convolutional layers.The encoder output dimensionality is retained across the information bottleneck.
  • Configuration selection: Transformer-module positions were selected through cross-validation, with more than two modules increasing complexity without performance benefits.Candidate configurations were formed from pre-selected ART-block positions.
  • Configuration selection: Transformer modules were evaluated in base and large variants differing in depth, latent dimensionality, attention heads, and MLP size.The base variant has 12 layers and the large variant has 24 layers.

F. Experiments

Experiments evaluate ResViT on multi-contrast MRI and MRI-to-CT synthesis, compare it with task-specific baselines, and examine architectural and training components through ablations. Qualitative figures and quantitative tables assess synthesis quality across representative tasks.

  • Experiments: IXI experiments cover one-to-one and many-to-one synthesis among T1-, T2-, and PD-weighted MRI sequences.
  • Experiments: MRI-to-CT experiments derive pelvic CT images from source MR images and compare task-specific ResViT with seven baseline methods.
  • Qualitative results: In IXI examples, ResViT produces lower artifact and noise levels and sharper tissue depiction than competing methods.The figure shows T1, T2 →PD and T2, PD →T1 many-to-one tasks alongside sources and reference targets.
  • Quantitative evaluation: Table IV reports PSNR and SSIM for task-specific many-to-one and one-to-one IXI synthesis tasks.Boldface identifies the top-performing model for each task, with values reported as mean±std across test subjects.
  • Ablation studies: Ablations assess ResViT’s architectural design elements and training strategies on representative IXI, BRATS, and MRI-to-CT tasks.The study includes PSNR, SSIM, and FID measurements.
  • Ablation studies: Additional ablations examine tied transformer weights, transformer placement, ImageNet initialization, delayed insertion, skip connections, and sampling modules.
  • Interpretability analyses: Contextual and input feature maps are compared using L2 norms and channel-compression combination weights.Attention Rollout maps are also visualized to inspect transformer focus during synthesis.
  • Qualitative results: In BRATS examples, ResViT shows better-delineated tissue boundaries and lower artifact/noise levels, especially in pathological regions.The representative tasks are T1, T2 →FLAIR and T2, FLAIR →T1.

IV. RESULTS

ResViT achieved strong performance across task-specific and unified missing-modality synthesis in brain MRI, with improved visual quality and pathology depiction relative to competing models.

  • Task-specific synthesis: ResViT achieved the highest performance in IXI many-to-one and one-to-one tasks (p<0.05).It outperformed convolutional, attention-augmented, and transformer models on average.
  • Task-specific synthesis: ResViT achieved the highest performance in BRATS many-to-one and one-to-one tasks (p<0.05), except T2→FLAIR where A-UNet had slightly higher SSIM.Average gains were reported against convolutional and attention-augmented models.
  • Task-specific synthesis: ResViT’s pathology depiction signals the value of ART blocks for maintaining local precision and contextual consistency.The comparison discusses improved synthesis of pathological regions relative to transformer-based alternatives.
  • Unified synthesis: ResViTuni successfully consolidated varying source-target configurations while producing lower artifacts and sharper target images than unified baselines.The unified model was evaluated on many-to-one tasks in IXI and BRATS.

B. Across-Modality Synthesis

Across MRI-to-CT synthesis and ablation studies, ResViT delivered strong performance, while component removals generally reduced image quality or perceptual fidelity.

  • Across-modality synthesis: ResViT yielded the highest performance for pelvic T2-weighted MRI→CT synthesis across subjects (p<0.05).It outperformed convolutional, attention-augmented, and transformer models on average.
  • Across-modality synthesis: ResViT maintained elevated performance near CT bone structures, where contextual representations may be especially important for MRI-CT synthesis.The reported visual advantage concerned accurate tissue depiction and relevant morphology.
  • Component ablations: Removing transformer or convolutional modules generally reduced PSNR and SSIM, while removing adversarial loss produced notably higher FID.The pattern held across representative tasks, with stated exceptions for some metrics in MRI→CT.
  • Training and transformer ablations: Removing weight tying or transformer modules from ART blocks generally reduced performance, supporting the reported transformer configuration and weight-sharing strategy.The only noted exception was similar SSIM for the variant retraining A6 in T1, T2→PD.
  • Architectural ablations: Removing ART residual skip connections or replacing learned down/upsampling generally reduced performance, supporting residual learning and the proposed sampling scheme.Similar SSIM was reported for one unlearned down/upsampling variant in MRI→CT.
  • Interpretability: ResViT attention maps concentrated on lesions and pelvic bone, which were also regions with substantially reduced synthesis errors relative to pGAN.The result links attention focus with improved depiction of healthy and pathological tissues.

V. DISCUSSION

The discussion presents ResViT as an adversarial multimodal synthesis model that combines convolutional localization with transformer context and supports unified modality configurations, while identifying scope boundaries and future extensions.

  • Model contribution: ResViT aggregates convolutional and transformer branches within a residual bottleneck to preserve local precision and contextual sensitivity.This design addresses the limited long-range modeling associated with conventional convolutional operators.
  • Model contribution: Weight sharing lowers model complexity, while unification supports numerous source-target configurations without training separate models.These strategies target computational burden and practical deployment across modality combinations.
  • Empirical support: ResViT outperformed several state-of-the-art convolutional and transformer models in one-to-one and many-to-one MRI and MRI-CT tasks.The models were trained with an identical loss function to emphasize architectural influences.
  • Implementation considerations: ResViT was initialized with transformers pretrained on 16x16 feature maps, and longer sequences incur quadratic processing and memory costs.Different image resolutions can require adjusting sequence length, patch size, or downsampling rate.
  • Scope and future work: The study considered registered modalities paired across subjects, limiting the demonstrated setting to those data conditions.Future extensions include registration blocks, cycle consistency, unpaired data, semi-supervised training, undersampled acquisitions, and multi-scale decoding.
Loading 2106.16031v3…