PaperlayerView exact revision on arXiv

Source-linked AI summary

Adaptive Latent Diffusion Model for 3D Medical Image to Image Translation: Multi-modal Magnetic Resonance Imaging Study

Jonghun Kim, Hyunjin Park

arXiv:2311.00265v1eess.IVcs.CV

TL;DR

Acquiring multiple MRI modalities can be difficult, although their complementary information is important for medical image analysis. The paper uses conditioned 3D latent diffusion with MS-SPADE for one-to-many translation without patch cropping, and reports successful synthesis and better quantitative performance than other models.

  • Problem

    Multiple MRI modalities provide complementary information, but scan cost, time, and safety can limit their availability, and existing translation approaches have 3D-consistency and flexible-target limitations.

    Source: §1 · pp. 1–2

  • Method

    A conditioned 3D latent diffusion model uses MS-SPADE to transform source latents toward target style, enabling one source modality to generate multiple targets without patch cropping.

    Source: §1 · p. 2

  • Results

    The model demonstrated synthesis across source-target modalities and outperformed existing approaches in quantitative and qualitative evaluations.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    One model can support one-to-many 3D modality synthesis, whereas other compared models perform one-to-one synthesis.

    Source: §1 · p. 2, §6 · p. 8

Abstract

from arXiv · show

Multi-modal images play a crucial role in comprehensive evaluations in medical image analysis providing complementary information for identifying clinically important biomarkers. However, in clinical practice, acquiring multiple modalities can be challenging due to reasons such as scan cost, limited scan time, and safety considerations. In this paper, we propose a model based on the latent diffusion model (LDM) that leverages switchable blocks for image-to-image translation in 3D medical images without patch cropping. The 3D LDM combined with conditioning using the target modality allows generating high-quality target modality in 3D overcoming the shortcoming of the missing out-of-slice information in 2D generation methods. The switchable block, noted as multiple switchable spatially adaptive normalization (MS-SPADE), dynamically transforms source latents to the desired style of the target latents to help with the diffusion process. The MS-SPADE block allows us to have one single model to tackle many translation tasks of one source modality to various targets removing the need for many translation models for different scenarios. Our model exhibited successful image synthesis across different source-target modality scenarios and surpassed other models in quantitative evaluations tested on multi-modal brain magnetic resonance imaging datasets of four different modalities and an independent IXI dataset. Our model demonstrated successful image synthesis across various modalities even allowing for one-to-many modality translations. Furthermore, it outperformed other one-to-one translation models in quantitative evaluations.

1. Introduction

Multi-modal MRI provides complementary information, but obtaining every modality can be difficult, while existing translation approaches face limitations in 3D consistency and flexible target generation. The paper proposes a conditioned 3D latent diffusion model with a switchable block for translating one source modality to multiple targets.

1. Introduction

  • Multi-modal MRI provides complementary information for identifying biomarkers, but scan cost, time, and safety can make modalities unavailable.Missing modalities can affect diagnosis and treatment quality.Source: §1 · p. 1
  • Existing missing-modality models may be difficult to generalize across scenarios, while 2D translation misses adjacent-slice information and patch-based 3D methods may lose global information.Prior translation methods also commonly require separate models for different target modalities.Source: §1 · p. 2
  • The proposed LDM performs 3D translation without patch cropping, using MS-SPADE to transform source latents toward target style and conditioning to synthesize desired modalities.This design supports one-to-many translation with a single model.Source: §1 · p. 2
  • The model translates one source modality to multiple 3D targets and achieved the highest image-translation performance on BraTS2021 and IXI.The contribution summary reports the highest performance on both datasets despite supporting multiple modalities.Source: §1 · p. 2

2. Related Works

Prior work includes adversarial translation, conditional normalization for style transfer, and diffusion models for image generation. This paper combines latent diffusion with a switchable SPADE block to support target-flexible 3D translation.

2. Related Works

  • GAN-based translation methods include paired and unpaired approaches, but the cited methods generally require a separate model for each target modality.Examples discussed include Pix2Pix, CycleGAN, NICEGAN, RegGAN, Ea-GAN, and ResViT.Source: §2 · p. 2
  • MS-SPADE applies normalization differently by target modality, converting source latents to target-like latents for the LDM to predict target latents.SPADE uses spatially varying affine transformations, while conditional normalization supports style transfer and image synthesis.Source: §2 · p. 3
  • Latent diffusion performs generation in a compressed representation, reducing complexity and enabling efficient 3D training without patch cropping.The approach trains an autoencoder first, then trains diffusion in its latent space.Source: §2 · p. 3
  • The training overview separates five-loss autoencoder compression from diffusion training, where the autoencoder is frozen and only the UNet trains on concatenated latents.The forward diffusion process is fixed.Source: Figure 3 · p. 4

3. Method

The method combines latent-space compression, target-style transformation, and diffusion to translate 3D medical images. Modality conditioning supports selecting the desired target while one model handles multiple targets from a source modality.

3. Method

  • The two-stage pipeline first transforms source latents toward the target style, then uses latent diffusion to reduce the gap to true target latents while prioritizing changes to the source image.The autoencoder compresses images; MS-SPADE performs the style transformation, and the diffusion model is trained using target-like and noisy target latents.Source: §3 · p. 4

3.1. Image compression to compute latents

  • The compression stage uses a perceptual latent representation and target-specific SPADE normalization; its VQGAN-based training combines reconstruction, quantization, perceptual, adversarial, and cycle-consistency losses.During inference, the learned target style parameters are used for normalization without requiring the target image as input.Source: §3.1 · p. 4

3.2. Diffusion Model

  • The diffusion model takes target-like latents alongside noisy target latents and predicts the noise at each time step, with the image-compression autoencoder frozen during diffusion training.The forward diffusion process is fixed, and the noisy target latents can be obtained efficiently from the encoder during training.Source: §3.2 · p. 5

3.3. Modality Conditioning

  • The model conditions cross-attention on a one-hot target-modality vector, and inference combines source-derived target-similar latents with random noise to generate the selected target modality.The source image is encoded and passed through MS-SPADE before the resulting latent conditions diffusion.Source: §3.3 · p. 5

4. Experiments

Experiments use BraTS 2021 and IXI data, with implementation based on VQGAN and latent diffusion. Evaluation reports three synthesis metrics on 3D volumes, including for 2D methods after stacking their generated slices.

4.1. Datasets

  • BraTS 2021 training uses 1,251 subjects with four MRI modalities, and evaluation uses its 219-subject validation set; IXI adds T1, T2, and PD data split into 459 training and 115 test subjects.The authors also describe IXI as a validation dataset for their model.Source: §4.1 · pp. 5–6

4.2. Implementation Details

  • The image-compression model uses VQGAN and the diffusion model uses LDM, implemented with PyTorch and MONAI.Training used AdamW with a learning rate of 2 × 10^-6, 1,000 diffusion steps, and four A100 80GB GPUs.Source: §4.2 · p. 6

4.3. Evaluation Metrics

  • Synthesis quality is evaluated with PSNR, NMSE, and SSIM against ground-truth targets on an independent test set, reporting mean and standard deviation for 3D volumes.For 2D methods, generated slices are stacked into a 3D volume for comparison.Source: §4.3 · p. 6

5. Results

The proposed model supports one-to-many 3D modality translation and outperforms competing models in quantitative and qualitative evaluations. Across modality pairs, translation performance varies with the source modality, and the ablation study reports the best performance when all design components are combined.

5.1. Comparison of Ours to Baselines

  • The model supports one-to-many synthesis and surpasses other models in quantitative evaluation, with qualitative outputs closest to ground truth, including challenging tumor regions.The qualitative comparison also reports that 3D methods avoid the stripe patterns seen with slice-wise 2D inference; the proposed method requires no patch cropping.Source: §5.1 · pp. 6–7, Figure 6 · p. 7

5.2. Multi Modalities Image Translation

  • Across BraTS modality pairs, the best source varies by target: T1ce for T1 and FLAIR, FLAIR for T2, and T1ce for FLAIR.The paper links T1ce's strong source performance to its anatomical and tumor information, while noting contrast-agent injection can be challenging and alternatives are feasible.Source: §5.2 · pp. 7–8, Figure 6 · p. 7

5.3. Ablation Study

  • The BraTS T1→T2 ablation reports the best translation performance when the diffusion model, palette, MS-SPADE, and image-compression components are used together.Adding the palette improves on diffusion alone, while MS-SPADE without diffusion has limited translation performance; KL regularization is computationally expensive for 3D training and yields slightly blurry compression.Source: §5.3 · p. 8, Table 3 · p. 8, Figure 7 · p. 8

6. Conclusion

The model synthesizes 3D medical images across source and target modalities, including one-to-many translation, and outperforms existing approaches in quantitative and qualitative evaluations.

6. Conclusion

  • The model performs one-to-many 3D image synthesis without patch cropping and outperforms existing approaches in quantitative and qualitative evaluations.The authors note that its 3D nature may make it computationally expensive; future work includes multiple-source translation and validation on other imaging such as computed tomography.Source: §6 · p. 8

A. Hyperparameter setting

The appendix describes the 3D network setup, including the autoencoder and diffusion-model architectures, and points to tables for their detailed structures and hyperparameters.

A. Hyperparameter setting

  • All networks use 3D inputs; the autoencoder follows VQGAN, and the diffusion model uses a UNet-based architecture.The appendix refers to Tables A and B for the autoencoder and bottleneck MS-SPADE structure, and Table C for diffusion-model hyperparameters.Source: A. Hyperparameter setting · p. 12, Table C · p. 12

B. Dataset

The study trains and evaluates on BraTS 2021 and also tests on IXI, which provides a separate dataset with three MRI modalities.

B. Dataset

  • BraTS 2021 training uses 1,251 subjects and four MRI modalities, with evaluation on 219 validation subjects.Scans are 240 × 240 × 155 at 1 × 1 × 1 mm3 resolution.Source: B. Dataset · p. 12
  • IXI includes 574 subjects with T1, T2, and PD scans; 459 are allocated for training and 115 for testing.Scans are 256 × 150 × 256 at 0.9375 × 0.9375 × 1.2 mm3 resolution.Source: B. Dataset · p. 12

C. Comparison Methods details

The comparison included established 2D translation models and 3D counterparts, with separate image dimensions and batch sizes for the two approaches.

C. Comparison Methods details

D. Additional Experimental Results

The IXI experiments compared source modalities for translation to each target. Quantitatively, the most effective source varied by target, while qualitative differences were minimal.

D. Additional Experimental Results

Loading 2311.00265v1…