PaperlayerView exact revision on arXiv

Source-linked AI summary

Conditional Diffusion Models for Semantic 3D Brain MRI Synthesis

Zolnamar Dorjsembe, Hsing-Kuo Pao, Sodtavilan Odonchimed, Furen Xiao

arXiv:2305.18453v5eess.IVcs.CVcs.LG

TL;DR

Medical imaging needs methods that address limited data and privacy concerns while producing meaningful 3D brain images. Med-DDPM uses segmentation-mask conditioning in a diffusion model to synthesize semantic 3D brain MRI. It produces diverse, anatomically realistic images and supports segmentation performance close to real-image training, with further gains when combined with real images.

  • Problem

    Medical imaging faces data scarcity, annotation costs, heterogeneity, and privacy concerns, while meaningful high-resolution 3D brain synthesis remains challenging.

  • Method

    Med-DDPM is a conditional diffusion model that concatenates segmentation masks with noisy images for pixel-level controllable 3D brain MRI synthesis.

  • Results

    Med-DDPM generates diverse, anatomically accurate images and achieves 0.0146 MSE and 0.6132 MS-SSIM, with qualitative and segmentation evaluations supporting its performance.

  • Takeaways & Limitations

    The semantic conditioning supports potential data augmentation and image anonymization while enabling generation of diverse brain images and all four MRI modalities.

Abstract

from arXiv · show

Artificial intelligence (AI) in healthcare, especially in medical imaging, faces challenges due to data scarcity and privacy concerns. Addressing these, we introduce Med-DDPM, a diffusion model designed for 3D semantic brain MRI synthesis. This model effectively tackles data scarcity and privacy issues by integrating semantic conditioning. This involves the channel-wise concatenation of a conditioning image to the model input, enabling control in image generation. Med-DDPM demonstrates superior stability and performance compared to existing 3D brain imaging synthesis methods. It generates diverse, anatomically coherent images with high visual fidelity. In terms of dice score accuracy in the tumor segmentation task, Med-DDPM achieves 0.6207, close to the 0.6531 accuracy of real images, and outperforms baseline models. Combined with real images, it further increases segmentation accuracy to 0.6675, showing the potential of our proposed method for data augmentation. This model represents the first use of a diffusion model in 3D semantic brain MRI synthesis, producing high-quality images. Its semantic conditioning feature also shows potential for image anonymization in biomedical imaging, addressing data and privacy issues. We provide the code and model weights for Med-DDPM on our GitHub repository (https://github.com/mobaidoctor/med-ddpm/) to support reproducibility.

I. INTRODUCTION

Medical imaging faces data scarcity, annotation costs, heterogeneity, and privacy concerns, while meaningful high-resolution 3D brain synthesis remains challenging. Med-DDPM addresses this gap with mask-conditioned diffusion for controllable semantic 3D brain MRI synthesis.

  • Medical imaging research is constrained by scarce and heterogeneous data, annotation costs, and privacy concerns.
  • GAN-based medical image synthesis can suffer unstable training, mode collapse, diminished gradients, and inadequate 3D spatial context.
  • Existing diffusion studies advanced unconditional 3D brain MRI synthesis, but semantic 3D medical image synthesis remained a significant research gap.
  • Med-DDPM incorporates segmentation masks into DDPMs for pixel-level controllable, high-resolution 3D brain image synthesis.
  • The method is evaluated on raw clinical MRI and BraTS2021 data, including synthesis of all four MRI modalities from segmentation masks.
  • The authors position Med-DDPM as a potential data augmentation and anonymization tool and provide a synthetic dataset, code, and model weights.

II. METHOD

Med-DDPM extends 3D DDPMs into conditional volumetric synthesis by concatenating segmentation masks with noisy images throughout diffusion. A modified 3D U-Net denoises these conditioned inputs to generate semantically guided MRI volumes.

  • Diffusion process: The forward process adds Gaussian noise to training images according to a variance schedule over timesteps.
  • Noise schedule: A cosine noise schedule with offset s is used to prevent sudden fluctuations and excessively small schedule values near timestep zero.
  • Denoiser architecture: The denoiser is a modified 3D U-Net incorporating volumetric components, embeddings, ResBlocks, SiLU activations, group normalization, attention, and fully connected layers.
  • Semantic conditioning: The segmentation mask is channel-wise concatenated with each noisy image, giving the denoiser semantic control over generated anatomy and pathology.
  • Sampling: During sampling, the conditioned input is passed to the trained noise predictor, which supports iterative reverse diffusion toward an MRI volume.
  • Mask representation: The segmentation mask uses one-hot encoding for background, head, and tumor classes to avoid ordinal bias from numeric labels.

A. Loss Function

The study uses an L1 loss for noise prediction because L2 produced noisier images and is more sensitive to outliers.

  • L2 loss produced noisier images than L1 in the authors’ experiments.L2 uses squared differences and is sensitive to outliers, whereas L1 uses absolute differences.
  • The model predicts the original added noise using an L1 loss over all image pixels.The total pixel count is n = w · h · d.

A. Datasets and Image Preprocessing

The evaluation uses unnormalized clinical whole-head brain MRIs and corresponding lesion masks, with registration, cropping, padding, and resizing applied during preprocessing.

  • The dataset contains unnormalized contrast-enhanced T1-weighted whole-head MRIs with corresponding segmentation masks for brain lesions.Images were acquired from patients undergoing Cyberknife radiosurgery at NTUH.
  • Preprocessing applied registration, cropping, and padding before resizing images to 128x128x128 with 1.5x1.5x1.5mm slice spacing.

B. Experiment Details

The experiments compare Med-DDPM with prior synthesis methods using quantitative image metrics, expert visual assessment, and tumor-segmentation performance under synthetic and real-data training scenarios.

  • Experiment setup: Med-DDPM was compared primarily with 3D DiscoGAN and 3D Pix2Pix, while other methods were excluded when their stacked 3D results were unsatisfactory.
  • Segmentation evaluation: A 3D U-Net was trained on synthetic and real images for tumor segmentation and evaluated using Dice, IoU, Recall, and Precision.The segmentation models were trained for 100 epochs with BCE loss and batch size 4.
  • Quantitative evaluation: Quantitative image evaluation used 3D-FID, MSE, MMD, and MS-SSIM, with a pretrained 3D ResNet for feature extraction.The 3D-FID implementation was adopted because conventional FID depends on a 2D Inception-v3 network.
  • Qualitative evaluation: Two neurosurgeons visually assessed 40 images consisting of 20 real and 20 synthetic samples.Their classifications were based solely on visual assessment.

D. Generated Images

Visual and quantitative comparisons indicate that Med-DDPM produces realistic, detailed, and diverse 3D brain MRIs, including varied outputs from manipulated or identical masks.

  • Overall quality: Med-DDPM produced more realistic overall 3D brain MRI images than the baseline methods in visual comparison.Other methods showed blur, texture artifacts, asymmetry, coarse anatomical features, or unnaturally wide vessels.
  • Tumor synthesis: Med-DDPM generated highly realistic images with visible brain features and tumor regions, while 3D Pix2Pix produced blurry outputs from unseen test masks.Med-DDPM sometimes generated incomplete tumor ring enhancement, a feature also present in real images.
  • Tumor synthesis: Med-DDPM produced more realistic peritumoral edema than 3D DiscoGAN and captured intricate details resembling real images.
  • Output diversity: Med-DDPM generated diverse images from manipulated masks, whereas 3D Pix2Pix and 3D DiscoGAN showed artifacts, limited anatomical detail, and similar brain features across samples.The similar outputs from the GAN models suggest mode collapse and limited image diversity.
  • Quantitative evaluation: Med-DDPM achieved an MSE of 0.0146 and an MMD of 28.2507 in quantitative evaluation.The reported MSE was the lowest among the compared models, while the MMD indicated matching to the target-domain distribution.

1) Quantitative Results:

Med-DDPM maintained balanced quantitative performance across synthetic-image quality metrics, although its 3D-FID indicates room for brain-specific feature-extraction optimization.

  • 0.6132 MS-SSIM was closest to the real-data score of 0.5864, indicating strong structural similarity.
  • 48.9729 was the lowest 3D-FID for 3D StyleGAN, while 24.9456 was the lowest MMD for 3D Pix2Pix.Med-DDPM instead maintained balanced performance across metrics.

2) Qualitative results:

Qualitative assessments found Med-DDPM images more realistic than baseline outputs, while segmentation results showed strong performance across real, synthetic, and combined datasets.

  • Qualitative assessment: Med-DDPM images appeared more realistic than baseline outputs, though experts detected vessel-continuity inconsistencies and missing tumor mass effects.
  • Segmentation performance: Med-DDPM consistently outperformed 3D DiscoGAN and 3D Pix2Pix in segmentation-model evaluations.
  • Segmentation performance: 0.6207 Dice on 1,000 synthetic images exceeded 3D DiscoGAN at 0.4685 and 3D Pix2Pix at 0.3171.
  • Segmentation performance: 0.6675 Dice with 1,000 real and 2,000 synthetic images exceeded the 0.6531 real-image baseline.
  • Precision: 3D DiscoGAN reached precision scores of 0.91 on synthetic images and 0.95 with 1,000 real plus 2,000 synthetic images.

F. 3D Multimodal MRI Synthesis Experiment

The multimodal experiment tested whether Med-DDPM could generate all four brain MRI modalities from a segmentation mask, using a preprocessed BraTS2021 dataset and channel-wise conditioning.

  • Experiment design: The experiment generated T1, T1CE, T2, and Flair simultaneously from a segmentation mask.
  • Data preparation: 193 artifact-free BraTS2021 images were cropped and padded to 192x192x144 for training.
  • Data preparation: Tumor labels were remapped so labels 1 to 3 represented tumor parts and label 4 represented brain area.
  • Conditioning: Four-channel modality inputs were concatenated channel-wise with four-channel one-hot segmentation masks during training.
  • Results: Generated multimodal images closely resembled real MRI modalities and showed higher fidelity than the baseline GAN model.

IV. DISCUSSION

The discussion presents Med-DDPM as a strong 3D semantic brain-MRI synthesis method with benefits for image quality and segmentation augmentation, while identifying remaining fidelity limitations.

  • Med-DDPM outperformed 3D StyleGAN, HA-GAN, LDM, and 3D-α-WGAN-GP in structural integrity and realistic representation of tissue and tumors.
  • 0.0146 MSE was the lowest reported value, while 0.6132 MS-SSIM was closest to the 0.5864 real-image score.
  • Expert assessments found Med-DDPM images closer to real brain MRIs than 3D Pix2Pix and 3D DiscoGAN.
  • Vessel-continuity inconsistencies, absent tumor mass effects, and higher 3D-FID identify areas for future improvement.
  • Synthetic images from Med-DDPM improved segmentation performance particularly when combined with real images, supporting data augmentation.
  • Med-DDPM conditions diffusion on segmentation masks to generate diverse, anatomically accurate 3D brain MRIs and all four modalities from one mask.
Loading 2305.18453v5…