PaperlayerView exact revision on arXiv

Source-linked AI summary

Diffusion Models for Medical Image Analysis: A Comprehensive Survey

Amirhossein Kazerouni, Ehsan Khodapanah Aghdam, Moein Heidari, Reza Azad, Mohsen Fayyaz, Ilker Hacihaliloglu, Dorit Merhof

arXiv:2211.07804v3eess.IVcs.CV

TL;DR

Medical imaging lacks a comprehensive review focused on diffusion-model applications despite rapid advances and growing interest. This survey synthesizes the field’s foundations, formulations, applications, and taxonomies, concluding that diffusion models are versatile clinical tools while identifying representation learning and architecture design as open challenges.

  • Problem

    Existing reviews address selected medical-imaging applications or modalities, but none focuses comprehensively on diffusion-model applications in medical imaging.

  • Method

    The survey synthesizes diffusion models through DDPM, NCSN, and SDE formulations, organizes studies across applications and medical-imaging perspectives, and reviews clinical uses and open issues.

  • Results

    The survey covers diffusion-model applications across anomaly detection, segmentation, denoising, classification, reconstruction, registration, generation, and other medical-imaging tasks.

  • Takeaways & Limitations

    Diffusion models are presented as valuable and versatile clinical tools for a wide range of imaging challenges, with use expected to expand.

  • Takeaways & Limitations

    Diffusion models remain constrained by limited semantically meaningful latent representations and unresolved architecture-design choices, including transformer use.

Abstract

from arXiv · show

Denoising diffusion models, a class of generative models, have garnered immense interest lately in various deep-learning problems. A diffusion probabilistic model defines a forward diffusion stage where the input data is gradually perturbed over several steps by adding Gaussian noise and then learns to reverse the diffusion process to retrieve the desired noise-free data from noisy data samples. Diffusion models are widely appreciated for their strong mode coverage and quality of the generated samples despite their known computational burdens. Capitalizing on the advances in computer vision, the field of medical imaging has also observed a growing interest in diffusion models. To help the researcher navigate this profusion, this survey intends to provide a comprehensive overview of diffusion models in the discipline of medical image analysis. Specifically, we introduce the solid theoretical foundation and fundamental concepts behind diffusion models and the three generic diffusion modelling frameworks: diffusion probabilistic models, noise-conditioned score networks, and stochastic differential equations. Then, we provide a systematic taxonomy of diffusion models in the medical domain and propose a multi-perspective categorization based on their application, imaging modality, organ of interest, and algorithms. To this end, we cover extensive applications of diffusion models in the medical domain. Furthermore, we emphasize the practical use case of some selected approaches, and then we discuss the limitations of the diffusion models in the medical domain and propose several directions to fulfill the demands of this field. Finally, we gather the overviewed studies with their available open-source implementations at https://github.com/amirhossein-kz/Awesome-Diffusion-Models-in-Medical-Imaging.

1 Introduction

This survey addresses the absence of a medical-imaging-focused review of diffusion-model applications by systematically covering the field’s literature, methods, applications, and open challenges. It organizes studies across applications, algorithms, organs, and imaging modalities.

  • Taxonomy: It provides a systematic taxonomy separating variational-based and score-based diffusion models and grouping applications into nine medical-imaging task categories.The categories include translation, reconstruction, registration, classification, segmentation, denoising, generation, anomaly detection, and other applications.
  • Multi-perspective categorization: The survey further classifies papers by proposed algorithm, organ of interest, and imaging modality rather than restricting its analysis to applications alone.This multi-perspective organization is presented as a new taxonomy in Figure 5.
  • Motivation and uniqueness of this survey: The survey fills a gap left by prior reviews, which focused on particular applications, modalities, or computer-vision diffusion models rather than medical-imaging applications.Its literature coverage includes relevant papers through October 2022 and selected newer techniques through April 2023.
  • Search strategy: Its search strategy queries DBLP, Google Scholar, and Arxiv Sanity Preserver, then filters results to remove false positives and retain relevant papers.The search spans peer-reviewed, conference, workshop, non-peer-reviewed, and preprint publications.
  • Paper organization: The paper combines theoretical foundations, clinical significance, application-specific comparisons, and discussion of challenges and open issues.Its organization covers diffusion-model concepts, clinical benefits, medical-imaging applications, and future directions.

2 Theory

Diffusion models address the generative-learning trade-off by combining broad mode coverage with high-quality sampling, while incurring iterative sampling costs. Their theory is organized around forward noising and reverse recovery, with variational, score-based, and SDE perspectives.

  • Generative learning landscape: Generative models balance high-quality sampling, mode coverage, sample diversity, and computational efficiency, with different model families favoring different criteria.GANs generate high-quality samples rapidly but have limited mode coverage, whereas VAEs and normalizing flows cover modes but generally produce lower-quality samples.
  • Diffusion models: Diffusion models progressively add noise until data approaches a Gaussian distribution, then reverse the process to recover the original data structure.The forward process perturbs the data distribution, while reverse diffusion denoises the perturbed samples.
  • Variational perspective: DDPMs define forward diffusion as a Gaussian-noising Markov chain and learn a parameterized reverse process that generates samples from the data distribution.The forward chain uses successive diffusion steps and a variance schedule; the reverse transitions are modeled as Gaussian distributions and trained through a variational bound.
  • Variational perspective: DDPM training can instead predict the added noise, connecting the simplified objective to generative score networks.Ho et al. replace direct parameterization of the reverse-process mean with a model that predicts the noise term.
  • Score perspective: Score-based models estimate log-density gradients across noise levels, including NCSNs and SDEs as two main subcategories.NCSNs estimate scores for perturbed data distributions at different noise levels, while SDEs provide a broader stochastic-process framework.
  • Score perspective: NCSNs use Gaussian perturbations at multiple scales and a shared noise-conditioned network to address inaccurate scores and scalability problems in high-dimensional data.The approach makes data distributions more amenable to score-based modeling and estimates scores corresponding to all noise levels.

3 Clinical Importance

Diffusion models address clinical imaging needs arising from annotation bottlenecks, class imbalance, privacy concerns, and limited access to diverse medical data. Surveyed studies report realistic synthetic images, promising augmentation benefits, and broad potential across biomedical imaging challenges.

  • Clinical motivation: Diffusion models can generate diverse, realistic-looking medical images to help address class imbalance, educational needs, and data-security concerns.Synthetic images can provide unique examples for teaching and practice while reducing reliance on sensitive real data.
  • Evidence from applications: Studies report promising practical use of synthetic data, including preserved statistical properties, privacy protection, and improved classifier performance when synthetic and real data are combined.The cited studies cover electronic health records, histology images, and skin classifiers.
  • Evidence from applications: Pathologists could not reliably distinguish real from diffusion-generated histopathology images, supporting their use as realistic augmentation for medical research.Most correctly identified synthetic images were classified with lower confidence.
  • Inverse imaging: Diffusion models are a recent generative prior for biomedical inverse imaging problems, where the goal is to infer underlying physical properties from observations or measurements.The passage frames inverse imaging as reconstruction guided by assumptions about the underlying object or system.
  • Clinical outlook: The survey concludes that diffusion models are valuable and versatile in clinical settings, addressing a wide range of imaging challenges with expected future expansion.The conclusion presents continued growth as a future direction for medical imaging and research.

4 Diffusion Models in Action

The survey organizes diffusion-based medical imaging research across applications and additional dimensions such as modality, organ, and reverse-process algorithm. Examples include modality translation, missing-modality synthesis, and structure-preserving translation, while the taxonomy exposes limitations and methods across studies.

  • Taxonomy: The taxonomy covers nine application categories, including translation, reconstruction, registration, classification, segmentation, denoising, generation, anomaly detection, and other applications.Figure 5 maps studies across these nine sub-fields.
  • Taxonomy: Each study is additionally classified by imaging modality, organ of interest, and the specific reverse-process algorithm used in the diffusion model.This expands the taxonomy beyond application labels to capture algorithmic and clinical dimensions.
  • Image-to-image translation: Diffusion models are applied to missing-modality synthesis and cross-modal translation, including MRI-to-CT conversion for diagnostic imaging.The cited example addresses the clinical need for complementary modalities while avoiding some additional acquisition burden.
  • Image-to-image translation: Figure 6 compares visual and quantitative MRI-to-CT translation methods, including DDPM, ODE, EM, and Monte Carlo-averaged variants.Zoomed regions are marked with red and green boxes, and Monte Carlo outputs average ten samples.
  • Image-to-image translation: UMM-CSGM synthesizes a missing modality from all remaining modalities using a conditional score-based generative approach.On BraTS19, it produced higher-fidelity images with more structural information than the cited state-of-the-art methods.
  • Image-to-image translation: FGDM addresses structural-detail loss during diffusion-based translation through frequency-domain filters and can translate without source-domain training data.The approach is described as zero-shot and trained exclusively on target-domain data.

4.2 Reconstruction

Diffusion models address reconstruction challenges caused by incomplete measurements, slow acquisition, artifacts, and ill-posed inverse problems. Medical reconstruction approaches use learned priors and measurement-consistency mechanisms to recover images from accelerated or limited-angle data.

  • MRI reconstruction is time-consuming and vulnerable to motion artifacts, while supervised methods can be non-robust to train/test distribution changes.
  • CSGM uses score-based generative models trained on MRI images as priors for reconstructing realistic data from undersampled MRI through posterior sampling with Langevin dynamics.
  • Score-MRI trains a continuous-time score function on magnitude images and uses a reverse VE-SDE conditioned on measurements for reconstruction.
  • DOLCE addresses limited-angle CT reconstruction by conditioning diffusion on filtered back projection outputs and adding a sinogram consistency term.
  • Registration: DiffuseMorph combines diffusion and deformation networks end-to-end, using deformation scores and spatial information to estimate realistic continuous deformation fields.

4.4 Classification

Medical image classification remains challenging for diffusion models despite its importance for identifying structures and anomalies. DiffMIC addresses this by combining image embeddings with global and local conditional priors in a latent diffusion framework.

  • Classification supports identifying and characterizing structures and anomalies, but improving classification with diffusion models remains a significant challenge.
  • DiffMIC encodes input images into feature embeddings and uses Dual-granularity Conditional Guidance to capture global and local prior information.

4.5 Segmentation

Diffusion models are applied to medical segmentation to address limited access to diverse pixel-annotated data and to represent uncertainty or generate training data. The surveyed methods synthesize labels or images, learn vessel information, and support large three-dimensional volumes.

  • Medical segmentation needs diverse pixel-annotated data, but labels are restricted by the time, cost, and expertise required; diffusion models can synthesize labeled data.
  • brainSPADE uses a label generator and an image generator to synthesize labeled brain MRI images for training segmentation models.
  • DiffMIC’s Dual-granularity Conditional Guidance generates global and local priors from raw images and ROIs to guide the diffusion process.
  • DARL learns background image distributions and generates vessel masks or synthetic angiograms through two complementary paths, forming a cycle for vessel learning.
  • CIMD addresses ambiguous segmentation with a probabilistic diffusion model and evaluates three datasets across different modalities using Collective Insight Score.
  • PatchDDM is a memory-efficient patch-based diffusion model for large 3D volumes and generates meaningful tumor segmentations with fewer computational resources than traditional diffusion models.

4.6 Denoising

Diffusion models are used for medical denoising because acquisition noise and artifacts can obscure clinically important structures. Surveyed methods target OCT speckle, PET noise, and diffusion-MRI limitations through unsupervised, conditional, or self-supervised diffusion strategies.

  • Noise and artifacts can corrupt medical images, with especially serious effects when imaged objects are small and have low contrast.
  • DenoOCT-DDPM performs unsupervised OCT despeckling using self-fusion to create clearer reference images and learn speckle-noise patterns.
  • DDPM-MR-PETCon has the lowest surface error in the reported comparison, followed by DDPM-PETMR.
  • PET-DDM produced state-of-the-art denoising results compared with a U-Net-based network in terms of PSNR and SSIM.
  • Diffusion MRI denoising imposes a self-supervised, statistic-based strategy and performs denoising through conditional generation to address long acquisition times and low SNR.
  • DDM, VM, and VM-Diff are visually compared for generating temporal cardiac images, with deformed intermediate frames constructed from source and target images.

4.7 Image Generation

Diffusion models support medical image generation across synthetic imaging, temporal reconstruction, chest X-rays, histopathology, and multimodal MRI. Applications adapt diffusion through deformation modules, privacy-aware sampling, genotype guidance, or latent representations.

  • 4.7 Image Generation: Medical image generation includes synthetic 2D/3D images and reconstruction of 3D cells from 2D cell images.These applications illustrate the breadth of diffusion-based image-generation tasks surveyed.
  • 4.7 Image Generation: Diffusion Deformable Model generates intermediate temporal frames from source and target images to support 4D anatomical tracking.It combines a DDPM module for latent-code learning with a deformation module that renders intermediate images.
  • 4.7 Image Generation: Latent diffusion generates class-conditional chest X-rays while using a sampling strategy intended to preserve sensitive biometric information.The generated images were evaluated on thoracic abnormality classification, where the approach outperformed GAN-based methods.
  • 4.7 Image Generation: DDPMs were investigated for synthesizing histopathology images, including images containing varied morphological and genotype-guided characteristics.The passage identifies this as the first use of DDPMs for histopathology image generation.
  • 4.7 Image Generation: CoLa-Diff addresses multimodal MRI synthesis by using a conditioned latent diffusion model rather than operating directly in the original image domain.The latent architecture is proposed to reduce the memory demands that limit unimodal diffusion approaches.

4.8 Anomaly Detection

Diffusion-based anomaly detection generally generates a healthy image or counterfactual and compares it with the input to localize abnormalities. The reviewed methods differ in supervision, noise process, latent representation, and reconstruction strategy.

  • 4.8 Anomaly Detection: Weakly supervised DDIM-based detection translates healthy or diseased inputs toward healthy images before identifying anomaly regions by subtraction.The method uses image-to-image translation as the first stage of the detection process.
  • 4.8 Anomaly Detection: AnoDDPM trains a DDPM only on healthy medical samples and detects anomalies from the difference between generated output and input images.Simplex noise significantly enhances performance compared with Gaussian noise in this setting.
  • 4.8 Anomaly Detection: CDPM generates healthy counterfactuals from factual inputs because training diffusion models only on healthy data produced poor segmentation performance.It encodes the input into a latent space, reverses diffusion under healthy-state conditioning, and derives an anomaly heatmap by subtraction.
  • 4.8 Anomaly Detection: BAnoDDPM uses an autoencoder and latent diffusion to compress input images, enhance their latent codes, and decode results into pixel space.Its overview describes anomaly-detection processing through autoencoding, diffusion, reverse diffusion, and decoding.
  • 4.8 Anomaly Detection: Patch-based DDPMs estimate healthy brain anatomy from local patches because simultaneously noising the entire image can hinder reconstruction of intricate brain structures.Spatial context guides the patch-based reconstruction process.
  • 4.8 Anomaly Detection: HierarchicalDet detects abnormal teeth in panoramic X-rays through bounding boxes and predicts quadrant, enumeration, and diagnosis labels.Its two-stage diffusion-based detector refines boxes and then predicts tooth-related labels.

4.9 Other Applications and Multiple Tasks

Beyond the main application categories, diffusion models are applied to brain CT quantification, vertebral-fracture grading, MRI denoising and super-resolution, adversarial defense, and hierarchical tooth detection. These applications often require task-specific adaptations.

  • 4.9 Other Applications and Multiple Tasks: Diffusion models quantify brain midline shift in head CT images through a semi-supervised learning framework.The survey places this work among applications that do not fit a single predefined category.
  • 4.9 Other Applications and Multiple Tasks: R2D2+ applies an SDE-based multi-successive paradigm to MRI denoising and super-resolution without requiring paired clean and noisy images.It starts reverse diffusion from the initial noisy image and uses non-parametric covariance-eigenvalue estimation.
  • 4.9 Other Applications and Multiple Tasks: A diffusion-based defense method uses a multiscale image pyramid and Gaussian noise injection to reverse distortions that cause skin-lesion misclassification.The method addresses adversarial vulnerability in computer-aided skin-cancer diagnosis.
  • 4.9 Other Applications and Multiple Tasks: HierarchicalDet combines diffusion-based box refinement with hierarchical prediction of tooth number, quadrant number, and bounding box.The second stage concatenates inferred boxes from the first stage with noisy boxes instead of refining complete noisy boxes directly.

4.10 Comparative Overview

The survey compares diffusion models by algorithm, conditioning strategy, application focus, and practical limitations. Diffusion is prominent in reconstruction, denoising, and generation, while broader tasks often require additional modifications.

  • 4.10 Comparative Overview: The taxonomy categorizes reviewed papers by DDPMs, NCSNs, and SDEs, highlighting each algorithm’s concepts, objectives, and practical use cases.Table 1 organizes the reviewed diffusion-model literature according to directly used or inspired algorithms.
  • 4.10 Comparative Overview: Conditioning the reverse diffusion process is among the most studied strategies for obtaining desired medical-imaging outputs.The survey describes image-based and measurement-based constraints as examples of conditioning types.
  • 4.10 Comparative Overview: Diffusion models face slow sampling and high computational cost, motivating accelerations such as DDIM’s non-Markovian sampling with negligible quality degradation.The limitation is computational burden rather than a lack of output quality.
  • 4.10 Comparative Overview: Simplex noise improves anomaly-detection quality over Gaussian noise, while high-frequency-only diffusion improves MRI reconstruction stability and quality.These examples show that task-specific diffusion design choices can affect output quality.
  • 4.10 Comparative Overview: Diffusion models are used in medical imaging because of their effectiveness, ease of implementation, and high output quality.The survey connects these properties with the need for high-resolution images containing accurate local information.
  • 4.10 Comparative Overview: Reconstruction, denoising, and generation receive more attention because diffusion’s noise-then-denoise formulation aligns with their objectives.Other tasks, including segmentation, registration, and text-to-image translation, have received comparatively less attention in the surveyed literature.
  • 4.10 Comparative Overview: Adapting diffusion models to tasks such as text-to-image translation may require auxiliary networks with strong text-encoding capabilities.The survey therefore describes task-specific modification as a practical boundary on direct reuse.

5 Future Direction and Open Challenges

Future work should address diffusion models’ representation, architecture, privacy, modality coverage, and optimization challenges while exploiting their growing medical-imaging relevance.

  • Diffusion models remain appealing for medical imaging because they can operate without labeled data.
  • Future research should explore underrepresented modalities beyond the predominantly used CT and MRI studies.The survey specifically identifies ultrasound as a modality with physical and imaging-system limitations that diffusion models may help address.
  • Representation space: Diffusion models’ latent spaces are less semantically meaningful, hindering semantic manipulation, reconstruction, and interpolation tasks.Conditional denoising score matching with an additional encoder is presented as one approach for producing interpretable latent features.
  • Architecture design: Medical-imaging diffusion research needs more architectural innovation beyond prevailing CNN-based and baseline DDPM or score-based designs.Transformers may model non-local interactions and long-range dependencies, but their use in diffusion models remains at an early stage.
  • Privacy concerns: Privacy-preserving diffusion systems should account for the possibility that models memorize and reproduce individual training images.Federated learning is discussed as a way to train on decentralized medical data without exposing sensitive information.
  • Optimization: The survey identifies computationally expensive reverse-process optimization and reinforcement-learning-based inversion as additional directions for investigation.Reinforcement learning can search for diffusion parameters that maximize a reward while penalizing deviations from observed data.

6 Conclusion

The survey synthesizes diffusion-model research across medical-imaging applications and formulations, then outlines future research avenues. It also cautions that the rapidly growing field remains early-stage and changeable.

  • The survey covers diffusion models across anomaly detection, segmentation, denoising, classification, reconstruction, registration, generation, and other medical-imaging tasks.
  • It organizes existing models around three primary formulations: DDPMs, NCSNs, and SDEs.
  • The paper outlines possible avenues for future research in medical diffusion modeling.
  • The survey is intended as a starting point and reference because diffusion-based medical-imaging research is rapidly growing but still early-stage and subject to change.Some cited papers are preprints, although the authors sought high-quality research from reputable sources.
Loading 2211.07804v3…