Source-linked AI summary
Metadata-Aware Adaptation of a Generative Foundation Model for Conditional CMR Synthesis
Marc Rodríguez, Grzegorz Skorupko, Nay Aung, Steffen E Petersen, Karim Lekadir, Polyxeni Gkontra
TL;DR
Limited and imbalanced CMR data make metadata-faithful synthesis challenging. The paper adapts a pretrained latent diffusion model using metadata prompts and three metadata-aware strategies, achieving improved distributional fidelity while disease-specific conditioning remains most difficult.
Problem
CMR synthesis must address limited, imbalanced datasets while ensuring generated images faithfully reflect clinically meaningful patient metadata.
Method
The framework fine-tunes a pretrained latent diffusion model conditioned on structured patient metadata and slice position, integrating Metadata-Free CFG, Contrastive Batching, and Inverse-Frequency Sampling.
Results
FID of 37.47 improved by 57.04% over the same model without the proposed strategies and by 28.68% over a geometry-conditioned CMR diffusion baseline.
Takeaways & Limitations
The framework generated realistic and diverse short-axis CMR images, with conditioning more reliable for balanced attributes than for highly imbalanced pathology.
Abstract
from arXiv · showhide
Synthetic image generation is a promising strategy to address data scarcity and the underrepresentation of clinically important phenotypes in medical imaging, yet generating images that faithfully reflect meaningful patient characteristics remains challenging. In this work, we investigate metadata-conditioned cardiac magnetic resonance (CMR) synthesis using a pretrained latent diffusion model, encoding structured clinical metadata and slice position as textual prompts to guide CMR generation. To improve metadata adherence and address the imbalance of clinical attributes, we integrate three strategies: Metadata-Free Classifier-Free Guidance (CFG), Contrastive Batching, and Inverse-Frequency Sampling. The framework was fine-tuned and evaluated on 59,058 short-axis CMR from the UK Biobank using paired image similarity, distributional fidelity, and subgroup-level analyses. The combined approach achieved a Fréchet Inception Distance (FID) of 37.47, improving by 57.04\% over the same model fine-tuned without these strategies and by 28.68\% over a previous text-conditioned CMR diffusion baseline requiring cardiac geometry as additional input, while relying solely on patient metadata. This distributional gain, driven mainly by Metadata-Free CFG, came with a modest reduction in paired similarity, suggesting that the model prioritizes population-level realism over exact image reproduction. Subgroup analyses demonstrated improved alignment across demographic and acquisition-related metadata, with disease-specific conditioning being the most challenging task. These findings demonstrate the potential of generative foundation models for clinically meaningful CMR synthesis while highlighting the need for more effective metadata-aware conditioning strategies. Our code is available at https://github.com/rodriguezmarc/conditional-cmr.
I. INTRODUCTION
CMR datasets are constrained by limited availability, diversity, and quality, motivating synthetic generation that preserves clinically meaningful patient metadata. This work adapts a pretrained latent diffusion model for metadata-conditioned CMR synthesis while addressing imbalance during training.
- CMR datasets are limited and imbalanced because of privacy restrictions, annotation costs, and scarce patient populations or disease phenotypes.
- Data availability, diversity, and quality have become primary bottlenecks for robust, clinically applicable medical AI systems.
- Synthetic image generation can support augmentation, privacy-preserving experimentation, and fairness-aware dataset balancing, but visual plausibility alone does not ensure metadata fidelity.
- The framework encodes structured patient metadata and slice position as textual prompts for a latent diffusion model adapted to the CMR domain.
- The proposed pipeline fine-tunes a pretrained latent diffusion model using techniques intended to improve metadata adherence and mitigate class imbalance.
II. METHODS
The framework integrates three metadata-aware strategies into a latent diffusion pipeline for conditional CMR synthesis. These strategies strengthen conditioning and improve representation of diverse metadata values during training and inference.
- The pipeline combines Metadata-Free Classifier-Free Guidance, Contrastive Batching, and Inverse-Frequency Sampling.The strategies target metadata conditioning and class imbalance through complementary inference and training procedures.
- Metadata-Free Classifier-Free Guidance removes metadata segments from the negative prompt while preserving acquisition context to strengthen metadata conditioning.
B. Contrastive Batching (CB)
The training design increases metadata variation within batches and weights selection toward underrepresented metadata values. Balancing is applied at the metadata level because sample rarity can differ across attributes.
- B. Contrastive Batching (CB): Contrastive Batching constructs batches with different values of a selected metadata attribute when sufficient diversity is available.This increases within-batch metadata variation and reduces dominance by frequent metadata values.
- B. Contrastive Batching (CB): Inverse-Frequency Sampling assigns weights from metadata-value frequencies to increase representation of underrepresented values.
- B. Contrastive Batching (CB): Inverse-Frequency Sampling caps the inverse-frequency weight at τ = 2.0 to prevent extremely rare values from affecting results excessively.
- B. Contrastive Batching (CB): During batch construction, metadata values are sampled proportionally to inverse-frequency weights, prioritizing minority values.
- B. Contrastive Batching (CB): Balancing operates across conditioning metadata because a sample may be rare for one attribute but common for another.Combined with Contrastive Batching, value selection remains contrastive while incorporating inverse-frequency probabilities.
A. Dataset and Prompt Construction
The study uses UK Biobank short-axis cine CMR slices paired with structured metadata and standardized text prompts. The dataset is patient-level split and contains substantial imbalance, especially for pathology, BMI, and age.
- A. Dataset and Prompt Construction: UK Biobank data comprise mid-ventricular short-axis slices and two adjacent slices extracted from end-systolic cine CMR acquisitions.
- A. Dataset and Prompt Construction: Table I reports paired and distributional fidelity metrics, including MAE, MS-SSIM, FID, FRD, and diversity ratios for incremental strategy additions.
- A. Dataset and Prompt Construction: The dataset is split at the patient level using a 90:10 ratio, producing 53,298 training images and 5,760 testing images.
- A. Dataset and Prompt Construction: Each sample uses a standardized prompt containing acquisition context, slice position, pathology, and BMI group metadata.
- A. Dataset and Prompt Construction: Pathology, BMI, and age are substantially imbalanced, whereas sex is relatively balanced and slice position is inherently balanced.
B. Model and Training Workflow
The model was initialized from public Stable Diffusion weights and fine-tuned as a Stable Diffusion-like latent diffusion system for conditional CMR synthesis.
- Public Stable Diffusion weights initialized the model because pretrained MINIM weights were unavailable.The implementation used CompVis/stable-diffusion-v1-4 weights from Hugging Face.
- A frozen VAE mapped images to latent space while a fine-tuned U-Net denoised CMR latents.
C. Evaluation Protocol
Evaluation measured paired-image fidelity, distributional fidelity, and subgroup-level performance using complementary image and metadata analyses.
- Evaluation dimensions: Subgroup analyses evaluated performance across metadata categories and compared generated and real distributions using distributional matrices.
- Evaluation dimensions: Paired fidelity compared generated images with references using MAE and MS-SSIM.MAE measured pixel-level differences, while MS-SSIM assessed anatomical similarities.
- Evaluation dimensions: Distributional fidelity compared generated and real test sets using FID and FRD.
IV. RESULTS AND DISCUSSION
The combined strategy configuration improved distributional fidelity while preserving realistic CMR appearance and producing BMI-conditioned variation and diversity across random seeds.
- Overall results: FID 37.47 improved 57.04% over the same model without the proposed strategies and 28.68% over Skorupko et al.’s geometry-conditioned baseline.The combined model relied on metadata conditioning while the comparison baseline additionally required cardiac geometry.
- Overall results: MF-CFG drove most of the distributional improvement, reducing FID from 87.22 to 44.84 before further sampling gains.
- Overall results: MAE 0.260 and MS-SSIM 0.171 indicate preserved short-axis CMR appearance without exact reconstruction behavior.Metadata-only conditioning is not intended to reproduce a specific reference image.
- Qualitative results: BMI-conditioned images showed progressively greater adipose tissue from underweight to obese categories.
- Qualitative results: Three images generated from one obese prompt with different random seeds demonstrated diversity under identical conditioning.
B. Subgroup Performance Analysis
Subgroup agreement was strongest for better-represented demographic and acquisition attributes and weakest for pathology, although matching subgroups consistently outperformed mismatched ones.
- Subgroup performance: Pathology had the highest diagonal mean FID at 118.87, making disease categories the least accurate same-subgroup generation task.The passage notes that pathology underrepresentation and FID sensitivity to small sample sizes may partly contribute.
- Subgroup performance: Diagonal mean subgroup FID ranged from 44.90 for sex and 49.48 for slice position to 80.55 for BMI and 92.45 for age.
- Subgroup performance: For every metadata attribute, diagonal mean FID was lower than the corresponding off-diagonal value.This indicates closer alignment between generated and matching real subgroups than between generated and different real subgroups.
V. CONCLUSION
The proposed framework conditions short-axis CMR synthesis on structured patient metadata and slice position without requiring cardiac geometry, improving distributional fidelity over two baselines. However, conditioning remains less reliable for imbalanced clinical variables, particularly pathology, leaving robust control under class imbalance as an open challenge.
- The framework conditions image generation on structured patient metadata and slice position without requiring cardiac geometry as input.
- It improved distributional fidelity over both a fine-tuned Stable Diffusion baseline and a previously published CMR text-conditioned diffusion baseline.
- Subgroup conditioning was more reliable for balanced attributes such as sex and slice position than for highly imbalanced clinical variables, particularly pathology.
- Metadata-Free Classifier-Free Guidance drove the main distributional improvement, while Contrastive Batching and Inverse-Frequency Sampling provided complementary refinements.
- Robust and fully controllable metadata conditioning under class imbalance remains an open challenge.