PaperlayerView exact revision on arXiv

Source-linked AI summary

SciFusion: Physics-Guided Compositional Diffusion for Multimodal 3D Scientific Image Generation

Luoxi Wang, Alessio Quercia, Hanno Scharr, Arya Bangun

arXiv:2610.30313v1eess.IVcs.LG

TL;DR

Large-scale 3D scientific imaging data are difficult to collect, and modality-specific generative models limit transfer across heterogeneous domains. SciFusion combines factorized conditioning with domain-specific physics guidance in a unified latent diffusion framework. Across 15 modalities, it improves generation quality over flattened multimodal conditioning and supports zero-shot synthesis of unseen combinations.

  • Problem

    Large-scale 3D scientific imaging datasets are difficult to collect, while modality-specific models require dedicated data and limit knowledge transfer across heterogeneous imaging domains.

    Source: §1 · pp. 1–2

  • Method

    SciFusion uses latent diffusion with domain, object, and contrast factors that can be recombined, alongside domain-specific physical guidance.

    Source: §1 · p. 2, §3.2 · p. 3, §3.3 · p. 4

  • Results

    Across 15 medical and scientific imaging modalities, compositional conditioning and physics guidance reduce I3D-FVD from 11.80 to 9.90 relative to a flattened multimodal baseline.

    Source: §5 · p. 8

  • Takeaways & Limitations

    Reusable factors allow zero-shot synthesis of unseen combinations of known factors, including configurations sharing characteristics with existing modalities.

    Source: §5 · p. 8

    Zero-shot transfer may be insufficient when a target has substantial acquisition-specific or distributional differences from factors observed during training.

    Source: §5 · p. 8

Abstract

from arXiv · show

Existing 3D generative approaches are typically developed for individual imaging modalities, requiring substantial modality-specific data and limiting their ability to exploit shared structure across scientific imaging domains. We present SciFusion, a multimodal framework for unified 3D scientific image generation built on a latent diffusion backbone, with two key contributions. First, we introduce a compositional conditioning scheme that factorizes each modality into domain, object, and contrast representations. Sharing these factors across related modalities enables knowledge transfer across heterogeneous imaging domains and allows the factors to be recombined at inference time to synthesize unseen domain-object-contrast compositions in a zero-shot manner. Second, we incorporate domain-specific physics-informed objectives: Fourier spectral consistency for MRI, Radon projection consistency for CT, and multislice diffraction consistency for electron microscopy, to encourage generated samples to respect modality-dependent image formation processes. Extensive experiments across 15 medical and scientific imaging modalities demonstrate that SciFusion improves generation fidelity and cross-domain generalization over modality-specific and flattened multimodal conditioning strategies, while enabling zero-shot synthesis of unseen compositions without training on those target compositions. These results demonstrate that compositional, physics-guided conditioning provides a scalable framework for unified 3D scientific image generation, particularly for emerging modalities where large-scale training data are difficult to obtain.

1. Introduction

Scientific imaging datasets are fragmented because acquiring large-scale 3D data is costly and constrained by specialized equipment and medical privacy. SciFusion addresses the challenge of unifying heterogeneous modalities while preserving their shared structure and distinct imaging properties.

  • Large-scale 3D datasets are difficult to collect because acquisition is expensive, equipment is specialized, and medical applications face privacy constraints.These limitations fragment datasets across imaging domains and restrict data-driven approaches.Source: §1 · p. 1
  • Existing 3D generative models often target individual datasets, objects, or modalities, requiring separate training and substantial modality-specific data.This limits knowledge transfer across related imaging domains.Source: §1 · p. 1
  • Treating modalities as independent classes limits generalization to unseen conditions, while merging them directly can create ambiguous conditioning and weaken modality-specific characteristics.A useful representation should capture modality-defining factors and allow their recombination.Source: §1 · pp. 1–2
  • SciFusion combines a latent diffusion backbone, compositional conditioning, and physics-guided objectives for 3D generation across MRI, CT, and EM.Source: §1 · p. 2
  • Its factorized conditioning enables zero-shot generation of unseen domain, object, and contrast combinations without additional modality-specific training data.The framework also uses domain-specific physical guidance and evaluates observed and unseen modality combinations.Source: §1 · p. 2

2. Related Works

Prior 3D volumetric generation methods improve synthesis efficiency with latent representations but largely target specific medical domains or predefined conditions. Physics-guided generative work likewise remains concentrated within individual imaging domains, motivating transferable representations paired with domain-specific physical guidance.

2.1. 3D Volumetric Image Generation

  • Latent representations support more scalable volumetric generation, but existing methods mainly target specific medical domains or predefined modality conditions.Examples include 3D MedDiffusion, MedLoRD, MAISI, and MAISI-v2.Source: §2.1 · p. 2

2.2. Physics-Guided Generative Models

  • Physics-guided generative approaches have been developed primarily within individual imaging domains, where constraints are coupled to particular acquisition processes.Extending them across heterogeneous scientific imaging is challenging because modalities follow distinct physical formation mechanisms.Source: §2.2 · p. 2

3. Method

SciFusion trains a shared 3D latent diffusion model and controls generation through compositional conditioning and domain-specific physical guidance. Its factorized conditions support seen and unseen domain–object–contrast combinations at inference.

3.1. Latent Diffusion Framework

  • The frozen VQGAN compresses each volume spatially while preserving depth, and a conditional 3D U-Net predicts velocity from noisy latents, timestep, and conditioning.Training uses the v-prediction formulation and a latent diffusion objective.Source: §3.1 · pp. 2–3

3. Method

  • A shared diffusion backbone jointly models heterogeneous imaging modalities, with generation controlled by compositional conditioning and domain-specific physical guidance.Training first learns shared VQGAN latents, then trains a conditional diffusion model on frozen latents.Source: §3 · p. 2, §3.1 · p. 3

3.2. Compositional Conditioning Mechanism

  • Each modality is represented by domain, object, and contrast embeddings whose sum forms a shared condition, allowing factors learned in related modalities to be recombined.The factorized representation supports zero-shot generation of unseen combinations without an additional modality-specific embedding.Source: §3.2 · pp. 3–4

3.3. Physics-Informed Training Objective

  • MRI guidance aligns Fourier power spectra, while CT guidance enforces consistency between Radon-domain projections of generated and reference volumes.The constraints are applied through domain-dependent operators within mixed-domain training.Source: §3.3 · p. 4
  • For EM, a multislice diffraction operator sequentially modulates and propagates the wave through volume slices, and the loss aligns generated and reference far-field diffraction patterns.The physical regularization is applied during training and does not alter inference.Source: §3.3 · p. 4

4. Results

Across 15 modalities, SciFusion’s compositional conditioning improves generation across diffusion backbones, and its physics-guided unified model adds further gains. It also synthesizes unseen modalities without target samples or additional optimization.

4.1. Benchmark Overview

  • The benchmark covers 15 MRI, CT, and EM modalities, with EM treated as out-of-distribution relative to the medical imaging domains.The evaluation includes diverse anatomical structures and imaging contrasts.Source: §4.1 · p. 4

4.2. Comparison Across Conditioning Strategies

  • For 3DMedDiff, compositional conditioning reduces VideoMAE-FVD from 91.27 to 26.85 and I3D-FVD from 33.40 to 11.01.Across the evaluated backbones, it reduces both VideoMAE-FVD and I3D-FVD; SciFusion also has the lowest VMAE-FVD, VMAE-KID, I3D-FVD, I3D-KID, and JEDI scores among compared methods, with results macro-averaged over 15 modalities.Source: §4.2 · pp. 5–6, Table 1 · p. 5
  • Selected generated volumes illustrate SciFusion outputs across modalities, while the reported qualitative comparisons include eight of the 15 modalities.The supplementary materials provide examples for all modalities and slices.Source: §4.2 · p. 6, Figure 2 · p. 6

4.3. Zero-shot Compositional Generation

  • On brain PD, training-free zero-shot generation matches or exceeds PEFT on JEDI (1.68 vs. 1.75) without target-modality samples or additional optimization.The two unseen targets are knee T1-weighted MRI and brain PD-weighted MRI; zero-shot generation produces visually plausible anatomy and modality-specific appearance.Source: §4.3 · p. 7

4.4. Ablation Study

  • The unified model reduces the model count from 15 to 1 while achieving comparable or better quality, and compositional conditioning further reduces I3D-FVD from 11.80 to 11.10 and VideoMAE-FVD from 28.63 to 27.30.Adding physics-guided regularization yields the best overall performance, with improvements across all evaluation metrics.Source: §4.4 · pp. 7–8, Table 3 · p. 8
  • A physical-loss weight of 2 · 10^-4 underperforms across domains, while 5 · 10^-4 achieves the best overall performance on every metric averaged across MRI, CT, and EM.For EM, 4 · 10^-4 gives better FVD/KID, but 5 · 10^-4 gives better JEDI (1.50 vs. 2.13); the authors adopt 5 · 10^-4 as a shared default.Source: §4.4 · p. 8, Table 4 · p. 8

5. Conclusion

SciFusion addresses uneven data availability across imaging domains and acquisition settings through shared compositional structure and physics objectives tailored to image formation. It improves generation quality and supports reuse for unseen compositions, subject to the target’s fit with factors observed during training.

5. Conclusion

  • SciFusion mitigates data imbalance by exploiting shared structure across modalities and adding domain-specific physical guidance.The motivation is that new contrasts, protocols, or configurations may have limited data even when other settings are well represented.Source: §5 · p. 8
  • I3D-FVD fell from 11.80 to 9.90 versus a flattened multimodal baseline, with compositional conditioning and physics guidance both contributing.The comparison covers 15 medical and scientific imaging modalities.Source: §5 · p. 8
  • Recombining known domain, object, and contrast factors supports synthesis of unseen compositions without training on the target composition.For a new contrast or protocol on an established anatomical target, generated samples can provide an initial source for method development while a complete target-specific dataset is collected.Source: §5 · p. 8
  • Zero-shot transfer may be insufficient when a target has substantial acquisition-specific or distributional differences from factors observed during training.The authors propose lightweight target-specific guidance or few-shot adaptation when limited real samples are available.Source: §5 · p. 8

A. Dataset and Preprocessing Details

The dataset setup spans MRI, CT, and electron microscopy, with modality-specific preprocessing and acquisition or simulation details. Brain IXI and Stanford Knee MRI data are reserved for zero-shot assessment rather than training.

A. Dataset and Preprocessing Details

B. Model Architecture Details

SciFusion first learns a shared latent space with a VQGAN, then trains a conditional 3D latent diffusion model using compositional embeddings and domain-specific physics guidance. The setup combines unified training across modalities with a warm-up before physical constraints are applied.

B. Model Architecture Details

  • Experiments used HPC nodes with 8 NVIDIA A100 GPUs, each with 80GB VRAM, and 128 CPU cores.Source: B. Model Architecture Details · p. 11
  • A 3D VQGAN learns a shared latent space from the complete multimodal dataset without modality-specific supervision or target-modality adaptation.Source: B. Model Architecture Details · p. 11
  • The VQGAN maps 16 × 256 × 256 input volumes to 8-channel latents at 16 × 32 × 32 resolution, using a 16,384-entry codebook with embedding dimension 8.Source: B. Model Architecture Details · p. 11
  • VQGAN training combines reconstruction, perceptual, adversarial, and 3D feature-based Jukebox losses over 1,000 epochs.Training uses batch size 1, with adversarial training introduced after a 10-epoch warm-up.Source: B. Model Architecture Details · p. 11
  • The conditional 3D U-Net diffusion model operates on frozen VQGAN latents with 8 channels at 16 × 32 × 32 resolution.It uses channel dimensions [64, 128, 256, 512], two residual blocks per resolution, and attention at the two highest-resolution stages.Source: B. Model Architecture Details · pp. 11–12
  • Compositional conditioning combines 64-dimensional domain, object, and contrast embeddings to represent 3, 12, and 9 categories, respectively.The combined representation supports unseen domain–object–contrast combinations without additional target-modality training.Source: B. Model Architecture Details · p. 12
  • A unified diffusion objective incorporates domain-specific physical constraints, with the physical loss warmed up from epoch 100 through epoch 150.This schedule lets the model first learn the general latent distribution before introducing physics-based constraints.Source: B. Model Architecture Details · p. 12

C. Additional Experiments

The supplementary backbone breakdown shows compositional conditioning improving most available backbone–domain combinations, with backbone-specific exceptions. Generated-volume figures provide qualitative examples across modalities.

C. Additional Experiments

  • Figure 4 shows representative generated volumes across all modalities, with rows for modalities and 16 columns of axial slices from one volume.Source: Figure 4 · p. 13
  • Compositional conditioning improves most available backbone–domain combinations: 10 of 12, with 3DMedDiff improving on all 15 metric–domain combinations.MedLord on MRI and EM is the stated exception, where independently learned per-modality embeddings remain competitive; the table averages results over modalities within each domain.Source: C. Additional Experiments · p. 12, Table 8 · p. 14
  • Table 8 compares backbone-specific baseline conditioning with compositional conditioning across MRI, EM, and CT domain averages.For MedLord, the baseline uses independently learned modality embeddings; other backbones retain their original baseline conditioning.Source: Table 8 · p. 14

Related papers

Loading 2610.30313v1…