PaperlayerView exact revision on arXiv

Source-linked AI summary

Discrete Diffusion Bridges for Spatiotemporally Aligned Image Translation and Generation

Xing Xie, Jiawei Liu, Shijun Zhou, Huijie Fan, Zhi Han, Yandong Tang, Liangqiong Qu

arXiv:2608.29997v1cs.CV

TL;DR

Standard discrete diffusion misaligns with image translation and generation by discarding source structure in a pure-mask state and masking tokens in an order that conflicts with inference. DDB addresses this with hybrid absorption and an information-guided schedule, and experiments report broad task applicability, structural fidelity, edit alignment, and high quality under extremely low sampling steps.

  • Problem

    Standard discrete diffusion creates spatial and temporal misalignment for image translation and generation by discarding source priors and using random masking despite difficulty-aware inference.

  • Method

    DDB uses a source-conditioned hybrid absorbing state and an information-guided masking schedule to construct a spatiotemporally consistent bridge between domains.

  • Results

    DDB reports balanced edit alignment and structural fidelity across I2I tasks, competitive T2I performance, broad applicability, and excellent quality at extremely low sampling steps.

  • Takeaways & Limitations

    DDB provides a unified trajectory design for discrete latent spaces that preserves source context while supporting diverse translation and generation settings.

Abstract

from arXiv · show

We propose Discrete Diffusion Bridges (DDB), a novel framework designed to resolve the fundamental spatiotemporal misalignment of standard discrete diffusion in image translation and generation. By corrupting data into a pure mask state via a random schedule, the conventional forward process induces a twofold misalignment: spatially, this pure-mask destination entirely discards the rich structural priors of the source image; temporally, the random masking order inherently contradicts the ``easy-first, hard-last'' decoding mechanism used during inference. To address this, DDB constructs a direct and efficient trajectory between domains. Spatially, we introduce a hybrid absorption mechanism that redefines the absorbing state to a stochastic mixture of mask and source tokens, effectively injecting source prior as spatial anchors into the latent space. Temporally, we design an information-guided noise schedule that quantifies semantic variation to prioritize the corruption of high-information regions at earlier timesteps. This ensures the model learns to resolve difficult semantic changes using robust context from invariant regions. Extensive experiments validate the versatility and robustness of our framework across diverse generative paradigms. DDB effectively balances edit alignment with structural fidelity across both text-guided semantic manipulation and pure structural image translation, while inherently complementing text-to-image generation and guaranteeing robust high-quality decoding under extremely low sampling steps. Code and models are available at \href{https://github.com/HKU-HealthAI/DDB}{https://github.com/HKU-HealthAI/DDB}.

1 Introduction

DDB identifies spatial and temporal misalignment in standard discrete diffusion for image translation and generation, then addresses both with source-aware absorption and difficulty-aware scheduling. Experiments report broad applicability across generative paradigms while balancing edit alignment, structural fidelity, and low-step quality.

  • Motivation: Standard discrete diffusion discards source structure through a pure-mask state and uses random masking that conflicts with easy-first, hard-last inference.This misalignment can force models to hallucinate invariant structures rather than transform existing source features.
  • Experimental Validation: Extensive experiments report balanced edit alignment and source-context preservation in I2I translation, competitive T2I performance, and excellent quality at extremely low sampling steps.The reported scope includes semantic manipulation, structural mapping, and diverse generative paradigms.
  • DDB Framework: DDB reformulates image translation and generation as a spatiotemporally consistent trajectory between data domains.The framework is presented as a unified response to the identified misalignment.
  • DDB Framework: Hybrid Absorption replaces the pure-mask destination with a stochastic mixture of mask and source tokens, injecting source priors as spatial anchors.The mechanism is intended to preserve spatial structure and ground generation in source context.
  • DDB Framework: The Information-Guided Noise Schedule prioritizes masking high-information or complex regions early, aligning training objectives with the difficulty-aware inference process.This schedule directs difficult semantic changes to be resolved when sufficient contextual information is available.

2 Related Work

Related work positions discrete diffusion as a parallelizable, iterative token-refinement paradigm for visual generation and contrasts it with continuous diffusion approaches for image translation and generation. Prior continuous methods include conditional pure-noise-to-image models and direct source-to-target trajectories built with residuals or diffusion bridges.

  • Discrete Diffusion: Discrete diffusion extends masked-token reconstruction into visual generation through iterative token refinement and offers a highly parallelizable alternative to autoregressive models.The related-work passage connects this paradigm to masked language modeling and visual systems such as MaskGIT and MUSE.
  • Continuous Diffusion: Continuous diffusion has been applied to image translation and generation, including conditional mappings from pure noise to clean targets.Palette and SR3 are cited as early examples of image-conditioned continuous diffusion models.
  • Continuous Diffusion: Later continuous approaches establish direct source-to-target trajectories using residuals and diffusion bridges for cross-domain mapping.The passage names DDBM, RDDM, DDIB, and BBDM as examples.

3 Method

DDB constructs a spatiotemporally consistent bridge by injecting source-aware spatial anchors and scheduling corruption according to regional information. Its forward and reverse processes align hybrid-state training with confidence-based inference.

  • Reverse Process: Inference along the Bridge: During reverse inference, DDB starts from the hybrid state, retains confident predictions, and resamples uncertain regions from the hybrid distribution.This iterative decoding strategy follows the constructed bridge and matches the stochastic source injection used during training.
  • Motivation: Misalignment in Standard DDM: Standard discrete diffusion drives target tokens to a pure mask state, severing source correspondence and applying masking uniformly across spatial locations.This creates spatial disconnect and an unordered corruption process for translation and generation.
  • Forward Process: Constructing the Bridge: DDB replaces the pure-mask endpoint with Hybrid Absorption, a stochastic mixture of source tokens and mask tokens controlled by source injection ratio λ.When λ > 0, retained source tokens act as explicit spatial anchors; λ = 0 recovers standard discrete diffusion.
  • Forward Process: Constructing the Bridge: The hybrid forward transition conditions corruption on both target tokens and source-conditioned hybrid states, ensuring the endpoint retains structural priors.The categorical transition selects either the target token or the hybrid absorbing state according to the binary corruption mask.
  • Forward Process: Constructing the Bridge: The Information-Guided Schedule stochastically prioritizes high-information tokens for early absorption while mixing information-based and uniform sampling to preserve contextual diversity.The stochasticity parameter ρ controls the uniform component, while exactly ⌊N · γ(t)⌋ mask indices are sampled at timestep t.
  • Forward Process: Constructing the Bridge: DDB derives token-level information maps from source-target differences for translation or local grayscale variance for source-free generation.Pixel-level maps are average-pooled to the tokenizer resolution before guiding the schedule.

4 Experiments Analysis and Results

DDB is evaluated across instruction-based and pure image translation, text-to-image generation, and component analyses. Across these settings, it balances semantic editing with source preservation, improves structural mapping and report-to-image generation, and remains effective under limited sampling steps.

  • Experimental Setup: Experiments cover instruction-based translation, pure image translation, and text-to-image generation using multiple datasets and task-specific metrics.The evaluation includes OmniEdit, Graph-200K, CDD-11, SynthRAD, IXI, and MIMIC-CXR, with metrics spanning DINO, CLIP-T, Edit Score, PSNR, SSIM, LPIPS, FID, MS-SSIM, and CLIP-Score.
  • Semantic Manipulation: DDB achieves the highest overall Edit Score on semantic manipulation by balancing edit semantic alignment with source and background preservation.This evaluation covers image editing, style transfer, and subject-driven generation.
  • Structural Mapping: DDB consistently outperforms baselines on restoration, modality translation, and super resolution, with higher PSNR and SSIM and lower LPIPS.These tasks test multi-task restoration, cross-domain anatomical consistency, and recovery of fine-grained details.
  • Text-to-Image Generation: DDB achieves superior FID, MS-SSIM, and CLIP-Score on report-to-image generation, while visual results show more realistic anatomy and fewer structural artifacts.The task uses medical reports from MIMIC-CXR as input.
  • Ablation Analysis: An injection ratio of 0.5 with random-ratio training provides the best balance between edit alignment and source preservation.Larger ratios improve DINO, smaller ratios favor CLIP-T, and the equal mixture produces the highest overall Edit Score.
  • Ablation Analysis: Stochastic proportional source-mask mixing performs best because it provides contextual robustness and decouples spatial randomness from the temporal schedule.Information-metric analysis further finds Image Difference best for source-anchored translation, while Target variance is optimal for report-to-image generation.

5 Conclusion

DDB reframes discrete generation as a direct, efficient bridge between data domains, addressing spatiotemporal misalignment through spatial anchors and difficulty-aware timing. Experiments support its versatility across translation and generation tasks.

  • DDB replaces the random void-to-image trajectory with a direct bridge between data domains.The framework targets spatiotemporal misalignment in standard discrete diffusion.
  • Hybrid absorption provides spatial source anchors, while an information-guided schedule creates a difficulty-aware temporal path.Together, these designs synchronize training objectives with inference dynamics.
  • Experiments demonstrate DDB’s versatility and superiority across a wide range of tasks.The conclusion positions the framework as relevant to unified multimodal translation and generation.

S1 Overview

The supplementary materials provide implementation details, additional experiments and analyses, ablations, and further visual results.

  • The supplement includes more implementation details covering datasets and experiment settings.
  • It adds experimental results on I2I translation, framework analysis, and ablation studies.
  • Additional visual results are provided in Section S4.

S2 More Implementation Details

The experiments cover diverse image editing, generation, restoration, modality translation, super-resolution, report-to-image, and inpainting datasets. Training uses a fully parameterized Lumina DiMOO backbone with standardized 512×512 preprocessing and multi-GPU optimization.

  • Image Editing and Style Transfer: OmniEdit provides 1.2M training pairs and 700 test cases for instruction-based editing and style transfer.Tasks include object manipulation, attribute changes, environment changes, and global style transfer.
  • Subject-Driven Generation: Graph-200K uses 182,000 training pairs and 1,000 test pairs for subject-driven generation.The task contextualizes isolated subjects in text-described backgrounds.
  • All-in-One Restoration: CDD-11 contains 11 composite degradation combinations, with 20,790 training pairs and 2,310 testing pairs for restoration.
  • Modality Translation: SynthRAD2023 evaluates paired CT-to-MRI and MRI-to-CT translation using 83,864 training pairs and 555 testing pairs.The scans are strictly aligned and come from identical patients.
  • Implementation: Training uses PyTorch on 4 NVIDIA A100 GPUs, batch size 8 per GPU, learning rate 2e-5, and full-scale Lumina DiMOO parameter training.Images are center-cropped to 512×512 and data are pretokenized.

S3 Additional Experimental Results

Additional experiments show strong performance across editing, inpainting, unified multi-task translation and generation, and alignment analyses. DDB’s information-guided schedule and spatial anchoring are linked to improved localization and stable cross-domain decoding.

  • Editing Performance: DDB achieves the highest overall Edit Score across nearly all evaluated OmniEdit editing subtasks.The evaluation covers attribute modification, background swap, and environment change, with Edit Score, CLIP-T, and DINO metrics.
  • Editing Performance: DDB balances precise semantic edit alignment with strict source-context preservation across editing instructions.
  • Mask Inpainting: DDB consistently outperforms baselines on CelebA-HQ inpainting with irregular masks.The result supports recovery of coherent semantic and structural information under severe spatial corruption.
  • Unified Image Translation and Generation: Under unified training, DDB outperforms the baseline across all evaluated translation and generation tasks.The setting jointly trains one shared-parameter model across seven benchmarks.
  • Alignment Mechanism: Discrete diffusion decodes easy, high-confidence contexts before difficult semantically altered regions.Editing prioritizes invariant backgrounds, while style transfer reconstructs source-consistent subjects before style-variant backgrounds.
  • Alignment Mechanism: The information-guided schedule corrupts high-information regions first, matching regions identified as hard to decode during inference.The mechanism uses edit difference to quantify cross-domain variation.
  • Alignment Mechanism: Spatial source anchors help DDB reveal target-domain structure faster and localize intended semantic modifications more accurately.The aligned framework is contrasted with baseline failures in identifying correct manipulation regions.
  • Ablation: The optimal stochastic masking ratio for image editing is ρ=0.3.Ratios from 0 to 0.5 were compared on OmniEdit.

S4 More Visual Comparisons

Additional visual comparisons evaluate DDB across multiple OmniEdit editing tasks and broader image translation and generation settings. The figures compare DDB with Lumin-DiMOO on OmniEdit and show visual results for super resolution, modality translation, and report-to-image generation.

  • OmniEdit: The OmniEdit comparisons cover attribution modification, weather change, object swap, object add, object remove, and style transfer.Object swap is shown in two figures.
  • OmniEdit: The OmniEdit figures use Lumin-DiMOO as the baseline and report that DDB more effectively translates input images to the target domain.This comparison is stated for attribution modification, weather change, object swap, object add, object remove, and style transfer.
  • Other Translation Tasks: Additional visual comparisons examine super resolution on IXI and modality translation on SynthRAD2023.These results are presented in Figures 15 and 16.
  • Other Translation Tasks: A further visual comparison evaluates report-to-image generation on the MIMIC-CXR dataset.The report-to-image results are presented in Figure 17.
Loading 2608.29997v1…