Source-linked AI summary
Omni-Diffusion: Unified Multimodal Understanding and Generation with Masked Discrete Diffusion
Lijiang Li, Zuwei Long, Yunhang Shen, Heting Gao, Haoyu Cao, Xing Sun, Caifeng Shan, Ran He, Chaoyou Fu
TL;DR
Most multimodal language models rely on autoregressive backbones, motivating alternatives for unified multimodal intelligence. Omni-Diffusion uses mask-based discrete diffusion to jointly model text, image, and speech tokens, matching or surpassing existing autoregressive multimodal systems across benchmarks.
Problem
Most existing multimodal intelligence approaches rely on autoregressive architectures, leaving substantial room to explore alternative probabilistic modeling approaches.
Method
Omni-Diffusion uses mask-based discrete diffusion to model the joint distribution of multimodal tokens for unified comprehension and generation across text, images, and speech.
Results
Across diverse benchmarks, Omni-Diffusion performs comparably to or better than existing autoregressive multimodal systems processing two or more modalities.
Takeaways & Limitations
The results support diffusion models as promising foundation-model backbones for multimodal AI systems.
Takeaways & Limitations
The model requires further improvement for broader downstream tasks such as instruction-based visual content editing.
Abstract
from arXiv · showhide
While recent multimodal large language models (MLLMs) have made impressive strides, they predominantly employ a conventional autoregressive architecture as their backbone, leaving significant room to explore effective and efficient alternatives in architectural design. Concurrently, recent studies have successfully applied discrete diffusion models to various domains, such as visual understanding and image generation, revealing their considerable potential as a promising backbone for multimodal systems. Drawing inspiration from these pioneering studies, we introduce Omni-Diffusion, the first any-to-any multimodal language model built entirely on mask-based discrete diffusion models, which unifies understanding and generation across text, speech, and images. Omni-Diffusion employs a unified mask-based discrete diffusion model to directly capture the joint distribution over discrete multimodal tokens. This approach supports not only bimodal tasks but also more complex scenarios involving multiple modalities. On a diverse set of benchmarks, our method outperforms or performs on par with existing multimodal systems that process two or more modalities, highlighting the significant promise of diffusion models in powering the next generation of multimodal foundation models. Project webpage: https://omni-diffusion.github.io.
1 Nanjing University, 2 Tencent Youtu Lab, 3 CASIA
The passage introduces recent multimodal large language models (MLLMs).
- The passage introduces recent multimodal large language models (MLLMs).
1. Introduction
Omni-Diffusion addresses the dominance of autoregressive architectures in multimodal intelligence by using a unified mask-based discrete diffusion model for any-to-any comprehension and generation. It models joint distributions over multimodal tokens and introduces specialized training and inference techniques for multimodal alignment and variable-length generation.
- Most existing multimodal systems rely on autoregressive architectures, leaving substantial room for alternative probabilistic modeling approaches.
- Diffusion models are emerging as a promising alternative to classical autoregressive architectures for natural language processing.
- Omni-Diffusion is the first any-to-any multimodal language model built on a mask-based discrete diffusion model for unified comprehension and generation.It learns the joint distribution of multimodal semantic tokens from text, image, and speech data.
- Omni-Diffusion uses a three-stage progressive training pipeline and attenuated tail-pad masking to support multimodal alignment and variable-length generation.Its inference techniques include a position penalty for generation order and visual quality, plus special token pre-infilling for spoken dialogue.
- The model aligns modalities in a shared semantic representation space by modeling a joint distribution over multimodal discrete tokens.
2. Related Work
Related work advances unified multimodal systems for comprehension and any-to-any generation, while discrete diffusion models provide an alternative foundation for language, images, and visual understanding. Mask-based discrete diffusion models sequences through mask-token prediction, alongside related discrete flow-matching approaches for multimodal language models.
- Unified Multimodal Models: Unified multimodal comprehension models support diverse modalities through modality alignment, encoder connections, or duplex communication mechanisms.OneLLM aligns eight modalities; Video-SALMONN connects audio-visual encoders through a Q-former; VITA introduces duplex communication.
- Any-to-Any Frameworks: Any-to-any frameworks extend multimodal models beyond comprehension to arbitrary input and output modalities.CoDi aligns multiple modalities within continuous diffusion, while AnyGPT processes discrete tokens across modalities with a unified LLM.
- Discrete Diffusion Models: Mask-based discrete diffusion models have shown strong performance in natural language processing, image generation, and visual understanding.These models learn target discrete-token distributions through mask-token prediction by corrupting clean training sequences with mask replacements.
- Discrete Flow Matching: Discrete Flow Matching is mathematically similar to mask-based discrete diffusion and has been used to build multimodal large language models.NeXT-Omni models transformation from a random token sequence to sequences sampled from real data, although the two approaches differ in decoding processes.
3. Method
Omni-Diffusion unifies multimodal understanding and generation by modeling text, speech, and image as discrete tokens within a mask-based diffusion framework. Its method combines modality-specific tokenization with unified training, progressive multimodal alignment, and decoding strategies for controllable generation.
- Unified Multimodal Diffusion: Omni-Diffusion directly models a unified distribution over multimodal discrete tokens instead of using an additional output model to project textual features into generated data.This unified representation supports comprehension and generation across modalities.
- Unified Multimodal Diffusion: Text, speech, and images are tokenized, wrapped with modality-specific boundary tokens, concatenated into one sequence, and randomly corrupted with [MASK] tokens at a timestep-dependent ratio.The corruption ratio is derived from a timestep sampled uniformly from [0, 1].
- Unified Multimodal Diffusion: Training uses cross-entropy only on masked positions, with one unified mask-token prediction objective and no modality-specific optimization.The model predicts the clean sequence from the corrupted sequence.
- Tokenization and Backbone: The implementation uses MAGVIT-v2 image tokens with downsampling factor f = 16 and codebook size 8192, plus SenseVoiceSmall and GLM-4-Voice for speech.Dream-7B provides the pretrained discrete diffusion backbone, whose vocabulary is expanded by 16384 speech tokens and 8192 image tokens.
- Progressive Training: A three-stage progressive pipeline expands training from visual-language pre-alignment to speech-vision-language alignment and SDVI-based cross-modal improvement.The SDVI dataset contains spoken visual question answering and speech-to-image data, including 30,000 speech-image pairs.
- Generation Strategies: Attenuated Tail-Pad Masking reduces the mask ratio for pad tokens using γ < 1, while Special Token Pre-Infilling inserts [begin-of-speech] at index 0.25L to guide spoken dialogue generation.These strategies address excessive pad-token generation and control output formatting.
4. Experiment
Omni-Diffusion is evaluated across multimodal perception, generation, alignment, inpainting, and sampling-efficiency tasks. Results show strong or comparable performance across modalities, alongside cross-modal generation, fine-tuning-free inpainting, and efficient few-step sampling.
- Speech Tasks: Omni-Diffusion matches specialized speech models on ASR and surpasses them on TTS across LibriSpeech and LibriTTS benchmarks.Evaluation uses WER and compares against CosyVoice, GLM-4-Voice, and AnyGPT.
- Visual Tasks: Omni-Diffusion achieves strong visual understanding and generation, comparable to specialized visual LLMs while supporting more modalities and tasks.The evaluation covers VQA and text-to-image generation against visual and any-to-any multimodal LLMs.
- Visual Tasks: Compared with existing any-to-any models, Omni-Diffusion provides better visual understanding and superior text-image alignment, with visual quality comparable to external diffusion-based methods.These results are reported for the VQA and text-to-image evaluations.
- Speech-Vision Alignment Evaluation: Using 10,000 synthesized-speech captions, Omni-Diffusion demonstrates unified speech-image-text alignment through speech-to-image generation and spoken visual interaction.Qualitative examples show spoken responses to spoken image-related questions and capture image semantics and object relationships.
- Inpainting: Mask-token prediction enables inpainting without additional fine-tuning or inpainting training samples by replacing unknown regions with [MASK] tokens.The model generates harmonious visual content conditioned on the unmasked image regions.
- Sampling Efficiency: Reducing inference steps significantly lowers latency while preserving strong generation performance and enabling high-quality images with extremely few time steps.Sampling is evaluated for text-to-image and TTS using CLIP-T/CLIP-I and WER, respectively.
5. Conclusion
Omni-Diffusion is an any-to-any multimodal language model built purely on mask-based discrete diffusion models. It unifies comprehension and generation across text, image, and speech while matching or surpassing existing autoregressive methods.
- 5. Conclusion: Omni-Diffusion is an any-to-any multimodal language model built purely on mask-based discrete diffusion models.It models a joint distribution over multimodal tokens.
- 5. Conclusion: The model performs unified comprehension and generation across text, image, and speech.Its multimodal scope supports both understanding and generation across these modalities.
- 5. Conclusion: Extensive experiments show performance comparable to or better than existing autoregressive methods.The conclusion presents these results as evidence of the promise of diffusion models for multimodal language modeling.
A. Implementation Details · B. Additional Experiments
Omni-Diffusion is initialized from Dream-7B-Instruct and extended with dedicated image and speech components within a three-stage progressive training pipeline. The provided implementation details identify the datasets in Table 4 and specify AdamW optimization with β1 = 0.9 and β2 = 0.95.
- A. Implementation Details: The model is initialized with weights from the pre-trained Dream-7B-Instruct discrete diffusion language model.This provides the pretrained discrete diffusion language-model backbone.
- A. Implementation Details: MAGViT-v2 is incorporated for image tokenization.It handles modality-specific image processing.
- A. Implementation Details: SenseVoiceSmall is incorporated for speech encoding.It provides modality-specific speech processing on the encoding side.
- A. Implementation Details: The GLM-4-Voice decoder is incorporated for speech decoding.It provides modality-specific speech processing on the decoding side.
- A. Implementation Details: Omni-Diffusion uses a three-stage progressive training pipeline whose datasets are detailed in Table 4.Table 4 summarizes the datasets used in Omni-Diffusion.
- A. Implementation Details: Optimization uses AdamW with β1 = 0.9 and β2 = 0.95.These optimizer hyperparameters are the values provided in the implementation passage.
B.1. Quantitative evaluation on Spoken VQA
Omni-Diffusion demonstrates strong spoken interaction with visual content on a Spoken VQA evaluation created by converting MME text questions into speech.
- Quantitative evaluation on Spoken VQA: Omni-Diffusion maintains strong spoken visual interaction capability on the Spoken VQA task.Because no Spoken VQA benchmark exists, the evaluation converts MME text questions into speech using CosyVoice2.
B.2. Ablation Study
The ablation study examines position penalty and special token pre-infilling, finding that both strategies improve relevant multimodal generation quality. Position penalty benefits text-to-image generation, while special token pre-infilling prevents overly brief spoken responses in visual interactions.
- Ablation overview: The ablation study evaluates several key design choices in Omni-Diffusion.The study focuses on position penalty and special token pre-infilling.
- Effect of position penalty: Removing position penalty decreases visual quality on the COCO text-to-image benchmark.The strategy is intended to prevent repetitive patterns and improve image generation quality.
- Effect of special token pre-infilling: Special token pre-infilling improves spoken interaction with visual content by preventing overly brief responses.Without this strategy, the model tends to generate overly brief responses, as shown in Figure 8.
B.3. Additional Evaluation on Image Generation · B.4. Latency comparison with AR baselines · B.5. More Example of Image Generation
Omni-Diffusion is evaluated for image-generation quality on DPG-Bench, latency against autoregressive baselines for text-to-image and TTS, and additional text- and speech-conditioned image examples. The evaluations indicate competitive image-generation performance and improved efficiency when inference steps are reduced.
- B.3. Additional Evaluation on Image Generation: Omni-Diffusion’s image-generation quality is evaluated on DPG-Bench against SD-series image-generation models and Show-o.The comparison is presented in Table 7.
- B.3. Additional Evaluation on Image Generation: The DPG-Bench evaluation reports that Omni-Diffusion performs nicely compared with models designed specifically for image generation.The result is used to illustrate Omni-Diffusion’s multimodal generation capability.
- B.4. Latency comparison with AR baselines: Inference latency is compared with autoregressive models for text-to-image generation and text-to-speech across Tables 8 and 9.Text-to-image latency is measured as average seconds per MSCOCO image, while TTS latency uses average real-time factor.
- B.4. Latency comparison with AR baselines: TTS performance and latency are evaluated on LibriTTS, with latency estimated using RTF = Inference Latency / Speech Duration.The passage defines RTF as the ratio of inference latency to speech duration.
- B.4. Latency comparison with AR baselines: Reducing inference steps gives Omni-Diffusion better generation efficiency than autoregressive baselines.This efficiency comparison is reported for the TTS evaluation.
- B.5. More Example of Image Generation: Additional generated-image examples are provided for text-to-image and speech-to-image conditioning in Figures 9 and 10.These examples illustrate image generation conditioned on text and speech.
C. Limitation and Future Direction
Omni-Diffusion supports perception and generation across various modalities but remains limited in downstream task coverage, motivating future scaling of data and model parameters.
- C. Limitation and Future Direction: The model’s downstream capabilities could be extended to tasks such as instruction-based visual content editing.The authors identify broader downstream-task support as a future improvement area.
- C. Limitation and Future Direction: Future work will explore scaling the training data and model parameters.This scaling is proposed to improve the model beyond its current capabilities.