PaperlayerView exact revision on arXiv

Source-linked AI summary

Dynin-Omni: Omnimodal Unified Large Diffusion Language Model

Jaeik Kim, Woojin Kim, Jihwan Hong, Yejoon Lee, Sieun Hyeon, Mintaek Lim, Yunseok Han, Dogeun Kim, Hoeun Lee, Hyunggeun Kim, Jaeyoung Do

arXiv:2604.00007v1cs.CLcs.AI

TL;DR

Unified multimodal models face serialization constraints from autoregressive decoding and orchestration overhead from external modality-specific generators. Dynin-Omni addresses this gap with masked diffusion over a shared discrete token space and multi-stage training, achieving strong and balanced performance across 19 multimodal benchmarks while remaining competitive with specialized experts.

  • Problem

    Existing unified models rely largely on autoregressive or compositional designs that serialize heterogeneous modalities or delegate generation to external components.

  • Method

    Dynin-Omni uses masked diffusion over a shared discrete token space with a native unified backbone and modality-disentangled multi-stage training.

  • Results

    Dynin-Omni consistently outperforms leading unified models across modalities while remaining competitive with modality-specialized experts.

  • Takeaways & Limitations

    Masked diffusion provides a unified paradigm for omnimodal understanding and generation across text, image, speech, and video.

  • Takeaways & Limitations

    Perception-centric alternatives commonly implement speech generation through auxiliary decoders or modality-specific pipelines rather than unified generative modeling.

Abstract

from arXiv · show

We present Dynin-Omni, the first masked-diffusion-based omnimodal foundation model that unifies text, image, and speech understanding and generation, together with video understanding, within a single architecture. Unlike autoregressive unified models that serialize heterogeneous modalities, or compositional unified models that require orchestration with external modality-specific decoders, Dynin-Omni natively formulates omnimodal modeling as masked diffusion over a shared discrete token space, enabling iterative refinement under bidirectional context. Dynin-Omni adopts a multi-stage training strategy with model-merging-based modality expansion and omnimodal alignment. We evaluate Dynin-Omni across 19 multimodal benchmarks spanning language reasoning, image generation and editing, video understanding, and speech recognition and synthesis. Dynin-Omni achieves 87.6 on GSM8K, 1733.6 on MME-P, 61.4 on VideoMME, 0.87 on GenEval, and 2.1 WER on LibriSpeech test-clean, consistently outperforming existing open-source unified models while remaining competitive with strong modality-specific expert systems. These results demonstrate the potential of masked diffusion as a unified paradigm for any-to-any modeling, providing a flexible foundation for real-time omnimodal systems, unified cross-modal retrieval and generation, and embodied multimodal agents.

1 Introduction

Dynin-Omni addresses serialization and orchestration limits in unified multimodal generation with a native masked-diffusion architecture. Its shared token space, staged training, and broad evaluations support balanced cross-modal performance.

  • Autoregressive models serialize all modalities token by token, limiting parallel generation for non-sequential modalities such as images.
  • Compositional systems improve flexible generation but introduce modality-specific components, heterogeneous objectives, and orchestration bottlenecks.
  • Masked diffusion enables bidirectional attention and parallel token refinement, making it suitable for modalities without strict left-to-right ordering.
  • Dynin-Omni natively unifies text, image, speech, and video capabilities through iterative masked diffusion over a shared discrete token space.It uses lightweight modality detokenizers rather than separate modality-specific generative models.
  • A modality-disentangled multi-stage training strategy separates modality expansion from capability scaling while preserving semantic consistency and flexible-length generation.
  • Dynin-Omni achieves competitive or state-of-the-art results across reasoning, multimodal understanding, image generation and editing, and speech tasks while narrowing the gap with specialized experts.

2 Related Work

Related work spans diffusion language models, perception-centric systems, compositional unified models, and native unified models. Dynin-Omni uses masked diffusion to unify token-level understanding and generation without external generators.

  • 2.1 Diffusion Language Models: Diffusion language models replace left-to-right decoding with iterative denoising, enabling any-order generation, intermediate editing, global coherence, and parallel token updates.
  • 2.2 Perception-centric and Unified Models: Perception-centric models extend multimodal inputs to video, speech, and audio but remain primarily optimized for perception and understanding.
  • 2.2 Perception-centric and Unified Models: Their speech generation commonly relies on auxiliary decoders or modality-specific pipelines rather than a unified generative framework.
  • 2.2 Perception-centric and Unified Models: Compositional unified models coordinate a shared backbone with external modality-specific generators, producing staged generation through separate modules.
  • 2.2 Perception-centric and Unified Models: Native unified models directly model multimodal perception and generation within one backbone, with earlier systems often using autoregressive image–text formulations.
  • 2.2 Perception-centric and Unified Models: Dynin-Omni encodes and decodes all modalities through a shared token space under one masked-diffusion objective, using parallel inference without external generators.

3 Dynin-Omni

Dynin-Omni represents text, images, videos, and speech in a shared discrete token space processed by one masked-diffusion backbone. A shared objective and lightweight frozen reconstruction modules support unified cross-modal modeling, while image-token reuse transfers visual knowledge to video understanding.

  • 3.1 Omnimodal Tokenization: Dynin-Omni encodes text, images, videos, and speech into a shared discrete token space for one masked-diffusion backbone.The backbone uses shared embeddings and Transformer parameters across modalities.
  • 3.1.2 Omnimodal Tokenization: Frozen detokenizers deterministically reconstruct modality outputs from predicted discrete tokens, while sequence-level generation remains inside the masked-diffusion backbone.This design avoids additional iterative generative stages and separate modality-specific generation pipelines.
  • 3.1.2 Omnimodal Tokenization: Images and videos share the same visual tokenizer, with each video formed by concatenating independently tokenized frames in temporal order.This preserves a unified visual vocabulary rather than introducing a separate video-tokenization branch.
  • 3.1.2 Omnimodal Tokenization: Image-understanding improvements consistently translate to stronger video-understanding performance, suggesting knowledge transfer through shared visual tokenization.The reported transfer is empirical rather than attributed to a separate video-specific tokenizer.
  • 3.2 Masked Diffusion Modeling: Masked diffusion corrupts a unified text, visual, and speech token sequence and reconstructs masked tokens from remaining context using bidirectional attention.The forward process samples a masking ratio and excludes prompts, modality delimiters, and role indicators from masking.
  • 3.2.1 Training Objective: The training objective minimizes negative log-likelihood only for ground-truth tokens at masked positions.An indicator function selects positions whose corrupted tokens equal [MASK].

4 Training Recipe

Dynin-Omni uses three training stages to expand modality coverage, align modalities, merge models, and progressively improve omnimodal capabilities. The recipe combines shared-token modeling with staged supervision and model merging to stabilize adaptation and preserve existing abilities.

  • Training overview: Three stages cover modality adaptation, omnimodal alignment, and enhanced capability learning while optimizing all parameters with fixed modality tokenizers and detokenizers.The stages progressively decouple modality expansion from cross-modal integration and capacity growth.
  • Stage 1: Modality adaptation: Stage 1 extends the backbone to video understanding and speech understanding and generation through a dedicated speech vocabulary.All modality tokens are organized in a unified token space; vision vocabulary includes image and video tokens.
  • Stage 1: Modality adaptation: A shared embedding matrix places modality tokens in one latent space without modality-specific embedding layers or projection heads.Video frames reuse the visual codebook, whose embeddings encode spatial and temporal semantics.
  • Stage 1: Modality adaptation: Text-centric alignment trains video and speech inputs against text using video captioning, ASR, and TTS, with <EOS> supervision deferred until Stage 2 and Stage 3.Stage 1 samples eight video frames at 128 × 128 resolution and supports speech segments up to approximately 10 seconds.
  • Stage 2: Omnimodal SFT: Model merging accelerates convergence and lowers final loss across T→T, T→I, and V→T, with 6:4 merging stabilizing V→T adaptation.The merged model retains up to 84% of original-backbone performance on existing modalities and uses α = 0.6 in the experiments.
  • Stage 2: Omnimodal SFT: Model merging addresses Stage 1 catastrophic forgetting by combining the adapted model with the original backbone before omnimodal SFT.For mismatched dimensions, new modality dimensions are inherited from Stage 1 while original dimensions are retained; identical dimensions are linearly interpolated.
  • Stages 2–3: Capability scaling: Stage 2 jointly trains text, image, video, and speech tasks, while Stage 3 increases difficulty, resolution, output quality, and speech duration under continual fine-tuning.Stage 3 uses 480 × 480 image understanding, 512 × 512 generation, 336 × 336 editing, and speech durations up to approximately 21 seconds.

5 Data Composition

Dynin-Omni allocates data progressively across three stages: modality-specific alignment first, balanced cross-modal supervision second, and high-quality capability scaling third. The datasets span video captioning, speech, dialogue, mathematical reasoning, and explicit reasoning traces.

  • Data allocation: Stage 1 prioritizes modality-specific datasets, Stage 2 increases cross-modal instruction data, and Stage 3 emphasizes reasoning traces and curated multimodal samples.This composition follows a progressive scaling strategy across the training pipeline.
  • Stage 1: Modality alignment: Stage 1 aligns video and speech with text using WebVid-10M and approximately 14K hours of speech data for ASR and TTS.WebVid-10M contains 10.7M video–text pairs; speech data combines GigaSpeech, LibriSpeech, and CommonVoice.
  • Stage 2: Omnimodal SFT: Stage 2 performs joint supervised fine-tuning across text, image, video, and speech using dialogue, general-knowledge, and mathematical-reasoning datasets.Text supervision includes Evol-Instruct, Magpie-Pro, Open-Platypus, OpenR1-Math, and OpenHermes-2.5.
  • Stage 3: Capability scaling: Stage 3 enhances advanced reasoning, long-form generation quality, and multimodal robustness with explicit chain-of-thought supervision and thinking-mode control tokens.It uses reasoning datasets including Llama-Nemotron Post-Training, OpenR1-Math-220K, Mixture-of-Thoughts, and OpenMathReasoning.

6 Evaluation

Dynin-Omni is evaluated across textual reasoning, multimodal understanding, image generation and editing, and speech recognition and synthesis. It generally outperforms prior unified models while remaining competitive with modality-specific experts.

  • Evaluation setup: The evaluation covers textual reasoning, multimodal understanding, image generation and editing, and speech recognition and synthesis.Baselines include modality-specific experts, perception-centric omnimodal models, and unified understanding-and-generation systems.
  • Textual reasoning: Dynin-Omni achieves comparable linguistic reasoning to text-only expert models while supporting omnimodal understanding and generation.It surpasses prior diffusion-based language models and leads unified models on GSM8K, MATH, and GPQA while remaining competitive on MMLU and ARC-C.
  • Multimodal understanding: Dynin-Omni maintains balanced image and video understanding, attaining the highest POPE and MME-P scores among unified models and matching or surpassing unified baselines on several video benchmarks.It remains competitive on GQA, MMMU, MMB, and ActNet-QA, while matching or surpassing baselines on Next-QAmc, MVBench, TempCompass, and VideoMME.
  • Image generation: Dynin-Omni remains competitive with specialized image-generation systems on GenEval and DPG-Bench despite supporting additional modalities.Image synthesis directly predicts discrete image tokens in the shared vocabulary, followed by deterministic detokenization.
  • Image editing: Dynin-Omni achieves strong image-editing performance, particularly for object replacement and removal, while remaining stable on additive and attribute-based modifications.This performance is obtained within the broader omnimodal setting that also supports speech and video capabilities.
  • Speech understanding and synthesis: Dynin-Omni ranks among the strongest unified ASR systems and significantly outperforms prior unified TTS systems without modality-specific speech decoders.Dedicated speech models remain the strongest ASR and TTS specialists, but Dynin-Omni retains competitive speech performance within one masked-diffusion backbone.

7 Analysis

The analysis examines vocabulary-merging strategies and diffusion sampling budgets, showing how each affects capability retention, modality expansion, reasoning, and generation quality.

  • 7.1 Merging Strategies: The three compared recipes are Shared Merging, Stage 1-Only Merging, and Modality-Disentangled Merging for vocabulary-extension parameters.Shared Merging interpolates original-vocabulary parameters and inherits new rows from Stage 1; Stage 1-Only inherits all vocabulary-related parameters from Stage 1.
  • 7.1 Merging Strategies: Modality-disentangled merging retains backbone parameters for the original vocabulary while taking newly introduced speech dimensions from Stage 1.Remaining matching-dimensionality backbone weights are interpolated between the backbone and Stage 1.
  • 7.1 Merging Strategies: Modality Disentanglement provides a stable initialization that mitigates catastrophic forgetting while preserving newly acquired modality-specific competence for Stage 2 fine-tuning.The analysis adopts Modality Disentanglement with α = 0.6 for initialization.
  • 7.2 Effects of Diffusion Sampling Steps: Reasoning accuracy depends strongly on diffusion timesteps: very low sampling steps produce near-random GSM8K performance, while higher steps improve accuracy before plateauing around 512–1024.The results indicate diminishing returns beyond a certain refinement budget.
  • 7.2 Effects of Diffusion Sampling Steps: Image-generation quality improves rapidly from 4 to 32 GenEval steps and saturates around 64–128, while image editing performs strongly at 8–32 steps.DPGBench remains relatively stable across 4–128 steps, and further editing refinement yields marginal gains beyond 128 steps.

8 Conclusion

Dynin-Omni unifies text, image, speech, and video capabilities through masked diffusion over a shared token space and a single architecture. Its multi-stage training strategy supports stable modality expansion and omnimodal supervised fine-tuning.

  • 8 Conclusion: Dynin-Omni unifies text, image, and speech understanding and generation with video understanding in one shared architecture.The model uses a unified discrete token space and iterative masked diffusion.
  • 8 Conclusion: Masked diffusion avoids autoregressive serialization constraints and compositional systems’ orchestration overhead from external decoders.All modalities are modeled through iterative masked diffusion over the unified discrete token space.
  • 8 Conclusion: Modality-disentangled model merging, scheduled padding prediction learning, and joint omnimodal supervised fine-tuning form the multi-stage training strategy.The strategy is presented as enabling stable modality expansion.

B.1 Dataset Curation

The appendix identifies the datasets and stage-wise training configurations used by Dynin-Omni, while Figure 14 documents the model-merging alternatives under vocabulary extension.

  • B.1 Dataset Curation: Table 10 reports the datasets used at each training stage, including their sources and sizes.Its task notation uses unified input–output directions such as V→T, and synthetic data are internally generated.
  • B.1 Dataset Curation: Table 11 summarizes the hyperparameters and configurations of the training recipe across stages.The passage describes the table as detailing stage-wise training configurations.
  • B.1 Dataset Curation: Figure 14 compares Shared Merging, Stage 1-Only Merging, and Modality-Disentangled Merging under vocabulary extension.The figure distinguishes how original-vocabulary and new speech-specific parameters are sourced across strategies.

B.2 Implementation Details

The implementation details state that Dynin-Omni adopts modality-disentangled merging among three vocabulary-merging strategies.

  • B.2 Implementation Details: Among three vocabulary-merging strategies, the implementation adopts the modality-disentangled strategy shown in Figure 14(c).This choice is reported in the stage-wise training configuration discussion.
Loading 2604.00007v1…