PaperlayerView exact revision on arXiv

Source-linked AI summary

UniDDT: Unifying Multimodal Understanding and Generation with Decoupled Diffusion Transformer

Shuai Wang, Liang Li, Yang Chen, Ruopeng Gao, Yao Teng, Limin Wang

arXiv:2606.16255v1cs.CV

TL;DR

Existing unified multimodal models struggle to reconcile understanding and generation, fragmented visual spaces, and task-specific training data. UniDDT addresses these issues with unified semantic encoding, a decoupled diffusion decoder, and dual data structures, achieving strong results across generation and understanding benchmarks.

  • Problem

    Existing unified multimodal models face conflicts between understanding and generation, fragmented visual spaces, and task-specific training data that overlooks text-image duality.

  • Method

    UniDDT combines a Noisy ViT encoder and LLM for unified semantic encoding with a separate diffusion decoder and latent visual space.

  • Results

    UniDDT achieves 0.87 overall GenEval, 86.9 DPG, 1699.5 MME perception, and 76.5 overall SEEDbench scores.

  • Takeaways & Limitations

    UniDDT demonstrates a decoupled yet unified design for multimodal understanding and generation, supporting semantic consistency and scalability.

  • Takeaways & Limitations

    Because its original image-text data mainly uses model-generated captions, Native-UniDDT has limited understanding and instruction-following ability and is restricted to image captioning.

Abstract

from arXiv · show

Unified Multimodal Models (UMMs) have emerged as a critical direction for general-purpose multimodal intelligence, integrating understanding and generation into a single framework. However, existing UMMs face prominent challenges: (1) the inherent learning conflicts between visual understanding and generation tasks, leading to suboptimal modeling in both tasks; (2) different understanding and generation visual spaces impeding scalability; (3) over-reliance on task-specific data that neglects the duality of text-image understanding and generation. To address these challenges, we propose UniDDT, which leverages a Noisy ViT encoder along with an LLM to unify semantic encoding for visual generation and understanding tasks, while employing a separate diffusion decoder to decouple diffusion decoding from text decoding. With this Noisy ViT encoder, UniDDT is able to leverage the latent space as a unified visual representation, enabling seamless compatibility between understanding and generation tasks. Thus, the scalability within the generation tasks and the semantic expressiveness within understanding tasks can be balanced. Also, we construct dual data structures from the same image-text pairs, fostering interdependence between the generation and understanding data to exploit their inherent duality. Extensive experiments demonstrate that UniDDT achieves effective unification of multimodal understanding and generation with enhanced semantic consistency and scalability. For visual generation tasks, our UniDDT achieves 0.87 GenEval score and 86.9 DPG overall score. For multimodal understanding tasks, our UniDDT achieves 1699.5 score on MME benchmark and 76.5 overall score on SEEDbench.

1. Introduction

UniDDT addresses modeling, visual-space, and training-data limitations in unified multimodal models by combining unified semantic encoding with decoupled visual decoding. Its reported results span both visual generation and multimodal understanding benchmarks.

  • Motivation: Unified multimodal models integrate understanding and generation, but hybrid autoregressive-diffusion designs face substantial differences between these tasks.These designs use autoregressive modeling for text generation and diffusion for visual generation.
  • Challenges: Existing approaches assemble task-specific models, use fragmented visual spaces, and train understanding and generation components on separate data.These limitations hinder synergy, complicate workflows, impede scaling, and neglect the duality of the tasks.
  • Method: UniDDT uses a Noisy ViT encoder and LLM for unified semantic encoding, while a separate diffusion decoder decouples visual generation from text decoding.The architecture provides a unified visual space and a decoupled but unified understanding-generation design.
  • Results: 0.87 overall score on GenEval and 86.9 score on DPG benchmark demonstrate UniDDT's visual generation performance.These results are reported for VLM-UniDDT.
  • Results: 1699.5 perception score on MME Benchmark and 76.5 overall score on SEEDbench demonstrate UniDDT's multimodal understanding performance.These results are reported for VLM-UniDDT.

2. Related Works

Related work spans visual-language models built on LLMs, latent diffusion models for image generation, and unified multimodal approaches that combine discrete autoregressive and diffusion modeling. These methods align visual information with language or latent spaces, but unified systems differ in how they represent pixels and generate text and images.

  • Visual Language Models: Visual Language Models use LLMs with next-token prediction and pretrained visual encoders to align raw pixels with language embeddings.Early approaches used raw pixels or causal discrete visual tokens, while others combined visual encoders for fine-grained perception.
  • Visual Generative Models: Latent diffusion models typically combine a VAE and diffusion model trained on a VAE-shaped latent space.Visual foundation models can also be used to align intermediate representations and improve performance.
  • Unified Multimodal Models: Discrete token-based unified multimodal models convert pixels into discrete visual tokens and train under a unified next-token-prediction paradigm.These models are inspired by large language models and extend token-based modeling to multimodal inputs.
  • Unified Multimodal Models: AR-diffusion hybrids use autoregressive modeling for text generation and diffusion modeling for image generation to mitigate generative-performance loss.Another direction integrates specialized large multimodal models with diffusion models rather than relying solely on native unified frameworks.

3. Method

UniDDT unifies multimodal understanding and generation through a Noisy ViT encoder, LLM backbone, and dedicated diffusion decoder, using a shared latent visual space. Its staged training exploits dual image-text formats and generation intermediates to improve semantic consistency and generation quality.

  • Architecture: UniDDT combines a Noisy ViT encoder, LLM backbone, and diffusion decoder to unify semantic perception while dedicating diffusion decoding to visual generation.The Noisy ViT and LLM process shared visual semantics for understanding and generation, while the diffusion decoder handles image synthesis.
  • Architecture: The Noisy ViT extracts semantic features from noisy latents and timesteps, which the LLM causally encodes for understanding or refines for diffusion-based generation.For understanding, the LLM autoregressively decodes text; for generation, refined visual features condition the diffusion decoder.
  • Diffusion Decoder: The diffusion decoder estimates velocity from noisy latents, timesteps, and refined visual features, and can be trained using only those refined features even with the encoder and LLM frozen.The decoder follows stacked Attention and FFN blocks and uses attention rather than AdaLN-zero; full-attention refiner blocks improve training stability.
  • Unified Visual Space: UniDDT uses the latent space as its principal visual space because its understanding performance is comparable to pixels while its generation performance is significantly better.The paper reports minimal understanding degradation in latent space and no observed scaling advantage for pixel space over latent space.
  • Training Strategy: UniDDT first distills a pretrained vision-language teacher into the Noisy ViT, freezes the encoder and LLM to warm up the diffusion decoder, then jointly trains all modules.Joint training samples understanding and generation formats from the same text-image pair, applying cross-entropy to text understanding and diffusion loss to image generation.
  • Training Strategy: During post-training, UniDDT freezes the Noisy ViT and LLM, trains only the diffusion decoder, and maximizes understanding-branch likelihood on intermediate generation states.This duality-based procedure is intended to improve generation quality and semantic consistency.

4. Experiments

Experiments show that UniDDT balances multimodal understanding and visual generation through a unified visual space, while staged training and duality-based post-training improve generation scaling. The model achieves strong generation quality and remains effective for understanding under acceptable noise levels.

  • Visual Space: Pixel space slightly outperforms latent space for understanding, but latent space provides better visual-generation scaling during pretraining.Generation performance shows clear scaling across training stages and spaces, while pixel-space scaling is not better than latent-space scaling in warmup and joint training.
  • Multimodal Understanding: VLM-UniDDT understands noisy inputs well under acceptable noise levels, while Native-UniDDT is evaluated only for captioning because its naive image-text data does not support instruction following.Understanding evaluation fixes timestep t = 1.0, although joint-training understanding timesteps are randomly sampled from [0, 1].
  • Visual Generation: 0.89 GenEval and 87.1 on DPGBench are achieved by Native-UniDDT-XL, while VLM-UniDDT reaches 0.87 on GenEval and 86.9 on DPGBench.Native-UniDDT-L achieves 0.88 GenEval and 86.6 DPG-Bench before scaling to Native-UniDDT-XL.
  • Visual Generation: The strong GenEval and DPG-Bench results indicate robust object-level compositionality, prompt-following, and semantic alignment without compromising generation quality through decoupled diffusion decoding.UniDDT remains competitive with, and often superior to, dedicated generative models and existing unified multimodal models.
  • Training Ablations: Training the diffusion decoder succeeds with frozen Noisy ViT and LLM components, improves with additional compute, and duality-based post-training exhibits clear generation scaling.Post-training significantly boosts generation performance, with visual generation improving steadily.

5. Limitation

The authors identify limitations in Native-UniDDT’s understanding and instruction-following capabilities due to caption-based training text, and note unexamined architectural and formulation choices that may improve performance. Consequently, understanding results are reported only for VLM-UniDDT, while stronger VAEs and JiT’s prediction formulation remain promising directions.

  • Data and model limitations: Caption-generated text limits Native-UniDDT’s understanding and instruction-following capabilities, leaving it capable only of image captioning.The authors therefore do not report Native-UniDDT’s understanding performance and provide results only for VLM-UniDDT.
  • Architectural and formulation limitations: A stronger VAE and JiT’s prediction formulation were not explored in the experiments, leaving room for further improvement.The experiments preceded JiT’s release, so the pixel-space experiments did not consider its proposed prediction formulation.

6. Conclusion

UniDDT addresses conflicting understanding and generation objectives, fragmented visual spaces, and training data that underuses text-image duality. Its decoupled yet unified design enhances semantic consistency and highlights objective decoupling, unified representation, and task duality as key directions for UMMs.

  • Existing UMMs face conflicting understanding and generation objectives, fragmented visual spaces that hinder scalability, and task-specific data that fails to exploit text-image duality.
  • UniDDT is proposed as a native UMM with a decoupled yet unified framework addressing these challenges.
  • UniDDT enhances semantic consistency between understanding and generation, supporting a perspective for next-generation unified model design.
Loading 2606.16255v1…