Source-linked AI summary
LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning
Zebin You, Shen Nie, Xiaolu Zhang, Jun Hu, Jun Zhou, Zhiwu Lu, Ji-Rong Wen, Chongxuan Li
TL;DR
Existing multimodal models rely predominantly on autoregressive modeling, motivating exploration of purely diffusion-based alternatives. LLaDA-V combines masked diffusion with visual instruction tuning for multimodal understanding, achieving stronger data scalability than LLaMA3-V and state-of-the-art results among hybrid and purely diffusion-based MLLMs.
Problem
Existing MLLMs predominantly rely on autoregressive models, while purely diffusion-based multimodal understanding remains largely unexplored.
Method
LLaDA-V combines a vision encoder and MLP connector with LLaDA’s masked diffusion language model for visual instruction tuning and multimodal dialogue processing.
Results
Across 18 multimodal tasks, LLaDA-V achieves superior performance on 11 tasks versus LLaMA3-V and state-of-the-art results among hybrid and purely diffusion-based MLLMs.
Takeaways & Limitations
The findings support further exploration of large language diffusion models for multimodal understanding.
Takeaways & Limitations
For high-resolution images, splitting and resizing segments before SigLIP2 processing may reduce visual-representation efficiency and accuracy.
Abstract
from arXiv · showhide
In this work, we introduce LLaDA-V, a purely diffusion-based Multimodal Large Language Model (MLLM) that integrates visual instruction tuning with masked diffusion models, representing a departure from the autoregressive paradigms dominant in current multimodal approaches. Built upon LLaDA, a representative large language diffusion model, LLaDA-V incorporates a vision encoder and MLP connector that projects visual features into the language embedding space, enabling effective multimodal alignment. Our empirical investigation reveals several intriguing results: First, LLaDA-V demonstrates promising multimodal performance despite its language model being weaker on purely textual tasks than counterparts like LLaMA3-8B and Qwen2-7B. When trained on the same instruction data, LLaDA-V is highly competitive to LLaMA3-V across multimodal tasks with better data scalability. It also narrows the performance gap to Qwen2-VL, suggesting the effectiveness of its architecture for multimodal tasks. Second, LLaDA-V achieves state-of-the-art performance in multimodal understanding compared to existing hybrid autoregressive-diffusion and purely diffusion-based MLLMs. Our findings suggest that large language diffusion models show promise in multimodal contexts and warrant further investigation in future research. Project page and codes: https://ml-gsai.github.io/LLaDA-V-demo/.
1 Introduction
LLaDA-V extends masked diffusion language modeling to multimodal understanding and shows competitive performance, scalability, and benchmark results against autoregressive, hybrid, and diffusion-based models.
- Existing diffusion-based MLLMs either rely on autoregressive language models or have limited language modeling capacity and suboptimal performance.
- LLaDA-V combines a vision encoder and MLP connector with LLaDA to map visual features into the language embedding space for joint visual-text processing.
- LLaDA-V shows stronger data scalability than LLaMA3-V across several benchmarks, especially multidisciplinary knowledge and mathematical reasoning tasks.
- 11 tasks favor LLaDA-V over LLaMA3-V despite its slightly weaker language tower, while MMStar results are comparable to Qwen2-VL at 60.1 vs. 60.7.
- LLaDA-V is introduced as a purely diffusion-based MLLM for multimodal understanding.
- LLaDA-V achieves state-of-the-art multimodal performance among hybrid autoregressive-diffusion and purely diffusion-based MLLMs.
2 Preliminaries
The preliminaries frame LLaDA-V through masked diffusion language modeling and visual instruction tuning, two components that provide its language and multimodal foundations.
- Large Language Diffusion Models: Discrete diffusion models provide an alternative language-modeling paradigm to the predominantly autoregressive approach, with masked diffusion as a promising variant.
- Large Language Diffusion Models: LLaDA has demonstrated performance comparable to strong autoregressive models such as LLaMA3-8B-Instruct while retaining masked diffusion properties.
- Large Language Diffusion Models: LLaDA uses a masked diffusion process that corrupts tokens with masks and iteratively reconstructs them from an all-mask sequence.
- Visual Instruction Tuning: Visual instruction tuning combines a vision tower, an MLP connector, and an LLM to project image representations into the LLM word-embedding space.
- Visual Instruction Tuning: Visual instruction tuning enables strong multimodal understanding with fewer than 1M image-text pairs.
3 Method
LLaDA-V extends masked language diffusion to multimodal instruction tuning by integrating visual features with the LLaDA language embedding space and predicting masked response tokens. Its method combines specialized attention and iterative diffusion inference with multi-stage training for visual understanding and multimodal reasoning.
- Architecture and objective: LLaDA-V maps vision-tower features into LLaDA’s language embedding space through an MLP connector for joint visual-textual processing.The model extends visual instruction tuning to a language diffusion model and supports multi-turn multimodal dialogues.
- Architecture and objective: During training, image features and prompts remain clean while response tokens are randomly masked and predicted.The objective predicts masked response tokens conditioned on visual features and dialogue prompts.
- Attention design: Bidirectional attention achieves superior performance to the alternative attention design examined in ablation studies.The design enables access to the entire dialogue context during mask prediction.
- Inference: At inference, each response begins fully masked and is iteratively refined through the reverse masked-diffusion process as mask levels decrease.LLaDA-V generates dialogue responses through diffusion sampling rather than autoregressive next-token prediction, using low-confidence remasking.
- Training strategy: LLaDA-V uses three training stages covering language-image alignment, visual instruction tuning, and multimodal reasoning enhancement.The stages progressively align visual and language representations, develop instruction following, and improve reasoning flexibility across diverse multimodal data.
- Training strategy: The instruction-tuning stages include approximately 10M single-image samples and approximately 2M diverse samples spanning images and video.These stages expand from single-image understanding to multi-image and temporal multimodal scenarios.
4 Experiment
LLaDA-V is evaluated through staged training, data-scaling experiments, broad multimodal benchmarks, and attention-mask ablations. It scales better than LLaMA3-V on several tasks and outperforms diffusion-based alternatives, while remaining weaker on some scene-understanding comparisons.
- 4.1 Experimental Settings: LLaDA-V is trained through sequential stages using alignment, single-image, onevision, reasoning-focused, and combined multimodal datasets.The first stage trains only the projector; subsequent stages train the full model.
- 4.2 Data Scalability of LLaDA-V: LLaDA-V demonstrates stronger data scalability than LLaMA3-V across several benchmarks, particularly for multidisciplinary knowledge and mathematical reasoning.Both models use the same instruction data and training pipeline for this comparison.
- 4.3 Benchmark Results: LLaDA-V outperforms hybrid and pure diffusion models and surpasses LLaMA3-V on 6 of 9 multidisciplinary and mathematical benchmarks.It is stronger than LLaMA3-V on many knowledge, reasoning, multi-image, and video tasks, but less competitive on chart/document and real-world scene tasks.
- 4.2 Data Scalability of LLaDA-V: 1M samples let LLaDA-V outperform LLaMA3-V trained with 9M samples on MMMU-Pro, while LLaDA-V lags on AI2D and RealworldQA.The comparison favors LLaDA-V on some reasoning tasks but not on chart/document and real-world scene understanding.
- 4.1 Experimental Settings: The evaluation covers multidisciplinary knowledge, mathematical reasoning, chart and document understanding, real-world scenes, and multi-image and video understanding.The experiments compare LLaDA-V with autoregressive, hybrid autoregressive-diffusion, and pure diffusion models across 18 benchmarks.
- 4.4 Ablation Study: The no-mask bidirectional attention strategy outperforms dialogue-causal masking on 7 of 12 benchmarks and is adopted in LLaDA-V.The authors hypothesize that access to the entire dialogue context improves performance.
5 Related Work
Related work places LLaDA-V within masked diffusion language modeling and multimodal architectures spanning autoregressive, hybrid, and pure diffusion approaches. The paper builds on LLaDA’s scaling results while addressing its unexplored multimodal capabilities.
- Diffusion Language Models: Masked diffusion models have shown competitiveness with autoregressive models, and LLaDA scales this approach to 8B parameters.LLaDA is described as rivaling modern LLMs across a wide range of downstream tasks while remaining slightly inferior to LLaMA3-8B in language performance.
- Multimodal Understanding: LLaDA-V’s stronger performance across more tasks than the LLaMA3 baseline motivates extending large language diffusion models to multimodal understanding.This comparison is notable because LLaDA’s language performance remains slightly below LLaMA3-8B.
- Multimodal Understanding: Existing multimodal diffusion approaches are grouped into autoregressive, autoregressive-diffusion hybrid, and pure diffusion models.The related-work discussion identifies D-DiT as a closely related pure diffusion model.
6 Conclusion
LLaDA-V extends large language diffusion models to multimodal understanding and reports strong performance and data scalability, while acknowledging image-processing and hallucination limitations.
- LLaDA-V is a purely diffusion-based MLLM for both training and sampling, extending large language diffusion to multimodal understanding.
- LLaDA-V achieves superior performance among hybrid autoregressive-diffusion and purely diffusion-based models.
- LLaDA-V delivers better data scalability and performance across more benchmarks than LLaMA3-V under the same training strategy.
- High-resolution image processing may reduce visual-representation efficiency and accuracy because image segments are split, resized, and concatenated.Unlike Qwen2-VL, LLaDA-V does not use native dynamic resolution support.
- Like many advanced MLLMs, LLaDA-V may generate hallucinations, including factually incorrect content or information absent from the input.
A The Formulation of Masked Diffusion Models
Masked diffusion models corrupt tokens through independent masking and generate text by progressively reconstructing masked content in a reverse process.
- The forward process independently masks each token according to noise level t, where K is vocabulary size and N is sentence length.
- LLaDA-V sets α_t = 1−t, so each token is masked with probability t and remains unchanged with probability 1−t.
- The reverse process starts from a fully masked sequence and gradually transforms masked tokens into meaningful content.
- With α_t = 1−t, unmasked tokens remain unchanged while masked tokens either stay masked with probability s/t or are replaced with predicted content.
- Training minimizes an upper bound on the negative log-likelihood.
- At each reverse step, the model first identifies masked positions and samples their token values from the model distribution.
- LLaDA-V follows LLaDA by re-masking newly sampled tokens with the lowest confidence scores.
B Experiments
The implementation reuses official codebases and datasets from several established multimodal and evaluation projects.
- LLaDA-V leverages official codebases and datasets from MAmmoTH, VisualWebInstruct, LLaVA-NeXT, and LMMS-EVAL.
B.1 Model Architecture
LLaDA-V uses a bidirectional diffusion-language tower with a SigLIP2 vision encoder and MLP projector, replacing causal attention with bidirectional attention.
- Language tower: LLaDA replaces LLaMA3’s causal transformer with a bidirectional transformer, so it does not support KV caching.
- Implementation resources: Table 5 lists the code repositories and datasets leveraged in the implementation.
- Attention masks: Figure 4 contrasts standard causal, dialogue causal, and LLaDA-V’s bidirectional attention masks.In LLaDA-V, tokens can attend to all tokens in the sequence; PRM denotes prompt and RES denotes response.
- Vision tower: LLaDA-V’s vision tower uses SigLIP2 at 384×384 resolution and produces 729 visual tokens per image.
- Projector: A randomly initialized two-layer MLP serves as the projector for the visual features.
B.2 Attention Mask
LLaDA-V considers alternatives to the conventional causal attention mask for multimodal dialogue processing. These alternatives vary how tokens attend within and across dialogue turns.
- B.2 Attention Mask: The conventional autoregressive mask lets each token attend only to itself and preceding tokens.
- B.2 Attention Mask: The Dialogue Causal Mask permits bidirectional attention within each dialogue turn while preserving causality across turns.
- B.2 Attention Mask: LLaDA-V also evaluates a No Mask alternative alongside the conventional and dialogue-causal designs.
B.3 Case Studies
The case studies illustrate LLaDA-V’s visual understanding through descriptions of a serene Swiss Alps landscape. The example identifies the scene’s setting, objects, and spatial layering.
- B.3 Case Studies: The example depicts a lush path, wooden fence, walking person, distant church, valley, and mist-shrouded mountains in the Swiss Alps.
- B.3 Case Studies: The church appears in the background within a green valley framed by towering mountains.
- B.3 Case Studies: The scene is organized into foreground path, middle-ground valley, and background mountains.