PaperlayerView exact revision on arXiv

Source-linked AI summary

LaViDa: A Large Diffusion Language Model for Multimodal Understanding

Shufan Li, Konstantinos Kallidromitis, Hritik Bansal, Akash Gokul, Yusuke Kato, Kazuki Kozuka, Jason Kuen, Zhe Lin, Kai-Wei Chang, Aditya Grover

arXiv:2505.16839v4cs.CV

TL;DR

Autoregressive VLMs are difficult to parallelize and to control for bidirectional or structured generation, while diffusion models offer an underexplored alternative for multimodal tasks. LaViDa equips diffusion language models with visual inputs and supporting training and inference techniques, achieving competitive multimodal performance alongside speed-quality and controllability benefits. Its reported results include improved COCO captioning and constrained poem completion, though performance remains below larger or more data-rich state-of-the-art VLMs.

  • Problem

    Autoregressive VLMs have slow sequential inference and struggle with bidirectional or structurally constrained generation, motivating diffusion-based multimodal modeling.

  • Method

    LaViDa equips a diffusion language model with a vision encoder and adds complementary masking, Prefix-DLM caching, and timestep shifting for multimodal instruction following.

  • Results

    LaViDa achieves competitive performance across multimodal benchmarks, with +4.1 CIDEr and 1.92× speedup on COCO captioning and +59% on Poem Completion.

  • Takeaways & Limitations

    LaViDa offers a diffusion-based alternative to autoregressive VLMs with flexible speed-quality tradeoffs, controllability through text infilling, and bidirectional reasoning.

  • Takeaways & Limitations

    LaViDa remains behind larger or more data-rich state-of-the-art VLMs, leaving scaling to larger models and datasets as future work.

Abstract

from arXiv · show

Modern Vision-Language Models (VLMs) can solve a wide range of tasks requiring visual reasoning. In real-world scenarios, desirable properties for VLMs include fast inference and controllable generation (e.g., constraining outputs to adhere to a desired format). However, existing autoregressive (AR) VLMs like LLaVA struggle in these aspects. Discrete diffusion models (DMs) offer a promising alternative, enabling parallel decoding for faster inference and bidirectional context for controllable generation through text-infilling. While effective in language-only settings, DMs' potential for multimodal tasks is underexplored. We introduce LaViDa, a family of VLMs built on DMs. We build LaViDa by equipping DMs with a vision encoder and jointly fine-tune the combined parts for multimodal instruction following. To address challenges encountered, LaViDa incorporates novel techniques such as complementary masking for effective training, prefix KV cache for efficient inference, and timestep shifting for high-quality sampling. Experiments show that LaViDa achieves competitive or superior performance to AR VLMs on multi-modal benchmarks such as MMMU, while offering unique advantages of DMs, including flexible speed-quality tradeoff, controllability, and bidirectional reasoning. On COCO captioning, LaViDa surpasses Open-LLaVa-Next-8B by +4.1 CIDEr with 1.92x speedup. On bidirectional tasks, it achieves +59% improvement on Constrained Poem Completion. These results demonstrate LaViDa as a strong alternative to AR VLMs. Code and models will be released in the camera-ready version.

1 Introduction

LaViDa adapts discrete diffusion models to vision-language understanding, targeting autoregressive models’ limitations in inference speed and constrained generation. It combines a vision encoder with diffusion modeling and introduces techniques for efficient training, cached inference, and quality-preserving speed control.

  • Motivation: Autoregressive VLMs generate sequentially, limiting parallelism and making bidirectional or structurally constrained outputs difficult.These limitations affect tasks such as text infilling, constrained poem completion, and structured JSON extraction.
  • Motivation: Discrete diffusion models progressively transform masked tokens into meaningful text through a learned reverse process.Their bidirectional context and adjustable diffusion steps motivate their use for controllable and speed-flexible vision-language generation.
  • Approach: LaViDa is the first family of diffusion-based VLMs, integrating visual features into a pretrained diffusion language model through a vision encoder.The model uses a two-stage pipeline consisting of pretraining followed by supervised fine-tuning.
  • Technical challenges: Complementary masking addresses inefficient diffusion training by ensuring critical answer tokens contribute to learning.Standard masking can leave semantically important visual-question-answer tokens unmasked and excluded from the loss.
  • Technical challenges: Prefix-DLM enables caching of multimodal prompts, addressing repeated recomputation over visual and text tokens during diffusion decoding.This targets the practical inference bottleneck caused by long multimodal prompts.
  • Results: +59% on Poem Completion, +4.1 CIDEr on COCO image captioning, and 1.92× speedup demonstrate LaViDa’s constrained-generation and speed-quality advantages.Across MMMU, MathVista, ChartQA, and ScienceQA, LaViDa achieves competitive performance against autoregressive VLMs.

2 Background and Related Works

The paper situates LaViDa within vision-language and diffusion-language modeling. VLMs commonly connect a large language model to a vision encoder, while discrete diffusion models replace left-to-right generation with masked-token corruption and iterative reconstruction.

  • Vision-language models: VLMs extend large language models with visual understanding, commonly using a vision encoder connected to a language model through staged training.Pretraining typically aligns text-image pairs, followed by instruction-focused fine-tuning.
  • Diffusion language models: Discrete diffusion language models gradually corrupt discrete text into mask tokens and learn a reverse process that reconstructs meaningful text.This distinguishes their generation process from autoregressive next-token prediction.
  • Diffusion language models: The diffusion language-modeling loss is computed over masked tokens because the unmasked-token term has a closed-form representation independent of model parameters.This objective underlies the diffusion formulation used by LaViDa.
  • LaViDa context: LaViDa’s overall design combines a vision encoder, diffusion language model, and MLP vision projector.Figure 2 separates the image-encoding pipeline from the diffusion-language-modeling pipeline.

3 Method

LaViDa combines visual embeddings with a noncausal diffusion Transformer and develops complementary masking, Prefix-DLM caching, and schedule shifting for multimodal training and inference. These designs target loss coverage, prompt recomputation, and low-step sampling quality.

  • 3.1 Model Architecture: LaViDa connects a vision encoder to a diffusion language model through an MLP projection network.The architecture follows the common vision-language pattern while replacing causal language modeling with diffusion modeling.
  • 3.1 Model Architecture: The vision encoder processes five image views and applies pooling to reduce the visual sequence to 980 embeddings per image.The views comprise four crops of a resized image plus one view of the original image.
  • 3.1 Model Architecture: The diffusion Transformer uses noncausal attention and takes projected vision embeddings, a prompt, and a partially masked response as input.Its final linear layer produces token-wise logits for reconstructing the unmasked response.
  • 3.2 Training Algorithms: Complementary masking uses disjoint corrupted spans so all answer tokens eventually contribute to training, while copied vision embeddings improve efficiency.The method addresses sparse or semantically critical answer tokens being omitted from the loss.
  • 3.3 Inference Algorithms: LaViDa begins inference with L mask tokens and iteratively unmasks them over K timestamps, with L and K controlling generation length and computation.The number of functional evaluations determines how many tokens are generated per forward pass.
  • 3.3 Inference Algorithms: Prefix-DLM restricts visual and prompt tokens to attend within the prefix while answer tokens attend globally, enabling multimodal KV caching.This design yields a speedup of up to 3.9× on COCO captioning tasks.
  • 3.3 Inference Algorithms: Schedule shifting uses a convex schedule that unmasks more tokens earlier, improving performance when sampling uses few diffusion steps.The method ensures at least one token is unmasked per step.

4 Experiments

LaViDa is evaluated across vision-language understanding, reasoning, controllable text infilling, speed-quality tradeoffs, and design ablations. The experiments show competitive benchmark performance alongside gains from specialized training, caching, timestep scheduling, and complementary masking.

  • Main Results: LaViDa-L achieves the highest MMMU score among comparable models, at 43.3, while LaViDa performs competitively across general, reasoning, OCR, and science tasks.LaViDa also achieves the best and second-best ScienceQA scores, 81.4 and 80.2, respectively, but lags some autoregressive models on OCR because average pooling loses fine-grained spatial information.
  • Reasoning Distillation: LaViDa-Reason outperforms the stage-2 model across MathVista, MathVerse, and MathVision, with the largest gain on MathVision at +18% relative improvement.The reasoning-specialized model is trained with 19.2K chain-of-thought examples distilled from VL-Rethinker-7B.
  • Text Infilling: Both LaViDa variants achieve 100% constraint satisfaction on constrained poem completion, while autoregressive baselines remain below 50%.LaViDa-FIM additionally adapts token counts per line, supporting variable-length completions under line-level constraints.
  • Speed vs. Quality Trade Off: Adjusting discretization steps K provides a tunable speed-quality tradeoff for COCO captioning, while Prefix-DLM caching reduces latency by up to 3.9× with marginal performance cost.The experiments vary K across 32, 24, 16, and 8 steps, corresponding to NFE values from 100% to 25%.
  • Speed vs. Quality Trade Off: The convex timestep-shifting schedule with α = 3^-1 performs best among evaluated schedules, reaching 21.05 accuracy at NFE=50% on MathVision, 30% above 16.12.The same schedule behavior is also observed during chain-of-thought inference with LaViDa-Reason.
  • Ablation Studies: Complementary masking improves all evaluated benchmarks, including a 67% relative improvement on ScienceQA, with an 8% training slowdown.Higher-resolution inputs improve overall performance, with larger gains on OCR than on generic vision tasks.

5 Conclusion

LaViDa is presented as the first family of vision-language models based on discrete diffusion models, combining visual inputs with diffusion-language generation. The paper concludes that its training and inference techniques outperform naive diffusion adaptation for visual tasks.

  • LaViDa uses complementary masking, Prefix-DLM caching, and timestep shifting to improve training efficiency, inference speed, and sample quality.
  • LaViDa supports text-infilling and bidirectional context while achieving competitive performance against autoregressive models.
  • LaViDa offers controllable sampling schedules by varying diffusion steps, with discretization adjusted to decode at least one token per step.

B Additional Experiment Details and Results

The additional experiments describe LaViDa’s training and evaluation setup and provide qualitative evidence for text-infilling applications. These applications include structured extraction, image-based editing, and movie-script completion, with infilling presented as more straightforward than careful autoregressive prompting.

  • Training and Evaluation Setup: Pretraining uses LCS-558K with 558K image-text pairs, while fine-tuning primarily uses Open-LLaVa-Next data with adjusted source weights.
  • Training and Evaluation Setup: The evaluation uses LMMS-Eval with its default prompts, reporting benchmark splits and generation lengths in Table 6.
  • Qualitative Text-Infilling Results: LaViDa demonstrates text infilling for extracting image attributes in JSON, editing image-grounded sentences, and completing movie scripts.
  • Qualitative Text-Infilling Results: Text infilling is described as more straightforward with diffusion than with autoregressive models requiring careful prompting.

B.3 Math Reasoning

The math-reasoning experiments extend LaViDa-Reason with teacher-generated long chain-of-thought data and examine inference speed-quality tradeoffs. Across MathVision schedule settings, convex schedules perform best, while LaViDa’s MathVista frontier dominates the Open-LLaVA-Next-8B baseline in speed and quality.

  • Data and Training Setup: LaViDa-Reason is trained on long chain-of-thought traces generated by VLRethinker-7B and filtered for correct final answers.
  • Speed-Quality Tradeoff: Convex schedules perform best across different sampling-step choices for chain-of-thought math reasoning on MathVision.
  • Data and Training Setup: The appendix notes that the relevant data will be released in the camera-ready version.
  • Speed-Quality Tradeoff: LaViDa’s MathVista inference frontier strictly dominates Open-LLaVA-Next-8B in both throughput and accuracy.

B.4 Prefix-DLM

Prefix-DLM is used as an inference-only attention-mask strategy intended to combine parallel diffusion decoding with KV-cache efficiency. Training variants preserve the same loss and dynamics but incur substantial overhead, so the authors retain full-attention training.

  • Inference Algorithm: Prefix-DLM restricts image and prompt queries to prefix keys and values while allowing partially masked answer tokens to interact with the prefix and answer sequence.
  • Inference Algorithm: Unlike block-wise causal diffusion, Prefix-DLM preserves bidirectional context for the answer tokens while enabling KV-cache use.
  • Training Algorithm: Prefix-DLM-FT variants produce identical loss values and training dynamics, differing only in implementation efficiency.
  • Training Algorithm: The fastest Prefix-DLM-FT training version is 62% slower than the full-attention baseline because of batch-dependent masking overhead.
  • Training Algorithm: Because Prefix-DLM and Prefix-DLM-FT have mostly identical performance, the authors use full attention during training and retain Prefix-DLM as an inference option.

B.5 Ablation Studies

The ablation study compares SigLip, CLIP, and MetaCLIP as vision encoders for LaViDa. SigLip achieves the strongest overall performance after 1,500 training steps.

  • SigLip achieves the strongest overall performance among the evaluated vision encoders.The comparison includes SigLip, CLIP, and MetaCLIP after 1,500 training steps, roughly 200k training samples.
  • SigLip shows notable gains on VQAv2, ScienceQA, and AI2D.

B.6 Additional Scaling

Additional scaling experiments examine LaViDa on OCR and general understanding tasks. Doubling the training schedule improves performance, though further scaling was left for future work because of compute constraints.

  • Doubling LaViDa’s training schedule produces considerable performance gains on general understanding and OCR tasks.The results are reported in Table 12.
  • Further scaling experiments were left to future work because of compute constraints.

D Limitations

LaViDa’s main limitations concern scalability, OCR performance, context length, and inherited model or data risks. The authors identify gaps against larger or more data-rich VLMs and unresolved long-context limitations.

  • LaViDa remains behind state-of-the-art open-source VLMs such as LLaVa-OneVision and Qwen2.5-VL.These models use either more training data or larger model sizes, and the authors call for studying diffusion-model scaling.
  • LaViDa performs slightly worse than baselines on OCR tasks, mainly because average pooling compresses visual information.Removing pooling would exceed the context lengths of the LLaDa and Dream base models for longer multimodal inputs.
  • Dream’s longer-context extension remains weaker than LLaDa-8B and Llama-3-8B on the 4,096-context needle-in-a-haystack task.Dream-7B scores 72.4 Acc, compared with 91.6 Acc for LLaDa-8B and 95.4 Acc for Llama-3-8B.
  • The authors state that LaViDa may inherit biases from its base models and training data and may suffer from hallucinations.They recommend using the model for research on diffusion-based vision-language applications rather than other purposes.

F License Information for Data and Models

The paper reports licenses and sources for the datasets and models used, including compound licensing conditions and research-only restrictions. It also documents dataset disclaimers and generated training data.

  • Table 13 lists licenses and sources for the datasets and models used in the study.Entries marked with † or ‡ indicate compound or inferred licensing conditions, while some datasets lack published licenses.
  • CC-3M is provided without warranties, and Google disclaims liability for damages resulting from its use.
  • The training data combines ShareGPT4V and AVG-LLaVA under differing licensing conditions and was used strictly for academic research.ShareGPT4V is subject to CC BY-NC 4.0 and additional model-license restrictions, while AVG-LLaVA is Apache 2.0.
  • An LLM generated the mathematical reasoning data used for stage-3 training.The paper refers readers to Appendix B.3 for details.
  • The paper compares LaViDa with MMaDa using MME, MMMU, MMB, and average image-captioning latency.
Loading 2505.16839v4…