PaperlayerView exact revision on arXiv

Source-linked AI summary

Dimple: Discrete Diffusion Multimodal Large Language Model with Parallel Decoding

Runpeng Yu, Xinyin Ma, Xinchao Wang

arXiv:2505.16990v3cs.CV

TL;DR

Existing multimodal large language models rely mainly on autoregressive generation, while discrete diffusion training has supervision and trajectory-coverage challenges. Dimple addresses this gap with hybrid or diffusion-based initialization strategies and adds efficient, controllable inference techniques, achieving performance comparable to autoregressive baselines and state-of-the-art performance among dMLLMs.

  • Problem

    Most existing multimodal large language models rely exclusively on autoregressive generation, while discrete diffusion training provides limited supervision over tokens and generation trajectories.

  • Method

    Dimple uses autoregressive training followed by diffusion training, Dimple+ uses autoregressive multimodal initialization with diffusion training, and the paper introduces Confident Decoding, Prefilling, and Structure Prior.

  • Results

    Dimple and Dimple+ achieve performance comparable to autoregressive baselines under the same training budget, while Dimple+ achieves state-of-the-art performance among dMLLMs.

  • Takeaways & Limitations

    The results support the feasibility of multimodal discrete diffusion models with parallel decoding, improved inference efficiency, and structure-aware generation.

  • Takeaways & Limitations

    Dimple-7B is trained on less than 1/20th of the data used by state-of-the-art models, and full attention retains quadratic memory and computational complexity.

Abstract

from arXiv · show

In this work, we propose Dimple, the first Discrete Diffusion Multimodal Large Language Model (DMLLM). We observe that training with a purely discrete diffusion approach leads to significant training instability, suboptimal performance, and severe length bias issues. To address these challenges, we design a novel training paradigm that combines an initial autoregressive phase with a subsequent diffusion phase. This approach yields the Dimple-7B model, trained on the same dataset and using a similar training pipeline as LLaVA-NEXT. Dimple-7B ultimately surpasses LLaVA-NEXT in performance by 3.9%, demonstrating that DMLLM can achieve performance comparable to that of autoregressive models. To improve inference efficiency, we propose a decoding strategy termed confident decoding, which dynamically adjusts the number of tokens generated at each step, significantly reducing the number of generation iterations. In autoregressive models, the number of forward iterations during generation equals the response length. With confident decoding, however, the number of iterations needed by Dimple is even only $\frac{\text{response length}}{3}$. We also re-implement the prefilling technique in autoregressive models and demonstrate that it does not significantly impact performance on most benchmark evaluations, while offering a speedup of 1.5x to 7x. Additionally, we explore Dimple's capability to precisely control its response using structure priors. These priors enable structured responses in a manner distinct from instruction-based or chain-of-thought prompting, and allow fine-grained control over response format and length, which is difficult to achieve in autoregressive models. Overall, this work validates the feasibility and advantages of DMLLM and enhances its inference efficiency and controllability. Code and models are available at https://github.com/yu-rp/Dimple.

1 Introduction

Dimple introduces multimodal discrete diffusion models that combine autoregressive and diffusion training to address limitations of pure diffusion training. The paper also develops techniques for faster, more controllable inference while demonstrating competitive performance with autoregressive baselines.

  • 1 Introduction: Dimple and Dimple+ are Discrete Diffusion Multimodal Large Language Models designed to add multimodal understanding to discrete diffusion generation.Dimple uses autoregressive training followed by diffusion training, while Dimple+ applies diffusion training to an autoregressive multimodal initialization.
  • 1 Introduction: The proposed first-autoregressive-then-diffusion strategy is more effective for multimodal understanding than pure diffusion training.Ablation results report that autoregressive-then-diffusion training outperforms pure diffusion tuning across all evaluated benchmarks.
  • 1 Introduction: Dimple and Dimple+ match the performance of autoregressive baselines under the same training budget, with Dimple+ achieving state-of-the-art performance among dMLLMs.The models retain parallel decoding and structure-aware generation capabilities.
  • 1 Introduction: The reverse diffusion process uses bidirectional attention to iteratively denoise noisy token sequences, with a weighted cross-entropy training loss.The loss is implemented as masked language modeling with a time-dependent random mask ratio and supervision on masked tokens.
  • 1 Introduction: Pure discrete diffusion training provides incomplete supervision across tokens and generation timesteps, unlike autoregressive next-token training.Diffusion training supervises only masked tokens and samples only a single or few timesteps, leaving the full generation trajectory uncovered.

3 Methodology

The methodology mitigates discrete diffusion training limitations through autoregressive initialization or pretraining, then applies diffusion-based multimodal training. It also introduces adaptive confidence-based decoding and prefilling to improve inference efficiency while retaining generation quality.

  • Training Strategies: Two strategies mitigate discrete diffusion training limitations: autoregressive training before diffusion, or autoregressive MLLM initialization followed by diffusion training.The first targets multimodal understanding; the second adds parallel decoding while retaining pretrained multimodal ability.
  • Dimple Training: Dimple uses a three-stage pipeline: Autoregressive Alignment, Autoregressive Instruction Tuning, and Diffusion Instruction Tuning.It starts from Dream, Qwen2.5-VL’s vision encoder, and a randomly initialized two-layer projector.
  • Dimple Training: During Dimple’s autoregressive phases, causal attention and next-token prediction are used; diffusion training restores full attention and diffusion-based loss.The alignment phase trains only the projector, while instruction tuning trains all components except the vision encoder.
  • Dimple+ Training: Dimple+ initializes from Qwen2.5-VL and uses a single diffusion instruction-tuning phase with full attention, updating all components except the vision encoder.Its training data combines LLaVA-NEXT instruction data with 1.3M text-only instruction-following samples.
  • Inference Techniques: Confident Decoding updates every position above confidence threshold γ, or the highest-confidence position when none qualifies.This adapts the number of tokens decoded per iteration to prediction confidence, enabling simultaneous high-confidence updates while avoiding uncertain updates.
  • Inference Techniques: Prefilling reuses prompt key-value pairs after the first generation step, reducing per-step complexity from O((L_prompt + L_answer)^2) toward O(L_answer^2).Because dMLLM decoding uses full attention, the technique is not strictly lossless.

4 Experiments

Experiments show that Dimple models match or exceed autoregressive and other discrete-diffusion multimodal baselines, while prefilling and confident decoding improve inference efficiency with limited performance loss.

  • Benchmark Performance: Dimple achieves 62.4% average performance, 1.8% above its autoregressive baseline, under matched training conditions.The comparison uses the same data, initialization, and total training iterations for each model and baseline.
  • Benchmark Performance: Dimple+ significantly outperforms other dMLLM baselines on most benchmarks and establishes state-of-the-art dMLLM performance.Dimple+ uses 2.6M training samples, compared with more than 17M for LLaDA-V.
  • Prefilling: Prefilling causes only a 0.8% average performance drop across benchmarks while providing substantial inference-speed improvements.The ablation evaluates response lengths of 4 and 8 with and without prefilling.
  • Confident Decoding: Confident Decoding reduces decoding steps to 1.25–2.87 for response length 8, yielding approximately 3× to 6× speedup.The method dynamically adjusts the number of decoded tokens per iteration using confidence.
  • Confident Decoding: Compared with fixed-step strategies, Confident Decoding preserves performance comparable to 8-step baselines using about 2 steps.Reducing other methods to 2 steps causes performance drops of up to 10% on CharQA for MaskGIT and Entropy.

5 Generation Behaviors

Dimple’s generation behavior differs from autoregressive decoding by supporting direct structure priors, early answer formation, structured formats, and controlled response lengths.

  • Structured Reasoning: Structure priors fix tokens at arbitrary positions before generation, enabling direct control over generated content.This capability differs from the indirect guidance used by autoregressive chain-of-thought strategies.
  • Structured Reasoning and Early Answering: Dimple decodes the correct answer “scissors” at the 10th iteration before completing the full response.Later iterations complete the surrounding reasoning trajectory.
  • Structured Output: Structure priors such as “{date:” and “time:” define a JSON-like response layout that Dimple follows successfully.The priors specify the desired output format and required attributes.
  • Parallel Decoding: Confident Decoding dynamically varies tokens per iteration, decoding 9 date-and-year tokens simultaneously and completing 22 generated tokens in 7 iterations.This corresponds to one-third as many iterations as generated tokens in the example.
  • Length Control: For target response lengths of 16 and 32, end-position structure priors let the model adjust its reasoning span to fill the available token budget.The forced suffix is “Thus, the answer is \box{”.

6 Related Work

Related work develops continuous and discrete diffusion approaches for language generation, hybridizes diffusion with autoregression, and studies their training objectives and controllability.

  • Diffusion Spaces: Prior work explores continuous diffusion spaces, including partial noising, continuous relaxations, embedding-space diffusion, and continuous-time formulations.These approaches address sequence generation through continuous representations or relaxations of categorical distributions.
  • Discrete Diffusion: Discrete diffusion research includes SDDM and RDM, with methods tailored to linguistic structure and flexible stepwise sampling.The paper follows masked language modeling for its language diffusion process.
  • Training Objectives: MDLM, MD4, and SEDD show that weighted masked-language-modeling and score-entropy losses can train diffusion language models.These objectives provide alternatives for learning discrete diffusion models.
  • Hybrid Models: Hybrid approaches such as AR-Diffusion and SSD-LM combine diffusion and autoregressive modeling to capture benefits of both paradigms.SSD-LM conducts diffusion in the natural vocabulary space using simplex projection.

7 Conclusion

Dimple uses diffusion multimodal generation while matching autoregressive baselines, with training, evaluation, and response-length procedures detailed in the supplementary material. Its main limitation is substantially smaller training data than the strongest models.

  • Dimple and Dimple+ match their autoregressive baselines while providing parallel decoding and structure-aware generation.
  • Dimple-7B is trained on less than 1/20th of the data used by Qwen2.5-VL and Eagle2, limiting comparison with the most powerful models.
  • Dimple training uses autoregressive alignment and instruction tuning followed by diffusion-based instruction tuning.
  • Diffusion tuning masks response tokens according to a linear schedule proportional to the diffusion time step.
  • Dream generates outputs at a predefined response length, padding shorter textual responses instead of using an [EOS] token.
  • Evaluation uses deterministic MaskGIT decoding, with response lengths 4, 8, and 16 tested for most datasets and 64 fixed for MMVet.

S2 Confident Decoding

Confident Decoding selects masked positions according to prediction confidence rather than updating a fixed number of tokens. It decodes confident positions together and uses fallback selection when the threshold is unmet.

  • Confident Decoding dynamically selects masked positions based on prediction confidence.
  • At each step, positions exceeding threshold γ are decoded together; otherwise, a fallback algorithm selects positions within the decoding budget.
  • Confidence may be computed from maximum probability, entropy, or the margin between the two highest probabilities.
  • The algorithm takes logits, temperature, threshold γ, fallback count K, and masked-token count N, then outputs an updated token sequence.
  • The implementation computes confidence from prediction probabilities before temperature, top-p, or top-k revisions.

S3 Ablation on Dimple Training

Ablations show that autoregressive preparation improves multimodal alignment and instruction tuning relative to purely diffusion-based training. The hybrid AT+DT strategy also reduces length sensitivity observed in pure diffusion models.

  • Training Strategies: The hybrid pipeline combines Autoregressive Alignment, Autoregressive Tuning, and Diffusion Tuning, unlike the two-phase pure diffusion pipeline.
  • After Alignment: AA-aligned models outperform DA-aligned models across all captioning metrics on 128 held-out LLaVA-CC3M images.GPT-4o rated accuracy, completeness, relevance, and conciseness on a 1–100 scale.
  • After Instruction Tuning: AT outperforms DT on 7 out of 9 benchmarks under the matched one-epoch learning-rate comparison.
  • After Instruction Tuning: AT-only models retain a gap between autoregressive training and diffusion-based inference, motivating the subsequent diffusion-tuning stage.
  • After Instruction Tuning: AT-then-DT outperforms pure diffusion tuning across all benchmarks in the corresponding comparison.
  • Length Bias: Pure diffusion shows severe length bias, with ChartQA accuracy falling from 42.7% to 8.6% as response length rises from 8 to 32.
  • Length Bias: AT+DT remains relatively stable across response lengths, whereas pure diffusion may continue generating after a correct answer.

S4 Sensitivity Analysis on Generation Parameters

The sensitivity analysis examines response length, decoding steps, confidence threshold, temperature, and top-p. Across most datasets, response length affects performance but generally has only a minor impact.

  • The analysis varies response length L, decoding steps k, confidence threshold γ, temperature t, and top-p p.
  • Response Length: With k = L, γ = 1, t = 0, and p = 0.01, response length generally has a minor performance impact across most datasets.

2. Steps (k)

The analysis examines how decoding parameters affect performance and speed. Smaller k trades performance for faster decoding, while confident decoding is stable across a broad γ range; temperature matters more than top-p.

  • Steps (k): Reducing k degrades performance, revealing a trade-off between decoding speed and performance.The experiment fixes L = 8, γ = 1, t = 0, and p = 0.01 while varying k.
  • Steps (k): Confident decoding remains stable for γ ∈[0.8, 1.0] and does not significantly affect performance across most datasets.The experiment fixes L = 8, t = 0, and p = 0.01.
  • Steps (k): Performance is best when t < 0.2, whereas higher temperature values significantly harm performance.The experiment fixes L = k = 8, γ = 1, and p = 0.95.
  • Steps (k): Performance is largely unaffected by top-p, suggesting low sensitivity to p.The experiment fixes L = k = 8, γ = 1, and t = 0.2.

S5 Further Discussion on Related Works

The discussion places Dimple among multimodal and parallel-decoding approaches while highlighting its adaptive decoding, efficiency techniques, and practical scope limitations. It also notes data, scalability, and safety boundaries.

  • Related dMLLMs: Concurrent dMLLMs include UniDisc, MMaDA, LLaDA-V, and LaViDA, with some jointly modeling multimodal and text generation through discrete diffusion.UniDisc and MMaDA are identified as unified discrete diffusion models.
  • Parallel Decoding: Prior parallel-decoding methods select positions using confidence scores or random subsets, while their per-step decoding count follows a predefined schedule.Confidence-based methods retain high-confidence predictions; other methods randomly select positions.
  • Parallel Decoding: Confident Decoding uses a fixed confidence threshold to dynamically determine how many tokens to update at each timestep.It avoids decoding tokens whose predicted confidence does not exceed the threshold and adapts speed to the input query.
  • Limitations: Dimple-7B uses less than 1/20th of the data used by Qwen2.5-VL and Eagle2, and full attention remains quadratic in memory and computation.The paper identifies scaling data and parameters, and optimizing inference, as future directions.
  • Broader Impact: Dimple remains susceptible to hallucination and generation bias, with safety concerns arising from both its underlying DLM and vision encoding process.The document also restricts release to academic and non-commercial research purposes and discourages harmful use.

S7 Examples

The examples show that decoding randomness affects meaningful response length, but the effect depends on the task. Mathematical reasoning tends to remain long, perception-only tasks short, and flexible visual-understanding tasks longer with randomness.

  • Examples: Randomness in decoding changes meaningful response length by altering the proportion of padding and mask tokens.Without randomness, Dimple tends to generate shorter responses; with randomness, it is more likely to generate longer responses.
  • Examples: Mathematical reasoning tasks produce relatively long answers regardless of decoding randomness because they require multi-step reasoning.This behavior is illustrated in Table S11.
  • Examples: Pure OCR recognition and counting tasks tend to yield short answers regardless of randomness because they primarily require perception.These examples are associated with Tables S12 and S13.
  • Examples: Tasks combining perception with visual understanding or image description produce much longer responses when decoding includes randomness.The comparison covers Tables S14 to S16; the setup uses response length 128, MaskGIT, and temperatures 0 versus 0.4.
  • Examples: The supplementary material includes examples of Dimple generations with explicit instruction and in-context learning.These examples are listed in Tables S17 and S18.
Loading 2505.16990v3…