PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual Information-Guided Parallel Decoding for Diffusion Multimodal Large Language Models

Insu Lee, Wooje Park, Wonseok Shin, Jinwoo Son, Byonghyo Shim

arXiv:2608.26580v1cs.CVcs.CL

TL;DR

dMLLM token selection must account for visual grounding because early decoded tokens become context for later predictions, while existing approaches may rely on certainty or textual influence alone. VIG-Sampler uses token-to-image attention and diversity-aware subset selection, outperforming Info-Gain Sampler across captioning and VQA benchmarks, including with fewer decoding steps.

  • Problem

    Existing dMLLM samplers may prioritize linguistically likely or influential tokens without explicitly accounting for their grounding in the input image.

  • Method

    VIG-Sampler ranks tokens by attention to image tokens and penalizes candidates with image-attention distributions similar to previously selected tokens.

  • Results

    Across 7 captioning and VQA benchmarks with 3 open-source dMLLMs, VIG-Sampler outperforms Info-Gain Sampler by an average of 19.3 CIDEr points on captioning and 7.3 accuracy points on VQA.

  • Takeaways & Limitations

    Visual information-guided token selection improves parallel dMLLM decoding quality and can preserve performance with fewer decoding steps.

Abstract

from arXiv · show

Diffusion multimodal large language models (dMLLMs) have recently emerged as a new decoding paradigm for multimodal generation. Starting from a fully masked sequence, dMLLMs progressively decode the sequence by unmasking a subset of the remaining masked positions at each step. Since the selected tokens serve as the prediction context for subsequent steps, deciding which tokens to decode is crucial to the quality of the final output. The most common strategy prioritizes tokens based on a certainty measure that tends to favor tokens frequently observed in the training data. Recent approaches instead order tokens according to their influence on subsequent predictions, but do not explicitly account for the input image. We propose the Visual Information-Guided Sampler (VIG-Sampler), which prioritizes tokens based on their attention to image tokens. We further impose a constraint that penalizes candidate tokens whose image-attention distributions are similar to those of previously selected tokens, thereby increasing the information gain of the decoded subset. Extensive experiments on 7 captioning and VQA benchmarks with 3 open-source dMLLMs demonstrate the effectiveness of VIG-Sampler, which outperforms the Info-Gain Sampler by an average of 19.3 CIDEr points across the captioning benchmarks and surpasses it on COCO Caption while using only half as many decoding steps.

Introduction

dMLLMs decode multimodal responses iteratively, making token-selection order critical because early tokens condition later predictions. VIG-Sampler addresses visually ungrounded or redundant selections by using image attention to guide parallel decoding, improving results across multimodal benchmarks.

  • dMLLMs progressively unmask tokens without a fixed positional order, unlike autoregressive models that decode left to right.
  • Certainty-based selection can decode frequent but semantically uninformative tokens early, prematurely constraining subsequent response structure.Examples include copulas, punctuation marks, and end-of-text tokens.
  • A sitting-dog example shows that selecting the visually unsupported token “frisbee” before “sits” can steer later predictions toward an image-inconsistent answer.The resulting response may describe the dog playing frisbee despite the image showing a sitting dog.
  • VIG-Sampler ranks tokens using image attention and penalizes candidates whose attention distributions resemble those of already selected tokens.This combines visual grounding with complementary information gain during parallel decoding, without additional training or model forward passes.

Visual Information-Guided Sampler

VIG-Sampler selects masked tokens using image attention as a cue for visual grounding and informativeness, while discouraging redundant attention patterns within each parallel-decoded set.

  • Decoding process: dMLLMs iteratively predict all masked positions, then commit a subset of k positions for parallel decoding.The selected tokens become context for subsequent decoding steps.
  • Information gain: Information gain measures the reduction in average entropy of predictions at remaining masked positions after unmasking a selected token set.Higher information gain indicates greater reduction in uncertainty for subsequent predictions.
  • Limitations of prior selection: Information-gain evaluation across candidate subsets requires extra forward passes, while information gain alone may select tokens weakly grounded in the visual input.The computational cost is especially problematic for long multimodal inputs because self-attention scales quadratically with sequence length.
  • Motivating observations: Higher image-attention mass corresponds to higher information gain and a greater proportion of visually grounded tokens.Image-attention mass is the sum of attention weights over image tokens, and visual dependence is assessed by removing the image.
  • Motivating observations: Shared information ratio increases with mean pairwise image-attention similarity, indicating that similar attention distributions correspond to overlapping token information.The relationship is evaluated for selected set sizes k ∈{2, 4, 8}.
  • VIG-Sampler: VIG-Sampler reweights confidence using normalized image-attention mass, then penalizes pairwise image-attention similarity when selecting the k-token set.The guidance strength is controlled by γ, while similarity regularization is controlled by λ; no additional training or model forward passes are required.

Experiments

Across three open-source dMLLMs, VIG-Sampler achieves the strongest overall results across captioning and VQA settings, with larger gains at higher decoding budgets.

  • The evaluation compares VIG-Sampler with six sampling baselines across LaViDa, MMaDA, and LLaDA-V.
  • VIG-Sampler achieves the highest mean CIDEr across all model–budget settings and the best DetailCaps and VQA averages in five of six settings.
  • At k = 8, VIG-Sampler exceeds the second-best baseline by 12.7 mean CIDEr points on LaViDa and 14.5 on MMaDA.
  • At k = 8, corresponding CAPTURE gains over the second-best baseline are 8.4 on LaViDa and 4.7 on MMaDA.
  • VIG-Sampler at k = 8 matches or surpasses certainty-based sampling at k = 2 on COCO Caption with LaViDa and Flickr30K with MMaDA.

Analysis

The analysis links VIG-Sampler’s gains to higher cumulative information gain and complementary visual token selection, while showing robust performance across hyperparameters and generation lengths.

  • Figure 4 compares cumulative information gain across decoding steps for k ∈{2, 4, 8}.
  • VIG-Sampler achieves the highest cumulative information gain across all commit budgets.
  • Confidence and Info-Gain may commit overlapping tokens such as “penny” and “coin”, producing plausible but incorrect or repetitive continuations.
  • VIG-Sampler instead selects complementary, visually grounded tokens and generates a caption consistent with the image.
  • Performance remains robust across a broad range of γ and λ values, although some configurations outperform the fixed default γ = 1 and λ = 3.
  • VIG-Sampler consistently outperforms Info-Gain on COCO Caption and TextVQA across all tested generation lengths N at k = 4.

Conclusion

VIG-Sampler is a training-free parallel decoding strategy that combines visual attention with joint information-aware selection, improving quality across multimodal benchmarks and supporting fewer decoding steps.

  • VIG-Sampler uses token-to-image attention to guide training-free parallel token selection.
  • Its visual reward-guided scoring and joint information-aware set selection prioritize visually grounded and informative tokens.
  • Table 3 reports hyperparameter sensitivity on COCO Caption using LaViDa, with † marking the default setting.
  • Table 4 reports robustness to N using LaViDa at k = 4.
  • Across seven captioning and VQA benchmarks with three dMLLMs, VIG-Sampler consistently improves over existing sampling baselines.
  • The conclusion highlights stronger gains under larger commit budgets and preserved generation quality with fewer decoding steps.

Related Work

Prior dMLLM research has addressed efficiency, visual grounding, and token selection, but relatively little work has focused on jointly selecting visually grounded and informative tokens.

  • MLLMs support captioning, visual question answering, and visual reasoning by projecting visual information into a pretrained LLM’s representation space.
  • With dLLMs, multimodal models denoise sequences conditioned on multimodal inputs as an alternative to autoregressive generation.
  • Some dMLLM studies improve inference efficiency by pruning redundant masked or visual tokens during generation.
  • Other methods improve visual grounding through model-internal bias correction or classifier-free guidance.
  • Relatively little prior work addresses the joint selection of tokens that are both visually grounded and informative for subsequent predictions.
  • Common token-selection strategies use predictive certainty, entropy, or top-two probability margins to determine committed positions and token counts.

Additional Details

The paper evaluates VIG-Sampler across multimodal benchmarks and specifies its decoding, sampler, and computational settings. It also derives a greedy rule for selecting token subsets using image-attention similarity regularization.

  • Motivation study: The motivation study uses 100 COCO Captions validation images to analyze visual grounding, image-attention mass, and image-attention similarity.For visual-grounding analysis, masked-token predictions are compared with and without image embeddings.
  • Evaluation setup: VIG-Sampler is evaluated on four captioning and three VQA benchmarks covering general description, open-vocabulary generalization, fine-grained description, and visual question answering.The benchmarks include COCO Caption, Flickr30K, NoCaps, DetailCaps, and three VQA datasets.
  • Generation settings: Generation lengths vary by model and benchmark, including N = 32 for LaViDa and MMaDA except DetailCaps, where block decoding uses N = 128 and block length 16.LLaDA-V uses N = 16, 32, or 128 depending on the benchmark.
  • Greedy selection rule: The set-selection derivation shows that the greedy marginal-gain rule accounts for candidate image-attention similarity with already selected positions.Terms independent of the candidate do not affect which position maximizes marginal gain.
  • Decoding procedure: Algorithm 1 repeatedly runs one forward pass, scores masked positions, greedily selects k positions, and commits their predicted tokens before the next step.The procedure runs for T = N/k decoding steps.

Additional Experiments and Analysis

Additional experiments test VIG-Sampler across models, generation lengths, inference costs, and qualitative token-selection patterns. The analyses show robust performance, modest overhead, and more distinct visual grounding than comparison methods.

  • Generation-length robustness: VIG-Sampler maintains strong overall performance across different generation lengths at k = 2, demonstrating robustness to the choice of N.These comparisons use LaViDa and contrast VIG-Sampler with Info-Gain.
  • Inference cost: 17.9 GB peak GPU memory is used by VIG-Sampler and the baselines, while VIG-Sampler’s wall-clock time remains near confidence-based sampling and below Info-Gain.The reported results indicate only modest computational overhead for visual guidance.
  • Qualitative analysis: Confidence-based selection chooses uninformative tokens with diffuse image attention, whereas VIG-Sampler selects answer-relevant tokens grounded in distinct image regions.The qualitative examples compare token choices and their image-attention patterns.
  • Penalty ablation: Disabling the similarity penalty makes selected tokens attend to nearly identical image regions and provide little additional information beyond one another.The default penalty is intended to encourage complementary selected tokens.
  • COCO Caption examples: Qualitative COCO Caption examples show conventional methods producing incorrect tokens, while some visually aligned captions receive relatively low CIDEr scores.The figure annotates decoding steps and correct versus incorrect committed tokens.
Loading 2608.26580v1…