PaperlayerView exact revision on arXiv

Source-linked AI summary

Context-Aware Cluster Decoding: Semantic Anchor-Driven Coherence in dMLLMs

Yikai Zhao, Qiyan Zhao, Jiaquan Zhang, Xiaofeng Zhang, Xiaosong Yuan, Pengzhou Cheng

arXiv:2608.22367v1cs.CL

TL;DR

dMLLM long-form generation suffers semantic drift and repetition because confidence-only scoring ignores decoded-neighbor support and block partitioning can exclude strong semantic anchors. CACD combines multiplicative confidence-context scoring with block-free cluster commitment and architecture-aware calibration. Across three dMLLMs and four benchmarks, it delivers consistent quality gains and hallucination reduction, with larger advantages in some longer-generation settings.

  • Problem

    dMLLM long-form generation often suffers semantic drift and repetition because existing decoders can ignore decoded-neighbor support and exclude high-readiness anchors through block partitioning.

  • Method

    CACD is a training-free decoding method that combines softmax confidence with neighbor proximity, commits contextually ready positions as block-free clusters, and calibrates thresholds for architecture-specific confidence distributions.

  • Results

    Across three dMLLMs and four benchmarks, CACD consistently improves quality and reduces hallucination relative to Original, with larger gains in several longer-generation settings.

  • Takeaways & Limitations

    Visual integration strategies shape dMLLM confidence landscapes, making neighbor support and architecture-aware calibration relevant considerations for decoding-method design.

  • Takeaways & Limitations

    Per-model threshold calibration is required, and the experiments establish cross-architecture generalization at approximately 8B scale but not cross-scale generalization.

Abstract

from arXiv · show

Diffusion multimodal large language models (dMLLMs) frequently produce long-form outputs marred by semantic drift and repetition, with quality generally degrading as output length increases. We identify two structural deficiencies in existing decoding methods as primary drivers of these failures: confidence-based scoring ignores decoded-neighbor support, and block partitioning prevents access to high-readiness semantic anchors, together causing tokens to be committed before their local context is sufficiently established. We propose \ours{} (\textbf{C}ontext-\textbf{A}ware \textbf{C}luster \textbf{D}ecoding), a training-free decoding method that scores each masked position by a multiplicative composite of softmax confidence and neighbor proximity, promoting contextually ready tokens above isolated candidates while suppressing low-confidence positional noise, operating block-free to keep high-readiness anchors globally accessible. \ours{} further applies architecture-aware calibration to handle confidence heterogeneity induced by diverse visual integration strategies. Experiments on three dMLLMs across four benchmarks demonstrate consistent quality gains and hallucination reduction over Original, with larger gains in several longer generation settings, highlighting the importance of neighbor support and visual integration strategy for future dMLLM decoding method design. Our code is openly available at https://github.com/zhaoyk-sysu/CACD-dMLLM.

1 Introduction

dMLLMs often suffer semantic drift and repetition in long-form generation because confidence-only scoring and block partitioning can commit tokens without sufficient local semantic support. CACD addresses these structural problems with neighbor-aware, block-free cluster decoding and architecture-aware calibration.

  • Motivation: Long-form dMLLM outputs exhibit semantic drift and repetitive phrasing, with quality gaps widening monotonically as output length increases.Figure 1 illustrates repeated paragraphs, while Figure 2 reports length-dependent degradation.
  • Motivation: Existing decoders overlook local semantic connections between decoded tokens and masked neighbors, leaving subsequent positions without sufficient support.Positions with decoded neighbors are more contextually ready, but commitment order can defer them in favor of weaker candidates.
  • Structural deficiencies: Block partitioning excludes the globally highest-readiness anchor in 75.0% of first-round steps and 27.3% of all steps.These figures come from 90 LLaVABench-COCO samples using block_length=128.
  • CACD: CACD scores masked positions using a multiplicative composite of softmax confidence and local semantic context, commits qualifying tokens as clusters, and preserves global anchor accessibility.The framework is presented as plug-and-play and uses the joint score to rescue contextually ready tokens while suppressing positional noise.
  • Architecture-aware design: Distinct visual integration strategies shape confidence landscapes that existing text-only decoding methods do not account for.The paper identifies this architecture-related confidence heterogeneity as relevant to future dMLLM decoding design.

2 Related Work

Prior work improves commitment order through confidence scoring, block refinement, spatial frontiers, or temporal histories, but leaves bilateral neighbor support incompletely integrated into dynamic commitment scoring. CACD is positioned as a single-pass, neighbor-aware alternative to these approaches.

  • Diffusion decoding: Diffusion decoding starts from a fully masked sequence and iteratively commits subsets of positions selected from parallel predictions.Selection can use confidence or other criteria.
  • Commitment order: Prior studies show that commitment order affects generation quality, but do not provide an inference-time mechanism maintaining correct order under complex visual context and long outputs.HDLM reinforces coarse-to-fine semantic ordering during training rather than supplying the described inference-time mechanism.
  • Decoding order and block structure: Block-based methods refine boundaries but retain sequential structure that can prevent access to the highest-readiness anchor candidates.WavefrontDiffusion models spatial distance but remains confidence-based, while AHD uses a multi-step history buffer.
  • Confidence-based scoring: Confidence-based methods improve thresholding and dynamic scoring but treat confidence as the sole signal without measuring neighborhood support.Related contextual methods use cross-step consistency or frontier constraints rather than bilateral neighbor support in the commitment score.

3 Motivation

The motivation analysis identifies decoded-neighbor support and unrestricted anchor access as key determinants of prediction difficulty. CACD combines these ideas through context-aware scoring, full-sequence cluster commitment, and propagated local influence.

  • 3.1 Local Semantic Connections Determine Prediction Quality: Neighbor support is associated with prediction difficulty beyond raw confidence, with entropy dropping 5.3× from isolated to fully-neighbored positions.The analysis covers 1,000 MathVista samples and reports a commit-round gap peaking at ∆=12.4 rounds under pure confidence scoring.
  • 3.1 Local Semantic Connections Determine Prediction Quality: CACD uses neighbor proximity as a multiplicative amplifier on confidence so contextually ready tokens can outrank isolated candidates.This design reflects the measured entropy advantage of positions whose neighbors are already decoded.
  • 3.1 Local Semantic Connections Determine Prediction Quality: Pure confidence scoring can strand low-confidence, high-context tokens in persistent [MASK] voids that weaken support for neighboring positions.The commit-round gap peaks at ∆=12.4 rounds, and the resulting void compounding worsens with sequence length.
  • 3.2 Block Structure Can Exclude Globally Preferred Anchor Candidates: Block-based methods may exclude high-readiness anchors, forcing weaker tokens to commit first and leaving neighboring positions without strong local support.Fixed sequential partitions cannot guarantee access to the strongest available anchor, even with accurate scoring.
  • 3.2 Block Structure Can Exclude Globally Preferred Anchor Candidates: CACD operates block-free on the full sequence, commits positions above threshold as a cluster, and lets committed anchors boost adjacent masked positions.The unified approach addresses block exclusion and void compounding through cluster commitment and localized context propagation.

4 Method

CACD combines confidence with decoded-neighbor proximity, commits globally accessible clusters, and calibrates decoding across heterogeneous dMLLM confidence distributions. Its pipeline prioritizes semantic anchors before propagating local context outward.

  • Context-Aware Scoring: Decoded positions define left and right neighbor distances, which are aggregated through harmonic decay into a context score.When no decoded neighbor exists, the context score is zero and the score reduces to pure confidence.
  • Context-Aware Scoring: CACD scores each masked position using a composite of token confidence and neighbor proximity, rather than confidence alone.The context score amplifies confidence for candidates near decoded neighbors and suppresses isolated positional noise.
  • Cluster-Based Commitment: CACD operates block-free on the full sequence, committing every position above the current threshold as a cluster and turning committed tokens into anchors.This preserves access to high-readiness anchors across structural boundaries and propagates context to neighboring positions.
  • Architecture-Aware Calibration: Architecture-aware calibration addresses heterogeneous confidence landscapes, including means of 0.63 for MMaDA, 0.47 for LaViDa, and 0.12 for LLaDA-V.More than 73% of LLaDA-V tokens fall below c_i=0.1, motivating calibration across architectures.
  • Architecture-Aware Calibration: Dynamic thresholds favor precision early and completion later, while progress-dependent EOS suppression prevents premature short outputs.The threshold schedule lowers thresholds in later phases to avoid deadlock; EOS penalties vanish once progress reaches the stated threshold.

5 Experiments

Across multimodal and text-only evaluations, CACD improves generation, reasoning, classification, and hallucination-related metrics, with larger gains as outputs become longer. Ablations and baseline comparisons support the roles of contextual scoring, dynamic scheduling, and architecture-aware calibration.

  • Long-form generation: CACD achieves the highest LLaVABench-COCO score across all three dMLLMs, improving LaViDa from 51.1 to 88.7 (+37.6).The corresponding gains are +23.6 on LLaDA-V and +13.7 on MMaDA relative to Original.
  • Hallucination reduction: CACD reduces CHAIRs on MMaDA from 24.58 to 19.76 (−4.82) and achieves LLaDA-V’s lowest CHAIRi at 3.81 with Recall of 37.30.Wavefront performs better than CACD on both MMaDA CHAIR metrics, while qualitative examples show fewer hallucinated objects with preserved visual content.
  • Reasoning and classification: CACD attains the highest MathVista accuracy on MMaDA and LLaDA-V and the best ScienceQA accuracy across all three models.MathVista improves from 23.5% to 28.0% (+4.5%) on MMaDA and from 30.6% to 33.5% (+2.9%) on LLaDA-V.
  • Baseline comparison: AdaBlock and Wavefront underperform Original on several combinations, including Wavefront on MMaDA MathVista at 23.0% versus 23.5%.The paper attributes this mismatch to text-only decoding assumptions overlooking confidence landscapes shaped by visual integration.
  • Extension to text-only dLLMs: CACD improves text-only LLaDA-8B-Instruct metrics, raising GSM8K accuracy from 61.00 to 63.00 and IFEval instruction-level strict accuracy from 64.74 to 68.59.AlpacaEval standard and length-controlled win rates also increase from 4.50 to 6.00 and from 10.95 to 11.45.
  • Length scaling and efficiency: CACD reduces average NFE to 48.00 on GSM8K, 112.73 on IFEval, and 121.75 on AlpacaEval, yielding speedups of 2.68×, 1.20×, and 1.03×.Runtime gain is modest on AlpacaEval despite reduced NFE.
  • Length scaling and efficiency: A 44.8% relative improvement at 1024 tokens (47.8 versus 33.0) shows the quality gap widens monotonically with output length.At 16 tokens, Original and CACD are close at 45.9 versus 52.2; Wavefront’s longer outputs do not explain its lower score.
  • Ablation: Removing context scoring lowers MMaDA LLaVABench-COCO from 49.0 to 46.4, while removing dynamic thresholds lowers it to 46.7 and removing EOS suppression to 48.2.The context weight β=1.0 performs consistently across the three architectures.

6 Conclusion

CACD is a training-free decoding method that improves long-form dMLLM generation through multiplicative confidence-context scoring, cluster commitment, and globally accessible anchors. Across three dMLLMs and four benchmarks, it consistently improves quality and reduces hallucination, with benefits compounding for longer outputs.

  • CACD improves long-form dMLLM generation using a multiplicative confidence-context score and block-free cluster commitment.The method promotes contextually ready tokens while keeping high-readiness anchors globally accessible.
  • Experiments across three dMLLMs and four benchmarks show consistent quality improvement and hallucination reduction with CACD.
  • CACD’s advantage compounds as outputs grow longer, addressing long-form degradation through context-aware commitment ordering.

7 Limitations

The paper identifies practical efficiency and evaluation-scope limitations, including model-dependent runtime and memory benefits and the need for per-model threshold calibration. Its experiments establish cross-architecture generalization at approximately 8B scale, but not cross-scale generalization.

  • 7 Limitations: Per-model threshold calibration is required because the optimal τ varies with each architecture’s confidence distribution.Automatic calibration remains an important future direction.
  • 7 Limitations: CACD’s runtime and memory benefits are model dependent, and effective caching mechanisms may further improve overall efficiency.
  • 7 Limitations: Experiments demonstrate cross-architecture generalization at approximately 8B scale, but not cross-scale generalization.Evaluation at larger model scales is left for future work.

C Confidence Distribution Analysis

Confidence distributions differ substantially across the evaluated dMLLMs, and block diffusion can structurally prevent high-readiness anchors from being selected. These findings motivate architecture-aware calibration and CACD’s global decoding strategy.

  • Confidence Distribution Analysis: MMaDA, LaViDa, and LLaDA-V have substantially different confidence distributions, limiting raw-threshold transferability across architectures.MMaDA’s mean confidence is 0.63, LaViDa’s is 0.47, and LLaDA-V’s is 0.12, with over 73% of its tokens below ci=0.1.
  • Confidence Distribution Analysis: LLaDA-V has a strongly left-skewed confidence distribution, with more than 73% of tokens below ci=0.1.
  • Confidence Distribution Analysis: Block diffusion restricts high-readiness anchors behind rigid sequential block boundaries, whereas CACD evaluates the full sequence globally each round.

D Hyperparameter Sensitivity

Hyperparameter studies support balancing confidence with neighborhood context, using architecture-sensitive thresholds and fixed defaults for other controls. CACD’s cluster commitment also reduces forward evaluations substantially, with gains varying across models and tasks.

  • β Sensitivity: β=1.0 provides the best context amplification across all three models, while β>1.0 yields diminishing returns.Over-weighting neighborhood context suppresses the token’s own confidence signal.
  • β × τ Interaction: The optimal β=1.0, τ=0.9 configuration achieves the highest MMaDA MathVista accuracy of 28.0%.τ is tuned independently because it operates directly on raw confidence distributions.
  • τ Sensitivity: Architecture-specific τ thresholds are required because confidence distributions differ substantially across models.
  • δ Sensitivity: δ=0.3 is the universal default because too-small values increase deadlock while too-large values admit low-confidence tokens early.
  • ρ Sensitivity: ρ=0.8 is adopted as the universal default; score remains stable across ρ while output length increases monotonically.
  • Efficiency: On LLaVABench-COCO at 512 tokens, CACD reduces NFE from 512.0 to 45.9 and provides an 11.2× measured speedup.The reduction comes from committing multiple positions within the same decoding round.
  • Efficiency: CACD requires the fewest forward evaluations on every evaluated benchmark for both LaViDa and LLaDA-V, with reductions varying across models and tasks.

E.1 Peak GPU Memory

CACD reduces forward evaluations, but its runtime and memory advantages vary by model rather than improving every efficiency measure uniformly.

  • CACD’s practical efficiency tradeoffs depend on the model.On MMaDA, CACD provides the largest speedup, while Original uses less peak memory.
  • On LaViDa, CACD requires the fewest forward evaluations and uses the least peak memory, while AdaBlock is slightly faster.
  • On LLaDA-V, CACD again requires the fewest forward evaluations and uses the least peak memory, while AdaBlock provides the largest speedup.
  • NFE reduction does not imply uniform superiority in runtime or memory usage.

F Text Only Decoding Efficiency

CACD reduces decoding cost across the text-generation benchmarks, although measured speedup varies with task and output structure. Qualitative examples also show more complete and less repetitive outputs in representative multimodal cases.

  • Decoding efficiency: CACD reduces NFE on all three text-generation benchmarks, but measured speedup varies with task and output structure.NFE is averaged over fixed 100-example subsets under the same hardware environment.
  • Qualitative examples: In the Measuring Cup and Tennis Court examples, CACD anchors visual tokens and propagates context outward to produce correct answers or descriptions.
  • Qualitative examples: For a Clock Tower example, CACD yields a well-structured response with specific readings and multiple candidate interpretations, unlike Original’s repetition and missing time readings.
  • Qualitative examples: CACD produces a complete, non-repetitive Wii Controller description with broader scene coverage than Original.
Loading 2608.22367v1…