PaperlayerView exact revision on arXiv

Source-linked AI summary

Parallelism and Generation Order in Masked Diffusion Language Models: Limits Today, Potential Tomorrow

Yangyang Zhong, Yanmei Gu, Zhengqing Zang, Xiaomeng Li, Yuqi Ding, Xibei Jia, Yuting Shen, Zhenzhong Lan, Liwang Zhu, Weiping Liu, Junlin Zhou, Haisheng Liu, Zhong Xin Yu, Pengxin Luo, Donglian Qi, Yunfeng Yan, Junbo Zhao

arXiv:2601.15593v2cs.CLcs.AIcs.LG

TL;DR

Current MDLMs are promising for parallel and arbitrary-order generation, but their realized capabilities and accuracy remain unclear. This paper evaluates eight models across 58 benchmarks, measures parallelism and generation order, and develops a theoretical Generate-then-Edit perspective; it finds persistent accuracy limitations alongside adaptive decoding and non-linear-order potential.

  • Problem

    The extent to which current MDLMs realize parallelism and arbitrary-order decoding, and how these behaviors relate to quality, remains insufficiently characterized.

  • Method

    The paper evaluates eight MDLMs on 58 benchmarks and analyzes decoding with Average Finalization Parallelism, Kendall’s τ, theoretical factorization analysis, and Generate-then-Edit design insights.

  • Results

    MDLMs trail AR models because parallel factorization weakens inter-token dependencies, while parallelism and generation order adapt to task domain, reasoning stage, and correctness.

  • Takeaways & Limitations

    MDLMs show potential for non-linear causal tasks through easy-to-hard ordering and for accuracy recovery through fast sampling followed by parallel refinement.

  • Takeaways & Limitations

    Controlled scores are not leaderboard-equivalent, and hardware-specific optimization can affect observed practical behavior.

Abstract

from arXiv · show

Masked Diffusion Language Models (MDLMs) promise parallel token generation and arbitrary-order decoding, yet it remains unclear to what extent current models truly realize these capabilities. We characterize MDLM behavior along two dimensions -- parallelism strength and generation order -- using Average Finalization Parallelism (AFP) and Kendall's tau. We evaluate eight mainstream MDLMs (up to 100B parameters) on 58 benchmarks spanning knowledge, reasoning, and programming. The results show that MDLMs still lag behind comparably sized autoregressive models, mainly because parallel probabilistic modeling weakens inter-token dependencies. Meanwhile, MDLMs exhibit adaptive decoding behavior: their parallelism and generation order vary significantly with the task domain, the stage of reasoning, and whether the output is correct. On tasks that require "backward information" (e.g., Sudoku), MDLMs adopt a solution order that tends to fill easier Sudoku blanks first, highlighting their advantages. Finally, we provide theoretical motivation and design insights supporting a Generate-then-Edit paradigm, which mitigates dependency loss while retaining the efficiency of parallel decoding.

1 Introduction

MDLMs target faster, more flexible decoding than autoregressive models, but their practical capabilities and decoding behavior remain insufficiently characterized. This paper addresses the gap through large-scale evaluation and mechanism-level analysis of parallelism and generation order.

  • Motivation: MDLMs enable parallel token prediction and potentially non-monotonic generation orders, while practical systems commonly decode autoregressively across blocks.This combines within-block diffusion with block-level autoregressive execution for efficiency and KV-cache reuse.
  • Research Gap: Existing studies provide limited, strategy-focused evidence and often conflate generation order with parallelism intensity.The paper identifies systematic decoding-trajectory characterization as missing.
  • Approach: The study evaluates recent MDLMs and strong AR baselines on 58 benchmarks spanning Knowledge, Math, Reasoning, and Coding.It also analyzes decoding behavior using parallelism and order metrics.
  • Findings: The paper attributes a consistent accuracy gap to conditional-independence approximations in parallel decoding and reports adaptive behavior across structure-heavy spans and semantic pivots.Correct predictions show higher parallelism, according to the contribution summary.

2 Related Work

Prior work develops discrete-diffusion models, adapts autoregressive models to diffusion, and studies decoding strategies. However, existing evaluations remain limited in scale or confound decoding order with parallelism.

  • Model Development: Discrete-diffusion research includes native diffusion training and adaptations of pretrained autoregressive models.Examples include LLaDA lines, Dream7B, LLaDA 2.0, SDAR, Trado, DiRL, and OpenPangu-Diffusion.
  • Inference Analysis: Prior inference studies link quality drops to conditional independence, propose unmasking strategies, and evaluate decoding order with setup-sensitive metrics.The cited work often relies on limited leaderboards or hardware-dependent throughput.
  • Open Gap: Earlier research often uses small-scale datasets, while decoding-order analyses may conflate order with the number of tokens generated per step.These limitations motivate a unified, large-scale evaluation.

3 Preliminaries

AR inference predicts tokens strictly left-to-right, whereas masked-diffusion inference updates selected masked positions in parallel. Practical MDLMs combine within-block diffusion with autoregressive block sequencing.

  • Autoregressive Modeling: AR language models factorize sequences left-to-right, conditioning each token on the full preceding prefix.The factorization defines strictly sequential inference.
  • Masked-diffusion Inference: Masked-diffusion inference selects an update set of masked positions and predicts those tokens in one forward pass.The update mechanism is the basis for within-step parallelism.
  • Evaluation Context: Figure 1 compares AR and DLM performance across six evaluation dimensions, while Figure 2 compares models at matched parameter scales.The supplied captions identify the comparison scopes but do not report detailed values.
  • Masked-diffusion Inference: Within-step parallelism assumes conditional independence among simultaneously updated tokens, preventing them from conditioning on one another in that step.This mean-field approximation is explicit in the inference formulation.
  • Block-wise Framework: Block-wise MDLMs partition sequences into contiguous blocks, apply masked diffusion within each block, and condition on previously generated blocks.This design supports variable-length generation and efficient KV caching.

4 Reality Check: MDLMs Still Trail AR LMs at Scale

A unified evaluation of MDLMs across diverse benchmarks finds a persistent performance gap relative to AR models, with excessive parallelism identified as a principal accuracy bottleneck. The analysis links this gap to conditional-independence factorization and shows that dependency strength grows with parallel decoding granularity.

  • Experimental Protocol: Eight representative MDLMs are evaluated against frontier AR models on 58 benchmarks spanning six capability dimensions.The study uses a unified platform and reports mean accuracy across test instances.
  • Empirical Results: A performance gap persists between current MDLMs and AR counterparts across aggregated capability dimensions.The comparison is conducted under a unified inference pipeline and controlled evaluation protocol.
  • Empirical Trade-off: Smaller block sizes yield superior performance, and near-AR MDLM performance typically requires restricted block sizes.OpenPangu-7B-Diffusion uses block size 4 for peak precision in the reported comparison.
  • Parallel Factorization Limit: Excessive decoding parallelism compromises predictive accuracy by forcing stronger factorization of inter-token dependencies.The theoretical account identifies conditional independence as an intrinsic source of approximation error.
  • Decoding Dynamics: Figure 4 compares inter-chunk parallelism and sequential ordering across task domains, with additional Kendall’s τ and parallel-AFP groupings in the appendices.The caption directs readers to Appendix A.3.1 and A.3.2 for those groupings.
  • Parallel Factorization Limit: As block size B increases, inter-token dependency complexity and the CTC-induced error bound increase monotonically.The analysis presents this as an architectural limitation rather than merely insufficient model capacity.

5 Decoding Dynamics of MDLMs: Disentangling Parallelism and Generation Order

MDLM decoding exhibits domain- and stage-dependent variation in parallelism and generation order. The analyses show structured parallel filling, mostly sequential reasoning, and localized departures from left-to-right order at semantic or structural transitions.

  • Metrics for Decoding Dynamics: AFP measures average finalized tokens per effective decoding step, while Kendall’s τ measures alignment between token surface order and finalization order.AFP ranges from approximately 1 for autoregressive decoding to n for fully parallel decoding; τ<1 indicates non-monotonic generation.
  • Intra-block Patterns: Logic-intensive Math, Reasoning, and Code tasks generally follow near-sequential generation, with τ>0.8 in non-repetitive samples.The reported pattern is attributed to sequential dependency in these domains.
  • Inter-block Patterns: Correct Knowledge and Math samples become increasingly parallel as reasoning progresses, while coding follows an arch-shaped τ trajectory.Across domains, the final block shows a τ drop associated with generating EOS before backfilling.
  • Semantic Parallel Composition: Parallel-generated sequences are dominated by formulaic tokens such as newlines, punctuation, delimiters, and transitional markers with minimal semantic information gain.The paper connects these patterns to near-zero conditional total correlation when tokens are determined mainly by global context.
  • Semantic Generation Order: MDLMs generate structural backbones before fine-grained modifiers, indicating a non-linear order that prioritizes globally certain anchors.POS-based analysis on Knowledge outputs supports a structure-first, detail-second decoding pattern.
  • Parallelism Cases: Parallel intensity peaks for deterministic content and increases as Knowledge and Math reasoning converges, while local τ minima align with semantic or structural transitions.These transitions include shifts from chain-of-thought to final answers, paragraph breaks, and code-to-comment boundaries.

6 Discussion: Unlocking the Potential of MDLMs

The discussion identifies parallelism, non-sequential generation order, and iterative editing as MDLM capabilities with unrealized potential. It argues that generate-then-edit can address dependency loss, while current inference and masking strategies remain limiting factors.

  • Parallelism Potential: Acceleration: MDLMs parallelize structured content, but current speedups are limited by inference frameworks and require diffusion-specific optimization.The paper identifies structured and long-form generation as important targets for future acceleration.
  • Sequential Potential: Non-Sequential Dependency: Sudoku, molecule design, and protein design experiments show MDLMs using any-order inference to fill high-certainty anchor points before resolving global uncertainty.The reported strategy is described as easy-to-hard and leverages bidirectional attention.
  • Sequential Potential: Non-Sequential Dependency: Uniform masking does not reflect internal causal hierarchies, leaving a future challenge to introduce heuristic biases that model reverse causality.The paper links this issue to a larger search space and numerous invalid training paths.
  • Editing Potential: A Theoretical Path to Bridging the Accuracy Gap: Generate-then-edit is proposed to recover accuracy lost under parallel conditional independence while retaining parallel refinement.The paradigm combines exploratory generation with batch editing and is theoretically described as outperforming single-pass inference within equivalent time.

7 Conclusion

The paper presents a large-scale empirical and theoretical study of MDLM decoding dynamics. It reports limits from parallel factorization alongside potential for non-linear causal modeling and iterative editing.

  • Conclusion: The study evaluates state-of-the-art MDLMs across 58 tasks and analyzes a 100B-parameter LLaDA model to characterize parallel and arbitrary-order generation.The authors describe the work as the first systematic and large-scale evaluation of state-of-the-art MDLMs.
  • Conclusion: The findings expose limits of current parallel factorization while highlighting potential for non-linear causal modeling and efficient iterative editing.The conclusion frames these results as a blueprint for developing logically flexible and computationally efficient non-autoregressive models.

Limitations

The evaluation uses a unified local pipeline for controlled relative comparisons, but its scores are not leaderboard-equivalent and may reflect hardware-specific effects.

  • Limitations: Controlled scores should be interpreted as relative comparisons because prompts, decoding constraints, and answer matching can differ from original reports.Hardware-specific optimization can also affect practical behavior; OpenPangu may behave differently on NVIDIA hardware without Ascend-specific kernels.

We thank Ant Group for providing the computing

The study evaluates diffusion and autoregressive models across broad benchmark suites and analyzes their decoding behavior, task performance, and constraint-solving capabilities. Results describe structured generation patterns and show strong diffusion performance on several globally constrained tasks.

  • Model Selection: Eight state-of-the-art MDLMs, including LLaDA, Trado, and SDAR, are compared with autoregressive baselines across model scales reaching 100B parameters.The models include both block-wise and global diffusion decoding paradigms, with inference block sizes documented for comparison.
  • Evaluation Scope: The evaluation spans coding, mathematics, knowledge, language understanding, reasoning, and agentic capabilities across 58 benchmarks.The benchmark suite includes tasks ranging from program synthesis and mathematical problem solving to factual recall and advanced reasoning.
  • Decoding Order: Part-of-speech decoding follows a staged pattern: simple tokens and structural cues appear early, core sentence content emerges intermediately, and descriptive modifiers arrive later.Early tags include numbers, names, symbols, punctuation, and conjunctions; nouns, verbs, pronouns, and auxiliaries dominate intermediate steps.
  • Decoding Order: The model appears to build basic structure before details, rather than simply following tag frequency.Highly frequent tags such as NOUN and PUNCT are not necessarily earliest, while some less common tags appear at the beginning.
  • Constraint Solving: On Sudoku, fine-tuned Dream-7B reached 80 after 10 epochs, surpassing zero-shot 100B LLaDA at 78.With 50 training examples, Dream-7B reached 65 at epoch 5, while Qwen3-8B reached 55 only after 50 epochs.
  • Constraint Solving: Diffusion models are described as iteratively refining an entire Sudoku grid, whereas autoregressive models solve sequentially and are more susceptible to error propagation.The reported comparison attributes the difference to diffusion models’ compatibility with bidirectional constraints and AR models’ causal masking.

A.6.2 Cross-math puzzle

The cross-math puzzle evaluates models on tightly coupled arithmetic constraints requiring simultaneous validity across rows and columns. The supplied results and editing design compare diffusion and autoregressive approaches under small-data training and sequence refinement settings.

  • Task Structure: MATHMATRIX puzzles contain interlocking arithmetic equations whose row and column constraints must all be satisfied simultaneously.Changing one digit can invalidate both row-wise and column-wise expressions, creating high coupling.
  • Experimental Setup: The dataset contains 150 generated puzzles, split into 50 training samples and 100 test samples for few-shot learning and generalization.The same 50-sample training and 100-sample test protocol is used for the cross-math evaluation.
  • Experimental Setup: Diffusion and autoregressive architectures are evaluated in both zero-shot and fine-tuned settings using standardized hyperparameters and official training frameworks.Zero-shot baselines include Qwen3-next-80B and LLaDA-mini-16B.
  • Task Structure: The task prompt requires filling missing numbers so that every equation in both rows and columns is valid.The puzzle is represented as a 5×5 grid with alternating number and operator rows or columns.
  • Editing Results: Preliminary LLaDA2.0-mini-16B experiments report potential gains in generation quality and inference throughput from the editing mechanism.The broader evaluation reports Accuracy, AFP, and TPS, with AFP serving as a hardware-agnostic speed metric.
  • Editing Mechanism: The editing paradigm trains models to predict subsequent masked tokens while refining previously decoded tokens toward the ground truth.Its two-stage objective combines Mask-to-Token prediction, Token-to-Token editing loss, and teacher-forcing refinement.
  • Inference Strategies: The proposed inference modes either edit newly predicted and previously generated tokens in one pass or use a second pass for sequence refinement.Forward=1 performs parallel editing, while Forward=2 separates new-token decoding from whole-sequence refinement.

A.7.4 Experimental Analysis

The editing-enhanced MDLM operates in distinct quality, balance, and speed regimes, while theory explains how iterative thresholded editing can restore dependencies lost by one-step parallel factorization.

  • Experimental Analysis: 73.27 average score in Quality-Priority Mode significantly outperforms the baseline, but multi-pass refinement lowers AFP and inference speed.The setting uses m2t = 0.8, t2t = 0.3, and Forward=2.
  • Experimental Analysis: 72.60 accuracy in Balanced Mode exceeds the baseline while preserving near-identical speed, with AFP ≈3.20.This regime uses m2t = 0.5, t2t = 0.3, and Forward=1.
  • Experimental Analysis: AFP 6.03 vs. 3.34 and TPS 1302.0 vs. 589.8 on CruO show nearly doubled throughput in Speed-Priority Mode.This regime relaxes thresholds to m2t = 0.5 and t2t = 0.
  • Theoretical Analysis: A single parallel step discards conditional dependency information equal to conditional total correlation under the factorized approximation.This gap persists even when per-token predictors match the correct marginals.
  • Theoretical Analysis: Under the stated weak-dependence assumption, thresholded editing has a unique stationary distribution and converges geometrically.The contraction result is formalized by Theorem A.2.
  • Theoretical Analysis: Iterative thresholded editing can express cross-token dependencies across iterations because updates condition on the full previous sequence and depend on the state.The resulting stationary distribution can be non-factorized, unlike a single parallel factorized step.

S is sampled from Q

The theory connects iterative editing to runtime through a trade-off: higher parallelism reduces per-stage cost but may require more rounds because it can worsen dependence and mixing.

  • Runtime Analysis: Effective parallelism is L/m, where m is the number of sequential decoding stages; autoregressive decoding has m = L, while one-step parallel decoding has m = 1.For partially parallel decoding, m equals the number of blocks B.
  • Runtime Analysis: The runtime model assumes per-stage cost Tstep(m) does not increase as parallelism increases.Equivalently, Tstep(m) is nonincreasing in m.
  • Runtime Analysis: Under contraction α(m) < 1, Theorem A.4 provides a runtime condition for achieving total-variation error δ with editing.The condition compares editing runtime against a sequential baseline with m0 stages and no editing.
  • Runtime Analysis: Higher parallelism reduces Tstep(m) but can increase the number of editing rounds by worsening the contraction coefficient α(m).Editing matches or exceeds baseline speed when throughput gains dominate the deterioration in mixing.
Loading 2601.15593v2…