Reliable Parallel Decoding in Masked Diffusion Language Models

1 INTRODUCTION

• Delaying answers until other tokens are committed improved accuracy from 57.0% to 62.3%, and requiring operands before dependent results raised it to 72.0%.Confidence peaks near sequence boundaries can lock in an incorrect answer before its supporting calculations are established.Source: §1 · p. 1, Figure 1 · p. 2
• Fixed-block methods do not adapt to unresolved dependencies, while adaptive confidence or entropy rules assess predictions without their supporting context.The introduction identifies using the current forward pass to decide what should wait and what can be committed together as an unmet need.Source: §1 · pp. 1–2
• Downstream predictions are less reliable when upstream information remains uncertain, but tokens resolved together within one forward pass can be committed in parallel.Predictions consistent across final layers without losing probability are more likely to be correct.Source: §1 · p. 2
• Across mathematical reasoning and code-generation benchmarks on LLaDA and Dream, RPD had the highest throughput in all eight settings, running 2.4–6.1× faster than default decoding while maintaining or improving accuracy.The method orders commitments by upstream uncertainty rather than using a fixed block schedule.Source: §1 · p. 2
• RPD selects candidates using layerwise stability and final confidence, then commits them under a cumulative entropy budget rather than a fixed block schedule.The method is training-free and defers predictions whose upstream information remains too uncertain.Source: §1 · p. 2

2 UNDERSTANDING PARALLEL COMMITMENT

• The study uses controlled masking experiments and layerwise prediction analyses to examine upstream uncertainty and reliable joint commitment.For arithmetic traces, selected upstream digits and all final-answer digits are masked, and each state is evaluated with one forward pass.Source: §2 · p. 3

2.1 UPSTREAM UNCERTAINTY AND DOWNSTREAM ERRORS

• 17.3–29.9 percentage points higher answer error rates occur for high-entropy than low-entropy upstream masking patterns.Paired 95% confidence intervals were all above zero.Source: §2.1 · p. 3

2.2 LAYERWISE STABILITY AND PREDICTION RELIABILITY

• Predictions persisting across more final layers generally have lower error rates, even within the same final-confidence range.Persistence counts consecutive final layers predicting the same token.Source: §2.2 · p. 4
• For persistence Ki = 5–8, error rates rise from 4.6% to 23.7% as confidence drop increases.Longer persistence corresponds to fewer errors, while larger confidence drops correspond to more errors.Source: §2.2 · p. 4

3.1 CANDIDATE SELECTION WITH LAYERWISE STABILITY

• As confidence drop rises, error rate goes from 14.2% to 46.4% for Ki = 2, versus 1.8% to 6.9% for Ki ≥9.Under a fixed stability threshold, longer persistence tolerates a larger confidence drop up to Kmax, while w weights the confidence-drop penalty.Source: §3.1 · p. 5
• Stability screening matters most at intermediate confidence: very high-confidence predictions are already reliable, while low-confidence predictions can persist across layers yet be wrong.Source: §3.1 · p. 5
• High-confidence predictions bypass stability screening, while lower-confidence candidates need sufficient confidence and stability; all candidates remain subject to the entropy budget.A, B, and C pass the stability test, whereas D changes across layers and E is excluded for a large confidence drop despite persistence.Source: §3.1 · p. 5

3.2 PARALLEL COMMITMENT WITH CUMULATIVE ENTROPY

• For each candidate, RPD bounds cumulative entropy over preceding masked positions that remain masked; earlier candidates selected for joint commitment are excluded because they provide context.The sum includes preceding non-candidates and rejected candidates.Source: §3.2 · p. 5
• If no candidate meets the budget, RPD commits the highest-confidence prediction among the first W masked positions, guaranteeing progress at each iteration.In the illustrated case, A and B commit together while C waits because its preceding entropy exceeds β.Source: §3.2 · p. 5
• Full RPD uses the whole canvas and cumulative entropy to order commitments; RPD-block uses fixed 32-token blocks without that budget.RPD-block isolates layerwise stability and offers a faster operating point, while full RPD further delays candidates with uncertain upstream context.Source: §3.2 · p. 5

4.1 EXPERIMENTAL SETUP

• The evaluation uses LLaDA-8B-Instruct and Dream-7B-Instruct on GSM8K, MATH-500, HumanEval, and MBPP, with shared prompts and evaluators.Accuracy and pass@1 measure quality; NFE and tokens per second measure model evaluation cost and practical decoding speed.Source: §4.1 · p. 6

4.2 MAIN RESULTS

• RPD variants achieve the highest throughput in all eight model–task settings, with 2.4–6.1× speedups over default samplers while ranking first or second in accuracy.Relative to Fast-dLLM, RPD improves accuracy in seven settings and remains within 0.2 points in the remaining setting.Source: §4.2 · pp. 6–7
• Full RPD improves accuracy over RPD-block in all eight settings at a modest throughput cost, favoring adaptive upstream-uncertainty ordering over fixed blocks.Full RPD removes block structure and uses a cumulative entropy budget to order commitment.Source: §4.2 · p. 7

4.3 QUALITY–EFFICIENCY TRADE-OFFS

• On LLaDA, RPD lies above Fast-dLLM’s trade-off curve on all four benchmarks and on or above EB-Sampler’s, with the clearest margin on MBPP.On Dream, RPD lies at the high-accuracy end of the frontier, matching the most conservative baseline configurations.Source: §4.3 · pp. 7–8, Figure 4 · p. 7

4.4 ABLATION STUDIES

• Cumulative entropy matches or exceeds block-wise accuracy in all four settings, with the largest gain, +2.6 points, on LLaDA GSM8K.Without ordering constraints, full-canvas confidence decoding lowers accuracy from 76.35 to 54.59 on LLaDA GSM8K and from 81.73 to 35.56 on Dream GSM8K.Source: §4.4 · p. 8, Table 2 · p. 8
• Combining prediction persistence and confidence drop retains cumulative-entropy accuracy while reducing NFE by 4–11% and matching or increasing throughput in all four settings.Prediction persistence alone reduces NFE by 26–42% but can sharply lower accuracy; confidence drop removes most of those unreliable predictions.Source: §4.4 · p. 8, Table 2 · p. 8

4.5 DECODING BEHAVIOR AND OVERHEAD

• RPD commits positions in an approximately left-to-right order on the full canvas, unlike Full-canvas’s X-shaped pattern that often commits answers before supporting reasoning.The cumulative entropy budget recovers a dependency-respecting order without a fixed block schedule.Source: Figure 5 · p. 9, Figure 5a
• Layerwise projections and scoring add little runtime overhead: RPD runs at 1.03× Fast-dLLM runtime, while the backbone forward pass dominates.LoPA, EB-Sampler, and DAPD require 1.45×, 1.24×, and 1.16× Fast-dLLM runtime, respectively.Source: §4.5 · p. 8, Figure 5 · p. 9

Parallel decoding in masked diffusion language models

• Fast-dLLM (Wu et al., 2026) and EB-Sampler use confidence thresholds or entropy budgets, while LoPA (Xu et al., 2025) and Ripple-Pivot Search (Ye et al., 2026) search over decoding decisions.RPD instead assesses tokens using upstream uncertainty and layerwise predictions from the current forward pass, without exploring alternative continuations.Source: §5 · p. 9

Dependencies and commitment order

• DAPD (Kim et al., 2026) avoids jointly updating attention-coupled positions, while DOS orders positions by dependence on observed context.RPD distinguishes dependencies needing further decoding steps from those resolvable within one forward pass.Source: §5 · p. 9

Layerwise predictions and stability

• Unlike tuned lens (Belrose et al., 2023) and DoLa, RPD uses within-pass layerwise prediction consistency without trained probes or layer-contrast modification.Stability-Weighted Decoding (Wu & Huang, 2026) scores distributional changes between consecutive denoising steps.Source: §5 · p. 9

6 CONCLUSION

• RPD orders commitments with a cumulative entropy budget and layerwise stability, achieving the highest evaluated throughput while maintaining or improving accuracy without fixed blocks.The diagnostics find downstream predictions less reliable when upstream information remains uncertain, while predictions stable across layers are more likely to be correct.Source: §6 · p. 9

A.1 PROMPTS

• Each benchmark input is the raw field in a fixed, task-specific zero-shot prompt, with the model’s native chat template and no added demonstrations or assistant prefill.Generation is capped at 256 tokens, and the same prompts are shared by all decoding methods.Source: §A.1 · p. 12
• LLaDA receives no system turn, while Dream’s tokenizer template prepends “You are a helpful assistant.”Figures 6–9 provide the four benchmark instructions verbatim, and Figures 10–11 show the chat wrapping.Source: §A.1 · p. 12, Figure 6 · p. 12, Figure 7 · p. 12

• 2,000 GSM8K-Aug training examples were sampled deterministically, with verified arithmetic traces and filters ensuring eligible upstream digits and nontrivial final answers.Examples required at least 48 eligible upstream digit positions; answers appearing in the question or specified equations were excluded.Source: §A.2 · p. 12
• Each question generated 64 states from 16 masking patterns at four mask counts, with identical character-level digit masks applied to both models.The mask counts were 4, 8, 16, and 32; all target-answer digits and selected upstream digits were masked together.Source: §A.2 · p. 13
• Answer correctness required an exact decoded numerical match, while layerwise correctness used the final-layer argmax against the gold token.Diagnostic probabilities were measured before temperature scaling, sampling noise, or top-p truncation; each model had 531,264 target-token observations for layerwise analysis.Source: §A.2 · p. 14
• The layerwise analysis projected intermediate hidden states through the final normalization and language-model head, then measured the final prediction’s consecutive agreement suffix and probability drop.It analyzed layers 17–32 for LLaDA and 15–28 for Dream; earlier matches separated by a disagreement were excluded.Source: §A.2 · p. 14

• High-entropy configurations had higher error rates than low-entropy configurations for both models at every mask count.Rates were averaged over question-level groups formed from the five lowest- and five highest-entropy patterns among retained mixed-outcome questions.Source: §A.3.1 · pp. 14–15
• 28.5, 24.2, 24.8, and 28.6 percentage points for LLaDA and 17.3, 17.9, 18.3, and 29.9 for Dream were the paired high-minus-low error differences at 4, 8, 16, and 32 masks.All paired 95% confidence intervals were above zero.Source: §A.3.1 · p. 15, Table 3 · p. 15

A.3.2 LAYERWISE AGREEMENT AND PROBABILITY DROP

• Within Dream’s [0.6, 0.7) confidence band, error fell from 26.5% at K_i ≥2 to 4.6% at K_i ≥10; tolerable confidence drop also depended on persistence.The combined score therefore caps persistence and penalizes confidence drop rather than thresholding either quantity alone.Source: §A.3.2 · p. 15

A.5 DETAILED EXPERIMENTAL CONFIGURATION

• Zero-shot prompts used each model’s native chat template; GSM8K and MATH-500 requested derivations, while MBPP and HumanEval requested Python implementations.Answer equivalence or execution tests scored outputs; throughput timing counted forward calls and excluded model loading, tokenization, and scoring.Source: §A.5 · p. 16

A.5.1 MATCHED BASELINE CONFIGURATION.

• Accelerated baselines were matched at 32-token blocks and a 256-token budget, with Fast-dLLM, EB-Sampler, LoPA, and DAPD using their specified selection rules.Unless specified for Dream’s native sampler, matched baselines used greedy sampling without top-p truncation.Source: §A.5.1 · pp. 16–17

A.4 COMMITMENT ORDER UNDER MODEL-SPECIFIC DECODING

• With LLaDA, 32-token blocks produce a left-to-right diagonal, while full-canvas greedy selection leaves scattered late commits among earlier positions.The block schedule, rather than the confidence rule, enforces the order.Source: §A.4 · pp. 15–16, Figure 12 · p. 17

A.6 IMPLEMENTATION DETAILS OF RPD

• RPD admits predictions at confidence ≥0.9 directly, tests predictions from 0.6 to 0.9 for stability, and excludes predictions below 0.6.Only the stability threshold and confidence-drop weight are model-specific; both were frozen across benchmarks, and the stability score caps persistence at Kmax = 6 while penalizing confidence drop.Source: §A.6 · pp. 17–18

A.4 COMMITMENT ORDER UNDER MODEL-SPECIFIC DECODING

• Dream’s default temperature-0.1, top-p-0.9 sampler yields approximately left-to-right commits, but zero-temperature lowest-entropy selection commits terminal positions first and fills right to left.The ordering score must use the same transformed distribution from which the token is drawn.Source: §A.4 · p. 16, Figure 13 · p. 18

A.6 IMPLEMENTATION DETAILS OF RPD

• Roughly 8–10% of commits use the stability route; its admitted-token statistics are selection-induced, and layer projections reuse current-pass hidden states without extra forward calls.The stability rule imposes a confidence-drop ceiling of about 0.167 for LLaDA and 0.175 for Dream.Source: §A.6 · pp. 18–19

A.7 TRAJECTORIES VISUALIZATION

• 95 forward passes gave RPD the same broad left-to-right progression in fewer steps than Fast-dLLM’s 114 or one-token decoding’s 256 on the illustrated GSM8K document.The figure plots the commit step for each output position, with lower steps indicating earlier commitment.Source: Figure 16 · p. 20, §A.7 · p. 20