PaperlayerView exact revision on arXiv

Source-linked AI summary

Diffusion Language Models Change Speed Across Tasks and Languages

Mohsen Fayyaz, Ali Modarressi, Daniel Israel, Lucas Bandarkar, Nanyun Peng

arXiv:2610.25835v1cs.CLcs.AIcs.LG

TL;DR

The paper examines why diffusion language models with equal-length outputs can require different decoding effort. It measures forward passes per generated token across tasks and models, finding that effort varies with task and output properties and is associated with correctness. These patterns also inform a decoding-speed-weighted voting strategy that improves math-answer accuracy without additional overhead.

  • Problem

    Diffusion models can generate multiple tokens per forward pass, but decoding effort varies for equal-length outputs, and its behavior across tasks and contexts is not well understood.

    Source: §1 · p. 1

  • Method

    The authors measure forward passes per generated token across two adaptive diffusion models and tasks, and examine output novelty, word rarity, multilingual generation, and answer correctness.

    Source: §1 · p. 1, §2 · p. 2, §8 · p. 8

  • Results

    Decoding effort is lower for copying or light editing than for open-ended writing, and wrong math answers take more passes per token than correct ones; speed-weighted voting improves five-answer accuracy by 2.3 points.

    Source: §1 · p. 2, §8 · p. 8

  • Takeaways & Limitations

    Decoding effort can guide inference strategies: weighting candidate math answers by decoding speed improves accuracy over five-answer majority voting without additional overhead.

    Source: §1 · p. 2

    The study covers two adaptive diffusion models; the authors note that findings can be extended as more such models become available.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Diffusion language models can write several tokens in one forward pass, and they decide during generation how many passes to spend. Two outputs of the same token length can therefore vary a lot in cost. To understand the underlying factors of this cost variation, we analyze two diffusion models across 10 tasks. We find that the number of forward passes per token depends heavily on the task. Text copied or lightly edited from the input is cheap, while open-ended writing is costly. In DiffusionGemma, generating a token for story writing requires more than three times as many forward passes as for grammar correction. Two output properties explain much of this gap: how much of it is new relative to the input, and how rare its words are within the task. The same holds within a single response when thinking is involved. The final answer is cheap when it repeats the model's reasoning trace and costly when it adds something new. In multilingual generation, we find that highly tokenized languages also require more passes per token, adding to the cost they already pay with more tokens. Finally, cost tracks correctness: on the same math problem, wrong answers take more passes per token than correct ones, and weighting candidate answers by decoding speed beats majority voting by up to 4.2 points at no extra cost. Taken together, our results show that diffusion's computational advantage depends on how much of the output is already determined, while its decoding effort offers a practical signal for making inference more efficient and reliable.

1 Introduction

Diffusion decoding effort varies substantially with the task: input-constrained or recurring-pattern text is cheaper than open-ended writing. The paper examines task, within-response, multilingual, and correctness-related variation in that effort.

1 Introduction

  • Diffusion models use fewer forward passes per token for input-constrained or recurring-pattern text than for open-ended tasks such as creative writing.This makes decoding effort an additional dimension of computational efficiency beyond output length.Source: §1 · p. 1
  • Incorrect mathematical answers require more forward passes per token than correct answers, even after controlling for problem difficulty and generation length.Final answers often take fewer decoding steps when they reproduce information already established in the reasoning trace.Source: §1 · p. 2
  • Languages requiring more tokens also tend to require more decoding passes per token, compounding their tokenization disadvantage.The paper describes this combined burden as a double multilingual tax on inference efficiency.Source: §1 · p. 2
  • Weighting candidate math answers by final-answer decoding speed improves accuracy over plain majority voting by 2.3 points with five answers, at zero additional overhead.With two answers, the reported improvement is 4.2 points.Source: §1 · p. 2
  • The findings characterize when diffusion decoding is efficient and motivate inference strategies guided by task structure and denoising signals.The study frames decoding effort as useful for understanding how computational effort is distributed across tasks and within generations.Source: §1 · p. 2

2 Setup

The setup measures diffusion decoding cost as forward passes per 100 generated tokens and compares tasks with different degrees of constraint from the input or earlier output. It also includes rewriting tasks beginning in Section 3.1.

2 Setup

  • The two models use adaptive step counts, committing tokens only when they meet a target confidence threshold.DiffusionGemma generates blocks of 256 tokens, while DreamReasoner generates blocks of 32.Source: §2 · p. 2
  • Cost is the number of forward passes per 100 meaningful generated tokens, averaged across task prompts and normalized by generated length rather than block capacity.The count excludes trailing end-of-sequence and padding tokens; this normalization accounts for early stopping and supports comparison across block sizes.Source: §2 · p. 2
  • Seven tasks are grouped by how much they constrain the answer: reproducing, chained generation, or open-ended generation.The examples include grammar correction and extractive summarization; code editing, code generation, and math; and user queries and story generation.Source: §2 · pp. 2–3
  • Three rewriting tasks are added from Section 3.1, where answers draw on input information but express it in new words.They are German-to-English translation, clinical-note generation, and abstractive chapter summarization.Source: §2 · p. 3

3 Task Dependent Cost

Decoding cost varies sharply by task: copying or lightly editing input is cheaper than open-ended writing. Across tasks, cost rises with new information, combining output novelty and task-level word rarity, though this measure misses some cross-language copying and discourse-level unpredictability.

3 Task Dependent Cost

  • Story generation costs 11.43 forward passes per token versus 3.72 for grammar correction in DiffusionGemma; DreamReasoner shows the same task-cost ranking.Across reproducing, chained, and open-ended tasks, DiffusionGemma averages 4.0, 5.8, and 9.8 forward passes per 100 tokens; DreamReasoner averages 22.1, 56.2, and 87.0.Source: §3 · p. 3

3.1 What Makes an Output Cheap to Decode

  • Novelty measures segments per output word, ranging from near zero for long copied fragments to one for text generated entirely from scratch.For DiffusionGemma, novelty is 2% for extractive summarization and 99% for story generation; novelty alone correlates with cost at +0.72 for DiffusionGemma and +0.79 for DreamReasoner.Source: §3.1 · p. 3
  • Novelty and surprisal capture distinct properties: math and code have similarly high novelty to abstractive summaries but lower word rarity and decoding cost.For DiffusionGemma, novelty is 83%, 85%, and 83% for math, code, and abstractive summarization; surprisal is 7.0 and 7.9 bits versus 9.2, while cost is 6.2 and 6.2 versus 9.5 forward passes per 100 tokens.Source: §3.1 · pp. 3–4
  • New information (NI) tracks decoding cost better than novelty or surprisal alone in both models.Cost rises with NI across tasks; correlations rise from +0.72 for novelty to +0.76 for NI in DiffusionGemma and from +0.79 to +0.85 in DreamReasoner.Source: §3.1 · pp. 4–5, Figure 2 · p. 4
  • NI’s fit has limits: translation costs 6.0 versus story generation’s 11.4 despite similar NI, while word-level surprisal misses discourse-level unpredictability.The authors attribute translation’s lower cost to reuse of input content across languages that surface-copy novelty misses; story and abstractive-summary unpredictability can instead be discourse-level.Source: §3.1 · p. 5

4 Analysis: Copying from One’s Own Thinking

Final answers are cheaper when they reuse the model’s thinking, while answer cost rises on harder problems mainly for incorrect solutions. The relationship between novelty and cost also reverses for answers that add substantially new text.

4 Analysis: Copying from One’s Own Thinking

  • In one competition-math example, thinking cost 3.81 passes per token versus 2.88 for the final answer, which had 15% novelty relative to thinking.The example used 10,339 thinking tokens and a 1,241-token answer; the answer’s novelty relative to the question was 87%.Source: §4 · p. 5

4.1 Across Tasks

  • Across tasks, less novel final answers cost less relative to thinking (r = +0.89, p = 0.001), and novelty-cost correlations are positive in all 10 tasks.The within-task correlations range from +0.06 to +0.90 and are significant in seven tasks; Figure 4 plots novelty against the answer-versus-thinking cost gap.Source: §4.1 · pp. 5–6, Figure 4 · p. 5
  • For WildBench and story writing, answers are 68% and 76% novel relative to thinking and cost more than thinking (+0.60 and +2.32).At intermediate novelty, code edit and clinical notes cost about the same as thinking; the gap closes and then reverses as novelty rises.Source: §4.1 · p. 6

4.2 Across Difficulty

  • Incorrect answers require +1.20 additional forward passes per 100 tokens after controlling for contest, problem position, and token counts; matched runs still require +0.83 more passes.Correct answers remain equally cheap across difficulty levels, while wrong-answer costs rise; contest position is a coarse difficulty heuristic, not a calibrated measure across contests.Source: §4.2 · pp. 6–7

5 Multilingual Effort

In DiffusionGemma translations, languages with more tokenizer pieces per word also require more forward passes per token. This compounds the extra token cost of longer token sequences.

5 Multilingual Effort

  • The study translates the same 200 English FLORES sentences into 25 languages using DiffusionGemma with thinking off and default settings.The Figure 6 comparison shows 20 languages that separate words with spaces; results for all 25 are in Figure D.1.Source: §5 · p. 7
  • Armenian needs 24.7 forward passes per 100 tokens versus Spanish’s 15.7, in addition to using 75 versus 36 tokens per sentence.The resulting total sentence costs are 16.9 versus 5.3 forward passes, respectively.Source: §5 · p. 7
  • Across 20 space-delimited languages, tokens per word correlate with passes per token (r = +0.65, p = 0.002) and with lower chrF++ quality (r = −0.81).The authors describe the combined token-count and per-token effect as a double multilingual tax on inference efficiency.Source: §5 · p. 7

6 Decoding Cost as a Measure of Certainty

The authors weight candidate answers by their decoding speed and compare the resulting vote with plain majority voting. The weighted vote improves accuracy, with no additional decoding overhead.

6 Decoding Cost as a Measure of Certainty

  • Each answer’s vote is weighted by answer tokens per forward pass, using decoding speed already counted during generation and adding no cost.Only text generated after thinking ends is counted for the answer weight.Source: §6 · pp. 7–8
  • With five answers, speed-weighted voting raises accuracy from 71.4% to 73.7%, with a 95% bootstrap interval of [0.3, 4.5] points for the gain.Across answer counts, it beats plain majority vote by 4.2 points with two answers and 2.3 points with five.Source: §6 · p. 8
  • Weighting by the inverse number of thinking tokens performs similarly to plain majority vote, at 71.0% accuracy with five answers.This comparison indicates that the reported gain is not merely a preference for shorter reasoning.Source: §6 · p. 8

7 Related Work

Prior work studies diffusion decoding behavior and data-dependent parallelism, but the cited work does not address how diffusion models allocate forward passes. The authors distinguish their question from related analyses of speculative and adaptive decoding.

Diffusion LLM interpretability

Data-dependent parallelism

  • Speculative decoding research establishes data-dependent speed and examines task benchmarks, language fairness, and acceptance rates across domains.The cited work includes Leviathan et al. (2023), Sandler et al. (2025), and Fu et al. (2026).Source: §7 · p. 8
  • ParallelBench argues masked diffusion adaptive samplers fail to adjust well to task difficulty, but does not study newer uniform models such as DiffusionGemma.The passage contrasts that work’s scope with uniform diffusion models.Source: §7 · p. 8

8 Conclusions and Future Work

Across two models, copying and light editing require fewer passes per token than open-ended writing, while output novelty and word rarity explain part of the gap. Decoding effort also varies within responses and tracks math-answer correctness, motivating speed-guided inference strategies.

8 Conclusions and Future Work

  • In DiffusionGemma, story generation takes 3.1 times as many passes per token as grammar correction; copying and light editing are cheaper than open-ended writing.Output copying and task-relative word commonness explain part of the difference.Source: §8 · p. 8
  • Final answers tend to need fewer steps when they repeat information already established in the model’s thinking.Within tasks, greater copying from the reasoning trace is associated with cheaper final answers relative to that trace.Source: §8 · p. 8
  • Lower-resource languages pay twice, with both more tokens and more passes per token.Source: §8 · p. 8
  • Wrong math answers take more passes to decode than correct answers, and speed-weighted voting improves five-answer majority voting by 2.3 percentage points at no additional cost.Source: §8 · p. 8
  • Future work could use decoding effort for verification, additional sampling, early stopping, and adapting block sizes or stopping rules to task structure and text reuse.The authors also suggest selectively using autoregressive or diffusion decoding according to what remains to be generated.Source: §8 · p. 8

Limitations

The study covers two models because few available models meet its requirements for adaptive decoding and performance across reasoning, translation, and long-form writing.

Limitations

  • The findings are bounded by two suitable models; the authors note that they can be extended as more models with the required capabilities become available.The study requires adaptive step counts and models capable of reasoning, translation, and long-form writing.Source: Limitations · p. 9

A Models and Settings

The study uses each model’s published decoding settings and defines how passes and output tokens contribute to its cost measurements.

A Models and Settings

  • Each denoising step on a block counts as one forward pass; blocks contain 256 positions for DiffusionGemma and 32 for DreamReasoner.Source: A Models and Settings · p. 12
  • The context-reading pass is excluded, and output tokens run to the first <eos> without counting special tokens.Source: A Models and Settings · p. 12
  • Task cost is computed per output and averaged over task outputs; error bars are 95% bootstrap intervals using 10,000 item resamples.For groups of tasks, the reported value averages the task means.Source: A Models and Settings · p. 12

B Tasks and Data

The appendix documents task data and prompts, generation runs and exclusions, multilingual translations, a self-correction example, and supplementary materials. The task setup distinguishes copied or lightly edited outputs from rewriting and open-ended generation.

B Tasks and Data

  • Task data were fixed with a selection seed; corrected essays and extractive summaries largely reuse input wording, and prompts constrained summary and story lengths.The table covers all ten tasks, with the first seven in Figure 1 and three rewriting tasks added later.Source: B Tasks and Data · p. 12, Table B.1 · p. 14

B.1 Prompts

  • The grammar-correction prompt forbids changing correct text or altering content, while extractive summarization requires verbatim source sentences selected to cover key findings.Source: §B.1 · pp. 12–13
  • Other prompts request outputs such as a 700–900-word story, an English translation, and a clinical note with specified section headings.Source: §B.1 · p. 13

C Runs and Exclusions

  • Each model generated one output per item with seed 1; outputs that remained at the token limit and outputs shorter than five tokens were excluded from averages.Most long outputs were repetition loops; retaining them would lower DreamReasoner’s story cost from 95.47 to 54.7, while DiffusionGemma’s code-generation cost would change from 6.21 to 6.4.Source: C Runs and Exclusions · p. 13

D Multilingual Runs

  • DiffusionGemma translated 200 English FLORES sentences into 25 languages written in 16 scripts; all 5,000 generations finished, and reference translations supplied the tokenizer-based measurements.Fertility is not defined for five languages written without spaces; translation quality is measured against references with chrF++.Source: D Multilingual Runs · pp. 13–14

E An Example of Self-Correction

F E1 elaboration: Use of AI Assistants

  • AI assistants helped polish manuscript text, find citations and related work, and assist with code generation and plotting.Source: F E1 elaboration: Use of AI Assistants · p. 14
  • The multilingual figure plots tokens per sentence against forward passes per sentence, with dashed lines marking equal total cost and English shown as a reference.The accompanying table reports per-sentence tokens and passes and cost per 100 tokens, with languages sorted by passes per sentence.Source: Figure D.1 · p. 14, Table D.1 · p. 16

Related papers

Loading 2610.25835v1…