PaperlayerView exact revision on arXiv

Source-linked AI summary

ReFusion: A Diffusion Large Language Model with Parallel Autoregressive Decoding

Jia-Nan Li, Jian Guan, Wei Wu, Chongxuan Li

arXiv:2512.13586v2cs.CLcs.AIcs.LG

TL;DR

Autoregressive models are limited by sequential inference, while masked diffusion models face KV-cache overhead and incoherent generation from token-level independence assumptions. ReFusion combines inter-slot diffusion selection with intra-slot autoregressive infilling and reorders generated slots. Across seven benchmarks, it reports gains over prior MDMs and performance that challenges strong ARMs while retaining a speed advantage.

  • Problem

    Autoregressive models decode sequentially, while masked diffusion models face KV-cache overhead and incoherence when conditionally dependent tokens are generated in parallel.

    Source: §1 · p. 1

  • Method

    ReFusion partitions sequences into slots, selects slots through diffusion, infills within slots autoregressively, and reorders generated slots ahead of remaining masks.

    Source: §1 · pp. 2–3

  • Results

    Across seven benchmarks, ReFusion gains 34% in average performance and exceeds LLaDA and Dream by over 18× in throughput; it beats Qwen3-8B on GSM8K and MBPP by 3.68 absolute points while being 2.33× faster on average.

    Source: §1 · p. 2

  • Takeaways & Limitations

    ReFusion reports state-of-the-art MDM results and often outperforms strong ARMs while being significantly faster, with full KV-cache reuse and tractable slot-level learning.

    Source: §1 · p. 3, §6 · p. 10

    Completed slots cannot be remasked or refined, so errors within them cannot be corrected in later iterations.

    Source: E LIMITATIONS · p. 23

Abstract

from arXiv · show

Autoregressive models (ARMs) are hindered by slow sequential inference. While masked diffusion models (MDMs) offer a parallel alternative, they suffer from critical drawbacks: high computational overhead from precluding Key-Value (KV) caching, and incoherent generation arising from learning dependencies over an intractable space of token combinations. To address these limitations, we introduce \textsc{ReFusion}, a novel masked diffusion model that integrates sequence reorganization into the causal attention framework. By elevating parallel decoding from the token level to a higher slot level, \textsc{ReFusion} interleaves inter-slot diffusion-based selection with intra-slot autoregressive infilling, while reordering newly generated slots ahead of the remaining masks after each iteration. Consequently, this design simultaneously unlocks full KV cache reuse and reduces learning complexity from an intractable token combination space to a manageable slot-level permutation space. Extensive experiments on seven diverse benchmarks show that \textsc{ReFusion} not only overwhelmingly surpasses prior MDMs with a 34\% performance gain and an over 18$\times$ speedup on average, but also bridges the performance gap to strong ARMs while maintaining a 2.33$\times$ average speedup.

1 INTRODUCTION

Autoregressive models decode sequentially, while masked diffusion models permit parallel generation but face KV-cache overhead and incoherence from conditional-independence failures. ReFusion reorganizes sequences into slots, combining diffusion-based selection and autoregressive infilling; experiments report strong performance and throughput against prior MDMs and strong ARMs.

1 INTRODUCTION

  • MDMs enable parallel decoding and flexible generation orders, but bidirectional attention prevents KV caching and independent token sampling can yield incoherent combinations.The text attributes incoherence to modeling dependencies over an exponential token-combination space; for example, independently chosen tokens can produce “right once” from alternatives “at once” and “right now.”Source: §1 · pp. 1–2
  • ReFusion partitions masked sequences into slots, selects slots by diffusion, infills within them autoregressively, and moves generated slots ahead of remaining masks.The design reuses KV states for previously generated tokens and reduces learning from token combinations to slot permutations, while serializing adjacent tokens to mitigate conditional-independence violations.Source: §1 · p. 2
  • ReFusion training permutes clean slots, places them before masked slots, and combines autoregressive and denoising losses to supervise every token.Traditional MDMs learn only from masked positions, whereas this hybrid objective is described as boosting data efficiency.Source: §1 · p. 2
  • 34% average performance gain over LLaDA and Dream accompanies over 18× faster throughput, while ReFusion outperforms Qwen3-8B on GSM8K and MBPP by 3.68 absolute points and is 2.33× faster on average.These results span seven benchmarks across math, code generation, and general-purpose understanding and reasoning; Figure 1 plots pass@1 against throughput on MBPP relative to Qwen3-8B.Source: §1 · p. 2, Figure 1 · p. 2
  • Controlled experiments attribute ReFusion’s gains to its architectural and training innovations rather than initialization or data advantages.The introduction also presents inter-slot parallel decoding with intra-slot autoregressive decoding as the model’s combined design.Source: §1 · p. 3

2 RELATED WORK

Prior work addresses MDM efficiency through approximate caching, mixed attention, or causal attention, while decoding strategies select tokens by confidence or external verification. ReFusion’s slots combine flexible inter-slot generation with serial intra-slot decoding, distinguishing them from block-based methods.

2 RELATED WORK

  • Prior efficient-MDM designs approximate KV reuse, mix bidirectional and causal attention, or use causal attention alone; token-level permutation in Eso-LMs is reported to make learning intractable and hinder performance.Block methods can permit caching between blocks but retain parallel bidirectional generation within each block; D2F parallelizes succeeding blocks, with performance limited by lacking inter-block lookahead attention.Source: §2 · p. 3
  • Confidence-based decoding uses token probability, entropy, or probability margins, while external verification can add the overhead of maintaining and querying another model.Confidence heuristics rely on model scores being calibrated; external approaches include ARM prefix validation and reward-model guidance.Source: §2 · p. 3
  • Unlike block-based methods’ serial inter-block and parallel intra-block order, ReFusion uses parallel inter-slot and serial intra-slot decoding to reduce learning complexity.Slots retain global generation flexibility and full KV cache reuse; blocks sacrifice global flexibility and bidirectional intra-block attention precludes KV caching and risks incoherence.Source: §2 · pp. 3–4
  • Slots can be nested within blocks, making the slot design hierarchically compatible with block-based methods.The paper characterizes this compatibility as positioning ReFusion as a more generalizable framework.Source: §2 · p. 4

3 PRELIMINARY

The preliminaries contrast autoregressive next-token generation with masked diffusion’s mask-and-denoise training and iterative parallel inference. ARMs require sequential passes, whereas MDMs use fewer passes but have more expensive passes because they cannot use KV caching.

3 PRELIMINARY

  • ARMs factorize sequence probability through strict left-to-right dependencies and train by minimizing next-token negative log-likelihood.Inference requires T forward passes to generate T tokens, so latency scales with sequence length.Source: §3 · p. 4
  • MDMs train by masking tokens with probability t sampled uniformly from [0, 1] and learning to reconstruct the original sequence.The objective is described as a reconstruction objective over the masked sequence.Source: §3 · p. 4
  • MDM inference progressively generates tokens from a fully masked sequence, using fewer forward passes than an ARM but more expensive passes due to incompatibility with KV caching.Source: §3 · p. 4

4 METHODOLOGY

ReFusion reorganizes sequences into slots, selecting slots with diffusion while decoding tokens within each slot autoregressively. Its inference and training procedures mirror this design to support cache reuse and supervise both clean and masked tokens.

4.1 SEQUENCE REORGANIZATION

  • Reordering generated slots ahead of masks supports full KV cache reuse, while original position IDs keep attention invariant to the reorder.Within each slot, standard autoregressive KV caching is used during token-level decoding.Source: §4.1 · pp. 4–5
  • Slots group continuous, non-overlapping tokens so diffusion can select globally while autoregressive decoding captures strong local dependencies.The design is motivated by the rapid decay of dependency strength with token distance.Source: §4.1 · p. 5

4.2 INFERENCE

  • Inference selects slots using the most likely token’s probability at the first position, then drafts selected slots for autoregressive verification and prediction.Valid prefixes are retained; unverified suffixes are remasked and predicted until the selected slots are complete.Source: §4.2 · pp. 5–6

4.3 TRAINING

  • Training masks random slots, permutes clean slots ahead of them, and combines autoregressive loss on clean slots with denoising loss on masked slots.This objective supervises every token and improves data efficiency compared with traditional MDMs, where clean tokens provide context but no direct supervision.Source: §4.3 · pp. 6–7

5 EXPERIMENTS

Across seven benchmarks, REFUSION is evaluated against autoregressive and masked-diffusion baselines, with controlled comparisons, hyperparameter analysis, and a code-generation case study. It outperforms MDM baselines in performance and throughput and often challenges strong ARMs.

5.1 EXPERIMENTAL SETUP

  • The evaluation covers seven benchmarks across understanding, reasoning, math, and code, measuring accuracy or pass@1 and throughput in tokens per second.Throughput is measured on a single A100 GPU with batch size 1.Source: §5.1 · pp. 7–8

5.2 MAIN RESULTS

  • 2.33× average speedup over Qwen3-8B across all tasks; REFUSION also beats it by 3.68 absolute points on GSM8K and MBPP.Across seven benchmarks, REFUSION outperforms MDM baselines in performance and throughput; on HumanEval it reaches 78.66% pass@1, nearly 22 points above the next-best MDM.Source: §5.2 · p. 8

5.3 CONTROLLED COMPARISON

  • In controlled Qwen3-8B-initialized retraining on 120K samples, REFUSION exceeds Qwen3-8B by about 6 HumanEval points and is 1.9× faster.All models used the same initialization and their respective original objectives; the subset was drawn from a 3.7M-sample dataset.Source: Table 2 · p. 8, §5.3 · p. 9
  • Against Dream-7B-Instruct, a REFUSION variant reports 2.23% average performance gain and 11.05× speedup despite using fine-tuning without pre-training.The comparison uses Dream’s native Qwen2.5-7B backbone; REFUSION performs lower on knowledge-intensive MMLU-Pro and ARC-C tasks.Source: Table 3 · p. 8, §5.3 · p. 9

5.4 ANALYSIS OF HYPERPARAMETERS

  • The hyperparameters create performance-efficiency trade-offs, with a broad sweet spot over τslot ∈ [0.5, 1.0], τtoken ∈ [0.1, 0.9], and k ∈ {8, 32}.Higher τslot improves performance but has non-monotonic throughput effects; higher τtoken lowers throughput, while larger k can improve both performance and speed.Source: §5.4 · pp. 9–10, Figure 4 · p. 9

5.5 CASE STUDY

  • In the MBPP case study, REFUSION generates multiple slots concurrently and can construct a central for-loop before initializing a local variable.At iteration 8, it generates four slots simultaneously and accepts four tokens per slot in one pass; the for-loop appears at iteration 5, before “sum = 1” at iteration 6.Source: §5.5 · p. 10, Figure 5 · p. 10

6 CONCLUSION

REFUSION combines sequence reorganization with causal attention, pairing diffusion-based selection and autoregressive infilling. Across seven benchmarks, it establishes a new MDM state of the art and often outperforms strong ARMs while being faster.

6 CONCLUSION

  • REFUSION uses sequence reorganization with causal attention to combine diffusion-based selection and autoregressive infilling.The design enables full KV cache reuse and makes the training objective tractable by simplifying the generation space’s combinatorial complexity.Source: §6 · p. 10
  • Across seven benchmarks, REFUSION establishes a new state of the art for MDMs and often outperforms strong ARMs while being significantly faster.The paper concludes that structuring parallel generation can achieve MDM throughput potential without sacrificing generation quality.Source: §6 · p. 10
  • Future directions include scaling the model and data and using reinforcement learning to optimize planning for complex, multi-step reasoning.Source: §6 · p. 10

A METHODOLOGICAL DETAILS

ReFusion combines flexible slot-level selection with autoregressive generation within each slot, targeting the efficiency and coherence trade-offs of prior masked diffusion models. Its inference formalization selects draft slots, verifies and infills them, and repeats until no masks remain.

A.1 COMPARISON BETWEEN REFUSION AND REPRESENTATIVE MDMS

  • LLaDA's bidirectional attention prevents KV caching and its token-level masking patterns create an intractable learning objective, while BD3-LMs constrain generation order between blocks.BD3-LMs retain bidirectional attention and token-level generation within blocks, limiting intra-block cache use and global flexibility.Source: §A.1 · p. 15
  • ReFusion preserves any-order generation between slots while using causal autoregressive decoding within slots, enabling full KV cache reuse and reducing learning complexity to slot-level permutations.Generated slots are reordered ahead of remaining masks; unlike LLaDA and BD3-LMs, this design aims to combine global flexibility with cache reuse.Source: §A.1 · pp. 15–16

A.2 LOCALITY OF INTER-TOKEN DEPENDENCY

  • Dependency strength, measured by the change in prediction distribution after revealing a token, decays with token distance and more rapidly in denser contexts.The pilot study estimates this change using Jensen-Shannon divergence on GSM8K test sequences.Source: §A.2 · p. 16, Figure 6 · p. 16

A.3 INFERENCE FORMALIZATION

  • At each iteration, ReFusion drafts masked slots, scores them, and selects slots above a threshold—or the highest-scoring slot if none qualifies—for infilling.The score targets slots that are strongly constrained by context and weakly interdependent.Source: §A.3 · pp. 16–17
  • The infilling stage verifies a concatenated draft's longest high-probability prefix, accepts complete slots when possible, and otherwise refines selected slots independently.Unverified suffixes are re-masked for another select-and-infill iteration; completed slots' KV caches are concatenated, and the final output is restored to original order.Source: §A.3 · p. 17

B.1 IMPLEMENTATION DETAILS

The implementation and analyses describe ReFusion's training setup, inference settings, comparisons, ablations, trade-off frontier, and scaling behavior. Results include speed and performance comparisons, while scaling throughput rises with data size but performance is not strictly monotonic under a fixed-epoch schedule.

B.1 IMPLEMENTATION DETAILS

  • 3.7M training samples, a 4,096-token maximum sequence length, and approximately 10.68K H20 GPU-hours define the reported training setup.Training used a global batch size of 512, a learning rate of 2e-5, and randomly selected slot sizes from {4, 8, 16, 32}.Source: §B.1 · p. 17

B.2 HYPERPARAMETER SETTING

  • Inference exposes four adjustable settings: slot-selection threshold, token-acceptance threshold, slot size, and block size.The specific task settings are presented in Table 5.Source: §B.2 · p. 18, Table 5 · p. 18

C.1 COMPARISON WITH CLOSED-SOURCE AND DIVERSE ARCHITECTURES

  • ReFusion reaches 1.66× GPT-4o Mini's throughput and exceeds Nova Micro by 2.8 points on MBPP.The passage also reports results approaching those of larger, more powerful closed-source MDMs.Source: §C.1 · p. 18

C.2 ABLATION ON CERTAINTY SCORING

  • Across seven benchmarks, first-token probability, mean slot probability, and mean slot entropy produce highly comparable performance.The first-token probability remains the default because it has lower computational overhead.Source: §C.2 · pp. 18–19, Table 7 · p. 19

C.3 ABLATION ON KV CACHE REUSE

  • 1.16–1.33× faster across benchmarks, ReFusion's default KV-reuse approach maintains stable performance and slightly improves it on several benchmarks.The comparison is against a variant that recomputes KV states for full contextualization.Source: §C.3 · p. 19, Table 8 · p. 19

C.4 TRADE-OFF FRONTIER ANALYSIS

  • As tokens generated per forward pass increase, LLaDA and Dream show sharp performance declines, while ReFusion's performance curve is substantially flatter.The passage attributes the baseline decline to failure of conditional independence and says ReFusion more reliably identifies conditionally independent tokens.Source: §C.4 · pp. 19–20, Figure 7 · p. 19

C.5 SCALING WITH DATA SIZE

  • On MBPP, throughput rises from approximately 51 TPS with 120K training samples to over 81 TPS with 14M, while GSM8K accuracy peaks at 2M samples before slightly decreasing at 3.7M.The analysis trains for one epoch on varying data sizes; the authors note performance scaling is not strictly monotonic under this fixed-epoch schedule.Source: §C.5 · p. 20

C.6 ANALYSIS OF GENERATION LENGTH AND LATENCY

ReFusion's outputs are shorter than those of the Qwen3-8B ARM baseline and no longer than or comparable to other MDMs, while its measured inference latency is substantially lower across tasks. The reported comparison therefore does not attribute its quality gains to generating longer sequences.

C.6 ANALYSIS OF GENERATION LENGTH AND LATENCY

  • 124 tokens on MMLU-Pro—roughly 5× fewer than Qwen3-8B—accompanies superior performance, and ReFusion's measured latency is substantially lower across all tasks.Generated outputs are consistently shorter than Qwen3-8B's and either shorter or comparable to those of other MDMs.Source: §C.6 · p. 20

C.7 ANALYSIS OF BLOCK SIZE

ReFusion's block size affects both performance and throughput non-monotonically, reflecting trade-offs in selection horizon, modeling complexity, and computation. The analysis identifies block sizes from 32 to 128 as a robust operating range for surpassing Qwen3-8B on both performance and throughput.

C.7 ANALYSIS OF BLOCK SIZE

  • Performance and throughput vary non-monotonically with block size: larger blocks extend the selection horizon but increase modeling difficulty and latency from padded positions.More opportunities for parallelism can be outweighed by processing longer sequences containing positions not yet generated.Source: §C.7 · pp. 20–21
  • Block sizes from 32 to 128 form a reported sweet spot where ReFusion consistently surpasses Qwen3-8B in both performance and throughput.As block size increases from 32 to 512, pass@1 decreases by only approximately 0.2% relative to its peak.Source: §C.7 · p. 21

D.1 CODE GENERATION CASE STUDY

The MBPP case study presents ReFusion’s code as better structured and higher quality than existing MDMs. Its inference proceeds through iterative slot selection and autoregressive infilling, with reordering and invariant position IDs supporting the process.

D.1 CODE GENERATION CASE STUDY

  • Figure 9 compares pass@1 and throughput relative to Qwen3-8B across block sizes, highlighting the range where ReFusion surpasses it.Source: Figure 9 · p. 21
  • ReFusion’s MBPP output is described as better-structured, higher-quality code than existing MDMs, with Table 10 showing responses to the same problem as Figure 5.The account attributes this to diffusion-based global structure sketching and autoregressive local infilling.Source: §D.1 · p. 21, Table 10 · p. 22

D.2 STEP-BY-STEP VISUALIZATION OF INFERENCE

  • Inference iteratively selects high-quality masked slots by confidence and autoregressively completes the selected slots as a batch.The process is presented as a two-stage “select-and-infill” cycle.Source: §D.2 · p. 21
  • After infilling, decoded slots move ahead of remaining masks while retaining position IDs from their original sequence indices.The model uses these consistent position IDs with RoPE to perceive relative token positions despite input-buffer reordering.Source: §D.2 · pp. 22–23

E LIMITATIONS

Generated slots are immutable, so errors in completed slots cannot be corrected. The paper proposes sub-slot remasking as future work, notes its added inference complexity, and states that LLM assistance was limited to linguistic refinement.

E LIMITATIONS

  • Once a slot is generated, it cannot be remasked or refined, leaving errors in completed slots uncorrected.Source: E LIMITATIONS · p. 23
  • Sub-slot remasking could support iterative refinement, but would require more complex inference and dynamic slot sizing; an efficient fine-grained strategy remains a challenge.Source: E LIMITATIONS · p. 23

F THE USE OF LARGE LANGUAGE MODELS

Related papers

Loading 2512.13586v2…