PaperlayerView exact revision on arXiv

Source-linked AI summary

dInfer: An Efficient Inference Framework for Diffusion Language Models

Yuxin Ma, Lun Du, Lanning Wei, Kun Chen, Qian Xu, Kangyu Wang, Guofeng Feng, Guoshan Lu, Lin Liu, Xiaojing Qi, Xinyuan Zhang, Zhen Tao, Haibo Feng, Ziyun Jiang, Ying Xu, Zenan Huang, Yihong Zhuang, Haokai Xu, Jiaqi Hu, Zhenzhong Lan, Junbo Zhao, Jianguo Li, Da Zheng

arXiv:2510.08666v3cs.CLcs.AI

TL;DR

dLLM adoption is limited by high denoising costs, difficult parallel scaling, and the lack of standardized inference infrastructure. dInfer addresses this gap with a modular framework combining component-level algorithms and system optimizations, achieving high throughput while maintaining output quality. Its reported results include over 1,100 TPS on HumanEval and more than 10× speedup over Fast-dLLM under comparable performance.

  • Problem

    dLLM deployment lacks a unified, efficient inference framework and standardized evaluation despite the models’ inherent parallelism.

  • Method

    dInfer decomposes inference into model, iteration manager, decoding strategy, and KV-cache components with specialized algorithms and system optimizations.

  • Results

    dInfer exceeds 1,100 TPS on HumanEval and 800 TPS across six benchmarks, with over 10× speedup over Fast-dLLM and 2−3× over vLLM’s Qwen2.5-3B at comparable quality.

  • Takeaways & Limitations

    dInfer provides an open-source practical toolkit and standardized platform for accelerating dLLM research and development.

Abstract

from arXiv · show

Diffusion-based large language models (dLLMs) have emerged as a promising alternative to autoregressive (AR) LLMs, leveraging denoising-based generation to enable inherent parallelism. Even more and more open-sourced dLLM models emerge, yet their widespread adoption remains constrained by the lack of a standardized and efficient inference framework. We present dInfer, an efficient and extensible framework for dLLM inference. dInfer decomposes the inference pipeline into four modular components--model, diffusion iteration manager, decoding strategy, and KV-cache manager--and integrates novel algorithms for each component alongside system-level optimizations. Through this combination of algorithmic innovations and system enhancements, dInfer achieves substantial efficiency gains without compromising output quality on LLaDA-MoE. At batch size 1, it surpasses 1,100 tokens per second on HumanEval and averages over 800 tokens per second across six benchmarks on $8\times$ H800 GPUs. Compared to prior systems, dInfer delivers a $10\times$ speedup over Fast-dLLM while maintaining similar model performance. Even compared to the AR model (with a comparable number of activation parameters and performance) QWen2.5-3B, which is highly optimized with the latest vLLM inference engine, dInfer still delivers a $2$-$3\times$ speedup. The implementation of dInfer is open-sourced at https://github.com/inclusionAI/dInfer.

1 Introduction

dLLMs offer parallel denoising but face high computation costs, quality trade-offs, and fragmented evaluation. dInfer addresses these challenges with a modular framework, new decoding and cache algorithms, and system optimizations, achieving major throughput gains while maintaining model performance.

  • dInfer modularizes dLLM inference into model, diffusion iteration manager, decoding strategy, and KV-cache management components.The framework provides APIs for flexible combinations of algorithms and supports multiple dLLM variants.
  • dInfer is more than 10× faster than Fast-dLLM and 2−3× faster than vLLM’s Qwen2.5-3B when model performance is comparable.Figure 1 compares average TPS across six benchmarks and peak HumanEval speed.
  • dInfer combines iteration smoothing, hierarchical and credit decoding, vicinity KV-cache refresh, and system-level optimizations to improve inference efficiency.The system also supports tensor and expert parallelism, compilation, CUDA Graphs, and loop unrolling.
  • Over 1,100 TPS on HumanEval and over 800 TPS across six benchmarks are achieved at batch size 1 on 8× H800 GPUs while maintaining accuracy.The reported evaluation measures tokens per second per sequence.
  • dInfer is presented as an open-source demonstration that dLLM inference can surpass autoregressive models at batch size 1.The implementation is available through the project repository.

2 Framework Design

dInfer organizes dLLM inference around modular iteration, decoding, model, and KV-cache components. Its algorithms address parallel decoding and cache staleness while preserving flexible pipeline composition.

  • dLLMs require specialized cache management because bidirectional attention makes token representations evolve across denoising steps.Static KV reuse is infeasible, while recomputing the full sequence creates substantial computational overhead.
  • The framework combines model, diffusion iteration manager, decoding strategy, and KV-cache management components for customizable inference pipelines.These components can be combined through flexible algorithm choices.
  • 2.1 Diffusion Iteration Manager: Iteration smoothing retains previous token representations and fuses them with later embeddings to improve confidence and mitigate KV-cache-related degradation.It extends blockwise diffusion iteration by enabling cross-iteration information flow.
  • 2.2 Decoding Strategy: Hierarchical decoding partitions masked spans recursively, while credit decoding commits tokens with consistently stable historical confidence scores.Both strategies improve parallel decoding efficiency without retraining the underlying model.
  • 2.3 KV-cache management: Vicinity KV-cache refresh updates masked tokens and nearby neighbors during denoising, followed by a full cache update after block completion.The strategy uses semantic locality to balance cache-update cost with global consistency.
  • 2.4 Model support: dInfer supports LLaDA-MoE, LLaDA-1.5, and LLaDA-Instruct, and introduces Trajectory Distillation to improve parallel decoding in LLaDA-MoE.Trajectory Distillation fine-tunes models using effective decoding trajectories from their own generation process.

3 Implementation Details

dInfer uses parallel execution, compilation, CUDA Graphs, loop unrolling, early termination, and control-flow-free decoding to reduce inference overhead. These optimizations target both single-sequence execution and diffusion-iteration scheduling.

  • Tensor parallelism distributes dense computations, while expert parallelism improves LLaDA-MoE utilization even at batch size 1.The two forms of parallelism are combined to improve GPU utilization.
  • PyTorch compilation and NVIDIA CUDA Graphs improve inference efficiency by over 200% when tensor and expert parallelism are enabled.Kernel fusion and graph execution reduce PyTorch execution overhead.
  • Loop unrolling reduces CUDA stream bubbles and boosts diffusion-iteration efficiency by about 5-10%.It allows Python to launch CUDA kernels continuously without stream-synchronization blocking.
  • Early termination halts decoding after an EOS token and improves inference efficiency by 15-40%.Remaining blocks are filled with EOS to avoid redundant computation.
  • Control-flow-free decoding and elimination of tensor-to-Python transfers make decoding compatible with loop-unrolling optimizations.This design supports high-throughput diffusion iteration execution.

4 Evaluations

The evaluation measures parallel decoding with TPF and overall inference efficiency with TPS across six datasets and compares dInfer with Fast-dLLM, vLLM, and Qwen2.5-3B. dInfer improves throughput while maintaining comparable or higher performance in the reported configurations.

  • Evaluation setup: The study uses six datasets spanning code generation, mathematical reasoning, and instruction-following agent tasks.Efficiency is assessed with TPF per sequence for parallel decoding within an iteration and TPS per sequence for overall inference efficiency.
  • Evaluation setup: Experiments compare dInfer and Fast-dLLM under configurations with and without KV cache, using different decoding and cache-management strategies.All experiments use batch size 1 on 8× NVIDIA H800 GPUs with generation length 1024 and block size 64.
  • Accuracy and configuration: 54.33 average accuracy without KV cache is higher than Fast-dLLM and comparable to Qwen2.5-3B in vLLM and reported LLaDA-MoE results.With KV cache enabled, dInfer reports 53.96 accuracy versus 52.15 for Fast-dLLM while delivering 6× speedup.
  • Trajectory distillation: 847.22 average TPS for LLaDA-MoE-TD exceeds the non-distilled baseline’s 680.71 TPS and is more than three times Qwen2.5-3B in vLLM.The trajectory-distilled model combines with dInfer’s algorithmic optimizations to improve efficiency across six benchmarks.
  • Performance comparison: 680.71 TPS gives dInfer a 2–3× improvement over vLLM and more than a tenfold enhancement over Fast-dLLM, with similar performance scores.Table 1 reports 680.71 versus 277.45 TPS against vLLM and 680.71 versus 63.61 against Fast-dLLM, alongside performance scores of 53.96 versus 53.52 for Fast-dLLM.

5 Conclusion

The paper presents dInfer as an efficient, extensible dLLM inference framework that combines modular design with decoding and cache optimizations. Experiments on LLaDA-MoE report throughput above 1,100 TPS on 8× H800 GPUs while maintaining output quality.

  • 5 Conclusion: dInfer combines modular inference components with hierarchical decoding, credit decoding, iteration smoothing, and vicinity KV-cache refresh to address dLLM computation and parallel-decoding bottlenecks.The framework is positioned as both a practical toolkit and a standardized platform for dLLM research and development.
  • 5 Conclusion: Over 1,100 TPS on 8× H800 GPUs demonstrates the reported throughput of dInfer on LLaDA-MoE while maintaining output quality.The conclusion characterizes this as state-of-the-art throughput in the reported evaluation.

A.1 IterSmooth: Iteration Smoothing

IterSmooth reuses logits from masked positions as distribution-level embedding signals instead of discarding them after argmax decoding. A scheduled mixing weight and decode threshold provide conservative early behavior while increasing guidance later.

  • A.1 IterSmooth: Iteration Smoothing: IterSmooth converts masked-position logits into expected embeddings and injects them into mask-token embeddings, enriching uncertain positions with dense signals.The method operates only on masked positions to avoid shifting the training distribution elsewhere.
  • A.1 IterSmooth: Iteration Smoothing: The mixing weight α_t grows from a small initial value toward a preset maximum, while the decode threshold decays from 1.0 toward a target across iterations.This schedule stabilizes inputs early and progressively increases distribution-level guidance later.
  • A.1 IterSmooth: Iteration Smoothing: 30–40% more tokens decoded per diffusion iteration is observed with IterSmooth, alongside improved final generated-text quality.The reported gain comes from leveraging the full probability distribution rather than only argmax tokens.

B.1 Hierarchical decoding

Naive parallel decoding can degrade quality because simultaneously generated tokens violate conditional independence and create semantic inconsistencies. Hierarchical Decoding addresses this with recursive, confidence-based partitioning of masked spans.

  • B.1 Hierarchical decoding: Naive parallel decoding often causes semantic inconsistencies because simultaneously generated tokens violate the conditional independence assumption.This quality problem motivates the hierarchical strategy.
  • B.1 Hierarchical decoding: Hierarchical Decoding recursively partitions masked spans and seeks to decode at least one confident token in each region per forward pass.The strategy is training-free and is inspired by divide-and-conquer.
  • B.1 Hierarchical decoding: Non-contiguous decoding increases spacing between masked tokens, reducing local dependencies and improving semantic consistency.Selecting positions near span centers also causes undecoded regions to shrink recursively.

B.2 Credit decoding

CreditDecoding accelerates dLLM parallel decoding by tracking token consistency across steps and using that history to prioritize stable, previously underconfident predictions. It preserves the underlying decoding policy while remaining compatible with common inference optimizations.

  • CreditDecoding reduces redundant computation by helping tokens that stabilize early but remain below confidence thresholds avoid repeated re-masking and reevaluation.It is training-free and accelerates convergence in parallel decoding.
  • Credits quantify how consistently each token has been favored during generation, providing a temporal prior for its likelihood of being correct.
  • Discounting earlier confidence prevents errors from accumulating, while a concave transformation gives larger boosts to low- or moderate-confidence predictions.These choices help correct but underconfident tokens stabilize earlier.
  • The accumulated credit is fused with logits as a prior, boosting consistently predicted tokens and suppressing fluctuating or transiently confident ones.This enhances decoding stability, particularly for long-sequence and reasoning tasks.
  • CreditDecoding replaces the original distribution without changing the sampling policy, preserving compatibility with threshold decoding, top-k sampling, KV-cache, and compiler optimizations.Credits are maintained within the current decoding block to reduce interference from uncertain future context and improve scalability.

C.1 Inference Acceleration via Trajectory Compression

Trajectory Compression fine-tunes a dLLM to jump between non-consecutive states along verified generation trajectories, reducing the number of sampling steps. The resulting model improves tokens per forward substantially across mathematical reasoning and other domains.

  • Trajectory Compression addresses latency from iterative multi-step sampling by training the model to jump between non-consecutive states in an optimal generation trajectory.The resulting model is called LLaDA-MoE-TD.
  • The method first distills high-quality trajectories by retaining only generation paths whose final outputs pass an external correctness verifier.This produces the training set Tgold.
  • Compressed transition learning fine-tunes the model to predict a multi-step transition from an earlier state si to a later state sj instead of a standard single-step transition.Training targets are tokens masked in si but revealed in sj.
  • The compressed-transition objective minimizes the negative log-likelihood of predicting the newly revealed tokens between sampled trajectory states.Variable-length sequences are padded to the model’s maximum context length.
  • 99.8% higher TPF on mathematical reasoning and 45.3% average TPF improvement across other domains show that Trajectory Compression reduces dLLM inference latency.The method trains the model to execute large jumps during inference.

D Detailed Configuration of Experiments

Experiments use tailored combinations of decoding and optimization methods for different models and KV-cache settings, with ablations covering three configurations.

  • Experimental settings: Each experimental setting combines decoding and optimization methods to balance model performance and inference efficiency.The combinations are tuned separately for model characteristics and cache usage.
  • Hyperparameters: Threshold decoding uses a confidence threshold of 0.8, while hierarchical decoding uses thresholds of 0.92 and 0.62.The hierarchical decoder specifies both a decoding threshold and a lower boundary threshold.
  • Hyperparameters: Iteration smoothing uses a continuation weight of 0.3, and vicinity KV-Cache refreshment uses prefix and after looks of 16 with warmup times = 4.These settings define the reported smoothing and cache-refresh configurations.
  • Experimental settings: The experiments compare enabled methods across tailored configurations summarized in Table 3.A checkmark indicates that a method is applied in the corresponding setting.
  • Ablation studies: Ablation studies evaluate decoding algorithms on LLaDA-MoE without KV-Cache, with KV-Cache, and on LLaDA-MoE-TD with KV-Cache.These settings correspond to Tables 4, 5, and 6, respectively.
Loading 2510.08666v3…