Source-linked AI summary
DARE: Diffusion Large Language Models Alignment and Reinforcement Executor
Jingyi Yang, Yuxian Jiang, Xuhao Hu, Shuang Cheng, Biqing Qi, Jing Shao
TL;DR
dLLM post-training remains fragmented across model-specific implementations, making iteration, reproduction, and fair algorithm comparison difficult. DARE provides a shared execution stack built on verl and OpenCompass that supports multiple post-training methods and both masked and block diffusion models. Across LLaDA and Dream experiments, algorithm rankings vary by task and backbone, illustrating the value of matched infrastructure for comparison.
Problem
dLLM post-training is fragmented across model forks, rollout implementations, reward interfaces, and benchmark evaluations, making iteration, reproduction, and fair comparison difficult.
Method
DARE is a unified post-training and evaluation framework integrating SFT, PEFT, preference optimization, and dLLM-specific reinforcement learning for masked and block diffusion models.
Results
Across LLaDA and Dream experiments, no single algorithm dominates uniformly across tasks, and method rankings change with the backbone.
Takeaways & Limitations
DARE provides a reusable environment for comparing heterogeneous dLLM post-training algorithms under matched execution and evaluation infrastructure.
Abstract
from arXiv · showhide
Diffusion large language models (dLLMs) are emerging as a compelling alternative to dominant autoregressive models, replacing strictly sequential token generation with iterative denoising and parallel generation dynamics. However, their open-source ecosystem remains fragmented across model families and, in particular, across post-training pipelines, where reinforcement learning objectives, rollout implementations and evaluation scripts are often released as paper-specific codebases. This fragmentation slows research iteration, raises the engineering burden of reproduction, and makes fair comparison across algorithms difficult. We present \textbf{DARE} (\textbf{d}LLMs \textbf{A}lignment and \textbf{R}einforcement \textbf{E}xecutor), an open framework for post-training and evaluating dLLMs. Built on top of verl~\cite{sheng2024hybridflow} and OpenCompass~\cite{2023opencompass}, DARE unifies supervised fine-tuning, parameter-efficient fine-tuning, preference optimization, and dLLM-specific reinforcement learning under a shared execution stack for both masked and block diffusion language models. Across representative model families including LLaDA, Dream, SDAR, and LLaDA2.x, DARE provides broad algorithmic coverage, reproducible benchmark evaluation, and practical acceleration. Extensive empirical results position that DARE serves as a reusable research substrate for developing, comparing, and deploying post-training methods for current and emerging dLLMs.
1 Introduction
DARE addresses fragmentation in dLLM post-training by unifying diverse model families, post-training methods, system optimizations, and evaluation within one execution stack.
- dLLM reinforcement-learning implementations are fragmented across model forks, rollout code, reward interfaces, and evaluation scripts.This fragmentation slows iteration, complicates fair comparison, and raises the engineering barrier for researchers.
- DARE unifies supervised fine-tuning, parameter-efficient fine-tuning, preference optimization, and dLLM-specific reinforcement learning.The shared framework supports both masked and block diffusion language models.
- DARE supports LLaDA, Dream, SDAR, LLaDA-MoE, and LLaDA2.x through one infrastructure stack.The framework is built on verl for training and OpenCompass for evaluation.
- DARE treats rollout, training, and evaluation acceleration as first-class components of dLLM post-training.Its optimizations include model-aware attention and serving backends, fused loss kernels, and dLLM-aware evaluation extensions.
- The framework is presented as a reusable substrate for integrating, comparing, and evaluating dLLM post-training methods rather than as a new optimization objective.Its contributions emphasize unified infrastructure, broad method coverage, system optimization, and reproducible comparison.
2 Related Works
The paper reviews masked and block diffusion language models and the post-training methods they motivate, highlighting distinct modeling and systems requirements across these families.
- Masked Diffusion Language Models: Masked diffusion language models corrupt categorical tokens through processes such as absorbing-mask or uniform noise and learn to reverse that corruption.The mask predictor is trained by minimizing the negative evidence lower bound objective.
- Masked Diffusion Language Models: Large masked diffusion language models require diffusion-specific rollout and training pipelines because generation differs fundamentally from autoregressive left-to-right decoding.Their generation process motivates specialized execution infrastructure.
- Block Diffusion Language Models: Block diffusion models combine intrablock diffusion with inter-block autoregression and support variable-length generation with key-value caching.A sequence is partitioned into contiguous blocks, with each block denoised conditioned on preceding clean blocks.
- Block Diffusion Language Models: Block diffusion introduces distinct requirements for rollout backends, block-wise verification, and online policy updates.The coexistence of masked and block diffusion families motivates a unified dLLM executor.
- Post-Training Methods: Recent dLLM post-training work adapts preference alignment, GRPO-style optimization, and progressive-refinement objectives to masked diffusion models.Examples include VRPO, d1, Coupled-GRPO, and MDPO.
3 DARE Framework
DARE provides a shared post-training executor for masked and block diffusion language models, integrating multiple training methods, RL algorithms, model families, and evaluation components. It separates shared workflow structure from model- and algorithm-specific hooks while adding model-aware acceleration for training and rollout.
- Framework Architecture: DARE builds on verl for distributed training and OpenCompass for evaluation, adding dLLM-specific actor, rollout, reward, model, and execution wrappers.The framework keeps the outer execution skeleton shared and restricts customization to model- or algorithm-specific components.
- Supported Model Families: DARE supports LLaDA, Dream, LLaDA-MoE, SDAR, LLaDA2.0, and LLaDA2.1 across masked- and block-diffusion paradigms.These families require different rollout backends, training optimizations, and sometimes different policy-update paths.
- Unified Post-Training: The executor covers supervised fine-tuning, parameter-efficient fine-tuning, preference optimization, and multiple dLLM-specific reinforcement learning algorithms.Integrated RL methods include VRPO, D1, Coupled-GRPO, MDPO, CJ-GRPO, SPG, BGPO, and EBPO.
- Unified Post-Training: DARE shares rollout, reward, log-probability, advantage, and actor-or-critic-update stages while exposing diffusion-specific customization through defined algorithm hooks.The hooks specify elements such as corruption processes, trajectory construction, likelihood estimation, and policy loss.
- Acceleration Backends: Fast-dLLM with FlashAttention reduces MDLM rollout latency from about 161.6 seconds to about 73.4–73.5 seconds, yielding approximately 2.2× speedup.The same decoupled training-and-rollout backend design reports approximately 4× end-to-end RL pipeline speedup for the MDLM path.
- Acceleration Backends: The supported BDLM path provides more than 14× RL pipeline acceleration using LMDeploy or SGLang for rollout and block-aware fused training operators.DARE also supports online rollout-policy updates to address synchronization between rollout and actor components.
4 Empirical Results
DARE enables matched comparisons of post-training algorithms across LLaDA and Dream, showing that the strongest method depends on task and backbone. Training curves further indicate stability differences, with some ELBO-based methods more sensitive to estimator noise and sampling budgets.
- LLaDA as backbone: On LLaDA-8B-Instruct, CJ-GRPO leads GSM8K, Coupled-GRPO leads MATH and Countdown, VRPO leads HumanEval and MBPP, and BGPO leads Sudoku.The best-performing algorithm changes across math, code, and planning tasks.
- Dream as backbone: On Dream-7B-Instruct, CJ-GRPO leads GSM8K and MATH, while Coupled-GRPO leads HumanEval and MBPP; SPG is much weaker on math and code.These results show that algorithm performance varies substantially across model families.
- Cross-task comparison: No single algorithm dominates uniformly across tasks, and the same method can change rank when the backbone changes.The matched executor makes this cross-task and cross-backbone variation directly visible.
- Training stability: d1, Coupled-GRPO, and CJ-GRPO usually exhibit more stable reward curves, whereas ELBO-based methods can become fragile under limited Monte Carlo budgets.Estimator variance can destabilize optimization even when the final objective remains competitive on some tasks.
5 Conclusion and Future Work
DARE provides a reusable execution environment that unifies dLLM model families, post-training methods, rollout implementations, reward interfaces, and evaluation pipelines. The paper identifies broader model integration, additional algorithms and stability techniques, efficiency ablations, and deployment-oriented evaluation as future directions.
- Conclusion: DARE unifies model families, post-training methods, rollout implementations, reward interfaces, and evaluation pipelines under one reusable stack.The framework supports broad model and benchmark coverage and direct comparison of heterogeneous post-training algorithms.
- Future Work: Future work includes integrating diffusion vision-language and omni-language models, adding new estimators and control policies, and expanding efficiency and deployment-oriented evaluations.These directions cover model, algorithm, and systems development.