PaperlayerView exact revision on arXiv

Source-linked AI summary

DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation

Shansan Gong, Ruixiang Zhang, Huangjie Zheng, Jiatao Gu, Navdeep Jaitly, Lingpeng Kong, Yizhe Zhang

arXiv:2506.20639v2cs.CL

TL;DR

Coding with diffusion language models remains under-explored, particularly in decoding behavior and reinforcement-learning methods. The paper analyzes DiffuCoder, a 7B code model, and introduces coupled-GRPO, whose complementary mask sampling improves likelihood estimation and raises EvalPlus performance by 4.4%.

  • Problem

    Training and inference mechanisms for diffusion language models in coding remain under-explored, including how their decoding differs from autoregressive models.

  • Method

    The paper trains DiffuCoder and analyzes its decoding behavior before introducing coupled-GRPO, which uses complementary mask noise for diffusion-native RL training.

  • Results

    Coupled-GRPO improves DiffuCoder’s EvalPlus score by 4.4% with training on 21K samples, while higher temperature diversifies generation order and improves pass@10.

  • Takeaways & Limitations

    dLLMs can adapt their causal generation behavior without semi-AR decoding, and coupled-GRPO provides an RL framework aligned with diffusion’s non-autoregressive nature.

Abstract

from arXiv · show

Diffusion large language models (dLLMs) are compelling alternatives to autoregressive (AR) models because their denoising models operate over the entire sequence. The global planning and iterative refinement features of dLLMs are particularly useful for code generation. However, current training and inference mechanisms for dLLMs in coding are still under-explored. To demystify the decoding behavior of dLLMs and unlock their potential for coding, we systematically investigate their denoising processes and reinforcement learning (RL) methods. We train a 7B dLLM, \textbf{DiffuCoder}, on 130B tokens of code. Using this model as a testbed, we analyze its decoding behavior, revealing how it differs from that of AR models: (1) dLLMs can decide how causal their generation should be without relying on semi-AR decoding, and (2) increasing the sampling temperature diversifies not only token choices but also their generation order. This diversity creates a rich search space for RL rollouts. For RL training, to reduce the variance of token log-likelihood estimates and maintain training efficiency, we propose \textbf{coupled-GRPO}, a novel sampling scheme that constructs complementary mask noise for completions used in training. In our experiments, coupled-GRPO significantly improves DiffuCoder's performance on code generation benchmarks (+4.4\% on EvalPlus) and reduces reliance on AR bias during decoding. Our work provides deeper insight into the machinery of dLLM generation and offers an effective, diffusion-native RL training framework. https://github.com/apple/ml-diffucoder.

1 INTRODUCTION

This work studies open-source diffusion language models for code, focusing on how their decoding differs from autoregressive generation and how reinforcement learning can better exploit that behavior. It introduces DiffuCoder and coupled-GRPO, which improves code-generation performance while avoiding semi-autoregressive decoding.

  • Motivation: Open-source dLLM coding mechanisms remain under-interpreted, while existing post-training methods show marginal gains or rely heavily on semi-AR decoding.Masked diffusion models refine sequences in parallel, supporting global planning, but their coding training and inference behavior remains unclear.
  • DiffuCoder: DiffuCoder is a 7B-scale masked diffusion model specialized for code and trained on 130B effective tokens.The model serves as a testbed for analyzing dLLM behavior and developing diffusion-native post-training methods.
  • Decoding analysis: Higher sampling temperature diversifies both token choices and generation order, allowing DiffuCoder to move beyond strict left-to-right decoding.Increasing temperature from 0.2 to 1.2 also improves pass@10 accuracy in the reported experiments.
  • Reinforcement learning: Coupled-GRPO uses paired complementary mask noise to estimate token likelihoods efficiently without relying on semi-AR decoding.The method targets lower estimation variance and further improves the instruct-tuned model while strengthening its non-AR generation pattern.
  • Results: Coupled-GRPO boosts DiffuCoder’s EvalPlus score by 4.4% with training on only 21K samples.The result supports reinforcement learning aligned with diffusion principles.

2 PRELIMINARIES AND NOTATIONS

The preliminaries describe masked diffusion as iterative denoising and cast its reinforcement-learning formulation as a Markov decision process. GRPO can then be adapted to diffusion models using group-based reward estimation and token-level policy optimization.

  • Masked diffusion models: Discrete diffusion gradually corrupts data through categorical transitions and trains denoising parameters by minimizing an ELBO-based negative log-likelihood.For absorbing diffusion, transitions replace tokens with a special [MASK] token; continuous-time formulations express the objective as weighted cross-entropy.
  • Markov decision process: Diffusion denoising can be represented as an MDP with states, actions, transitions, an initial distribution, and rewards.In conditional masked generation, the state includes the condition, timestep, and noisy sequence, while the action is the previous denoising state.
  • Markov decision process: The conditional-generation reward is applied at the final denoising step because fine-grained intermediate rewards are usually difficult to quantify.The resulting policy gradient optimizes the expected return over denoising trajectories.
  • GRPO: GRPO samples grouped outputs, estimates the value baseline from their mean reward, and uses relative advantages without training a separate value function.Its loss can be applied to masked diffusion models by combining the GRPO objective with the diffusion MDP formulation.
  • Coupled sampling: Coupled sampling pairs complementary masks so every completion token is unmasked in exactly one of two forward passes.This design targets more efficient probability estimation while retaining full token coverage.

3 DIFFUCODER

DiffuCoder is developed through staged training and evaluated across diverse Python code benchmarks against autoregressive, diffusion, and commercial models. Its continual training yields performance comparable to strong AR code models, while instruction tuning exposes a remaining improvement gap for dLLMs.

  • Training pipeline: DiffuCoder’s training pipeline includes adaptation pre-training, mid-training as an annealing phase, and instruction tuning.The stages connect pre-training and post-training while progressively adapting the model for code generation.
  • Training pipeline: The model is adapted from Qwen-2.5-Coder using a 400B-token code corpus, 16B annealing tokens, and 436K supervised fine-tuning samples.These data sources support continual pre-training, mid-training, and instruction tuning.
  • Evaluation: Evaluation covers HumanEval, MBPP, EvalPlus, and BigCodeBench across completion and instruction settings.The benchmarks use Python tasks to assess code correctness and quality.
  • Performance: After continual training on 130B code tokens, DiffuCoder performs on par with Qwen2.5-Coder and OpenCoder.The comparison includes autoregressive code models, general dLLMs, and commercial models.
  • Motivation for RL: The instruction-tuning improvement gap between AR models and dLLMs motivates exploring reinforcement-learning post-training methods.Earlier diffusion RL approaches relied heavily on semi-AR decoding, which deviates from diffusion’s global nature.
  • Performance: Table 1 compares 7/8B code models across generation paradigms and reports absolute score changes for instruct models relative to their bases.EvalPlus is computed as the average of HumanEval+ and MBPP+, with results from public reports marked separately.

4 UNDERSTANDING MASK DIFFUSION MODELS BASED ON DIFFUCODER

This section measures how closely masked diffusion decoding follows left-to-right generation and examines how model training, modality, and temperature affect that behavior. dLLMs retain autoregressive structure but can generate flexibly and increasingly non-autoregressively under suitable conditions.

  • AR-ness metrics: Local AR-ness measures consecutive next-token behavior, while global AR-ness measures whether the model unmasks the earliest remaining positions.Higher values indicate more autoregressive generation; local AR-ness becomes harder to satisfy over longer consecutive spans, while global AR-ness becomes easier as more early positions are allowed.
  • Decoding behavior: Unlike AR models, dLLMs recover a substantial fraction of tokens out of left-to-right order while retaining non-trivial autoregressive structure.AR models have both metrics equal to 1, whereas dLLMs show flexible decoding orders with AR-ness values between fully non-AR and fully AR behavior.
  • Model origins: Adapted dLLMs tend to be more autoregressive than models trained from scratch because they inherit left-to-right dependencies from AR pretraining.Lower AR-ness may enable more parallel generation, while higher AR-ness can sometimes support performance through causal decoding strategies.
  • Data modalities: Code generation has lower mean and higher variance in global AR-ness than math generation, consistent with more globally planned token recovery.The paper relates this pattern to code’s intrinsic structure and programmers’ back-and-forth refinement during implementation.
  • Training stages: After 65B-token Stage 1 training, AR-ness is relatively low; later high-quality training initially increases causal bias, then declining AR-ness accompanies improved task performance.After GRPO, global AR-ness decreases and performance drops less when decoding uses half as many steps.
  • Sampling temperature: Higher sampling temperatures encourage more parallel generation and can substantially increase pass@k by diversifying both token choices and generation orders.For DiffuCoder, a suitable temperature range such as 1.0 to 1.2 improves pass@k relative to low-temperature sampling that produces high pass@1 but limited diversity.

5 COUPLED-GRPO

This section introduces coupled-GRPO, a diffusion-native adaptation of GRPO designed to improve token log-likelihood estimation while controlling sampling variance and training cost. Experiments compare it with full-mask, d1, and decoupled baselines across reward learning and benchmark performance.

  • Motivation: Diffusion GRPO requires Monte Carlo token-probability estimates, but these estimates add substantial training overhead and variance.The diffusion objective bounds negative log-likelihood through an ELBO, with losses depending on sampled diffusion states and masked-token positions.
  • Baselines: The d1 baseline masks all completion tokens and uses one forward pass to estimate each token’s probability.This is equivalent to sampling once at diffusion step t = T and yields the baseline sequence of token log-probabilities.
  • Baselines: Masking condition tokens makes completion-probability estimates unreliable and does not produce stable reward improvement in the code experiments.The method therefore reverts to full completion masking, although that baseline remains biased by the entropy sink’s causal preference.
  • Coupled-GRPO: Coupled-GRPO adds paired complementary mask noise so every token receives a learning signal under realistic partial-masking contexts.The scheme improves probability accuracy while adding 2λ samples relative to the baseline; the experiments use λ = 1.
  • Coupled-GRPO: Coupled sampling can be interpreted as antithetic-variates variance reduction, with leave-one-out advantages and verified rewards supporting training.Verified rewards include code-format validity and execution pass rate over test cases as a correctness reward.
  • Results: DiffuCoder-Instruct achieves higher pass@10 at temperature 1.2 than at 1.0, while RL shifts the optimal evaluation temperature from 0.2 toward 0.3 or 0.4.Coupled-GRPO is also sensitive to rollout temperature, linking sampling diversity to training behavior.

6 RELATED WORK

Related work situates the paper within discrete text diffusion, diffusion-based code generation, and reinforcement learning with verifiable rewards. These lines of research establish the model family, application domain, and post-training context.

  • Text diffusion models: Discrete diffusion models introduced discrete noise for text, later developing into masked diffusion models and larger diffusion language models.This progression reflects efforts to accommodate discrete text while scaling diffusion approaches.
  • Code generation: Diffusion-based code generation progressed from small-scale CodeFusion to commercial-scale systems that approach leading AR code models while generating faster.The cited commercial systems include Mercury and Gemini.
  • Reinforcement learning: RL with verifiable rewards and GRPO has improved math reasoning and code generation, while diffusion-specific RL remains a developing area.Prior work includes RLVR with GRPO and diffusion-model approaches such as VRPO.

7 CONCLUSION

DiffuCoder shows that diffusion-native decoding and reinforcement learning can exploit flexible generation order rather than imposing semi-autoregressive behavior. Coupled sampling provides complete token coverage and supports more efficient estimation through complementary masks.

  • 7 CONCLUSION: DiffuCoder demonstrates that sampling temperature affects both token selection and generation order, creating diverse samples for optimization.The model’s decoding can therefore vary in how closely it follows an autoregressive order.
  • 7 CONCLUSION: The coupled-GRPO procedure is designed as a diffusion-native alternative to semi-autoregressive decoding for policy optimization.Its probability estimates are computed from paired complementary masks rather than an imposed semi-AR generation process.
  • 7 CONCLUSION: Coupled-GRPO uses complementary masks so every token is evaluated exactly once across each paired sampling step.The two masks hide complementary token subsets, covering the full completion sequence.
  • 7 CONCLUSION: Coupled sampling improves estimation efficiency by ensuring non-zero score contributions for every token in each coupled pair.This addresses the standard estimator’s need for many samples when individual masks cover only subsets of tokens.

A.4 THEORETICAL ANALYSIS OF COUPLED-GRPO

The theoretical analysis frames coupled-GRPO as a variance-reduction method for estimating per-token scores used in GRPO policy updates. Its complementary sampling scheme guarantees coverage while preserving the quantities needed for importance sampling.

  • A.4 THEORETICAL ANALYSIS OF COUPLED-GRPO: Coupled-GRPO supports policy optimization by sampling completions, computing rewards and advantages, and updating the model with coupled probability estimates.The algorithm uses GRPO iterations with complementary masks before gradient descent updates.
  • A.4 THEORETICAL ANALYSIS OF COUPLED-GRPO: Coupled-GRPO estimates per-token scores to compute policy probability ratios used in the GRPO objective.The score is defined as an expectation over a diffusion timestep and mask, then used for the importance-sampling ratio.
  • A.4 THEORETICAL ANALYSIS OF COUPLED-GRPO: Because the scoring function is non-zero only for masked tokens, standard Monte Carlo estimation can require many samples to produce reliable scores for all tokens.Individual masks cover only a subset of positions in any sample.

A.4.2 PROOFS

The proofs establish that the antithetic coupled estimator is unbiased and has lower variance than standard Monte Carlo estimation. The reduction follows from the negative covariance induced by complementary mask indicators.

  • A.4.2 PROOFS: The coupled estimator is unbiased for the desired per-token quantity.The antithetic pair has the same distribution as the original sample, so linearity of expectation preserves the target expectation.
  • A.4.2 PROOFS: Complementary binary masks make exactly one scoring term non-zero for each token within a coupled pair.This mutual exclusivity produces complete token coverage and drives the covariance result.
  • A.4.2 PROOFS: The covariance between paired scoring functions is guaranteed to be negative when the expected score is positive.The product of the paired mask indicators is always zero, while each scoring function is non-negative.
  • A.4.2 PROOFS: The coupled estimator therefore has strictly lower variance than the standard Monte Carlo estimator under the stated positive-score assumption.The variance difference is positive because the complementary estimators contribute a negative covariance term.

B.1 TRAINING DETAILS

DiffuCoder’s training combines large-scale code pretraining, mid-training, instruction tuning, and coupled-GRPO reinforcement learning. The accompanying analyses examine decoding entropy, generation order, and timestep choices relevant to speed and non-autoregressive behavior.

  • B.1 TRAINING DETAILS: Coupled-GRPO training uses 21K hard samples with verifiable tests, rollout temperature 1.2, and coupled timesteps sampled from [0.2, 0.8].The timestep range avoids extreme loss values observed outside the empirically selected interval.
  • B.1 TRAINING DETAILS: The weighted reward combines code correctness and format validity, assigning 2.0 weight to code reward and 0.5 to format reward.Code pass rate is evaluated only when the completion satisfies the format condition.
  • C.1 ENTROPY PATTERN: The entropy analysis visualizes decoding entropy by token index and diffusion step, with sequence length matched to the number of diffusion timesteps.The figure’s two-dimensional heatmaps expose how confidence varies throughout decoding.
  • C.1 ENTROPY PATTERN: Higher temperature produces less autoregressive generation, with later code regions and padding-related positions often recovered before key code content.Figure 10 colors characters by recovery order, from earliest red to latest purple.
  • B.1 TRAINING DETAILS: Reducing decoding timesteps to half the sequence length is presented as a 2x speedup setting for comparing model variants.The figure labels the default sequence-length setting as 1x and the shortened setting as 2x.

C.4 COUPLED GRPO TRAINING

The discussion highlights training-cost and completion-length behavior for DiffuCoder, alongside reward-curve comparisons between coupled-GRPO and relevant baselines.

  • Completion length did not consistently increase during GRPO training, unlike reported behavior in AR GRPO training.
  • DiffuCoder’s end-to-end GRPO training takes twice as long as training the AR model Qwen2.5-Coder.
  • Figure 12 compares reward curves for coupled-GRPO against d1, decoupled GRPO, and regular GRPO with Qwen2.5Coder+SFT.

D DISCUSSIONS

The paper identifies limitations involving data quality, instruction-template diversity, programming-language coverage, long reasoning, and incomplete entropy analysis.

  • The experiments rely entirely on open-source community datasets that may not provide the highest data quality.
  • Relatively fixed and homogeneous GRPO instruction templates may limit generalization by encouraging reliance on specific prompt formats.
  • Training and evaluation primarily focus on Python, leaving extension to multiple programming languages as future work.
  • DiffuCoder was not trained on long-reasoning tasks, which remain challenging because of limited sequence length and slower dLLM inference.
  • Further token-entropy analysis is proposed to clarify entropy dynamics and their potential impact on reward optimization.
Loading 2506.20639v2…