PaperlayerView exact revision on arXiv

Source-linked AI summary

CanvasAnneal: Curriculum Reinforcement Learning for Diffusion Language Models

Blake Olson, Yuhang Song, Emmett McQuinn, Yuan Shangguan

arXiv:2609.13060v1cs.LG

TL;DR

Diffusion language models have parallel-generation strengths, but reinforcement learning from fully masked noise faces an exploration bottleneck. CanvasAnneal injects teacher reasoning traces into the initial canvas and gradually removes that guidance. It improves over diffu-GRPO on some reasoning and tool-use benchmarks, with task-dependent gains.

  • Problem

    Diffusion RL must explore from a fully masked, unconditioned canvas, with credit assignment spanning token positions and reverse steps and early rollout variance hindering viable reasoning or tool-call discovery.

    Source: §1 · pp. 1–2

  • Method

    CanvasAnneal initializes prompt groups with teacher-trace context using suffix masking and a dynamic Beta curriculum, while retaining pure-noise rollouts as guidance is annealed.

    Source: §1 · p. 2, §4.2 · p. 5, §4.3 · p. 6

  • Results

    CanvasAnneal improves over diffu-GRPO on MATH500 and Countdown and accelerates reward improvement on xLAM and Countdown, while effects on GSM8K and tool-use settings are mixed.

    Source: §1 · p. 2, §6 · p. 11

  • Takeaways & Limitations

    The experiments suggest curriculum-guided initialization can improve diffusion RL optimization and downstream accuracy in some settings, but its effectiveness depends on task and training distribution.

    Source: §6 · p. 11

    The ablations do not exhaustively evaluate interactions among trace-injection frequency, curriculum schedule, and masking structure; a full factorial decomposition is left to future work.

    Source: §5.3 · p. 10

Abstract

from arXiv · show

Diffusion Language Models (DLMs) offer promising parallel generation capabilities but lag behind autoregressive models in complex reasoning and tool-use tasks. While Reinforcement Learning (RL) has recently been applied to enhance DLMs, standard RL approaches suffer from an exploration bottleneck. To address this, we inject reasoning priors from a stronger teacher model to guide RL exploration. In this paper, we introduce CanvasAnneal, a curriculum-guided diffusion RL framework. During the initial RL phase, we warm-start exploration by injecting teacher-generated reasoning traces into the initial diffusion canvas. As training progresses, we gradually remove this guidance and require the model to generate more of the reasoning trajectory independently. Across mathematical reasoning and tool-use benchmarks, CanvasAnneal improves over standard diffu-GRPO on MATH500, Countdown, and Tau2 and substantially accelerates reward improvement on several tasks, while gains are task-dependent. Our results suggest that structured training-time guidance can alleviate exploration bottlenecks in diffusion RL and speed up convergence on harder tasks.

1 Introduction

Diffusion RL struggles to explore from fully masked noise, despite the parallel-generation strengths of diffusion language models. CanvasAnneal addresses this by annealing teacher-provided reasoning guidance during training, with task-dependent benchmark gains.

1 Introduction

  • DLMs refine complete sequences in parallel, enabling arbitrary-order infilling, iterative editing, flexible speed-quality trade-offs, and relief from sequential alignment issues.The passage contrasts this with autoregressive token-by-token generation in a left-to-right order.Source: §1 · p. 1
  • DLM reinforcement learning must assign credit across token positions and reverse diffusion steps, while high-entropy noise causes early rollout variance that impedes viable reasoning and tool-call discovery.The difficulty arises because DLMs begin sampling from a fully masked, unconditioned noise canvas.Source: §1 · pp. 1–2
  • CanvasAnneal injects teacher-generated reasoning traces into the initial diffusion canvas, then progressively reduces teacher context until initialization is nearly fully masked.The curriculum warm-starts exploration during early RL training.Source: §1 · p. 2
  • CanvasAnneal improves over diffu-GRPO on MATH500 and Countdown and accelerates reward improvement on xLAM and Countdown, but gains are mixed on GSM8K and tool-use tasks.The method is evaluated on mathematical reasoning and tool-use tasks.Source: §1 · p. 2

2 Related Work

Related work addresses diffusion language modeling, diffusion-specific RL optimization, teacher-guided exploration, and curriculum learning. CanvasAnneal differs by annealing teacher context over the diffusion canvas while retaining unassisted rollouts.

2 Related Work

  • Discrete diffusion models refine sequences in parallel through masked transitions, but complex reasoning remains challenging because exact sequence likelihoods are intractable.DLMs also support dynamic infilling and flexible speed-quality trade-offs (Nie et al., 2025; Ye et al., 2025).Source: §2 · p. 2
  • Diffusion RL methods address credit assignment across denoising steps and token positions using tractable objectives, trajectory or state optimization, and revised likelihood or policy objectives.The passage describes diffu-GRPO (Zhao et al., 2025a), DCOLT and MDPO (Huang et al., 2025; He et al., 2025), and later variants.Source: §2 · p. 2
  • CanvasAnneal uses contiguous suffix masking and a training-progress-dependent Beta curriculum, unlike IGPO’s failure-adaptive scattered inpainting and BREAD’s autoregressive expert-prefix branching (Zhao et al., 2025b).It retains a fixed fraction of pure-noise rollouts and uses the same partially masked canvas across a prompt group; teacher guidance is training-only.Source: §2 · p. 3
  • CanvasAnneal applies curriculum learning to sequence corruption, progressively closing the gap between teacher-guided scaffolding and autonomous denoising without task-difficulty annotations or supervised fine-tuning pipelines.The passage frames this as structural guidance over sequence corruption rather than task-level prompt filtering (Bengio et al., 2015; Parashar et al., 2026b).Source: §2 · p. 3

3 Preliminaries

The preliminaries describe masked diffusion as token corruption followed by reverse prediction, then introduce a tractable GRPO approximation for policy optimization. Group-relative rewards and token-level likelihood surrogates support the diffusion setting.

3 Preliminaries

  • The forward process corrupts a target sequence over discrete timesteps by replacing tokens with [MASK] according to a monotonically increasing schedule that ends in an all-masked canvas.The masked-token cross-entropy objective is described for training the reverse model.Source: §3 · p. 3
  • The reverse process uses pθ(x0 | xt) to predict clean-sequence logits from a corrupted state, with the training objective evaluated over masked-token positions.M(xt) denotes the indices of tokens masked in xt.Source: §3 · p. 3
  • For each prompt, diffu-GRPO samples a group of responses and approximates per-token likelihood using a perturbed, partially masked prompt and a fully masked response.The passage defines ⊕ as sequence concatenation.Source: §3 · p. 3
  • Each response receives a scalar reward and a group-relative advantage shared across its tokens; diffu-GRPO uses token-level importance ratios from the one-step likelihood approximation.The objective applies PPO-style clipping to token-level ratios and a KL penalty toward a fixed reference policy.Source: §3 · p. 4

4 Methods

CanvasAnneal uses teacher-generated reasoning as a training-time scaffold, then progressively reduces that guidance while preserving fully unassisted rollouts. Its methods combine trace generation, suffix masking, a dynamic masking curriculum, group-level policy optimization, and native teacher-free inference.

4.1 Teacher Reasoning Trace Generation

  • Offline Gemini 3.1 Pro traces provide intermediate reasoning steps without the final answer or tool command, and are length-calibrated to about 256 tokens.Mathematical and function-calling rationales use different tags.Source: §4.1 · p. 4

4.2 Reasoning Injection into Diffusion Canvas

  • Suffix masking preserves a contiguous prefix of the teacher rationale while masking the remaining suffix, requiring the policy to generate the rest of the reasoning and answer.As the masking fraction increases, the amount of teacher-provided context decreases.Source: §4.2 · p. 5

4.3 Curriculum Annealing via Dynamic Beta Mixture

  • A prompt-group curriculum mixes pure-noise canvases with time-varying Beta-sampled mask ratios, shifting from mostly retained teacher traces toward larger independently generated portions.The Beta parameters move from (1, 5), with expected mask fraction 1/6, to (5, 1), with expected mask fraction 5/6; pure-noise probability is 0.25 for xLAM and 0.75 for math benchmarks.Source: §4.3 · p. 6

4.4 Policy Optimization and Native Inference

  • All rollouts in a prompt group share the same sampled masking ratio and initial canvas, while the surrogate objective evaluates both fixed teacher-prefix and generated positions.This keeps within-group context constraints identical for advantage computation; the paper motivates including both token sets by diffusion's parallel credit assignment.Source: §4.2 · pp. 5–6, §4.3 · p. 6
  • At test time, the teacher is omitted and the model generates from a fully masked canvas using native iterative denoising.Inference uses m = 1.0 and requires no additional teacher inference.Source: §4.4 · p. 6

5 Experiments

Experiments evaluate CanvasAnneal on mathematical reasoning and tool-use tasks, using LLaDA-7B-A1B-Instruct and comparisons with base, SFT, and RL configurations. Results vary by task: CanvasAnneal improves on several benchmarks, while ablations and discussion emphasize retaining unassisted rollouts and tailoring guidance to the task.

5.2 Results

  • CanvasAnneal improves over standard diffu-GRPO on MATH500 by 6.0, 2.0, and 0.4 percentage points at generation lengths 128, 256, and 512, and on Countdown by 2.73, 3.91, and 1.96 points; diffu-GRPO remains stronger on GSM8K.The reported test accuracy confidence intervals are approximately ±2.5% for GSM8K, ±4.3% for MATH500, and ±6.1% for Countdown.Source: §5.2 · p. 8, Table 1 · p. 9
  • Without SFT, CanvasAnneal exceeds the SFT+RL d1-LLaDA baseline on MATH500 at lengths 256 and 512, but SFT+RL remains competitive or stronger in other settings.Combining SFT with CanvasAnneal yields the best result in several individual configurations, including 70.96% on GSM8K at length 256, 36.20% on MATH500 at length 128, and 48.05% on Countdown at length 128.Source: §5.2 · p. 8
  • CanvasAnneal raises Tau2 scores from 6.14% to 7.90% on Retail, 12.00% to 18.00% on Airline, and 12.30% to 13.20% on Telecom, while BFCL results are mixed.On BFCL, CanvasAnneal scores 59.08% versus 53.33% on Simple, 65.50% versus 66.00% on Multiple, and 60.50% versus 65.00% on Parallel, compared with diffu-GRPO.Source: §5.2 · p. 8, Table 2 · p. 9
  • On xLAM, CanvasAnneal reaches reward 1.0 in 7.4× fewer steps, and on Countdown reaches reward 0.5 in 18.8× fewer steps than diffu-GRPO.On MATH it reaches peak reward 228 updates earlier, but final reward is 0.6% lower; diffu-GRPO has higher peak, mean, and final rewards on GSM8K.Source: §5.2 · pp. 8–9, Figure 2 · p. 10

5.3 Ablation Studies

  • Increasing the share of pure-noise prompt groups generally improves shorter-length GSM8K and MATH500 results, while the 75% No-Trace / 25% Injected mixture leads in selected settings.The 75% No-Trace / 25% Injected configuration is strongest on GSM8K at length 128 and MATH500 at length 256, but other ratios lead in other settings.Source: §5.3 · p. 9, Table 3 · p. 10
  • Dynamic Beta schedules outperform uniform sampling on MATH, with α : 1 →10, β : 10 →1 reaching 39.00% on MATH and 44.53% on Countdown, while uniform sampling leads GSM8K at 66.34%.The evaluated schedules use a generation length of 256 tokens; their relative performance remains task-dependent.Source: §5.3 · p. 9, Table 4 · p. 10
  • At generation length 256, suffix masking scores 68.54%, 41.00%, and 49.61% on GSM8K, MATH500, and Countdown, versus 60.58%, 37.80%, and 42.19% with random masking.Suffix masking retains a contiguous teacher-rationale prefix; random masking also requires reconstructing missing intermediate tokens.Source: §5.3 · p. 10

5.4 Discussion

  • Teacher traces guide training without teacher inference at test time, but the discussion reports task-dependent gains and identifies unassisted rollouts as a useful supplement to guided ones.The ablations do not exhaustively evaluate interactions among trace frequency, schedule, and masking structure; a full factorial decomposition is left to future work.Source: §5.3 · p. 10, §5.4 · p. 11

6 Conclusion

CanvasAnneal uses partially observed teacher reasoning traces as training-time scaffolds, progressively increasing their masking while retaining fully masked prompt groups. Its results vary by task: it improves over diffu-GRPO on some reasoning and tool-use benchmarks, but not all.

6 Conclusion

  • CanvasAnneal progressively increases masking of teacher reasoning traces while retaining a fixed fraction of fully masked prompt groups during training.At inference, the teacher is removed and generation uses a fully masked canvas with the standard diffusion process.Source: §6 · p. 11
  • Across mathematical reasoning benchmarks, CanvasAnneal improves over diffu-GRPO on MATH500 and Countdown, but not GSM8K.Tool calling improves consistently on Tau2, while performance remains competitive on BFCL; overall effectiveness depends on task and training distribution.Source: §6 · p. 11

A Training and Implementation Details

The implementation uses eight NVIDIA A100 GPUs and reports hyperparameters for SFT, RL, and curriculum schedules. Evaluation also includes additional generation lengths and low-confidence remasking.

A Training and Implementation Details

  • Training uses 8 × NVIDIA A100 (80GB) GPUs, and Table 5 summarizes core SFT and RL hyperparameters.The table caption also identifies curriculum schedules among its hyperparameter categories.Source: A Training and Implementation Details · p. 15, Table 5 · p. 15
  • Evaluation adds generation lengths of 128 and 512 tokens, using 64 and 256 denoising steps, respectively.Generation uses low-confidence remasking: at each step, the model ranks predicted tokens by softmax probability and selectively unmasks the highest-confidence tokens.Source: A Training and Implementation Details · p. 15

A.1 Reward Functions

Candidate completions are evaluated against ground-truth answers using task-specific multi-component reward functions, with correctness and formatting terms varying by task.

A.1 Reward Functions

  • GSM8K reward combines 2.0·Rcorrect, 0.5·Rstrict_format, 0.5·Rsoft_format, 0.5·Rint_output, and Rxml_count.Rxml_count grants 0.125 for each correctly matched XML tag.Source: §A.1 · p. 15
  • MATH reward combines 2.0 · Rcorrect and 0.5 · Rformat, with formatting requiring both \boxed{} and answer tags.Source: §A.1 · p. 15
  • Countdown reward combines 1.0 · Rcorrect_eval and 0.1 · Rvalid_format; the expression must use each provided number exactly once.Source: §A.1 · p. 15
  • xLAM reward combines 2.0 · Rcorrect_json, 0.5 · Rstrict_format, and Rxml_count.Strict formatting requires valid <think> blocks and proper <tool_call> JSON syntax.Source: §A.1 · p. 15

B Quantitative Analysis of RL Training Dynamics

CanvasAnneal improves training reward and sample efficiency on several tasks, with the size and direction of gains varying by task. Reported dynamics show faster threshold attainment on xLAM and Countdown, earlier peak performance on MATH, and weaker GSM8K rewards.

B Quantitative Analysis of RL Training Dynamics

Loading 2609.13060v1…