Consolidating Reinforcement Learning for Multimodal Discrete Diffusion Models

1 INTRODUCTION

• Parallel token generation complicates coherent stochastic rollouts and importance estimation, while existing text and image methods offer only partial solutions.Prior masking heuristics can violate conditioning assumptions, and Monte Carlo estimators remain computationally expensive.Source: §1 · pp. 1–2
• MaskGRPO extends GRPO to multimodal discrete diffusion through modality-specific innovations in sampling and importance estimation.Its language estimator uses fading-out masking to focus on uncertain later tokens; its visual approach uses probabilistic decoding for diverse, high-quality rollouts.Source: §1 · p. 2
• MaskGRPO nearly doubles prior RL gains on mathematical reasoning and coding while improving text-image alignment and visual fidelity.The paper argues that effective policy optimization in discrete diffusion requires modality-aware samplers and estimators.Source: §1 · p. 2

2.1 DISCRETE DIFFUSION MODEL

• Discrete diffusion gradually corrupts tokens toward an absorbing state, and its linear-scheduler objective is an ELBO for the denoising log-likelihood.The text also assumes perfect reconstruction of clean data when deriving the sampling rule.Source: §2.1 · p. 2

2.2 GRPO FOR AUTOREGRESSIVE MODEL

• GRPO samples response groups from an old policy, computes group-relative advantages, and optimizes a clipped reward objective with KL regularization.The approach avoids explicit value-function modeling and reduces sensitivity to absolute reward scales.Source: §2.2 · pp. 2–3

2.3 ACCOMMODATING GRPO FOR DISCRETE DIFFUSION MODEL

• For discrete diffusion, progressively reversing a completion yields subsequence-level importance estimates, with re-masking strategies including uniform random and AR-like fading-out.The AR-like reversal treats the first k response tokens as a re-masked completion at a corresponding timestep.Source: §2.3 · p. 3, Figure 2 · p. 4
• The likelihood difference for newly unmasked tokens is approximated by full-sequence loss differences, enabling a calculable importance estimate.Prior DDM approaches either disrupt conditional dependencies or require costly Monte Carlo estimation.Source: §2.3 · p. 3

3.1 IMPORTANCE ESTIMATION

• Importance estimation targets informative token fluctuations rather than confident low-mask predictions, using AR-like fading-out for later text tokens and modality-aware reversing.The method also clamps the sampling range to exclude low mask ratios and uses low-discrepancy estimators for language and vision.Source: §3.1 · p. 4

3.2 ROLLOUT ADAPTION

• For text generation, the semi-autoregressive sampler uses block-wise decoding with low-confidence re-masking to reflect language’s sequential structure.The method describes this sampler as supporting generation quality.Source: §3.2 · p. 5
• For images, probabilistic token emergence avoids fixed per-prediction decoding quantities, while highly truncated reversing addresses strong global token correlations.The paper reports better texture and expressiveness for Emerge samples than comparison samples under identical sampling parameters.Source: §3.2 · p. 5, Figure 3 · p. 5

3.3 ALGORITHM FRAMEWORK

• MaskGRPO samples completions, generates masked variants for each completion, estimates reward-based advantages, and updates the policy using importance and KL terms.The framework uses the designed reverse function with controllable randomness to generate masked variants.Source: §3.3 · p. 6

4.1 SETUP

• Experiments optimized LLaDA-8B-Instruct and MMaDA-8B-Base across math, coding, and image generation, using 6,000 language-task steps and 1,500 image-generation steps.The evaluations included GSM8K, MATH500, MBPP, GenEval, DPG-Bench, and human-preference metrics.Source: §4.1 · pp. 6–7

4.2 PERFORMANCE AND COMPARISON RESULTS

• Over 5% absolute improvement in solution accuracy on GSM8K, MATH500, and MBPP nearly doubled prior methods’ RL gains with fewer training steps.MaskGRPO used 6,000 steps versus 7,000+ for prior methods.Source: Table 1 · p. 7, §4.2 · p. 7
• MaskGRPO achieved consistent human-preference improvements and improved text–image alignment; GenEval and DPG-Bench results showed discrete models can approach leading commercial systems.Qualitative comparisons report better artistic style, photographic detail, and overall atmosphere with RL under identical sampling parameters.Source: Table 2 · p. 7, §4.2 · p. 7, Table 3 · p. 8

4.3 DISCUSSION

• For GSM8K, both no timestep truncation and overly aggressive truncation reduced training stability; the authors chose γ = 0.6 to promote stable learning.The truncation ratios tested were 0.2, 0.4, 0.6, and 0.8.Source: §4.3 · p. 8, Figure 5 · p. 9
• In vision tasks, γ = 0.4 caused training failure, γ = 0.8 was selected, and relatively small clip ranges produced better training dynamics than language-task settings.The authors attribute modality-specific design needs to global correlations and higher entropy in visual-token prediction.Source: §4.3 · p. 8, Figure 5 · p. 9
• The proposed AR-like reversal consistently outperformed TraceRL, whose path-dependent formulation constrains exploration and biases sequence-level importance estimates.AR-like reversal balances autoregressive bias and randomness and can capture front fluctuations at smaller masking ratios.Source: §4.3 · pp. 8–9, Figure 6 · p. 9
• Before RL, the emerge sampler scored 0.51 vs. 0.56 for vanilla MaskGIT on GenEval; after RL, it scored 0.84 vs. 0.77.Before RL, emerge samples had better textures but sometimes unstable or deformed object boundaries; these instabilities were largely eliminated after RL.Source: §4.3 · p. 10

• For language, semi-autoregressive decoding can achieve state-of-the-art performance, despite scaled DDM training using fully random noising [Arriola et al. (2025); Nie et al. (2025)].Source: §5 · p. 10
• For visual diffusion, low-confidence re-masking can improve sample quality but may limit diversity and complicate reliable likelihood estimation [Chang et al. (2022); Zheng et al. (2025); Ma et al. (2025a)].Source: §5 · p. 10
• GRPO’s importance sampling is intractable for discrete diffusion because its likelihood is not factorized, limiting direct application of reward-based optimization [Zhu et al. (2025)].Source: §5 · p. 10

• MaskGRPO extends GRPO for discrete diffusion with fading-out masking for text and probabilistic decoding for images.Source: §6 · p. 10
• Experiments report improvements in reasoning accuracy, text-image alignment, and sample diversity.Source: §6 · p. 10
• The authors conclude that modality-specific samplers and estimators matter for policy optimization across multimodal discrete diffusion.Source: §6 · p. 10

A.1 ELBO ESTIMATION

• The intractable DDM log-likelihood is approximated from an evidenced lower bound, with LLaDA-1.5 providing a low-variance estimation.Source: §A.1 · p. 13
• The time-weighted term is implemented as an average over masked tokens, with t_j = j/ν and y_tj sampled from q(y_tj; y, t_j).Source: §A.1 · p. 13

A.2 IMPORTANCE ESTIMATION

• The appendix frames its importance-estimation discussion as a viable implementation shared by current RL works, not a strict proof.Source: §A.2 · p. 13
• For a partially masked completion, the importance term describes deviation between model predictions at masked positions; the appendix relates x to context and remaining masked tokens, and y to newly unmasked tokens.Source: §A.2 · p. 14
• Because the newly unmasked token set is small and inner-loop computation is limited, the appendix approximates at t_j = 1, when all those tokens are pending.Source: §A.2 · p. 14
• The resulting expression equates the prediction-loss difference on the partially masked completion with that on the full completion.Source: §A.2 · p. 14

B.1 REWARD FUNCTION

• Image rewards sum UnifiedReward, HPSv3, and CLIP Score; CLIP Score is retained to stabilize training and mitigate reward hacking.UnifiedReward scores alignment, HPSv3 assesses visual quality and alignment, and CLIP Score measures text-image feature similarity.Source: §B.1 · p. 14
• For GSM8K and MATH500, text rewards combine answer correctness and properly enclosed reasoning-format checks.Correct final answers receive 2; correctly formatted reasoning receives 0.5, and each reward is 0 otherwise.Source: §B.1 · p. 14
• For MBPP, the reward combines format compliance with correctness tested against predefined test cases.Source: §B.1 · p. 14

B.2 EVALUATION

• Language tasks use zero-shot prompting and report performance with pass@1; image generation decodes 1,024 tokens over 32 steps with classifier-free guidance 3.5.The image sequence represents a 512 × 512 image.Source: §B.2 · p. 15
• The MBPP protocol addresses result variance attributed to differing generation settings, prompts, and dataset subsets, which impede direct model comparisons.Source: §B.2 · p. 15
• MBPP evaluation uses the first 500 official-dataset samples, the lm-eval prompt format, zero-shot prompting, and gen_len/step/block_len = 256/256/32.Performance is reported as pass@1, the percentage of test cases passed on the first attempt.Source: §B.2 · p. 15

B.3 DATA USAGE

• Language training uses standard GSM8K and MATH500 sets and Acecode-87K for MBPP; the authors reimplemented diffu-GRPO and report its best MBPP performance on Acecode-87K.They note that diffu-GRPO reported using KodCodeLight-RL-10K, with multiple traversals inferred from its reported 7,500 steps.Source: §B.3 · p. 15
• Image-generation training uses roughly 150K prompts: 90K sampled from midjourney-prompts and 60K compositional GenEval-style prompts from Blip3-o, with one training epoch.Source: §B.3 · p. 15
• Optional SFT on Blip3-o instruction-tuning data precedes RL and is marked in the tables; the authors report improved prompt following and clearer spatial relationships and object boundaries.SFT runs for about 1,000 steps with global batch size 128 and learning rate 3e−6.Source: §B.3 · p. 15

B.4 CLARIFICATION

• Language experiments use LLaDA-8B-Instruct and image generation uses MMaDA-8B-Base; the authors do not pursue language experiments with MMaDA because its reasoning-enabled MixCoT checkpoint performs poorly and is not comparable to prior LLaDA-based work.MMaDA-8B-MixCoT reaches 48% GSM8K accuracy, about 30% below LLaDA-8B-Instruct under the same sampling parameters; MaskGRPO improves its result by over 6%, but the authors say the comparison is not comparable.Source: §B.4 · p. 15

C MORE RESULTS

• The section also provides additional math-reasoning and coding examples alongside its generated portraits.Source: C MORE RESULTS · p. 16
• Table 4 covers DPG-Bench, with SFT marking models tuned on BLIP3-o data before RL; the supplied caption gives no table values.Source: Table 4 · p. 16, C MORE RESULTS · p. 16
• Figure 7 presents portrait samples for identical prompts and sampling parameters, contrasting the default MaskGIT-style sampler before RL with the emerge sampler after RL.The settings are CFG=3.5 and 64 steps.Source: Figure 7 · p. 16