Reinforcement Learning-driven Diffusion Models: Language, Vision, and Beyond

• Conventional diffusion models struggle with complex reasoning, autonomous exploration of generation pathways, and alignment with human preferences or task objectives.Their denoising is tied to static training distributions and reconstruction-centric MSE loss.Source: §1 · p. 1
• RLHF and DPO fine-tune pretrained diffusion models toward complex instructions and human preferences, while RLVR targets specialized robot and science domains.The passage also notes GRPO's use for long-form reasoning and efficiency.Source: §1 · p. 2
• RL post-training shifts diffusion toward experience-driven interaction, closed-loop planning, and reward maximization rather than static imitation and MSE minimization.The survey describes multi-trajectory inference scaling and policy exploration based on cumulative reward estimates.Source: §1 · p. 2
• The survey introduces standard and RL-driven diffusion principles to establish their Markovian theoretical commonality.Source: §1 · p. 2
• For diffusion-based LLMs, the survey covers masking and unmasking, the ELBO objective, and RL methods organized by trajectory level and reward density.Source: §1 · p. 2
• The multimodal review examines RL technology development and RL-related tasks as complementary perspectives.Source: §1 · p. 2
• The robotics review covers foundational principles, diffusion-policy RL training, and generative experts in Vision-Language-Action models.Source: §1 · p. 2
• The scientific-model review organizes work by modality, including molecular sequences, 3D structures, crystals, physics-constrained fields, and scientific imaging.The survey also points toward future research on robust, interpretable RL-diffusion systems.Source: §1 · p. 2

2.1.1 Discrete-Time Paradigm: DDPMs

• DDPMs gradually transform data into Gaussian noise and learn reverse denoising transitions, with training simplified to a noise-prediction mean-square-error objective.Latent diffusion applies the process in compressed latent space to improve efficiency.Source: §2.1.1 · p. 2

2.1.2 Continuous Generalization: Score and Flow Matching

• Score matching and flow matching generalize diffusion as continuous-time dynamics, while rectified flow linearly interpolates data and noise and regresses their velocity difference.The noise-prediction loss is equivalent to estimating the score function.Source: §2.1.2 · p. 3

2.2.1 Why Do Diffusion Models Require RL?

• RL directly optimizes rewards, addressing limits of reward-weighted finetuning and rejection sampling while accommodating non-differentiable or sparse objectives.Reward-weighted finetuning struggles with out-of-distribution solutions and typically needs paired conditional data; rejection sampling depends on high-reward outputs already being probable.Source: §2.2.1 · p. 3

2.2.2 Preference-Based RL Optimization

• Preference-based methods use human rankings as proxy rewards: RLHF trains a reward model from preference triples, whereas DPO optimizes directly from pairwise preferences.The described RLHF process combines supervised pretraining, reward-model training, and preference-alignment optimization.Source: §2.2.2 · p. 3

2.2.3 Policy-Based RL Optimization

• In policy-based RLVR, each denoising step is an action, the trajectory is a decision sequence, and terminal rewards come from verifiable criteria.The denoising distribution is the policy and the Gaussian prior is the initial-state distribution.Source: §2.2.3 · p. 3, Figure 3 · p. 3
• PPO constrains update magnitudes with a clipped surrogate objective, while GAE estimates advantages and provides a tunable bias–variance trade-off.The return is defined as the advantage plus the value estimate.Source: §2.2.3 · pp. 3–4
• GRPO removes the separate value network and uses group-relative advantages across outputs, with a clipped surrogate objective and optional KL regularization.The section characterizes GRPO as a lightweight, stable alternative to PPO for generative RL.Source: §2.2.3 · p. 4

3.1 Preliminaries of Diffusion-based LLMs

• dLLMs generate text by iteratively unmasking a fully masked sequence, and train the reverse process by maximizing an ELBO aligned with forward masking.The example training pipeline includes pretraining, instruction-response SFT, and sampling from masked to unmasked text.Source: §3.1 · p. 4

3.2 The Role of RL in Diffusion-based LLMs

• RL4dLLM methods are classified along trajectory level, which sets credit-assignment granularity, and reward density, which describes supervisory-signal distribution.The review distinguishes terminal outcome rewards from stepwise process rewards, including dense intermediate supervision.Source: §3.2 · pp. 4–5

3.2.1 GRPO-Related Post Training

• d1 [22] estimates per-token likelihood by unmasking one timestep per iteration, while later methods reduce variance, improve efficiency, or inject reasoning hints.DiffuCoder [30] averages likelihoods from paired complementary masks; wd1 [26] uses a double-polar objective, and IGPO [41] inserts partial ground-truth reasoning traces into sampling.Source: §3.2.1 · p. 6

3.2.2 Beyond Standard Token-level GRPO

• Beyond token-level GRPO, methods use sequence-level likelihood proxies, direct denoising-logit optimization, stabilization, or self-distillation to adapt RL to dLLMs.ESPO [34] treats a generated sequence as one action and uses ELBO as its likelihood proxy; StableDRL [37] uses unconditional clipping and self-normalization.Source: §3.2.2 · p. 6

3.2.3 Aligning with Human Feedback

• For human preference alignment, VRPO [47] reduces gradient-estimation variance, while ELBO-KTO [48] uses unpaired feedback to lessen reliance on costly preference pairs.VRPO shares timesteps and masks across policies, allocates the full budget to timesteps, and increases gradient-estimation samples.Source: §3.2.3 · p. 7

3.2.4 Advancing Chain-of-Thought Reasoning Abilities

• CoT methods treat unmasking as latent reasoning: SAPO [50] rewards incremental progress, while DMPO [32] enhances CoT through distribution matching.DCoLT [20] uses outcome-based CoT optimization and top-K ranking to predict unmasking positions at inference.Source: §3.2.4 · p. 7

3.2.5 RL-Adapted Frameworks

• RL-adapted frameworks use intermediate traces, likelihood bounds, or pseudo-rewards to stabilize or guide dLLM post-training.TraceRL [51] derives rewards from intermediate traces and aggregates neighboring steps; RFG [52] uses log-likelihood ratios to bias trajectories without additional training.Source: §3.2.5 · p. 7

4 RL FOR DIFFUSION MULTIMODAL MODELS

• RL in dMMs uses human feedback, aesthetic evaluators, or pretrained vision-language models to guide visual quality, semantic consistency, and cross-modal relevance.The review considers both RL technology development and RL-related multimodal tasks.Source: §4 · p. 7
• The reviewed objectives extend beyond standard SFT to include complex and subjective goals for multimodal generation.These goals include visual quality, semantic consistency, and cross-modal relevance.Source: §4 · p. 7
• RLHF and DPO are described as predominant human-alignment methods, while RLVR is presented as an emerging trend for complex tasks and long-horizon reasoning.The figure caption characterizes RLVR as significantly enhancing dMM capacity in those areas.Source: Figure 6 · p. 7

4.1 Underlying RL Technology Development

• The three approaches address different signal types: learned rewards, direct preference comparisons, and rule-based verifiable rewards.The review presents them as complementary categories for aligning generation with human preferences and task-specific goals.Source: §4.1 · p. 7

4.1.1 Reward-based RLHF

• RLHF frames preference alignment as reward optimization using feedback, learned reward models, and RL fine-tuning, but static rewards can limit generalization and induce reward hacking or instability.The review describes policy-gradient fine-tuning, denoising-trajectory MDPs, and direct reward-driven gradient propagation as implementation approaches.Source: §4.1.1 · pp. 7–8

4.1.2 Reward-free DPO

• DPO directly optimizes diffusion models using preferred and dispreferred sample pairs, avoiding explicit reward estimation and improving stability and sample efficiency.The review notes that reliance on static preference data limits adaptability.Source: §4.1.2 · p. 8

4.1.3 Rule-based RLVR

• RLVR adds verifiable, dynamic rewards for task-driven optimization, with approaches based on stochastic reparameterization, randomized policy rollouts, or implicit policy improvement.Inference-optimization frameworks and adaptive sampling control also use verifiable feedback without full retraining.Source: §4.1.3 · p. 8

4.2 RL-driven Diffusion Multimodal Tasks

• RL-driven multimodal methods address generation, editing, and unified understanding-generation tasks across image, video, 3D, and audio.Source: §4.2 · p. 8

4.2.1 RL-driven Multimodal Generation

• Image-generation methods align global aesthetic rewards, add process-level control for compositional reasoning, and optimize denoising trajectories for faster inference.The section groups methods by aesthetic alignment, compositional reasoning, and inference acceleration.Source: §4.2.1 · pp. 8–9
• Flow-GRPO improves global aesthetics and compositionality, while text rendering shows only moderate gains over base models.The cited figure identifies text rendering as a task where outcome-based rewards alone provide limited improvement.Source: Figure 8 · p. 9
• Video alignment methods optimize motion smoothness, flicker reduction, and subject consistency to improve human-preference alignment, while physics-aware RL rewards guide generated dynamics using trajectories, velocities, and conservation or contact consistency.Geometry-aware rewards support 3D/4D world consistency through camera smoothness and reprojection consistency. For joint audio-video generation, OmniNFT uses modality- and region-aware feedback.Source: §4.2.1 · p. 10
• For 3D generation, preference rewards and feedback-driven optimization target geometric consistency, while reward-gradient and score matching connect 3D generators to 2D rewards.Source: §4.2.1 · p. 11
• For audio and motion, RL methods use preference feedback to align generated content and physics-based rewards to promote physically valid motion.Audio approaches target semantic alignment, lyric intelligibility, prompt consistency, and musicality; motion approaches balance text adherence and physical validity.Source: §4.2.1 · p. 11

4.2.2 RL-driven Image Editing

• In image editing, RL methods optimize editing control, improve reward alignment with user intent, and preserve image structure during semantic edits.Recent approaches also address multi-turn revisions and scarce editing-specific rewards.Source: §4.2.2 · p. 11

4.2.3 RL in Unified Understanding and Generation

• Unified multimodal systems use RL to optimize comprehension and generation together, including reasoning-conditioned generation and self-reflective refinement.Other work extends unified frameworks to model assessment and applications such as autonomous driving.Source: §4.2.3 · p. 12

5 RL FOR DIFFUSION ROBOTICS

• Robotics research integrates RL with diffusion through policy training, where DMs serve as policies optimized by RL gradients, and VLA models, where diffusion generates actions from multimodal perception.Source: §5 · p. 12

5.1 Foundations of RL for Diffusion in Robotics

• Diffusion policies model state-conditioned action distributions through iterative denoising, avoiding predefined action-distribution forms and supporting multimodal continuous actions.The reverse process predicts noise from a noisy action, environmental state, and denoising step.Source: §5.1.1 · p. 13

5.2 Diffusion Policy-based RL Training

• Diffusion-policy RL spans online and offline settings, with online methods using environmental feedback and offline methods avoiding the cost and safety risks of real-time interaction.Offline approaches must also limit policy drift beyond the support of the collected data.Source: §5.2 · pp. 14–15
• Online methods combine policy-gradient and denoising-structure optimization, with additional approaches using preference feedback, imitation learning, and residual policies.Multi-Loco is reported to narrow the sim2real gap in online robotic control tasks.Source: §5.2 · p. 14
• Offline diffusion RL models action distributions and trajectories from collected datasets, with methods addressing data support, offline fine-tuning, risk sensitivity, and discrete-continuous actions.Source: §5.2 · p. 15

5.3 Generative Experts in VLA Models

• VLA systems use diffusion or flow matching as action experts to support diverse trajectories, while RL offers an alternative to imitation learning for improving policy scalability.The section reviews RL methods for VLA control, including world-model-based and value-gradient approaches.Source: §5.3 · p. 16

6 RL FOR DIFFUSION SCIENCE MODELS

• Scientific diffusion models use reward signals to control outputs toward physical laws or function-specific objectives.Source: §6 · p. 16

6.1 Discrete Molecular Sequence Generation

• For biomolecular sequences, RL steers discrete diffusion toward structural and functional objectives, including protein foldability, DNA fitness, and antibody binding.Methods also use stepwise preference learning, KL regularization, and value-weighted likelihood distillation.Source: §6.1 · p. 16
• Recent sequence-design methods combine function, structural validity, and sequence naturalness rather than maximizing a single proxy reward.DoMinO limits drift from pretrained generators; RIDER uses self-consistency rewards to align RNA sequences with target structures.Source: §6.1 · p. 17

6.2 Continuous 3D Structure Generation

• For continuous 3D molecular design, RL uses physics- and property-based rewards to optimize conformations and compounds across binding, drug-likeness, synthesizability, and diversity.Uncertainty-aware reward shaping is used to balance conflicting objectives and avoid high-risk chemical spaces.Source: §6.2 · p. 17

6.3 Periodic Crystal Structure Generation

• Crystal-generation methods use formation-energy rewards and KL regularization to target stable structures while preserving crystallographic symmetry.CrystalGRPO highlights a conflict between Top-1 quality and Top-K candidate coverage for polymorph discovery.Source: §6.3 · p. 17

6.4 Continuous Physics-Constrained Field Modeling

• Physics-constrained field methods align generated fields with PDE residuals, boundary conditions, or differentiable weak-form constraints.The section distinguishes on-policy reward optimization, off-policy posterior matching, and direct differentiable constraint optimization.Source: §6.4 · p. 17

6.5 Scientific Imaging and Inverse Reconstruction

• Scientific imaging covers both reconstruction and measurement-acquisition policies, while other RL methods directly fine-tune medical diffusion generators.Cold Diffusion is a non-RL reconstruction baseline; ASMR learns an RL policy for selecting informative k-space measurements.Source: §6.5 · p. 17

7 CONCLUSION

• The survey maps RL-driven diffusion models across diffusion-based LLMs, multimodal frameworks, robotic systems, and scientific modeling.It centers on the post-training era.Source: §7 · p. 17
• It emphasizes test-time scaling, experience-driven adaptation, autonomous exploration, and reward mechanism design as pivotal dimensions of RL integration.The survey aims to synthesize principles, identify underexplored frontiers, and propose pathways for future work.Source: §7 · p. 17
• The survey may miss recent breakthroughs, so it proposes a crowdsourced community platform for real-time updates.Source: §7 · p. 17