Source-linked AI summary
Qwen-Image-2.0-RL Technical Report
Yixian Xu, Kaiyuan Gao, Yuxiang Chen, Yilei Chen, Zecheng Tang, Zihao Liu, Zikai Zhou, Deqing Li, Hao Meng, Kuan Cao, Jiahao Li, Jie Zhang, Liang Peng, Lihan Jiang, Ningyuan Tang, Shengming Yin, Tianhe Wu, Xiaoyue Chen, Yan Shu, Yanran Zhang, Yi Wang, Yu Wu, Yujia Wu, Zekai Zhang, Zhendong Wang, Xiao Xu, Kun Yan, Chenfei Wu
TL;DR
Supervised diffusion training does not directly capture human preferences for visual quality and prompt faithfulness. Qwen-Image-2.0-RL addresses this with RLHF, task-specific rewards, GRPO training, and on-policy distillation, achieving 57.84 on Qwen-Image-Bench and higher arena Elo ratings for text-to-image and editing.
Problem
Supervised diffusion objectives do not directly capture human preferences for compositional harmony, texture, prompt faithfulness, and stylistic coherence.
Method
The pipeline combines VLM-based composite rewards, GRPO-based reinforcement learning, hybrid CFG, and on-policy distillation to unify task-specialized image-generation and editing policies.
Results
57.84 overall score on Qwen-Image-Bench (+2.61 over the base model), with Elo ratings of 1193 in text-to-image and 1349 in image editing.
Takeaways & Limitations
Qwen-Image-2.0-RL shows consistent gains in aesthetic quality, prompt adherence, and editing accuracy across text-to-image and image-editing tasks.
Abstract
from arXiv · showhide
We present Qwen-Image-2.0-RL, a post-training pipeline that applies reinforcement learning from human feedback (RLHF) and on-policy distillation (OPD) to improve both the visual quality and instruction-following capability of the Qwen-Image-2.0 diffusion model. To provide reliable reward signals, we construct task-specific composite reward models by fine-tuning vision-language models with a pointwise scoring paradigm and chain-of-thought reasoning. For text-to-image generation, the reward models cover alignment, aesthetics, and portrait fidelity dimensions. For image editing tasks, the reward system addresses instruction-following accuracy and face identity preservation. Building on this reward system, we develop a scalable GRPO-based RL training framework, incorporating a hybrid classifier-free guidance (CFG) strategy to preserve pre-trained knowledge, prompt curation via intra-group reward range filtering, and per-category reward weight calibration. To merge the task-specialized RL policies for T2I and editing, we propose on-policy distillation as the final training stage, which consolidates multiple teachers into a single student model through trajectory-level velocity matching. Extensive evaluation shows that Qwen-Image-2.0-RL achieves 57.84 overall score on Qwen-Image-Bench (+2.61 over the base model), Elo ratings of 1193 in text-to-image arena (+78) and 1349 in image edit arena (+93), demonstrating consistent gains in aesthetic quality, prompt adherence, and editing accuracy.
1 Introduction
Qwen-Image-2.0-RL addresses the gap between supervised diffusion outputs and human aesthetic expectations through task-aware rewards, scalable GRPO training, and on-policy distillation. The resulting model improves benchmark, text-to-image, and image-editing performance.
- Motivation: Supervised diffusion training leaves a gap from human preferences because denoising score matching does not directly capture composition, texture, prompt faithfulness, and stylistic coherence.RLHF is presented as a way to improve alignment with these human aesthetic dimensions.
- Challenges: Reliable diffusion-model RL requires composite, task-aware rewards spanning T2I aesthetics and prompt adherence alongside editing instruction accuracy and identity preservation.The introduction identifies diverse quality dimensions across fundamentally different tasks as a central challenge.
- Contributions: The pipeline fine-tunes Qwen VLMs with chain-of-thought pointwise scoring to build layered T2I rewards and combined editing rewards.T2I rewards target alignment, aesthetics, and portraits; editing rewards address instruction accuracy and identity preservation.
- Contributions: The GRPO-based framework uses multi-reward advantages, hybrid CFG, prompt filtering, and per-category reward weighting to balance training stability, knowledge preservation, and optimization.Hybrid CFG guides rollout sampling but is excluded from the policy objective.
- Contributions: On-policy distillation merges task-specialized T2I and editing teachers into one student through trajectory-level velocity matching, avoiding cross-task conflicts and reward-model dependence.This consolidation addresses deployment requirements for a single model without sacrificing per-task quality.
- Results: 57.84 overall score on Qwen-Image-Bench (+2.61 over the base model), 1193 Elo in text-to-image arena (+78), and 1349 Elo in image edit arena (+93) demonstrate gains across tasks.The reported improvements cover aesthetic quality, prompt adherence, and editing accuracy.
2 Backgrounds
Diffusion and flow-matching models generate images by learning reverse dynamics from progressively noised data. Recent methods adapt reinforcement learning to these models by representing denoising as an MDP or optimizing with forward-process velocity predictions while constraining policy drift.
- Diffusion and Flow Matching: Diffusion models learn reverse dynamics that recover data from progressively noised samples, following the flow-matching framework.The forward process corrupts clean data with Gaussian noise, while a neural network approximates the conditional velocity field.
- Diffusion and Flow Matching: The flow-matching objective minimizes squared error between predicted and target velocities over timesteps, data samples, and Gaussian noise.The target velocity is ϵ − x0, and the loss is denoted LFM(θ).
- Reinforcement Learning for Flow Matching: Flow-GRPO formulates multi-step denoising as an MDP, evaluates grouped samples with a reward model, and normalizes advantages using group reward statistics.Because deterministic ODE sampling prevents direct GRPO application, Flow-GRPO introduces an equivalent stochastic sampler whose discretized transitions have tractable Gaussian densities.
- Reinforcement Learning for Flow Matching: DiffusionNFT optimizes policy behavior from the forward diffusion process using current, old, and reference velocity predictions.It constructs positive and negative velocity predictions and uses a clipped group-relative advantage with a KL penalty to limit deviation from the pretrained reference.
3 Reward Modeling
This section presents task-specific composite reward models for T2I generation and image editing, combining VLM-based assessment with model-based identity scoring. It establishes pointwise reward training as the default paradigm because it yields better visual quality and fewer artifacts than pairwise training.
- Reward System: Task-specific composite rewards combine VLM-based semantic and aesthetic scorers with model-based fine-grained identity preservation for T2I and editing.The system is tailored to different evaluation dimensions across both task types.
- Training Paradigm Comparison: Pointwise reward training produces consistently better visual quality, finer texture detail, and fewer artifacts than pairwise training under matched evaluation dimensions and model pools.The comparison isolates annotation format and attributes the improvement to absolute, calibrated scores rather than relative preferences.
- T2I Reward Models: T2I rewards cover image-text alignment, aesthetics, and portrait fidelity, progressing from prompt adherence to visual quality and human-subject correctness.Portrait scoring includes facial proportions, identity-preserving details, texture realism, and common anatomical failure modes.
- Image Editing Reward Models: Editing rewards assess instruction-following for modifications such as replacement, attribute changes, and style transfer, while a dedicated model-based scorer detects subtle facial identity shifts.The identity scorer supplies embedding-level preservation signals complementary to the VLM’s global semantic consistency assessment.
4 Training
The training pipeline separately optimizes T2I and editing policies with task-specific rewards, then merges them through on-policy distillation. It combines hybrid CFG, reward normalization, timestep selection, prompt curation, and category-specific calibration to stabilize and target RL optimization.
- Training pipeline: Separate RL policies are trained for T2I generation and image editing, then merged into one deployable model through on-policy distillation.The policies use their respective reward compositions before consolidation.
- Hybrid CFG strategy: CFG in rollout only is adopted because CFG-guided sampling preserves coherent, high-quality candidates while CFG-free optimization stabilizes gradients and reduces computational overhead.Using CFG in both stages causes collapse, whereas omitting CFG entirely degrades stylization, world knowledge, celebrity likeness, and style-specific generation.
- Reward computation: Reward advantages combine multiple models through weighted, per-prompt-group normalization, making composite rewards invariant to absolute scale differences across reward dimensions.The reward weights satisfy ∑K k=1 wk = 1, and normalization prevents any single dimension from dominating because of its numerical range.
- Prompt and reward calibration: Prompts are retained only when their intra-group composite-reward range exceeds a threshold, and category-specific reward weights tailor optimization to domains such as portraits and typography.Portrait prompts emphasize portrait fidelity, whereas typography prompts emphasize alignment accuracy.
- On-Policy Distillation: On-policy distillation unifies task-specialized teachers into one student through trajectory-level velocity matching, addressing cross-task performance degradation from separate RL policies.The approach maintains distinct T2I and editing teachers and selects the appropriate teacher according to task type.
5 Evaluation
Qwen-Image-2.0-RL is evaluated using automated quality metrics on standardized benchmarks and human preference rankings in text-to-image and image-edit arenas. RL training improves Elo ratings across all evaluated dimensions, including the overall T2I rating.
- Evaluation setup: The evaluation combines automated quality metrics on standardized benchmarks with human preference rankings in competitive arenas.The study evaluates both benchmark-based quality and pairwise user preferences between anonymized outputs.
- Text-to-image generation results: Qwen-Image-Bench assesses text-to-image generation across Quality, Aesthetics, Alignment, Real-world Fidelity, and Creative Generation using Q-Judger.Q-Judger was trained on over 130K human-labeled image-prompt pairs annotated by 80 professional artists.
- Human preference evaluation: 1115 to 1193 (+78) marks the overall T2I Elo improvement after RL training, with gains reported across all dimensions.The strongest listed gain is 3D Modeling (+93).
6 Related Works
The work builds on reinforcement-learning methods for diffusion and flow-matching models, reward-modeling approaches for preference alignment, and on-policy distillation for consolidating heterogeneous capabilities. It specifically introduces a hybrid CFG strategy for the Qwen-Image-2.0 architecture.
- Reinforcement learning for diffusion models: Diffusion RL methods extend GRPO or use forward-process formulations to align generation with human preferences, while addressing optimization challenges in flow matching.Flow-GRPO treats multi-step generation as a Markov decision process, GRPO-Guard uses regulated clipping, and AWM and DiffusionNFT introduce online RL with forward-process formulations.
- Reinforcement learning for diffusion models: The paper builds on prior diffusion-RL methods by introducing a hybrid CFG strategy specifically designed for Qwen-Image-2.0.The cited passage identifies hybrid CFG as the work’s architecture-specific contribution within this line of research.
- Reward models for diffusion models: Reward modeling has progressed from limited CLIP-based approaches to scalar predictors trained with Bradley–Terry ranking loss, including ImageReward, PickScore, HPSv2, and HPSv3.The passage notes that early CLIP-based methods have poor cross-task generalization and fixed architectures.
- On-policy distillation: On-policy distillation trains students on self-generated trajectories under teacher supervision to consolidate heterogeneous capabilities and avoid the seesaw effect of multi-reward RL.GKD established the framework for LLMs, while Flow-OPD and DiffusionOPD extended OPD to flow-matching models.
7 Conclusion
Qwen-Image-2.0-RL combines RLHF with OPD to improve visual quality and instruction-following. Its contributions include task-tailored composite rewards, adapted GRPO training, and qualitative gains from RL and OPD across generation and editing.
- Qwen-Image-2.0-RL combines RLHF with OPD to improve visual quality and instruction-following capabilities.
- Its VLM-based composite reward system covers T2I and TI2I using rubrics for aesthetics, prompt adherence, portrait fidelity, instruction-following, and visual consistency.
- An adapted GRPO training framework is another stated contribution of the Qwen-Image-2.0-RL approach.
- RL improves visual quality over the pre-trained baseline, while OPD further surpasses mixed RL by avoiding cross-task interference in T2I generation.
- Qwen-Image-2.0-RL achieves the best face identity preservation and instruction-following accuracy in portrait editing, whereas Mix-RL shows identity drift or incomplete edits.
A Derivation of the On-Policy Distillation Objective
The section derives the on-policy distillation loss by minimizing an efficiently computable upper bound on the 2-Wasserstein distance between student and teacher output distributions. Under smooth-flow and teacher-velocity regularity assumptions, this bound reduces to trajectory-level velocity matching, whose discrete approximation recovers the practical OPD objective.
- Distributional metric: The derivation uses the 2-Wasserstein distance because diffusion output distributions are implicit through ODE solves, making path-measure KL divergence generally intractable.Direct W2 computation is also intractable in high dimensions, so the method minimizes an efficiently computable upper bound.
- Assumptions: The W2 bound relies on unique continuously differentiable student and teacher flows and Lipschitz regularity of the teacher’s velocity field.These are stated as assumptions (A1) and (A2) for applying the flow-distance result.
- Continuous-time objective: Minimizing the W2 upper bound with respect to the student parameters reduces to a trajectory-level velocity matching objective.The exponential factor depends on the teacher velocity field’s Lipschitz regularity and not on the training objective.
- Discrete implementation: Discretizing the student ODE over N timesteps and summing velocity-matching terms along the student trajectory yields the practical OPD loss in Eqn. (14).The trajectory points provide samples from the student distributions at the corresponding timesteps.