PaperlayerView exact revision on arXiv

Source-linked AI summary

Pref-GRPO: Pairwise Preference Reward-based GRPO for Stable Text-to-Image Reinforcement Learning

Yibin Wang, Zhimin Li, Yuhang Zang, Yujie Zhou, Jiazi Bu, Chunyu Wang, Qinglin Lu, Cheng Jin, Jiaqi Wang

arXiv:2508.20751v2cs.CV

TL;DR

Pointwise-reward GRPO and coarse T2I benchmarks limit stable optimization and fine-grained diagnosis. The paper introduces PREF-GRPO for pairwise preference fitting and UNIGENBENCH for detailed semantic evaluation, reporting stronger semantic alignment without sacrificing image quality.

  • Problem

    Pointwise reward normalization can create illusory advantages and reward hacking, while existing T2I benchmarks lack fine-grained semantic evaluation.

  • Method

    PREF-GRPO compares generated images pairwise and uses win rates instead of absolute scores, while UNIGENBENCH evaluates fine-grained testpoints across themes and dimensions.

  • Results

    PREF-GRPO improves UNIGENBENCH overall by 5.84%, with gains of 12.69% on Text and 12.04% on Logical Reasoning, while achieving the best score on four image-quality metrics.

  • Takeaways & Limitations

    UNIGENBENCH exposes remaining weaknesses in higher-order semantic composition, while PREF-GRPO provides more stable training and stronger semantic consistency across in-domain and out-of-domain benchmarks.

Abstract

from arXiv · show

Recent advancements highlight the importance of GRPO-based reinforcement learning methods and benchmarking in enhancing text-to-image (T2I) generation. However, current methods using pointwise reward models (RM) for scoring generated images are susceptible to reward hacking. We reveal that this happens when minimal score differences between images are amplified after normalization, creating illusory advantages that drive the model to over-optimize for trivial gains, ultimately destabilizing the image generation process. To address this, we propose Pref-GRPO, a pairwise preference reward-based GRPO method that shifts the optimization objective from score maximization to preference fitting, ensuring more stable training. In Pref-GRPO, images are pairwise compared within each group using preference RM, and the win rate is used as the reward signal. Extensive experiments demonstrate that PREF-GRPO differentiates subtle image quality differences, providing more stable advantages and mitigating reward hacking. Additionally, existing T2I benchmarks are limited by coarse evaluation criteria, hindering comprehensive model assessment. To solve this, we introduce UniGenBench, a unified T2I benchmark comprising 600 prompts across 5 main themes and 20 subthemes. It evaluates semantic consistency through 10 primary and 27 sub-criteria, leveraging MLLM for benchmark construction and evaluation. Our benchmarks uncover the strengths and weaknesses of both open and closed-source T2I models and validate the effectiveness of Pref-GRPO.

I. INTRODUCTION

The paper identifies unstable pointwise-reward GRPO optimization and coarse T2I evaluation as key limitations, then proposes PREF-GRPO and UNIGENBENCH to address them.

  • Motivation: Pointwise reward maximization can produce reward hacking, with rising reward scores alongside deteriorating image quality during prolonged training.The paper attributes this to tightly clustered within-group scores whose normalization amplifies small gaps.
  • UNIGENBENCH: UNIGENBENCH evaluates 600 prompts across 5 primary themes, 20 sub-themes, 10 primary dimensions, and 27 sub-dimensions.Its testpoints support fine-grained assessment rather than only aggregate primary-dimension scores.
  • PREF-GRPO: PREF-GRPO replaces absolute reward maximization with pairwise preference fitting using image win rates as the optimization reward.A pairwise preference model compares images within each generated group.
  • Results: PREF-GRPO improves semantic alignment by 5.84% overall on UNIGENBENCH, including gains of 12.69% on Text and 12.04% on Logical Reasoning.The reported improvements occur without sacrificing perceptual quality.
  • Results: UNIGENBENCH shows that Style and World Knowledge are relatively strong across models, while fine-grained compositional capabilities remain bottlenecks.The cited bottlenecks include Logical Reasoning, Grammar, and Compound.

III. MOTIVATION FOR PAIRWISE PREFERENCE REWARD

The paper explains reward hacking as an objective mismatch caused by amplified pointwise-score differences and motivates pairwise preference fitting as a more stable alternative.

  • Illusory Advantage: Pointwise reward models compress scores for visually similar images, so normalization turns negligible gaps into disproportionately large advantages.This makes update directions depend excessively on small reward differences.
  • Objective Mismatch: Absolute reward maximization encourages policies to chase reward-model bias rather than genuine text-image alignment or visual quality.Observed artifacts include oversaturated images under HPSv2 and an unnaturally dark style under UnifiedReward.
  • Pairwise Preference: Pairwise comparison is motivated by its greater reliability for close-quality images than independently assigning calibrated absolute scores.It focuses optimization on relative ordering rather than score magnitude.
  • Method Setup: The section frames PREF-GRPO as a reformulation of the GRPO objective from absolute score maximization to pairwise preference fitting.The surrounding technical development introduces GRPO for flow-matching models before presenting the new objective.

B. Illusory Advantage in Pointwise-Reward GRPO

The illusory advantage arises when near-identical pointwise rewards are divided by a very small within-group standard deviation, magnifying noise and destabilizing optimization.

  • Cause: Pointwise reward models often assign tightly clustered scores to images generated from the same prompt.The resulting within-group standard deviation is small.
  • Mechanism: The gradient norm contains an explicit factor of 1/σr, exposing why small reward variance amplifies updates.The derivation isolates reward normalization while dropping clipping and KL terms.
  • Mechanism: Small score gaps become disproportionately large normalized advantages when divided by a small group standard deviation.The effect makes normalization highly sensitive to small perturbations.
  • Consequences: Amplification causes excessive optimization toward spurious reward differences and increases sensitivity to reward-model noise and bias.The paper names this failure mode the illusory advantage phenomenon.

C. Pairwise Preference Reward-based GRPO

PREF-GRPO replaces absolute reward-score maximization with pairwise preference fitting, using image win rates as rewards to produce more stable policy-optimization signals and reduce reward hacking.

  • C. Pairwise Preference Reward-based GRPO: PREF-GRPO uses a Pairwise Preference Reward Model to compare generated images pairwise and reformulate GRPO as preference fitting.For each ordered image pair, the model determines which image is preferred rather than relying on absolute reward scores.
  • C. Pairwise Preference Reward-based GRPO: Each image’s win rate across pairwise comparisons replaces the scalar reward in the GRPO advantage.The win rate is the proportion of comparisons in which an image is preferred, and it is inserted into the advantage computation.
  • C. Pairwise Preference Reward-based GRPO: Win rates spread across [0, 1], increasing within-group reward variance and making high- and low-quality samples more discriminable.High-quality samples approach 1 while lower-quality samples approach 0, yielding a more robust reward distribution for advantage estimation.
  • C. Pairwise Preference Reward-based GRPO: Relative rankings reduce sensitivity to small reward fluctuations and biases compared with raw score optimization.The preference signal is intended to capture nuanced quality differences while reducing susceptibility to reward hacking.

V. UNIGENBENCH

UNIGENBENCH is a fine-grained T2I benchmark designed to diagnose semantic consistency across diverse creative scenarios, dimensions, and explicit testpoints rather than relying on coarse aggregate scores.

  • Motivation and scope: UNIGENBENCH addresses sparse sub-dimension coverage and coarse reporting in existing T2I benchmarks.Its purpose is to reveal where models’ semantic understanding breaks down through finer-grained assessment.
  • Evaluation dimensions: The benchmark covers 10 primary dimensions and 27 sub-dimensions for substantially finer-grained semantic diagnosis.The dimensions include capabilities such as logical reasoning, facial expressions, pronoun reference, hand actions, and inter-entity relationships.
  • Prompt themes: It spans 5 major prompt themes and 20 sub-themes, covering practical generation scenarios and open-ended creative settings.The themes are Art, Illustration, Creative Divergence, Design, and Film&Storytelling, alongside diverse subject categories.
  • Assessment: Structured testpoint descriptions enable precise, interpretable assessment of whether intended semantic targets are satisfied.The benchmark’s prompt themes and evaluation dimensions are illustrated through representative examples.
  • Evaluation dimensions: The benchmark decomposes attributes, actions, relationships, and layout into specific testable aspects rather than reporting only dimension-level averages.Examples include quantity, material, hand actions, composition, inclusion, 2D placement, and 3D spatial arrangement.

B. Benchmark Construction and Evaluation Pipeline

UNIGENBENCH combines controlled sampling of themes, subjects, and evaluation dimensions with an MLLM pipeline for systematic prompt generation and fine-grained assessment.

  • Benchmark Construction and Evaluation Pipeline: The pipeline uses MLLMs for both large-scale prompt generation and scalable, fine-grained T2I model evaluation.This dual use is intended to reduce costly human annotation while maintaining efficiency and reliability.
  • Benchmark Construction and Evaluation Pipeline: For each prompt, the construction process samples a theme and subject category uniformly, then selects 1 to 5 evaluation testpoints.The sampled tuple controls the semantic constraints and targeted evaluation aspects for the generated prompt.

1) Prompt and Testpoint Description Generation:

The prompt-generation stage feeds sampled semantic categories and testpoints to an MLLM, which produces a constrained natural-language prompt and structured descriptions for evaluation.

  • Prompt and Testpoint Description Generation: The pipeline passes a sampled theme, subject category, and testpoint subset to the MLLM as semantic constraints.The sampled elements determine the prompt’s intended content and fine-grained evaluation targets.
  • Prompt and Testpoint Description Generation: The MLLM generates both a natural-language prompt and a structured description set for the selected testpoints.Each description specifies how the corresponding semantic requirement should be assessed.
  • Prompt and Testpoint Description Generation: The examples combine complex entities and relations, such as a panda in a spacesuit and a nebula butterfly resting on its helmet.The associated descriptions assess world knowledge, pronoun reference, imagination, composition, and size.
  • Prompt and Testpoint Description Generation: During evaluation, the MLLM receives each generated image, its prompt, and testpoint descriptions, then outputs binary scores and textual rationales.Each testpoint is evaluated independently in the context of the image and prompt.

2) T2I Model Evaluation:

The evaluation pipeline scores generated images at fine-grained testpoints, aggregates these into sub- and primary-dimension scores, and compares models using shared evaluation settings and independent quality metrics.

  • Testpoint evaluation: Each testpoint receives a score for whether the generated image satisfies its corresponding description.The evaluation also captures qualitative reasoning alongside quantitative testpoint performance.
  • Score aggregation: Sub-dimension scores are computed from successful testpoint satisfactions, while primary-dimension scores average their sub-dimensions.This preserves both fine-grained and broad performance measurements.
  • Training comparisons: FLUX.1-dev is trained with Flow-GRPO using HPSv2, CLIP, or UnifiedReward as pointwise rewards, while PREF-GRPO uses UnifiedReward-Think as its preference reward model.The same base model supports comparisons between score maximization and pairwise preference optimization.
  • Experimental settings: Training uses 5k prompts, while inference generates 1024×1024 images with 30 sampling steps and identical prompts and initial noise across methods.The 5k training prompts are disjoint from the 600 UNIGENBENCH evaluation prompts.
  • Evaluation metrics: Evaluation uses UNIGENBENCH, GenEval, and T2I-CompBench for semantic consistency, plus UnifiedReward, ImageReward, PickScore, and Aesthetic for image quality.ImageReward, PickScore, and Aesthetic are independent of the training rewards.

B. Main Results: PREF-GRPO vs. Reward-Maximization Baselines

PREF-GRPO improves semantic consistency over reward-maximization baselines and generalizes across benchmarks without sacrificing image quality. UNIGENBENCH further reveals distinct strengths and weaknesses among open- and closed-source models through primary- and sub-dimension analysis.

  • PREF-GRPO results: 5.84% higher overall UNIGENBENCH score, including +12.69% on Text and +12.04% on Logical Reasoning, is achieved by PREF-GRPO versus UnifiedReward.These gains are reported against the strongest score-maximization baseline.
  • PREF-GRPO results: PREF-GRPO consistently outperforms reward-maximization baselines on GenEval and T2I-CompBench while achieving the best score on all four image-quality metrics.The results indicate semantic gains without an image-quality trade-off.
  • Qualitative comparison: Pointwise baselines show oversaturation or dark-style artifacts, whereas PREF-GRPO avoids these reward-hacked patterns and preserves FLUX.1-dev’s visual style.Combining HPSv2 and CLIP only partially reduces reward hacking.
  • Benchmarking results: Closed-source models lead on 7 of 10 UNIGENBENCH primary dimensions, especially those requiring deeper semantic grounding.Style and World Knowledge are comparatively saturated across many models.
  • Benchmarking results: Qwen-Image is the strongest open-source model overall, while open-source systems show greater variance and weaker performance on Logical Reasoning, Grammar, and Compound.The remaining gap is characterized as fine-grained instruction grounding and multi-constraint satisfaction rather than primarily image quality.
  • Benchmarking results: UNIGENBENCH functions as both a leaderboard and a diagnostic benchmark that exposes where current T2I models fail.Its benchmark results span representative closed-source and open-source systems.
  • Fine-grained diagnosis: Fine-grained sub-dimension scores expose differences hidden by primary-dimension averages, with GPT-4o stronger on semantic interpretation and Imagen-4.0-Ultra stronger on spatial and compositional structure.Qwen-Image has the most balanced open-source profile, while HiDream-I1-Full remains weaker on reasoning-intensive dimensions.

VII. ANALYSES AND ABLATIONS

Pointwise reward maximization can produce rising reward scores alongside deteriorating image quality, while PREF-GRPO maintains more stable training and avoids the observed visual degradation.

  • Reward hacking: HPSv2 reward continues rising while image quality collapses around step 160 with severe oversaturation.HPSv2 exhibits the same failure mode more abruptly than UnifiedReward.
  • Reward hacking: UnifiedReward scores rise while images drift toward an unnatural dark style and perceptual quality degrades during training.This is the observed reward-hacking pattern for UnifiedReward.
  • Mechanism: Compressed within-group pointwise scores amplify tiny gaps into large normalized advantages, driving optimization toward spurious reward cues.The mechanism is linked to illusory-advantage amplification under GRPO.
  • Mitigation: PREF-GRPO maintains larger intra-group reward variance and avoids the visual drift seen with pointwise-score GRPO during prolonged training.Qualitative results also show preservation of the original FLUX.1-dev visual style.

B. Point-Score vs. Pairwise-Preference Win Rates

Win-rate fitting isolates a key source of stability in PREF-GRPO: changing the optimization target from absolute scores to relative preferences mitigates reward hacking, with native pairwise rewards providing additional benefit at moderate cost.

  • Objective ablation: Converting UnifiedReward scores into pairwise win rates replaces score maximization while keeping the underlying reward model unchanged.This controlled variant tests whether stability comes from the optimization objective rather than only the reward model.
  • Objective ablation: Point-score-derived win-rate training substantially alleviates UnifiedReward’s dark-style reward hacking, while native PREF-GRPO achieves the best overall performance.The comparison supports relative-preference optimization as the principal stabilizing factor.
  • Objective ablation: Native pairwise preference rewards outperform score-derived win rates, indicating cleaner and more faithful relative signals than rankings from compressed pointwise scores.Scalar-reward practitioners can use win-rate fitting as a drop-in stability improvement.
  • Computation cost: At G=8, pairwise rewards increase per-step reward time from 3 s to 7 s; at G=16, reward time rises from 5 s to 22 s despite 7.5× more requests.Parallel reward serving makes wall-clock growth sublinear in pairwise comparisons.
  • Computation cost: PREF-GRPO achieves a favorable computation–performance trade-off, with strong gains at G=8 and limited wall-clock overhead.The principal concern is theoretical O(G^2) comparison complexity rather than practical infeasibility in the reported setting.
  • Joint optimization: Adding CLIP improves semantic consistency but slightly reduces perceptual quality without triggering obvious reward hacking.Pairwise preference fitting acts as a stabilizing regularizer for the auxiliary score signal.

E. Robustness to Noisy Preferences

PREF-GRPO remains effective when pairwise preference outcomes are moderately noisy, while UNIGENBENCH’s evaluator is assessed against human judgments to support benchmark reliability.

  • Noisy preference robustness: p=0.1 pairwise-outcome flips model preference-label noise during training, approximating typical human disagreement rates on image preferences.Each pairwise outcome is independently flipped with probability 0.1.
  • Noisy preference robustness: PREF-GRPO consistently outperforms point score-based baselines under noisy preferences, with only a small degradation from its clean setting across all metrics.The reported result supports robustness to moderate label corruption in practical preference annotation settings.
  • Evaluator validation: Gemini-2.5-Pro is validated against human majority labels using 400 held-out prompt-image-testpoint triplets spanning all 10 primary dimensions.Five annotators independently judge each triplet using the benchmark’s binary evaluation protocol.
  • Evaluator validation: Human validation reports inter-annotator agreement, Gemini-versus-human accuracy, macro-F1, and Cohen’s κ as complementary evaluator-reliability measures.These measures respectively address human ambiguity, direct agreement, class imbalance, and chance agreement.
Loading 2508.20751v2…