Pref-GRPO: Pairwise Preference Reward-based GRPO for Stable Text-to-Image Reinforcement Learning

I. INTRODUCTION

• Within-group normalization amplifies tightly clustered pointwise scores into illusory advantages, encouraging optimization for trivial cues or reward-model biases and destabilizing image generation.This explains how reward scores can rise while image quality deteriorates during prolonged training.Source: §I · p. 2
• PREF-GRPO replaces absolute-score maximization with pairwise preference fitting: a preference model compares each image pair, and each image’s win rate becomes its reward signal.The paper says this produces more separable rewards and informative, stable advantages, reduces sensitivity to marginal score gains, and better reflects human comparisons.Source: §I · p. 2, §II · p. 3
• UNIGENBENCH results show strong open- and closed-source models perform relatively well on Style and World Knowledge but underperform on Logical Reasoning prompts involving causal or contrastive relations.The paper also identifies Grammar and Compound as fine-grained compositional bottlenecks, while describing Style and World Knowledge as saturated.Source: Figure 3 · p. 2, §I · p. 3
• +5.84% overall on UNIGENBENCH, including +12.69% on Text and +12.04% on Logical Reasoning; PREF-GRPO also improves on GenEval and T2I-CompBench without sacrificing perceptual quality.The paper reports that PREF-GRPO alleviates reward hacking while improving semantic alignment.Source: §I · p. 3

II. RELATED WORK

• Preference-driven T2I methods include DiffusionDPO and training-free DyMO, while Flow-GRPO [25] and DanceGRPO [3] apply GRPO to flow-matching models; score-maximizing approaches remain prone to reward hacking.The paper positions PREF-GRPO as shifting GRPO training from score maximization to pairwise preference fitting.Source: §II · p. 3

I. INTRODUCTION

• UNIGENBENCH evaluates semantic consistency across 600 prompts, 5 themes and 20 sub-themes, with 10 primary dimensions and 27 fine-grained sub-dimensions scored through explicit testpoints.Its automated construction and evaluation pipeline uses Gemini-2.5-Pro; existing benchmarks mainly report primary-dimension scores or cover sparse subsets of sub-dimensions.Source: §I · pp. 2–3, §II · p. 3, Table I · p. 3

II. RELATED WORK

• Prior T2I benchmarks assess composition, world knowledge, or text rendering, but generally offer coarse primary-dimension evaluations and limited subtask coverage.The introduction notes CLIP- or detection-based metrics in widely adopted benchmarks and additional dimensions such as text rendering in TIIF-Bench [7].Source: §I · p. 2, §II · p. 3

III. MOTIVATION FOR PAIRWISE PREFERENCE REWARD

• Near-identical pointwise scores can become disproportionately large GRPO advantages after normalization by a tiny group standard deviation, creating illusory advantages.The resulting updates follow amplified numerical fluctuations rather than substantial image-quality differences.Source: §III · p. 3
• Absolute-reward maximization encourages models to chase reward-model bias; HPSv2 favors over-saturated images and UnifiedReward an unnatural dark style, while clipping only dampens updates.These behaviors reflect objective mismatch, not merely optimization instability; prior work also reports reward hacking [1] [3].Source: §III · pp. 3–4

IV. PREF-GRPO

• Pairwise preference fitting replaces score maximization with win-rate rewards, using more reliable relative judgments and stable ordering rather than absolute score magnitudes.The approach is suited to close-quality comparisons, reduces sensitivity to calibration bias and small fluctuations, and better matches comparative human evaluation.Source: §III · p. 4, §IV · p. 4

A. Flow Matching GRPO

• At inference, flow matching is formulated as an MDP with prompt, time, and latent state; denoising actions advance deterministic transitions, and reward is given only at the terminal state.The time interval is discretized into T steps.Source: A. Flow Matching GRPO · p. 4
• GRPO on flow-matching models uses group-relative advantages to stabilize policy updates and maximizes a clipped, KL-regularized objective.Source: A. Flow Matching GRPO · p. 4
• Flow-GRPO converts deterministic flow dynamics to an equivalent SDE for stochastic exploration, then applies an Euler–Maruyama update.Source: A. Flow Matching GRPO · p. 4

• Pointwise-reward GRPO normalizes each generated image’s reward within its group, and tightly clustered scores produce a small standard deviation that magnifies minor score gaps.The method uses pointwise reward models to score groups of images at each update step.Source: B. Illusory Advantage in… · p. 4
• When within-group scores are nearly identical, dividing small reward differences by a small standard deviation creates large normalized advantages and makes updates overly dependent on noisy differences.The normalization is highly sensitive to small perturbations in reward scores.Source: B. Illusory Advantage in… · p. 5
• The authors call this amplification the illusory advantage phenomenon: it causes reward hacking through excessive optimization and sensitivity to reward-model noise.The policy may pursue extreme reward-hacked behaviors or exploit model flaws instead of improving genuine image quality.Source: B. Illusory Advantage in… · p. 5

• PREF-GRPO uses PPRM [13] to fit pairwise image preferences rather than maximize absolute reward scores.This shifts the optimization objective from absolute scores to relative preferences among generated images.Source: C. Pairwise Preference Reward-based GRPO · p. 5
• PPRM comparisons between images in each group determine each image’s win rate, which replaces scalar rewards in the GRPO advantage.The method compares each image against every other image using ordered pairs.Source: C. Pairwise Preference Reward-based GRPO · p. 5
• Win rates spanning [0, 1] increase within-group reward variance, sharpen advantage estimation, and reduce sensitivity to reward-model noise, mitigating reward hacking and stabilizing training.High-quality samples approach 1 and lower-quality samples approach 0, making the reward distribution more discriminative.Source: C. Pairwise Preference Reward-based GRPO · p. 5
• By mirroring relative human judgments, PREF-GRPO captures fine-grained image-quality distinctions often missed by pointwise scoring and provides a more faithful policy-improvement signal.The relative comparisons also account for prompt alignment.Source: C. Pairwise Preference Reward-based GRPO · p. 5

• Existing benchmarks [7] often cover few sub-dimensions and report only primary-dimension scores, limiting interpretability and diagnosis of model strengths and weaknesses.For example, they include only one sub-dimension for Relationship and Grammar.Source: §V · p. 5
• UNIGENBENCH evaluates 10 primary dimensions and 27 sub-dimensions, enabling finer-grained diagnosis than benchmarks reporting only dimension-level scores.Source: §V · p. 5
• Its prompt set spans 5 major themes and 20 sub-themes, using 600 prompts with 1 to 5 explicit testpoints each.The themes range from realistic rendering to open-ended creative composition.Source: §V · p. 5
• Structured testpoint descriptions paired with each prompt support precise, interpretable MLLM assessment of whether intended semantic targets are satisfied.Source: §V · p. 5

A. Prompt Themes and Evaluation Dimensions

• UNIGENBENCH spans five major themes and 20 subcategories, covering subjects from animals and objects to scenes and special entities.Its prompt design aims to cover practical generation scenarios and open-ended creative settings rather than a narrow prompt distribution.Source: A. Prompt Themes and Evaluation… · p. 5
• UNIGENBENCH defines 10 primary evaluation dimensions and 27 sub-dimensions, including often-overlooked capabilities such as pronoun reference, hand actions, and logical reasoning.The finer decomposition is intended to diagnose strengths and weaknesses more precisely than dimension-level averages alone.Source: A. Prompt Themes and Evaluation… · p. 6
• The 10 dimensions cover semantic faithfulness in style, world knowledge, attributes, actions, relationships, layout, compound concepts, grammar, logical reasoning, and text.They respectively assess visual-form adherence, factual consistency, properties, behaviors, grounded relations, spatial accuracy, coherent concept integration, language-sensitive grounding, causal or inferential understanding, and explicit text generation.Source: A. Prompt Themes and Evaluation… · p. 6
• Testpoints subdivide attributes, actions, relationships, and layout to identify specific semantic-generation failures rather than only coarse dimension-level averages.Examples include quantity, expression, and color; hand actions and animal behaviors; composition and similarity; and 2D placement and 3D spatial arrangement.Source: A. Prompt Themes and Evaluation… · p. 6
• UNIGENBENCH uses 600 prompts, each targeting 1 to 5 testpoints, to balance broad semantic coverage with interpretability and evaluation efficiency.The prompts remain focused and evaluation-friendly while collectively spanning a broad set of semantic capabilities.Source: A. Prompt Themes and Evaluation… · p. 6

B. Benchmark Construction and Evaluation Pipeline

• The MLLM-based pipeline supports systematic prompt generation and scalable, fine-grained T2I evaluation without costly human annotation.The paper describes the process as efficient and reliable for benchmark construction and model assessment.Source: B. Benchmark Construction and… · p. 6
• For each prompt, the pipeline uniformly samples a theme and subject category, then selects 1–5 evaluation dimensions as testpoints.The selected testpoints target specific fine-grained evaluation aspects.Source: 1) Prompt and Testpoint Description… · p. 6

1) Prompt and Testpoint Description Generation:

• Given the selected theme, subject category, and testpoints, the MLLM generates a semantically constrained prompt and descriptions specifying how each testpoint should appear.The generated prompt must conform to the selected theme and subject category.Source: 1) Prompt and Testpoint Description… · pp. 6–7

2) T2I Model Evaluation:

• For each generated image, the MLLM considers its prompt and testpoint descriptions, returning a binary score and textual rationale for every testpoint.This captures quantitative testpoint performance and qualitative reasoning.Source: 2) T2I Model Evaluation · p. 7
• Each sub-dimension score is the proportion of benchmark occurrences for which the model satisfies the corresponding testpoint description.The numerator counts successful matches and the denominator counts occurrences of that testpoint.Source: 2) T2I Model Evaluation · p. 7
• A primary-dimension score is the average of its sub-dimension scores, capturing both fine-grained and broader performance.The aggregation follows the per-testpoint binary scoring stage.Source: 2) T2I Model Evaluation · p. 7

A. Experimental Setup

• The study uses FLUX.1-dev with UnifiedReward-Think as PREF-GRPO’s pairwise preference model, and compares it with Flow-GRPO trained using HPSv2, CLIP, or UnifiedReward.Source: A. Experimental Setup · p. 7
• Training uses the same 5k prompts for all methods, disjoint from UNIGENBENCH evaluation prompts, and evaluation gives all methods identical prompts and initial noise.Training uses 64 H20 GPUs, 25 sampling steps, 8 rollouts per prompt, 4 gradient accumulation steps, and a learning rate of 1×10−5; inference uses 1024×1024 images, 30 sampling steps, and guidance scale 3.5.Source: A. Experimental Setup · p. 7
• Evaluation measures semantic consistency on UNIGENBENCH, GenEval, and T2I-CompBench, alongside image quality using UnifiedReward, ImageReward, PickScore, and Aesthetic.Four images are generated per test prompt and their scores are averaged; ImageReward, PickScore, and Aesthetic are disjoint from the training rewards.Source: A. Experimental Setup · p. 7

B. Main Results: PREF-GRPO vs. Reward-Maximization Baselines

• PREF-GRPO improves UNIGENBENCH overall score by 5.84% over UR, with Text (+12.69%) and Logical Reasoning (+12.04%) gains.The authors attribute this pattern to pairwise preferences providing a cleaner gradient signal on fine-grained semantic grounding and compositional understanding.Source: B. Main Results: PREF-GRPO vs.… · p. 7, Table II · p. 8
• PREF-GRPO consistently outperforms reward-maximization baselines on GenEval and T2I-CompBench and scores best on all four image-quality metrics.ImageReward, PickScore, and Aesthetic are independent quality metrics not used as training rewards.Source: B. Main Results: PREF-GRPO vs.… · p. 7, Table III · p. 8
• Qualitatively, HPSv2 optimization yields oversaturation and UnifiedReward yields an unnaturally dark style, while PREF-GRPO avoids these artifacts and faithfully renders prompt semantics.Combining HPSv2 and CLIP scores partially reduces, but does not eliminate, reward hacking.Source: B. Main Results: PREF-GRPO vs.… · p. 7, Figure 8 · p. 8

C. Benchmarked T2I Models on UNIGENBENCH

• The benchmark covers representative closed- and open-source models spanning diffusion, MMDiT, autoregressive, and unified architectures, using official APIs or released checkpoints with default inference settings.The evaluated set includes models such as GPT-4o [15], Imagen-4.0-Ultra, Qwen-Image [18], and HiDream-I1-Full [19].Source: C. Benchmarked T2I Models on UNIGENBENCH · p. 8

D. Benchmarking Results on UNIGENBENCH

• Style and World Knowledge are saturated across many models, while UNIGENBENCH exposes remaining failures in higher-order semantic composition that coarse overall scores can hide.Many models score above 0.8 on these dimensions; the benchmark also serves as a diagnostic tool for current T2I model weaknesses.Source: D. Benchmarking Results on UNIGENBENCH · pp. 8–9
• Qwen-Image [18] is the strongest open-source model overall, but open-source models vary more across primary dimensions and lag on Logical Reasoning, Grammar, and Compound.HiDream-I1-Full [19] is competitive on Action, Layout, and Attribute; the remaining gap reflects fine-grained instruction grounding and satisfying multiple constraints coherently.Source: D. Benchmarking Results on UNIGENBENCH · p. 8
• GPT-4o [15] and Imagen-4.0-Ultra lead on 7 of 10 primary dimensions, while closed-source models are strongest overall (Tab. V).Their advantage is most pronounced on Logical Reasoning, Relationship, Compound, and Text.Source: D. Benchmarking Results on UNIGENBENCH · p. 8, Table V · p. 9
• GPT-4o [15] leads closed-source models on expression, non-contact interaction, Grammar, and Logical Reasoning, whereas Imagen-4.0-Ultra [17] leads on quantity, contact interaction, composition, and 3D spatial arrangement (Layout).The sub-dimension breakdown shows that top systems differ in abstract semantic interpretation versus spatial and compositional structure (Tab. VII).Source: D. Benchmarking Results on UNIGENBENCH · p. 9, Table VII · p. 10
• Qwen-Image [18] has the most balanced open-source sub-dimension profile and leads in a majority, while HiDream-I1-Full [19] is competitive on geometry and action but weaker on reasoning-intensive dimensions (Tab. VII).Models with similar primary-dimension averages can still differ substantially in the sub-dimensions they master.Source: D. Benchmarking Results on UNIGENBENCH · p. 9, Table VII · p. 10

A. Reward Hacking in Pointwise GRPO

• UnifiedReward and HPSv2 scores rise during training as generated images degrade; HPSv2 quality collapses around step 160 with severe oversaturation.UnifiedReward training also drifts toward an unnatural dark style, revealing a mismatch between reward growth and generation quality.Source: A. Reward Hacking in Pointwise GRPO · p. 10, Figure 9 · p. 10
• Compressed within-group pointwise scores let GRPO amplify tiny gaps into large normalized advantages, and HPSv2 hacks earlier than UnifiedReward amid lower score variance.The amplification can steer the policy toward spurious reward cues rather than genuine alignment or aesthetic gains.Source: A. Reward Hacking in Pointwise GRPO · p. 10
• PREF-GRPO maintains larger intra-group reward variance and avoids the visual degradation and reward-hacked artifacts seen with pointwise-score GRPO.The reported comparisons say it preserves the original FLUX.1-dev visual style during prolonged training.Source: A. Reward Hacking in Pointwise GRPO · p. 10

B. Point-Score vs. Pairwise-Preference Win Rates

• The controlled variant keeps UnifiedReward unchanged, converting its group scores into pairwise win rates to isolate the effect of the optimization objective.It compares every image pair and uses the resulting win rates as training rewards instead of maximizing scores directly.Source: B. Point-Score vs. Pairwise-Preference… · p. 10

VII. ANALYSES AND ABLATIONS

• Point-score-derived win-rate training consistently outperforms direct score maximization, while PREF-GRPO achieves the best overall performance.The comparison supports relative-preference optimization as a stabilizing factor, strengthened by native pairwise preference rewards.Source: B. Point-Score vs. Pairwise-Preference… · p. 11, Table VIII · p. 11

B. Point-Score vs. Pairwise-Preference Win Rates

• Native pairwise preference rewards outperform score-derived win rates, while win-rate fitting offers scalar-reward users a drop-in stability improvement.The paper attributes the added benefit to cleaner, more faithful relative signals from native pairwise supervision.Source: B. Point-Score vs. Pairwise-Preference… · p. 11

C. Wall-Clock Overhead of Pairwise Rewards

• The overhead experiment compares UnifiedReward point scores with PREF-GRPO pairwise win-rate rewards across group sizes, using 4 H800 GPUs for training and 4 for reward inference.The setup isolates per-step reward cost from the multi-node overhead of the 64-GPU main setup; pairwise rewards require O(G^2) within-group comparisons.Source: C. Wall-Clock Overhead of Pairwise… · p. 11
• At G=8, pairwise rewards increase per-step reward time from 3 s to 7 s; at G=16, requests rise from 64 to 480 (7.5×) while time rises from 5 s to 22 s (3.1×).The passage attributes the far-sublinear wall-clock growth to amortization through parallel reward serving.Source: C. Wall-Clock Overhead of Pairwise… · p. 11, Table IX · p. 11
• PREF-GRPO is characterized as a favorable computation–performance trade-off: improved generation quality and mitigated reward hacking with limited wall-clock overhead.The passage identifies theoretical complexity, rather than practical infeasibility, as the main concern with pairwise rewards.Source: C. Wall-Clock Overhead of Pairwise… · p. 11

D. Joint Optimization with Auxiliary Score Rewards

• The robustness experiment evaluates PREF-GRPO under 10% preference noise, with the noisy variant flipping each pairwise outcome at probability p=0.1 during training.The supplied passage gives the noise condition but no robustness outcome.Source: Table XI · p. 12
• The joint-optimization experiment adds a CLIP score reward to PREF-GRPO to test whether pairwise preference fitting can stabilize score-based supervision while CLIP supplies semantic guidance.CLIP is described as focusing more on semantic consistency between the prompt and generated image.Source: D. Joint Optimization with Auxiliary… · pp. 11–12
• Joint PREF-GRPO and CLIP improves semantic consistency on UniGenBench and GenEval, slightly lowers image-quality metrics, and avoids obvious reward hacking.The results suggest pairwise preference fitting stabilizes the auxiliary score signal, allowing it to provide complementary semantic guidance without collapsing to reward-hacked solutions.Source: Figure 11 · p. 11, D. Joint Optimization with Auxiliary… · p. 12, Table X · p. 12

E. Robustness to Noisy Preferences

• Under pairwise-outcome flips with probability p=0.1, PREF-GRPO’s performance drops only slightly across metrics and remains stronger than all point score-based baselines.The injected noise roughly matches typical human disagreement rates on image preferences.Source: E. Robustness to Noisy Preferences · p. 12

F. Human Validation of the UNIGENBENCH Evaluator

• The evaluator agreement study sampled 400 held-out prompt-image-testpoint triplets, with 40 triplets for each of 10 primary dimensions, and used five annotators’ majority vote as the reference.Models of varying strengths supplied the predictions; annotators judged each image against an explicit testpoint using the benchmark’s binary protocol.Source: F. Human Validation of the UNIGENBENCH… · p. 12
• Gemini-2.5-Pro shows strongest human agreement on Style, Attribute, and Layout, while Relationship, Compound, and Logical Reasoning are more challenging.The reported measures are inter-annotator agreement, accuracy against the human majority, macro-F1, and Cohen’s κ.Source: Table XII · p. 12, F. Human Validation of the UNIGENBENCH… · p. 12
• The evaluator study uses complementary metrics: inter-annotator agreement as a practical human reference ceiling, accuracy for direct human agreement, macro-F1 for class imbalance, and κ for chance agreement.These measures provide complementary views of evaluator reliability.Source: F. Human Validation of the UNIGENBENCH… · p. 12

VIII. CONCLUSION

• PREF-GRPO stabilizes training, alleviates reward hacking, and improves semantic consistency in-domain and out-of-domain, while UNIGENBENCH provides fine-grained evaluation of semantic consistency.Together, the method and benchmark offer a more reliable training objective and a more informative evaluation protocol for T2I reinforcement learning.Source: §VIII · p. 12
• UNIGENBENCH highlights persistent T2I weaknesses in relationship understanding, compositional instruction following, logical reasoning, and text rendering.The conclusion identifies these as especially demanding dimensions for strong models.Source: §VIII · p. 12