PaperlayerView exact revision on arXiv

Source-linked AI summary

The Image as Its Own Reward: Reinforcement Learning with Adversarial Reward for Image Generation

Weijia Mao, Hao Chen, Zhenheng Yang, Mike Zheng Shou

arXiv:2511.20256v1cs.CV

TL;DR

Scalar preference rewards for image generation can misalign with human perception and be exploited without improving image quality. Adv-GRPO jointly trains a generator and an adversarial reward using reference images, and also uses visual foundation-model rewards. The paper reports gains across image quality, text alignment, and aesthetics.

  • Problem

    Scalar reward models can reflect visual-attribute biases rather than human preferences, allowing higher scores without genuine image-quality improvement.

    Source: §1 · pp. 1–2

  • Method

    Adv-GRPO jointly optimizes the generator and an adversarial reward model supervised with reference images, and uses visual foundation-model rewards such as DINO.

    Source: §1 · p. 2, §3.3 · p. 6

  • Results

    The method is reported to alleviate reward hacking while maintaining competitive standard-benchmark performance, with visual foundation-model rewards improving image quality, text–image alignment, and aesthetics.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The conclusion reports that reference-based rewards and visual foundation models provide guidance for improving image quality, aesthetics, and text alignment across diverse reward settings.

    Source: §5 · p. 11

    Existing reward-model biases remain: PickScore optimization may sacrifice image quality, while OCR optimization may reduce aesthetic fidelity.

    Source: §1 · p. 2

Abstract

from arXiv · show

A reliable reward function is essential for reinforcement learning (RL) in image generation. Most current RL approaches depend on pre-trained preference models that output scalar rewards to approximate human preferences. However, these rewards often fail to capture human perception and are vulnerable to reward hacking, where higher scores do not correspond to better images. To address this, we introduce Adv-GRPO, an RL framework with an adversarial reward that iteratively updates both the reward model and the generator. The reward model is supervised using reference images as positive samples and can largely avoid being hacked. Unlike KL regularization that constrains parameter updates, our learned reward directly guides the generator through its visual outputs, leading to higher-quality images. Moreover, while optimizing existing reward functions can alleviate reward hacking, their inherent biases remain. For instance, PickScore may degrade image quality, whereas OCR-based rewards often reduce aesthetic fidelity. To address this, we take the image itself as a reward, using reference images and vision foundation models (e.g., DINO) to provide rich visual rewards. These dense visual signals, instead of a single scalar, lead to consistent gains across image quality, aesthetics, and task-specific metrics. Finally, we show that combining reference samples with foundation-model rewards enables distribution transfer and flexible style customization. In human evaluation, our method outperforms Flow-GRPO and SD3, achieving 70.0% and 72.4% win rates in image quality and aesthetics, respectively. Code and models have been released.

1 Introduction

Scalar reward models can misalign with human visual preferences and invite reward hacking in text-to-image generation. Adv-GRPO uses reference images and adversarial or visual-foundation-model rewards to guide generation, with reported gains across image quality, alignment, and aesthetics.

1 Introduction

  • Scalar reward models can bias generations toward attributes such as oversaturated colors or excessive text emphasis, allowing higher scores without genuine quality improvement.The paper describes this mismatch as reward hacking and notes Flow-GRPO underperforms the base model in some human-evaluation aspects under PickScore and OCR rewards.Source: §1 · p. 2
  • Adv-GRPO jointly optimizes the generator and an adversarial reward model, using high-quality reference images as positive supervision for the reward model.The generator is optimized with GRPO; for rule-based rewards such as OCR, reference images are used in a multi-reward optimization scheme.Source: §1 · pp. 2–3
  • DINO is fine-tuned as a reward model whose output features provide dense visual signals for guiding generations toward improved aesthetics, text alignment, and visual fidelity.The paper contrasts these signals with a single scalar from existing reward models.Source: §1 · pp. 2–3
  • The framework extends to multiple existing reward models and explores visual foundation models as rewards for the base text-to-image model.The overview also presents reference-domain alignment as a route to style customization.Source: Figure 1 · p. 2, §1 · p. 3
  • Visual foundation model rewards produce comprehensive improvements in image quality, text–image alignment, and aesthetics.The introduction reports these gains alongside effective mitigation of reward hacking with existing reward models and competitive standard-benchmark performance.Source: §1 · p. 3

2 Related Work

Prior work applies online RL methods, including GRPO variants, to image generation, while reward-model research spans human-preference and rule-based signals. These approaches still face reward hacking and training-stability challenges, motivating the paper’s adversarial framework with reference samples.

2.1 Reinforcement Learning for Image Generation

  • Prior image-generation RL includes DPO, PPO, and GRPO methods such as DanceGRPO and Flow-GRPO, with some work refining SDE processes for stability.The passage also notes that GRPO removes the need for an additional value network compared with PPO.Source: §2.1 · p. 3
  • Despite these advances, image-generation RL methods still face reward hacking and training instability; this paper contrasts its adversarial training framework with reference samples.Source: §2.1 · p. 3

2.2 Reward Models for Image Quality Assessment

  • Human-preference and rule-based reward models provide different supervision, but both can overfit to specific biases rather than capture true perceptual quality.Human-preference models include CLIP-based variants, while rule-based examples include OCR and GenEval.Source: §2.2 · pp. 3–4

3 Method

The method extends GRPO for text-to-image generation by jointly training a generator and an adversarial reward model, then uses visual foundation-model features to provide global and local image rewards. It also combines reference-image similarity with deterministic task rewards when adversarial training is unsuitable.

3.1 Preliminary

  • GRPO treats each diffusion reverse step as an action in a Markov decision process and optimizes groups of generated samples using normalized within-group reward advantages.The clipped objective includes a KL penalty weighted by β to stabilize training.Source: §3.1 · p. 4

3.2 GRPO with Adversarial Reward

  • Adv-GRPO jointly trains the generator by GRPO and a discriminator reward on reference-positive/generated-negative images, triggering discriminator fine-tuning when generated mean reward exceeds reference mean.The authors state that this co-training mitigates reward hacking and encourages overall visual fidelity.Source: §3.2 · pp. 4–5
  • For rule-based rewards such as OCR or GenEval, the method combines the task reward with CLIP similarity to a reference image, balancing task specificity and visual fidelity.The weight λ controls the trade-off, and the formulation is described as preventing rule-based objectives from dominating.Source: §3.2 · p. 5

3.3 Visual Foundation Models As Reward

  • A frozen visual backbone and lightweight classification head produce a reward combining global [CLS] features with randomly sampled local patch features, weighted by λg and λl.The global–local formulation pairs high-level semantics and structural consistency with local texture and detail; patch sampling is described as promoting diverse local structures while maintaining computational efficiency.Source: §3.3 · p. 6

4 Experiments

Across reward settings, Adv-GRPO improves human-rated image outcomes over Flow-GRPO and SD3. DINO reward improves results across tasks, while reference-image experiments test data efficiency and style customization.

4.1 Implementation Details / 4.2 Evaluation Protocol

  • Experiments used SD3 and Flow-GRPO baselines, PickScore, OCR, GenEval and DINO-similarity metrics, and 12 experts judging 400 prompts in 12,000 comparisons.The evaluation covered aesthetics, prompt alignment, and image quality.Source: §4.1 · p. 7, §4.2 · pp. 7–8

4.3 Main Results

  • 70% image-quality win rate under PickScore and 85.3% aesthetics win rate under OCR beat Flow-GRPO; versus SD3, PickScore aesthetics reaches 72.6% and OCR alignment 77.6%.PickScore optimization still tends to sacrifice image quality, while OCR optimization slightly compromises aesthetics.Source: Figure 4 · p. 7, §4.3 · p. 9
  • 72.4% aesthetic win rate versus SD3; against Flow-GRPO, Adv-GRPO reaches 66.3% in quality with DINO similarity reward and 75.2% in aesthetics.With PickScore reward, it reaches 93.5% win rate in quality against Flow-GRPO.Source: Figure 6 · p. 8, §4.3 · p. 9
  • OCR accuracy rose from 0.59 to 0.69 and GenEval score from 0.61 to 0.69 with adversarial DINO reward versus SD3.Table 2 reports the multi-task comparison; visual comparisons across task prompts appear in Figure 8.Source: Table 2 · p. 9, §4.3 · p. 9

4.4 Ablation Study

  • Against SFT, DINO-reward Adv-GRPO achieved over 70% win rates in aesthetics and image quality, and supports targeted optimization for text readability or visual appeal.The reported comparison includes Figure 6(d), Figure 9, and Table 4.Source: §4.4 · p. 9, Table 4 · p. 10
  • Comparable DINO similarity and stable visual quality and style consistency were observed with as few as 200 reference images.The study compared datasets of 200, 500, and 1,000 samples.Source: §4.4 · p. 9, Table 3 · p. 10
  • KL constraints lowered reward scores and visual quality, while multi-reward optimization could mitigate hacking but made reward-weight balancing challenging; Adv-GRPO outperformed that baseline.The KL coefficient was sensitive: large values restricted optimization, while small values could not prevent reward hacking.Source: §4.4 · p. 10

4.5 Style Customization via Adv-GRPO

  • Using anime-style and sci-fi reference datasets, the method transferred SD3 toward the target styles while preserving semantic structure and image quality.The customization pipeline uses image inputs and visual foundation models such as DINO to guide learning.Source: Figure 11 · p. 10, §4.5 · p. 11

5 Conclusion

The conclusion presents an adversarial-reward RL framework for text-to-image generation, using high-quality references and visual foundation models to guide generation. It reports improved alignment with visual preferences and gains across image quality, aesthetics, and text alignment, alongside supplementary materials.

5 Conclusion

  • The framework uses high-quality references to better align reward models with human visual preferences and mitigate reward hacking.Visual foundation models such as DINO provide additional guidance for image quality, aesthetics, and text alignment.Source: §5 · p. 11
  • Experiments support the framework’s effectiveness and generality across diverse reward settings.Source: §5 · p. 11

Appendix

  • The appendix provides additional visualizations, style-transfer examples, SigLIP optimization experiments, implementation details, reward curves, and human-evaluation procedures.Source: Appendix · p. 15

A More Implementation Details

The implementation specifies training and evaluation settings for DINO and PickScore rewards.

A More Implementation Details

  • DINO training uses a 7:3 global-to-local loss and reward weighting and updates the discriminator 10 times per generator update.SD3 uses LoRA rank 32, scaling factor 64, Gaussian initialization, CFG scale 4.5, and bfloat16; PickScore fine-tuning occurs only when generated-image rewards exceed reference-image rewards.Source: A More Implementation Details · p. 15

B Visualizations Under Our Method

The visualizations illustrate reward-hacking mitigation under PickScore and OCR, and image-generation qualities under DINO.

B Visualizations Under Our Method

  • The method produces higher overall visual quality than Flow-GRPO in visualizations addressing PickScore and OCR reward hacking.Source: B Visualizations Under Our Method · p. 15
  • DINO-reward visualizations show stronger composition, richer color saturation, improved aesthetics, and more diverse background details.The passage presents these examples as evidence of robustness and generalization.Source: B Visualizations Under Our Method · p. 15

C More Visualizations on Style Customization

The style-customization examples show the base model’s style transferred to anime using anime reference images.

C More Visualizations on Style Customization

D Using SigLIP for Optimization

The optimization pipeline uses SigLIP as a discriminator and SD3 as the generator, with a lightweight head classifying reference and generated images. Its successful performance supports generalization beyond DINO, despite SigLIP providing only global representations.

D Using SigLIP for Optimization

E Reward Curve

Training converges in approximately 1000 steps, and generated-image rewards remain above those of the QWen reference images throughout training.

E Reward Curve

F Human Evaluation

The human evaluation compares model-generated images on quality, aesthetics, and text-image alignment, using expert judgments and annotation quality controls. The supplied passages also describe visualizations of optimization outcomes and style customization, but do not report human-evaluation win rates.

F Human Evaluation

  • Experts compare images from two models and select the better image on quality, aesthetics, and text-image alignment.The evaluation interface is shown in Figure 18.Source: F Human Evaluation · p. 15
  • Additional visualizations cover reward-hacking alleviation under PickScore and OCR rewards, DINO results on benchmark OCR and GenEval prompts, and reward curves for three reward models.The curves cover PickScore, OCR accuracy, and DINO cosine similarity.Source: Figure 12 · p. 16, Figure 14 · p. 16, Figure 17 · p. 17
  • Visualizations report consistently higher image quality with DINO, overall image-quality improvements with SigLIP versus SD3, and transfer of an anime style from reference images.Source: Figure 13 · p. 16, Figure 15 · p. 17, Figure 16 · p. 17
  • The benchmark contains 10 comparison-task groups and 100 questions, with 300 annotations from three independent experts per question.Each group is evaluated by 12 experts.Source: F Human Evaluation · pp. 16–17
  • Annotator calibration, cross-checking, adjudication, and refinement of scoring guidelines are used to reduce ambiguity and promote consistent judgments.Source: F Human Evaluation · p. 17
Loading 2511.20256v1…