PaperlayerView exact revision on arXiv

Source-linked AI summary

GARDO: Reinforcing Diffusion Models without Reward Hacking

Haoran He, Yuxiao Ye, Jie Liu, Jiajun Liang, Zhiyong Wang, Ziyang Yuan, Xintao Wang, Hangyu Mao, Pengfei Wan, Ling Pan

arXiv:2512.24138v1cs.LGcs.AIcs.CV

TL;DR

Visual preference is difficult to specify precisely, so diffusion-model RL relies on imperfect proxy rewards and can reward-hack. GARDO selectively regularizes uncertain samples, adapts its reference policy, and promotes diversity; experiments report mitigating reward hacking while balancing sample efficiency and exploration across proxy rewards and unseen metrics.

  • Problem

    Visual tasks lack precisely specified ground-truth objectives, so proxy rewards can imperfectly capture human preference and leave RL fine-tuning vulnerable to reward hacking.

    Source: §1 · p. 1

  • Method

    GARDO selectively applies KL regularization to high-uncertainty samples, periodically updates its reference model, and amplifies advantages for high-quality samples with high diversity.

    Source: §1 · p. 2, §4.2 · p. 4, §4.3 · p. 5

  • Results

    Across multiple tasks, proxy rewards, and unseen metrics, experiments report that GARDO mitigates reward hacking while balancing sample efficiency, exploration, and diversity.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The reported experiments demonstrate a way to balance sample efficiency, exploration, and diversity while mitigating reward hacking in image-generative-model fine-tuning.

    Source: §1 · p. 2, §6 · p. 9

    GARDO depends on auxiliary reward models for uncertainty estimation, and its scalability to resource-intensive video generation remains an open question.

    Source: §6 · p. 9

Abstract

from arXiv · show

Fine-tuning diffusion models via online reinforcement learning (RL) has shown great potential for enhancing text-to-image alignment. However, since precisely specifying a ground-truth objective for visual tasks remains challenging, the models are often optimized using a proxy reward that only partially captures the true goal. This mismatch often leads to reward hacking, where proxy scores increase while real image quality deteriorates and generation diversity collapses. While common solutions add regularization against the reference policy to prevent reward hacking, they compromise sample efficiency and impede the exploration of novel, high-reward regions, as the reference policy is usually sub-optimal. To address the competing demands of sample efficiency, effective exploration, and mitigation of reward hacking, we propose Gated and Adaptive Regularization with Diversity-aware Optimization (GARDO), a versatile framework compatible with various RL algorithms. Our key insight is that regularization need not be applied universally; instead, it is highly effective to selectively penalize a subset of samples that exhibit high uncertainty. To address the exploration challenge, GARDO introduces an adaptive regularization mechanism wherein the reference model is periodically updated to match the capabilities of the online policy, ensuring a relevant regularization target. To address the mode collapse issue in RL, GARDO amplifies the rewards for high-quality samples that also exhibit high diversity, encouraging mode coverage without destabilizing the optimization process. Extensive experiments across diverse proxy rewards and hold-out unseen metrics consistently show that GARDO mitigates reward hacking and enhances generation diversity without sacrificing sample efficiency or exploration, highlighting its effectiveness and robustness.

1. Introduction

Visual rewards are imperfect proxies for human preference, making RL fine-tuning vulnerable to reward hacking. GARDO selectively regularizes uncertain samples, adapts its reference model, and promotes diversity; experiments report balancing sample efficiency, exploration, diversity, and mitigation of reward hacking.

1. Introduction

  • Imperfect visual-reward proxies can produce reward hacking: proxy scores rise while actual image quality degrades.Model-based rewards are limited by their training distribution, while rule-based rewards evaluate specific attributes rather than overall quality.Source: §1 · p. 1
  • Static KL regularization can make updates less sample-efficient and constrain exploration around a potentially suboptimal reference policy.As the online policy improves, divergence from the static reference can cause the KL penalty to dominate the RL loss.Source: §1 · pp. 1–2
  • GARDO gates KL penalties to high-uncertainty samples, periodically updates its reference model, and amplifies advantages for high-quality, diverse samples.Uncertainty is estimated from reward-function disagreement; diversity amplification is calibrated not to dominate or reverse the original advantage.Source: §1 · p. 2
  • Experiments across multiple tasks, proxy rewards, and unseen metrics report that GARDO balances sample efficiency, exploration, and diversity while mitigating reward hacking.The paper describes this balance as demonstrated across experiments on multiple tasks.Source: §1 · p. 2

2. Related Works

Prior work fine-tunes diffusion models using reward signals through policy-gradient RL, preference optimization, and reward backpropagation. Existing approaches address reward hacking through policy constraints, reward-model scaling, or pairwise preferences, each with stated limitations.

Fine-Tuning Diffusion Models via Rewards

Reward Hacking

  • Imperfect reward models can be exploited for high scores that fail human intent; KL regularization can limit severe hacking but often slows convergence and hinders exploration [10, 30].The passage gives extreme saturation and visual noise as examples of adversarial outputs [31].Source: §2 · p. 2
  • RewardDance scales reward models but does not remove out-of-distribution vulnerability [60], while Pref-GRPO is limited to preference models and incurs substantial pairwise-comparison costs [54].Source: §2 · p. 2

3. Preliminaries

The preliminaries formulate denoising as a sequential decision process and describe the GRPO setup used for diffusion and flow-model fine-tuning.

3.1. Denoising as an MDP

  • The denoising process is formulated as an MDP in which each step uses the conditioning, time, and current noisy image as state, and predicts the preceding image as action.A nonzero reward is given only at the final denoising step, and the transition is deterministic.Source: §3.1 · p. 3

3.2. RL on Diffusion and Flow Models

  • The paper uses GRPO as its base RL algorithm, grouping rollouts conditioned on the same input to estimate advantages and optimize the policy with a surrogate objective.The preliminaries identify the importance-sampling ratio, behavior policy, clipping range, and KL regularization term in that setup.Source: §3.2 · p. 3

4. Method

Proxy-reward mismatch can drive reward hacking, while static KL regularization can constrain sample efficiency and exploration. GARDO combines gated and adaptive regularization with diversity-aware optimization to address these competing objectives.

4.1. Reward Hacking in RL-Based Fine-tuning

  • Proxy-reward mismatch can make a policy score well while failing the true reward; Flow-GRPO sacrifices perceptual quality in the OCR example, and static KL can constrain learning and exploration.Source: §4.1 · p. 3

4.2. Gated and Adaptive Regularization

  • The optimal KL-regularized distribution depends on both proxy reward and reference policy; with equal reference probabilities, proxy reward alone determines sample odds.Source: §4.2 · p. 4
  • GARDO applies KL selectively to high-uncertainty samples rather than penalizing all samples; roughly 10% of the highest-uncertainty samples sufficed to prevent reward hacking.Source: §4.2 · pp. 4–5

4.3. Diversity-Aware Optimization

  • In the Gaussian-mixture example, full GARDO reaches maximum reward and discovers all high-reward clusters, including a mode assigned 0.1× the reference density.The central cluster has only 0.1× the reference model's probability density compared with other modes.Source: Figure 3 · p. 5, §4.3 · p. 6

4.2. Gated and Adaptive Regularization

  • Periodic reference-model resets, triggered by a KL threshold or a maximum number of gradient steps, keep the regularization target current and are intended to sustain improvement.Source: §4.2 · p. 5

4.3. Diversity-Aware Optimization

  • Diversity-aware shaping multiplies positive advantages by nearest-neighbor feature-space isolation, rewarding samples that are both high-quality and novel without rewarding low-quality images for diversity.Source: §4.3 · p. 6
  • Removing standard-deviation normalization avoids amplifying small, often meaningless reward differences when group rewards are similar, reducing training sensitivity to reward noise.Source: §4.3 · p. 6

5. Experiments

Experiments test GARDO on GenEval and OCR proxy tasks, unseen metrics, and diversity, with additional evaluation of counting beyond the training range. GARDO preserves sample efficiency while improving unseen-reward performance, diversity, and emerging counting behavior.

5. Experiments

  • Experiments use SD3.5-Medium with GRPO, with additional results on Flux.1-dev and DiffusionNFT; images are generated at 512×512 resolution.GenEval and OCR are proxy tasks; Aesthetic, PickScore, ImageReward, ClipScore, and HPSv3 assess unseen performance.Source: §5 · p. 6

5.1. Results Analysis

  • Without standard-deviation normalization, unseen-reward performance improves over baseline while largely preserving sample efficiency and proxy rewards, but still falls short of the reference model.The authors describe this change as insufficient on its own to fully resolve reward hacking.Source: §5.1 · p. 6
  • GARDO matches the KL-free baseline’s sample efficiency while preserving high unseen rewards, sometimes surpassing the reference model; standard KL regularization costs over 10 proxy-reward points at equal compute.Figure 4 presents the learning-curve and out-of-distribution comparison; Table 1 covers proxy and unseen rewards.Source: Figure 4 · p. 7, §5.1 · p. 7, Table 1 · p. 7
  • At equal training steps, GARDO follows prompts with high-quality images, whereas vanilla GRPO produces noisy images with blurred backgrounds and Gibbs artifacts.Figure 6 compares qualitative generations across prompts.Source: §5.1 · p. 7, Figure 6 · p. 8
  • GenEval diversity scores rise from 19.98 to 24.95 with diversity-aware optimization.Figure 5 visualizes the diversity improvement.Source: Figure 5 · p. 7
  • Gated and adaptive KL alone matches the KL-free baseline’s convergence speed and reaches a 0.91 OCR score at 600 steps without compromising unseen rewards.The reference is dynamically updated, while KL targets high-uncertainty samples; Figure 8a shows around 10% are penalized, and Figure 7 shows examples.Source: §5.1 · p. 8, Figure 7 · p. 8, Figure 8 · p. 9
  • GARDO’s unregularized multiple-reward comparison has significantly lower sample efficiency on the primary OCR proxy than GARDO.The comparison uses 0.8×OCR + 0.1×Aesthetic + 0.1×ImageReward.Source: §5.1 · p. 8

5.2. Emerging Behavior

  • Trained on prompts containing 1–9 objects, GARDO significantly improves counting accuracy for 10–11 objects, where the base model consistently fails and has near-zero accuracy on 10.Figure 9 visualizes GARDO counting 11 objects; Table 2 reports counting accuracy.Source: §5.2 · pp. 8–9, Table 2 · p. 9, Figure 9 · p. 9

6. Conclusion and Limitations

The conclusion presents GARDO’s gated and adaptive regularization and diversity-aware strategy, while identifying auxiliary reward models and video scalability as limitations.

6. Conclusion and Limitations

  • GARDO combines gated and adaptive regularization with diversity-aware optimization to address reward hacking in diffusion-model fine-tuning.The conclusion associates this approach with improved sample efficiency and emerging behaviors.Source: §6 · p. 9
  • The authors identify dependence on auxiliary reward models for uncertainty estimation as a primary limitation.Whether the approach scales to resource-intensive video generative models remains open.Source: §6 · p. 9
  • Scalability to resource-intensive video generative models remains an open question.The conclusion frames this as future investigation, not a demonstrated failure.Source: §6 · p. 9

A. Implementation Details

The implementation details specify training settings, adaptive and gated KL parameters, and the reported computation setup.

A.1. Details of GARDO

  • Training uses Flow-GRPO with timestep settings of 10 for sampling and 40 for evaluation, batch size 6, group size 24, learning rate 3e−4, and LoRA α = 32, r = 64.The same initial noise is used for images generated from a given prompt.Source: §A.1 · p. 13
  • The adaptive KL reference resets when KL loss exceeds 1e−4 or after 100 gradient steps since the previous reset.Source: §A.1 · p. 13
  • Gated KL initially penalizes the 10% of samples with highest uncertainty, then adjusts the percentage using uncertainty observations cached over a 20-epoch window.The percentage increases by 10% above the cached maximum and decreases by 10% below the cached minimum.Source: §A.1 · p. 13

A.2. Computation Specification

  • Training uses 8 NVIDIA A800 GPUs and takes about 49 seconds per gradient step, including sampling and training.Source: §A.2 · p. 13

A.3. Evaluation Metrics

The appendix describes the proxy and unseen evaluation tasks and outlines the GARDO training procedure. It also reports a DiffusionNFT GenEval comparison favoring GARDO on sample efficiency and out-of-distribution rewards.

A.3. Evaluation Metrics

  • GenEval evaluates six compositional image-generation tasks, while OCR measures text-rendering accuracy using Flow-GRPO training and test sets.GenEval uses the Flow-GRPO experimental protocol and official evaluation pipeline.Source: §A.3 · p. 13
  • The GARDO algorithm samples prompt groups, generates images from shared initial noise, obtains advantages and uncertainty estimates, and applies a percentile-based KL gate before updating the policy.The reference is reset when the KL threshold or maximum reset-step condition is reached, and the gate percentage adapts to the uncertainty window.Source: §A.3 · p. 13
  • The OCR reward is max(1 − Ne/Nref, 0), where Ne is minimum edit distance and Nref is the target character count.Source: §A.3 · p. 13
  • On DiffusionNFT GenEval tasks, the caption reports GARDO surpassing baselines in both proxy-reward sample efficiency and out-of-distribution rewards.Source: Table 3 · p. 14
  • DrawBench uses 200 prompts across 11 categories, with four generated images per prompt; Aesthetic, PickScore, ImageReward, ClipScore, and HPSv3 assess unseen performance.Diversity is measured as mean pairwise cosine distance between Dinov3 image embeddings.Source: §A.3 · p. 14

B.1. Results on DiffusionNFT [70]

On DiffusionNFT, GARDO achieves the strongest balance between sample efficiency and reward-hacking mitigation, while retaining or improving unseen rewards and diversity.

B.1. Results on DiffusionNFT [70]

  • At 400 steps, GARDO reaches 0.95 GenEval accuracy without reducing unseen rewards versus the reference model and has the highest diversity among baselines.It also boosts an unseen reward such as Aesthetic.Source: §B.1 · p. 14
  • At 400 steps, KL-regularized DiffusionNFT reaches only 0.72 accuracy on the GenEval proxy task, reflecting compromised sample efficiency.Source: §B.1 · p. 14
  • Vanilla DiffusionNFT still shows reward hacking, with reduced performance on the unseen Aesthetic, picksocre, and HPSv3 tasks.Source: §B.1 · p. 14

B.2. Results on Flux.1-dev

On Flux.1-dev with HPSv2 as the proxy reward, GARDO retains high sample efficiency and performs best in the trade-off with unseen rewards.

B.2. Results on Flux.1-dev

  • GARDO achieves higher sample efficiency than KL-regularized Flow-GRPO on Flux.1-dev.Source: §B.2 · p. 14
  • GARDO outperforms both Flow-GRPO and the base model on unseen rewards in this Flux.1-dev evaluation.HPSv2 is the proxy reward, and HPDv3 is the training dataset.Source: §B.2 · p. 14
  • On Flux.1-dev, GARDO performs best in the trade-off between HPSv2 proxy reward and out-of-distribution unseen rewards.Source: Figure 10 · p. 14, §B.2 · p. 14

B.3. More Qualitative Results

Across the training process, GARDO maintains visual quality without sacrificing proxy-reward performance, while Flow-GRPO and GRPO show reward-hacking artifacts.

B.3. More Qualitative Results

GARDO (Ours)

  • Figure 14 presents GARDO-generated images along the training process.Source: Figure 14 · p. 17

Related papers

Loading 2512.24138v1…