PaperlayerView exact revision on arXiv

Source-linked AI summary

Adversarial Training of Reward Models

Alexander Bukharin, Haifeng Qian, Shengyang Sun, Adithya Renduchintala, Soumye Singhal, Zhilin Wang, Oleksii Kuchaiev, Olivier Delalleau, Tuo Zhao

arXiv:2504.06141v2cs.LG

TL;DR

Reward models can give high scores to out-of-distribution, low-quality responses, creating a reward-hacking risk. Adv-RM uses reinforcement learning to find such responses and add them to reward-model training; the resulting models support three times as many RLHF steps without reward hacking compared with conventional reward models.

  • Problem

    Reward models may score out-of-distribution, low-quality responses highly, enabling reward hacking that reduces alignment with human values.

    Source: §1 · p. 1

  • Method

    Adv-RM trains a reinforcement-learning policy to find OOD, low-quality responses that receive high scores from the target RM, then incorporates the adversarial samples into RM training.

    Source: §1 · p. 2

  • Results

    Three times as many RLHF steps as conventional-RM runs proceeded without reward hacking, and the resulting aligned models were more performant.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Adversarial examples can augment RM training without additional human annotation by pairing an in-distribution response as chosen with an adversarial response as rejected.

    Source: §1 · p. 2

    Adv-RM relies on reward ensembles and may miss adversarial responses when all ensemble RMs incorrectly predict they are high quality.

    Source: §6 · p. 10

Abstract

from arXiv · show

Reward modeling has emerged as a promising approach for the scalable alignment of language models. However, contemporary reward models (RMs) often lack robustness, awarding high rewards to low-quality, out-of-distribution (OOD) samples. This can lead to reward hacking, where policies exploit unintended shortcuts to maximize rewards, undermining alignment. To address this challenge, we introduce Adv-RM, a novel adversarial training framework that automatically identifies adversarial examples -- responses that receive high rewards from the target RM but are OOD and of low quality. By leveraging reinforcement learning, Adv-RM trains a policy to generate adversarial examples that reliably expose vulnerabilities in large state-of-the-art reward models such as Nemotron 340B RM. Incorporating these adversarial examples into the reward training process improves the robustness of RMs, mitigating reward hacking and enhancing downstream performance in RLHF. We demonstrate that Adv-RM significantly outperforms conventional RM training, increasing stability and enabling more effective RLHF training in both synthetic and real-data settings.

1 Introduction

Adv-RM uses reinforcement learning to find high-reward, OOD, low-quality responses and incorporates them into reward-model training. The resulting models are more robust to reward hacking, allowing RLHF to run three times as many steps as with conventional reward models without hacking.

1 Introduction

  • Measuring RM robustness is difficult because language attacks require adversarial preference pairs, while measuring reward hacking in RLHF can require costly human evaluation.Small misalignments can go undetected in human evaluation.Source: §1 · pp. 1–2
  • RMs can assign high scores to OOD responses, which may be low quality; uncertainty-based regularization may be limited because uncertainty does not reliably indicate in-distribution response quality.Reward hacking occurs when policies exploit unintended shortcuts in RM scores, reducing alignment with human values.Source: §1 · pp. 1–2
  • Adv-RM trains an adversarial policy with reinforcement learning to find OOD, low-quality responses that the target RM scores highly; attacks on SOTA RMs exceeded 80% success.Examples included responses without punctuation and responses containing random text.Source: §1 · p. 2
  • Three times as many RLHF steps as conventional-RM runs proceeded without reward hacking, resulting in more performant aligned models.Adv-RM models also showed greater robustness to attacks and reward hacking.Source: §1 · p. 2
  • Adversarial samples augment RM training without additional human annotation, using an in-distribution response as chosen and the adversarial response as rejected.The evaluation covered adversarial attacks, synthetic RLHF, and RLHF on real datasets.Source: §1 · p. 2
  • The RL-based attack approach generates adversarial samples without human domain knowledge, providing a means to evaluate RM robustness.The authors describe this as the first such approach to their knowledge.Source: §1 · p. 2

2 Related Work

Prior work addresses reward hacking with reward ensembles, reward-model changes, and adversarial attacks or training. The cited approaches have limits involving prevention, human annotation, task scope, or suitability for reward-model preference pairs.

2 Related Work

  • Reward ensembles can slow reward hacking but do not prevent it; the experiments compare MEAN and MEAN MINUS STD aggregation.RRM augments training data with random responses from other prompts, an approach the authors find effective.Source: §2 · pp. 2–3
  • Amodei et al. proposed adversarial reward models using RM–human disagreement but did not implement the idea; Kirchner et al.'s verifier approach is limited to rule-based tasks.The proposed reliance on human annotators was described as expensive and difficult; Wu et al. use manually defined input transformations for RM benchmarks.Source: §2 · p. 3
  • Textfooler synonym replacement and StyleAdv style changes usually do not attack RMs successfully because they ignore paired preference data and target weaker models such as BERT.Source: §2 · p. 3

3 Methodology

The methodology uses standard RLHF components and ensemble disagreement to train policies that expose high-reward, OOD responses. It then uses those responses to train reward models, guided by evidence that uncertainty is more informative about quality for high-uncertainty samples.

3.1 Background

  • Standard RLHF trains a Bradley–Terry reward model on human preference pairs, then optimizes a policy for reward while keeping it close to the initial SFT policy.Policy optimization can use reinforcement-learning algorithms such as PPO, RLOO, or GRPO.Source: §3.1 · p. 3

3.2 Learning to Attack

  • Adv-RM trains an adversarial policy with reinforcement learning to find responses that score highly under a target RM but differ from another RM’s assessment.The method uses a prompt-dependent reward threshold and a weighted score difference to encourage high target scores alongside disagreement and low scores from the second RM.Source: §3.2 · p. 4
  • Adv-RM filters generated responses for high target reward and uncertainty above the SFT-response baseline, marking them as high-reward, OOD attacks.The uncertainty cutoff is a z-score greater than 1.96 relative to responses sampled from πSFT.Source: §3.2 · p. 4

3.3 Preliminary Analysis: Uncertainty as a Quality Signal

  • Uncertainty correlates with gold quality at -0.70 for Adv-RM samples, but only -0.05 for πSFT samples, making it more informative among high-uncertainty responses.The figure compares uncertainty with gold score and shows similar target-RM scores for Adv-RM and SFT data.Source: Figure 1 · p. 5, §3.3 · p. 5

3.4 Adversarial Training of Reward Models

  • Adversarial preference pairs mark selected SFT responses as preferred and higher-reward, high-uncertainty adversarial responses as rejected; iterative retraining stops when attacks no longer succeed after two rounds.Each round adds around 1,000 pairs to the current dataset and trains a new reward model from scratch.Source: §3.4 · p. 5

4 Experiments: Adversarial Attacks on Reward Models

The experiments evaluate whether Adv-RM can find adversarial preference pairs and compare its attacks with baselines across synthetic and real reward-model settings. Adv-RM attacks all three tested state-of-the-art real reward models, while results also show model-specific attack patterns and limited transfer.

4.1 Evaluation

  • The evaluation defines success through a preference pair where the target RM favors a response that the gold reward judges negatively; synthetic evaluation adds a stricter criterion when responses are similar.For human or LLM judges, the study uses criterion (4); in synthetic settings, success intuitively means above-average SFT reward but a gold score in the bottom 5% of SFT scores.Source: §4.1 · pp. 5–6

4.2 Experimental Setup

  • The synthetic setup uses Nemotron-70B-Reward as the gold RM and proxy reward models trained on relabeled Helpsteer-2-Preferences; real attacks target three top RewardBench models.Real-setting attack success is evaluated with human judges, Llama 405B, and Deepseek-R1.Source: §4.2 · p. 6
  • The synthetic protocol uses the gold reward model for attack-success evaluation, whereas real-setting evaluation relies on human or model judges as proxies for gold reward.The paper notes that LLM judges may themselves be fooled, potentially underestimating attack success.Source: §4.2 · p. 6, §4.3 · p. 8

4.3 Main Results: Adversarial Attacks

  • In the synthetic setting, Adv-RM consistently finds adversarial samples with four attacks per prompt, while TextFooler and StyleAdv have low success despite generating 50 attacks per prompt.Over-optimized policies can produce successful attacks, but their responses are often high quality and only mildly quirky.Source: §4.3 · pp. 6–7, Table 1 · p. 6
  • Adv-RM attacks all three tested state-of-the-art reward models, with each target RM favoring adversarial pairs that human or model judges reject.LLM judges may underestimate attack success because they can also be fooled by adversarial examples.Source: Table 2 · p. 7, §4.3 · pp. 7–8
  • Adv-RM exposes distinct reward-model vulnerabilities: prompt repetition for Skyworks-Gemma, mid-text gibberish for Llama-Nemotron, and missing punctuation for Nemotron 340B.Figure 2 presents adversarial examples for top RewardBench models; its Z-scores normalize reward against the average Llama-3.1-8b-Instruct reward for each prompt.Source: Figure 2 · p. 7, §4.3 · p. 8
  • Attacks transfer weakly between Skyworks 27B models and not between the different Nemotron reward models.Figure 3 presents the attack-transferability results.Source: §4.3 · p. 8, Figure 3 · p. 8
  • The synthetic-setting ablation indicates that all components of Adv-RM are helpful.Table 3 reports the attack ablation.Source: §4.3 · p. 8, Table 3 · p. 8

5 Experiments: Downstream Policy Performance

Adv-RM-trained reward models support more stable downstream RLHF than the reported baselines in both synthetic and real settings. Repeated adversarial training improves attack robustness, while gains depend on the amount of adversarial data and training rounds.

Experimental Setup

  • The experiments optimize Llama-3.1-8B-Instruct policies with RLOO and evaluate synthetic performance by Gold Reward and real-data performance on 128 held-out prompts using multiple judges.Baselines include conventional RLHF, ensemble mean, ensemble mean minus uncertainty (Ens (STD), also called UWO), and RRM.Source: §5 · p. 8

Results: Synthetic Setting

  • In the synthetic setting, Adv-RM policies achieve higher gold reward with minimal reward hacking, little length hacking, and performance at least as good as the baselines.The reported gain is mainly greater robustness to reward hacking, with only a small improvement in best-case policy performance.Source: §5 · p. 8, Figure 4 · p. 9

Results: Real RLHF Setting

  • Adv-RM improves over the initial model for 2–3 times more steps than RLHF and produces the most stable real-data training among the baselines.RRM is a strong baseline, but its training is less stable; Adv-RM and RRM are described as complementary.Source: §5 · p. 8, Figure 5 · p. 9

Robustness Against Attacks

  • Policies trained with multiple rounds of Adv-RM become increasingly robust to attacks.The robustness results are shown in the real RLHF setting with Deepseek-R1 as judge.Source: §5 · pp. 8–9, Figure 6 · p. 9

Ablation

  • Around 1,000 Adv-RM samples yield the best downstream performance, while adding more hurts; two training rounds generally perform best, with minimal gain afterward.The authors hypothesize that limited diversity in Adv-RM data may cause overfitting when too much is added.Source: §5 · p. 9, Figure 6 · p. 9

6 Conclusion and Limitations

Adv-RM attacks state-of-the-art reward models and its examples improve RLHF stability and performance when incorporated into reward-model training. The approach relies on reward ensembles and incurs substantial computational overhead.

6 Conclusion and Limitations

  • Adv-RM uncovers cases where state-of-the-art reward models give high scores to low-quality responses, and incorporating its examples improves RLHF stability and performance.The conclusion introduces Adv-RM as an adversarial training framework for reward models.Source: §6 · p. 9
  • Adv-RM may miss adversarial responses if all reward models in its ensemble incorrectly predict they are high quality, and costs approximately 3× standard RLHF training.The authors identify more robust out-of-distribution detection as a need and consider the computational overhead justified by robustness gains.Source: §6 · p. 10

C Adversarial Training Hyperparameters

The appendix reports shared settings for adversarial-policy and reward-model training, including optimizer, batch-size, and checkpoint-selection choices.

C Adversarial Training Hyperparameters

  • Adversarial policies use Adam with a constant learning rate of 5e −7, batch size 64, four samples per prompt in RLOO, λ1 = 1, tuned λ2 ∈[8, 10, 12], and C = −25.The adversarial policy is initialized from llama-3.1-8B-instruct.Source: C Adversarial Training Hyperparameters · p. 14
  • Reward models train for one epoch with batch size 128 and learning rates tuned in [1e −6, 5e −6, 7e −6, 1e −5].Intermediate checkpoints are selected by RewardBench scores, with evaluation every 10 steps.Source: C Adversarial Training Hyperparameters · p. 14

D Adversarial Attack Baselines

The appendix describes four attack baselines and their construction: synonym substitutions, style rewrites, response swaps, and extended policy optimization.

Textfooler

  • Textfooler generates 100 attacks per prompt-response pair by replacing individual words with synonyms, then retains attacks that most increase the reward-model score.The approach follows Jin et al. (2020).Source: D Adversarial Attack Baselines · p. 14

StyleAdv

RRM

RM over-optimization

  • RM over-optimization trains policies for three times the steps needed to reach best performance: 150 steps in the synthetic setup and 240 in the real-data setup.Best performance occurs at 50 steps in the synthetic setup.Source: D Adversarial Attack Baselines · p. 14

E Robust Policy Optimization Baselines

The baselines optimize reward models using either the mean ensemble reward or a mean-minus-standard-deviation target; RRM augments the training data before policy optimization.

E Robust Policy Optimization Baselines

F Computational Cost.

Experiments report computational costs, describe judge- and human-based attack evaluations, and present RewardBench scores in Table 4.

F Computational Cost.

  • Conventional RLHF takes 224 A100 hours, while Adv-RM training with two rounds takes 672 A100 hours.Experiments used Nvidia A100 80G GPUs and NeMo-Aligner.Source: F Computational Cost · p. 14

G Evaluating Adversarial Attacks

  • The LLM-judge evaluation uses Llama 3 405B and Deepseek-R1 to rank response pairs, with preference required to hold in both pair orders across 128 prompts.The prompt requests a one-sentence comparison of helpfulness and harmlessness, followed by the selected response letter.Source: G Evaluating Adversarial Attacks · p. 15
  • Human evaluation uses four annotators, blinded to attack method, who label pairs as better, worse, or similar in quality; each method has 50 annotated samples.Source: G Evaluating Adversarial Attacks · p. 15

H Reward Bench Scores

I Adversarial Policy Training Curve

The synthetic-setting training curve shows πadv increasing Rθ1 while minimizing Rθ2, generating OOD responses with low gold reward that qualify as adversarial samples.

I Adversarial Policy Training Curve

Loading 2504.06141v2…