RRM: Robust Reward Model Training Mitigates Reward Hacking

1 INTRODUCTION

• Reward hacking occurs when a model maximizes its reward without aligning with intended human preferences, reflecting the reward model’s status as an imperfect proxy for judgment.The introduction also links this challenge to out-of-distribution generalization.Source: §1 · p. 1
• Traditional reward-model training can conflate prompt-dependent quality signals with context-free response artifacts because it uses only contextual response pairs and cannot control every artifact.The paper notes that verbosity, formatting, and response patterns can act as artifacts.Source: §1 · pp. 1–2
• The paper formulates reward-model training causally and augments training data with responses from other examples to balance artifacts between chosen and rejected responses.The stated aim is to learn real quality rather than artifact patterns.Source: §1 · p. 2, Figure 1 · p. 2
• The paper identifies distinguishing contextual preference signals from spurious artifacts and improving policies trained on robust reward models as its central contributions.It also presents a causal graph and data augmentation to mitigate artifacts learned by the reward model.Source: §1 · p. 2

2 PRELIMINARIES

• Preference learning treats human judgments as an oracle that assigns a probability that one response is preferred over another for a prompt.The goal is to optimize preference under a query-budget constraint.Source: §2 · p. 2
• Bradley–Terry pointwise reward models derive pairwise preference probabilities from response rewards, while pairwise ranking models directly predict preferences from a prompt and response pair.The paper uses a pairwise ranking model, estimating preference from the probability of decoding “A.”Source: §2 · p. 3
• Preference-alignment algorithms optimize a policy against a reward function and reference policy; this work uses DPO for stable evaluation.The cited algorithms include PPO, SLiC, DPO, RSO, and IPO.Source: §2 · p. 3
• Reward hacking has been addressed through artifact-specific methods, reward-model consistency and ensembles, policy-training reward adjustments, and model merging; this paper focuses on a causal reward-model approach.The related-work discussion notes that ensembles can mitigate, but not eliminate, reward hacking.Source: §2 · p. 3
• Causal inference represents variables and causal relationships with a directed acyclic graph, using d-separation to express conditional independence.Under the stated definition, d-separation corresponds to conditional independence for distributions represented by the graph.Source: §2 · pp. 3–4

3.1 CAUSAL FRAMEWORK

• The DAG distinguishes contextual signal S, which depends on the prompt and both responses, from artifact A, which depends only on the responses.Under H0, the framework specifies conditional independence between A and C given the responses, or given S.Source: §3.1 · pp. 4–5
• Traditional reward-model training cannot always distinguish whether A affects preference labels, including when A and S are perfectly correlated.The proof illustrates that perfect correlation permits reparameterization that makes the two hypotheses indistinguishable.Source: §3.1 · p. 4

3.2 DATA AUGMENTATION

• Augmentation permutes responses across examples: contextual responses win, two non-contextual responses tie, and original preferences govern when both fit the prompt.The construction uses fixed-response prompt variation and fixed-label variation of the losing response to perturb contextual signals and artifacts.Source: §3.2 · p. 5
• Training with the augmented triplets entails no causal edge from response artifacts to preference labels in the paper’s DAG.This is Proposition 3.2’s conclusion for training on the original dataset and the specified augmented triplets.Source: §3.2 · p. 6

3.3 CONNECTION TO EXISTING WORKS

• The framework treats ODIN as a special case for one observed artifact, length, and extends beyond single observed artifacts.The paper also describes its artifact-residual approach as equivalent to length-controlled AlpacaEval-2, while allowing extension to unobserved artifacts.Source: §3.3 · p. 6
• Unlike length-adjusted policy optimization, the proposed artifact-free reward model does not require an explicit length-adjustment factor in alignment.The paper also contrasts its cross-example combinations with prior consistency-focused augmentation and retrieval-augmented inference.Source: §3.3 · p. 6

4.1 SETTINGS

• RRM uses a 700K-pair RLHFlow mixture; Gemma-2-9b-it training filters augmented pairs into 2.4M examples before one-epoch AdamW training.DPO policies use on-policy Gemma-2-9b-it responses labeled by RM or RRM and best-worst response pairs.Source: §4.1 · pp. 6–7

4.2 MAIN RESULTS

• 3.54% absolute accuracy gain over RM is reported for RRM on Reward-Bench; it improves Chat Hard and Safety but sacrifices Reasoning.Source: §4.2 · p. 7, Table 2 · p. 8
• RRM has the best MT-Bench first-turn and AlpacaEval-2 performance over ODIN and RM, with shorter outputs than RM; neutral examples slightly improve both.Source: Table 3 · p. 8, §4.2 · p. 8

4.3 LENGTH ANALYSIS

• About 60% of RM-training chosen responses are longer; RRM training data has more balanced chosen/rejected lengths across bins.Source: §4.3 · p. 8, Figure 3 · p. 9
• Across AlpacaEval-2 prompts, policies induced by RRM generate shorter responses than RM, consistent with correction of lengthy bias.Source: §4.3 · p. 8, Figure 4 · p. 9

4.4 DELIBERATELY DESIGNED ARTIFACTS

• BoN policies induced by RRM are more robust than RM to injected response artifacts.The experiments add a prefix artifact to chosen training responses and vary artifact injection probability in policy responses from 0.05 to 0.5.Source: §4.4 · p. 9

5 RELATED WORKS

• RRM-induced BoN policies are more robust than RM-induced policies to injected response artifacts.Figure 5 tests injected-artifact rates of 5%, 10%, 20%, and 50% on AlpacaEval-2 prompts.Source: Figure 5 · p. 10
• Direct alignment algorithms optimize a supervised target without first constructing a reward model, while reward-model-based methods rank generated responses for best-of-n sampling.The passage notes that fixed-dataset direct alignment is offline and often off-policy, whereas reward-model methods can distill from selected responses.Source: §5 · p. 10
• Prior reward-hacking approaches include model merging, reward decomposition, and methods addressing observable artifacts such as length.The paper positions RRM as addressing the distinction between prompt-driven preferences and context-free artifacts.Source: §5 · p. 10

6 CONCLUSION

• The proposed techniques enhance reward-model test accuracy and the quality of the policies they induce.The conclusion attributes the approach to a causal framework and a data-augmentation method derived from it.Source: §6 · p. 10
• The identified problem is that current reward training cannot distinguish contextual signals from context-free artifacts.Source: §6 · p. 10
• Future work will explore filtering augmented pairs and matching artifacts when constructing response pairs.Source: §6 · p. 10

A.1 ADDITIONAL LENGTH ANALYSIS OF REWARD MODEL TRAINING DATASETS

• SHP, HelpSteer, and UltraFeedback have longer chosen responses than rejected responses on average.Source: §A.1 · p. 15
• HH-RLHF, SHP, HelpSteer, and UltraFeedback show bias toward rejected responses in the first length bin.Source: §A.1 · p. 15
• The analysis compares chosen and rejected response-length distributions for each individual data source.Source: §A.1 · p. 15

A.2 DATA AUGMENTATION PYTHON CODE

• The augmentation code takes each example’s context and preferred and rejected responses, and uses a neutral flag to mark tie labels.The sample code shuffles data copies and constructs augmented response combinations.Source: §A.2 · p. 15, §A.4 · p. 17

A.3 TRAINING DETAILS FOR ODIN

• ODIN is trained on Gemma-2-9b-it for one epoch with AdamW, a 2e-6 learning rate, cosine scheduling, and global batch size 128.Flash-Attention and DeepSpeed Zero-Stage 3 are used to support the stated batch size.Source: §A.3 · p. 15

A.4 PRELIMINARIES IN CAUSAL INFERENCE

• The appendix reports that SHP, HelpSteer, and UltraFeedback show lengthy bias toward chosen responses and together account for more than half of the training data.Source: Figure 6 · p. 16
• The causal Markov condition links a DAG to distributions in which each variable is conditionally independent of non-effects given its direct causes.Source: §A.4 · p. 17
• Counterfactuals are unobserved outcomes under an alternative treatment; the mean treatment effect compares expected outcomes under treatment and no treatment.The appendix identifies the two potential outcomes as Y0 and Y1.Source: §A.4 · pp. 17–18

• RRM shows consistent gains over RM when ranking Gemma-2-9b-it responses in AlpacaEval-2 best-of-n policies.Table 5 describes quality improvements with shorter responses over RM across aligned policies.Source: §A.5 · p. 18, Table 5 · p. 18
• RewardBench shows an average 1.75% accuracy improvement for RRM over RM on Chat and Chat Hard.The authors also report regressions on Safety and Reasoning, hypothesizing that context-free aspects of those tasks may contribute.Source: Table 4 · p. 18, §A.5 · p. 18

• A mixed-artifact experiment wraps each chosen response in bold with p = 0.1 and trains RM and RRM on the resulting dataset.Source: §A.6 · p. 18
• BoN policies induced by RRM are more robust than RM-induced policies to injected response artifacts.The emoji experiment adds emoji to sampled Gemma-2-9b-it responses at rates from 5% to 50%.Source: §A.6 · p. 18

A.7 DISCUSSION ON DATA FILTERING STRATEGIES

• BoN policies induced by RRM are more robust to injected emoji artifacts than the comparison policies.The figure compares artifact rates across policies after emoji are appended to responses at varying rates.Source: Figure 7 · p. 19
• Using rejected responses as new chosen examples or assigning 0.5 winning probability to safe–unsafe pairs may cause unwanted AI behavior.The discussion notes that preference labels may sometimes appropriately reflect prompt-independent signals such as safety or diversity.Source: §A.7 · pp. 18–19
• Proposed future safeguards include filtering chosen responses for safety and using AI feedback to ensure augmented examples meet quality and non-contextual preference rules.The suggested rules include safety, style, and factuality.Source: §A.7 · p. 19

B ETHICS STATEMENT

• The ethics statement frames avoiding response-length and format biases as a fairness objective.Source: B ETHICS STATEMENT · p. 19

• Training code, experimental datasets, hyperparameters, and evaluation approaches are stated to be publicly available for reproducibility.Source: C REPRODUCIBILITY STATEMENT · p. 19