Towards Understanding the Influence of Reward Margin on Preference Model Performance

1 Introduction

• Reward models can be exploited as policy inputs evolve away from the static human preference data used for training.The introduction identifies this mismatch as a basis for reward hacking or reward over-optimization.Source: §1 · p. 1
• Out-of-distribution policy drift and ambiguous or incorrect preferences challenge reward-model accuracy; reported inter-annotator agreement for InstructGPT is 72.6%.The introduction attributes preference uncertainty partly to low agreement among annotators.Source: §1 · p. 1
• Traditional ranking objectives order responses but do not quantify the degree of superiority, limiting their ability to distinguish real-world preference differences.The authors argue this shortcoming limits accurate capture of genuine human preferences.Source: §1 · pp. 1–2
• A margin score quantifies preference differences and is intended to mitigate noisy labels by assigning more discrepant scores to substantially different generations.The introduction says the approach may make the model less susceptible to closely ranked but misleading responses.Source: §1 · p. 2
• Reward confidence offers a way to estimate preference differences without detailed, exhaustive human labels.The method uses confidence levels to explore subtle differences in preference.Source: §1 · p. 2
• Margin-based training improves reward-model effectiveness and reward prediction accuracy, with evaluation also comparing Best-of-N win rates against a baseline.The introduction describes annotating actual margins between selected and non-selected responses.Source: §1 · p. 2

2.1 Reward Modeling

• The paper focuses on preference margins within the reward model, whereas Wang et al. (2024) analyze how varying preference strengths in data affect reward-model performance.The cited analysis centers on incorrect, ambiguous, and normal preferences; the paper presents its margin focus as applicable across data types.Source: §2.1 · p. 2

2.2 Reward Hacking

• Reward hacking can yield high reward scores without intended criteria, producing linguistically poor, overly verbose, or preference-misaligned outputs and complicating checkpoint selection.The passage connects this dynamic to Goodhart’s Law (Gao et al., 2023).Source: §2.2 · pp. 2–3
• Prior reward-hacking approaches include smaller models for learning reward biases, policy-weight interpolation, and reward-model ensembles, which mitigate but do not eliminate hacking.The paper instead focuses on the reward-model learning objective.Source: §2.2 · p. 3

3 Preliminaries

• RLHF is presented as supervised fine-tuning, preference sampling with reward-model training, and reinforcement-learning fine-tuning using PPO.The paper says its focus is the latter two stages.Source: §3 · p. 3
• Human evaluators choose between two responses to a query, yielding chosen and rejected responses for comparative preference modeling.The responses are generated by the supervised fine-tuned model.Source: §3 · p. 3
• The preference setup is treated as binary classification and leads to a negative log-likelihood loss.The reward network is initialized from the SFT model and gains a linear layer that produces a scalar reward.Source: §3 · p. 3

4.1 Evaluating the Different Margin Values on Performance of Reward Models

• The reward model is trained to score chosen responses above rejected ones, with the difference rψ(x, yc) − rψ(x, yr) quantifying their preference discrepancy.The section describes assigning GPT-4-labeled margins to query-response pairs and incorporating those scores into a new reward model.Source: §4 · p. 3, §4.1 · pp. 4–5

4 Investigating the Reward Margin on Preference Modeling

• Model reward-margin distributions have positive means and rightward skew; more effective models show lower kurtosis, indicating a broader distribution of margins.The authors also report instances with negative margins, potentially due to dataset noise or anomalies.Source: §4 · pp. 3–4

4.1 Evaluating the Different Margin Values on Performance of Reward Models

• GPT-4 annotations of an HH subset showed a range of preference degrees, unlike Figure 1(a) predictions clustered near zero, which suggest weak confidence and differentiation.The labels distinguish four categories: Distinctly Superior, More Effective, Slightly Better, and Negligibly Better/Unsure.Source: §4.1 · pp. 4–5, Figure 2 · p. 4

4.1.1 Experiment Results

• The model trained with detailed margin scores substantially outperformed the model without margin information on the GPT-4 annotation set and HH test set.Table 1 reports accuracy for both evaluation sets, but the supplied passages do not give the accuracy values.Source: Table 1 · p. 5, §4.1.1 · p. 5
• A wider reward margin tends to separate response quality more distinctly, while narrow margins accompany difficulty distinguishing high- from low-quality responses and a plateau in performance improvement.The authors present the final reward-difference distribution in Figure 1(b).Source: §4.1.1 · p. 5

• Pairwise binary preference labels are efficient but miss the range of preference differences, which is difficult and costly to annotate directly.Source: §5 · p. 5
• The method estimates preference differences from a local approximation of reward confidence, avoiding the need for exhaustive human labels.Source: §5 · p. 5
• Batch-margin equations aim to increase predicted preference differences while supporting computational efficiency and real-time updates.The batch loss uses the mean margin of the batch.Source: §5 · pp. 5–6
• Threshold filtering applies the margin-adjusted loss to samples below the batch-average margin and retains the original loss for samples above it.This design is intended to shift the overall margin right while preserving the original loss for higher-margin samples.Source: §5 · p. 6

6.1 Setups

• Evaluation uses six preference datasets and reward models ranging from Pythia-410M to Llama-7B-chat.The study assesses accuracy and reward distributions; dataset-specific train/test divisions are used.Source: §6.1 · p. 6, §6.2 · p. 6

6.2 Main performance

• The method consistently improves accuracy across models and preference benchmarks.Table 2 reports accuracy (%) across preference benchmarks; the supplied results text gives no table cell values.Source: §6.2 · p. 7, Table 2 · p. 7
• Llama-7B-chat has the highest accuracy, but its relative improvement over baseline is smaller than Pythia-410M’s.The authors suggest method efficiency may plateau at higher model scales, warranting further examination.Source: §6.2 · p. 7

7 Experiment of Best of N

• Best-of-N generates multiple SFT responses per input and selects the one with the highest predicted reward.The evaluation uses 1B and 6.9B policy models, several reward-model sizes, and GPT-4 plus a larger reward model to judge helpfulness and harmlessness.Source: §7 · p. 7, §7.1 · pp. 7–8

7.2 Experimental results

• Win rates initially improve with N, but each reward model has a peak after which gains plateau or decline.Pythia-410M peaks at N=64 and Pythia-2.8B at N=8; all models exceed a 50% win rate.Source: Figure 3 · p. 8, §7.2 · p. 8
• The optimal N is not directly proportional to reward-model size, and larger policy models may benefit less when they substantially exceed reward-model size.The upward trend is notably pronounced for Pythia-410M in Figure 3(b).Source: §7.2 · p. 8

8 Conclusion

• The method adaptively adjusts reward margins based on reward confidence, avoiding the need for exhaustive human labels.The conclusion links margin settings to distinguishing high- and low-quality responses without overly penalizing subtle quality variations.Source: §8 · p. 8