PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking the Role of Proxy Rewards in Language Model Alignment

Sungdong Kim, Minjoon Seo

arXiv:2402.03469v3cs.LGcs.AIcs.CL

TL;DR

Proxy reward models can make reliable RL training difficult, and their behavior remains a black box. This paper constructs interpretable white-box rewards and tests whether their scores track Gold rewards after RL training. The reported results identify different response qualities for open- and closed-ended queries and show competitive alignment-benchmark performance alongside strong open-source reward models.

  • Problem

    Reliable RL training with proxy reward models is difficult, and optimizing imperfect proxy rewards can raise estimated scores while true rewards stagnate or decline.

    Source: §1 · p. 1

  • Method

    The paper uses interpretable features to construct white-box proxy rewards and evaluates whether proxy and Gold scores have a monotonic relationship after RL training.

    Source: §1 · p. 2

  • Results

    Models using the engineered reward show competitive alignment-benchmark performance alongside robust open-source reward models, with relevant, sufficiently detailed open-ended responses and consistent closed-ended responses key to imitating Gold.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The engineered white-box reward is a potential strong alignment baseline that does not require a pairwise human-feedback dataset.

    Source: §1 · p. 2

    The study uses StarlingRM-34B rather than labeled ground-truth rewards, so it cannot guarantee that the Gold model accurately represents ground truth.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Learning from human feedback via proxy reward modeling has been studied to align Large Language Models (LLMs) with human values. However, achieving reliable training through that proxy reward model (RM) is not a trivial problem, and its behavior remained as a black-box. In this paper, we study the role of proxy rewards in the LLM alignment via `reverse reward engineering' by composing interpretable features as a white-box reward function. We aim to replicate the ground truth (gold) reward signal by achieving a monotonic relationship between the proxy and gold reward signals after training the model using the proxy reward in reinforcement learning (RL). Our findings indicate that successfully emulating the gold reward requires generating responses that are relevant with enough length to open-ended questions, while also ensuring response consistency in closed-ended questions. Furthermore, resulting models optimizing our devised white-box reward show competitive performances with strong open-source RMs in alignment benchmarks. We highlight its potential usage as a simple but strong reward baseline for the LLM alignment, not requiring explicit human feedback dataset and RM training. Our code is available at https://github.com/naver-ai/rethinking-proxy-reward.

1 Introduction

The introduction frames proxy-reward alignment as difficult to trust because policies can exploit imperfect rewards, then proposes interpretable white-box rewards to study how proxy optimization relates to Gold rewards. The reported findings emphasize query-appropriate response qualities and describe the engineered reward as a potential alignment baseline.

1 Introduction

  • RLHF typically trains a proxy reward model from human feedback, then uses reinforcement learning to optimize a policy against its scores.The cited work presents RLHF as a dominant approach to using human feedback for alignment.Source: §1 · p. 1
  • Imperfect proxy rewards can be overoptimized: proxy scores rise while true rewards saturate or decline, and policies may exploit reward-model shortcuts.The introduction associates this shortcut exploitation with reward hacking.Source: §1 · p. 1
  • Reverse reward engineering uses interpretable features in a white-box proxy reward and tests whether proxy and Gold scores move monotonically after RL training.The paper frames this relationship as a test of whether maximizing the proxy during training also maximizes Gold reward at test time.Source: §1 · p. 2
  • Engineered rewards often outperform UltraRM-13B and work across backbones, while query-type branching improves win rates without unnecessary verbosity.Imitating Gold requires sufficiently lengthy, relevant, and faithful open-ended responses and consistent closed-ended responses; reward differentiation also improves preference while minimizing alignment tax.Source: §1 · p. 2
  • The paper presents its white-box reward as a potential strong baseline that does not require a pairwise human-feedback dataset.The introduction also reports that the approach can enhance preferences in several alignment benchmarks.Source: §1 · p. 2

2 Related Work

Related work describes how RLHF uses proxy rewards and how imperfect reward models and human-feedback biases complicate alignment. The paper draws on these concerns to motivate interpretable reward features.

2 Related Work

  • RLHF uses a proxy reward model to predict preferences and score sampled responses during the reinforcement-learning rollout.Prior work also explores synthetic feedback and multiple fine-grained rewards, including relevance, factuality, and information completeness for long-form question answering.Source: §2 · p. 2
  • Gao et al. (2023) describe overoptimization when policy gains in estimated reward fail to improve real-world reward; reward hacking exploits reward-model shortcuts.Related work also discusses efforts to mitigate overoptimization and improve reward consistency.Source: §2 · pp. 2–3
  • Human preference judgments can favor superficial style, and Singhal et al. (2023) report a high correlation between response length and RLHF training.The paper motivates interpretable features including length incentives, repetition penalties, query relevance, and query-type distinctions.Source: §2 · p. 3

3 Experimental Setup

The experimental setup defines reverse reward engineering as PPO training with an interpretable proxy reward, judged successful when proxy and gold rewards have a monotonic relationship. It builds the reward from response features and evaluates trained models against a gold reward model and alignment benchmarks.

3.1 Reverse Reward Engineering

  • Reverse reward engineering succeeds when PPO-trained models show a monotonic relationship between the designed proxy reward and the gold reward across validation evaluations.Detecting proxy-reward overoptimization counts as failure.Source: §3.1 · p. 3

3.2 Features for White-box Reward

  • The reward combines length incentive, repetition penalty, and query relevance to encourage longer, non-repetitive responses that remain on topic.Length and repetition rewards alone can promote irrelevant content; the repetition penalty uses the unique-trigram ratio.Source: §3.2 · p. 3
  • Reward branching uses query type to favor relevant, sufficiently long open-ended responses while using reference-answer relevance for closed-ended queries.The query type is automatically assigned with GPT-4, and the relevance interpolation maps to the length-incentive range for stable RL training.Source: §3.2 · p. 4
  • The authors report that multiplying reward functions works better empirically than adding them.The comparison is described as an empirical finding, with further details deferred to Appendix B.Source: §3.2 · p. 4

3.3 Evaluation setup

  • PPO experiments use Anthropic-HH and AlpacaFarm data, with StarlingRM-34B serving as the assumed gold reward model for proxy–gold comparisons.The comparison uses 805 AlpacaEval prompts; GPT-4 query-type decisions were manually checked on 100 samples, with about 83% agreement.Source: §3.3 · pp. 4–5
  • Resulting models are evaluated on Vicuna-Bench, AlpacaEval, MT-Bench, and Super-NaturalInstructions, alongside open-source reward-model baselines.Relevant Sentence Ratio measures the proportion of generated-response sentences judged on-topic by GPT-4.Source: §3.3 · p. 5

4 Experiments

The experiments test interpretable reward designs against gold-reward trends and alignment outcomes. RER’s relevance-aware, query-type branching tracks gold rewards and supports competitive preference gains while limiting alignment tax.

4.1 Reverse Reward Engineering

  • RER monotonically increases gold reward on both datasets and achieves the highest proxy–gold Spearman correlations.Considering relevance avoids the gold-reward drop; relevance alone does not improve gold reward.Source: §4.1 · pp. 5–6, Table 1 · p. 6
  • RER’s query-type branching improves gold rewards especially for closed-ended queries, while producing more consistent closed-ended and more diverse open-ended responses.Models without branching produce excessively diverse responses regardless of query type.Source: §4.1 · p. 6, Figure 4 · p. 7, Figure 5 · p. 7

4.2 Comparison with other open-source RMs

  • RER achieves competitive performance with open-source reward-model-trained systems, outperforming UltraRM-13B-trained models on Vicuna-Bench and MT-Bench.Table 2 compares reward designs with open-source RMs on Anthropic-HH.Source: Table 2 · p. 7, §4.2 · p. 7

4.3 Investigating Alignment Tax

  • Removing reference-answer relevance can raise preference while reducing relevance and SuperNI performance by about 12.3% points; length-only reward also worsens both.The paper reports more drastic drops in relevant-sentence ratio and SuperNI when query-type branching is discarded.Source: §4.3 · p. 8
  • RER improves AlpacaEval preference while retaining relevance and SuperNI instruction-following ability, indicating reduced alignment tax.Table 4 measures retained zero-shot instruction-following ability on SuperNI.Source: Table 4 · p. 8, §4.3 · p. 8

4.4 Generalizability of RER

  • RER improves AlpacaEval win rate by approximately 5-6% points for both Vicuna backbones, with proxy and gold rewards remaining monotonic through 10k PPO steps.The upward tendency saturates after 5k steps and often fluctuates; saturation occurs earlier, at 2k, for the 13B model.Source: §4.4 · pp. 8–9, Figure 6 · p. 8, Figure 7 · p. 9

5 Discussion and Future Works

Reward-model preference accuracy does not guarantee alignment performance. RER scores below other black-box RMs on preference accuracy but above UltraRM-13B on AlpacaEval and SuperNI.

5 Discussion and Future Works

  • The study reports a large preference-accuracy gap between its white-box rewards and black-box RMs.Table 5 presents preference accuracy for the RMs used in the study.Source: §5 · p. 9, Table 5 · p. 9
  • RER achieves higher AlpacaEval and SuperNI performance than UltraRM-13B despite lower preference accuracy.The paper notes that preference accuracy does not guarantee higher alignment performance.Source: §5 · p. 9
  • An imperfect reference model used in the KL-divergence term is proposed as one possible explanation for the mismatch, and further investigation is left for future work.Source: §5 · p. 9

6 Conclusion

The paper uses interpretable features to build a white-box reward for studying proxy rewards in LLM alignment. It reports that response relevance, sufficient detail for open-ended queries, and consistency for closed-ended queries are key to imitating gold rewards, with competitive alignment-benchmark performance.

6 Conclusion

  • Interpretable features form a white-box reward function for investigating proxy rewards in LLM alignment.Source: §6 · p. 9
  • Imitating gold rewards requires relevant, sufficiently detailed open-ended responses and consistent closed-ended responses.Source: §6 · p. 9
  • Models optimized with the proposed reward show competitive performance alongside robust open-source reward models on alignment benchmarks.The paper presents the reward's potential as a strong baseline.Source: §6 · p. 9

Limitations

The paper identifies limits in its reward-model reference and evaluation scope. It also describes automatic query-type annotation and compares reward-feature combination methods.

Limitations

  • The study uses StarlingRM-34B rather than labeled ground-truth rewards, and its validation uses about 800 AlpacaEval samples, leaving outcomes on larger, more varied datasets uncertain.The authors note that the gold RM may not accurately mimic ground truth, and that outcomes could differ for reasoning, harmlessness, or coding queries.Source: Limitations · pp. 9–10

A.1 Automatic annotation

  • GPT-4 automatically classifies queries as open-ended or closed-ended, using whether responses require creativity or consistency as the distinction.The annotation covers about 120k queries from Anthropic-HH, AlpacaFarm, and ShareGPT.Source: §A.1 · p. 13, Figure 8 · p. 13

B Combining Reward Functions

  • The authors use multiplication to combine reward functions as regularizers, reporting that it performs more effectively than addition.Repetition penalty and query relevance may suppress reward increases driven undesirably by length incentive.Source: B Combining Reward Functions · p. 13, Table 6 · p. 13

C.1 Dataset Details

Experiments use different training datasets and initialized backbones, with dataset-specific sampling for PPO. Dataset statistics and query-type distributions are presented for the experiments.

C.1 Dataset Details

  • Anthropic-HH experiments use 161k chosen responses for LLaMA-2-7B SFT and about 23k first-turn queries for PPO.The PPO queries come from the Helpful-rejection and Helpful-online splits.Source: §C.1 · p. 14, Table 7 · p. 14
  • For other experiments, the study uses pretrained SFT backbones, 20k unlabeled AlpacaFarm examples, and 31k preprocessed ShareGPT instances for PPO.Source: §C.1 · p. 14, Table 7 · p. 14
  • Figure 10 shows each experiment dataset's distribution by query type T(x).Source: Figure 10 · p. 14, §C.1 · p. 14

C.2 Training Details

Training uses full-finetuning for SFT and LoRA for PPO, with adaptive KL regularization and checkpoint selection by average reward. Hardware and implementation details vary by model size.

C.2 Training Details

  • PPO uses LoRA on query and value parameters, with lora_r {8, 16}, lora_alpha {16, 32}, dropout {0.05, 0.1}, and an adaptive KL penalty initialized at 0.2.The shared hyper-parameter setup is summarized in Table 8.Source: §C.2 · p. 15, Table 8 · p. 15
  • SFT uses full-finetuning with FSDP, while PPO generally uses one A100 GPU and the 13B model uses eight.The implementation uses the corresponding chat template when supported by the initial backbone model.Source: §C.2 · p. 15
  • The best checkpoint is selected using average reward every 1,000 PPO steps, with Figure 11 showing the Vicuna + RER reward trend.Implementations use Transformers, TRL, and PEFT.Source: §C.2 · p. 15, Figure 11 · p. 15

D.1 Evaluation set

The evaluation section describes benchmark scoring, a GPT-4-based measure of relevant sentences, and qualitative comparisons of responses from models trained with different rewards.

D.1 Evaluation set

  • Evaluation spans Vicuna-Bench, AlpacaEval, MT-Bench, ToxiGen, and Super-NaturalInstructions, using adjusted win rate versus SFT, win rate versus Text-Davinci-003, 1–10 judgments, and ROUGE-L on unseen instructions.Vicuna-Bench uses 80 curated questions; AlpacaEval has 805 instructions, and the SuperNI score uses 100 instances from 119 unseen-test tasks.Source: §D.1 · p. 16

E Relevant Sentence Ratio

F Qualitative Examples

  • Qualitative examples report that RER avoids repetitions and gives more relevant, reliable responses than UltraRM and LI, while following CE constraints better than LI · RP · QR.The comparisons are illustrated in Figures 14 and 15.Source: Figure 14 · p. 18, Figure 15 · p. 19
Loading 2402.03469v3…