InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modeling

1 Introduction

• Reward misgeneralization can make reward models rely on ranking-correlated spurious features that are irrelevant to human preferences, undermining generalization and RL stability.The paper identifies this as a primary cause of reward overoptimization.Source: §1 · pp. 1–2
• Existing mitigation approaches include KL constraints, larger or composite reward models, and preference-dataset or length-bias interventions, but the cited approaches do not address reward misgeneralization.The comparison is framed as the motivation for InfoRM’s information-theoretic approach.Source: §1 · p. 2
• InfoRM uses a variational information bottleneck objective to filter information irrelevant to human preferences from an IB latent representation.The representation mediates between reward-model outputs and human preferences.Source: §1 · p. 2
• The paper links overoptimization to outliers in InfoRM’s IB latent space and proposes CSI to quantify deviations in generated-sample distributions.CSI is presented as an indicator for parameter adjustment and online mitigation strategies such as early stopping.Source: §1 · p. 2
• InfoRM significantly outperforms standard reward models in RLHF, particularly in mitigating reward hacking, and its overoptimization detection metric is reported as effective and robust.The framework is evaluated across a wide range of settings and reward-model scales.Source: §1 · p. 3

2.1 Reward Overoptimization in RLHF

• Reward overoptimization occurs when proxy-RM optimization improves proxy scores while actual human-preference performance often begins to deteriorate after an initial period.The paper notes KL penalties, RM scaling or ensembles, dataset optimization, and length-bias interventions, while arguing these approaches leave preference-irrelevant information in reward modeling unaddressed.Source: §2.1 · p. 3
• InfoRM targets reward misgeneralization and offers a tool for detecting overoptimization during RL, supporting parameter selection without a gold RM and strategies such as early stopping.The paper contrasts this with mitigation approaches that do not directly target preference-irrelevant information.Source: §2.1 · p. 3

2.2 Information Bottleneck-Family Methods

• The paper introduces the IB principle to RLHF reward modeling, deriving a variational bound for ranking and using the compact latent space to develop an overoptimization detection tool.It describes this as the first effort to connect IB with RLHF and demonstrate its effectiveness for LLMs.Source: §2.2 · p. 3

3.1 Preliminary

• Standard reward modeling learns a proxy RM from human preference pairs, using a Bradley–Terry preference distribution and a negative log-likelihood loss.The chosen and rejected responses form the pairwise samples; poor generalization to evolving RL-stage distributions motivates the information bottleneck approach.Source: §3.1 · p. 4

3.2 Information-Theoretic Reward Modeling

• InfoRM maximizes preference information in latent S while penalizing information in S that is irrelevant to rankings, with β controlling the trade-off.The objective is optimized through a variational lower bound; the latent representation serves as an information bottleneck between inputs and rankings.Source: §3.2 · pp. 4–5
• As KL divergence increases in later RL stages, Standard RM’s gold score declines while InfoRM’s gold and proxy scores continue growing.The figure presents simulated RLHF results for 1.4B proxy RMs.Source: Figure 4 · p. 5
• The framework uses pϕ(s|x) to extract latent representations and qψ(y|s) to predict rankings from them.In practice, the representation function is modeled by an LLM with an extra head, and the ranking predictor by an MLP; the latent distribution is a diagonal-covariance multivariate Gaussian.Source: §3.2 · p. 5

4.1.1 Setup

• The simulations use a fixed gold reward model for preference labels, train proxy models with and without 25% label noise, and optimize policies with PPO.The simulation data come from AlpacaFarm, and the experiments use 70M, 410M, and 1.4B proxy reward models.Source: §4.1 · p. 5, §4.1.1 · pp. 5–6

4.1.2 Main Results

• With 25% label noise, InfoRM maintains stability while Standard RM shows compromised stability and notable reward overoptimization.The simulation uses a 1.4B proxy reward model and compares noisy and noiseless settings.Source: §4.1.2 · p. 6
• InfoRM improves performance beyond simply enlarging the reward model, and its benefits are complementary to those from scaling.Figure 5 compares proxy reward-model sizes using final gold rewards.Source: §4.1.2 · p. 6, Figure 5 · p. 6
• On out-of-distribution Flan data, InfoRM maintains relatively stable performance while Standard RM deteriorates, supporting generalization across datasets.Figure 5 contrasts RL on in-distribution Alpaca and out-of-distribution Flan.Source: §4.1.2 · pp. 6–7, Figure 5 · p. 6
• Across KL penalty values in a 25%-noise simulation, InfoRM outperforms Standard RM with KL in resistance to hacking, training stability, and RLHF performance.For Standard RM with KL, increasing the penalty initially improves stability, but values above 0.001 compromise final RLHF performance.Source: §4.1.2 · p. 7, Figure 6 · p. 7

4.2.1 Setup

• Real-world experiments use actual human-preference data for general dialogue and Reddit TL;DR data for summarization, with GPT-4 used to evaluate win ratios.Dialogue evaluation includes in-distribution Anthropic-RLHF-HH and out-of-distribution AlpacaFarm data.Source: §4.2 · p. 7, §4.2.1 · p. 7

4.2.2 Main Results

• Under GPT-4 evaluation, InfoRM outperforms Standard RM, Standard RM with KL, Ensemble RM, and WARM, and also improves results when combined with Ensemble RM and WARM.Table 1 compares win, tie, and lose ratios for RLHF models using different reward models.Source: Table 1 · p. 7, §4.2.2 · p. 8

5.1 Outlier Behavior of Overoptimizaed Samples in IB Latent Space

• GPT-4 assesses overoptimization using guidelines based on excessive caution, deviation from user intent, and repetitive or meaningless text.The authors describe identifying overoptimized samples as challenging and under-explored.Source: §5.1 · pp. 8–9
• In the t-SNE visualization, most GPT-4-judged overoptimized samples deviate from pre-RLHF samples, and InfoRM shows fewer post-RLHF outliers than Standard RM.The authors report this pattern as consistent with their simulated and real-world results; they also note validation and parameter-sensitivity analyses in the appendices.Source: §5.1 · p. 9

5.2 Detection of Outlier Emergencies and Overoptimization by the CSI Indicator

• CSI quantifies latent-space deviations by clustering RLHF outputs, finding each cluster’s centroid and nearest SFT output, then summing distances weighted by cluster size.The authors use DBSCAN, which does not require a predetermined number of clusters.Source: §5.2 · p. 9
• Between 600–700 steps, Standard RM’s CSI rises sharply and stays elevated, while InfoRM’s CSI remains much lower.The increase corresponds to outlier emergence; the authors report that CSI is sensitive to that emergence and can detect overoptimization.Source: Figure 8 · p. 9, §5.2 · pp. 9–10
• The detection mechanism depends on InfoRM’s compact IB latent space; the authors report that non-IB reward models are incompatible because their outliers correlate weakly with overoptimized samples.Source: §5.2 · p. 10
• The authors propose CSI as a model-based metric that can guide parameter adjustment and reward-model optimization, including dataset selection and algorithm design.They also describe an automated early-stopping algorithm based on CSI as a potential online mitigation strategy.Source: §5.2 · p. 10

• InfoRM uses a variational information bottleneck objective to filter preference-irrelevant information, addressing reward misgeneralization, and adds CSI to detect overoptimization from the IB latent space.Source: §6 · p. 10
• Experiments across scenarios and model sizes demonstrate InfoRM’s effectiveness in mitigating reward overoptimization, while evaluations across datasets validate CSI’s detection effectiveness.Source: §6 · p. 10

• Reward overoptimization occurs when policy optimization diverges from human objectives, reducing helpfulness through outcomes such as excessive caution.Source: Broader Impacts · p. 10
• The work introduces an information bottleneck in reward modeling to reduce overoptimization and an indicator to support online mitigation strategies.Source: Broader Impacts · p. 10

Limitations

• Scaling InfoRM beyond 7 billion parameters remains unexplored; monitoring has latency and requires inference on test datasets, while GPT-4 win rates depend on prompt structure.The authors identify real-time, lightweight detection and more reliable automated judgments as future research directions.Source: Limitations · p. 10

A Derivation for the Loss of Our InfoRM

• The appendix derives InfoRM’s variational IB objective using an encoder-distributed latent representation and a decoder that estimates human preference rankings.The latent distribution is modeled as a multivariate Gaussian with diagonal covariance; its mean and covariance are determined by the encoder output.Source: A Derivation for the Loss of Our InfoRM · p. 15

B Upper Bound of the Generalization Error for Our InfoRM

• Theorem 1 bounds expected generalization error using mutual information between the latent representation and observations, as well as latent-space dimensionality.Source: B Upper Bound of the Generalization… · p. 16

C Further Validations for Our Overoptimization Detection Machanism

• The detection mechanism is validated on sixteen diverse datasets spanning a wide range of scenarios.Source: C Further Validations for Our… · p. 16

C.1 Validations for Outlier Behavior of Overoptimizaed Samples in IB Latent Space

• The analyses compare pre- and post-RLHF response distributions and GPT-4-judged overoptimized samples in InfoRM’s latent space.Figures 9–11 span the datasets listed in their captions.Source: §C.1 · p. 17, Figure 9 · p. 17, Figure 10 · p. 18
• Across the evaluated datasets, overoptimized samples consistently appear as prominent outliers in InfoRM’s IB latent space, while InfoRM reduces post-RLHF outliers.The visualizations cover the datasets in Figures 9–11.Source: §C.1 · p. 17, Figure 9 · p. 17, Figure 10 · p. 18

C.2 Validations for Outlier Emergencies and Overoptimization Detection by the CSI Indicator

• Across diverse datasets, abrupt CSI changes coincide with outlier emergence, and InfoRM produces significantly lower CSI values than Standard RM.The reported results are shown across the datasets in Figures 12 and 13.Source: §C.2 · p. 20, Figure 12 · p. 20, Figure 13 · p. 21

D Analysis of Irrelevant Information Filtering Using Our InfoRM

• Across the evaluated datasets, InfoRM-based RLHF produces significantly shorter outputs than Standard RM, supporting mitigation of length bias.Average response length is compared at different RLHF steps in Figure 14.Source: D Analysis of Irrelevant Information… · pp. 21–22, Figure 14 · p. 22
• In datasets with prevalent harmful data, InfoRM significantly reduces excessive caution, where models refuse benign requests.The paper points to qualitative case studies in Appendix K.Source: D Analysis of Irrelevant Information… · p. 22

E.1 Impact on Overoptimization Detection Mechanism

• Across different IB dimensionalities and β settings, overoptimized samples consistently remain outliers in InfoRM’s latent space.The authors report this as robustness of the detection mechanism to hyperparameter variation.Source: §E.1 · p. 23, Figure 15 · p. 23, E Sensitivity Analysis of… · p. 23

E.2 Impact on RLHF performance

• InfoRM achieves optimal RLHF performance with IB dimensionality 128 and β=0.1; at these settings, latent-space deviation and CSI are lower than with other configurations.The response-distribution and CSI comparisons are shown in Figures 17 and 18.Source: §E.2 · pp. 23–24, Figure 16 · p. 23, Figure 17 · p. 24

F Universality of Our Overoptimization Detection Machanism

• InfoRM latent-space outliers correspond to overoptimized samples, but Standard RM outliers do not necessarily do so; CSI may therefore not apply to RMs without IB.The paper attributes the difference to InfoRM’s more compact latent space versus Standard RM’s more dispersed space.Source: F Universality of Our Overoptimization… · pp. 24–25, Figure 19 · p. 25

G Early Stopping Algorithm Based on the Proposed CSI Metric

• The CSI early-stopping procedure stops training when the current CSI change relative to the previous-step average exceeds a maximum tolerable rate, empirically set to 10.Otherwise, training continues.Source: G Early Stopping Algorithm Based on… · p. 25

H More Real-World Results with Different Hyper-parameters

• Across reported hyperparameter configurations, InfoRM consistently demonstrates significant advantages in GPT-4 evaluation of RLHF models.Table 2 reports results for different reward models and hyperparameters; its best settings per group are highlighted.Source: H More Real-World Results with… · p. 25, Table 2 · p. 26

I Performance of InfoRM on Reward Model Benchmarks

• InfoRM is comparable to Standard RM on in-distribution benchmarks and significantly outperforms it on out-of-distribution reward-model benchmarks.The benchmarks are Anthropic-Helpful and Anthropic-Harmless in-distribution, and AlpacaEval and Truthful QA out-of-distribution.Source: I Performance of InfoRM on Reward… · p. 26, Table 3 · p. 26

J.3 Training Setup

• InfoRM encodes hidden states into an IB representation for reward prediction and trains with preference loss plus β-weighted bottleneck loss; CSI weights cluster-to-reference distances by cluster size.The implementation pseudocode describes the reward-model architecture and CSI calculation.Source: §J.3 · pp. 27–28
• Simulation fine-tuning used one SFT epoch and batch size 64; reward modeling used one epoch, while PPO batch size was 16 in simulation and 64 in real-world experiments.The experiments used pre-trained checkpoints and aligned model architectures and hyperparameters with their original counterparts.Source: §J.3 · pp. 26–27

J.4 GPT-4 Evaluation

• The GPT-4 evaluation appendix provides prompts for AlpacaFarm evaluation and identifying hacking samples.Source: Figure 20 · p. 28

K.1 Examples from AlpacaFarm Dataset

• AlpacaFarm examples illustrate incomplete-information errors, with green highlighting information covered by InfoRM but missed by competing methods.The examples are shown in Figures 21–23.Source: K Qualitative Examples in Real-World… · p. 29, Figure 21 · p. 29, Figure 22 · p. 30

K.2 Examples from Anthropic Helpful Dataset

• Figures 24–26 present qualitative RLHF-model outputs from different reward models on AlpacaFarm, highlighting information covered by InfoRM but missed by competing methods as incomplete-information errors; Figure 24 also marks repeat-information and excessive-caution errors.Source: Figure 24 · p. 32, Figure 25 · p. 33, Figure 26 · p. 34

K.3 Examples from Anthropic Harmless Dataset

• The harmless-dataset examples mark incomplete-information and repeat-information errors in model outputs.Figures 27–29 show these annotations.Source: Figure 27 · p. 35, Figure 28 · p. 35, Figure 29 · p. 36