PaperlayerView exact revision on arXiv

Source-linked AI summary

BiasReducer: Adaptive Bias Mitigation for Reward Models

Shuang Liu, Yongliang Miao, Yanguang Liu, Haoyi Xiong, Mengnan Du

arXiv:2609.32720v1cs.LG

TL;DR

Reward models can reward superficial response attributes, while existing mitigation either requires retraining or a bias specified in advance. BIASREDUCER learns attribute-specific reward-head edits and selects them for each new dataset. Across five reward models, its multi-edit variant improves three benchmarks and outperforms two training-based baselines.

  • Problem

    Reward models may favor superficial response attributes, while training-based mitigation can require new data and computation and existing editing methods usually require a bias specified in advance.

    Source: §1 · p. 1

  • Method

    BIASREDUCER learns interpretable attribute-specific reward-head corrections and selects which to apply to each new dataset.

    Source: §1 · p. 1, §3 · p. 3

  • Results

    Across five reward models, BIASREDUCER-M improves RM-Bench-Hard, JudgeBiasBench, and Arena-StyleConflict by 8.3, 18.0, and 6.9 percentage points on average, outperforming two training-based baselines.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The gains transfer to downstream LLM training, reducing unnecessary verbosity and sycophancy.

    Source: §6 · p. 9

    For a new dataset, edit selection uses responses, hidden representations, and original reward scores, but not preference labels.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Reward models score responses from large language models (LLMs) and guide LLM training toward human preferences. However, reward models can favor superficial attributes such as length or confidence, leading LLMs to produce higher-scoring but not more correct responses. Existing mitigation methods either retrain the reward model or apply a fixed correction to one known bias, such as a preference for longer responses. Retraining requires additional data and computational resources, while existing editing methods require the target bias to be specified in advance and use a fixed edit for that bias. To this end, we propose BiasReducer, a lightweight framework that edits only the linear reward head and selects the relevant edits for each new dataset. First, BiasReducer uses a sparse autoencoder (SAE)-style encoder to learn which attributes (e.g., length and confidence) the reward model is sensitive to. Second, it learns how to reduce the reward model's dependence on each attribute by determining which direction to adjust the reward head and how much to adjust it. Third, for a new dataset, it ranks the attributes by their influence on reward scores, selects the relevant ones, and edits the reward model accordingly. BiasReducer consistently improves reward-model robustness to biases toward superficial response attributes. Across five reward models, BiasReducer-M improves the three benchmarks by 8.3, 18.0, and 6.9 percentage points on average, outperforming the two training-based baselines. The gains transfer downstream, reducing unnecessary verbosity and sycophancy while maintaining comparable judged quality.

1 Introduction

Reward models can favor superficial response attributes, motivating corrections that adapt to which attributes matter in each new dataset. BIASREDUCER edits the linear reward head and reports stronger benchmark results than training-based baselines.

1 Introduction

  • Reward models may favor length, formatting, confidence, or sycophancy over correctness or helpfulness, encouraging reward hacking during response selection or downstream training.The cited examples include longer or more confident responses receiving higher rewards despite being less correct or helpful.Source: §1 · p. 1
  • Training-based mitigation can require substantial new data and computation, while editing methods usually require a target bias specified in advance despite its relevance varying by dataset.This motivates choosing corrections for each new dataset without retraining or applying the same preselected attribute correction.Source: §1 · p. 1
  • BIASREDUCER edits only the linear reward head, learning attribute-specific corrections and selecting which to apply for each new dataset.Its steps learn interpretable attribute representations, determine edit direction and strength, then select relevant edits for a new dataset.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • Across five reward models, BIASREDUCER-M improves RM-Bench-Hard, JudgeBiasBench, and Arena-StyleConflict by 8.3, 18.0, and 6.9 percentage points on average, outperforming two training-based baselines.The reported gains also transfer to downstream LLM training, reducing unnecessary verbosity and sycophancy.Source: §1 · p. 2
  • The framework separates reusable attribute-specific corrections from the decision about which corrections to apply to a particular dataset.The contributions describe semantic supervision for attribute representations and dataset-specific edit selection.Source: §1 · p. 2

2 Related Work

Prior work studies reward-model bias, sparse representations, and mitigation through training or model editing. The described approaches leave open how to connect interpretable attributes with reusable, dataset-specific corrections.

Reward hacking and reward-model bias

  • Reward models can favor length, formatting, confidence, or agreement with the user over correctness or helpfulness, contributing to reward hacking when used for selection or training.Examples of reward hacking include optimizing rewarded attributes instead of better responses (Gao et al., 2023; Coste et al., 2024; Rafailov et al., 2024).Source: §2 · p. 3

Sparse representations

  • SARM and SparseRM use sparse features for explaining reward decisions and preference modeling, but prior sparse-representation and concept-erasure methods do not connect interpretable attributes to reusable bias-mitigation edits.The cited methods are SARM and SparseRM (Zhang et al., 2026; Liu et al., 2026a).Source: §2 · p. 3

Mitigating reward-model bias

  • Training-based methods modify reward architectures, objectives, data, or fine-tuning, while editing methods modify trained models by removing a bias-associated subspace or direction.The section identifies training methods including ODIN, RRM, CROME, and counterfactual fine-tuning (Chen et al., 2024; Liu et al., 2025a; Srivastava et al., 2026; Bharadwaj et al., 2026), and editing methods (Liu et al., 2026b; Fein et al., 2026).Source: §2 · p. 3
  • Training-based methods require retraining and new data construction, while existing editing methods use a fixed correction for a bias specified in advance.These limitations leave the correction unchanged even when different attributes matter across target distributions.Source: §2 · p. 3

3 Methodology

BIASREDUCER learns attribute-specific reward-head edits from source preference data, then ranks and selects corrections for a new dataset using responses and original reward scores without preference labels.

3 Methodology

  • The framework’s overall workflow reuses edits learned from source data and selects among them for each new dataset without retraining the reward model.The methodology describes attribute representation learning, correction estimation, and dataset-specific selection as its stages.Source: §3 · p. 3

3.1 Problem Statement

  • For a scalar reward model with a linear head, BIASREDUCER keeps all model parameters fixed except the reward-head weight; new-dataset selection does not use pairwise preference labels.The new dataset may provide responses, hidden representations, and original reward scores.Source: §3.1 · p. 3

3.2 Learning Representations of Response Attributes

  • An SAE-style encoder sparsifies preference contrasts, while semantic supervision aligns selected latent coordinates with predefined attributes and controlled rewrites establish their directions.Behavioral supervision links a coordinate to natural attribute differences; interventional supervision uses controlled rewrites, and an artifact term captures changes shared across those rewrites.Source: §3.2 · pp. 3–4

3.3 Reducing the Reward Model’s Dependence on Response Attributes

  • For each attribute, BIASREDUCER estimates a co-varying hidden-state direction and chooses an edit strength that induces reward-score change while limiting preference-accuracy drop on validation data.The resulting attribute-specific direction and strength are collected in an edit bank for later selection.Source: §3.3 · p. 4

3.4 Selecting What to Edit for a New Dataset

  • On a new dataset, BIASREDUCER ranks attributes using reward variation with the attribute and the reward-score change from its corresponding edit, combining the two rankings by rank sum.A smaller rank sum gives higher selection priority; BIASREDUCER-S selects one eligible attribute, whereas BIASREDUCER-M combines edits for up to kmax attributes.Source: §3.4 · p. 4

4 Experiments

Experiments evaluate BiasReducer across five reward models and three benchmarks, finding that BIASREDUCER-M outperforms training-based baselines and that gains transfer to downstream response selection and policy training.

4.1 Experimental Setup

  • Experiments cover five reward models, three benchmarks, and comparisons against the original model plus two fine-tuning baselines.The models use Skywork-Reward-Preference-80K-v0.2 as source data for learning attribute representations and edits.Source: §4.1 · p. 5

4.3 Downstream Reward Optimization

  • Edited rewards reduce verbosity and sycophancy in response selection and policy training while preserving judged response quality.The downstream evaluation uses best-of-N response selection and GRPO policy training.Source: §4.3 · p. 5

4.2 Main Results

  • BIASREDUCER-M gains 8.3, 18.0, and 6.9 percentage points on average, exceeding training-based baselines across the three benchmarks.It also provides 4.1, 13.2, and 2.3 percentage points beyond RRM, the stronger baseline on average.Source: §4.2 · p. 5, Table 1 · p. 6

5 Analysis and Insights

The analyses find that semantic supervision, reward-aware edit construction, and dataset-specific selection contribute to stronger editing. Performance also remains relatively stable across edit-complexity settings.

5.1 Semantic Supervision Improves Attribute Alignment and Editing

  • Semantic supervision raises mean held-out attribute correlation from 0.26 to 0.55; removing behavioral supervision lowers attribute-axis correlation from 0.55 to 0.17.Removing intervention supervision reduces sign accuracy from 0.99 to 0.75, while artifact supervision still improves editing across all three benchmarks.Source: §5.1 · p. 7, Table 3 · p. 7

5.2 Reward-Aware Directions and Attribute-Specific Strengths Improve Editing

  • Covariance-based directions yield average gains of 6.2, 15.7, and 6.0 points across the three benchmarks, versus 0.5, 0.4, and 0.6 with decoder vectors.On JudgeBiasBench with Mistral-7B, separate attribute-specific strengths raise the gain from 8.4 to 11.8 points versus one shared strength.Source: §5.2 · p. 7, Table 4 · p. 8

5.3 Different Datasets Benefit from Different Attribute-Specific Edits

  • Selected attribute corrections vary across datasets and reward models, and BIASREDUCER-M combines different corrections across the three benchmarks.The figure panels show selection patterns across models and benchmarks, plus attribute-selection frequency and comparisons of selection signals.Source: §5.3 · p. 7, Figure 3 · p. 8
  • Rank-sum selection includes the best eligible correction in 86% of JudgeBiasBench cases, versus 71–74% for either signal alone, and gives the largest average gains.With 100 examples per subset, selection matches the full-pool decision in 89% of trials, rising to 97% with 200 examples.Source: §5.3 · p. 8, Figure 3 · p. 8, Figure 4 · p. 9
  • Using 100 examples per subset matches the full-pool attribute-selection decision in 89% of trials, increasing to 97% with 200 examples.Figure 4 also reports pairwise-accuracy gains for BIASREDUCER-S and BIASREDUCER-M.Source: §5.3 · p. 8, Figure 4 · p. 9

5.4 BIASREDUCER Remains Stable Across Edit-Complexity Settings

  • Changing kmax from 2 to 6 leads to only modest differences, while dictionary size and active-coordinate count have more benchmark-dependent effects.Reducing active coordinates lowers gains across all three benchmarks; reducing dictionary size improves RM-Bench-Hard but lowers gains on the other two.Source: §5.4 · p. 9, Table 5 · p. 9

6 Conclusion

BIASREDUCER learns reusable attribute-specific corrections and selects which to apply for each new dataset without retraining. It improves three tested benchmarks and transfers downstream to best-of-N selection and GRPO training.

6 Conclusion

  • BIASREDUCER learns reusable attribute-specific corrections and selects which to apply for each new dataset without retraining the reward model.The conclusion identifies semantic supervision, reward-aware edit directions and strengths, and dataset-specific selection as contributors to performance.Source: §6 · p. 9
  • Across five reward models, BIASREDUCER-M improves the three tested benchmarks by 8.3, 18.0, and 6.9 percentage points on average, outperforming two training-based baselines.Source: §6 · p. 9
  • The gains transfer downstream: edited rewards reduce unnecessary verbosity and sycophancy in best-of-N selection and GRPO training.Source: §6 · p. 9

A.1 Reward Models

  • Experiments use five publicly available scalar reward models from released checkpoints, modifying only the linear reward-head weight while keeping other parameters fixed.Source: §A.1 · p. 13

A.2 Source Preference Data

BIASREDUCER uses Skywork-Reward-Preference-80K-v0.2 to learn attribute representations and estimate attribute-specific edits, with data split between training and validation roles.

A.2 Source Preference Data

  • Skywork-Reward-Preference-80K-v0.2 supplies the preference data for learning attribute representations and determining attribute-specific reward-model edits.Source: §A.2 · p. 13, Table 7 · p. 13
  • The training split supports representation learning and edit-direction estimation, while validation data selects how far to edit along each direction.Source: §A.2 · p. 13
  • Before splitting, source examples overlapping evaluation benchmarks are removed using normalized prompt matching or shared sequences of 13 consecutive words.Source: §A.2 · p. 13

A.3 Evaluation Benchmarks

The evaluation spans three response-bias benchmarks and uses response data without preference labels to select edits. The method learns attribute-aligned representations, then determines reward-head edits from source data.

A.3 Evaluation Benchmarks

  • The three benchmarks cover bias-focused response preferences and style conflicts, including a 1,000-pair Arena-StyleConflict set constructed from human preferences.Arena-StyleConflict retains cases where humans prefer a response that is at least 1.3× shorter and no more Markdown-formatted than its counterpart.Source: §A.3 · p. 13, Table 8 · p. 14
  • RM-Bench-Hard tests three cells where the benchmark-preferred response has a lower style level; JudgeBiasBench covers seven superficial-quality bias subsets.The JudgeBiasBench subsets are length, authority, beauty, assertiveness, sycophancy, sentiment, and concreteness.Source: §A.3 · p. 14

A.4 Evaluation and Attribute Edit Selection Protocol

  • Edit selection uses responses, hidden representations, and original scores without pairwise preference labels, then fixes the edited reward head for benchmark evaluation.RM-Bench-Hard and Arena-StyleConflict use one selection decision each; JudgeBiasBench selections are made separately for its seven subsets.Source: §A.4 · p. 14

A.5 Response Attributes

  • BIASREDUCER represents seven predefined surface attributes—length, verbosity, formatting, emoji/exclamation use, confidence, hedging, and sycophancy—with measurable text signals and, mostly, controlled rewrites.The attributes are motivated by prior work on response-level biases.Source: §A.5 · p. 14

A.6 Baseline Implementations

  • The comparisons include RRM-style further fine-tuning and intervention fine-tuning using attribute-specific counterfactual pairs mixed with source preferences.Both baselines further fine-tune released reward-model checkpoints.Source: §A.6 · pp. 14–15

B.2 Semantic Supervision for Response Attributes

  • For each attribute, behavioral measurements and controlled rewrites supervise a designated interpretable coordinate; a separate artifact coordinate captures changes shared across rewrites.Behavioral signals track naturally occurring attribute differences, while interventions specify the direction of deliberate attribute changes.Source: §B.2 · pp. 17–18

B.3 Learning Attribute-Aligned Representations

  • A sparse encoder–decoder reconstructs preference contrasts with signed top-k activations, and its losses encourage reconstruction, sparsity, distinct directions, attribute alignment, and reduced cross-attribute leakage.The reward model itself remains unchanged during representation training.Source: §B.3 · p. 18

B.4 Semantic Checks After Representation Learning

  • Validation checks compare natural attribute variation with controlled rewrites, while source data determines each edit’s direction and the largest admissible amount under a preference-accuracy constraint.The checks include rewrite consistency, matched controls, split-half stability, expected direction, and approximate symmetry for length changes; attributes without an admissible nonzero edit are ineligible.Source: §B.4 · p. 19, §B.5 · pp. 19–20

B.6 Dataset-Specific Edit Selection and Multi-Edit Composition

BIASREDUCER ranks eligible attribute edits for each new dataset using two signals, then applies selected edits subject to a source-accuracy constraint. The accompanying analyses describe ablations of semantic supervision and edit-design choices.

B.6 Dataset-Specific Edit Selection and Multi-Edit Composition

  • For a new dataset, BIASREDUCER ranks eligible attributes by the rank sum of reward dependence and edit impact, using responses, representations, and original scores without preference labels.A smaller rank sum gives an attribute higher selection priority.Source: §B.6 · p. 20
  • BIASREDUCER-S applies the top-ranked eligible edit, while BIASREDUCER-M considers edits sequentially and reduces edit amounts or skips candidates when needed to satisfy the cumulative source-accuracy tolerance.Composition stops at the edit cap or after considering all eligible attributes; the default cap is four.Source: §B.6 · pp. 20–21

B.7 Complete Algorithm

  • The complete procedure learns representations, determines source-data edit directions and amounts, ranks edits on the new dataset, and edits the reward head.The algorithm specifies source-validation accuracy as the constraint for retaining candidate edit amounts.Source: §B.7 · p. 22

C.1 Semantic Supervision

  • The semantic-supervision ablations compare the supervised representation with a vanilla SAE and remove behavioral, intervention, and artifact supervision terms individually.The comparisons report representation metrics and downstream pairwise-accuracy changes, but the supplied passages do not give their values.Source: §C.1 · p. 23, Table 12 · p. 23, Table 13 · p. 24

C.2 Edit Direction and Amount

  • The edit-design analyses vary covariance-based versus decoder-vector directions, fixed versus attribute-specific edit amounts, and natural response representations versus preference contrasts.The direction comparison holds the selected attribute fixed; the representation choices for estimation and selection are varied independently.Source: §C.2 · pp. 23–24

C.3 Dataset-Specific Edit Selection

The additional analyses show that selected attributes vary across datasets and reward models, and describe comparisons of dataset-specific selection, selection signals, and settings. Downstream best-of-N evaluation uses fixed candidate pools scored by the original or edited reward heads.

C.3 Dataset-Specific Edit Selection

  • For the same Qwen3-1.7B model, sycophancy ranks highest on RM-Bench-Hard and the JudgeBiasBench sycophancy subset, while length ranks highest on Arena-StyleConflict.The examples illustrate dataset-dependent attribute relevance; BIASREDUCER-M can combine edits for multiple highly ranked attributes.Source: §C.3 · pp. 24–25, Table 17 · p. 29
  • The analyses compare dataset-specific selection with fixed sycophancy or source-selected attributes, and compare reward dependence, edit impact, and their rank-sum combination.Oracle coverage is evaluated against benchmark preference labels, which are not used during edit selection.Source: §C.3 · p. 25, Table 18 · p. 29, Table 19 · p. 30
  • JudgeBiasBench selection is recomputed from progressively larger response-pool samples and compared with full-pool selections and resulting pairwise-accuracy gains.The table description specifies five random draws for each sample size.Source: §C.3 · p. 25, Table 20 · p. 31

C.4 Sensitivity to Editing and Representation Settings

  • Sensitivity analyses vary the multi-edit cap, dictionary size, and active-coordinate count, with pairwise-accuracy changes reported across all three benchmarks.The default edit cap is four; the dictionary and active-coordinate experiments rerun the full pipeline.Source: §C.4 · p. 25, Table 21 · p. 32, Table 22 · p. 33

C.3 Dataset-Specific Edit Selection

  • Figure 5 shows attribute rankings varying across both reward models and benchmarks, with radar axes representing predefined attributes and rankings mapped from 1 to 0.JudgeBiasBench values are averaged over its seven subsets.Source: §C.3 · p. 24, Figure 5 · p. 26

D.1 Best-of-N Selection

  • For best-of-N evaluation, the reward heads score the same fixed candidate pools, with Mistral-7B scoring responses from Qwen3-4B and Llama-3.2-3B-Instruct policies.The candidate counts are 16, 64, and 256; edit selection and downstream evaluation use separate prompt sets.Source: §D.1 · p. 34, Table 23 · p. 34

D.2 RL Policy Training

The RL policy experiments compare training with the original and BIASREDUCER-M-edited reward heads. An independent evaluator separately assesses response quality and undesirable attributes.

D.2 RL Policy Training

  • GRPO trains Qwen3-4B and Llama-3.2-3B-Instruct with either the original or BIASREDUCER-M-edited reward head on Arena prompts.Runs within each reward-model–policy setting share initialization, prompts, KL penalty, and schedule; only the reward head differs.Source: §D.2 · p. 34
  • The reported RL outcomes are evaluated after 80 training rounds on 100 held-out prompts using an independent evaluator.The main GRPO results are reported in Table 2, which is not included among these passages.Source: §D.2 · p. 34

D.3 Independent Response Evaluation

  • Claude Sonnet 5 rates quality separately from binary judgments of unnecessary verbosity, unsupported confidence, and sycophancy.Quality is rated from 1–10 for correctness, completeness, clarity, and calibration; the evaluator does not affect editing or training.Source: §D.3 · pp. 34–35

E Implementation Details

The implementation uses shared settings across reward models and benchmarks, while training and edit-selection procedures follow fixed source-data and validation-budget choices.

E Implementation Details

  • The same implementation settings are used across reward models and evaluation benchmarks, with default settings and hyperparameters summarized in Table 24.Source: E Implementation Details · p. 36, Table 24 · p. 36
  • With the reward model fixed, the encoder–decoder trains for 200 epochs without early stopping or learning-rate scheduling, using intervention and artifact batches sampled with replacement.Edit directions and amounts are determined from source training and validation data and fixed before dataset-specific edit selection.Source: E Implementation Details · p. 36
  • For each attribute, edit amounts are searched over Λ = {0.125, 0.25, 0.375, 0.5, 0.625, 0.75, 0.875, 1.0, 1.25, 1.5, 1.75, 2.0}.The selected amount must satisfy the source-validation preference-accuracy budget τsrc = 0.02.Source: E Implementation Details · p. 36

Related papers

Loading 2609.32720v1…