PaperlayerView exact revision on arXiv

Source-linked AI summary

Patterning in Practice: Debiasing Reward Models with Susceptibilities

George Wang, Elizabeth Donoway, Daniel Murfet

arXiv:2609.00699v1cs.LG

TL;DR

Reward models can amplify stylistic biases already present in preference data, motivating methods that intervene directly on training distributions. This paper uses susceptibility-based patterning to reweight preference pairs before retraining, achieving substantial RM-Bench Hard improvements, avoiding or diagnosing side effects depending on the target, and transferring weights across related models.

  • Problem

    Reward models amplify length, formatting, and other stylistic biases in preference data, creating a need for more fine-grained debiasing than human curation alone provides.

  • Method

    Patterning uses a susceptibility matrix and its Moore–Penrose pseudo-inverse to compute per-pair reweightings for a desired change in benchmark-loss observables, followed by retraining on the corrected distribution.

  • Results

    +14.2 ± 1.2 pp on RM-Bench Hard is achieved by the three-observable target, while the fifteen-observable target yields a smaller substantial gain without the safety-response regression; Gemma 2 9B weights also transfer within the Gemma family.

  • Takeaways & Limitations

    Patterning provides an interpretable data-level intervention whose computed weights can be reused across same-family model scales, though target choice trades off debiasing gains, safety-response regression, and RewardBench 2 cost.

  • Takeaways & Limitations

    The susceptibility estimates have poorly characterized finite-sample bias because their SGLD-based posterior approximation lacks strong theoretical control.

Abstract

from arXiv · show

Reward models trained on human preferences are known to suffer from length, formatting, and other stylistic biases. In this paper we use patterning, which reweights each preference pair according to its measured effect on posterior expectation values of benchmark losses (its susceptibility), to debias a Gemma 2 9B Instruct reward model trained on Skywork-Reward-Preference v0.2. We obtain $+14.2 \pm 1.2$ pp on RM-Bench Hard, the split where style cues point against correctness (mean $\pm$ s.e.\ over 5 seeds), with overall RM-Bench accuracy preserved, comparable to the strongest Hard-split gain reported by the closest published comparator (SteerRM, $+13.2$ pp). We demonstrate in a simple case that the reweighting is interpretable by tracing a side effect of the intervention (a regression on a safety subset of RM-Bench) to a small class of training pairs, which we confirm by ablation. The weights also transfer: those computed on Gemma 2 9B debias Gemma 2 2B and 27B with no recomputation, and transfer partially to Llama 3.1 8B. This is the first application of patterning, a program grounded in singular learning theory, beyond small models and synthetic tasks.

1 Introduction

Reward models inherit and amplify stylistic biases in preference data, motivating data-level debiasing without collecting new preferences. This paper applies patterning to a 9B reward model, achieving competitive Hard-split gains, interpretable reweightings, and transfer across models.

  • Motivation: Preference datasets choose longer responses 56–63% of the time, while reward models amplify this imbalance into a strong length–reward correlation.These stylistic biases persist across reward-model generations and can be exploited by trained policies.
  • Approach: Patterning converts the instruction “debias a reward model” into a concrete reweighting of fixed preference data using susceptibilities.Susceptibilities measure how trained-model properties respond to reweighting individual training pairs.
  • Results: +14.2 ± 1.2 pp on RM-Bench Hard is achieved on Gemma 2 9B, while overall RM-Bench accuracy changes by −0.4 ± 0.4 pp.The model is trained on Skywork-Reward-Preference v0.2; the result is comparable to SteerRM’s strongest reported +13.2 pp Hard-split gain.
  • Interpretability: Patterning weights identify a small class of preference pairs associated with a safety-response regression, which ablation confirms.This makes the reweighting analyzable as a practical data-attribution process.
  • Transferability: Gemma 2 9B patterning weights transfer strongly to Gemma 2 2B and 27B and moderately to Llama 3.1 8B.The transfer suggests the weights capture meaningful properties of the data as well as model-specific effects.
  • Novelty: This is patterning’s first application beyond small transformers and synthetic tasks, at the scale of a practical 9B-parameter reward model.Unlike interventions that directly specify a bias, patterning targets benchmark-measured losses.

2 Background

The paper frames reward-model bias through posterior expectations and susceptibilities, then reverses the resulting data-to-behavior map to design preference-data reweightings. RM-Bench’s style-controlled splits provide observables for diagnosing whether models rely on substantive features or confounded style cues.

  • Posterior diagnosis: The local posterior around a trained model checkpoint supplies expectation values of observables, here the model’s losses on RM-Bench splits.Posterior expectations are used because local loss-landscape geometry reflects how the model computes reward.
  • Susceptibility matrix: The susceptibility matrix has observables as rows and training pairs as columns, forming a Jacobian from data distributions to observable expectations.Reading this matrix forward diagnoses which data drives behavior; patterning reads it in reverse.
  • Patterning: Patterning specifies a target dµ for observable changes and uses the Moore–Penrose pseudo-inverse of χ to obtain per-pair reweighting.Here the target asks for worse Easy loss and better Hard loss, after which the reweighted data are used for retraining.
  • Reward modeling: Reward models assign scalar rewards to responses, and preference training uses Bradley–Terry likelihood with negative log-likelihood losses.The paper’s dataset is Skywork-Reward-Preference v0.2, whose sampling distribution is denoted q.
  • Pipeline: Figure 1 summarizes the pipeline from training on q through posterior susceptibility estimation, matrix inversion, per-pair weighting, and retraining.The intervention strength is controlled by α > 0.
  • Benchmark: RM-Bench evaluates reward models across chat, code, math, safety-refuse, and safety-response categories.Overall scoring pools the safety subcategories and macro-averages the four domains.
  • Benchmark splits: Easy pairs align style cues with correctness, Normal pairs match style, and Hard pairs place higher style rank on the rejected response.A model relying on length or Markdown formatting therefore tends to score well on Easy but poorly on Hard.
  • Susceptibilities: A susceptibility is the derivative of a posterior expectation along a deformation of the training-data distribution.It measures how observables respond when the preference distribution shifts toward a probe distribution q′.

3 Methodology

The method diagnoses reward-model bias through posterior expected RM-Bench losses, then reweights preference pairs to target a desired Easy–Hard loss shift before retraining. It also includes domain-specific targets, practical handling of weights, and a caveat about the unregularized pseudo-inverse.

  • Diagnosis: Patterning treats the expected gap between RM-Bench Easy and Hard losses as a diagnostic of reliance on surface features.Posterior sampling replaces the trained weights with a local random variable when evaluating expected performance.
  • Susceptibility and reweighting: The susceptibility matrix links preference-distribution perturbations to first-order changes in posterior observable expectations, and its pseudo-inverse supplies a minimum-L2-norm perturbation.The empirical counterpart is implemented as a reweighting of the observed preference pairs.
  • Weight construction: Per-pair weights are ρj = 1 + α(χ̂†dμ)j, so susceptibility profiles aligned with the target are upweighted and opposing profiles are downweighted.The convention fixes s = χ̂†dμ; therefore the first-order observable shift is αβdμ rather than dμ itself.
  • Caveat: The unregularized pseudo-inverse is used despite the caveat that its empirical estimator need not consistently estimate the population pseudo-inverse.The paper does not investigate whether ridge regularization changes the results.
  • Target design: The target raises Easy loss and lowers Hard loss, while fifteen-observable targets apply the same directional change within each domain and hold Normal loss fixed.The reported domain target uses (achat, acode, amath, arefuse, aresponse) = (1.5, 2.0, 1.0, 1.0, 1.0) with α = 350.
  • Training procedure: The study estimates susceptibilities from 74,508 filtered Skywork preference pairs and retrains Gemma and Llama instruction-tuned reward models with normalized reweighted Bradley–Terry loss.Negative weights are handled heuristically by swapping chosen and rejected responses and using their absolute weights.

4 Results

Patterning improves RM-Bench Hard performance while exposing and partially correcting domain-specific side effects through interpretable data reweighting. The intervention also transfers strongly across Gemma scales, but less effectively across architectures.

  • +14.2 ± 1.2 pp on RM-Bench Hard at dµ3, compared with SteerRM’s +13.2 pp comparator result.The dµ3 result costs −1.4 ± 0.4 pp on RewardBench 2, while overall RM-Bench accuracy is preserved.
  • 4.1.1 Interpreting the dµ3 reweighting: Patterning weights correlate most negatively with words, length, and paragraphs, but are not reducible to length-only reweighting.The largest negative correlations are words (r = −0.228), len (r = −0.221), and paragraphs (r = −0.131); a length-only intervention performs substantially worse.
  • 4.1.2 Examining a failure of patterning at dµ3: −44.1 pp safety-response Easy accuracy at dµ3 was traced to short-rejected-refusal pairs whose longer chosen responses received substantial downweighting.Clamping 2,112 such pairs recovered safety-response Easy from −44.1 pp to −13.3 pp, removing about two-thirds of the regression without reducing debiasing.
  • 4.2 Patterning at dµ15: +10.2 ± 1.2 pp on RM-Bench Hard at dµ15, while safety-response Easy declined only −1.7 pp instead of the dµ3 collapse.The finer target costs −4.3 ± 0.5 pp on RewardBench 2 and has a smaller correlation with length.
  • 4.3 Transferability of ρ to other models: Patterning weights transfer strongly from Gemma 2 9B to Gemma 2 2B and 27B, but only moderately to Llama 3.1 8B.Within-family transfer to Gemma 2 2B exceeds the native SteerRM Hard gain, whereas cross-architecture transfer to Llama does not.
  • 4.4 Summary: The dµ3 target produces stronger Hard debiasing, whereas dµ15 better avoids safety-response regression at a larger RewardBench 2 cost.The authors characterize dµ3 as an overenthusiastic anti-length intervention and dµ15 as a subtler intervention.

5 Limitations

The evaluation reuses RM-Bench to define and measure debiasing, while RewardBench 2 provides an independent check but functions as validation because it informed target and hyperparameter selection. The susceptibility estimates and intervention also face theoretical, linear-response, and computational limitations.

  • RM-Bench both defines the patterning target and measures debiasing, while RewardBench 2 is independent of the observables and training data but serves as validation rather than a held-out test.
  • Susceptibility estimates rely on SGLD samples whose relationship to the localized posterior is not under good theoretical control.Constant-step-size SGLD has non-vanishing bias, and standard convergence guarantees may be incompatible with singular-model loss landscapes.
  • At the experimental coupling, linear response is not guaranteed; its usefulness at order-one strength is an empirical finding rather than a theoretical consequence.The coupling is approximately 1.0 for dµ3 and 1.4 for dµ15, with substantial weight movement and some negative weights.
  • Approximately 2,000 B200 GPU-hours were required to estimate susceptibilities for Gemma 2 9B, and each patterning target requires full retraining.Transfer across models can amortize the susceptibility-estimation cost.

6 Related Work

Related work spans inference-time steering, preference-data curation and reweighting, influence-based attribution, and theoretical analyses of spurious features. Patterning differs by using benchmark-specified debiasing targets and individual preference-pair reweighting.

  • Reward model debiasing: SteerRM suppresses format-related features in pretrained SAE dictionaries at inference time, whereas patterning reweights preference data.
  • Reward model debiasing: Industry practice still centers on collecting and curating preference data, an approach with which patterning composes.
  • Data reweighting and selection: Patterning reweights individual preference pairs toward a specified debiasing direction, unlike DORM’s dataset-level objective for noise robustness and data quality.DORM also targets multi-objective regression reward models, while patterning uses the Bradley–Terry objective.
  • Influence functions and training data attribution: Influence functions estimate effects of individual training examples and have been applied to reward models for performance attribution and labeler-bias detection.

7 Conclusion

The paper frames indirect control over model generalization as an alignment obstacle and applies patterning to steer style-bias behavior at larger scale. It reports transferable debiasing, interpretable failure diagnosis, and a boundary for harder alignment targets.

  • 7 Conclusion: Indirect specification through data curation and reward design leaves multiple low-loss, high-reward generalizations possible, creating a control obstacle for alignment.
  • 7 Conclusion: Patterning extends steering toward specified behavior from small synthetic transformers to a 9B reward model addressing style bias.The authors describe this as a step toward alignment problems such as reward hacking.
  • 7 Conclusion: Patterning matches the strongest comparable RM-Bench Hard gains, and weights computed on Gemma 2 9B transfer across smaller and larger models in the same family.
  • 7 Conclusion: A refined target diagnoses and fixes the safety-response regression caused by a simple patterning target, illustrating iterative specification of undesirable generalization.
  • 7 Conclusion: Harder alignment properties may be more entangled with genuine quality signals, and downstream-policy observables are a proposed route toward directly targeting reward hacking.

A.1 RM-Bench landscape

The RM-Bench landscape compares published reward models with the Skywork-Reward-V2 series using fine-grained difficulty-level scores. In the paper’s setup, dµ15 improves Hard accuracy over the unpatterned baseline while modestly raising average accuracy.

  • A.1 RM-Bench landscape: Table 8 separates published reward models from the Skywork-Reward-V2 series and marks best results with the original preference data versus the 40M-pair preference set.
  • A.1 RM-Bench landscape: The unpatterned Gemma 2 9B baseline scores 87.6/70.6/42.4 on RM-Bench Easy/Normal/Hard, averaging 66.9.
  • A.1 RM-Bench landscape: The dµ15 result reaches 82.0/71.2/52.6 on Easy/Normal/Hard, averaging 68.6, with Hard accuracy rising by +10.2 pp.The average improves modestly relative to the unpatterned baseline.

B Training hyperparameters

The paper reports training and posterior-sampling hyperparameters for reward-model fine-tuning and SGLD-based susceptibility estimation.

  • Training hyperparameters for reward-model fine-tuning are provided in Table 9.
  • Susceptibilities are estimated from samples of a localized tempered posterior.
  • SGLD samples the posterior by adding Gaussian noise to gradient descent, using inverse temperature β and localization strength γ.
  • The implementation uses RMSprop-preconditioned SGLD, with coordinate updates based on minibatch gradients and RMSprop statistics.
  • SGLD sampling hyperparameters for Gemma 2 9B are provided in Table 10.

D Negative weights and the label-swap convention

Negative patterning weights are implemented by swapping the chosen and rejected responses and assigning the pair a positive absolute weight, but this convention does not exactly reproduce the signed objective’s gradient.

  • D Negative weights and the label-swap convention: Negative patterning weights arise for some preference pairs and require handling beyond a positive-weight retraining objective.
  • D Negative weights and the label-swap convention: The label-swap convention exchanges chosen and rejected responses and assigns the pair weight |ρ_j|.
  • D Negative weights and the label-swap convention: Swapping the responses multiplies the Bradley–Terry gradient direction by −e^f(w).
  • D Negative weights and the label-swap convention: The signed objective contributes ρ_j ∇_w ℓ_zj(w), whereas the swap convention contributes a rescaled negative branch.
  • D Negative weights and the label-swap convention: The negative branch is continuous at ρ_j = 0 but not differentiable there, with below-zero slope e^f(w) rather than 1.
  • D Negative weights and the label-swap convention: +10.8 pp improves RM-Bench Hard in a single run, while RewardBench 2 changes from 74.0 to 74.2.

F Length-only baseline

The length-only intervention captures a prominent surface correlation in patterning weights and improves Hard accuracy, but it sacrifices Easy and RewardBench 2 more severely than patterning.

  • F Length-only baseline: r = −0.228 is the dominant correlation between chosen-minus-rejected word-count difference and patterning weights.
  • F Length-only baseline: The calibrated length-only baseline downweights pairs with longer chosen responses and upweights pairs with longer rejected responses.
  • F Length-only baseline: The baseline is restricted to |ζ_j| ≤ 1 because patterning pulls extreme length differences toward ρ ≈ 0 rather than assigning extreme weights.
  • F Length-only baseline: RewardBench 2 falls −9.4 pp for the length-only baseline, compared with −1.5 pp for patterning in the same run.
  • F Length-only baseline: The length-only baseline collapses Easy by 41 pp and overshoots Hard by 30 pp, producing Hard accuracy of 72.0% versus Easy accuracy of 46.3%.
  • F Length-only baseline: Length-only reweighting raises Hard accuracy but acts indiscriminately, sacrificing more Easy and out-of-distribution preference accuracy than patterning.
  • F Length-only baseline: The baseline is a probe calibrated to patterning’s weight scale, and another multiplier c might recover a different trade-off.

G Surface features

The paper extracts response-level surface features and computes chosen-minus-rejected differences for correlation analysis, including a regex-based refusal feature.

  • G Surface features: Surface features are extracted for every chosen and rejected response, with ΔX defined as chosen minus rejected.
  • G Surface features: Pearson correlation r(ΔX, ρ) is computed over the 74,508-pair Skywork population.
  • G Surface features: The extracted feature set is listed in Table 14.
  • G Surface features: The refusal feature is computed by a case-insensitive search of the response plaintext using a specified regular expression.
  • G Surface features: The refusal regex matches phrases such as “I cannot,” “I apologize,” “I’m sorry,” and “I must decline.”

H SRR details

The SRR analysis isolates a small training-pair class associated with the safety-response regression and tests whether the effect reflects length alone or refusal-specific structure. The evidence links most SRR pairs to WildGuard and shows that clamping their weights recovers much of the regression.

  • Length-matched control: Length alone explains roughly 60% of the SRR deviation from baseline, while refusal-rejected structure contributes an additional ∼0.29 drop.The comparison uses length-matched non-refusal-rejected controls to separate anti-length effects from the refusal-specific residual.
  • SRR-clamp ablation: Clamping the 2,112 SRR pair weights to ρ_j = 1.0 recovers most of the safety-response regression: Easy improves from −44.1 to −13.3 pp and Normal from −16.8 to −4.5 pp.The ablation uses dµ = (2, −1, −2), α = 250 on Gemma 2 9B, averaging five seeds.
  • Sub-corpus provenance: 1,956 of the 2,112 SRR pairs, or 92.6%, come from WildGuard, the only sub-corpus with population mean ρ below the overall population.This provenance analysis identifies WildGuard as the dominant source of the SRR class.
  • Refusal-cell analysis: Patterning down-weights the unharmful refusal-on-rejected cell, where 1,955 of the 1,956 WildGuard SRR pairs occur, but favors the harmful safety-refuse cell.The harmful cell has mean ρ = +0.96 with 10.5% flipped, whereas the unharmful cell has mean ρ = +0.43 with 24.5% flipped.
  • Test-time correspondence: Safety-response Easy presents the same long-compliance-chosen and short-refusal-rejected structure as SRR training pairs, connecting the training class to the test-time regression.Both classes involve adversarially framed or keyword-suspicious prompts with c_refusal = 0, r_refusal = 1, and c_words > 2 r_words.
Loading 2609.00699v1…