PaperlayerView exact revision on arXiv

Source-linked AI summary

CleanRM: Reward Modeling from Noisy Feedback with a Few Clean Feedback for RLHF

Licheng Pan, Haocheng Yang, Yuzhou Chen, Yichen Sun, Yuan Lu, Yunfei Teng, Jiajun Fan, Guangyi Liu, Zhixuan Chu, Hao Wang

arXiv:2610.21138v1cs.AIstat.ML

TL;DR

Sample-wise feedback noise violates the uniform-noise assumptions of common denoising methods and can bias reward models' clean-risk estimates. CleanRM calibrates noise ratios using a few clean feedback and optimal transport, then trains with a corrected surrogate loss. Experiments report improved reward prediction and downstream RLHF alignment over competitive denoising baselines.

  • Problem

    Feedback noise varies by sample, while common denoising methods generally assume a uniform noise ratio, and noisy training can bias clean-risk estimates.

    Source: §1 · p. 1

  • Method

    CleanRM uses a few clean feedback and optimal transport to estimate sample-wise noise ratios, then trains with a noise-corrected surrogate loss.

    Source: §1 · pp. 1–2

  • Results

    CleanRM consistently improves reward prediction accuracy over competitive denoising baselines and yields stronger downstream RLHF alignment.

    Source: §5 · p. 9

  • Takeaways & Limitations

    CleanRM's reported gains in reward prediction and downstream RLHF alignment support its use for reward modeling under sample-wise feedback noise.

    Source: §5 · p. 9

    CleanRM assumes access to a few clean feedback, and its formulation targets pointwise rather than pairwise or listwise preference modeling.

    Source: §5 · p. 9

Abstract

from arXiv · show

Feedback noise is pervasive in preference data, arising from subjective human judgments, annotation mistakes, and imperfect LLM judges. Standard reward modeling trained on such feedback yields a biased estimate of the clean risk. Although classical denoising methods are directly applicable to noisy feedback learning, they generally assume a uniform noise ratio across samples. However, this assumption fails for feedback data, where noise is intrinsically sample-wise. To address this challenge, we propose CleanRM, which uses a few clean feedback to calibrate sample-wise noise ratios via optimal transport. Specifically, CleanRM aligns the joint distribution of preferences reverted from noisy feedback with that of clean feedback and then trains the reward model using a noise-corrected surrogate loss. Theoretically, we prove that the resulting objective is unbiased with respect to the clean risk. Extensive experiments demonstrate that CleanRM outperforms state-of-the-art denoising baselines and yields more effective downstream RLHF alignment.

1 Introduction

Preference feedback noise varies across samples, conflicting with the uniform-noise assumptions of common denoising methods. CleanRM uses a few clean feedback to estimate sample-wise noise ratios and train a reward model with a surrogate objective that is unbiased for clean risk.

1 Introduction

  • Sample-wise feedback noise violates the uniform-noise assumption of existing denoising methods and can bias clean-risk estimates, degrading downstream alignment.Noise varies with the response: clear instruction violations may elicit consistent judgments, while reasonable responses with different trade-offs can divide annotators.Source: §1 · p. 1
  • CleanRM uses a few clean feedback to calibrate sample-wise noise ratios, then applies a noise-corrected surrogate loss that is theoretically unbiased for clean risk.The clean feedback can be obtained through targeted expert auditing or repeated annotation.Source: §1 · pp. 1–2
  • CleanRM outperforms competitive denoising baselines and improves downstream RLHF alignment in extensive experiments.The paper also reports more accurate reward signals under noisy feedback.Source: §1 · p. 2

2 Preliminaries

RLHF trains a reward model from human feedback, then optimizes a policy using that model's rewards. This paper's preliminaries formalize binary point-wise reward modeling when observed feedback can differ from latent preferences.

2.1 Reinforcement Learning from Human Feedback

  • RLHF has two stages: train a reward model from human feedback, then fine-tune a policy to maximize the rewards it assigns.The reward model maps a prompt-response pair to a scalar preference score and supplies the policy-optimization signal.Source: §2.1 · p. 2

2.2 Problem Definition

  • The problem definition distinguishes latent true preferences from observed binary feedback, with sample-wise noise ratios describing each direction of label flipping.The ideal objective minimizes clean risk, but the clean training set is unobservable; training instead uses noisy feedback.Source: §2.2 · pp. 2–3
  • When noise ratios are high, naive noisy-risk optimization can be dominated by losses on corrupted labels and push the reward model to memorize them.The passage motivates estimating sample-wise noise ratios for loss correction, using a few clean feedback because the ratios are generally unidentifiable from noisy feedback alone.Source: §2.2 · p. 3

3 Methodology

CleanRM addresses sample-wise feedback noise by estimating noise ratios from noisy data calibrated against a few clean feedback, then training with a corrected surrogate loss. Its joint-distribution alignment addresses the limits of marginal or conditional comparisons, while the workflow separates estimation from reward-model training.

3.1 Motivation

  • Feedback noise varies across samples, unlike the uniform-noise assumption used by many denoising methods, and learning from only a few clean feedback remains an open challenge.Clear instruction violations can have consistent judgments, while reasonable responses that trade off qualities can receive more variable feedback.Source: §3.1 · pp. 3–4

3.2 Sample-wise Noise Ratio Estimation

  • CleanRM estimates two directional noise ratios per input, inverts noisy labels toward estimated true preferences, and aligns recovered noisy samples with clean feedback using joint Wasserstein distance.Joint alignment serves as a proxy for conditional preference alignment: it upper-bounds expected conditional discrepancy and can be estimated by pooling samples.Source: §3.2 · p. 4

3.3 Noise-Corrected Surrogate Reward Modeling

  • With correct noise ratios, the noise-corrected surrogate objective is unbiased for clean risk, provided the directional noise ratios sum to less than one.The method substitutes estimated ratios in practice; the theorem's unbiasedness guarantee assumes the ratios used are true.Source: §3.3 · p. 5

3.4 The Workflow of CleanRM

  • CleanRM first trains the noise-ratio estimator by aligning recovered noisy-side pairs with clean feedback, then trains the reward model using the corrected surrogate objective.The estimator predicts ratios and recovers preferences before constructing the joint distributions; the reward-model stage evaluates candidate-label losses and updates on the surrogate objective.Source: §3.4 · pp. 5–6

4 Experiments

The experiments evaluate CleanRM across five questions, using semi-synthetic sample-wise noise on three preference datasets and comparing it with denoising baselines. The setup reserves clean feedback for calibration and keeps test data clean for evaluation.

4 Experiments

  • The evaluation asks whether CleanRM performs well, why it works, generalizes across architectures, is sensitive to clean-feedback size, and improves downstream RLHF.These five questions cover baseline comparisons, ablations, backbone scales, noise levels, and downstream policy fine-tuning.Source: §4 · p. 6

4.1 Experimental setup

  • Experiments use HelpSteer, UltraFeedback, and PKU-SafeRLHF, reserve 5% of clean training data as clean feedback, and evaluate on uncorrupted test data.The remaining training data and validation data are corrupted using sample-wise rates derived from a naive reward model; that model is used only to generate semi-synthetic noise.Source: §4.1 · p. 6
  • Baselines span statistically consistent and heuristic denoising methods, each with or without clean feedback; Table 1 compares them with CleanRM under sample-wise noise.A Naive model trained directly on noisy feedback is also included as a reference.Source: §4.1 · p. 6, Table 1 · p. 7

Implementation Details.

CleanRM leads the benchmark comparisons, and ablations show that both sample-wise noise estimation and the unbiased surrogate contribute to its performance. The section also evaluates generalization across backbones, sensitivity to clean-feedback availability and noise, and downstream RLHF alignment.

Implementation Details.

  • All methods use an LLM backbone and an MLP reward head, with the same backbone initialization and reward-head dimensions; CleanRM adds a noise-ratio estimator.Training uses Adam for up to 600 epochs, early stopping patience 30, batch size 512, learning rate 5 × 10^-4, and Sinkhorn regularization 0.1.Source: Implementation Details · p. 7

4.2 Main Results (RQ1)

  • CleanRM improves R2 over TrainRef by +0.052 on HelpSteer, +0.013 on UltraFeedback, and +0.032 on PKU-SafeRLHF, while achieving the highest R2 and lowest MSE on all three benchmarks.The comparison is under sample-wise feedback noise.Source: §4.2 · p. 7

4.3 Ablation Study (RQ2)

  • On HelpSteer, replacing sample-wise estimation with a global noise ratio lowers R2 from 0.343 to 0.100, while replacing the unbiased surrogate with forward correction lowers it to 0.232.Both ablated variants underperform complete CleanRM across datasets and metrics.Source: §4.3 · p. 8, Table 2 · p. 8

4.4 Generalization across Model Architectures (RQ3)

  • Across Qwen3 and LLaMA3.1 backbone scales, CleanRM improves R2 by 0.010–0.017 and reduces MSE by 3.4%–7.7% relative to TrainRef on PKU-SafeRLHF.CleanRM also maintains improvements over the Naive baseline as model scale increases.Source: §4.4 · p. 8, Table 3 · p. 9

4.5 Hyperparameter Sensitivity (RQ4)

  • R2 and MSE improve as clean feedback increases and plateau around 0.6; gains from more clean feedback are more pronounced at noise ratio ρ = 0.15 than at ρ = 0.05.At ρ = 0.2 and Nclean clean = 0.2, R2 stays above 0.47.Source: §4.5 · p. 8, Figure 2 · p. 8

4.6 Downstream RLHF (RQ5)

  • On Qwen3-8B, CleanRM improves the FFT safety score by 79.5% over Naive and 16.3% over TrainRef, and achieves the best downstream RLHF performance across benchmarks and policy models.The policies are fine-tuned via GRPO; safety is evaluated across HarmBench, FFT, and DAN.Source: §4.6 · p. 9, Table 4 · p. 9

5 Conclusion

CleanRM addresses sample-wise feedback noise by calibrating noise ratios from a few clean feedback and using them in a clean-risk-unbiased surrogate. The paper situates this approach against statistically consistent and heuristic denoising methods, and reports improved reward accuracy and downstream RLHF alignment.

5 Conclusion

  • CleanRM uses a few clean feedback to calibrate sample-wise noise ratios through optimal transport and trains reward models with a noise-corrected surrogate that is unbiased for clean risk.Experiments report improved reward prediction accuracy over competitive denoising baselines and stronger downstream RLHF alignment.Source: §5 · p. 9

A Related Work

  • CleanRM combines clean-risk guarantees with empirical noise mitigation by calibrating sample-wise ratios through optimal transport and correcting supervision with a surrogate loss.The related-work overview frames these as strengths drawn from statistically consistent and heuristic approaches.Source: A Related Work · p. 14

A.1 Statistically Consistent Approaches

  • Global transition-matrix corrections assume uniform noise ratios and may retain finite-sample bias, whereas CleanRM aligns noisy and clean feedback in joint input-preference space.The passage also describes existing sample-wise approaches, while emphasizing response-pair-specific noise in reward modeling.Source: §A.1 · p. 14

A.2 Heuristic Approaches

  • Heuristic filtering and regularization can discard hard-but-correct feedback and lack an unbiased objective under sample-wise noise; CleanRM retains noisy samples and corrects their supervision.The passage describes sample selection and noise-aware regularization as common heuristic strategies.Source: §A.2 · p. 14

B.1 Proof of Theorem 2.1

The proof frames naive noisy risk through a preference-dependent, sample-wise corruption probability and distinguishes clean from noisy losses. It also states a joint-to-conditional Wasserstein bound and conditions for equality.

B.1 Proof of Theorem 2.1

  • The risk-decomposition proof defines a preference-dependent noise ratio and analyzes the noisy risk by conditioning on the clean sample and feedback-corruption process.The clean and noisy losses are defined using the true preference and its flipped label, respectively.Source: §B.1 · pp. 14–15

B.2 Conditional Wasserstein Bound

  • For p ≥ 1, joint-distribution Wasserstein discrepancy upper-bounds expected conditional-distribution Wasserstein discrepancy, with equality when p = 1 or the conditional term is constant in X.Source: §B.2 · p. 15
  • The joint-alignment proof uses a nonnegative discriminative ground cost: zero cost forces matched pairs to coincide and the coupling marginals to coincide.It then derives zero joint Wasserstein discrepancy and applies the conditional Wasserstein bound.Source: §B.2 · p. 15

B.3 Proof of Theorem 3.2

Under local true-preference identifiability and valid, non-degenerate estimated ratios, conditional-distribution alignment identifies the sample-wise noise ratios. The surrogate-loss derivation solves the unbiasedness condition when the directional noise rates sum to less than one.

B.3 Proof of Theorem 3.2

  • Under local true-preference identifiability and valid, uniformly non-degenerate estimated ratios, conditional-distribution convergence implies that estimated directional noise ratios converge in probability to their true values.The stated conditions include local matching of both true preference states and positive mass for each state in neighborhoods.Source: §B.3 · p. 16
  • For each clean preference state, the proof uses its corresponding conditional feedback-flip probability to establish convergence of the directional noise-ratio estimates.Source: §B.3 · p. 16

B.4 Derivation of the Noise-Corrected Surrogate Loss

  • The surrogate in Eq. (10) is obtained by solving the unbiasedness condition; when µ(x) + ν(x) < 1, the resulting linear system has a unique solution.Evaluating that solution at a noisy sample recovers the stated surrogate loss.Source: §B.4 · p. 17

B.5 Proof of Theorem 3.3

Theorem 3.3 establishes that, with true noise ratios whose sum is below one, the corrected loss is unbiased for each fixed clean sample. Consequently, the surrogate objective recovers clean risk in expectation.

B.5 Proof of Theorem 3.3

  • If µ(x) + ν(x) < 1 and Eq. (10) uses the true noise ratios, the corrected loss is unbiased for any fixed (x, r∗).Source: §B.5 · p. 17
  • The expected corrected loss for either true preference equals the corresponding clean loss, yielding pointwise unbiasedness under the feedback-corruption process.Source: §B.5 · p. 17
  • By linearity of expectation and pointwise unbiasedness, the surrogate objective recovers the clean risk in expectation.Source: §B.5 · p. 17

C.1 Dataset Descriptions

The experiments use preference corpora to assess reward modeling and safety suites to assess downstream policy behavior. Implementation details cover reward-model training, policy optimization, and automatic safety scoring.

C.1 Dataset Descriptions

  • Three preference corpora use binarized scalar proxies: HelpSteer helpfulness, UltraFeedback Overall Score, and PKU-SafeRLHF severity tier.Each corpus is split into a small clean feedback set and a noisy training set, with the original test split held out.Source: §C.1 · p. 18
  • Three safety suites assess red-teaming, toxicity elicitation, and in-the-wild jailbreak behavior after RLHF fine-tuning.The evaluation uses HarmBench, the Toxicity subset of FFT, and DAN.Source: §C.1 · p. 18

C.2 Implementation Details

  • The noise-ratio estimator and corrected reward model share a warm-started LLM backbone with separate MLP heads, then train in two stages.The estimator predicts noise ratios using widths [256, 64, 2], followed by reward-model training with widths [256, 64, 1].Source: §C.2 · p. 18
  • Downstream RLHF uses GRPO with Qwen3-8B and LLaMA3.1-8B policies, trained on PKU-SafeRLHF prompts for 20 update steps.A KL coefficient of 1 × 10−3 constrains policies toward their reference models.Source: §C.2 · p. 19
  • DeepSeek-V4 Flash labels generated responses safe or unsafe, and the labels are aggregated into a Safety Score.Higher Safety Score values indicate better safety alignment.Source: §C.2 · p. 19

D More Experimental Results

Supplementary experiments extend the sensitivity study to PKU-SafeRLHF and include a qualitative HarmBench case study. Across the tested PKU-SafeRLHF grid, CleanRM remains reliable, while gains from more clean feedback flatten under the most severe noise.

D More Experimental Results

  • The supplementary section also extends the UltraFeedback sensitivity study to PKU-SafeRLHF and adds a qualitative HarmBench case study of downstream policy behavior.The case study illustrates how CleanRM’s noise-corrected reward steers policies toward safer responses under adversarial jailbreak prompts.Source: D More Experimental Results · pp. 19–20

D.1 Hyperparameter Sensitivity

  • The PKU-SafeRLHF sensitivity sweep varies the clean feedback ratio over {0.2, 0.4, 0.6, 0.8, 1.0} and noise ratio over {0.05, 0.10, 0.15, 0.20}.The results are shown in Figure 3.Source: §D.1 · p. 20
  • R² stays above 0.80 and MSE below 0.05 across the PKU-SafeRLHF grid, including at Nclean=0.2.At ρ ∈ {0.05, 0.10, 0.15}, more clean feedback maintains or improves R² and reduces MSE; at ρ = 0.20, both curves largely flatten.Source: Figure 3 · p. 20, §D.1 · p. 20

D.2 Case Study on Downstream RLHF

The HarmBench case study contrasts policies steered by different reward models under adversarial jailbreak prompts. Baseline-steered policies comply with harmful requests, while CleanRM-steered policies refuse and redirect.

D.2 Case Study on Downstream RLHF

  • Across Qwen3-8B and LLaMA3.1-8B, baseline-steered policies comply with adversarial jailbreaks, while CleanRM-steered policies refuse and redirect.The tables show responses under ALPHABREAK and LUCIFER persona prompts.Source: §D.2 · p. 20, Table 5 · p. 21, Table 6 · p. 22
  • Under the LUCIFER persona, FasTEN and TrainRef responses continue past an initial ChatGPT refusal into harmful compliance.The case descriptions characterize both responses as providing actionable harmful guidance.Source: §D.2 · p. 23
  • The CleanRM case response refuses the request and identifies it as illegal and harmful.It provides no guidance on the harmful activity.Source: §D.2 · p. 23

Related papers

Loading 2610.21138v1…