PaperlayerView exact revision on arXiv

Source-linked AI summary

Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers

Xin-Qiang Cai, Wei Wang, Feng Liu, Tongliang Liu, Gang Niu, Masashi Sugiyama

arXiv:2510.00915v4cs.LGcs.AI

TL;DR

Imperfect automated verifiers can reject valid solutions and accept invalid ones, motivating corrections for verifier noise in policy learning. The paper models this unreliability explicitly and introduces backward and forward corrections, with reported advantages even under challenging non-iid noise.

  • Problem

    Automated verifiers are brittle and inevitably produce false positives and false negatives, compromising reinforcement learning with verifiable rewards.

  • Method

    The paper models verifier unreliability as a verifier-conditional noise channel and introduces backward and forward corrections at the policy-learning interface.

  • Results

    The proposed methods show clear advantages, including under a more challenging non-iid verifier noise setting.

  • Takeaways & Limitations

    Verifier-noise corrections can improve policy learning without requiring perfect verifier reliability.

  • Takeaways & Limitations

    The theorem is an oracle-rate population statement, and backward correction can have high variance when the total noise rate is high.

Abstract

from arXiv · show

Reinforcement Learning with Verifiable Rewards (RLVR) replaces costly human labeling with automated verifiers. To reduce verifier hacking, many RLVR systems binarize rewards to $\{0,1\}$, but imperfect verifiers inevitably introduce \emph{false negatives} (rejecting correct answers) and \emph{false positives} (accepting incorrect ones). We formalize verifier unreliability as a stochastic reward channel with asymmetric noise rates $\rho_0$ and $\rho_1$ -- the FP rate and the FN rate, respectively. From this abstraction we derive two lightweight corrections: (i) a \emph{backward} correction that yields an unbiased surrogate reward and thus an unbiased policy-gradient estimator in expectation, and (ii) a \emph{forward} correction that reweights score-function terms so the expected update aligns with the clean gradient direction and requires only the FN rate. We implement both as lightweight hooks in a group relative policy optimization pipeline, both corrections improve RLVR for math reasoning under synthetic and real verifier noise, with the forward variant being more stable under heavier noise. Finally, an appeals mechanism with a lightweight LLM verifier estimates the FN rate online and further improves performance.

1. Introduction

RLVR uses automated verifiers but suffers from false positives and false negatives that distort policy learning. This paper models verifier errors as reward noise, derives backward and forward corrections, and reports improved math-reasoning training under noisy verification.

  • Problem formulation: The paper models verifier unreliability as a stochastic reward channel with verifier-conditional false-positive and false-negative rates.The channel corrupts the latent true reward with asymmetric noise rates.
  • Methods: Backward correction inverts the noise process to produce an unbiased surrogate reward usable as a drop-in replacement for advantage estimation.Its expected policy-gradient estimator is unbiased when the noise rates are available.
  • Methods: Forward correction reweights policy-gradient terms so the expected update aligns with the clean gradient direction while requiring only an estimate of the false-negative rate.This avoids requiring the often harder-to-estimate false-positive rate [38].
  • Evaluation and contributions: The corrections are implemented as lightweight hooks in a GRPO-based RLVR pipeline and evaluated on competitive math-reasoning benchmarks [27].The contribution includes a formal verifier-channel model, two policy-gradient corrections, and online noise estimation for practical deployment.
  • Results: Both corrections outperform uncorrected training and approach noise-free performance, while forward correction converges fastest and most stably.An appeal mechanism using a low-cost LLM verifier estimates false negatives online and recovers near-oracle performance; both methods remain effective under non-i.i.d. noise.

2. Related Works

Prior work improves reasoning through prompting, search, reranking, verifier design, and label-noise correction. In contrast, this paper directly corrects the RLVR policy-gradient estimator so learning remains aligned with the clean objective under verifier noise.

  • Reasoning with LLMs: LLM reasoning research includes chain-of-thought prompting, self-consistency, decomposition, and search-based strategies such as Tree-of-Thoughts [39] [42].These approaches primarily alter inference-time reasoning or exploration rather than the noisy reward-learning interface.
  • RLVR and verifier reliability: RLVR pipelines combine on-policy reinforcement learning with programmatic or LLM-based verifiers, but LLM judges and brittle parsers introduce false positives and false negatives [21].LLM judges can exhibit systematic biases, while exact-match or brittle rule-based checkers miss algebraic equivalences and text-formatted answers.
  • Verifier reliability: Verifier-focused work improves judges, appeals, or evaluation suites, including TinyV, which recovers many rule-based false negatives with minimal overhead [38].This paper instead treats verifier errors as verifier-conditional noise and corrects policy optimization directly.
  • Learning with noisy labels: Learning-with-label-noise methods correct losses using noise-transition models or down-weight suspected noisy examples [22, 24, 30, 16].Related reinforcement-learning work models reward corruption and derives unbiased surrogate rewards for Q-learning.
  • Positioning: Unlike supervised-loss substitutions, this work instantiates backward and forward corrections inside the RLVR policy-gradient estimator to stabilize and align learning under verifier noise.The approach is complementary to inference-time strategies and reranking schemes [3].

3. Problem Setup and Algorithms

The paper models verifier errors as an asymmetric stochastic reward channel and derives two policy-gradient corrections: backward correction for unbiasedness and forward correction for directional alignment.

  • 3.1. Problem Setup: The verifier channel assumes instance-independent false-positive and false-negative rates with conditional independence, and is invertible when ρ0 + ρ1 < 1.When rates are instance-dependent, the formulas correct an effective marginal channel rather than eliminating all bias.
  • 3.1. Problem Setup: Naively optimizing noisy verifier rewards produces a biased policy gradient because the policy learns from verifier mistakes.
  • 3.2. Noisy Policy Gradient with Backward Correction (PGBC): PGBC inverts the verifier channel to construct a corrected reward that is unbiased for the clean reward when the true rates are known.The corrected reward can replace the noisy reward in standard policy-gradient formulations, although the theorem is an oracle-rate population result before finite-sample normalization, clipping, or KL terms.
  • 3.3. Noisy Policy Gradient with Forward Correction (PGFC): PGBC can become unstable when 1 − ρ0 − ρ1 is small and requires accurate estimates of both noise rates, motivating PGFC as a practical alternative.PGFC is introduced to mitigate these reward-estimation and rate-estimation challenges.
  • 3.3. Noisy Policy Gradient with Forward Correction (PGFC): PGFC reweights score-function terms using the observed noisy reward so its expected update is parallel to the clean policy gradient.The forward correction requires an estimate of the false-negative rate and avoids PGBC’s inverse operation, which can inflate variance when total noise is high.

4. Experiments

Experiments evaluate backward and forward reward corrections under synthetic, real-world, mis-specified, and non-iid verifier noise. Across math reasoning settings, both corrections recover performance lost to noisy rewards, with forward correction generally more stable.

  • 4.1. Experimental Setup: The evaluation uses three small backbones, six verifiable math suites, Pass@1 and Pass@8, and five random seeds with otherwise matched rollout settings.Corrections are implemented as drop-in hooks at the GRPO advantage-construction stage.
  • 4.2. Synthetic Noise: Both corrections substantially reduce the performance gap between noisy-reward training and the clean-reward oracle across models and math benchmarks.On DeepSeek-R1-Distill-Qwen-1.5B, PGBC and PGFC nearly match oracle performance; PGFC is consistently as good as or slightly better than PGBC.
  • 4.3. Real-World Verifier Noise: Real-world verifier results show that direct LLM-as-Judge rewards underperform rule-based rewards, while Appeals provides a modest lift and Appeals+PGFC delivers the strongest, most stable improvements [5] [29].Appeals+PGFC estimates the FN rate from appeals and applies forward correction directly to the policy gradient.
  • 4.4. Robustness to Noise Mis-Specification: Backward correction remains strong when noise is underestimated but degrades as estimated FP and FN rates are overestimated and 1−ˆρ0−ˆρ1 shrinks.Its inverse factor amplifies reward variance and estimation error when the effective denominator becomes small.
  • 4.4. Robustness to Noise Mis-Specification: Forward correction has a flatter response around the true FN rate because it changes per-sample weights without division, yielding lower-variance policy gradients.The robustness experiment varies estimated rates around injected noise of ρ0=0.1 and ρ1=0.2; forward correction requires only the estimated FN rate.
  • 4.5. Non-IID Verifier Noise: Under format-dependent non-iid verifier noise, both corrections still improve over uncorrected training, with PGFC achieving the strongest performance.The stress test uses a shared FP rate of ρ0=0.2 and format-dependent FN rates of ρ1=0.05 and ρ1=0.45.

5. Conclusion

The paper models verifier unreliability as asymmetric FP/FN noise and corrects it at the policy-learning interface. Both corrections improve RLVR under synthetic, real-world, and non-iid noise, while forward correction is more stable and needs only an FN estimate.

  • Problem: The paper addresses false positives and false negatives in binarized RLVR rewards, which can degrade the learning signal for reasoning models.It frames verifier unreliability as a conditional noise channel with FP rate ρ0 and FN rate ρ1.
  • Contributions: Backward correction debiases observed rewards into an unbiased gradient estimator, whereas forward correction reweights score-function terms to align expected updates with the clean gradient.Backward correction requires both FP and FN estimates; forward correction requires only the FN rate estimate.
  • Empirical Findings: Both corrections improve RLVR across synthetic and real verifier noise, and the forward variant is strongest and most stable in the reported experiments.An appeals mechanism using a lightweight verifier estimates the FN rate online and further improves performance.
  • Scope: The constant-noise model is only an approximation because real-world verifier noise can depend on content, formatting, prompt style, and verifier family.The paper motivates future work on instance-dependent reliability models and adversarially robust verifiers.

A. From REINFORCE-Style Corrections to GRPO Updates

The appendix analyzes how reward corrections interact with group-centered GRPO updates. Under stated independence and score-function assumptions, forward-corrected updates preserve the clean gradient direction, while implementation details determine whether corrections remain behaviorally distinct.

  • Binary-Proxy Equivalence: For binary rewards, PGBC can differ from the noisy reward only by an additive constant, while the implemented PGFC proxy may be a positive rescaling that centering or standardization cancels.Distinct behavior should therefore be audited on actual token-level advantages after centering and normalization.
  • Group Centering: Group centering preserves the expected update direction for i.i.d. group samples when the score-function expectation is zero.This result applies to scalar trajectory proxies and yields an expected centered update proportional to E[uG|x].
  • Forward Correction: PGFC is directionally correct when its weighted proxy uses w0=ρ1−1 and w1=ρ1 under the verifier reward-channel assumptions.The expected centered update remains aligned with the clean policy-gradient direction.
  • Standardization, Clipping, and KL: GRPO standardization acts as an adaptive positive scaling, whereas clipping and KL regularization stabilize updates but can move them away from the exact unclipped REINFORCE direction.The exact direction-preservation result covers centering; strict expectation equality after standardization requires additional concentration assumptions.
  • Practical Guarantees: The corrections target the reward-gradient component, while global scaling can be absorbed into the effective learning rate or interpreted as changing the reward–KL trade-off.Clipping may also intentionally limit step size when it activates.

B. Implementation Details

PGBC and PGFC enter GRPO through scalar trajectory coefficients, preserving their distinct utility- and score-correction roles. The implementations avoid current-group standardization and account for baselines, scaling, clipping, KL regularization, and verifier-noise assumptions.

  • GRPO integration: Both corrections are inserted at the scalar coefficient level, broadcast to response tokens, and passed directly to the clipped GRPO surrogate without conversion into a binary proxy.The usual KL penalty remains unchanged.
  • Correction placement: PGBC replaces the observed verifier reward with an estimate of the clean reward, while PGFC replaces the usual reward-derived advantage with a signed channel-calibrated multiplier.PGBC uses both estimated FP and FN rates; PGFC is a score-level correction requiring the FN rate for directional alignment.
  • PGBC: With correctly specified channel rates, PGBC preserves the expected score-function update and matches the clean reward-gradient component in expectation.The leave-one-out baseline is conditionally independent of the current sample’s score contribution; PGBC uses fixed or slowly moving positive scales rather than current-group standard deviation.
  • PGFC: When the FN rate is correct, PGFC preserves the clean reward-gradient direction and can calibrate its magnitude with a suitable positive scale, while leaving the KL gradient uncorrected.The scale and KL coefficient jointly determine the effective reward–KL trade-off.
  • Avoiding degeneracy: Applying current-group z-score standardization to binary affine correction quantities can make PGBC or PGFC exactly equivalent to the uncorrected noisy baseline.The implementations therefore use scalar coefficients with fixed or running positive scales and implementation-matched noisy controls.
  • Assumptions and diagnostics: The directional-alignment analysis assumes verifier errors are conditionally independent of policy score terms given correctness; trajectory-dependent errors add a covariance bias term.The paper recommends combining channel correction with verifier hardening or instance-dependent reliability modeling when this covariance may be non-negligible.

C.1. Performance under Pass@8 Metric

Across math reasoning evaluations with synthetic and real verifier noise, the corrections improve performance, with Appeals+PGFC using online FN-rate estimation for further gains. Direct LLM-as-Judge reward training underperforms the rule-based baseline, while corrections remain complementary to stronger checkers.

  • Verifier comparison: LLM-as-Judge as the primary reward source systematically underperforms the rule-based pipeline, consistent with susceptibility to biases and gaming behaviors.The comparison supports retaining the rule-based verifier as the primary reward signal in this setup.
  • Online estimation: Appeals+PGFC dynamically estimates the false-negative rate online from the appeal stream, directly applying forward correction to the policy gradient.A simple appeal mechanism alone provides only a small but consistent improvement while noisy signals still influence primary updates.
  • Real-world noise: Both corrections produce the highest and most robust pass@8 improvements across nearly all evaluated datasets under real-world verifier noise.The evaluation averages pass@8 over 16 samples and 5 random seeds.
  • Stronger verifiers: Backward and forward corrections continue to improve performance when paired with stronger verifier baselines, indicating complementarity with verifier engineering.The paper reports that PGFC improves over a stronger checker rather than merely replacing it.

C.3. Training Dynamics under Synthetic Noise

Training-dynamics analysis tracks reward variability, normalized advantages, and gradient norms to test whether correction changes optimization rather than only endpoint scores. Both PGBC and PGFC measurably alter these intermediate statistics relative to noisy training.

  • Measurement setup: The analysis monitors group reward standard deviation, normalized advantage standard deviation, and gradient norm across training steps under synthetic i.i.d. noise.Its purpose is to isolate active changes in advantage construction from merely shifted final scores.
  • Training dynamics: Both PGBC and PGFC measurably alter reward, normalized-advantage, and gradient-norm statistics throughout training relative to the uncorrected noisy baseline.These changes provide evidence that the corrections dynamically adjust the policy gradient during optimization.
  • Appeal efficiency: Appeals+PGFC achieves significant gains in pass@1 and pass@8 without additional verifier queries or extra computational overhead.It uses the same appeal stream as the standard Appeals baseline, and auxiliary TinyV calls remain a small fraction of training rollouts.
  • General capability retention: PGBC and PGFC recover most of the retention loss from uncorrected noisy rewards and remain close to Base/Oracle on auxiliary non-math evaluations.This assesses whether correction preserves broader benchmark capabilities after RLVR training.

D.1. Proof of Proposition 3.2

The proofs derive the verifier channel’s affine conditional expectation and use it to establish backward-reward unbiasedness and forward-update alignment. These results rely on the channel model and conditional independence assumptions used in the score-function analysis.

  • Verifier channel: The verifier reward has conditional expectation E[˜R | R*] = ρ0 + (1 − ρ0 − ρ1)R*, combining false positives and false negatives in an affine channel.The channel probabilities are P(˜R = 1 | R* = 0) = ρ0 and P(˜R = 0 | R* = 1) = ρ1.
  • Backward correction: Provided 1 − ρ0 − ρ1 > 0, backward correction inverts the affine channel so the corrected reward is unbiased for the latent clean reward.The proof establishes unbiasedness both conditionally and unconditionally.
  • Forward correction: Forward correction assigns outcome-dependent weights whose conditional expectations distinguish positive and negative clean rewards, yielding the required score-level identities.The proof evaluates the weighted noisy reward separately for R* = 1 and R* = 0.
  • Policy-gradient alignment: Under conditional independence between verifier noise and policy score terms given the clean reward, the expected corrected update aligns with the clean gradient up to a channel-dependent factor.The factorization E[w˜R G | R*] = E[w˜R | R*]E[G | R*] is central to the derivation.

E.4. Online FN-Rate Estimation Hyperparameters and Ablations

The appeals mechanism samples primary-verifier negatives for TinyV re-evaluation and estimates the FN rate online, with EMA smoothing and tunable cost–accuracy controls.

  • E.4. Online FN-Rate Estimation Hyperparameters and Ablations: EMA smoothing is applied to the online FN-rate estimate, with ablations targeting the mechanism’s cost–accuracy trade-off.The stated ablations focus on key knobs governing appeals cost and estimation accuracy.
  • Appeals protocol and estimator: The appeals protocol samples items labeled negative by the primary verifier with probability q and re-evaluates them using the lightweight TinyV verifier.At training step t, appealed negatives that TinyV flips positive contribute to FN-rate estimation.
  • Appeals protocol and estimator: The estimator combines appealed-negative flips with a Horvitz–Thompson estimator and a Beta prior to estimate the FN rate.The passages identify a Beta prior and the Horvitz–Thompson estimator as components of the online estimate.
  • Default hyperparameters: Real-world-noise runs use appeal probability q = 0.1, Beta prior pseudocounts α = β = 10−5, and EMA decay λ = 0.9.The expected TinyV calls per policy update are reported as q · |N (t) R | and serve as a cost proxy.

F. False Positive (FP) and False Negative (FN) Examples

The examples show that automated verifiers can reject correct answers because of formatting or representation mismatches, while LLM judges can accept malformed solutions.

  • False negatives and false positives: 52 FNs and 0 FPs out of 500 items occurred for the rule-based verifier, corresponding to marginal rates of 10.4% and 0%.These examples used Qwen2.5-Math-7B on MATH500.
  • False negatives and false positives: 81 FNs and 168 FPs out of 500 items occurred for the LLM verifier, corresponding to marginal rates of 16.2% and 33.6%.The LLM verifier exhibited both rejection of correct answers and acceptance of incorrect ones.
  • False negatives and false positives: An LLM judge produced a false positive by over-crediting a malformed response to the seven-island probability problem.The example’s correct probability is 448/15625, while the malformed response changes the arithmetic expression.
  • False negatives and false positives: A rule-based checker produced a false negative by rejecting a numerically correct answer because it expected a fraction rather than a decimal.The correct answer was 448/15625, equal to 0.028672, but the verifier rejected the representation.
Loading 2510.00915v4…