Source-linked AI summary
Can LLMs Learn to Reason Robustly under Noisy Supervision?
Shenzhi Yang, Guangcheng Zhu, Bowen Song, Sharon Li, Haobo Wang, Xing Zheng, Yingfan Ma, Zhongqi Chen, Weiqiang Wang, Gang Chen
TL;DR
Noisy supervision in RLVR is underexplored despite expert scarcity and rollout-dependent label reinforcement. This paper analyzes inactive and active noise, identifies Early Correctness Coherence, and proposes OLR, which refines labels using majority answers selected by rollout trends and historical consistency. Across noise ratios from 0.1 to 0.9, OLR consistently improves performance under both noise types across in-distribution and out-of-distribution evaluations.
Problem
RLVR relies on abundant carefully curated labels, but noisy-label robustness remains underexplored despite rollout-dependent reinforcement and expert scarcity.
Method
OLR progressively replaces potentially noisy labels with policy-generated majority answers when pass-rate slope is positive and historical consistency is stable.
Results
OLR consistently improves robustness across noise ratios from 0.1 to 0.9 under inactive and active noise on in-distribution and out-of-distribution evaluations.
Takeaways & Limitations
Early Correctness Coherence provides a basis for gradually refining RLVR training targets as correct majority answers emerge during policy improvement.
Takeaways & Limitations
The theoretical analysis assumes positive cross-sample coupling and a mean-drift condition, including a noise ratio below the threshold ρc.
Abstract
from arXiv · showhide
Reinforcement Learning with Verifiable Rewards (RLVR) effectively trains reasoning models that rely on abundant perfect labels, but its vulnerability to unavoidable noisy labels due to expert scarcity remains critically underexplored. In this work, we take the first step toward a systematic analysis of noisy label mechanisms in RLVR. In contrast to supervised classification, most RLVR algorithms incorporate a rollout-based condition: a label's influence on training is contingent on whether the current policy can generate rollouts that realize it, a property that naturally extends to noisy labels. Based on this observation, we distinguish two types of noise: inactive noisy labels, which reduce data efficiency, and active noisy labels, which are reinforced and risk skewing the model toward incorrect distributions. From experiments on training with noisy samples, we identify an Early Correctness Coherence phenomenon: although noisy samples begin to lag behind in later stages, accuracy on both clean and noisy samples increases similarly in early training. Motivated by this dynamic, we propose Online Label Refinement (OLR), which progressively corrects potentially noisy labels with majority-voted answers when two conditions hold: a positive slope in the majority answer's rollout pass rate and stable historical consistency across updates, enabling gradual self-correction as the policy improves. We evaluate OLR on six in-distribution mathematical reasoning benchmarks (AIME24/25, AMC, MATH-500, Minerva, and Olympiad) and three out-of-distribution tasks (ARC-c, GPQA-diamond, and MMLU-pro). Across noise ratios from 0.1 to 0.9, OLR consistently improves robustness under both inactive and active noisy-label settings, achieving average gains of 3.6% to 3.9% on in-distribution benchmarks and 3.3% to 4.6% on out-of-distribution evaluations.
1 Introduction
RLVR’s reliance on carefully curated labels leaves it vulnerable to noisy supervision, whose impact depends on whether the policy can realize the noisy label. The paper identifies early coherence between clean and noisy samples and proposes OLR to refine labels during training.
- Noise mechanisms: RLVR noisy labels are inactive when unrealizable by the current policy and active when realizable and therefore reinforced.Inactive noise wastes rollouts and reduces data efficiency, whereas active noise can steer the policy toward incorrect distributions.
- Online Label Refinement: OLR refines potentially noisy labels when the majority answer’s rollout pass rate has a positive slope and remains historically consistent.The two signals capture increasing model confidence and filter transient majority answers.
- Results: 3.6% and 3.3% average gains under inactive noise occur on six in-distribution mathematical benchmarks and three out-of-distribution tasks, respectively.OLR is evaluated across noise ratios from 0.1 to 0.9.
- Results: 3.9% and 4.6% average gains under active noise occur on in-distribution and out-of-distribution evaluations, respectively.The reported benchmarks include six mathematical reasoning tasks and ARC-c, GPQA-diamond, and MMLU-Pro.
- Training dynamics: Early Correctness Coherence describes similar accuracy increases for clean and noisy samples early in training before later divergence.The phenomenon suggests that correct answers already exist latently for noisy samples and become increasingly expressed during early training.
2 Related Work
Existing noisy-label learning methods largely target static classification datasets, while RLVR introduces open-ended labels, on-policy generation, and rollout-dependent training influence.
- Related work: Traditional noisy-label learning is established but rarely accommodates RLVR’s generation and on-policy dynamics.RLVR differs from static classification through its open-ended output space and policy-generated data.
- Related work: Noise transition matrix estimation is unsuitable for RLVR because noisy labels form an open set without clear class boundaries.
3 Method
The method formalizes rollout-dependent noisy labels, analyzes Early Correctness Coherence, and uses OLR to replace unreliable labels with increasingly reliable majority answers. OLR’s criteria and theoretical analysis support denoising during RLVR training.
- Problem formulation: RLVR trains an on-policy policy with sampled rollouts and verifier rewards, making label influence dependent on rollout feasibility.The framework uses K rollouts sampled from πθ and reward-based advantages.
- Problem formulation: A noisy label is inactive when πθ(ỹ|x)=0 and active when πθ(ỹ|x)>0.
- Early Correctness Coherence: Early Correctness Coherence occurs when clean and noisy samples improve similarly early in training before noisy-sample accuracy lags later.The paper attributes this behavior to positive cross-sample coupling through shared parameters.
- Online Label Refinement: OLR estimates solution probabilities from rollout majority answers and refines labels using positive convergence slope and historical consistency.Historical consistency filters stochastic or accidental majority answers, while a positive slope indicates increasing model confidence.
- Online Label Refinement: After an initial early learning phase, OLR uses an effective label for reward computation and replaces the original label when its criteria are satisfied.The method is incorporated with GRPO, and its additional time overhead is described as almost negligible.
- Theoretical analysis: OLR’s theoretical analysis states that replacement selects the correct solution with probability at least 1−ϵ and reduces the effective noise ratio below ρ.The resulting analysis increases the tolerable noise threshold from ρc to ρOLR.
4 Experiment
Experiments evaluate OLR across noise types, ratios, baselines, dataset sizes, training dynamics, and criterion sensitivity. OLR consistently improves robustness, with both slope and historical consistency contributing to performance.
- 4.1 Setup: Experiments inject active or inactive noise at ratios from 0.1 to 0.9 into DAPO-Math training data and evaluate in-distribution and out-of-distribution reasoning.The default setting uses 800 instances; a 4,000-instance setting uses a 50% noise ratio.
- 4.2 Experimental Results: 3.6% average gains on six in-distribution benchmarks and 3.3% on three out-of-distribution benchmarks occur under inactive noise; active-noise gains are 3.9% and 4.6%, respectively.These gains hold across noise ratios from 0.1 to 0.9.
- 4.2 Experimental Results: At 50% noise, OLR exceeds unsupervised baselines by 5.0% and 8.0% on in-distribution settings and by 3.3% on out-of-distribution tasks.The comparisons are reported for active and inactive noise, respectively, for the in-distribution gains.
- 4.2 Experimental Results: During active-noise training, majority-answer accuracy exceeds 60% early, plateaus near 60%, then rises above 80% after OLR.OLR-selected samples exceed 90% majority-answer accuracy, while unselected samples remain below 70%.
- 4.2 Experimental Results: 4.6% average in-distribution and 2.0% out-of-distribution improvements persist with 4,000 training samples at a 50% noise ratio.The improvements are averaged across both noise scenarios.
- 4.2 Experimental Results: The slope threshold performs best at 0.05 and the early-learning duration at T = 5 in most settings, while removing either criterion substantially reduces performance.No clear trend appears for out-of-distribution active-noise sensitivity; both criteria are reported as complementary and essential.
5 Conclusion
The paper analyzes noisy-label mechanisms in RLVR, identifies early correctness coherence, and proposes OLR for online denoising. Experiments report improved reasoning performance across noise ratios and benchmark types.
- 5 Conclusion: The paper classifies RLVR noisy labels as active or inactive, identifies Early Correctness Coherence, and proposes OLR to denoise labels during training.OLR is presented as the first RLVR method that denoises labels during training.
- 5 Conclusion: Experiments show significant denoising effectiveness and improved reasoning performance across varied noise ratios.The conclusion describes potential relevance beyond RLVR, including VLM and agent domains.
A Theoretical Proof
The appendix formalizes early correctness coherence and finite-rollout dynamics in noisy RLVR using concentration bounds, martingale decompositions, and log-ratio analysis.
- Theoretical setup: Theorem A.1 analyzes a dataset partitioned into clean and noisy samples under rollout-based RLVR dynamics.It assumes rollout probabilities, K rollouts per step, and binary rewards.
- High-probability guarantees: With sufficiently large K, the high-probability bounds ensure finite-rollout fluctuations remain controlled over the early convergence stage.The stated concentration conditions depend on log(T/δ)/K and constants determined by reward bounds.
- Concentration bounds: Finite-rollout estimates concentrate around their expectations through Hoeffding bounds for empirical means and variances.The analysis uses bounded rewards and Lipschitz dependence of normalized advantage estimates.
- Log-ratio dynamics: The active-noise log-ratio is decomposed into deterministic drift and a bounded martingale fluctuation.The martingale difference has zero conditional expectation, while its magnitude is controlled by reward bounds and advantage concentration.
A.4 Cross-Sample Coupling
Cross-sample coupling allows clean-sample updates to increase the probability of correct answers on noisy samples during early training.
- Cross-sample coupling: The analysis models coupling through the interaction between clean and noisy-sample policy gradients.The coupling condition requires a nonnegative expected gradient inner product bounded below by γ > 0.
- Early dynamics: When the initial correct-answer probability exceeds the active noisy-label probability, the correct-versus-noisy log-ratio can increase monotonically early.The result holds under the theorem’s early-phase assumptions and positive drift condition.
A.6 Active Noise Collapse Condition
The active-noise analysis identifies a collapse regime in which negative deterministic drift overwhelms finite-rollout fluctuations and active-noise solutions dominate.
- Collapse implication: As the negative drift dominates, active-noise solutions can dominate the policy with high probability.This is the stated collapse implication of the log-ratio dynamics.
- Collapse dynamics: When active-noise drift is negative, the correct-versus-noisy log-ratio decreases monotonically during the analyzed regime.The drift depends on the noise ratio, clean and noisy advantage magnitudes, and cross-sample coupling.
- Finite-rollout effects: For sufficiently large K, martingale fluctuations remain smaller than the deterministic negative drift.Azuma-Hoeffding bounds control the stochastic correction over the finite early-training horizon.
A.7 KL-Regularized Dynamics
KL regularization modifies the active-noise drift by adding a reference-policy term, thereby shifting the critical noise ratio and the associated phase boundary.
- KL-regularized drift: KL regularization adds a reference-policy contribution to the deterministic log-ratio drift.The added term is proportional to βD_KL(π_θ∥π_ref).
- Modified phase boundary: Theorem A.6 states that KL regularization shifts the critical noise ratio.The corresponding early correctness coherence and collapse conditions are modified.
- Phase behavior: The phase behavior remains governed by the sign of the total deterministic drift, with finite-rollout fluctuations controlled when K is sufficiently large.The appendix distinguishes positive-drift early coherence from negative-drift collapse.
- Empirical illustration: Figure 5 compares clean and noisy prediction proportions under 50% inactive and active noise using majority votes across rollouts.The noisy-sample columns distinguish correct answers, matches to the noisy label, and answers matching neither label.
A.8 OLR Improves Label Noise Tolerance
OLR’s analysis shows that refining labels can reduce effective noise and raise the noise level RLVR can tolerate. The proof combines label-selection correctness, noise-ratio reduction, and threshold improvement.
- A.8 OLR Improves Label Noise Tolerance: The proof concludes that OLR selects correct labels with high probability, reduces effective noise, and increases the maximum tolerable noise level.These three properties are the stated components of the theorem’s conclusion.
- A.8 OLR Improves Label Noise Tolerance: OLR selects the majority answer when rollout evidence supports its correctness under early correctness dominance.The analysis defines majority voting over K rollouts and uses a positive probability gap between the ground-truth answer and the strongest incorrect answer.
- A.8 OLR Improves Label Noise Tolerance: OLR reduces the effective noise ratio from ρ to ρ(1 − ∆), where ∆ is the probability that a noisy sample satisfies the replacement criteria.The remaining incorrect probability is approximately 1 − ∆, ignoring an exponentially small ϵ.
- A.8 OLR Improves Label Noise Tolerance: OLR increases the tolerable noise threshold from ρc to ρOLR_c = ρc/(1 − ∆).The conclusion summarizes this as an increase in the maximum tolerable noise level after effective noise reduction.
B Experiment Details
The experiments use specified RLVR training infrastructure, hyperparameters, and baseline families, including unsupervised and noise-robust methods. These baselines cover label smoothing, sample selection, pseudo-labeling, consistency, entropy, and confidence regularization.
- Infrastructure and hyperparameters: Experiments use 8 × NVIDIA A100 (80G) GPUs, eight rollouts, maximum response length 4096, learning rate 1e-6, and KL loss weight 1e-3.The slope threshold is δslope = 0.05 and the early-learning period is T = 5.
- Infrastructure and hyperparameters: The implementation is built on veRL3 with vLLM4 as the rollout generator.The passage identifies the software stack used for the experiments.
- Baseline methods: Unsupervised baselines include Label Smoothing, Small-loss Selection, Random Selection, TTRL, Co-Reward, Self-Certainty, Token-Level Entropy, and Sentence-Level Entropy.These methods respectively use smoothed targets, loss-based or random sampling, majority-voted pseudo-labels, consistency, confidence, or entropy objectives.
- Baseline methods: Additional baselines include Confidence Penalty and other noise-robust learning or transfer-friendly regularization techniques.Confidence Penalty discourages overconfident predictions to reduce overfitting to noisy labels.
C More Experiments
Additional experiments test OLR across model variants, training dynamics, hyperparameters, ablations, computational overhead, and prior noisy-label approaches. The analyses examine both its empirical behavior and its fit to RLVR’s open-ended on-policy setting.
- More experiments: OLR improves Qwen3-8B-Base performance, with a 4.4% average gain on OOD benchmarks across both noise scenarios.OLR also improves Deepseek-R1-Distill-Llama-8B across both in-distribution and out-of-distribution benchmarks.
- More experiments: Early Correctness Coherence supports two assumptions: clean samples have higher correct-label probability, while noisy-sample correctness also improves during early training.The analysis is conducted under both inactive and active noise scenarios.
- More experiments: Under 50% inactive noise, majority accuracy rises from 50% to over 70% before OLR increases it beyond 80%, a 10% gain.OLR-selected samples exceed 90% accuracy, whereas unselected samples remain below 80%.
- More experiments: Both OLR criteria together achieve near 100% selection accuracy; removing either criterion reduces accuracy by 20%.The criteria are positive pass-rate slope and historical consistency.
- More experiments: OLR reduces inactive noise by 25% during refinement and selects over 40% of both clean and noisy sets.The selection pattern avoids bias toward simple samples.
- More experiments: OLR adds majority counting and convergence-statistics updates, while rollout generation remains the dominant O(NKL) training cost.The additional overhead is described as negligible compared with rollout generation.
- More experiments: Traditional noisy-label methods rarely accommodate RLVR’s open-ended generation space and on-policy dynamics.The paper contrasts RLVR with classification methods based on transition estimation, loss correction, small-loss selection, and semi-supervised learning.
- More experiments: OLR is implemented as an online procedure that computes majority answers, pass rates, slopes, and historical majorities during training.These operations are part of the stated OLR algorithm.