PaperlayerView exact revision on arXiv

Source-linked AI summary

What If Consensus Lies? Selective-Complementary Reinforcement Learning at Test Time

Dong Yan, Jian Liang, Yanbo Wang, Shuo Lu, Ran He, Tieniu Tan

arXiv:2603.19880v2cs.LGcs.AI

TL;DR

Existing TTRL methods rely on positive pseudo-labels, leaving weak-consensus settings vulnerable to label noise. SCRL combines strict positive-label criteria with entropy-gated negative supervision and dynamic reward shaping. The paper reports that SCRL outperforms baselines, particularly on challenging problems and under constrained rollout budgets.

  • Problem

    Existing TTRL methods rely exclusively on positive pseudo-labeling, while dispersed answers can make majority-vote supervision unreliable.

    Source: §1 · p. 1, §3.2 · p. 4

  • Method

    SCRL combines strict consensus-based positive labeling, entropy-gated negative labeling, and rewards shaped by consensus strength and uncertainty.

    Source: §1 · p. 2, §3.2 · p. 4, §3.4 · p. 5

  • Results

    SCRL significantly outperforms baseline methods, particularly on challenging problems and under constrained rollout budgets.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The combined labeling mechanisms aim to reinforce high-confidence trajectories and penalize incorrect answers without discarding rare but potentially correct trajectories.

    Source: §6 · p. 9

    SCRL lacks theoretical analysis or guarantees specifying when its mechanisms provably outperform standard majority voting.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Test-Time Reinforcement Learning (TTRL) enables Large Language Models (LLMs) to enhance reasoning capabilities on unlabeled test streams by deriving pseudo-rewards from majority voting consensus. However, existing TTRL methods rely exclusively on positive pseudo-labeling strategies. Such reliance becomes vulnerable under challenging scenarios where answer distributions are highly dispersed, resulting in weak consensus that inadvertently reinforces incorrect trajectories as supervision signals. In this paper, we propose SCRL (Selective-Complementary Reinforcement Learning), a robust test-time reinforcement learning framework that effectively mitigates label noise amplification. SCRL develops Selective Positive Pseudo-Labeling, which enforces strict consensus criteria to filter unreliable majorities. Complementarily, SCRL introduces Entropy-Gated Negative Pseudo-Labeling, the first negative supervision mechanism in TTRL, to reliably prune incorrect trajectories based on generation uncertainty. Extensive experiments on multiple reasoning benchmarks demonstrate that SCRL achieves substantial improvements over baselines, while maintaining robust generalization and training stability under constrained rollout budgets. Our code is available at https://github.com/Jasper-Yan/SCRL.

1 Introduction

TTRL uses majority-vote consensus to learn from unlabeled test streams, but weak consensus can turn incorrect trajectories into supervision. SCRL combines selective positive labeling, uncertainty-gated negative labeling, and dynamic reward shaping; experiments report stronger baseline performance, especially under challenging conditions and constrained rollout budgets.

1 Introduction

  • TTRL uses diverse rollouts and majority-vote consensus to derive pseudo-rewards for policy updates on unlabeled test streams.It is presented as a response to the difficulty of acquiring high-quality manual supervision as task complexity and diversity grow.Source: §1 · p. 1
  • When answers are dispersed, majority-vote methods can reinforce incorrect trajectories, a vulnerability that grows under constrained rollout budgets.Weak consensus can lead policy optimization toward spurious solutions.Source: §1 · pp. 1–2
  • Selective Positive Pseudo-Labeling applies strict consensus and margin criteria, granting positive supervision only to sharply concentrated answers clearly separated from alternatives.This filters unreliable majorities when answer distributions are dispersed.Source: §1 · p. 2
  • SCRL adds negative labels for rare, uncertain answers and shapes rewards by consensus strength, pruning implausible solutions without discarding potentially correct rare answers.The paper presents negative supervision as the first such mechanism in TTRL.Source: §1 · p. 2
  • SCRL consistently outperforms baselines on multiple reasoning benchmarks, with especially pronounced gains on challenging problems and under constrained rollout budgets.Ablations and label-quality analyses also validate the necessity of each component.Source: §1 · p. 2

2 Related Work

The related work situates SCRL among reinforcement-learning approaches to reasoning and methods that adapt models on unlabeled test streams. It distinguishes SCRL through the combination of selective positive labeling and negative labeling to prune the search space without premature convergence.

2.1 RL for Reasoning

  • RLHF uses annotated preference data, RLVR uses verifiable signals, and RLIF derives intrinsic rewards from confidence, entropy, or self-consistency.SCRL is placed in the RLIF paradigm and uses both positive and negative signals from model output distributions.Source: §2.1 · pp. 2–3
  • SCRL has three components: selective positive pseudo-labeling, entropy-gated negative pseudo-labeling, and dynamic reward shaping.They respectively enforce consensus thresholds, identify rare high-uncertainty answers, and scale rewards with consensus while penalizing uncertainty.Source: Figure 2 · p. 3

2.2 Unsupervised Reasoning at Test Time

  • Prior TTRL work includes soft-weighted pseudo-labels, self-play verification, entropy-based update restrictions, entropy-fork rollouts, and multimodal extensions.These approaches target reward robustness, training stability, efficiency, or complex vision-language reasoning.Source: §2.2 · p. 3
  • SCRL addresses noise amplification from voting-based positive labels by applying strict criteria to positive labels and using negative labels to prune without premature convergence.This contrasts with approaches relying solely on voting-based positive-label assignment.Source: §2.2 · p. 3

3 Method

SCRL combines selective positive pseudo-labeling, entropy-gated negative pseudo-labeling, and dynamic reward shaping for test-time reinforcement learning. Its rules limit positive supervision to credible consensus and use uncertainty to identify trajectories for negative supervision.

3 Method

  • SCRL combines selective positive pseudo-labeling, entropy-gated negative pseudo-labeling, and dynamic reward shaping to mitigate label-noise amplification in unsupervised test-time reinforcement learning.The method uses Grouped Relative Policy Optimization (GRPO) as its main RL algorithm.Source: §3 · p. 3, §3.1 · p. 3

3.1 Preliminaries

  • GRPO samples response groups, assigns each response a reward, and constructs a group-normalized advantage shared across tokens for policy optimization.The importance-sampling ratio compares current and old policy probabilities for a token given the query and preceding response tokens.Source: §3.1 · pp. 3–4

3.2 Selective Positive Pseudo-Labeling

  • Selective positive pseudo-labeling labels the most frequent answer only when its proportion and margin over the runner-up meet strict criteria.Otherwise, SCRL abstains from positive supervision and shifts learning focus to negative pseudo-labeling.Source: §3.2 · p. 4

3.3 Entropy-Gated Negative Pseudo-Labeling

  • Entropy-gated negative pseudo-labeling targets answers with low frequency and above-average generation uncertainty, while preserving potentially correct rare answers with lower uncertainty.The method uses policy uncertainty to distinguish low-frequency valid answers from incorrect responses.Source: §3.3 · pp. 4–5

3.4 Dynamic Reward Shaping

  • Dynamic reward shaping scales reinforcement by consensus strength and adds uncertainty penalties to guide policy updates.The reward is described as reinforcing answers under credible consensus and eliminating trajectories that are both rare and uncertain.Source: §3.4 · p. 5

4 Experiments

Experiments evaluate SCRL across reasoning benchmarks and model families. SCRL consistently outperforms baselines, with especially strong results on challenging problems and under constrained rollout budgets.

4.1 Experimental Setup

  • Evaluation covers six reasoning datasets and five open-weight models, using pass@1 with 16 responses per question and 32 or 64 candidate rollouts.The setup also specifies training and labeling hyperparameters.Source: §4.1 · p. 5

4.2 Main Results

  • Across benchmarks, SCRL consistently beats baselines, with Qwen2.5-3B averaging 1.6% improvement under constrained budgets and AIME25 reaching 8.4% versus TTRL’s 2.6%.The main results are reported in Tables 1 and 2.Source: §4.2 · pp. 5–6, Table 1 · p. 6
  • With 64 candidate responses and 32 training samples, SCRL maintains a 0.7% average improvement and 3.8% improvement on AIME25, while also consistently improving GPQA.Source: §4.2 · p. 6
  • Qwen2.5-Math-7B gains average 7.7% under constrained budgets; on Minerva, SCRL reaches 41.6% while TTRL peaks at 14.5% before degradation.With an increased rollout budget, SCRL sustains a 7.9% average improvement.Source: §4.2 · pp. 6–7, Table 1 · p. 6
  • Across two Llama instruction-tuned models, SCRL averages 23.2% on Llama-3.2-1B-Instruct and 29.0% on Llama-3.1-8B-Instruct, outperforming the stated baselines.The results support generalization across the evaluated architectures and model scales.Source: Table 2 · p. 6, §4.2 · p. 7

5 Analysis

The analyses examine SCRL’s components, supervision signals, generalization, and operating conditions. They also identify a Minerva setting where selective labeling can withhold positive supervision.

5.1 Ablation Study

  • Ablations show selective positives matter on AIME25, negatives and entropy gating matter, and hard rewards degrade performance; stricter thresholds slightly hurt AMC.Removing consensus thresholds on AMC yields a slight performance gain.Source: §5.1 · p. 7, Table 3 · p. 7

5.2 Statistics of Label Estimation

  • On AMC, selective positives maintain higher accuracy than TTRL early; entropy-gated negative labels stay near-perfect as positive-label ratios rise and negative labels fall.The label-ratio changes accompany stronger consensus during training.Source: §5.2 · p. 8

5.3 Generalization Capabilities

  • On three out-of-distribution benchmarks, SCRL improves pass@1 over TTRL and maintains more stable pass@16, consistent with preserving exploration capacity.Both methods were trained on AIME25 using Qwen2.5-3B.Source: §5.3 · p. 8, Table 4 · p. 8

5.4 Analysis of Labeling Thresholds

  • Raising τpos from 0.25 to 0.375 consistently improves AIME25; raising τneg from 0.0625 to 0.125 generally improves AMC and MATH-500.The analysis uses Qwen2.5-Math-7B with 64 candidate responses and 32 training samples.Source: §5.4 · p. 8, Table 5 · p. 8

5.5 Entropy Analysis

  • Across mathematical benchmarks, correct and incorrect Qwen2.5-3B trajectories show a consistent entropy gap, supporting uncertainty-based negatives while preserving potentially correct rare answers.The reported entropy values are from the initial rollout phase.Source: §5.5 · p. 9, Table 6 · p. 9

5.6 Performance on Large Reasoning Models

  • On AIME25 with Qwen3-4B thinking mode, SCRL beats TTRL by +2.7% and +0.9% at maximum generation lengths of 10k and 15k tokens, respectively.The comparison covers long chain-of-thought reasoning with extended generation lengths.Source: §5.6 · p. 9, Table 7 · p. 9

5.7 Failure Analysis

  • On Minerva with Qwen2.5-3B, domain-knowledge gaps and fragmented equivalent answers make selective labeling over-conservative, often withholding positive supervision.The reported underperformance contrasts with SCRL’s results on other benchmarks.Source: §5.7 · p. 9

6 Conclusion

SCRL combines selective positive labeling with entropy-gated negative labeling to mitigate label-noise amplification in unsupervised test-time reinforcement learning. The paper reports improved baseline performance, robust generalization, and training stability.

  • SCRL combines strict-consensus positive labeling with entropy-gated negative labeling to reinforce high-confidence trajectories and prune low-frequency, high-uncertainty answers without discarding rare potentially correct ones.The paper describes entropy-gated negative labeling as the first negative-supervision mechanism in test-time reinforcement learning.Source: §6 · p. 9
  • SCRL is designed to mitigate label-noise amplification in unsupervised test-time reinforcement learning.Source: §6 · p. 9
  • Empirical results show SCRL significantly outperforms baselines while maintaining robust generalization and training stability.Source: §6 · p. 9

Limitations

The authors identify missing theoretical guarantees and reproducibility constraints on comparisons with concurrent methods. The appendix also specifies chat-template prompts for the two Qwen model variants.

Limitations

  • The authors provide empirical validation but no theoretical guarantees for when either SCRL mechanism outperforms standard majority voting.They identify the lack of theoretical analysis for selective positive and entropy-gated negative pseudo-labeling as future work.Source: Limitations · p. 10
  • Comparisons with concurrent TTRL methods are constrained because most have not released code and critical hyperparameters cannot be precisely aligned.Source: Limitations · p. 10

A.1 Prompt Design

  • The appendix uses architecture-specific chat templates, including step-by-step reasoning prompts and boxed final answers for the Qwen models.Qwen2.5-3B and Qwen2.5-Math-7B use separately specified templates.Source: §A.1 · p. 12

A.2 Benchmarks

The evaluation suite covers six reasoning datasets spanning difficulty levels and domains, including mathematics and graduate-level science. Figures 4 and 5 describe training dynamics across three mathematical benchmarks for two Qwen models.

  • Table 8 is identified as the training-hyperparameter configuration for Qwen2.5-3B and Qwen2.5-Math-7B using the verl framework.Source: Table 8 · p. 13
  • Figures 4 and 5 cover SCRL and TTRL training dynamics across three mathematical benchmarks for Qwen2.5-3B and Qwen2.5-Math-7B, respectively.Source: Figure 4 · p. 13, Figure 5 · p. 13
  • The suite includes AIME24, AIME25, AMC, MATH-500, and Minerva mathematics benchmarks, alongside GPQA’s graduate-level biology, physics, and chemistry questions.MATH-500 is described as a 500-problem subset spanning algebra, geometry, number theory, and combinatorics; GPQA contains 448 questions.Source: §A.2 · p. 13

A.3 Training Episode

For one MATH-500 configuration, the authors shorten the TTRL training episode after observing collapse at a longer setting. Table 8 lists hyperparameter configurations for both Qwen models.

A.3 Training Episode

  • For Qwen2.5-3B on MATH-500 with 32 candidate responses and 16 training samples, TTRL collapses at episode 10, so the episode is set to 4 for fair comparison.Other settings remain consistent with Section 4.1.Source: §A.3 · pp. 13–14

A.4 Training Hyperparameter

  • The paper states that its training hyperparameters are shown in Table 8, identifying that table as the location where its training hyperparameter information is presented.Source: §A.4 · p. 14

B Additional Results

Figures 4 and 5 show pass@1 accuracy (%) throughout training.

C Analysis of Consensus Margin and Entropy Penalty

The analysis examines how consensus-margin and entropy-penalty settings affect performance. The main configuration balances stricter filtering on hard tasks with supervision frequency on easier ones.

Entropy Penalty

  • λH = 0.1 achieves optimal performance on both benchmarks; a moderate entropy penalty outperforms λH = 0, while larger coefficients degrade performance.The penalty is described as balancing confident reasoning against sufficient exploration.Source: C Analysis of Consensus Margin and… · p. 14

Consensus Margin

  • On AIME25, τmarg = 0.25 maintains 22.6% performance, while a lower τmarg = 0.0625 performs better on AMC.Increasing τmarg from 0 to 0.125 substantially improves AIME25; overly strict margins may discard valid positive signals on easier problems.Source: C Analysis of Consensus Margin and… · p. 14

Analysis Setup

Related papers

Loading 2603.19880v2…