PaperlayerView exact revision on arXiv

Source-linked AI summary

How Far Can Unsupervised RLVR Scale LLM Training?

Bingxiang He, Yuxin Zuo, Zeyuan Liu, Shangziqi Zhao, Zixuan Fu, Junlin Yang, Cheng Qian, Kaiyan Zhang, Yuchen Fan, Ganqu Cui, Xiusi Chen, Youbang Sun, Xingtai Lv, Xuekai Zhu, Li Sheng, Ran Li, Huan-ang Gao, Yuchen Zhang, Bowen Zhou, Zhiyuan Liu, Ning Ding

arXiv:2603.08660v1cs.LGcs.CL

TL;DR

URLVR addresses the difficulty of scaling RLVR when ground-truth labels become costly or unavailable. The paper classifies reward sources, develops a unified theory and evaluates intrinsic and external methods. It finds intrinsic rewards sharpen existing model preferences and eventually collapse beyond confidence-correctness alignment, while external verification shows preliminary evidence of more scalable improvement.

  • Problem

    URLVR seeks to scale reinforcement learning beyond costly ground-truth supervision, but the potential and limitations of intrinsic reward methods remain unclear.

  • Method

    The paper combines a taxonomy of intrinsic and external rewards with unified theoretical analysis and experiments, including self-verification based on generation-verification asymmetry.

  • Results

    Intrinsic rewards sharpen initial model distributions and collapse when confidence misaligns with correctness, whereas self-verification shows sustained learning signals and stronger performance than trajectory-level entropy.

  • Takeaways & Limitations

    Intrinsic URLVR has bounded scaling but remains useful for test-time training, while external rewards grounded in independent verification offer a promising path beyond that ceiling.

  • Takeaways & Limitations

    Intrinsic methods cannot consistently push the model beyond what it already knows because their rewards derive entirely from the model’s internal state.

Abstract

from arXiv · show

Unsupervised reinforcement learning with verifiable rewards (URLVR) offers a pathway to scale LLM training beyond the supervision bottleneck by deriving rewards without ground truth labels. Recent works leverage model intrinsic signals, showing promising early gains, yet their potential and limitations remain unclear. In this work, we revisit URLVR and provide a comprehensive analysis spanning taxonomy, theory and extensive experiments. We first classify URLVR methods into intrinsic versus external based on reward sources, then establish a unified theoretical framework revealing that all intrinsic methods converge toward sharpening the model's initial distribution This sharpening mechanism succeeds when initial confidence aligns with correctness but fails catastrophically when misaligned. Through systematic experiments, we show intrinsic rewards consistently follow a rise-then-fall pattern across methods, with collapse timing determined by model prior rather than engineering choices. Despite these scaling limits, we find intrinsic rewards remain valuable in test-time training on small datasets, and propose Model Collapse Step to measure model prior, serving as a practical indicator for RL trainability. Finally, we explore external reward methods that ground verification in computational asymmetries, showing preliminary evidence they may escape the confidence-correctness ceiling. Our findings chart boundaries for intrinsic URLVR while motivating paths toward scalable alternatives.

1. Introduction

URLVR seeks to relieve the human-label bottleneck in RLVR by deriving rewards without ground-truth labels. This study analyzes its reward sources, intrinsic mechanism, empirical collapse behavior, and external alternatives.

  • URLVR derives reinforcement-learning rewards without ground-truth labels, extending post-training toward unlabeled-data scaling.
  • The study classifies URLVR rewards as intrinsic or external and combines taxonomy, theory, and extensive experiments.
  • Intrinsic methods sharpen the model’s initial distribution, amplifying existing preferences rather than discovering new knowledge.
  • Intrinsic URLVR exhibits a rise-then-fall pattern across methods, with collapse differing in timing rather than occurrence.
  • Intrinsic rewards remain useful for small, domain-specific test-time training, while external rewards based on generation-verification asymmetries show sustained improvement without intrinsic collapse patterns.
  • Intrinsic rewards can enable efficient test-time gains but risk reward hacking when model confidence misaligns with correctness, motivating scalable external rewards.

2. Taxonomy of Unsupervised RLVR

URLVR replaces difficult ground-truth supervision with proxy rewards and divides methods by whether rewards come from model-internal signals or external mechanisms. Intrinsic rewards formalize confidence or consensus, whereas external rewards exploit scalable data and computational verification.

  • URLVR targets verifiable tasks where ground-truth labels are difficult to obtain, using proxy rewards derived without human effort.
  • The taxonomy separates intrinsic rewards, generated from model signals, from external rewards, generated through data structure or computational asymmetries.
  • Intrinsic Reward Methods: Certainty-based rewards encourage high-confidence predictions using internal output distributions, including entropy, probability, and KL-based estimators.
  • Intrinsic Reward Methods: Ensemble-based rewards use agreement across multiple rollouts, with majority voting and related methods treating consistency as a proxy for correctness.
  • External Reward Methods: Intrinsic rewards are limited by the model’s existing knowledge, whereas external rewards can scale with unlabeled data or independent computation.
  • External Reward Methods: External methods are positioned as the more promising long-run direction, while diverse verifiable environments and new scientific-domain asymmetries remain open challenges.

3. The Sharpening Mechanism of Intrinsic Rewards

The paper’s theory shows that intrinsic rewards progressively sharpen the model’s initial distribution. This reinforces correct preferences when confidence tracks correctness but amplifies errors when that alignment is weak.

  • Intrinsic URLVR methods with diverse designs share a mechanism that sharpens the model’s initial distribution by reinforcing existing preferences.
  • Dynamics of One-Step Update: The representative majority-voting update samples rollouts, selects the most frequent answer, and rewards trajectories producing that answer.
  • Dynamics of One-Step Update: The KL-regularized objective uses a reference policy and regularization strength to define the policy update toward reward-favored outputs.
  • Convergence Towards Sharpening Initial Distribution: Each update increases probability mass on majority trajectories and proportionally diminishes other trajectories, creating a rich-get-richer dynamic.
  • Convergence Towards Sharpening Initial Distribution: Under majority stability and effective learning, the majority probability converges geometrically toward one, yielding a deterministic policy.
  • Sharpening reinforces good solutions when confidence aligns with correctness but amplifies errors and causes collapse when alignment is poor.

4. When Does Intrinsic URLVR Work?

Intrinsic URLVR works when the model’s initial confidence aligns with correctness, but it eventually collapses as training amplifies confidence rather than reliably correcting errors. Across methods, this produces a consistent rise-then-fall pattern whose failure modes vary by reward design and problem.

  • Early Success, Later Collapse: Intrinsic URLVR consistently follows a rise-then-fall pattern: early gains depend on confidence-correctness alignment, while collapse occurs when that alignment breaks down.Hyperparameter changes affect collapse timing, but not whether collapse occurs.
  • Early Success, Later Collapse: Proxy rewards can keep rising while reward accuracy and validation performance decline, revealing reward hacking through divergence between confidence optimization and correctness.Majority-voting training initially matches or exceeds ground-truth training before this divergence appears.
  • Distinct Failure Patterns: Five intrinsic methods exhibit distinct failures: Self-Certainty and Majority Voting degrade gradually, Probability favors brevity, and entropy methods reduce entropy through repetition rather than correctness.These differences reflect practical consequences of reward designs that all sharpen model distributions.
  • In-Distribution Per-Problem Sharpening: When the highest-reward rollout is correct, sharpening amplifies success; when it is wrong or inconsistent, training amplifies failure or gradually changes correct answers to wrong ones.Representative cases include amplifying success, amplifying failure, wrong-to-correct transitions, and correct-to-wrong degradation.
  • In-Distribution Per-Problem Sharpening: Training amplifies initial preferences within problems rather than correcting errors, with 22 of 25 problems sharpening their starting preference and only 3 flipping correctness.The three flips represented 12% of the problems, and sharpening occurred even when correctness changed.
  • Out-Of-Distribution Cross-Problem Generalization: Even when all training problems begin wrong, intrinsic sharpening can improve unseen problems, with test label accuracy increasing steadily from 0 to 1 when confidence aligns with correctness out of distribution.This shows that amplification on training problems can still generalize to OOD problems.

5. How Can Sharpening from Intrinsic URLVR Be Applied Safely?

Intrinsic URLVR is safest on small, domain-specific datasets, where it can produce localized adaptation without catastrophic collapse. Test-time training therefore offers a practical setting, while larger datasets trigger distribution shift and reward hacking.

  • Small datasets: ≤128 samples maintain stable performance without collapse, whereas ≥512 samples consistently exhibit reward hacking.DAPO-32 never collapses across three seeds, while DAPO-512 always does.
  • Small datasets: DAPO-32 reaches Majority Voting Reward →1 while preserving high Ground Truth Reward.The result indicates convergence on the sampled problems without collapse.
  • Small datasets: DAPO-32 reaches only 0.057 KL after 600 steps, while DAPO-512 reaches 2× higher distributional shift.Small subsets induce localized updates, whereas larger datasets cause global policy shift and collapse.
  • Test-time training: Test-time training on AMC23 avoids collapse while improving performance on both AMC23 and AIME24.Training on the approximately 17,000-problem DAPO dataset instead follows the familiar rise-then-fall pattern.
  • Test-time training: Even when almost all 32 filtered samples have incorrect initial majority votes, small-scale training produces effective learning without catastrophic collapse.The resulting gains appear on both AIME24 and AMC23.

6. How Can We Measure Model Prior?

Model Collapse Step measures how long intrinsic URLVR sustains reward accuracy before collapse and serves as an efficient indicator of model prior and RL trainability. It remains predictive across models and aggressive hyperparameter settings while reducing assessment cost.

  • Indicator: Model Collapse Step is defined as the training step where Reward Accuracy drops below 1%.Models with stronger priors remain stable longer before collapsing.
  • Model differences: Higher initial Actor Entropy does not predict better trainability: base models collapse faster than SFT variants despite starting more entropically.This suggests entropy is a consequence of sharpening rather than its determinant.
  • Predictive validity: Model Collapse Step correlates strongly with GT Gain and matches or surpasses pass@k’s predictive power.It also cannot be gamed by random guessing on multiple-choice questions.
  • Efficiency: Aggressive settings accelerate collapse in absolute steps but preserve relative model rankings and predictive power.The faster settings shorten assessment for models with strong priors by at least 50 steps.
  • Efficiency: Model Collapse Step uses 5.6× fewer tokens than full RL training while preserving the relative ranking of seven models.It requires no ground truth labels, making it applicable when verification is unavailable.

7. Discussion

Intrinsic rewards face a confidence-correctness ceiling, whereas external rewards grounded in verification procedures, unlabeled data, or generation-verification asymmetry offer more scalable alternatives. Self-verification provides preliminary evidence for this direction, especially with instruction-aligned models.

  • External rewards: External rewards scale with data and computation rather than saturating with model capacity.Their verification procedures can remain reliable as model outputs become more sophisticated.
  • Intrinsic limits: Intrinsic URLVR cannot consistently push models beyond what they already know because its rewards derive entirely from internal state.Weak confidence-correctness alignment causes existing biases to be amplified rather than corrected.
  • Self-verification: Self-verification exploits the asymmetry that generating correct solutions is hard while checking them is easy.This produces rewards without ground truth labels.
  • Self-verification: Self-Verification yields higher validation accuracy than Trajectory-Level Entropy for both evaluated models.Reward Accuracy initially drops around step 200, then recovers above 0.5 while Ground Truth Reward continues rising.
  • Instruction alignment: Instruction alignment improves starting accuracy, prompt robustness, and self-verification stability.The instruction model starts above 60% accuracy, improves beyond 80%, and succeeds with both prompts, while the base model works only with P2.

8. Conclusion

The paper concludes that intrinsic URLVR sharpens existing preferences: it enables gains when confidence matches correctness but eventually collapses beyond the model’s knowledge boundaries. Model Collapse Step offers rapid trainability assessment, while external rewards motivate scalable alternatives.

  • Conclusion: Intrinsic reward methods share a mechanism that sharpens output distributions by amplifying the model’s existing preferences.The paper frames this as the central commonality across diverse intrinsic designs.
  • Conclusion: Intrinsic URLVR enables efficient gains when initial confidence aligns with correctness but eventually collapses beyond the model’s knowledge boundaries.The conclusion characterizes collapse as a fundamental limitation rather than an engineering problem.
  • Conclusion: Model Collapse Step enables rapid assessment of RL trainability from early training dynamics.It is presented as a practical indicator of model prior.
  • Conclusion: External reward methods ground verification in computational asymmetries and unlabeled data rather than model confidence.The paper presents these methods as motivated paths toward scalable alternatives.

A.1.1. Empirical Validation 1: Validation of Ordering and Majority Stability

Empirical validation shows majority-based intrinsic training preserves its initial majority answer while monotonically concentrating probability on that answer. Under fixed rewards and sufficient updates, the same process can converge completely while validation performance collapses.

  • With N=1024 rollouts, all four tested problems showed strict monotonic increases in p_maj during early training.
  • The majority answer remained stable across all iterations for the four monitored problems.
  • After 50 steps, p_maj reached 98.54%-99.80%, demonstrating near-complete concentration and convergence toward deterministic policies.
  • After 1024 fixed-reward mini-updates, Majority Voting Reward reached 1.0, while AIME24, AIME25, and AMC23 validation performance dropped to zero.

A.2. Proof of Theorem 1

The proof establishes geometric convergence toward a deterministic policy under stable-majority and non-trivial-progress assumptions. A unified cross-entropy framework shows that diverse intrinsic rewards sharpen token- or answer-level distributions through different anchors and optimization directions.

  • Theorem 1 proof: Under majority stability and non-trivial progress, the probability mass assigned to the initial majority answer converges to one.
  • Theorem 1 proof: The error sequence is strictly decreasing and bounded below by zero, then converges to zero under a positive lower bound on update efficiency.
  • Theorem 1 proof: The convergence is geometric, with ideal rate ρ=e^-1/β when every update reaches the optimum.
  • Unified reward framework: Intrinsic rewards share a cross-entropy formulation over anchor and model distributions, with token-level and answer-level aggregation variants.
  • Unified reward framework: Uniform anchors encourage departure from randomness, while sharp anchors reinforce confident predictions; both mechanisms progressively sharpen distributions.

A.4. Generalized Sharpening Analysis via Unified Reward Framework

The generalized analysis identifies reward-confidence monotonicity as a shared sharpening mechanism for most intrinsic rewards. It creates positive feedback that concentrates probability on already preferred trajectories, while Self-Certainty requires separate treatment because its reward is not directly confidence-monotonic.

  • Reward-confidence monotonicity: For σ=−1 methods, dominant trajectories receive strictly higher rewards than non-dominant competitors, producing a self-reinforcing loop under KL-regularized updates.
  • Reward-confidence monotonicity: The positive reward gap persists because updates increase dominant-path probability, maintaining or widening the gap and driving distribution sharpening.
  • Implications: The framework predicts deterministic outputs whether initial confidence aligns with correctness or instead reinforces incorrect preferences.
  • Self-Certainty: Self-Certainty does not satisfy direct reward-confidence monotonicity because both high- and low-probability outputs can have high divergence from uniformity.
  • Reward-specific instantiations: Trajectory-Level Entropy, Probability, and Self-Certainty rewards increase the probability of sequences that are already high-confidence or high-likelihood under the prior policy.

B.3.2. Certainty-Based Methods

Certainty-based methods show method-specific stability differences but share a broad vulnerability to self-reinforcing collapse. Higher temperature and larger on-policy mini-batches can delay degradation for several methods, whereas larger rollout counts often accelerate convergence and collapse.

  • Temperature: At T=1.2, Token-Level Entropy, Trajectory-Level Entropy, and Probability significantly delayed collapse while maintaining exploration and gradual validation gains.
  • Temperature: Self-Certainty performed more stably at T=1.0, whereas T=1.2 produced excessive exploration without convergence and lower validation scores.
  • Mini-batch size: Mini-batch size 64, matching the global batch size, maintained the strongest policy-reward coupling and prevented premature collapse for three certainty-based methods.
  • KL regularization: KL regularization had minimal impact on training dynamics and downstream performance and generally failed to prevent eventual collapse.
  • Number of rollouts: Rollout counts N≥16 accelerated convergence and premature collapse for most certainty-based methods by amplifying confidence in potentially incorrect assessments.
  • Number of rollouts: Self-Certainty remained stable across rollout configurations, reflecting its distinct KL-divergence-based reward computation.

C.2.1. Horizontal Analysis: Training Stage Impact

Training stage strongly affects stability, with Qwen variants showing distinct robustness patterns and Llama variants eventually experiencing reward hacking at different times.

  • Qwen family: Qwen math-specialized and SFT models maintain Majority Voting Reward within 0.3-0.6, while base and instruct variants saturate at 1.0 by step 180.Math specialization and strong supervised fine-tuning provide more robust optimization foundations than raw base or non-math-aligned instruct models.
  • Llama family: All Llama variants eventually encounter reward hacking, with base models collapsing earliest at step 40, followed by math-specialized, SFT, and instruct models.Collapse timing differs across training stages rather than disappearing entirely.
  • Cross-family comparison: Qwen’s training dynamics provide greater fundamental stability than the compared Llama configurations.The family-level comparison identifies a broader architectural stability difference.

C.2.2. Vertical Analysis: Scale and Generation Effects

Scaling effects are not uniformly beneficial: smaller models and newer generations show greater stability, while larger datasets intensify reward hacking and intrinsic training can exceed majority-vote baselines.

  • Scale effects: Smaller models remain stable longer: Q3-1.7B outlasts Q3-4B, while Octo-3B outlasts Octo-8B by about 40 steps.The results suggest larger capacity increases sensitivity to noisy pseudo-rewards and accelerates degenerate convergence.
  • Generation effects: Qwen3 models show more controlled training dynamics and superior stability compared with Qwen2.5 counterparts.Q3-1.7B-Base demonstrates a more controlled Majority Voting Reward progression, likely reflecting better-calibrated uncertainty estimates and representations.
  • Training data effects: DeepScaleR-40k and ORZ-56k exhibit clear reward hacking, whereas smaller datasets remain steady or continue rising.This pattern indicates early intrinsic-reward improvements can give way to reward hacking when training extends over larger corpora.
  • Consistency versus capability: Intrinsic reward methods primarily reduce uncertainty, raising whether they improve capability or only output self-consistency.TTRL uses majority voting across m outputs as its supervisory signal, linking the concern to consistency alignment.
  • Consistency versus capability: TTRL-trained models significantly surpass base models’ majority-vote performance in accuracy.The comparison uses base maj@1024 against TTRL-trained pass@1 (avg@32) on AIME 2024 for Qwen2.5-Math-1.5B and 7B.
Loading 2603.08660v1…