PaperlayerView exact revision on arXiv

Source-linked AI summary

How Far Can Unsupervised RLVR Scale LLM Training?

Bingxiang He, Yuxin Zuo, Zeyuan Liu, Shangziqi Zhao, Zixuan Fu, Junlin Yang, Cheng Qian, Kaiyan Zhang, Yuchen Fan, Ganqu Cui, Xiusi Chen, Youbang Sun, Xingtai Lv, Xuekai Zhu, Li Sheng, Ran Li, Huan-ang Gao, Yuchen Zhang, Bowen Zhou, Zhiyuan Liu, Ning Ding

arXiv:2603.08660v1cs.LGcs.CL

TL;DR

Unsupervised RLVR seeks to scale LLM training when ground-truth labels are difficult or costly to obtain. This paper unifies intrinsic methods theoretically and experimentally, finding that they sharpen existing preferences and eventually collapse, while external verification offers a promising alternative.

  • Problem

    Unsupervised RLVR addresses training verifiable-task models when ground-truth labels are difficult or costly to obtain.

    Source: §1 · p. 4, §2 · p. 5

  • Method

    The paper classifies URLVR by reward source, develops a unified framework for intrinsic rewards, and examines external self-verification experimentally.

    Source: §2 · p. 5, §7 · p. 21

  • Results

    Intrinsic methods sharpen existing preferences and inevitably collapse beyond the model’s knowledge, whereas self-verification provides stronger, more stable rewards in experiments.

    Source: §7 · pp. 22–23, §8 · p. 23

  • Takeaways & Limitations

    Intrinsic URLVR is useful for exploiting existing knowledge but cannot scale indefinitely, while external rewards grounded in generation-verification asymmetry offer a more promising scaling direction.

    Source: §7 · pp. 21, 23, §8 · p. 23

    Intrinsic rewards cannot consistently push models beyond existing knowledge because they derive entirely from the model’s internal state and depend on confidence-correctness alignment.

    Source: §7 · p. 21

Abstract

from arXiv · show

Unsupervised reinforcement learning with verifiable rewards (URLVR) offers a pathway to scale LLM training beyond the supervision bottleneck by deriving rewards without ground truth labels. Recent works leverage model intrinsic signals, showing promising early gains, yet their potential and limitations remain unclear. In this work, we revisit URLVR and provide a comprehensive analysis spanning taxonomy, theory and extensive experiments. We first classify URLVR methods into intrinsic versus external based on reward sources, then establish a unified theoretical framework revealing that all intrinsic methods converge toward sharpening the model's initial distribution This sharpening mechanism succeeds when initial confidence aligns with correctness but fails catastrophically when misaligned. Through systematic experiments, we show intrinsic rewards consistently follow a rise-then-fall pattern across methods, with collapse timing determined by model prior rather than engineering choices. Despite these scaling limits, we find intrinsic rewards remain valuable in test-time training on small datasets, and propose Model Collapse Step to measure model prior, serving as a practical indicator for RL trainability. Finally, we explore external reward methods that ground verification in computational asymmetries, showing preliminary evidence they may escape the confidence-correctness ceiling. Our findings chart boundaries for intrinsic URLVR while motivating paths toward scalable alternatives.

1. Introduction

This study analyzes URLVR’s taxonomy, theory, and experiments, finding that intrinsic rewards sharpen initial model preferences and exhibit rise-then-fall behavior whose collapse depends on prior alignment. It identifies test-time training benefits and external reward methods as paths beyond intrinsic scalability limits.

Motivation

  • URLVR aims to scale post-training beyond human-provided labels, whose costs and feasibility become limiting as models approach or exceed specialized human expertise.This motivation parallels pretraining’s use of large-scale computation over unlabeled data.Source: §1 · p. 4

Theory

  • Intrinsic URLVR methods sharpen the model’s initial distribution, amplifying existing preferences rather than discovering new knowledge.The mechanism succeeds when initial confidence aligns with correctness but is limited when confidence and correctness diverge.Source: §1 · p. 4

Experiments

  • Across intrinsic methods, training consistently rises and then falls; methods differ in collapse timing, not whether collapse occurs.The timing is determined by the model prior rather than engineering choices.Source: §1 · p. 4

Test-Time Training

  • Intrinsic rewards remain useful for test-time training on small, domain-specific datasets, avoiding collapse even when initial preferences are wrong.These bounded gains reflect the model prior and can enable efficient, safe improvement while confidence misalignment risks reward hacking.Source: §1 · pp. 4–5

Model Prior Indicator

  • Model Collapse Step measures model prior and predicts RL trainability without requiring expensive training runs.It is derived from the rise-then-fall behavior of intrinsic URLVR.Source: §1 · p. 4

External Rewards

  • External URLVR methods use generation-verification asymmetries rather than internal model states and show sustained improvement without intrinsic collapse patterns.These methods address the fundamental scalability limits of intrinsic rewards and motivate more robust improvement pathways.Source: §1 · pp. 4–5

2. Taxonomy of Unsupervised RLVR

URLVR studies verifiable-task reinforcement learning using proxy rewards derived without human-provided ground-truth labels. It divides methods into intrinsic rewards from model behavior and external rewards from data structure or computational verification.

2. Taxonomy of Unsupervised RLVR

  • URLVR addresses the scalability bottleneck created by increasingly difficult ground-truth datasets through proxy rewards that avoid human efforts.The framework targets verifiable tasks where labels are difficult to obtain, extending scalable-reward efforts [Jayalath et al., 2025, Zhao et al., 2025b, Zuo et al., 2025].Source: §2 · p. 5
  • Based on reward source, URLVR methods divide into intrinsic rewards generated by the model and external rewards produced by data or computational mechanisms.The scope excludes general-domain self-rewarding methods while including approaches such as TTRL [Zuo et al., 2025] that match responses to verifiable answers.Source: §2 · p. 5

2.1. Intrinsic Reward Methods

  • Intrinsic rewards comprise certainty-based signals that favor confident predictions and ensemble-based signals that use agreement across rollouts as a correctness proxy.Certainty estimators include entropy, KL divergence, sequence probability, and top-token disparity, while ensemble methods include voting, semantic clustering, reasoning consistency, and multi-model collectives [Agarwal et al., 2025, Prabhudesai et al., 2025, Zuo et al., 2025].Source: §2.1 · pp. 5–7
  • Ensemble rewards incur rollout or multi-model computation and depend on suitable answer extractors and comparison rules, limiting their use beyond domains with clear formats.Mathematical tasks support voting naturally, whereas many general tasks lack appropriate criteria for consensus.Source: §2.1 · p. 7

2.2. External Reward Methods

  • External rewards use unlabeled-data structure or generation-verification asymmetries, deriving signals independently of the model’s internal state.Unlabeled-data methods include next-token, segment, reconstruction, meta-learning, and programmatically verifiable question-answering objectives [Dong et al., 2025, Wang et al., 2025, Li et al., 2025b].Source: §2.2 · pp. 7–8
  • Generation-verification methods exploit tasks where producing solutions is difficult but checking them is cheap and deterministic, spanning mathematics, code execution, and theorem proving.Examples include LADDER [Simonds and Yoshiyama, 2025], RLSR [Simonds et al., 2025], Absolute Zero [Zhao et al., 2025a], DeepSeekMath-V2 [Shao et al., 2025b], and AlphaProof.Source: §2.2 · p. 8
  • External rewards are positioned as more promising for long-run URLVR scaling because data supply and computational verification remain reliable as model capability increases.The open challenges are developing diverse verifiable environments and extending verification asymmetries to new scientific domains.Source: §2.2 · p. 9

3. The Sharpening Mechanism of Intrinsic Rewards

The section shows that intrinsic URLVR rewards sharpen the model’s initial distribution, amplifying correct solutions when confidence aligns with correctness but reinforcing errors when it does not. A representative majority-voting analysis explains the update dynamics and geometric convergence underlying this behavior.

3. The Sharpening Mechanism of Intrinsic Rewards

  • Intrinsic URLVR methods converge toward sharpening the model’s initial distribution, benefiting aligned confidence and correctness but systematically reinforcing errors when they are misaligned.The unified perspective extends beyond majority voting: intrinsic rewards manipulate cross-entropy between carefully chosen distributions.Source: §3 · p. 9, §3.2 · p. 11

3.1. Dynamics of One-Step Update

  • Majority-voting updates create a rich-get-richer dynamic: trajectories producing the majority answer gain probability, while other trajectories are proportionally diminished.The KL-regularized optimum amplifies majority trajectories by e^(1/β), whereas one gradient step moves only partway toward that optimum.Source: §3.1 · p. 10, §3.2 · p. 11

3.2. Convergence Towards Sharpening Initial Distribution

  • Under majority stability and effective learning, repeated one-step updates converge geometrically toward a deterministic policy concentrated on the initial majority answer.The convergence rate is ρ = e^−1/β, with the majority-answer probability converging to 1.Source: §3.2 · p. 11
  • The sharpening mechanism amplifies good solutions when the model’s initial confidence identifies correct answers, but causes model collapse when confidence is poorly aligned with correctness.Majority stability is assumed for sufficiently large rollout counts, and effective learning requires each update to increase majority-answer probability.Source: §3.2 · p. 11

4. When Does Intrinsic URLVR Work?

Intrinsic URLVR improves performance only while initial confidence aligns with correctness, then universally follows a rise-then-fall trajectory as sharpening amplifies misaligned preferences. This behavior varies by problem and reward design, yet sharpening can still generalize to unseen problems when their confidence-correctness alignment is favorable.

4. Fine-Grained Per-Problem Analysis

  • Training can amplify failures on one problem yet improve unseen problems, because confidence-correctness alignment varies across problems.The section’s overall finding is that success depends on alignment for each specific problem, not merely on whether the training problem’s initial answer is correct.Source: §4 · p. 11, §4.2.1 · p. 15

4.1.1. Early Success, Later Collapse

  • Intrinsic URLVR initially matches or exceeds ground-truth training, but proxy rewards keep rising while Reward Accuracy and validation performance decline, producing eventual collapse.Figure 2 shows divergence between optimizing confidence and optimizing correctness; majority-voting training also reduces Actor Entropy faster than ground-truth training.Source: §4.1.1 · p. 12, Figure 2 · p. 12
  • Across hyperparameter settings, nearly all intrinsic methods eventually degrade; even stabilized training collapses around 1,000 steps, indicating a fundamental limitation rather than an engineering problem.Tuning affects when collapse occurs, not whether it occurs.Source: §4.1.1 · pp. 12–13

4.1.2. Different Methods, Different Failures

  • Intrinsic rewards exhibit three distinct failure modes: gradual degradation, length collapse from probability rewards, and repetition collapse from entropy-based rewards.Self-Certainty and Majority Voting degrade more slowly; probability rewards favor short answers, while entropy rewards can encourage repetitive high-probability tokens.Source: §4.1.2 · p. 13, Figure 3 · p. 13

4.2.1. In-Distribution Per-Problem Sharpening

  • Individual problems follow four patterns—amplifying success, amplifying failure, wrong-to-correct, and correct-to-wrong—depending on whether the highest-reward sample is correct and consistent.These cases show that intrinsic training can guide correction, but can also degrade initially correct behavior when rewarded samples are inconsistent.Source: §4.2.1 · p. 14
  • Per-problem training amplifies initial preferences rather than correcting errors: among 25 problems, only 3 flipped greedy correctness, while 22 sharpened their initial preference.Both correct and incorrect answers became more confident, revealing sharpening regardless of whether correctness changed.Source: §4.2 · p. 14, §4.2.1 · p. 15

4.2.2. Out-Of-Distribution Cross-Problem Generalization

  • Despite low training Label Accuracy, all six training problems improved two unseen problems from wrong to correct, with Test Label Accuracy increasing steadily from 0 to 1.This one-shot OOD result shows sharpening can generalize when confidence aligns with correctness on unseen problems.Source: §4.2.2 · p. 15

5. How Can Sharpening from Intrinsic URLVR Be Applied Safely?

Intrinsic URLVR can be applied safely when training data is sufficiently small and domain-specific: small datasets avoid collapse through localized policy shifts, enabling effective test-time training even with incorrect majority votes.

5. How Can Sharpening from Intrinsic URLVR Be Applied Safely?

  • The section demonstrates that small, domain-specific datasets make intrinsic URLVR suitable for safe test-time training, even when training amplifies errors on an incorrect subset.Small datasets induce localized rather than systematic policy shift, and gains can occur even when training uses wrong problems.Source: §5 · pp. 15–16, §5.3 · p. 17

5.1. Small Datasets Prevent Model Collapse

  • Training on ≤128 samples avoids collapse, whereas datasets ≥512 consistently trigger reward hacking; DAPO-32 reaches Majority Voting Reward →1 while preserving high Ground Truth Reward across three seeds.Figure 6 supports the dataset-size comparison; DAPO-32 never collapses, while DAPO-512 always does.Source: §5.1 · p. 16, Figure 6 · p. 16
  • Small datasets induce localized rather than systematic policy shifts: DAPO-32 reaches 0.057 KL after 600 steps, while DAPO-512 reaches 2× higher and large datasets cause global policy shift and collapse.The limited drift sharpens confidence on specific samples without altering the global policy, preserving general reasoning on AIME24/AMC23.Source: §5.1 · pp. 16–17, Figure 7 · p. 16

5.2. Test-Time Training as a Safe Application

  • Test-time training on AMC23 avoids collapse: both Ground Truth Reward and Majority Voting Reward rise and stabilize while performance improves on AMC23 and AIME24, unlike DAPO-17k’s rise-then-fall pattern.These results help explain why recent intrinsic-reward methods focus on test-time rather than train-time settings [Prabhudesai et al., 2025, Zuo et al., 2025].Source: §5.2 · p. 17, Figure 8 · p. 17

5.3. Incorrect Majority Votes Still Improve Reasoning

  • Even when almost all 32 training samples have incorrect initial majority votes, small-scale training produces effective learning without catastrophic collapse and improves AIME24 and AMC23 performance.After 100 steps, Label Accuracy is consistently zero while Majority Voting Reward converges, supporting localized overfitting rather than systematic policy shift.Source: Figure 9 · p. 17, §5.3 · p. 18

6. How Can We Measure Model Prior?

Model Collapse Step measures how long intrinsic URLVR sustains reward accuracy before collapse, providing an efficient, label-free indicator of model priors and RL trainability. Across model families and evaluation settings, longer collapse times predict better supervised RL gains and can be measured rapidly with aggressive hyperparameters.

6.1. Pilot Study of Different Models

  • Intrinsic URLVR varies sharply across model families: Qwen SFT remains stable, whereas Qwen base and both LLaMA variants eventually collapse, with SFT delaying failure.Qwen SFT maintains Reward Accuracy above 0.8 while Qwen base approaches zero by step 200; LLaMA base fails by step 40, while its SFT variant rises before later collapse. These dynamics align with Shao et al. [2025a].Source: §6.1 · pp. 18–19, Figure 10 · p. 19

6.2. Model Collapse Step Accurately Predicts RL Gains

  • Model Collapse Step strongly correlates with GT Gain, matching or surpassing pass@k while remaining reliable on multiple-choice problems.Models that survive longer consistently achieve better standard supervised RL results; pass@k is less reliable and can approach 1 on multiple-choice tasks.Source: §6 · p. 18, Figure 11 · p. 19, §6.2 · p. 20

6.3. Model Collapse Step Rapidly Predicts RL Gains

  • Aggressive hyperparameters accelerate collapse but preserve model rankings, enabling Model Collapse Step to assess strong-prior models at least 50 steps sooner.Across mini-batch sizes {1, 8, 64} and rollout counts {8, 16, 32}, relative predictive power remains stable despite lower absolute collapse steps.Source: §6.3 · p. 20, Figure 12 · p. 20
  • Model Collapse Step uses 5.6× fewer tokens than full RL training, preserves relative model rankings, requires no ground-truth labels, and supports pre-RLVR trainability assessment when verification is unavailable.It captures the point where geometric amplification shifts from beneficial reinforcement of correct outputs to harmful amplification of errors, enabling short diagnostic runs with aggressive hyperparameters and calibrated early stopping.Source: §6.3 · p. 21

7. Discussion

The discussion identifies intrinsic URLVR’s scalability ceiling as confidence-correctness misalignment and motivates external rewards grounded in unlabeled data or generation-verification asymmetries. Self-verification provides preliminary evidence of stronger, scalable signals, especially when combined with instruction alignment.

Intrinsic limitations

  • Intrinsic URLVR cannot consistently exceed the model’s existing knowledge: weak confidence-correctness alignment causes it to amplify biases rather than discover new information.This limitation arises from rewards derived entirely from the model’s internal state, not merely from hyperparameter choices.Source: §7 · p. 21

External reward directions

  • External rewards offer complementary scalable paths by deriving signals from unlabeled-data structure or generation-verification asymmetries that scale with data and computation.Unlike intrinsic rewards, these signals need not saturate with model capacity.Source: §7 · p. 21

Self-verification case study

  • Self-Verification yields higher validation accuracy than Trajectory-Level Entropy for both models, while Ground Truth Reward rises after transient verifier exploitation (Figure 13).Reward Accuracy initially drops around step 200, then recovers and stabilizes above 0.5, indicating learning rather than persistent reward hacking.Source: §7 · p. 22, Figure 13 · p. 22

Instruction alignment

  • Instruction alignment raises Self-Verification from above 60% starting accuracy to over 80% and makes it effective with both prompts, unlike the base model.The instruction model also maintains stable Reward Accuracy across prompts, whereas the base model is highly prompt-sensitive.Source: §7 · pp. 22–23, Figure 14 · p. 22

Scaling implications

  • External verification remains reliable as outputs improve and can be generated cheaply at scale without human annotation, providing fresh signals from data and computation.Arithmetic checking, code execution, formal validation, and unlabeled-text structure are presented as scalable resources beyond intrinsic knowledge.Source: §7 · p. 23

8. Conclusion · A. Details for Section 3 · A.1. Justifying the Ordering 𝑝∗,(𝑘+1)

The paper concludes that intrinsic URLVR sharpens existing model preferences, enabling gains when confidence matches correctness but inevitably collapsing beyond the model’s knowledge boundaries. Its analysis motivates external rewards and uses early training dynamics, including Model Collapse Step, to assess trainability and model prior.

8. Conclusion

  • Intrinsic URLVR methods cannot scale indefinitely: they exploit existing knowledge but eventually collapse regardless of engineering choices or hyperparameter tuning.The shared mechanism amplifies the model’s existing preferences, so gains depend on confidence aligning with correctness.Source: §8 · p. 23
  • Model Collapse Step uses early training dynamics as a practical indicator of model prior and RL trainability.This enables rapid assessment of whether a model is likely to benefit from intrinsic rewards before collapse occurs.Source: §8 · p. 23
  • External reward methods ground verification in computational asymmetries and unlabeled data, with preliminary evidence that they may escape intrinsic methods’ confidence-correctness ceiling.The conclusion presents these methods as a direction for overcoming intrinsic URLVR’s scaling boundary.Source: §8 · p. 23

A.1. Justifying the Ordering 𝑝∗,(𝑘+1)

  • For fixed rewards, the KL-regularized RL objective has an optimal policy that places probability mass according to the reward-regularized solution.Infinite updates from the current policy would reach this optimum, whereas a single update moves only partway toward it.Source: §A.1 · p. 28
  • When rewards mark majority trajectories, one gradient update increases their log-probabilities, so the majority probability is monotonically increasing and bounded.The update is approximate because training uses stochastic gradient descent, and positive-advantage majority trajectories receive the increase.Source: §A.1 · p. 28
  • As rollout count N increases, empirical answer frequencies converge to their true probabilities, stabilizing the majority answer at the initial most-probable answer.Thus arg max_a π^(k)(a|x) remains maj_0(Y) throughout training; in practice, N=8 provides reasonable stability.Source: §A.1 · p. 28

A.1.1. Empirical Validation 1: Validation of Ordering and Majority Stability · A.1.2. Empirical Validation 2: Fixed Reward Convergence Validation

The experiments validate monotonic majority-probability ordering, stable majority answers, and convergence toward deterministic policies. Under fixed rewards, the theoretical optimum is achievable, but complete reward convergence coincides with zero validation performance.

A.1.1. Empirical Validation 1: Validation of Ordering and Majority Stability

  • The ordering and stability experiment used 1,024 rollouts on one MATH-500 problem for 50 steps, with four randomly selected problems monitored.The large rollout count was used to reduce majority-vote randomness while tracking majority-answer stability and p(k).Source: §A.1.1 · p. 28
  • Across the first 10 steps, all four problems showed a strict monotonic increase in p_maj at every step, confirming the ordering lower bound.The result is reported in Table 4 and supports the predicted monotonicity of majority probability during iterative training.Source: Table 4 · p. 28, §A.1.1 · p. 29
  • The majority answer remained stable across all training iterations for the four monitored problems.This stability accompanies the monotonic increase in majority probability and supports the ordering behavior tested in the experiment.Source: §A.1.1 · p. 29
  • At step 50, all four problems reached 98.54%-99.80% majority-probability concentration, validating geometric convergence toward deterministic policies.The same problems were trained for 50 steps, and the observed concentration matched the convergence predicted by Theorem 1.Source: Table 5 · p. 29, §A.1.1 · p. 29

A.1.2. Empirical Validation 2: Fixed Reward Convergence Validation

  • The fixed-reward experiment used 1,024 prompts, mini-batches of one, one-time N=8 rollouts, and 1,024 gradient updates from initial-rollout majority rewards.This setup isolates whether a single KL-regularized RL objective can reach its theoretical optimum when the reward signal remains constant.Source: §A.1.2 · p. 29
  • After 1024 mini-updates using the same fixed reward signal, Majority Voting Reward reached 1.0 (complete convergence), while validation performance on AIME24, AIME25, and AMC23 dropped to zero, confirming Equation (5)’s predicted convergence point is achievable with sufficient updates.Source: §A.1.2 · p. 29

A.2. Proof of Theorem 1 … A.3.3. Monotonicity Analysis

The proof shows that stable-majority intrinsic-reward training converges geometrically to a deterministic policy, while a unified cross-entropy framework explains why diverse intrinsic rewards progressively sharpen model distributions.

A.2. Proof of Theorem 1

  • Under stable-majority and non-trivial-progress assumptions, the error sequence decreases strictly to zero, proving convergence toward the majority-selected deterministic policy.The assumptions are (A1) maj_k = maj_0 and (A2) η_k ≥ η_min > 0; probability mass concentrates on trajectories whose answer equals maj_0(Y_0).Source: §A.2 · pp. 29–31
  • The convergence is geometric, with a rate determined by η_min and α = e^(1/β), and ideal updates achieve ρ = e^(-1/β).The ideal rate applies when η_k = 1 for every iteration.Source: §A.2 · p. 31

A.3. Unified Reward Framework

  • Intrinsic rewards share a unified paradigm: they manipulate cross-entropy between selected anchor distributions and model distributions.This framework consolidates diverse intrinsic reward constructions into a common distributional lens.Source: §A.3 · p. 31, Unified Reward Framework · p. 31

A.3.1. Framework Components

  • The framework specifies reward behavior through aggregation granularity, model and anchor distributions, a sign factor, and a monotonic transformation.Token-level distributions condition on prior generated tokens, whereas answer-level distributions represent complete semantic answers.Source: §A.3.1 · p. 32

A.3.2. Instantiations of the Framework

  • Representative intrinsic methods differ mainly in their choices of granularity, anchor, sign, and transformation, yet all manipulate cross-entropy to sharpen distributions.Table 6 organizes these instantiations, including token-level and answer-level variants.Source: §A.3.2 · p. 32, Table 6 · p. 32
  • Token-level methods encourage local confidence, answer-level methods enforce global answer consistency, and exponential transformations amplify sharpening more than identity transformations.Uniform anchors encourage departure from randomness, while sharp anchors reinforce high-probability paths.Source: §A.3.2 · p. 33

A.3.3. Monotonicity Analysis

  • Regardless of the sign factor, intrinsic rewards progressively sharpen distributions by either rewarding divergence from uniformity or reinforcing already confident predictions.Positive sign rewards increasing cross-entropy against typically uniform anchors; negative sign rewards alignment with typically sharp anchors.Source: §A.3.3 · p. 33

A.4. Generalized Sharpening Analysis via Unified Reward Framework … B.3.1. Majority Voting

This section analyzes intrinsic reward methods through a unified sharpening framework and evaluates their behavior using metrics for pseudo-reward quality, reward hacking, and confidence–correctness alignment. It also examines how Majority Voting hyperparameters affect exploration, policy–reward alignment, regularization, voting reliability, and collapse dynamics.

A.4. Generalized Sharpening Analysis via Unified Reward Framework

  • For σ = −1 methods, Reward-Confidence Monotonicity creates a self-reinforcing reward gap that drives geometric concentration toward dominant trajectories.The analysis assumes the model initially prefers and rewards the dominant trajectory; iterative KL-regularized updates then preserve or widen this gap.Source: §A.4 · pp. 33–34

Step 3: The Reinforcement Loop

  • Self-Certainty does not obey direct raw-confidence monotonicity, but its penalty on high-entropy distributions still favors peaked, low-entropy policies.Its mechanism therefore requires separate analysis from σ = −1 methods.Source: §A.4 · p. 33, Step 3: The Reinforcement Loop · p. 35

Step 1: Existence of a Positive Reward Gap

Step 3: The Reinforcement Loop

  • The reinforcement loop makes sharpening structurally inevitable for confidence-aligned σ = −1 rewards, driving deterministic outputs whether the prior is correct or incorrect.Increasing dominant-path probability increases or maintains its reward advantage, producing a persistent rich-get-richer dynamic.Source: Step 3: The Reinforcement Loop · p. 34

A.5. Optimal Policies Induced by other Intrinsic Rewards

  • Self-Certainty, entropy, trajectory-entropy, and probability rewards all increase the probability of sequences already favored by the prior policy, while EMPO concentrates mass on the high-probability answer.These optimal-policy analyses show that intrinsic rewards enhance prior-policy confidence at token, trajectory, or answer granularity.Source: §A.5 · pp. 35–37

B.1. Experimental Setup

  • The experiments implement five intrinsic rewards with GRPO and monitor training to identify the transition from beneficial sharpening to pathological collapse.The monitored signals compare pseudo-rewards with ground truth and measure whether confidence-based proxies track correctness.Source: §B.1 · pp. 38–39

B.2. Calculation of Training Dynamics

  • The training-dynamics metrics diagnose reward hacking and distribution sharpening using prompt-level labels, sample-level reward agreement, oracle rewards, majority-voting rewards, and certainty-based accuracy.These definitions distinguish pseudo-label degradation from corruption of the reward signal.Source: §B.1 · p. 39, §B.2.2 · pp. 39–40, §B.2.3 · p. 40

B.2.3. Certainty-Based Metrics

  • Certainty-based metrics select the highest-confidence response by maximizing r_cert(y_i,j) over candidate responses for prompt x_i.This defines the certainty-based response-selection criterion used in the analysis.Source: §B.2.3 · p. 40

B.3. Hyperparameter Tuning

  • The hyperparameter study varies training temperature, mini-batch size, KL regularization, and rollout count one at a time against fixed baseline settings.These four parameters directly influence intrinsic-reward training performance.Source: §B.3 · p. 40

B.3.1. Majority Voting

  • Training temperature T=1.0 provides the optimal balance, yielding steady early gains with delayed degradation versus lower- or higher-temperature alternatives.Low T∈{0.6, 0.8} rapidly sharpens logits and destabilizes Label Accuracy, whereas T=1.2 preserves stability longer but adds noise.Source: §B.3.1 · p. 40
  • Mini-batch size 1 collapses within 20 steps, while on-policy training with mini-batch 64 is maximally stable; sizes 16–32 improve gradually.Small batches create reward staleness by applying pseudo-rewards computed under the current policy to older-policy samples.Source: §B.3.1 · pp. 40–41, Figure 17 · p. 41
  • Adding KL regularization with β=0.005 provides only marginal early gains, increases variance, and delays collapse minimally to approximately 40 steps.Intrinsic sharpening and KL attraction to the reference policy create competing pressures that oscillate rather than producing durable stability.Source: §B.3.1 · pp. 41–42, Figure 18 · p. 42
  • Rollout counts N=32 and N=16 collapse within 180 and 220 steps, respectively, whereas N≤8 remains stable through the full epoch.Although N=4 can be competitive, N=8 is recommended because it improves voting reliability while retaining robust stability across problem types.Source: §B.3.1 · pp. 42–43, Figure 19 · p. 42

B.3.2. Certainty-Based Methods … C.2. Impact of Backbone Model

Certainty-based methods are highly sensitive to temperature, mini-batch size, and rollout count, while Self-Certainty is more stable but usually improves less. The appendices specify self-verification prompts and evaluate backbone effects across Qwen and Llama models, training stages, and sizes.

B.3.2. Certainty-Based Methods

  • Higher temperature T=1.2 delays collapse for entropy- and probability-based methods by sustaining exploration and improving certainty–correctness alignment.These methods maintain higher Actor Entropy, gradual validation gains, and higher Point-Biserial Correlation values at T=1.2.Source: §B.3.2 · p. 44
  • Self-Certainty instead performs better at T=1.0, because T=1.2 causes excessive exploration, persistent entropy, and lower validation and Label Accuracy scores.The methods therefore require method-specific temperature tuning despite converging toward similarly sharp distributions.Source: §B.3.2 · p. 44
  • Larger mini-batches prevent premature collapse for Token-Level Entropy, Trajectory-Level Entropy, and Probability methods, with pure on-policy training at mini-batch size 64 maintaining the best coupling.Certainty estimates become unreliable when applied to samples generated by earlier policy iterations.Source: §B.3.2 · p. 44
  • Self-Certainty is minimally sensitive to mini-batch size and rollout count, offering superior stability but limited performance improvement through KL-divergence-based logit comparisons.Its robustness contrasts with other certainty methods, whose estimates are more vulnerable to temporal policy mismatch and sample-size changes.Source: §B.3.2 · pp. 45, 48, Figure 24 · p. 45, Figure 32 · p. 48
  • KL regularization has minimal impact on certainty-based training and downstream performance, generally failing to prevent eventual collapse except for Self-Certainty.Entropy-based methods show only modest Label Accuracy improvements, suggesting slightly better compatibility with KL stabilization.Source: §B.3.2 · pp. 46–47
  • Rollout counts N≥16 accelerate convergence and premature collapse for most certainty methods, as larger samples amplify confidence in potentially incorrect assessments.Self-Certainty remains stable across rollout configurations but shows neither collapse nor significant improvement.Source: §B.3.2 · pp. 47–48, Figure 33 · p. 48, Figure 34 · p. 48

C.1. Prompts for Self-Verification

  • Self-verification prompts require checking expression validity, number usage, operator constraints, numerical equality to the target, and returning boxed True only when all checks pass.The prompt explicitly forbids generating a new expression and requires boxed False for invalid or failing expressions.Source: §C.1 · p. 49

C.2. Impact of Backbone Model

  • The backbone study compares 11 Qwen and Llama models across training stage, model size, and architectural generation using DAPO-17k, optimal Section B.3 hyperparameters, and Majority Voting.The cross-architecture design addresses reported architectural differences [Gandhi et al., 2025] and potential data contamination [Wu et al., 2025b].Source: §C.2 · pp. 49–50, Table 8 · p. 50

C.2.1. Horizontal Analysis: Training Stage Impact

Training-stage progression reveals a sharp architectural split: Qwen math-specialized and SFT models remain stable, whereas Llama variants eventually experience reward hacking. Qwen base and instruct models saturate earlier, while Llama base models collapse earliest.

C.2.1. Horizontal Analysis: Training Stage Impact

  • Qwen math-specialized and SFT models maintain Majority Voting Reward within 0.3-0.6, while base and instruct variants reach saturation (1.0) by step 180.Math specialization and strong supervised fine-tuning provide robust optimization foundations, establishing Qwen’s genuine stability advantage.Source: §C.2.1 · pp. 50–51, Figure 37 · p. 51
  • All Llama variants eventually succumb to reward hacking, with base models failing earliest at step 40, followed by math-specialized, SFT, and instruct versions.The differing collapse timing contrasts with Qwen’s stability across training stages.Source: §C.2.1 · p. 50, Figure 38 · p. 51

C.2.2. Vertical Analysis: Scale and Generation Effects · C.3. Impact of Training Dataset · C.4. Does Intrinsic Reward Methods Truly Improve Capabilities?

The analyses show that intrinsic-reward training is sensitive to model scale, generation, and dataset size, yet TTRL can improve capability beyond base-model self-consistency. Smaller or newer models train more stably, larger datasets trigger reward hacking, and TTRL-trained models exceed the base models’ majority-vote performance.

C.2.2. Vertical Analysis: Scale and Generation Effects

  • Smaller models consistently remain stable longer than larger variants, with Octo-3B outlasting Octo-8B by about 40 steps.Q3-1.7B also maintains stability significantly longer than Q3-4B, suggesting larger capacity amplifies sensitivity to noisy pseudo-rewards.Source: §C.2.2 · p. 51, Figure 39 · p. 52
  • Qwen3 models exhibit more stable and controlled training dynamics than Qwen2.5 counterparts.Q3-1.7B-Base shows more controlled Majority Voting Reward progression, plausibly due to better-calibrated uncertainty estimates and more reliable pseudo-reward computation.Source: §C.2.2 · pp. 51–52, Figure 40 · p. 52

C.3. Impact of Training Dataset

  • DeepScaleR-40k and ORZ-56k show clear reward hacking, whereas smaller datasets remain in steady or rising stages.All datasets were evaluated on Qwen3-1.7B-Base for one epoch using optimal hyperparameters and three validation benchmarks.Source: §C.3 · pp. 52–53
  • Extending intrinsic-reward training to much larger corpora inevitably encounters reward hacking after short-sighted early improvements.The dataset comparison indicates that early gains can reflect only the initial steady or rising phase of intrinsic-reward training.Source: §C.3 · p. 53

C.4. Does Intrinsic Reward Methods Truly Improve Capabilities?

  • Prevailing intrinsic-reward methods primarily reduce uncertainty, raising the question of whether they improve capability or merely output self-consistency.TTRL uses majority voting over m model outputs as a supervisory signal, which could appear limited by the base model’s majority-vote upper bound.Source: §C.4 · p. 53
  • TTRL-trained models’ pass@1 significantly exceeds the base models’ maj@∞ on AIME 2024, demonstrating capability gains beyond self-consistency alignment.The comparison used Qwen2.5-Math-1.5B and 7B, with base-model maj@1024 treated as an approximation to maj@∞.Source: Table 9 · p. 53, §C.4 · p. 54
  • TTRL was evaluated on AIME 2024 using 30 samples, batch size 30, and 100 training epochs across Qwen2.5-Math-1.5B and 7B.The experiments directly compared base-model maj@1024 with trained-model pass@1 (avg@32).Source: §C.4 · p. 54

Related papers

Loading 2603.08660v1…