PaperlayerView exact revision on arXiv

Source-linked AI summary

TraPO: A Semi-Supervised Reinforcement Learning Framework for Boosting LLM Reasoning

Shenzhi Yang, Guangcheng Zhu, Xing Zheng, Yingfan MA, Zhongqi Chen, Bowen Song, Weiqiang Wang, Junbo Zhao, Gang Chen, Haobo Wang

arXiv:2512.13106v1cs.LGcs.AI

TL;DR

RLVR relies on verifiable labels that are costly, while unsupervised internal rewards can reinforce incorrect reasoning. TraPO uses labeled examples to select unlabeled samples by pass-rate trajectory similarity. With 4K labeled and 12K unlabeled samples, it surpasses fully supervised RLVR trained on 45K labeled samples across all benchmarks.

  • Problem

    RLVR’s reliance on gold labels incurs high annotation costs, while unsupervised rewards based on internal signals can reinforce incorrect consensus without external ground truth.

    Source: §1 · pp. 1–2

  • Method

    TraPO selects reliable unlabeled samples by matching their pass-rate learning trajectories to those of labeled samples.

    Source: §3.2 · p. 6

  • Results

    With 4K labeled and 12K unlabeled samples, TRAPO surpasses fully supervised RLVR trained on 45K labeled samples across all benchmarks, using 10% of the labeled data.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The results support using a small labeled set to guide robust self-improvement on unlabeled data while balancing data efficiency and learning effectiveness.

    Source: §5 · p. 11

    Unsupervised proxy rewards are vulnerable when they diverge from correctness, as when majority voting favors an incorrect answer.

    Source: §3.1 · p. 5

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards (RLVR) has proven effective in training large reasoning models (LRMs) by leveraging answer-verifiable signals to guide policy optimization, which, however, suffers from high annotation costs. To alleviate this problem, recent work has explored unsupervised RLVR methods that derive rewards solely from the model's internal consistency, such as through entropy and majority voting. While seemingly promising, these methods often suffer from model collapse in the later stages of training, which may arise from the reinforcement of incorrect reasoning patterns in the absence of external supervision. In this work, we investigate a novel semi-supervised RLVR paradigm that utilizes a small labeled set to guide RLVR training on unlabeled samples. Our key insight is that supervised rewards are essential for stabilizing consistency-based training on unlabeled samples, ensuring that only reasoning patterns verified on labeled instances are incorporated into RL training. Technically, we propose an effective policy optimization algorithm, TraPO, that identifies reliable unlabeled samples by matching their learning trajectory similarity to labeled ones. Building on this, TraPO achieves remarkable data efficiency and strong generalization on six widely used mathematical reasoning benchmarks (AIME24/25, AMC, MATH-500, Minerva, and Olympiad) and three out-of-distribution tasks (ARC-c, GPQA-diamond, and MMLU-pro). With only 1K labeled and 3K unlabeled samples, TraPO reaches 42.6% average accuracy, surpassing the best unsupervised method trained on 45K unlabeled samples (38.3%). Notably, when using 4K labeled and 12K unlabeled samples, TraPO even outperforms the fully supervised model trained on the full 45K labeled samples on all benchmarks, while using only 10% of the labeled data. The code is available via https://github.com/ShenzhiYang2000/TRAPO.

1 Introduction

RLVR guides reasoning models with answer-verifiable rewards, but gold labels are costly and purely unsupervised rewards can reinforce incorrect reasoning. TraPO uses labeled examples to guide learning from unlabeled data, with strong results at a fraction of the labeled-data scale.

1 Introduction

  • RLVR generates multiple reasoning paths per question and rewards answers that match verifiable ground-truth labels to guide policy optimization.The resulting group-based advantage estimation guides exploration toward reasoning paths that produce correct final answers.Source: §1 · p. 1
  • With 4K labeled and 12K unlabeled samples, TRAPO surpasses fully supervised RLVR trained on 45K labeled samples across all benchmarks, using 10% of the labeled data.With 1K labeled and 3K unlabeled samples, it improves in-domain performance by 4.3% over the strongest unsupervised baseline trained on 45K unlabeled samples.Source: Figure 1 · p. 2, §1 · p. 3
  • Unsupervised RLVR uses internal signals such as self-certainty, entropy, or majority voting, but can reinforce incorrect consensus and degrade in late training.The introduction attributes this risk to rewards that lack external ground truth and can reinforce systematic biases.Source: §1 · p. 2

2 Related Work

Prior work spans supervised and unsupervised RLVR, semi-supervised learning, and reasoning-data selection. The paper argues that pass-rate learning dynamics offer a bridge between labeled and unlabeled examples where answer or feature similarity is inadequate.

Semi-supervised learning

  • In RLVR, examples have instance-specific solution spaces, making direct alignment of unlabeled and labeled samples through standard similarity methods impractical.The paper turns from what the model learns to how it learns, using pass-rate changes as a bridge.Source: §2 · p. 3

Unsupervised RLVR

  • Unsupervised RLVR uses self-generated signals such as entropy, self-confidence, or majority voting, but biased or noisy signals risk reinforcing incorrect behavior and model collapse.The paper positions its framework as anchoring intrinsic signals with labeled data.Source: §2 · p. 3

Reasoning data selection

  • Reasoning-data selection methods use external resources or internal signals, but internal metrics may not capture which examples genuinely benefit learning.The paper proposes probing learning dynamics to identify unlabeled instances that contribute to effective and robust training.Source: §2 · pp. 3–4

3 Method

The method anchors unlabeled RLVR with rewards verified on labeled data, then selects unlabeled samples whose pass-rate learning trajectories align with trusted labeled trajectories. A warm-up and trajectory-consistency analysis support this selection-and-training framework.

3.1 Semi-supervised Reinforcement Learning with Verifiable Rewards

  • When a proxy reward diverges from actual correctness, majority-voted incorrect responses can be reinforced and produce a feedback loop of increasingly confident errors.Unsupervised methods assume higher confidence implies a greater probability of correctness.Source: §3.1 · p. 5
  • Labeled rewards use ground-truth answers, while unlabeled rewards may use self-consistency; this separates correctness from internal agreement.That distinction is intended to prevent reinforcing outputs that are internally consistent but incorrect.Source: §3.1 · p. 5

3.2 Progressive Trajectory Guidance for Bridging Labeled and Unlabeled Data

  • TRAPO estimates each sample’s pass rate across training epochs and selects unlabeled samples whose trajectories align with those of labeled samples.Because unlabeled pass rates cannot be directly computed, the method uses a pseudo-pass-rate approximation.Source: §3.2 · p. 6
  • Reliable trajectories are chosen from the highest-similarity unlabeled samples or those exceeding a similarity threshold.The reliable database starts with labeled trajectories and is updated with selected unlabeled trajectories.Source: §3.2 · p. 6
  • Training warms up with labeled-only updates while accumulating unlabeled trajectories, then applies a selection mask to include reliable unlabeled samples.The selected samples are integrated into the GRPO-based training objective.Source: §3.2 · p. 7
  • The generalization analysis treats trajectory consistency as a regularizer that encourages unlabeled samples to follow learning dynamics similar to labeled data.The bound also depends on empirical labeled-data risk, domain shift, and average voting confidence.Source: §3.2 · p. 7

4 Experiment

The experiments evaluate TraPO across mathematical reasoning benchmarks and out-of-domain tasks, comparing supervised, unsupervised, and semi-supervised RLVR. Results cover overall performance, cross-domain unlabeled data, trajectory matching, parameter sensitivity, and training behavior on another model.

4.1 Setup

  • The evaluation uses OpenR1-Math-220k training data and six math benchmarks plus ARC-c, GPQA-diamond, and MMLU-Pro; baselines span supervised, unsupervised, and semi-supervised settings.AIME 2024/2025 and AMC use avg@32; the other benchmarks use pass@1.Source: §4.1 · pp. 7–8

4.2 Experimental Results

  • With 1K labeled math samples and 1K unlabeled non-math samples, TraPO beats the best unsupervised baseline by 1.8% ID and 3.5% OOD.It trails the fully supervised model trained on 2K labels by 0.9% on both metrics.Source: Table 2 · p. 9, §4.2 · p. 10
  • With 1K labeled and 3K unlabeled samples, TraPO beats the best unsupervised baseline trained on 45K unlabeled samples by 4.3% ID and 3.7% OOD accuracy.It also improves on the strongest naive semi-supervised baseline by 2.6% ID and 3.5% OOD, and surpasses the 1K-label supervised model by 3.2% ID and 4.0% OOD.Source: §4.2 · p. 9
  • TRAPO achieves 45.6 ID and 59.7 OOD accuracy, exceeding the fully supervised model trained on all 45K labels by 0.1% ID and 2.4% OOD, despite using only 10% of total labels.Source: §4.2 · p. 10
  • The top 10% of unlabeled samples by trajectory alignment outperform the bottom 10% by over 40%, and pseudo-dynamics matching retains a strong positive correlation with performance.Figure 4 compares performance with trajectory-matching scores using true and pseudo training dynamics.Source: Figure 4 · p. 9, §4.2 · p. 10
  • TraPO performs best using the top 30% of unlabeled samples; adding more increases noise and reduces gains, while it outperforms random selection and a token-level entropy baseline.The sensitivity analyses also report degraded performance from overly high top-p, too-low or too-high Γ, and short warm-up lengths.Source: §4.2 · pp. 10–11, Figure 5 · p. 10
  • On Llama-3.1-8B-Instruct, TraPO follows a trend similar to supervised training, while unsupervised training collapses within tens of steps.The passage attributes training stability to effective pseudo-supervision selection via trajectory matching.Source: Figure 6 · p. 10, §4.2 · p. 11

5 Conclusion

The conclusion presents TRAPO as semi-supervised RLVR that uses a small labeled set to guide self-improvement on unlabeled data. The appendix develops theoretical results about GRPO and learning dynamics under NTK and gradient-alignment assumptions.

5 Conclusion

  • TRAPO aligns labeled and unlabeled learning dynamics through pass-rate trajectory similarity to provide reliable pseudo-supervision and improve data efficiency.The conclusion reports that it outperforms baselines using only a fraction of labeled data.Source: §5 · p. 11

A.2 GRPO as Preference Optimization

  • With binary rewards, the expected GRPO loss reduces to a weighted preference-optimization objective.The responses are grouped as correct and incorrect, with group-specific weights.Source: §A.2 · p. 16

A.3.1 Change in Log-Probability

  • The gradient-dynamics analysis considers how a GRPO update on one question changes response log-probabilities for another question.It assumes an approximately constant NTK and gradient alignment for semantically similar questions in the same task family.Source: §A.3 · p. 17, §A.3.1 · p. 18

A.3.2 Main Generalization Result

Under the stated gradient-alignment assumptions, GRPO raises the log-probability of correct responses and lowers it for incorrect responses to similar questions. The appendix also connects pass-rate trajectory divergence to domain discrepancy.

A.3.2 Main Generalization Result

  • Correct-response log-probability increases while incorrect-response log-probability decreases for similar questions under the GRPO update and NTK alignment assumptions.The stated result applies in the infinite-width limit.Source: §A.3.2 · pp. 19–20

A.4 Unifying Trajectory Divergence and Domain Discrepancy

  • Gradient coherence implies similar pass-rate evolution and low trajectory divergence across questions.The appendix describes trajectory divergence as angular dissimilarity between confidence-evolution paths.Source: §A.4 · p. 20
  • Low pass-rate trajectory divergence implies low domain discrepancy, supporting transfer without explicit adversarial or feature-level alignment.The proposition bounds HΔH-divergence between labeled and unlabeled distributions using expected pass-rate trajectory divergence.Source: §A.4 · p. 21

A.5 Main Theorem: Generalization Bound

The theorem states a high-probability bound on expected target-distribution risk. Its proof combines a domain-adaptation decomposition with trajectory-based discrepancy and pseudo-labeling-error bounds.

A.5 Main Theorem: Generalization Bound

  • With probability at least 1 − δ, the theorem bounds the model's expected risk on the target distribution.The theorem statement also specifies bounded loss and a Lipschitz condition.Source: §A.5 · p. 22
  • The listed bound quantities include empirical risk on labeled source samples, an irreducible domain-shift term, and voting confidence.The passage identifies average voting confidence across unlabeled samples as based on G votes.Source: §A.5 · p. 22
  • The proof combines a domain-adaptation risk decomposition, a trajectory-based domain-discrepancy bound, and a pseudo-labeling-error bound.It combines the steps with a union bound.Source: §A.5 · pp. 22–23

A.6 Main Theorem: Convergence Analysis

The convergence analysis states conditions under which target risk approaches a stationary point of source risk. Its accompanying proof connects gradient coherence to trajectory coherence through log-probability updates and pass-rate evolution.

A.6 Main Theorem: Convergence Analysis

  • The convergence theorem concerns expected target risk under the Consistent Trajectory Learning Condition and stated assumptions.The theorem identifies the limiting point as a stationary point of source risk.Source: §A.6 · pp. 23, 25
  • Under the convergence argument, the residual term tends to zero, and trajectory-consistency risk approaches zero.The proof uses a decreasing trajectory-divergence assumption and a bounded-gradient condition.Source: §A.6 · pp. 24–25

A.7 Addition Proofs

  • The supplementary proof links gradient coherence to trajectory coherence by bounding log-probability updates, relating them to pass-rate evolution, and bounding trajectory-vector cosine distance.The restated lemma assumes small learning rates and an approximately constant NTK.Source: §A.7 · p. 25

B Discussion and Limitations

The discussion reports that semi-supervised training with 4K labeled and 16K unlabeled samples outperforms fully supervised training on 45K labeled samples. It also identifies model alignment as a condition for effective guidance and flags larger models and the choice of guiding examples as future directions.

Discussion

  • 4K labeled plus 16K unlabeled samples outperform fully supervised training on 45K labeled samples.The authors relate this to the benefits of thorough training on smaller curated datasets.Source: B Discussion and Limitations · p. 27

Limitations

  • Evaluation is limited to models under 7B parameters because of computational constraints.Applicability and scalability to larger models, such as 13B or beyond, remain for future study.Source: B Discussion and Limitations · p. 27

Discussion

  • The guidance from labeled data depends on how effective supervised training is for the model; the authors recommend TRAPO where model and labeled data are well aligned.When labeled RLVR training yields only marginal gains, its effect on unlabeled-data filtering is also limited.Source: B Discussion and Limitations · p. 28

Future Work

  • Which types of labeled examples best guide unlabeled-data training remains an open direction for future work.The authors propose investigating this question, similarly to active learning.Source: B Discussion and Limitations · p. 28

C Experiment Details

The experiments report comparisons across supervised baselines, model families, selection strategies, and DeepMath, alongside implementation and prompt choices. Across these tests, TRAPO improves on relevant baselines, while repeated runs show stable results and sample selections.

C.1 Detailed Setup; C.2 System Prompt

  • The default setup uses Qwen2.5-Math-7B, disables KL regularization, sets entropy coefficient to 0.01, and uses 8 rollouts per prompt; LLaMA-3.1-8B receives a simplified CoT prompt.The shared prompt asks most models for Thought and Solution sections; LLaMA-3.1-8B uses a simplified version without the <think> token.Source: §C.1 · p. 28, §C.2 · p. 29

C.3 Baseline Description

  • Baselines include unsupervised methods using majority voting, confidence, or entropy signals, alongside supervised RLVR methods such as Simple-RL, Oat-Zero, PRIME-Zero, and OpenReasonerZero.The fully supervised baseline uses on-policy RL within RLVR with Dr.GRPO, the same reward, and the same data.Source: §C.3 · p. 29

D.1 Comparison with More Supervised RLVR Baselines

  • TRAPO with 4K labeled and 12K unlabeled samples surpasses fully supervised baselines trained on 45K labeled data, beating Oat-Zero by 1.9% in-distribution and 14.5% out-of-distribution.Table 4 presents the comparison with fully supervised methods.Source: Table 4 · p. 30, §D.1 · p. 30

D.2 Extend TraPO to More Models

  • Across two additional models, TRAPO beats TTRL by 2.0% ID and 9.5% OOD on DeepSeek-R1-Distill-Qwen-1.5B, and by 1.2% ID and 0.9% OOD on LLaMA-3.1-8B-Instruct.On LLaMA-3.1-8B-Instruct, TRAPO also exceeds the fully supervised baseline by 0.5% ID; Tables 5 and 6 give model-specific results.Source: §D.2 · p. 30, Table 6 · p. 31, Table 5 · p. 31

D.3 TraPO Is a Universal Component

  • Adding trajectory-based sample selection to Sentence-level Entropy, Token-level Entropy, and TTRL consistently improves performance over naive semi-supervised counterparts across multiple benchmarks.Figure 7 compares the filtering approach with naive combinations of supervised and unsupervised methods.Source: §D.3 · pp. 30–31, Figure 7 · p. 32

D.4 Run TraPO on DeepMath

  • On DeepMath, TRAPO improves 1.5% over the best naive semi-supervised method with TTRL on ID, trails fully supervised training by 1.2% on ID, and exceeds it by 2.4% on OOD.The experiment uses 2K labeled and 8K unlabeled samples; Table 8 reports the results.Source: §D.4 · p. 31

D.5 Different Selection Strategies

  • With a fixed 30% selection ratio, random selection, sentence-level entropy, and self-certainty each perform significantly worse than TraPO on both ID and OOD test sets.Table 13 reports the comparisons across nine benchmarks.Source: §D.5 · p. 32

D.6 Stability of TraPO

  • Across three randomly shuffled runs, both results and selected samples were nearly identical for Qwen-2.5-7B trained with 1K labeled and 3K unlabeled samples.Table 16 reports the three-run results.Source: §D.6 · p. 32

D.7 Training Cost Analysis of TraPO

TraPO adds cosine-similarity computations over short vectors, which are negligible relative to rollout generation, so its theoretical time complexity matches fully supervised RLVR. Empirically, it incurs no substantial additional training cost; an alternative trajectory-matching variant can select ineffectively.

D.7 Training Cost Analysis of TraPO

  • TraPO and fully supervised RLVR have identical time complexity, dominated by forward sampling and GRPO updates.The additional cosine similarity computation over short vectors is negligible compared with rollout generation; Table 7 summarizes wall-clock times by data scale.Source: §D.7 · p. 32, Table 7 · p. 33
  • TraPO incurs no substantial additional training cost compared with supervised RLVR, and reaches its best checkpoint at nearly the same training step.The empirical comparison used the same epochs, batch sizes, and 8×H200 GPU configuration.Source: §D.7 · p. 33, Table 7 · p. 33

D.8 Different Ways of Utilizing Reliable Passrate Databases

  • Matching each unlabeled sample to the closest labeled pass-rate trajectory can select ineffectively because easy, difficult, and moderate problems may all appear similarly matched.The authors report this variant can make selection unstable; Table 17 compares it with average trajectory matching.Source: §D.8 · p. 33

E More Related Work

Prior semi-supervised approaches often combine supervised and unsupervised signals independently, while RLVR consistency signals can reinforce incorrect reasoning without external supervision. TRAPO instead uses labeled data to guide unlabeled-sample selection through learning-trajectory similarity.

E More Related Work

  • Earlier semi-supervised reinforcement-learning methods commonly combine reward and consistency objectives independently or pre-train on unlabeled data before labeled fine-tuning.The paper distinguishes this from using labeled data to steer which unlabeled samples contribute to training.Source: E More Related Work · p. 33
  • RLVR methods use answer-verification feedback, while unsupervised variants rely on internal consistency signals that can reinforce incorrect or degenerate reasoning and lead to model collapse.The cited signals include low token entropy, high self-certainty, and majority voting.Source: E More Related Work · p. 33

E More Related Work; F Pseudo Code

  • TRAPO selects unlabeled samples whose reward-model-space learning trajectories resemble reliable labeled trajectories, so unsupervised signals are used when aligned with externally validated behavior.The algorithm outlines labeled and unlabeled data, warm-up epochs, a threshold, and a top-p fraction as inputs, then computes pass rates and updates trajectories before selection.Source: E More Related Work · pp. 33–34

E More Related Work

  • The paper frames supervised guidance as a way to address limitations of independently combined signals and reports data-efficient performance against fully supervised and unsupervised baselines.The supplied passage states this conclusion without giving the supporting benchmark values.Source: E More Related Work · p. 34
Loading 2512.13106v1…