Source-linked AI summary
TTPO: Test-Time Policy Optimization
Aozhe Wang, Zhengxi Lu, Jianze Wang, Shangke Lv, Ying Liu, Weiming Lu, Jun Xiao, Yueting Zhuang, Hua Yang, Qianglong Chen, Yongliang Shen
TL;DR
Ground-truth-dependent RL and OPSD cannot directly support label-free test-time training, while majority-vote pseudo-labels can corrupt all-token distillation when wrong. TTPO routes agreeing rollouts to OPSD and disagreeing rollouts to grouped RL, with token-level selection in both branches. Without labels, it matches label-supervised OPSD across five competition-level benchmarks, raises Qwen3-1.7B from 38.0% to 45.2% in TTT, and generalizes across tasks.
Problem
RL and OPSD require ground-truth answers, while majority-vote pseudo-label errors can corrupt teachers and mislead every token during label-free test-time training.
Method
TTPO distills agreeing rollouts with OPSD, penalizes disagreeing rollouts with GRPO, and uses token-level selection to refine both branches.
Results
Without labels, TTPO matches or exceeds label-supervised OPSD across five competition-level benchmarks and raises Qwen3-1.7B from 38.0% to 45.2% average accuracy in TTT.
Takeaways & Limitations
TTPO provides label-free test-time training with dense guidance on positive rollouts, robust penalties on negative rollouts, and strong cross-task generalization.
Takeaways & Limitations
TTPO depends on majority-vote quality; very small sample budgets or problems with no correct rollout can make both branches receive noisy supervision.
Abstract
from arXiv · showhide
Recent prominent post-training methods, such as Reinforcement Learning (RL) and On-Policy Self-Distillation (OPSD), have driven rapid progress in mathematical reasoning for large language models, yet their reliance on ground-truth labels precludes test-time training (TTT). Replacing ground truth with majority-vote pseudo-labels is a natural alternative, yet it is fragile: an incorrect vote corrupts the teacher and misleads every token. We observe that this failure mode is asymmetric: rollouts that disagree with the pseudo-label are typically wrong regardless of whether the vote itself is correct. Building on this observation, we propose Test-Time Policy Optimization (TTPO), an asymmetric objective that distills agreeing rollouts via OPSD and penalizes disagreeing rollouts with Grouped RL. Token-level selection further refines both branches: distillation down-weights already-converged positions, while RL penalizes only confident errors. Both updates remain well-grounded even under frequent pseudo-label errors, and majority-vote routing yields tighter self-supervision as the model improves. Without any labels, TTPO matches label-supervised OPSD on five competition-level benchmarks, raises Qwen3-1.7B from 38.0% to 45.2% in TTT, yields +25.2% to +36.4% without thinking, and shows strong cross-task generalization.
1 INTRODUCTION
Existing RL and OPSD methods rely on ground-truth answers, which prevents their direct use in label-free test-time training. TTPO exploits an asymmetry in noisy majority-vote supervision by distilling agreeing rollouts and penalizing disagreeing ones.
- RL and OPSD assume ground-truth answers, preventing their use when test-time training labels never arrive.
- TTPO distills rollouts agreeing with the pseudo-label and applies GRPO penalties to disagreeing rollouts.The agreeing branch uses an answer-conditioned teacher, while the disagreeing branch relies on disagreement rather than the pseudo-label’s answer.
- Token-level selection down-weights converged distillation positions and focuses penalties on confident errors.
- Without labels, TTPO matches or exceeds label-supervised OPSD across five competition-level benchmarks and raises Qwen3-1.7B average accuracy from 38.0% to 45.2% in TTT.It also reports strong cross-task generalization and gains of +25.2% to +36.4% with thinking disabled.
- ∼85% of competition-level prompts have wrong pseudo-labels, yet ∼79% of disagreeing rollouts are also wrong.This makes disagreement useful for penalty-based supervision even when the majority vote is incorrect.
2 RELATED WORK
Related work combines reinforcement learning, self-distillation, and token-level gradient selection to improve reasoning supervision. In test-time training, TTRL extends pseudo-reward-based GRPO to unlabeled reasoning problems, while later methods address consensus sensitivity and token selection.
- TTRL samples multiple trajectories, derives pseudo-rewards by majority voting, and trains with GRPO in label-free reasoning test-time training.
- Self-distillation methods condition the same model on privileged training-time information to create a teacher without a separate teacher model.
- Recent methods combine teacher–student signals with RL through advantage scaling or related log-probability-gap objectives.
- Token-selection research finds that selective gradients can preserve distillation performance and prevent disproportionate or spurious reinforcement-learning updates.
3 METHOD
TTPO partitions rollout trajectories by a majority-vote pseudo-label, distilling agreeing samples with OPSD and penalizing disagreeing samples with GRPO. Token-level weighting and masking focus each branch on informative positions while limiting damage from incorrect pseudo-labels.
- 3.1 PRELIMINARIES AND PROBLEM SETUP: TTPO samples trajectories, extracts their answers, and partitions them into positive samples agreeing with the mathematically equivalent majority cluster and negative samples that disagree.The largest answer-equivalence cluster supplies the pseudo-label.
- 3.2 MOTIVATION: WHY ASYMMETRIC?: Because pseudo-labels are frequently wrong, TTPO applies OPSD to agreeing rollouts and GRPO penalties to disagreeing rollouts, reducing the spread of pseudo-label errors.The asymmetric routing relies on disagreement being informative even when the majority answer is incorrect.
- 3.3 POSITIVE SAMPLES: OPSD BRANCH: For positive samples, forward-KL distillation uses token weights based on normalized student entropy and teacher–student divergence, emphasizing uncertain or confidently wrong positions.The Soft-OR weight approaches zero when the student is already confident and aligned with the teacher.
- 3.4 NEGATIVE SAMPLES: GRPO BRANCH: For negative samples, group-relative advantages make GRPO penalize trajectories that disagree with the pseudo-label.Negative samples receive negative group advantages under binary majority-vote rewards.
- 3.4 NEGATIVE SAMPLES: GRPO BRANCH: Negative-sample token masking ranks tokens by raw negative log-probability and certainty, selecting the top-50% to prioritize confident anomalous outputs.This excludes typically high-probability locally correct tokens from indiscriminate penalties.
- 3.5 UNIFIED OBJECTIVE: The unified objective combines the OPSD and GRPO losses with a balancing weight λ, and the complete procedure updates the policy after both branches.Algorithm 1 summarizes pseudo-labeling, teacher–student computation, branch-specific losses, and the final update.
4 EXPERIMENTS
Experiments evaluate TTPO against label-dependent and label-free baselines across five competition-level benchmarks, then analyze its components, generalization, supervision, and self-evolution. TTPO consistently improves test-time training performance, with gains attributed to asymmetric updates and token-level selection.
- Experimental setup: Evaluation covers Qwen3-1.7B, 4B, and 8B on AIME 2025, AIME 2026, HMMT 2025, HMMT 2026, and BRUMO 2025.Results use Avg@12 with temperature 1.0; OPSD and TTPO are trained for 100 steps, while GRPO and TTRL use 500 steps.
- Main results: TTPO exceeds label-dependent OPSD across all three model scales, reaching 40.1 versus 39.7 on 1.7B, 58.6 versus 58.4 on 4B, and 62.6 versus 61.7 on 8B average.TTPO uses majority-vote pseudo-labels, whereas OPSD uses ground-truth labels in the OpenThoughts setting.
- Main results: 45.2 average on Qwen3-1.7B is TTPO’s label-free TTT result, exceeding OPSD-TTT by 3.3 and TTRL by 5.4.This represents a 7.2-point absolute gain over the base model; TTPO reaches 61.1 average on Qwen3-4B versus 60.7 for the Qwen3-8B base model.
- Ablation studies: Both token-level selection mechanisms improve over uniform updates by focusing distillation on informative positions and masking negative tokens that can cause collateral damage or optimization noise.Removing positive weighting dilutes gradients, while removing negative masking penalizes locally correct steps and lets anomalous tokens dominate.
- Ablation studies: The full update strategy, positive FKL with negative GRPO, reaches 48.9 and substantially outperforms alternative positive-negative assignments.Positive-only FKL reaches 46.7, while the reversed positive-GRPO, negative-FKL assignment reaches 37.2.
- Generalization and analysis: Training on any one benchmark consistently improves performance on the other two, supporting cross-benchmark transfer rather than problem-specific overfitting.TTPO with pseudo-labels also outperforms both TTPO with ground truth and OPSD with leakage in the reported upper-bound comparison.
- Generalization and analysis: Avg@12 rises toward the base Maj@12 while Maj@12 rises in tandem during training, indicating self-evolution beyond the initial majority-vote ceiling.The authors interpret this pattern as collective knowledge from majority voting being distilled into single-sample performance.
5 CONCLUSION
TTPO combines OPSD and RL under an asymmetric objective for label-free test-time training. Its token-level updates match ground-truth-supervised methods, outperform existing label-free approaches, and show self-evolution with cross-task generalization.
- Conclusion: TTPO combines dense OPSD guidance on agreeing rollouts with robust RL penalties on disagreeing rollouts under an asymmetric objective.Token-level selection is applied to both branches.
- Conclusion: TTPO matches ground-truth-supervised methods, substantially outperforms existing label-free approaches, and exhibits self-evolution with strong cross-task generalization.
A IMPLEMENTATION DETAILS
The implementation uses thinking-mode prompts and illustrates token-selective updates that focus learning on uncertain reasoning and confident errors.
- Prompt templates: OPSD and TTPO use a thinking-mode-on teacher with a thinking-mode-off student, while training and evaluation configurations are documented separately.The prompt templates wrap problems in the model’s chat template; the teacher can receive a pseudo-label reference solution.
- Token weighting: Token weighting down-weights low-entropy, low-divergence positions where the student has already converged.In the geometry example, deterministic coordinate values receive near-zero weight, while geometric insights receive higher weight.
- Complementarity: The two mechanisms avoid low-signal gradients from opposite directions: weighting filters converged positive tokens, while masking filters non-actionable negative tokens.Together they focus updates on where the model still needs to learn and where it is confidently wrong.
- Token masking: Token masking suppresses locally correct or uncertain positions and retains anomalous, context-inconsistent errors for negative-sample penalties.The example masks correct arithmetic and keeps errors such as applying a vertical displacement to a horizontal segment.
D.2 NON-THINKING EVALUATION
Non-thinking evaluation tests whether a thinking-mode teacher transfers reasoning to inference without thinking; TTPO produces larger gains than OPSD across model scales.
- Non-thinking evaluation: +25.2, +30.6, and +36.4 points are TTPO’s average improvements over the base model at 1.7B, 4B, and 8B scales, respectively.The corresponding OPSD improvements are +7.1, +5.8, and +3.5 points.
- Non-thinking evaluation: TTPO achieves dramatically larger gains than OPSD across all evaluated scales when thinking mode is disabled.Both methods are trained with a thinking-mode-on teacher, so the comparison evaluates transfer to non-thinking inference.
D.3 ADDITIONAL ABLATIONS
Ablations examine branch balancing and rollout selection, finding that λ=0.1 and a fixed 50/50 positive-negative training subset are effective design choices.
- RL weight: λ=0.1 gives the best performance among the tested RL weights.The GRPO loss is roughly an order of magnitude larger than the OPSD forward-KL loss, so λ balances branch gradients.
- Ktrain composition: A fixed 50/50 positive-negative split outperforms random sampling and a dynamic fraction for Ktrain.The fixed split maintains stable contributions from both branches as the positive fraction changes.
- Ktrain selection: Selecting the shortest completions performs best among the evaluated Ktrain selection strategies.Only the first 1,024 tokens receive gradient updates, so shorter trajectories expose a larger share of their reasoning within the gradient window.
- Ktrain selection: The top-signal strategy underperforms shortest selection despite choosing high-divergence positive samples and high-log-probability negative samples.The result indicates that larger divergence or higher confidence does not reliably identify more useful gradients in this ablation.
E.2 FKL SIGNAL ANALYSIS
FKL is applied to agreeing rollouts, while GRPO handles disagreements, avoiding the pseudo-label conflict that can make distillation harmful on negative samples.
- FKL signal: For positive samples, FKL is benign at worst and recovers standard OPSD when the pseudo-label is correct.When the pseudo-label is wrong, the branch becomes answer-agnostic distillation rather than direct supervision toward that answer.
- FKL signal: For negative samples, disagreement creates a conflict term that can suppress correct reasoning when the pseudo-label is wrong.This is the central failure mode of applying FKL to disagreeing rollouts.
- Asymmetric routing: GRPO on negative samples requires only that a rollout is outside the majority cluster, making it label-agnostic and usually correct for negative samples.Pure FKL on all samples can propagate a wrong pseudo-label to every trajectory, whereas asymmetric routing limits corruption to agreeing samples.
- Routing: Ground-truth routing can make both branches vanish on hard problems with few correct rollouts, whereas vote routing keeps a non-empty consensus branch active.The pseudo-label is reported to converge toward the ground-truth answer as the model improves.
F LIMITATIONS AND FUTURE WORK
TTPO’s current scope is limited by majority-vote reliability, a fixed training balance, and evaluation confined to mathematical reasoning with verifiable answers.
- Dependence on majority-vote quality: Small sample budgets or problems without any correct rollout degrade majority-vote supervision for both TTPO branches.The paper suggests adjusting the positive-negative ratio or falling back to pure RL when consensus is low.
- Domain scope: TTPO has been evaluated only on mathematical reasoning tasks with automatically verifiable final answers.Applying it to code generation or open-ended reasoning remains unexplored because those domains require execution-based verification or learned reward models.
- Dynamic training curriculum: A fixed asymmetric objective may become suboptimal as pseudo-label accuracy improves during training.The paper proposes dynamically adjusting the RL weight or positive-negative fraction to match training dynamics.