PaperlayerView exact revision on arXiv

Source-linked AI summary

TTRL: Test-Time Reinforcement Learning

Yuxin Zuo, Kaiyan Zhang, Li Sheng, Shang Qu, Ganqu Cui, Xuekai Zhu, Haozhan Li, Yuchen Zhang, Xinwei Long, Ermo Hua, Biqing Qi, Youbang Sun, Zhiyuan Ma, Lifan Yuan, Ning Ding, Bowen Zhou

arXiv:2504.16084v3cs.CLcs.LG

TL;DR

RL for reasoning models typically requires labeled data, but test-time learning on emerging unlabeled tasks requires rewards without ground-truth answers. TTRL uses majority-voting rewards from repeated model samples to train models during inference, improving performance across tasks and models, including a 211% AIME 2024 gain for Qwen2.5-Math-7B.

  • Problem

    TTRL addresses RL training on unlabeled test data, where reward estimation is required without ground-truth labels and existing RL methods rely heavily on labeled data.

  • Method

    TTRL performs test-time RL by repeatedly sampling outputs, using majority voting to estimate labels, and converting consensus into rule-based rewards.

  • Results

    211% improvement on AIME 2024 raises Qwen2.5-Math-7B from 12.9 to 40.2, while TTRL also consistently improves performance across models and tasks.

  • Takeaways & Limitations

    TTRL suggests that self-labeled rewards can support continual learning and reduce reliance on human annotations for scaling RL.

  • Takeaways & Limitations

    The paper leaves the effects of prior knowledge and hyperparameter configurations for deeper analysis and future ablation studies.

Abstract

from arXiv · show

This paper investigates Reinforcement Learning (RL) on data without explicit labels for reasoning tasks in Large Language Models (LLMs). The core challenge of the problem is reward estimation during inference while not having access to ground-truth information. While this setting appears elusive, we find that common practices in Test-Time Scaling (TTS), such as majority voting, yield surprisingly effective rewards suitable for driving RL training. In this work, we introduce Test-Time Reinforcement Learning (TTRL), a novel method for training LLMs using RL on unlabeled data. TTRL enables self-evolution of LLMs by utilizing the priors in the pre-trained models. Our experiments demonstrate that TTRL consistently improves performance across a variety of tasks and models. Notably, TTRL boosts the pass@1 performance of Qwen-2.5-Math-7B by approximately 211% on the AIME 2024 with only unlabeled test data. Furthermore, although TTRL is only supervised by the maj@n metric, TTRL has demonstrated performance to consistently surpass the upper limit of the initial model maj@n, and approach the performance of models trained directly on test data with ground-truth labels. Our experimental findings validate the general effectiveness of TTRL across various tasks and highlight TTRL's potential for broader tasks and domains. GitHub: https://github.com/PRIME-RL/TTRL

1 Introduction

TTRL addresses the challenge of applying RL to increasingly complex, unlabeled tasks by estimating rewards from model-generated outputs. It improves reasoning performance without labeled training data and can surpass majority-voting limits.

  • Motivation: RL methods for long-chain reasoning commonly depend on expensive human-annotated data, while emerging unlabeled tasks remain difficult.The cited examples contrast o3's 75.7% success on ARC-AGI-1 with 4% on ARC-AGI-2.
  • Approach: TTRL performs test-time RL on unlabeled data by generating experiences, estimating rewards, and improving the model over time.Its reward estimation uses repeated sampling and majority voting to create rule-based rewards.
  • Results: 211% improvement on AIME 2024 lifts Qwen2.5-Math-7B from 12.9 to 40.2, with a 76% average gain across four tasks.The gains use no labeled training data and further generalize to other tasks.
  • Results: Majority voting provides effective reward estimation for TTRL.This is identified as a central finding of the work.
  • Results: TTRL can exceed the initial model's maj@n upper limit and closely mirror direct training on test data with ground-truth labels.The comparison uses direct training as an upper bound that assumes access to ground-truth labels.
  • Results: TTRL enables efficient and stable RL in an unsupervised manner.The paper presents this as a key experimental takeaway.

2 Test-Time Reinforcement Learning (TTRL)

TTRL trains a pretrained model during inference on unlabeled test data by converting consensus among sampled outputs into rewards. The method combines repeated sampling, answer extraction, majority voting, and RL updates.

  • Setting: TTRL studies training a pretrained model during test time using RL without ground-truth labels.The paper names this setting Test-Time Reinforcement Learning.
  • Overview: TTRL combines Test-Time Scaling with Test-Time Training.Figure 2 identifies both components as part of the approach.
  • Methodology: Given prompt x, the policy samples outputs, repeated sampling produces candidate answers, and consensus provides a proxy label for reward construction.The environment evaluates alignment between a sampled output and the consensus output.
  • Methodology: The model parameters are updated through gradient ascent, enabling adaptation during inference without labeled data.The learning rate is denoted by η.
  • Majority Voting Reward: The majority-voting reward function extracts answers from outputs and assigns rewards based on agreement with the majority answer.The pseudocode uses answer extraction, counting, and a reward of 1 for matching outputs and 0 otherwise.
  • Majority Voting Reward: The reward pipeline generates outputs, extracts predicted answers P, estimates the most frequent prediction, and uses it as the label for rule-based rewards.The majority-voted prediction is computed using a scoring function over the extracted predictions.

3 Experiments

Experiments evaluate TTRL across diverse models and reasoning benchmarks, finding consistent gains, generalization beyond target tasks, compatibility with multiple RL algorithms, and improving training dynamics.

  • Main Results: TTRL improves performance across six models, four model families, two model types, three model sizes, and four challenging benchmarks.On AIME 2024, every evaluated model improves by at least 105%; a 1.5B model gains up to 40.3 points on MATH-500.
  • Main Results: Approximately 10-point gains on both evaluated reasoning models show that TTRL remains effective after extensive post-training.The evaluation includes additional large reasoning models, including Qwen3-8B in thinking mode.
  • Main Results: As model size increases from 1.5B to 7B and 7B to 32B, TTRL performance consistently improves.Larger models produce more accurate majority-voting rewards during self-improvement, supporting more effective learning on new data.
  • Main Results: TTRL produces substantial out-of-distribution improvements across all evaluated benchmarks after training on each target benchmark individually.The authors interpret this pattern as evidence of generalizable gains rather than overfitting trade-offs.
  • Main Results: Performance trajectories for GRPO, PPO, and PRIME on MATH-500 are closely aligned, indicating TTRL compatibility with different RL algorithms.The comparison covers GRPO, value-based PPO, and process-level PRIME.
  • Main Results: During AMC training, both pass@1 and maj@16 increase consistently, while TTRL converts voting-based pseudo-labels into reward signals.This dynamic improves supervision quality and decouples learning from the limitations of maj@n.

4 Analysis and Discussions

TTRL can surpass majority-voting supervision and approach labeled-data training, while its effectiveness depends on reward robustness, online learning, and sufficient priors; hyperparameter choices remain a failure point.

  • Performance limits: TTRL surpasses the initial model’s maj@n and can approach direct RL trained with ground-truth labels.These define the two empirical upper bounds used to assess TTRL.
  • Performance limits: TTRL can reach the empirical upper bound on challenging tasks using a 1.5B model.The result is presented as evidence for self-evolution through TTRL.
  • Why TTRL works: Reward accuracy can reach 92% despite 37% label accuracy because scattered incorrect outputs often receive correct verifier rewards through “Lucky Hit”.Multiple outputs make rewards denser and more robust than a single rollout output.
  • Failure conditions: TTRL inherits RL sensitivity to data difficulty, reliance on priors, and potential collapse, with reward noise and sparse unseen test data amplifying risks.The section identifies inappropriate hyperparameters and insufficient target-task knowledge as potential failure sources.
  • Failure conditions: As MATH-500 difficulty increases, performance improvement and length reduction ratios tend to decrease for Qwen2.5-Math-1.5B.The authors attribute this pattern to insufficient backbone prior knowledge for harder questions.

5 Related Works

Related work covers test-time scaling, test-time training, reinforcement learning for language models, and prior self-rewarding or self-play approaches on unlabeled data.

  • Test-Time Scaling: Test-time scaling increases inference computation through parallel or sequential generation and aggregates candidates using methods such as process reward models.Parallel approaches include self-consistency, best-of-N, tree search, and reward-guided search.
  • Test-Time Scaling: Rule-based rewards are more scalable than approaches requiring intensive process-level supervision.The comparison concerns two test-time scaling approaches described in the related work.
  • Test-Time Training: Prior test-time training work mainly studied video generation, video understanding, and some large-language-model applications.The paper distinguishes these studies from its integration of test-time training with reinforcement learning.
  • Reinforcement Learning: RL has been used to improve LLM instruction following through RLHF, PPO, and preference modeling, and has recently supported long-chain reasoning models.The related work positions TTRL within this broader RL literature.
  • Unlabeled-data learning: Earlier self-rewarding and self-play methods primarily target open-domain instruction following or use preference optimization rather than online RL for mathematical reasoning.The paper identifies this distinction as a gap addressed by TTRL.

6 Conclusion

The paper proposes TTRL, which trains language models with reinforcement learning on test data without ground-truth labels, using majority-voting rewards and showing consistent gains across models and tasks.

  • Conclusion: TTRL trains large language models with RL on test data without access to ground-truth labels.Its majority-voting reward function generates rule-based rewards from consensus among model predictions.
  • Conclusion: The experiments report consistent improvements across a variety of models and tasks.The authors frame TTRL as an initial step toward learning from continuous streams of experience.

7 Limitations and Future Works

The paper identifies prior knowledge and hyperparameter configuration as important unresolved limitations, and proposes theoretical, streaming-data, scaling, and broader-task extensions.

  • Limitations: The work is an initial exploration of test-time RL with self-labeled rewards, requiring deeper analysis of prior knowledge and hyperparameter effects.The authors identify both factors as critical to reinforcement-learning dynamics.
  • Future Works: Future work includes formal convergence analysis, especially regarding TTRL’s optimization toward two empirical upper bounds.This targets the theoretical basis of the reported performance limits.
  • Future Works: The authors propose extending TTRL to streaming data and real-time test-time adaptation.The goal is dynamic adaptation to continuously arriving data.
  • Future Works: Planned applications include large-scale self-supervised RL and more complex agentic or multi-step scientific-reasoning tasks.These directions test TTRL beyond the current evaluation scope.

A Additional Results

Additional results evaluate TTRL across reasoning models and benchmarks, reporting substantial gains and comparisons with other trained models.

  • Table 4 reports greedy-decoding pass@1 results for base and instruct models, alongside leading R1-Zero-Like models with similar backbones.
  • 159.3% improvement on AIME 2024 surpasses all models trained on large-scale datasets.
  • TTRL improves Qwen2.5-Math-7B by an average of 84.1% across three benchmarks.
  • Figure 12 presents entropy and accuracy curves for TTRL on AIME 2024 with Qwen2.5-Math-7B.

B Training Metrics

TTRL uses label-free training-time metrics to monitor optimization, while separate ground-truth-dependent metrics support deeper behavioral analysis.

  • Training-time metrics address the challenge of evaluating TTRL when test data lack ground-truth labels and guide checkpoint selection.
  • Entropy measures uncertainty in the model’s generation.
  • Majority Voting Reward computes rule-based rewards from the majority-voted label.
  • Majority Ratio measures the frequency of the most common answer within a rollout.
  • Ground-truth-dependent metrics include Label Accuracy, Reward Accuracy, and Ground-Truth Ratio for deeper training analysis.Label Accuracy compares estimated and ground-truth labels; Reward Accuracy compares majority-voting and ground-truth rewards; Ground-Truth Ratio counts ground-truth answers in rollouts.

C Terminology

Test-time scaling increases computational resources during test time and comprises complementary test-time training and test-time inference approaches.

  • Test-time scaling refers to increasing computational resources during test time.
  • Test-time scaling is categorized into test-time training and test-time inference, which are complementary approaches.

C.1 Test-Time Training (TTT)

Test-time training adapts a pretrained model during inference without target labels, while test-time inference generates and selects multiple candidate outputs using scoring strategies.

  • Test-Time Training (TTT): Test-Time Training adapts a pretrained model at inference time to improve generalization under distribution shifts.
  • Test-Time Training (TTT): TTT updates model parameters online by minimizing a label-free auxiliary loss without access to test labels.
  • Test-Time Inference (TTI): Test-Time Inference enhances inference performance by allocating additional computation to generate candidate outputs and select among them.
  • Test-Time Inference (TTI): The selection framework generates N candidates and chooses y* using a scoring function s(y, x).
  • Test-Time Inference (TTI): Scoring strategies include Majority Voting, Best-of-N with reward models, and Weighted BoN combining both approaches.
Loading 2504.16084v3…