Source-linked AI summary
T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling
Zhenyu Hou, Xin Lv, Rui Lu, Jiajie Zhang, Yujiang Li, Zijun Yao, Juanzi Li, Jie Tang, Yuxiao Dong
TL;DR
Existing imitation-focused approaches struggle with effective test-time scaling, while prior RL methods provide modest gains in complex reasoning. T1 addresses this by combining reasoning-rich CoT initialization with exploration-oriented RL scaling and stability controls. The resulting open-model systems achieve strong math-benchmark performance and show that larger inference budgets improve performance without additional verification.
Problem
Existing approaches rely mainly on imitation learning, while prior RL methods show modest gains and limited scalability for complex reasoning and test-time inference.
Method
T1 initializes models with synthesized trial-and-error and self-verification CoT data, then scales RL through diverse sampling, entropy encouragement, KL strategies, and penalties.
Results
T1 achieves strong performance across challenging reasoning benchmarks, including a 92.4 MATH500 score versus 90.6 for the previous best result, and shows inference scaling as budgets increase.
Takeaways & Limitations
Increasing RL training improves both reasoning accuracy and inference-scaling trends, allowing increased inference budgets to improve T1 performance without additional verification.
Abstract
from arXiv · showhide
Large language models (LLMs) have demonstrated remarkable capabilities in complex reasoning tasks. However, existing approaches mainly rely on imitation learning and struggle to achieve effective test-time scaling. While reinforcement learning (RL) holds promise for enabling self-exploration, recent attempts yield modest improvements in complex reasoning. In this paper, we present T1 to scale RL by encouraging exploration and understand inference scaling. We first initialize the LLM using synthesized chain-of-thought data that integrates trial-and-error and self-verification. To scale RL training, we promote increased sampling diversity through oversampling. We demonstrate that T1 with open LLMs as its base exhibits inference scaling behavior and achieves superior performance on challenging math reasoning benchmarks. More importantly, we present a simple strategy to examine inference scaling, where increased inference budgets directly lead to T1's better performance without any additional verification.
1. Introduction
T1 addresses limited RL progress and weak test-time scaling by encouraging exploration during training and controlling inference budgets. Built on open models, it improves mathematical reasoning and exhibits stronger inference scaling as RL training increases.
- Existing RL methods show modest gains in complex reasoning and limited scalability beyond earlier training stages.
- Repeated sampling with auxiliary verifiers does not update the policy model and fails to fundamentally improve reasoning ability.
- T1 initializes models with synthesized CoT data containing trial-and-error and self-verification to expand exploration before RL.
- T1 scales RL by oversampling diverse responses, adding entropy and KL strategies, and penalizing repetitive or nonsensical outputs.
- T1 measures inference scaling by separating intermediate reasoning from final answers and truncating reasoning to control inference budgets.
- T1 models outperform existing models across evaluated math benchmarks, including Qwen-32B outperforming QwQ-32B-Preview on MATH500, AIME2024, and Omni-MATH-500.
- T1 exhibits promising training and inference scaling, with performance improving as generation length and RL training increase.
2. Building T1 with RL Scaling
T1 combines reasoning-rich SFT with exploration-oriented RL scaling and stability controls. Its pipeline broadens sampled reasoning trajectories while penalizing undesirable outputs and regulating policy updates.
- RL pipeline: The RL pipeline samples multiple responses for each prompt, assigns rewards, and updates the policy to maximize the RL objective.
- Initializing reasoning patterns: SFT initializes the model with trial-and-error and verification patterns rather than only correct solution steps.
- Encouraging exploration: T1 expands RL exploration through hard sampling, high-temperature oversampling, response entropy bonuses, and relaxed KL regularization.
- Encouraging exploration: T1 generally samples K = 64 responses per prompt, compared with previous works commonly using K ≤8.
- Reward normalization: Leave-one-out reward normalization lets the policy learn from a richer set of experiences, improving generalization and reasoning performance.
- Stabilizing optimization: On-policy KL normalization and EMA reference updates keep policy changes incremental and controlled during RL training.
- Penalizing unexpected patterns: Negative reward −1 is assigned to repetitive, overlong, or garbage responses to prevent collapse and discourage undesirable patterns.
3. Evaluating T1
T1 improves performance across challenging math benchmarks, with reinforcement learning and broader response sampling providing substantial gains. Its training is sensitive to sampling temperature and benefits from penalties that stabilize generation.
- Performance Results: T1 evaluates open-model variants on AIME, Omni-MATH, MATH, and GPQA using Accuracy (Pass@1) as the primary metric.The benchmark suite includes college- and competition-level mathematical reasoning tasks.
- Performance Results: Over 10% improvement on Omni-MATH-500 and MATH500, and over 20% on AIME, is achieved by RL-trained T1 (Qwen2.5-32B) versus its T1-SFT version.These results show substantial gains from RL scaling over SFT.
- Performance Results: 92.4 Accuracy on MATH500 is achieved by T1 (Qwen2.5-32B), exceeding the previous best result of 90.6.T1 also reports competitive results on AIME compared with baselines.
- Performance Results: T1 improves out-of-domain GPQA performance despite training data consisting predominantly of mathematical content and almost no science-related data.The reported GPQA result uses no task-specific optimization during training.
- The Effect of Encouraging Exploration: More response samples during RL training increase exploration, lengthen responses, accelerate accuracy gains, improve reward at fixed KL divergence, and slow KL growth.Sampling more responses therefore yields a better KL–reward trade-off and improves MATH500 performance in fewer training steps.
- The Effect of Encouraging Exploration: Over 6% improvement is observed when increasing response samples to 64, whereas sampling only 4 responses provides little or no benefit at around 3%.Additional responses also significantly enhance generalization, including on GPQA.
- The Effect of Encouraging Exploration: Higher sampling temperatures support more stable RL training, while temperatures ≤1.0 often collapse after only a few steps.A moderately high temperature such as 1.2 performs better, but excessively high temperatures also hurt outcomes.
- The Effect of Encouraging Exploration: Applying penalties stabilizes generation length and reduces repetition and noise, whereas removing penalties causes explosive length growth after approximately 100 steps and deteriorates performance.The penalized model maintains appropriate generation lengths and consistent performance.
4. Understanding Inference Scaling in T1
T1 studies inference scaling by separating intermediate reasoning from final-answer generation and truncating responses to vary thinking budgets. Longer thinking improves benchmark performance, while stronger RL training makes this scaling behavior more pronounced and supports reflection-driven reasoning.
- Measuring inference scaling: T1 measures inference scaling by truncating single responses at different reasoning lengths and evaluating the resulting summarized answers.The method separates intermediate reasoning from final-answer generation, allowing inference cost to be varied through token budgets.
- Inference-budget effects: Across AIME, Omni-MATH-500, and MATH500, performance improves consistently as the average number of thinking tokens increases.On AIME, accuracy rises from 24% to 50% with more thinking tokens.
- RL and inference scaling: More RL training strengthens inference scaling, whereas the SFT model and under-trained RL policies show minimal gains from larger inference budgets.At 100% RL training, relative improvements are 66% on AIME, 30% on Omni-MATH-500, and 6% on MATH500.
- Reasoning behavior during RL: The policy generates increasingly longer responses during RL, and longer responses contain more diverse reasoning patterns associated with trial-and-error problem solving.The classified patterns include correcting mistakes, trying different approaches, and verification.
- Key reasoning steps: A reasoning trajectory can include uncertainty, an alternative approach, and final verification before the model becomes confident in its answer.The example illustrates how T1 explores and checks multiple reasoning paths within one response.
5. Related Work
Related work covers advances in language-model reasoning and scaling, including reasoning-oriented pretraining, imitation-based methods, and scaling through model size, training tokens, reinforcement learning, and test-time computation.
- Language model reasoning: Language-model reasoning has advanced through pretraining on extensive reasoning datasets, including mathematics and code.These datasets are described as improving foundational understanding of reasoning tasks.
- Scaling language models: Scaling research studies predictable improvements from increasing model size and training tokens, while RL and test-time scaling have attracted growing attention for reasoning.The related work situates reinforcement learning and inference-time computation within broader language-model scaling research.
6. Conclusion
The paper presents T1 as a scaled-reinforcement-learning approach for language-model reasoning and reports strong reasoning performance together with promising test-time scaling behavior. Its inference-scaling analysis links increased RL training with improved accuracy and stronger scaling trends.
- Conclusion: T1 enhances language-model reasoning through scaled reinforcement learning that promotes exploration while maintaining training stability through penalties and oversampling.The approach combines extensive exploration during RL with mechanisms intended to stabilize training.
- Conclusion: T1 demonstrates promising test-time scaling behavior and strong performance on challenging reasoning benchmarks, outperforming existing models.The conclusion reports these outcomes at the paper level without specifying individual benchmark values.
- Conclusion: The proposed analysis reveals that increased RL training improves both reasoning accuracy and inference-scaling trends.The relationship is examined through the connection between reasoning steps and model performance.
Impact Statement
The impact statement frames the work as an advance in language-model reasoning through reinforcement learning and inference scaling, with anticipated relevance to AI research and applications.
- Impact Statement: The paper positions its contribution as improving language-model reasoning capabilities through reinforcement learning and inference scaling.It anticipates positive impacts on language-model development and applications.
- Impact Statement: The authors state that the ethical and societal implications align with those commonly associated with advancing machine learning.The statement does not identify paper-specific ethical risks or benefits.
A.1. Setup
T1 is evaluated on reasoning datasets using Pass@1 accuracy, with training data drawn from public math datasets and split between SFT and RL. The setup also describes the box-solving example used to illustrate T1’s reasoning process.
- Data construction: Training prompts come from publicly available datasets, including MATH-train and NuminaMath, with approximately 12k examples allocated to SFT and the remainder to RL.The RL data are converted into Question–Label pairs through answer extraction and related preprocessing.
- Illustrative reasoning example: T1’s example response explores multiple solution approaches and uses verification strategies to check the final answer.The example concerns AIME2024 Question-15 and is presented as a full worked response.
- Training details: SFT uses three epochs with a 1e-5 learning rate, while RL samples 64 responses per prompt and updates after every 32 prompts.RL training uses a 1.5e-6 learning rate; the supplied passage truncates the remaining training details.
- Training details: Training and inference maximum generation lengths are 10,240 tokens for GLM-4-9B and Qwen2.5-14B, and 16,384 for Qwen2.5-32B.
- Evaluation: Evaluation covers AIME2024, Omni-MATH, MATH, and GPQA, using Accuracy (Pass@1) as the primary metric.MATH is evaluated on the predefined MATH500 subset.
- Illustrative reasoning example: The worked solution concludes that a = b = 2 and c = 23/4 satisfies both constraints and gives the minimal r^2 = 657/64.It reports the final answer p + q = 721 and states that alternative dimensions produce a larger r^2 or violate the constraints.
- Illustrative reasoning example: The quartic-solving portion uses factorization and Vieta’s formulas to establish that the product of the roots is 1736.The response explicitly presents Vieta’s formulas as a more straightforward alternative to involved factoring calculations.