PaperlayerView exact revision on arXiv

Source-linked AI summary

rStar2-Agent: Agentic Reasoning Technical Report

Ning Shang, Yifei Liu, Yi Zhu, Li Lyna Zhang, Weijiang Xu, Xinyu Guan, Buze Zhang, Bingcheng Dong, Xudong Zhou, Bowen Zhang, Ying Xin, Ziming Miao, Scarlett Li, Fan Yang, Mao Yang

arXiv:2508.20722v1cs.CL

TL;DR

Long chain-of-thought can struggle with subtle intermediate errors, motivating methods that use tools to validate reasoning. rStar2-Agent combines agentic RL infrastructure, GRPO-RoC, and a staged training recipe; its 14B model reports strong math performance, surpasses 671B DeepSeek-R1, and generalizes beyond mathematics.

  • Problem

    Long chain-of-thought can fail to detect subtle intermediate errors or correct flawed approaches on hard reasoning problems.

    Source: §1 · p. 3

  • Method

    rStar2-Agent trains a 14B model with agentic RL using a Python environment, GRPO-RoC resampling, and a staged recipe beginning with non-reasoning SFT.

    Source: §1 · p. 3, §2.2.3 · p. 7

  • Results

    80.6% pass@1 on AIME24 and 69.8% on AIME25 are reported for the 14B model, which surpasses 671B DeepSeek-R1 while producing shorter responses and generalizing beyond mathematics.

    Source: §6 · p. 19

  • Takeaways & Limitations

    The report presents agentic RL as a way for a 14B model to achieve frontier-level math reasoning and generalize to tasks beyond mathematics.

    Source: §1 · p. 3, §6 · p. 19

    On the 14B model, continued RL after peak accuracy at step 510 led to collapse in both policy and reward signals.

    Source: §5.3 · p. 16

Abstract

from arXiv · show

We introduce rStar2-Agent, a 14B math reasoning model trained with agentic reinforcement learning to achieve frontier-level performance. Beyond current long CoT, the model demonstrates advanced cognitive behaviors, such as thinking carefully before using Python coding tools and reflecting on code execution feedback to autonomously explore, verify, and refine intermediate steps in complex problem-solving. This capability is enabled through three key innovations that makes agentic RL effective at scale: (i) an efficient RL infrastructure with a reliable Python code environment that supports high-throughput execution and mitigates the high rollout costs, enabling training on limited GPU resources (64 MI300X GPUs); (ii) GRPO-RoC, an agentic RL algorithm with a Resample-on-Correct rollout strategy that addresses the inherent environment noises from coding tools, allowing the model to reason more effectively in a code environment; (iii) An efficient agent training recipe that starts with non-reasoning SFT and progresses through multi-RL stages, yielding advanced cognitive abilities with minimal compute cost. To this end, rStar2-Agent boosts a pre-trained 14B model to state of the art in only 510 RL steps within one week, achieving average pass@1 scores of 80.6% on AIME24 and 69.8% on AIME25, surpassing DeepSeek-R1 (671B) with significantly shorter responses. Beyond mathematics, rStar2-Agent-14B also demonstrates strong generalization to alignment, scientific reasoning, and agentic tool-use tasks. Code and training recipes are available at https://github.com/microsoft/rStar.

1 Introduction

Long chain-of-thought can fail on problems with subtle intermediate errors, motivating rStar2-Agent’s use of Python tools for exploration and verification. The paper combines scalable infrastructure, GRPO-RoC, and a staged training recipe; its 14B model reports strong math results and broader generalization.

1 Introduction

  • Long chain-of-thought can miss subtle intermediate errors or fail to correct a flawed initial approach on hard problems.Python tools can support alternative-solution exploration and intermediate-step verification, complementing self-reflection.Source: §1 · p. 3
  • Coding-tool errors can waste reasoning tokens, while outcome-only rewards may still reward trajectories with failed intermediate calls if the final answer is correct.These noisy rollouts and the high volume of concurrent tool calls also make agentic RL infrastructure and training slower to scale.Source: §1 · p. 3
  • A high-throughput isolated code environment handles 45K concurrent calls with average feedback latency of 0.3 seconds, supporting training on 64 MI300X GPUs in one week.A load-balanced scheduler allocates rollouts according to available GPU KV cache capacity.Source: §1 · p. 3
  • GRPO-RoC oversamples rollouts, keeps higher-quality positive trajectories, and uniformly downsamples negatives to address tool-induced noise under outcome-only rewards.The approach aims to preserve diverse failure signals while emphasizing cleaner positive supervision.Source: §1 · p. 3
  • The training recipe begins with non-reasoning SFT and then uses multi-stage RL with gradually increased difficulty and response lengths, reaching frontier-level math reasoning in 510 RL steps.The SFT stage targets instruction following, coding-tool use, and formatting rather than reasoning enhancement.Source: §1 · pp. 3–4
  • 80.6% AIME24 accuracy and 69.8% AIME25 accuracy are reported for rStar2-Agent-14B, which surpasses DeepSeek-R1 and generalizes beyond mathematics.The report also describes strong results on science reasoning and agentic tool-use tasks.Source: §1 · p. 4

2 Agentic Reinforcement Learning Methodology

The methodology trains reasoning through interactive Python use, then applies GRPO-RoC to address environment noise while retaining answer-only rewards. The authors report fewer tool errors, improved reasoning, and shorter responses with GRPO-RoC.

2.1 Smarter Reasoning in a Code Environment

  • rStar2-Agent trains native Python use through multi-turn rollouts, executing structured calls and returning outputs to context until an answer or turn limit.The model continues reasoning from the environment output after each tool call.Source: Figure 2 · p. 4, §2.1 · pp. 4–5

2.2.1 Preliminary: GRPO

  • GRPO updates the policy from grouped trajectories using verifier-assigned binary final-answer rewards; its recipe removes the KL penalty, relaxes upper clipping, and drops entropy loss.The authors say removing the KL penalty permits freer exploration, while the entropy loss is removed to prevent training instability.Source: §2.2.1 · p. 6

2.2.3 GRPO-RoC: Group Relative Policy Optimization with Resampling on Correct

  • GRPO-RoC reduces coding-tool errors in positively rewarded trajectories and is associated with improved reasoning and shorter responses.Figure 4 shows tool errors continuing to decline with training under GRPO-RoC, unlike the plateau under naive GRPO.Source: Figure 4 · p. 7, §2.2.3 · p. 8

2.2.2 Challenges in Agentic Reinforcement Learning

  • Outcome-only rewards can reinforce erroneous intermediate calls in correct trajectories; naive-GRPO error rates plateau near 15% for Qwen2.5-32B and 10% for Qwen3-14B.Such noisy tool feedback can divert tokens from reasoning to fixing tool errors.Source: §2.2.2 · pp. 6–7

2.2.3 GRPO-RoC: Group Relative Policy Optimization with Resampling on Correct

  • The authors retain answer-only rewards rather than step-level rewards or tool-error penalties, citing added complexity, reward-hacking risk, and possible constraint on early exploration.The proposed rollout filtering addresses noisy trajectories without changing the reward to penalize intermediate tool errors.Source: §2.2.3 · p. 7
  • RoC oversamples then downsamples: it favors successful traces with fewer tool or formatting errors while uniformly sampling failures to retain diverse learning signals.This asymmetric selection reinforces cleaner positive traces without discarding varied failure examples.Source: §2.2.3 · p. 7

3 Large-Scale Agentic RL Infrastructure

The infrastructure combines an isolated, high-throughput code environment with dynamic rollout scheduling to support large-scale agentic reinforcement learning. The scheduler addresses the workload imbalance and KV-cache overflow that make static allocation inefficient.

3.1 Reliable High-Throughput Code Environment

  • The code environment handles up to 45K concurrent tool calls per step, with 0.3-second average latency from issuing a call to receiving its result.The service is designed to support large-scale training without becoming a bottleneck.Source: Figure 6 · p. 9, §3.1 · p. 10
  • The isolated environment service offloads answer verification to run asynchronously, preventing CPU-intensive checks from stalling rollouts.The service is designed to isolate execution from the main RL training process while supporting concurrent tool calls.Source: §3.1 · p. 10

3.2 Load-Balanced Rollout Scheduler

  • Static allocation creates GPU idle time and synchronization delays as multi-turn rollouts vary in length and tool-call timing.KV-cache overflow can evict in-progress rollouts, requiring recomputation and wasting completed work.Source: §3.2 · p. 11
  • The dynamic scheduler assigns rollouts according to available KV-cache capacity, dispatches tool calls asynchronously, and assigns new work as capacity frees.This design aims to balance GPU workloads while avoiding KV-cache overflow and recomputation.Source: §3.2 · p. 11

4 Training Recipe

The training recipe pairs non-reasoning instruction-following and tool-use SFT with curated integer-answer math problems and progressively staged RL. It reaches frontier-level mathematical reasoning in 510 steps on 64 MI300X GPUs within one week.

4.1 Non-Reasoning Cold Start for Instruction Following

  • Non-reasoning SFT improves tool use, instruction following, and chat while maintaining comparable math performance to the base model.The recipe then uses multi-stage RL with progressively increasing training lengths.Source: Table 1 · p. 11, §4 · p. 11, §4.1 · p. 12

4.2 RL Data Curation

  • The RL dataset contains 42K cleaned, verifiable problems, selected for integer answers to reduce reward errors from difficult-to-verify equivalent expressions.Problems with unverifiable or incorrect answers and excessively large numerical answers were removed.Source: §4.2 · p. 12

4.3 Multi-Stage RL Training

  • Stage 1 retains an 8K response limit as average responses rise from around 1K to about 4K tokens; clipping later decreases as evaluation scores improve.The clipping ratio temporarily exceeded 10% multiple times, but the model self-adjusted while training continued at the 8K limit.Source: §4.3 · p. 12
  • Extending the response limit to 12K in Stage 2 raises average response length from 4K to 6K and improves AIME24 and AIME25 scores.Training then focuses on harder problems after more than 70% of a batch is rejected for having a perfect pass rate.Source: §4.3 · p. 13
  • Stage 3 filters out problems solved in all eight sampled rollouts, yielding 17.3K harder problems for focused training.Over 125 additional steps, average response length increases from 6K to 8K as performance improves.Source: §4.3 · p. 13
  • 510 RL steps across three stages advanced the 14B model to frontier-level mathematical reasoning, using 64 MI300X GPUs within one week.Performance began to saturate and could decline beyond the final stage’s 125 steps, so training stopped there.Source: §4.3 · p. 13

4.3.1 Unsuccessful Attempts And Lessons

  • Overlong filtering increased overlong rollouts, while retaining truncated rollouts with negative rewards helped guide the model to reduce repetition.N-gram repetition detection also hurt response length and reasoning scores because it could flag similar tool calls with different inputs that verify results.Source: §4.3.1 · p. 13
  • The recipe uses final-answer correctness as its reward and handles environment noise or formatting issues through resample-on-correct sampling rather than direct penalties.The authors caution that complex rule-based scoring can bias learning against useful intermediate behaviors.Source: §4.3.1 · p. 13

5 Experiments

Experiments test rStar2-Agent across math and other domains, compare agentic RL with alternative methods, and examine the training limits and behaviors associated with tool-based reasoning.

5.1 Setup

  • Experiments evaluate math, science reasoning, tool use, and alignment, with math and GPQA-Diamond sampled 16 times and reported as average pass@1 accuracy and response length.The 14B model uses non-reasoning SFT before RL; experiments use 64 AMD MI300X GPUs. The Qwen2.5-32B model runs only RL stages 1 and 2 due to resource limits.Source: §5.1 · pp. 13–14

5.2 rStar2-Agent-14B Main Results

  • rStar2-Agent-14B scores 80.6% on AIME24, outperforming DeepSeek-R1 by 0.8%, and reaches 69.8% on AIME25, showing strong performance across both benchmarks.On AIME24 and AIME25, it generates shorter responses than DeepSeek-R1-Zero, QWQ-32B, and official Qwen3-14B while attaining higher reasoning accuracy.Source: §5.2 · pp. 14–15
  • AIME24 rises from 3.3% after SFT to 72.1% in Stage 1, 77.0% in Stage 2, and 80.6% in Stage 3; AIME25 rises from 0% to 64.2%, 64.8%, and 69.8%.Stage 1 uses an 8k maximum response length and Stage 2 a 12k maximum; the paper reports Stage 1 surpasses the CoT-only DAPO baseline by 21.7% on AIME24 and 21.3% on AIME25.Source: Figure 8 · p. 14, §5.2 · p. 15
  • After math-only agentic RL, GPQA-Diamond accuracy rises from 42.1% to 60.9%, while non-reasoning tool-use and alignment performance remains comparable to the non-reasoning SFT baseline.The paper reports the model surpasses DeepSeek-V3 by 1.8% on GPQA-Diamond, but shows no improvement on the non-reasoning tool-use and alignment tasks.Source: §5.2 · p. 15, Table 5 · p. 15

5.3 Ablation Study and Discussions

  • On Qwen2.5-32B, rStar2-Agent exceeds ReTool by 2.4% on AIME24 and 8.0% on AIME25 despite using non-reasoning rather than reasoning-specific SFT.The comparison runs only RL stages 1 and 2; the authors expect continued Stage 3 training to improve performance further.Source: §5.3 · p. 16, Table 6 · p. 16
  • GRPO-RoC consistently improves AIME24 and AIME25 accuracy and reduces average training response length compared with GRPO with Tool.The authors attribute vanilla agentic RL's longer, lower-quality rollouts to ignoring environment noise; RoC prioritizes higher-quality positive rollouts.Source: §5.3 · p. 16, Figure 9 · p. 16
  • After reaching peak accuracy at step 510, continued Stage 3 RL training collapses both policy and reward signals.The authors explored changes to sampling temperature, maximum response length, tool interactions, and clipping, among other fixes.Source: §5.3 · p. 16

5.4 Analysis of Agentic Reasoning Behaviors

  • High-entropy tokens mark both exploration and self-reflection and, after code feedback, help the model diagnose errors, test alternatives, and refine its reasoning.In 64 sampled trajectories, coding-tool call tokens were usually low-entropy; whether this pattern generalizes to non-coding tools remains open.Source: Figure 10 · p. 17, Figure 11 · p. 18, §5.4 · p. 19

6 Conclusion

The conclusion presents rStar2-Agent as a 14B model that uses agentic reinforcement learning to reach performance comparable to 671B DeepSeek-R1. It highlights efficient training, strong math results with shorter responses, and reflection on tool feedback.

6 Conclusion

  • rStar2-Agent uses GRPO-RoC, scalable RL infrastructure, and a compute-efficient training recipe to train a 14B reasoning model through agentic reinforcement learning.The conclusion reports 80.6% pass@1 on AIME24 and 69.8% on AIME25, surpassing DeepSeek-R1 while producing shorter responses and generalizing beyond mathematics.Source: §6 · p. 19
  • Analysis finds that reflection on tool responses introduces exploration, self-reflection, and error correction in agentic reasoning.The authors plan to extend rStar2-Agent to broader reasoning domains and valuable environments.Source: §6 · p. 19
  • The authors publicly release the code, datasets, and recipes to support further research and collaboration.Source: §6 · p. 19
Loading 2508.20722v1…