Source-linked AI summary
Agent-R1: A Unified and Modular Framework for Agentic Reinforcement Learning
Mingyue Cheng, Shuo Yu, Daoyu Wang, Qingchuan Li, Xiaoyu Tao, Jie Ouyang, Yucong Luo, Yitong Zhou, Qi Liu, Enhong Chen
TL;DR
Agentic RL needs representations and training loops that preserve multi-round interaction with tools and environments without forcing trajectories into one growing token sequence. Agent-R1 uses step-level MDP transitions, structured trajectories, flexible context management, and modular interfaces; its results show support for varied tasks, optimizers, and context strategies, while its main limitations concern training cost and rollout-training synchronization.
Problem
Agentic RL makes single-response abstractions inadequate because multi-round interaction introduces long-horizon behavior, external feedback, sparse rewards, long contexts, and rollout-training representation mismatch.
Method
Agent-R1 uses step-level MDP transitions, structured step-level trajectories, flexible context construction, and unified interfaces across workflows, environments, infrastructure, and optimization.
Results
Across arithmetic, retrieval, embodied, and shopping settings, all four evaluated RL methods outperform the training-free baseline, with the best optimizer varying by task; context strategy also changes GSM8K performance.
Takeaways & Limitations
The framework provides a reusable substrate for heterogeneous agent environments and supports meaningful differences among optimization algorithms and context-management policies.
Takeaways & Limitations
Step-level traces can cause redundant computation from repeated long prefixes, and aligned rollout-training updates can bottleneck complex agents with slow or branching interactions.
Abstract
from arXiv · showhide
Large language models (LLMs) have rapidly evolved from single-turn text generators into the foundation of increasingly capable agents. As these agents take on more complex reasoning, decision making, tool use, and long-horizon tasks, reinforcement learning (RL) is becoming increasingly important for shaping their behavior. This shift is especially visible in agentic RL, where models must interact with tools and environments across multiple rounds rather than produce a single standalone response. In this regime, the usual view of a trajectory as one ever-growing token sequence becomes increasingly inadequate: it makes context evolution rigid and creates representation mismatches between rollout and training. This paper presents Agent-R1, a unified and modular framework for agentic RL built around step-level trajectory representation, flexible context management, and layered interfaces for workflows, environments and optimization. The key idea is to treat each interaction step as the basic reinforcement-learning transition, while keeping the optimization layer flexible: once the interaction is modeled at the step level, the framework can support token-level credit assignment, step-level credit assignment, or other compatible designs. These design choices make the framework compatible with a range of optimization strategies rather than tying it to a single algorithm. Together, these components provide a principled, extensible, and reusable substrate for agentic RL.
1 Introduction
Agentic RL extends LLM reinforcement learning to multi-round tool and environment interaction, making single growing token sequences inadequate. Agent-R1 addresses this with explicit step-level trajectories and flexible context management in a unified training substrate.
- Motivation: Agentic RL introduces multi-round interaction, delayed or sparse rewards, long horizons, and long contexts that make a single ever-growing token sequence inadequate.The model must adapt to external feedback over time rather than produce only one response or short reasoning trace.
- Core design: Agent-R1 models each interaction round as an RL transition and preserves rollout structure through structured step-level trajectory representation.This addresses the mismatch between message-based rollout storage and later text reconstruction, including possible retokenization drift.
- Core design: Agent-R1 combines explicit interaction units with explicit context-construction rules, supporting task-dependent memory policies while preserving rollout structure.Table 1 identifies this combination as the framework’s main distinction from representative agentic RL systems.
2 Preliminaries
Agentic RL requires an end-to-end loop connecting inference, tools, environments, trajectory replay, and optimization across multiple interaction steps. Agent-R1 formalizes this loop with step-level MDP transitions and explicit context construction.
- 2.2 From Supervised LLM Training to RL Training: LLM RL differs from supervised training because optimization depends on rollout, reward or preference evaluation, replay, and coordination between inference and training.Agentic RL extends this loop with environment feedback and multi-step trajectory construction.
- 2.3 From Single-Turn LLM RL to Agentic RL: Agentic RL connects multi-round rollout with trajectory replay and policy optimization while preserving observation, action, feedback, and termination across steps.Unlike single-turn RL, the framework must coordinate tools and environments throughout the interactive trace.
- 2.4 From LLMs to Agents: An MDP Perspective: Agent-R1 treats each interaction round as the native action unit, representing observations, actions, feedback, and context evolution as structured step-level traces.An action may be a natural-language response, a tool invocation, or a mixed output containing reasoning and external actions.
- 2.4 From LLMs to Agents: An MDP Perspective: The step-level formulation supports both terminal outcome rewards and intermediate process feedback within agent-environment transitions.This distinguishes the formulation from ordinary single-turn RL.
- 2.4 From LLMs to Agents: An MDP Perspective: Agent-R1 lets the environment construct the next observation with an explicit context rule rather than requiring append-only concatenation.The rule can preserve or transform structured interaction history while remaining within a defined transition.
3 Agent-R1
Agent-R1 unifies agentic RL around structured interaction steps, preserving replay fidelity while allowing flexible context construction and multiple optimization views.
- 3 Agent-R1: Agent-R1 connects agentic RL algorithms with workflow execution, rollout sampling, model serving, and large-scale optimization through a common training substrate.The framework is designed to bridge algorithmic and infrastructure concerns rather than reimplementing the execution stack for each method.
- 3.2 Step-level Trajectory Representation: Message-based replay can mismatch rollout tokens after text reconstruction, whereas flat token storage preserves tokens but leaves step boundaries implicit.Such mismatch can shift action boundaries, alter masks, and distort optimization log-probabilities.
- 3.2 Step-level Trajectory Representation: Structured step-level traces preserve rollout interaction boundaries, feedback, and complete actions while supporting faithful replay.Each step records the interaction structure needed to align actions, feedback, rewards, and optimization signals.
- 3.2 Step-level Trajectory Representation: Step-level traces support both token-level objectives over action tokens and step-level rewards or process supervision on corresponding transitions.Credit can be assigned to internal action tokens or broadcast from a step-level signal.
- 3.3 Flexible Context Management: Environment-specific context rules construct the next observation from structured history, allowing traces to be preserved, summarized, omitted, or transformed.The same step-level replay record can support different context constructions instead of requiring blind append-only concatenation.
4 Experiments
Agent-R1 transfers across heterogeneous agent tasks while preserving meaningful differences among optimizers and context-management strategies. Controlled GSM8K comparisons show that optimizer choice affects reward, accuracy, response length, and context construction affects learning quality.
- Different Application Scenarios: All four RL methods outperform the training-free baseline across arithmetic, retrieval, embodied, and shopping settings, while the best optimizer varies by task.GRPO leads on arithmetic, retrieval, and embodied settings, whereas PPO is strongest on shopping.
- Different Application Scenarios: Agent-R1 supports learning across GSM8K, HotpotQA, and ALFWorld, whose curves differ in speed, stability, and improvement pattern.GSM8K improves rapidly and stabilizes early, HotpotQA gains are slower and more fluctuating, and ALFWorld improves in pronounced late-stage jumps.
- Different RL Algorithms: GRPO and RLOO reach the strongest late-stage accuracy on GSM8K, while PPO remains more volatile and Reinforce++ achieves relatively high accuracy with substantially lower reward.The reward combines answer accuracy with a format-related component, and Reinforce++ produces much shorter later-stage responses.
- Different RL Algorithms: High task accuracy does not necessarily imply high reward, because optimizers can favor different response strategies under the same interaction setup.Agent-R1 keeps optimizer-specific behavior observable rather than washing it out.
- Context-Management Strategies: Sliding-window context performs best on GSM8K, while append-only replay is weaker and LLM-summarized context underperforms in this small-model setting.Sliding-window context retains the original question, recent tool output, and model analysis, producing more controlled responses without hurting task performance.
5 Future Directions
Agent-R1’s step-level view motivates future work on more efficient trajectory computation and stronger connections between structured traces and complex agent environments. These directions aim to preserve replay fidelity and context flexibility while improving scalability.
- Trajectory Representation: Step-level traces can introduce computational redundancy when related trajectories share long common prefixes during training.Future work may use prefix sharing or tree-structured merging to expose more efficient computation graphs while preserving replay fidelity and context flexibility.
- Complex Agent Environments: Future work should examine how increasingly complex agent environments connect to the training framework.The supplied discussion identifies this as a second direction but does not specify a concrete design.
6 Limitation Discussion
Agent-R1’s step-level traces and partially coupled rollout-training design impose efficiency limits. These constraints become more consequential for long-horizon agents and environments with slow, branching, persistent, or delayed interactions.
- Trajectory Computation: Step-level records can recompute shared prefixes across training samples, increasing cost for long-horizon agents.The paper identifies prefix sharing, KV-cache reuse, and tree-structured trajectory merging as possible remedies.
- Asynchronous Execution: Agent-R1 has not fully explored asynchronous rollout-training execution, which can make rollout a bottleneck in complex environments.The limitation is especially relevant when tools are slow, workflows branch, environments persist, or feedback is delayed.
7 Related Work
Related work spans both algorithms for multi-turn agentic RL and modular training frameworks that connect agents, rollout, optimization, and execution infrastructure. The field is moving from generic RL substrates toward increasingly agent-native abstractions.
- Agentic RL Algorithms: Agentic RL adapts PPO- and GRPO-style optimization to multi-turn interaction, sparse feedback, and environment-coupled behavior.Search-R1 and RAGEN illustrate why response-level rewards can be too weak for supervising long interaction chains.
- Agentic RL Training Frameworks: Agentic RL training frameworks include distributed RL substrates, low-intrusion agent integration, asynchronous execution, and scalable runtime-oriented systems.The cited examples include veRL, slime, Agent Lightning, rLLM, AReaL, MiniMax Forge, and Claw-R1.
- Training Infrastructure: A broader infrastructure trend connects inference engines and scalable training frameworks into unified loops that support tools, environment feedback, trajectory construction, and replay.Agent-R1 is positioned within this effort to bridge inference and training for interactive workflows.
8 Conclusion
Agent-R1 provides a unified, modular substrate for multi-turn LLM-agent reinforcement learning. It combines step-level modeling, structured trajectories, flexible context management, and modular interfaces while supporting multiple optimization algorithms.
- Agent-R1 unifies multi-turn RL training for LLM agents through step-level MDPs, structured trajectories, flexible context management, and modular interfaces.The framework is designed to remain analyzable, extensible, and reusable across agentic RL systems.
- The framework supports multiple algorithmic instantiations, including PPO and GRPO, on a common multi-turn RL substrate.
- A controlled GSM8K case study and the broader framework design demonstrate support for different algorithmic instantiations.