Source-linked AI summary
StraTA: Incentivizing Agentic Reinforcement Learning with Strategic Trajectory Abstraction
Xiangyuan Xue, Yifan Zhou, Zidong Wang, Shengji Tang, Philip Torr, Wanli Ouyang, Lei Bai, Zhenfei Yin
TL;DR
Long-horizon agentic RL is difficult because reactive decision making weakens exploration and credit assignment. StraTA adds an explicit trajectory-level strategy, jointly trains strategy generation and execution with hierarchical GRPO-style rollouts, and consistently improves benchmark performance, including 93.1% ALFWorld success, 84.2% WebShop success, and 63.5% on SciWorld.
Problem
Long-horizon agentic RL remains difficult because purely reactive decision making weakens exploration and credit assignment over extended trajectories.
Method
StraTA samples a compact strategy from the initial task state, conditions subsequent actions on it, and jointly trains strategy generation and execution with hierarchical GRPO-style rollouts.
Results
StraTA consistently improves sample efficiency and final performance over prompting-based and RL baselines across ALFWorld, WebShop, and SciWorld.
Takeaways & Limitations
Explicit trajectory-level strategy provides a unified design for improving exploration and learning in long-horizon agentic RL.
Takeaways & Limitations
The method assumes a non-cumulative setting in which each step is prompted from the current state rather than the full interaction history.
Abstract
from arXiv · showhide
Large language models (LLMs) are increasingly used as interactive agents, but optimizing them for long-horizon decision making remains difficult because current methods are largely purely reactive, which weakens both exploration and credit assignment over extended trajectories. In this work, we present Strategic Trajectory Abstraction (StraTA), a simple framework that introduces an explicit trajectory-level strategy into agentic reinforcement learning (RL). StraTA samples a compact strategy from the initial task state, conditions subsequent actions on that strategy, and trains strategy generation and action execution jointly with a hierarchical GRPO-style rollout design, further enhanced by diverse strategy rollout and critical self-judgment. Experiments on ALFWorld, WebShop, and SciWorld show that StraTA consistently improves both sample efficiency and final performance over strong baselines. StraTA reaches success rates of 93.1% on ALFWorld and 84.2% on WebShop. On SciWorld, StraTA attains a 63.5% overall score, outperforming frontier closed-source models.
1. Introduction
Long-horizon agentic RL is difficult because reactive policies must jointly manage immediate actions and overall plans under hierarchical decisions and delayed rewards. StraTA introduces a fixed trajectory-level strategy to guide subsequent execution and jointly trains strategy generation with action execution.
- Long-horizon agentic RL combines hierarchical action generation with sparse, delayed rewards, complicating exploration and credit assignment.
- Purely reactive policies choose actions from current states alone, often producing short-sighted exploration, unnecessary backtracking, and inconsistent behavior.
- StraTA samples a compact strategy from the initial task state and conditions every subsequent action on that fixed strategy.
- StraTA decomposes long-horizon decision making into generating a good strategy and executing actions that follow it.
- 93.1% success on ALFWorld and 84.2% on WebShop are achieved with a 7B backbone, while SciWorld reaches 63.5% overall.StraTA also reaches 100.0% on the SciWorld Lifespan subset and improves sample efficiency and final performance across benchmarks.
2. Related Work
Related work spans reinforcement fine-tuning and agentic RL, with the latter emphasizing online, long-horizon interaction. Existing work addresses exploration and credit assignment, while StraTA targets both through trajectory-level strategy guidance.
- Reinforcement fine-tuning: Reinforcement fine-tuning has progressed from PPO and GRPO foundations to methods improving stability, scalability, and intrinsic reward learning.
- Agentic reinforcement learning: Agentic RL trains long-horizon interactive behavior through trial and error, requiring online exploration and adaptation rather than fixed routines.
- Agentic reinforcement learning: Agentic RL differs from reinforcement fine-tuning by emphasizing long-horizon decision-making instead of single-turn reasoning.
- Agentic reinforcement learning: Research on agentic RL includes training frameworks, active exploration, and increasingly accurate credit assignment for multi-turn interaction.
- Agentic reinforcement learning: StraTA supports exploration and credit assignment with an explicit trajectory-level strategy guiding the entire episode.
3. Preliminaries
The paper formulates agentic tasks and LLM policies within finite-horizon MDPs, where natural-language states and actions produce trajectories with terminal rewards. It adopts group-based RL and GRPO-style clipped optimization for efficient training.
- Long-horizon agentic tasks: Long-horizon agentic tasks are modeled as finite-horizon MDPs with state and action spaces, transitions, rewards, horizon, and discount factor.
- Long-horizon agentic tasks: An episode forms a trajectory of state, action, and reward tuples, typically receiving only a terminal outcome reward with γ = 1.
- LLM-driven agent policy: The LLM policy maps each natural-language state to an action, while generating that action autoregressively as a token sequence.
- LLM-driven agent policy: The non-cumulative setting constructs each step from the current state rather than the full interaction history to control effective context length.
- Group-based reinforcement learning: GRPO removes the explicit critic and derives learning signals from relative reward comparisons within sampled trajectory groups.
- Group-based reinforcement learning: The clipped surrogate applies token-level importance ratios and advantages to optimize responses, with a generic formulation covering strategies or actions.
4. Method
StraTA introduces an explicit trajectory-level strategy into agentic RL, separating high-level planning from local action execution. It jointly optimizes both levels with hierarchical rollouts and adds mechanisms for strategy diversity, credit assignment, and training stability.
- Critical self-judgment and reward shaping: Critical self-judgment assigns a step-level auxiliary penalty to actions that neither follow the strategy nor advance task progress, addressing sparse-reward credit assignment.The method complements terminal outcome rewards with finer-grained signals while also using soft length and hard format penalties to stabilize generation and execution.
- Strategy-guided execution: StraTA generates a compact strategy at episode start and conditions every subsequent action on both that strategy and the current state.This changes reactive decision making into a structured process with separate strategy-generation and strategy-consistent action-execution objectives.
- Hierarchical rollout design: StraTA samples multiple strategies and multiple rollouts per strategy, creating one strategy-level group and multiple action-level groups for GRPO-style optimization.The strategy-level group compares plans, while each action-level group compares trajectories produced under the same plan.
- Hierarchical rollout design: The strategy-level reward averages the top-δ fraction of rollout outcomes, emphasizing achievable performance while reducing the influence of noisy low-quality executions.This is intended to make the reward reflect strategy quality more faithfully than an unfiltered average.
- Diverse strategy rollout: Farthest point sampling selects semantically distinct strategies from oversampled candidates, making strategy comparisons more informative and broadening exploration.Candidate strategies are embedded as normalized vectors, and selection greedily favors candidates with low maximum similarity to those already selected.
5. Experiments
StraTA is evaluated across ALFWorld, WebShop, and SciWorld against prompting-based and training-based baselines, with results covering performance, ablations, and computational efficiency. It consistently achieves strong task performance while adding only modest computational overhead.
- Experimental setup: StraTA is evaluated on ALFWorld, WebShop, and SciWorld, covering embodied manipulation, web navigation, and scientific experimentation.
- Main results: 93.1% ALFWorld success and 84.2% WebShop success are achieved by StraTA with the 7B backbone.With the 1.5B backbone, StraTA reaches 90.7% and 82.5% success on the same benchmarks.
- Main results: 63.5% overall score is achieved by StraTA on SciWorld, including 100.0% on the Lifespan subset.
- Ablation studies: 87.9% ALFWorld performance is reached by the Diverse variant, while the Judgment variant reaches 66.7% on WebShop.The study reports that the two components accelerate learning and improve performance in a largely orthogonal manner.
- Computational efficiency: StraTA is only marginally slower than GRPO in rollout and training, and its two core techniques add negligible overhead.Theoretical analysis states that the component overhead accounts for only 1/H of the main action rollouts.
6. Conclusion
StraTA addresses long-horizon agentic RL by adding an explicit trajectory-level strategy that separates high-level planning from low-level execution. Across three benchmarks, it improves sample efficiency and final performance, while its fixed-strategy design remains sensitive to strategy quality and substantial environmental change.
- StraTA generates a compact strategy from the initial task state and conditions subsequent actions on that fixed strategy.
- Hierarchical GRPO-style rollouts, diverse strategy rollout, and critical self-judgment jointly improve exploration and learning.
- StraTA consistently improves sample efficiency and final performance over strong prompting-based and RL baselines on ALFWorld, WebShop, and SciWorld.
- Its effectiveness depends on generated-strategy quality, and a fixed strategy may become restrictive when the environment changes substantially during an episode.
A.1. Effectiveness of Core Components
The ablation study evaluates StraTA’s core components through ALFWorld training curves and comparisons with ablated variants. Adding Diverse and Judgment further accelerates convergence, with the full system achieving the most stable and efficient training.
- StraTA is compared with ablated variants to evaluate the contribution of its core components.
- Adding the Diverse and Judgment techniques further accelerates convergence beyond the Vanilla variant.
- The full StraTA configuration achieves the most stable and efficient training.
A.2. Impact of Training Parameters
The ablations examine oversampling, strategy-quality aggregation, and self-judgment weighting, identifying settings that balance exploration and reliable evaluation. Across these factors, the main configuration uses σ=8, δ=0.5, and κ=0.1.
- Oversampling ratio: Increasing the oversampling ratio σ from 1 to 8 steadily improves performance on ALFWorld and WebShop.The authors attribute this trend to more diverse strategy exploration and more informative, discriminative reward signals.
- Aggregation ratio: δ=0.5 achieves the best performance among the tested aggregation ratios.δ=0.1 uses too few trajectories to estimate strategy quality reliably, whereas δ=1.0 ignores potential execution errors during action selection.
- Self-judgment weight: κ=0.1 provides the most effective trade-off among the tested self-judgment reward weights.κ=0.01 underuses critical self-judgment, while κ=1.0 places excessive reliance on self-judgment quality.
B. Additional Implementation Details
The implementation standardizes three agentic environments and uses prompt templates to generate a persistent global strategy, guide actions with it, and judge problematic steps. The prompts define the strategy as concrete, practical guidance maintained throughout the episode.
- Environmental setup: AgentGym provides a unified interface that standardizes observations, actions, interaction loops, and reward computation across ALFWorld, WebShop, and SciWorld.The benchmark setup covers embodied manipulation, web navigation, and scientific experimentation.
- Prompt templates: The global-strategy prompt requires a constant text snippet that guides future actions throughout the whole episode.It also requires the strategy to be concrete enough for strict following and practical enough to use existing information.
- Prompt templates: The action-taking prompt conditions each action on the generated strategy, while the self-judgment prompt identifies problematic steps in the action history.The judgment format returns the step numbers that fail to follow the strategy without contributing to final success.
B.3. Training Parameters
Additional details describe the implementation of diverse strategy selection and illustrate strategy-guided behavior across three environments. Semantic farthest-point sampling selects diverse strategies before rollouts, with selection cost negligible relative to LLM rollout cost.
- Diverse strategy rollout: Diverse strategy selection oversamples candidate strategies and uses farthest point sampling in semantic space to select maximally different strategies.Each strategy is encoded with a pretrained embedding model, and cosine similarity is computed through normalized-vector inner products.
- Diverse strategy rollout: O(σN^2d) is the stated complexity of diverse strategy selection, while practical selection takes only a few milliseconds per task.The reported selection cost is negligible relative to the cost of LLM rollouts.
- Case studies: In the ALFWorld case study, the strategy explores the room, retrieves tissuebox 1 from the coffeetable, and places it on the sidetable successfully.The final feedback states that the task was completed successfully.
- Case studies: In the WebShop case study, the strategy begins with a generic search, refines it using the criteria, and reaches a completed purchase task with score 1.0.The selected mattress is priced at $520.0, below the task’s $550.00 threshold.
C.3. SciWorld
The SciWorld case study demonstrates a strategy-guided sequence from the living room to the outside location, where the agent focuses on a baby elephant after inspecting the available environment. The task is completed successfully.
- Task completion: The agent observes the outside environment, which contains a baby elephant and other animals, then focuses on the baby elephant.The final feedback states that the task was completed successfully.
- Strategy generation: The generated strategy directs the agent to open the hallway door, gain visibility, and compare animal life spans outside.The strategy links initial navigation with the later comparison required by the task.
- Strategy-guided execution: The agent executes the plan by opening and traversing the hallway and greenhouse doors before reaching the outside location.The environment feedback confirms each move and door-opening action in sequence.