Source-linked AI summary
Group-in-Group Policy Optimization for LLM Agent Training
Lang Feng, Zhenghai Xue, Tingcong Liu, Bo An
TL;DR
Multi-turn LLM-agent training is difficult because long episodes and sparse or delayed rewards make individual-step credit assignment challenging. GiGPO addresses this with trajectory-level and repeated-state-level relative advantages, outperforming GRPO on ALFWorld and WebShop while preserving group-based RL’s efficiency properties.
Problem
Long-horizon LLM-agent episodes with sparse or delayed rewards make assigning credit to individual steps difficult.
Method
GiGPO combines episode-level trajectory grouping with retroactively constructed step-level groups formed from actions encountering identical environment states.
Results
> 12% on ALFWorld and > 9% on WebShop over GRPO, with search-based QA performance of 42.1% on 3B and 47.2% on 7B.
Takeaways & Limitations
GiGPO provides finer-grained per-step credit assignment while remaining critic-free and adding no LLM rollout or GPU memory overhead.
Takeaways & Limitations
GiGPO relies on state matching for anchor-group construction, which may be constrained in highly complex environments.
Abstract
from arXiv · showhide
Recent advances in group-based reinforcement learning (RL) have driven frontier large language models (LLMs) in single-turn tasks like mathematical reasoning. However, their scalability to multi-turn LLM agent training remains limited. Unlike static tasks, agent-environment interactions unfold over many steps and often yield sparse or delayed rewards, making credit assignment across individual steps significantly more challenging. In this work, we propose Group-in-Group Policy Optimization (GiGPO), a novel RL algorithm that achieves fine-grained credit assignment for LLM agents while preserving the appealing properties of group-based RL: critic-free, low memory, and stable convergence. GiGPO introduces a two-level structure for estimating relative advantage: (i) At the episode-level, GiGPO computes macro relative advantages based on groups of complete trajectories; (ii) At the step-level, GiGPO introduces an anchor state grouping mechanism that retroactively constructs step-level groups by identifying repeated environment states across trajectories. Actions stemming from the same state are grouped together, enabling micro relative advantage estimation. This hierarchical structure effectively captures both global trajectory quality and local step effectiveness without relying on auxiliary models or additional rollouts. We evaluate GiGPO on challenging agent benchmarks, including ALFWorld and WebShop, as well as tool-integrated reasoning on search-augmented QA tasks, using Qwen2.5-1.5B/3B/7B-Instruct. Crucially, GiGPO delivers fine-grained per-step credit signals, achieves performance gains of > 12% on ALFWorld and > 9% on WebShop over GRPO, and obtains superior performance on QA tasks (42.1% on 3B and 47.2% on 7B): all while maintaining the same GPU memory overhead, identical LLM rollout, and incurring little to no additional time cost.
1 Introduction
LLM agents extend language models into multi-turn environments, but sparse rewards and long horizons make step-level credit assignment difficult. GiGPO addresses this gap with hierarchical trajectory- and state-level relative advantages, achieving strong benchmark results without added rollout or memory overhead.
- Motivation: LLM agents perceive, reason, and act in multi-turn loops across embodied, web, and interactive environments.These settings require long-horizon planning and decision-making beyond static question answering.
- Motivation: Group-based RL is critic-free, memory-efficient, and scalable, but has mainly succeeded on single-turn tasks with immediate rewards.Its trajectory-level advantage estimation is less suited to delayed-reward agent interactions.
- Challenge: Long-horizon agent episodes can contain up to 50 steps and over 20k tokens, with rewards sometimes arriving only at episode end.Individual actions may affect outcomes much later, substantially complicating credit assignment.
- GiGPO: GiGPO computes macro relative advantages over complete trajectories and micro relative advantages over actions taken from repeated anchor states.The two-level design preserves global trajectory evaluation while enabling localized step-level credit assignment.
- Results: GiGPO achieves gains of > 12% on ALFWorld and > 9% on WebShop over GRPO, with search-based QA performance of 42.1% on 3B and 47.2% on 7B.Experiments use Qwen2.5-1.5B/3B/7B-Instruct and report fine-grained step-level credit signals.
2 Related Work
Research on LLM agents has progressed from prompting and classical RL toward parameter adaptation and group-based RL. GiGPO builds on group-based methods while targeting finer-grained credit assignment for interactive agents.
- LLM agents: Early LLM-agent systems used frozen models with prompting, memory, retrieval, and external tools across diverse interactive domains.Examples include program generation, smart-device operation, gameplay, and robot behavior control.
- RL for agents: Classical RL methods including DQN, PPO, and AWR have been applied to text games, Android control, ALFWorld, card games, and web tasks.WebShop methods such as ArCHer and AgentQ require intricate designs.
- Group-based RL: Group-based RL methods such as RLOO and GRPO avoid extra value functions by estimating advantages from samples sharing the same query.This supports large-scale training and has shown strong results in reasoning, search, and tool-use tasks.
- GiGPO: GiGPO retains critic-free learning and efficiency while introducing finer-grained credit assignment for LLM-agent training.Its hierarchical core is presented as orthogonal to existing group-based RL approaches.
3 Preliminaries
The paper models an LLM agent as a policy interacting with an environment over multi-step textual-action trajectories. Group-based RL replaces value-function estimation with relative advantages computed from rollout groups.
- Problem setup: At each step, the policy observes state s_t, generates textual action a_t, receives reward r_t, and transitions to state s_{t+1}.A trajectory contains these state-action-reward tuples over T steps.
- Problem setup: The policy π_θ(a_t|s_t, x) conditions textual actions on the current state and task prompt.Each action may span thousands of tokens, making token-level credit assignment difficult.
- Group-based RL: Group-based RL samples N candidate trajectories for a task and assigns each trajectory a scalar reward reflecting overall outcome quality.Advantages are computed from the rollout group rather than a separate value function.
- Group-based RL: GRPO normalizes each trajectory reward by the group mean and standard deviation.This design is memory-efficient and scales to large training batches and model sizes.
4 Training LLM Agents with GiGPO
GiGPO extends group-based RL to multi-turn agents by combining trajectory-level advantages with step-level comparisons among actions taken from recurring environment states. This preserves global performance signals while providing fine-grained credit assignment without extra rollouts.
- Episode-level advantages: GiGPO preserves episode-level grouping by sampling complete trajectories under identical task and initial-state conditions, then comparing their total returns.This provides a holistic signal about trajectory-wide task completion.
- Step-level advantages: Anchor state grouping constructs step-level groups by aggregating occurrences of identical environment states across trajectories and time steps.The procedure is offline and uses lightweight hashmap-based grouping rather than additional per-state rollouts.
- Step-level advantages: Step relative advantages compare the discounted outcomes of different actions taken from the same state, supplying localized feedback for individual decisions.Discounted returns capture an action’s future impact rather than relying only on immediate rewards.
- Illustration: In WebShop, repeated encounters with a search-results state can be grouped even when one trajectory succeeds and another fails.This illustrates how shared states enable relative comparison of actions with different eventual outcomes.
- Combined objective: GiGPO combines episode and step relative advantages into one hierarchical signal for policy optimization.The weighting coefficient ω balances global trajectory quality against local action effectiveness.
5 Experiment
GiGPO is evaluated across long-horizon embodied, web, and search-augmented QA tasks, with ablations, training-dynamics analysis, and computational profiling. The experiments examine performance, the roles of episode- and step-level advantages, evolving state-group structure, and added training cost.
- Experimental setup: GiGPO is evaluated on ALFWorld, WebShop, and search-augmented QA using Qwen2.5-1.5B/3B/7B-Instruct.The comparisons include closed-source models, prompting agents, PPO, GRPO, RLOO, and other RL baselines.
- Normalization analysis: Fnorm = 1 performs better than Fnorm = std on relatively difficult tasks such as Look, Pick2, and WebShop, but not uniformly elsewhere.Standard-deviation scaling can exaggerate gradients from overly difficult samples or imbalanced groups, harming update stability.
- Main results: 42.1% at 3B and 47.2% at 7B are reported for search-augmented QA, outperforming Search-R1 and StepSearch.Under three tool calls per query, the 7B model averages approximately 0.9 calls on single-hop tasks and 1.6 on multi-hop tasks.
- Ablation: Removing either episode or step relative advantages substantially degrades performance, with step-level removal especially affecting Cool, Pick2, and WebShop.The ablation compares GiGPO variants with and without standard-deviation normalization and either advantage component.
- Training dynamics: Step-level groups of size 1 remain below 35% throughout ALFWorld training, while large groups decrease and later concentrate around sizes 6 to 8.Groups with sizes 10≤|GS(˜s)|<50 decrease from 16.2% to 12.1%, and groups with |GS(˜s)|≥50 decrease from 5.6% to 3.1% by iteration 75.
- Computational budget: < 0.002% of per-iteration training time is added by anchor state grouping and step-relative advantage computation.The two additions take 0.01s and 0.53s per iteration, compared with 362.83s for dominant shared operations.
6 Conclusions and Limitations
GiGPO combines episode-level and step-level advantages for fine-grained credit assignment while preserving group-based RL efficiency and stability. The work also releases verl-agent and identifies state matching as a limitation with a GRPO fallback.
- 6 Conclusions and Limitations: GiGPO retroactively groups identical anchor states across trajectories to compute step-level relative advantages alongside episode-level advantages.The grouping uses anchor states as hash-table keys, while advantage combination requires only lightweight arithmetic.
- 6 Conclusions and Limitations: State matching may be difficult in noisy or subtly varying environments, but GiGPO degrades to GRPO when no states repeat across trajectories.The fallback preserves GRPO’s effectiveness and stability in credit assignment.
- 6 Conclusions and Limitations: verl-agent provides step-wise multi-turn rollouts, flexible memory control, and scalable reinforcement learning for long-horizon LLM agents.The framework avoids concatenating full interaction histories at every step.
- 6 Conclusions and Limitations: GiGPO preserves critic-free optimization, GPU-memory efficiency, and stable convergence without additional LLM rollouts.Its training procedure adds grouping and arithmetic operations to vanilla GRPO.
- 6 Conclusions and Limitations: Fnorm = 1 yields an unbiased estimator up to constant scaling, which can be absorbed into the learning rate without changing policy-gradient dynamics.This establishes the relationship between the fixed normalization and RLOO-style advantages.
E.1 Details of Training
The experiments use fixed rollout, optimization, reward, and model configurations across agent benchmarks, with prompt templates controlling runtime context and reasoning formats.
- ALFWorld: ALFWorld uses up to 50 steps, success reward 10, failure reward 0, invalid-action penalty -0.1, and group size 8.The setup samples 16 groups per rollout, producing 128 environments.
- WebShop: WebShop uses up to 15 steps, success reward 10, failure reward 0, and invalid-action penalty -0.1.Group-based methods use the same configured benchmark settings.
- Search-Augmented QA: Search-augmented QA uses at most 4 turns, success reward 1, failure reward 0, invalid-action penalty -0.01, and group size 5.The retriever and training configuration follow the Search-R1 settings described in the paper.
- Prompts: Prompt templates dynamically fill task, step, and observation slots, using history length 2 for ALFWorld and WebShop and full history for search-augmented QA.Reasoning and actions are delimited with <think> and <action> blocks; search agents additionally use <search>, <anwser>, and <information> tags.
E.3 Performance on Vision-Language Agents
GiGPO is evaluated on visual-language agents in Sokoban and EZPoints, where agents reason over visual and textual inputs. It outperforms prompting and GRPO on both tasks.
- Performance on Vision-Language Agents: 81.0% success on Sokoban and 100% success on EZPoints are achieved by GiGPO with Qwen2.5-VL-3B-Instruct.Sokoban tests spatial reasoning and long-term planning in a 6×6 grid-based puzzle.
- Performance on Vision-Language Agents: GiGPO significantly outperforms both the prompting baseline and GRPO on Sokoban and EZPoints.The comparison is reported in Table 3 for the two visual-language tasks.
E.4 Orthogonality to Single-Turn Group-Based RL
GiGPO can incorporate techniques from single-turn group-based RL. Combining it with DAPO produces GiGPOdynamic, which outperforms DAPO on WebShop.
- Orthogonality: GiGPO is presented as orthogonal to single-turn group-based RL advances, allowing complementary techniques to be integrated.The paper tests this claim by incorporating DAPO’s dynamic sampling and clip-higher techniques.
- Results: 75.0% on WebShop is achieved by GiGPOdynamic, outperforming DAPO after integrating DAPO techniques into GiGPO.DAPO itself improves over GRPO in the same comparison.
E.5 Sensitivity Analysis on ω
GiGPO’s WebShop performance depends on balancing episode-level and step-level advantages. The best setting emphasizes step-level feedback without overwhelming trajectory-level guidance.
- E.5 Sensitivity Analysis on ω: ω balances episode-level advantage AE and step-level advantage AS in GiGPO.The sensitivity study evaluates this hyperparameter on WebShop with Qwen2.5-1.5B-Instruct.
- E.5 Sensitivity Analysis on ω: ω = 0.8 is the performance optimum in the WebShop sensitivity analysis.Increasing ω initially improves performance, but performance declines beyond this value.
- E.5 Sensitivity Analysis on ω: Excessive emphasis on step-level signals can suppress useful trajectory-level guidance.The observed decline beyond the optimum indicates that both levels contribute to effective optimization.
- E.5 Sensitivity Analysis on ω: GiGPO is relatively insensitive to ω within the range [0.4, 1.2].This range demonstrates a reasonable degree of robustness to the weighting hyperparameter.
F.2 WebShop
The WebShop example illustrates multi-turn product-search behavior under detailed constraints. The agent filters by color, size, and price, but the shown results remain mismatched to the requested product type and fit.
- F.2 WebShop: The task requests blue, xx-large men’s tuxedo shirts under $40 with specified fit, sleeve, and material constraints.The initial observation is the search interface, so the agent must refine the query and product selection.
- F.2 WebShop: The initial results include short-sleeve T-shirts and other products rather than an exact match for the requested tuxedo shirt.The agent identifies the results as insufficiently filtered and selects a seemingly relevant product despite the mismatch.
- F.2 WebShop: The agent clicks b-blue and then xx-large before proceeding to Buy Now.Its reasoning treats the search as filtered by color, size, and price, although the product-type and fit mismatch remains noted.
F.3 Search
The search-augmented QA example shows ambiguity in identifying the song associated with “One Last Time.” Resolving the intended Ariana Grande video yields Matt Bennett as the answer.
- F.3 Search: The question asks who the guy is in the video for the performer of “One Last Time.”The wording is ambiguous because multiple songs titled “One Last Time” are considered.
- F.3 Search: The search process identifies both David Guetta’s and Ariana Grande’s songs titled “One Last Time.”The retrieved information describes different videos for the two songs.
- F.3 Search: Matt Bennett is identified as Ariana Grande’s boyfriend and the guy in her music video.The video depicts Grande and Bennett in a Fiat while attempting to leave the city.