PaperlayerView exact revision on arXiv

Source-linked AI summary

Group-in-Group Policy Optimization for LLM Agent Training

Lang Feng, Zhenghai Xue, Tingcong Liu, Bo An

arXiv:2505.10978v3cs.LGcs.AI

TL;DR

Sparse rewards and delayed action effects complicate step-level credit assignment for multi-turn LLM agents. GiGPO combines trajectory-level advantages with advantages from actions sharing environment states. It outperforms prior RL methods across agent benchmarks and search-augmented QA without additional LLM rollout or GPU memory overhead.

  • Problem

    Step-level credit assignment remains challenging for multi-turn LLM agents whose long episodes provide sparse rewards and delayed action effects.

    Source: §1 · p. 2

  • Method

    GiGPO combines episode-level relative advantages with step-level advantages computed by grouping actions that encounter identical environment states across trajectories.

    Source: §4 · p. 4, §4.2 · p. 5

  • Results

    GiGPO outperforms prompt-based agents and prior RL methods across ALFWorld, WebShop, and search-augmented QA, including > 12% gains over GRPO on ALFWorld.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    GiGPO’s performance gains do not require additional LLM rollout or GPU memory overhead.

    Source: §6 · p. 10

    Standard-deviation normalization, as used by GRPO [18], may introduce difficulty bias [19], giving low-variance trajectory groups disproportionately large gradients.

    Source: §4.1 · p. 4

Abstract

from arXiv · show

Recent advances in group-based reinforcement learning (RL) have driven frontier large language models (LLMs) in single-turn tasks like mathematical reasoning. However, their scalability to multi-turn LLM agent training remains limited. Unlike static tasks, agent-environment interactions unfold over many steps and often yield sparse or delayed rewards, making credit assignment across individual steps significantly more challenging. In this work, we propose Group-in-Group Policy Optimization (GiGPO), a novel RL algorithm that achieves fine-grained credit assignment for LLM agents while preserving the appealing properties of group-based RL: critic-free, low memory, and stable convergence. GiGPO introduces a two-level structure for estimating relative advantage: (i) At the episode-level, GiGPO computes macro relative advantages based on groups of complete trajectories; (ii) At the step-level, GiGPO introduces an anchor state grouping mechanism that retroactively constructs step-level groups by identifying repeated environment states across trajectories. Actions stemming from the same state are grouped together, enabling micro relative advantage estimation. This hierarchical structure effectively captures both global trajectory quality and local step effectiveness without relying on auxiliary models or additional rollouts. We evaluate GiGPO on challenging agent benchmarks, including ALFWorld and WebShop, as well as tool-integrated reasoning on search-augmented QA tasks, using Qwen2.5-1.5B/3B/7B-Instruct. Crucially, GiGPO delivers fine-grained per-step credit signals, achieves performance gains of > 12% on ALFWorld and > 9% on WebShop over GRPO, and obtains superior performance on QA tasks (42.1% on 3B and 47.2% on 7B): all while maintaining the same GPU memory overhead, identical LLM rollout, and incurring little to no additional time cost.

1 Introduction

GiGPO combines episode-level trajectory comparisons with step-level comparisons at repeated environment states to improve credit assignment for multi-turn LLM agents. It provides localized credit signals without auxiliary value models or per-step extra rollouts.

1 Introduction

  • LLM agents operate through multi-turn perception, reasoning, and action loops, requiring long-horizon planning and decision-making beyond language understanding.Applications include embodied assistants in simulated homes [5] [6], mobile and web navigation [7] [8] [9] [10], and interactive-game exploration [11] [12].Source: §1 · p. 1
  • RLOO [17] and GRPO [18] replace value-function estimation with within-group relative advantages, offering critic-free, low-memory optimization but succeeding mainly on single-turn tasks.Their established applications include mathematical problem solving [19] [20] and code generation [21], where rewards arrive immediately and credit assignment is straightforward.Source: §1 · pp. 1–2
  • Sparse rewards and delayed action effects complicate long-horizon agent credit assignment, while naïve group-based RL collapses distinctions between individual steps.An ALFWorld [5] episode may span up to 50 steps and over 20k tokens. The motivating challenge is finer credit assignment while preserving critic-free optimization, low memory use, and stable convergence.Source: §1 · p. 2
  • GiGPO computes episode-level macro advantages from total returns across complete trajectories sharing the same task and initial state, following GRPO [18].These advantages capture overall trajectory effectiveness and task-execution completeness.Source: §1 · p. 2
  • Anchor state grouping retroactively groups actions at repeated environment states across trajectories, assigning localized credit through relative action performance.Ineffective actions and loops naturally produce shared states under identical task and initial-environment conditions. Reusing these states avoids per-step extra rollouts and keeps GiGPO critic-free, without auxiliary value models.Source: §1 · p. 2
  • > 12% ALFWorld [5] and > 9% WebShop [22] gains over GRPO accompany GiGPO’s 42.1% (3B) and 47.2% (7B) search-based QA performance.Across experiments using Qwen2.5-1.5B/3B/7B-Instruct [3], GiGPO consistently outperforms prompt-based agents, actor-critic baselines, and prior group-based RL methods. The reported additional time cost is < 0.002%.Source: §1 · p. 2

2 Related Work

Prior work has shifted LLM agents from frozen-model prompting toward learning through environment interaction, while group-based RL estimates advantages without extra value functions. GiGPO builds on group-based RL to introduce finer-grained credit assignment while retaining critic-free efficiency and compatibility with existing approaches.

LLMs as decision-making agents

  • LLM-agent research has shifted from frozen-model prompting toward supervised fine-tuning [24] and RL, enabling learning directly from environment interaction.Earlier approaches used ReAct [29], Reflexion [30], memory and retrieval [12] [31], and external tools [32] [33] [34]. Applications include program generation [23], smart devices [24] [25] [26] [27], gameplay [11], and robot control.Source: §2 · p. 2

Reinforcement learning for LLM agents

  • RAGEN [51] concatenates states, intermediate reasoning, and actions into a unified trajectory-level GRPO response, but faces scalability challenges on long-horizon tasks.The passage highlights ALFWorld, which can involve up to 50 steps.Source: §2 · p. 3
  • Agent RL progressed from DQN in text games [36] to PPO [42] and AWR [43], while newer web-agent methods introduce additional computational overhead.Interactive settings include Android control, ALFWorld [5], and card games [45]. ArCHer and AgentQ [47] target WebShop [22] but require intricate designs involving additional value networks or Monte Carlo Tree Search.Source: §2 · p. 3
  • CoSo [10] improves agents through entropy-based RL, while LOOP [49] combines RLOO [17] with PPO-style updates, achieving state-of-the-art results in AppWorld.Source: §2 · p. 3

Reinforcement learning for large language models

  • GiGPO targets finer-grained credit assignment for LLM agents while preserving critic-free efficiency; its hierarchical structure is compatible with existing group-based RL approaches.The hierarchical mechanism is orthogonal to existing group-based methods, allowing GiGPO to incorporate them.Source: §2 · p. 3
  • Group-based RL estimates advantages from same-query samples without extra value functions, enabling large-scale training with strong results in mathematical reasoning [15], search [58] [59], and tool use [60] [61].Examples include RLOO [17], GRPO [18], Dr. GRPO [19], DAPO [20], and CPPO [57].Source: §2 · p. 3

3 Preliminaries

Multi-step LLM agents generate textual actions under sparse or delayed feedback, making token-level credit assignment difficult. Group-based RL estimates trajectory advantages from sampled-group statistics rather than separate value functions.

Problem setup

  • Sparse, delayed, or weak intermediate rewards make token-level credit assignment difficult across multi-step episodes containing potentially thousands of tokens per action.Feedback may consist only of success or failure signals at the end of an episode.Source: §3 · p. 3
  • An LLM policy conditions textual actions on the current state and task description, while the environment returns rewards and subsequent states.A complete episode is a trajectory of T state–action–reward triples; each action contains at most n vocabulary tokens.Source: §3 · p. 3

Group-based RL

  • Group-based RL samples N complete trajectories per task under the old policy and estimates advantages from group statistics instead of separate value functions like PPO [42].GRPO [18] normalizes each trajectory’s reward using the group’s mean and standard deviation. This memory-efficient design supports large training batches and model sizes.Source: §3 · p. 3

4 Training LLM Agents with GiGPO

GiGPO combines trajectory-level comparisons with step-level comparisons among actions taken from the same environment state. Its anchor state grouping reuses collected trajectories to provide fine-grained credit without extra rollouts.

4 Training LLM Agents with GiGPO

  • GiGPO introduces two-level grouping: episode advantages capture overall trajectory quality, while step advantages distinguish actions taken from the same environment state.Vanilla GRPO assigns one advantage per episode, obscuring individual action contributions; generating additional actions per state incurs substantial forward-pass overhead and difficulties evaluating unexecuted actions.Source: §4 · p. 4

4.1 Episode Relative Advantages

  • Episode advantages normalize each trajectory’s total return using the group mean and a normalization factor, providing a coarse, trajectory-wide credit signal.The policy collects N complete trajectories under a fixed task and identical initial states.Source: §4.1 · pp. 4–5
  • Standard-deviation normalization in GRPO [18] can introduce difficulty bias [19], disproportionately amplifying gradients from low-variance groups.This effect occurs frequently in long-horizon agent tasks and can destabilize updates. The alternative Fnorm = 1 yields an unbiased Leave-One-Out estimator and helps stabilize some challenging scenarios.Source: §4.1 · pp. 4–5

4.2 Step Relative Advantages

  • Anchor state grouping retroactively pools identical-state occurrences across trajectories and time steps, pairing them with their actions without extra rollouts.Each unique environment state serves as an anchor; grouping operates entirely offline using lightweight hashmaps.Source: §4.2 · p. 5
  • Discounted future returns enable step advantages to compare individual actions within each shared-state group, supplying micro credit beyond immediate rewards.This supports sparse-reward tasks while also leveraging per-step rewards in dense-reward settings.Source: §4.2 · pp. 5–6

4.3 Group-in-Group Policy Optimization

  • GiGPO balances episode and step advantages with a nonnegative coefficient, then uses a clipped policy objective with a KL penalty toward a reference policy.The objective uses an importance-sampling ratio, while β controls the KL-penalty strength.Source: §4.3 · p. 6

5 Experiment

GiGPO outperforms GRPO on ALFWorld and WebShop and strong baselines on search-augmented QA. Ablations support combining episode- and step-level advantages, while computational measurements show minimal additional time and unchanged memory and rollout costs relative to GRPO.

5.1 Experiment Setup

  • Evaluation spans embodied ALFWorld [5], shopping WebShop [22], and single- and multi-hop search-augmented QA, using prompting, closed-source, and RL baselines.Agent baselines include GPT-4o [1], Gemini-2.5-Pro [2], ReAct [29], Reflexion [30], PPO [42], RLOO [17], and GRPO [18]. QA baselines include R1-Instruct, Search-R1 [58], ZeroSearch [59], and StepSearch [69].Source: §5.1 · p. 7
  • Qwen2.5-1.5B/3B/7B-Instruct [3] models use ω = 1 without tuning, with matched RL hyperparameters for ALFWorld and WebShop.Rollout groups contain eight trajectories for ALFWorld/WebShop and five for QA, which uses at most four turns, Search-R1 [58] settings, and E5 [70] retrieval. Similarity-based GiGPO groups states whose longest matching subsequence similarity exceeds 0.9.Source: §5.1 · p. 7

5.2 Performance on ALFWorld and WebShop

  • 13.3% on ALFWorld and 10.6% on WebShop are GiGPOw/o std’s gains over GRPO at 1.5B.At 7B, the corresponding gains are 12.6% and 9.1%. Both normalization variants consistently surpass GRPO and RLOO; results are averaged over three random seeds.Source: §5.2 · p. 7, Table 1 · p. 8

5.3 Performance on QA tasks

  • 42.1% at 3B and 47.2% at 7B are GiGPO’s search-augmented QA results, outperforming Search-R1 and StepSearch.With at most three tool calls per query, 7B averages approximately 0.9 single-hop and 1.6 multi-hop calls, versus OTC [61]’s approximately 1.0 and 1.7, respectively. Grouping repeated query–information states helps eliminate redundant calls during training.Source: Table 2 · p. 8, §5.3 · pp. 8–9

5.2 Performance on ALFWorld and WebShop

  • Fnorm = 1 improves success on Look, Pick2, and WebShop, while standard-deviation normalization remains task-dependent and potentially useful with stable reward variance.Standard-deviation scaling can amplify gradients from difficult samples or imbalanced groups, destabilizing updates. The variants perform similarly on other tasks, consistent with [72].Source: §5.2 · p. 8

5.4 Ablation Study

  • Removing either episode- or step-level advantages substantially degrades performance, with structural ablations causing larger losses than changing normalization.On Qwen2.5-1.5B-Instruct, removing episode-level advantages hurts all tasks, while removing step-level advantages particularly hurts Cool, Pick2, and WebShop.Source: §5.4 · p. 9, Figure 4 · p. 9

5.5 Dynamics of Step-Level Group

  • Over 65% of states recur across trajectories throughout ALFWorld training, supporting anchor state grouping as group sizes evolve toward consistent trajectories.From iteration 10 to 75, groups sized 10–49 decrease from 16.2% to 12.1%, while groups sized at least 50 decrease from 5.6% to 3.1%, suggesting reduced behavioral redundancy. By iteration 140, sizes concentrate around 6–8 with eight trajectories per episode group, alongside success above 80%.Source: §5.5 · pp. 9–10, Figure 5 · p. 9

5.6 Computational Budget

  • 0.01s for anchor state grouping and 0.53s for step-relative advantages add little overhead per iteration, with unchanged GPU memory and rollout costs versus GRPO.Shared operations require 362.83s per iteration in the ALFWorld Qwen2.5-1.5B-Instruct test. Both methods use a single actor LLM without a critic.Source: §5.6 · p. 10, Figure 6 · p. 10

6 Conclusions and Limitations … D Pseudo Code

GiGPO combines episode-level and step-level advantages to improve long-horizon agent training without additional LLM rollouts or GPU memory overhead. Its state-matching limitation is accompanied by a fallback to GRPO, while verl-agent provides a scalable training framework.

6 Conclusions and Limitations

  • GiGPO outperforms prompt-based agents and prior RL methods across ALFWorld, WebShop, and search-augmented QA without additional LLM rollouts or GPU memory overhead.Hierarchical episode-level and step-level advantages provide fine-grained credit assignment while retaining training efficiency and stability.Source: §6 · p. 10, B Broader Impacts · p. 16
  • GiGPO falls back to GRPO when no states repeat, but noisy or subtly different states can impede anchor grouping.Similarity-based grouping partly mitigates this limitation; embedding-based representations and domain-specific structural equivalence remain future directions.Source: §6 · p. 10

A Open Source Codebase: verl-agent

  • The released verl-agent framework supports scalable long-horizon training through step-wise rollouts, customizable memory, and parallel gym-style environments rather than concatenating full histories.Built upon veRL [73], it lets developers select historical inputs such as key events, summaries, or external knowledge at each step.Source: A Open Source Codebase: verl-agent · p. 16, Figure 7 · p. 16
  • verl-agent supports diverse language and vision-language models, LoRA fine-tuning, multiple environments, and several RL algorithms.Model support includes Qwen3 [74], Qwen2.5, LLaMA3.2, and Qwen2.5-VL. Environments include Search, ALFWorld, WebShop, Sokoban, and Gym Cards; algorithms include GiGPO, GRPO, PPO, DAPO, and RLOO.Source: A Open Source Codebase: verl-agent · p. 16

B Broader Impacts

  • GiGPO’s critic-free, scalable design could broaden access to multi-step agents, but potential increases in autonomy warrant safeguards and responsible usage.Potential applications include virtual assistants, web automation, educational tools, and embodied AI; this methodological work has no direct downstream deployment.Source: B Broader Impacts · p. 16

C Unbiasedness

  • Fnorm = 1 yields an unbiased estimator up to constant scaling, corresponding to a rescaled RLOO [17] advantage.The derivation follows [19]; the constant advantage scaling can be absorbed into the learning rate without changing policy-gradient dynamics.Source: C Unbiasedness · pp. 16–17

D Pseudo Code

  • GiGPO constructs step-level groups using anchor-state hash keys and combines advantages through lightweight arithmetic, preserving critic-free, low-memory training and stable convergence.Hash-table aggregation introduces minimal overhead while enabling fine-grained credit assignment for long-horizon agents.Source: D Pseudo Code · p. 17

E Experiment Details · E.1 Details of Training · E.2 Prompts

Training configurations specify task-dependent rollout limits, rewards, and optimization settings, while GiGPO uses the same untuned weighting coefficient and discount factor across tasks. Prompt templates dynamically insert task context and separate reasoning from actions or search outputs.

E.1 Details of Training

  • 2048-token prompt limits and 50-step episode limits apply to ALFWorld; WebShop uses 4096-token prompt limits and 15-step episode limits.Both benchmarks cap responses at 512 tokens; mini-batch sizes are 256 for ALFWorld and 64 for WebShop.Source: §E.1 · pp. 17–18
  • 10 for success, 0 for failure, and -0.1 for invalid actions define the rule-based rewards for ALFWorld and WebShop.Both benchmarks use actor learning rate 1e-6, critic learning rate 1e-5 only for PPO, and KL-divergence loss coefficient 0.01.Source: §E.1 · pp. 17–18
  • 128 environments per rollout support ALFWorld and WebShop training, with group-based methods sampling 16 groups of 8.PPO instead uses 128 separate environments. Both benchmarks use rollout temperature 1.0 and validation temperature 0.4.Source: §E.1 · p. 18
  • 1 is GiGPO’s untuned weighting coefficient ω, and 0.95 is its discount factor γ across all three tasks.ALFWorld and WebShop run for 150 iterations, using 2×H100 GPUs for Qwen2.5-1.5B and 4×H100 for Qwen2.5-7B. Search-augmented QA runs for 200 iterations, using 4×H100 GPUs for Qwen2.5-3B and 8×H100 for Qwen2.5-7B.Source: §E.1 · p. 18
  • 4 turns and a group size of 5 configure search-augmented QA, with training data size set to 256.Maximum prompt and response lengths are 4096 and 512 tokens, actor learning rate is 1e-6, and mini-batch size is 512. Rewards are 1 for success, 0 for failure, and -0.01 for invalid actions; rollout/validation temperatures are 1.0/0.0, and the KL-divergence loss coefficient is 0.001.Source: §E.1 · p. 18

E.2 Prompts

  • Prompt templates populate semantic placeholders at runtime through Python’s .format(), incorporating task descriptions, step counts, and current observations.History length is 2 for ALFWorld and WebShop, while search-augmented QA receives the full history.Source: §E.2 · p. 18
  • Agent prompts separate step-by-step reasoning in <think> blocks from final decisions in <action> blocks, encouraging chain-of-thought deliberation.Search agents issue queries within <search>, provide answers within <anwser>, and receive retrieved evidence within <information> tags.Source: §E.2 · p. 18

E.3 Performance on Vision-Language Agents

GiGPO outperforms the prompting baseline and GRPO on Sokoban and EZPoints, where agents take sequential actions using visual observations and textual instructions.

E.3 Performance on Vision-Language Agents

  • The evaluation uses 6×6 Sokoban for spatial reasoning and long-term planning, and EZPoints in Gym Cards [39] for symbolic reasoning and multi-step planning.Sokoban requires pushing boxes onto designated goals. EZPoints requires constructing an arithmetic formula step-by-step from two playing cards to reach a target value of 12.Source: §E.3 · p. 19
  • All methods use Qwen2.5-VL-3B-Instruct [78], receiving an RGB image of the current environment state and a textual instruction before taking sequential actions.Source: §E.3 · p. 19
  • 81.0% success on Sokoban and 100% success on EZPoints place GiGPO above the prompting baseline and GRPO on both tasks.These results suggest generalization beyond language-only settings.Source: §E.3 · p. 19

E.4 Orthogonality to Single-Turn Group-Based RL

GiGPO can incorporate complementary single-turn group-based RL techniques. Integrating dynamic sampling and clip-higher from DAPO [20] produces GiGPOdynamic, which outperforms DAPO on WebShop.

E.4 Orthogonality to Single-Turn Group-Based RL

  • GiGPOdynamic integrates dynamic sampling and clip-higher from DAPO [20], which builds on GRPO.Source: §E.4 · p. 20
  • DAPO [20] also improves over GRPO, confirming the benefits of its techniques before their integration into GiGPO.Source: §E.4 · p. 20
  • 75.0% on WebShop is achieved by GiGPOdynamic, outperforming DAPO [20] using Qwen2.5-1.5B-Instruct.This result supports GiGPO's compatibility with complementary single-turn group-based RL advances.Source: §E.4 · pp. 19–20, Table 4 · p. 20

E.5 Sensitivity Analysis on ω · F Reasoning Behavior · F.1 ALFWorld

GiGPO’s WebShop sensitivity analysis favors ω = 0.8 while showing robustness across [0.4, 1.2]. An ALFWorld trajectory illustrates a GiGPO-trained agent finding an egg, heating it, and issuing its placement action.

E.5 Sensitivity Analysis on ω

  • ω = 0.8 yields the best WebShop performance; larger values reduce performance, suggesting excessive step-level emphasis suppresses useful trajectory-level guidance.Increasing ω initially improves performance through added fine-grained step-level rewards.Source: Table 5 · p. 20, §E.5 · p. 20
  • [0.4, 1.2] is the range over which GiGPO remains relatively insensitive to ω in WebShop experiments using Qwen2.5-1.5B-Instruct.The hyperparameter ω balances episode-level advantage AE and step-level advantage AS.Source: §E.5 · p. 20

F Reasoning Behavior

  • The reasoning-behavior section presents a complete multi-turn agent–environment trajectory from Qwen2.5-7B-Instruct trained with GiGPO.The example exposes the agent’s reasoning alongside its interactions with the environment.Source: F Reasoning Behavior · p. 21

F.1 ALFWorld

  • After finding no eggs in the fridge or on countertop 1, the agent checks countertop 2, where it identifies eggs.Its reasoning tracks previously inspected locations and considers alternative search locations.Source: §F.1 · p. 22
  • The agent takes egg 1 from countertop 2, opens microwave 1, and heats the egg using the microwave.The environment explicitly confirms both opening the microwave and heating the egg.Source: §F.1 · pp. 22–23
  • After heating the egg, the agent returns to countertop 1 and issues the action to place the heated egg there.Its reasoning notes that either countertop is acceptable and selects countertop 1.Source: §F.1 · p. 23

F.2 WebShop

This WebShop example traces an agent from a detailed tuxedo-shirt request to clicking Buy Now on a graphic T-shirt. Despite recognizing mismatches in product type and fit, it proceeds by selecting the requested color and size.

F.2 WebShop

  • The agent translates a detailed tuxedo-shirt request into a search query specifying color, size, price, product type, fit, and sleeve length.The request specifies b-blue, xx-large, a price below 40.00 dollars, polyester cotton, an elastic waist, loose/slim fit, long/short sleeves, and regular fit for gym workouts.Source: §F.2 · p. 24
  • The agent opens the first result, a JSPOYOU graphic T-shirt, reasoning that its description might indicate a tuxedo shirt.Its reasoning acknowledges that the item is not an exact match before inspecting it.Source: §F.2 · p. 24
  • The agent clicks Buy Now after selecting b-blue and xx-large, despite previously recognizing that the product type and fit do not match the request.The supplied trajectory ends with the purchase action; no subsequent outcome or reward is shown.Source: §F.2 · pp. 25–26

F.3 Search

This search example identifies Matt Bennett as the man in Ariana Grande’s “One Last Time” video. The agent reaches the answer after an incorrect initial attribution and a second, more specific search.

F.3 Search

  • Asked to identify the man in the video, the agent initially misattributes “One Last Time” to *NSYNC and issues a broad search.The results instead include David Guetta’s “Just One Last Time” and Ariana Grande’s “One Last Time.”Source: §F.3 · p. 26
  • The agent distinguishes the retrieved songs, assumes the question concerns Ariana Grande, and searches again using her name.Although the first results already named Matt Bennett, the agent incorrectly states that no specific name was provided.Source: §F.3 · p. 26
  • The agent’s final answer is Matt Bennett, who plays Ariana Grande’s boyfriend in the “One Last Time” music video.The retrieved description explicitly identifies Bennett as the boyfriend recording with a handheld camera.Source: §F.3 · p. 27

Related papers

Loading 2505.10978v3…