PaperlayerView exact revision on arXiv

Source-linked AI summary

Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Tasks

Shuo He, Lang Feng, Qi Wei, Xin Cheng, Lei Feng, Bo An

arXiv:2602.22817v1cs.LGcs.AI

TL;DR

Stepwise group-based RL can produce biased advantages when grouped steps have inconsistent historical contexts, while relying only on fully consistent groups increases variance. HGPO uses context-aware hierarchical grouping and adaptive weighting to balance these errors, and it outperforms prior RL approaches across ALFWorld and WebShop under matched training conditions.

  • Problem

    Historical context inconsistency can bias stepwise advantage estimation because steps sharing a current state may have different historical contexts, while Oracle steps are rare and small groups can destabilize training.

  • Method

    HGPO assigns each step to hierarchical groups based on historical-context consistency and adaptively aggregates their advantages, weighting more consistent groups more heavily.

  • Results

    HGPO consistently outperforms existing baselines on ALFWorld and WebShop with identical rollouts, the same GPU memory usage, and minimal additional time cost.

  • Takeaways & Limitations

    HGPO provides more reliable stepwise advantage estimates while balancing estimation bias and variance for long-horizon agentic policy optimization.

  • Takeaways & Limitations

    The default weighting parameter is α = 1, and better weighting based on hierarchical-group uncertainty remains an open direction.

Abstract

from arXiv · show

Group-based reinforcement learning (RL), such as GRPO, has advanced the capabilities of large language models on long-horizon agentic tasks. To enable more fine-grained policy updates, recent research has increasingly shifted toward stepwise group-based policy optimization, which treats each step in a rollout trajectory independently while using a memory module to retain historical context. However, we find a key issue in estimating stepwise relative advantages, namely context inconsistency, where steps within the same group may differ in their historical contexts. Empirically, we reveal that this issue can lead to severely biased advantage estimation, thereby degrading policy optimization significantly. To address the issue, in this paper, we propose Hierarchy-of-Groups Policy Optimization (HGPO) for long-horizon agentic tasks. Specifically, within a group of rollout trajectories, HGPO assigns each step to multiple hierarchical groups according to the consistency of historical contexts. Then, for each step, HGPO computes distinct advantages within each group and aggregates them with an adaptive weighting scheme. In this way, HGPO can achieve a favorable bias-variance trade-off in stepwise advantage estimation, without extra models or rollouts. Evaluations on two challenging agentic tasks, ALFWorld and WebShop with Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct, show that HGPO significantly outperforms existing agentic RL methods under the same computational constraints. Code is available at https://github.com/langfengQ/verl-agent/tree/master/recipe/hgpo.

1 INTRODUCTION

Stepwise policy optimization improves granularity but suffers when grouped steps have inconsistent historical contexts, biasing advantage estimates. HGPO addresses this with hierarchical, context-aware grouping and adaptive weighting, achieving strong benchmark performance under matched constraints.

  • Background: Stepwise optimization treats each rollout step independently while using memory to retain historical context, enabling scalable long-horizon RL training.This framework is contrasted with trajectory-wise optimization in Figure 1.
  • Motivation: Historical context inconsistency arises when steps sharing a current state differ in their historical contexts, biasing relative advantage estimates.The resulting bias can degrade policy optimization.
  • Motivation: Pilot analysis finds notable advantage-estimation bias for both trajectory-level and step-level methods, with larger bias at the trajectory level.The analysis compares estimates against Oracle groups whose steps share current states and identical historical contexts.
  • Approach: HGPO assigns steps to multiple hierarchical groups according to historical-context consistency and adaptively aggregates their group advantages.More context-consistent groups receive larger weights, while hierarchical grouping improves data utilization and reduces variance.
  • Results: HGPO consistently outperforms existing baselines on ALFWorld and WebShop while using identical rollouts, the same GPU memory, and minimal additional time.The evaluation uses Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct.

2 RELATED WORK

Research on LLM agents spans diverse environments and increasingly uses reinforcement learning for sustained, multi-turn decision-making. Recent work includes long-horizon and stepwise policy optimization frameworks.

  • LLM-based agents: LLM-based agents have been applied to device control, code generation, games, robotics, and other interactive domains.These applications extend autonomous decision-making beyond conventional text generation.
  • Reinforcement learning: Reinforcement learning has adapted LLM agents to dynamic environments, progressing from classic value-based methods to PPO, AWR, and newer approaches.Applications include mobile control, embodied tasks in ALFWorld, and card games.
  • Long-horizon agentic RL: Long-horizon agentic RL extends LLMs from single-turn generation to multi-turn decision-making with planning, reasoning, and memory capabilities.Applications include code generation, engineering, and GUI interaction.
  • Policy optimization: Recent policy optimization methods include long-horizon frameworks that optimize multi-turn rollouts and stepwise methods that treat each step independently.These methods form the context for more fine-grained policy optimization.

3 PRELIMINARIES

Long-horizon agentic tasks require multi-turn interaction under sparse delayed rewards. Policy optimization can operate over full trajectories or individual steps, with step-level grouping providing finer credit assignment.

  • Problem setup: An agent observes environment states and generates textual actions across multiple interaction turns to complete a task.The setup uses a scalar reward provided only at the final trajectory step.
  • Optimization frameworks: Trajectory-wise optimization concatenates full interaction histories, but growing turn counts rapidly increase context length and limit scalability.Stepwise optimization instead treats steps independently while retaining historical context through memory.
  • Group-based RL: Group-based RL computes advantages from sampled trajectory-group statistics without an additional value function.GRPO was originally designed for single-turn tasks and can be adapted to stepwise optimization.
  • Step-level estimation: Step-level groups cluster steps with identical current states, enabling advantages to be computed for finer within-trajectory credit assignment.This contrasts with trajectory-level computation, which assigns one advantage to every step in a trajectory.
  • Step-level estimation: The step-level estimator provides more fine-grained and effective credit assignment across steps within the same trajectory.

4 TRAINING AGENTS WITH HGPO FOR LONG-HORIZON AGENTIC TASKS

HGPO addresses context inconsistency in stepwise group advantage estimation by organizing same-state steps into hierarchical groups based on historical context, then adaptively combining their advantages. This offline procedure aims to balance estimation bias and variance without additional models or data collection.

  • Context inconsistency: Step-level groups can contain identical current states with different historical contexts, making their effective prompts differ and potentially biasing relative advantages.Oracle steps share both the current state and historical context, but restricting optimization to them is impractical because they are limited.
  • Context-aware hierarchical grouping: HGPO first groups steps sharing the same current state, then constructs multiple hierarchical groups according to progressively richer historical contexts.The hierarchy is defined through k-step context operators that return the historical states preceding the current state.
  • Context-aware hierarchical grouping: The hierarchy-of-groups procedure operates offline using hashmap lookups over existing rollouts, without additional models or data collection.When K = 0, the hierarchy degenerates to step-level grouping.
  • Adaptive weighting advantage estimation: HGPO estimates each hierarchical group’s advantage and adaptively aggregates information across groups, preferring higher-level groups with stronger context consistency.The final estimate fuses advantage information along the hierarchy rather than relying on a single grouping level.
  • Bias-variance trade-off: Theoretical bounds place HGPO’s bias between Oracle and trajectory-level estimates and its variance between step-level and trajectory-level bounds, yielding an intermediate trade-off.The stated bounds are bK ≤ Bias[AH] ≤ b0 ≤ BT and v0/(K + 1) ≤ Var[AH] ≤ vK ≤ VT.

5 EXPERIMENTS

HGPO is evaluated on ALFWorld and WebShop against competitive baselines, with results, efficiency, group utilization, parameter sensitivity, and ablations examining its effectiveness under matched training settings.

  • Experimental Results: HGPO consistently outperforms GiGPO across ALFWorld and WebShop settings, including both Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct.With Qwen2.5-1.5B-Instruct, average improvements over GiGPO are 4.01%, 1.08%, 2.81%, and 4.36% across the reported task and K settings.
  • Experimental Results: HGPO shows larger gains with Qwen2.5-1.5B-Instruct than with Qwen2.5-7B-Instruct, especially when rollouts are longer and advantage-estimation bias is larger.Average gains are 3.41% versus 0.74% for K = 2 and 2.72% versus 0.13% for K = 4.
  • Experimental Results: Increasing K improves both GiGPO and HGPO, indicating that HGPO benefits from retaining richer historical context.The reported result states that HGPO scales effectively as memory size increases.
  • Further Analysis: Higher-context groups contain fewer usable steps, while 0-context groups include a larger fraction of large groups and higher K shifts mass toward smaller groups.Oracle steps sharing identical historical contexts typically form small groups, increasing advantage-estimation variance.
  • Further Analysis: HGPO adds approximately 0.425 s versus GRPO and 0.472 s versus GiGPO per iteration, corresponding to less than 0.001% of total execution time.All methods use a single actor LLM and have identical GPU memory usage and rollout costs; HGPO’s main addition is advantage estimation.
  • Parameter Analysis: Adaptive weighting exhibits a bias-variance trade-off: α = 0 performs comparatively better at K = 2, whereas α = 1 and α = 2 improve when K increases to 4.The paper uses α = 1 as the default and notes that excessively emphasizing higher-level groups is not always optimal.

6 CONCLUSION

HGPO mitigates historical context inconsistency in long-horizon agentic reinforcement learning through context-aware hierarchical grouping and adaptive weighting advantage estimation. Experiments on ALFWorld and WebShop show substantial improvements over prompt-based agents and prior RL approaches.

  • HGPO is designed to mitigate historical context inconsistency in long-horizon agentic tasks.
  • The method combines context-aware hierarchical grouping with adaptive weighting advantage estimation.
  • HGPO provides a better advantage estimate for policy optimization.
  • HGPO substantially outperforms prompt-based agents and prior RL approaches on ALFWorld and WebShop.

B MORE DETAILS AND PROOF FOR THEOREM

The analysis models HGPO as an interpolation between step-level and oracle advantage estimators. Under stated bias and variance assumptions, the resulting estimator offers a bias-variance trade-off, while tighter bounds remain an open direction.

  • HGPO analyzes the k-th hierarchical-group advantage estimator through its bias and variance.
  • The assumptions order bias from trajectory-level and step-level estimation through hierarchical groups, while variance increases toward higher context depth.Trajectory-level quantities are denoted BT and VT; step-level quantities are b0 and v0.
  • HGPO trades off bias between the step-level estimator at k = 0 and the oracle estimator at k = K.
  • HGPO interpolates between step-level and oracle estimators in bias and variance, achieving a better trade-off.
  • Tighter bounds on HGPO’s bias and variance remain an open direction.

C.1 COMPARING METHODS •

The comparison covers closed-source LLM baselines, prompting-based agents, and reinforcement-learning methods, while the environments test multi-turn task completion and interactive shopping.

  • GPT-4o and Gemini-2.5-Pro are included as closed-source LLM baselines for multi-turn agentic tasks.
  • ReAct uses interleaved reasoning and acting, whereas Reflexion adds self-reflection and iterative improvement.
  • ALFWorld contains 4,639 household-task instances across six activity categories.
  • WebShop requires agents to search, navigate, and purchase suitable products on a simulated HTML shopping website.The environment contains over 1.1 million products and 12k user instructions.

C.3 DETAILS OF TRAINING

The experiments implement GiGPO and HGPO under shared training settings and report results across ALFWorld and WebShop. Training uses fixed episode limits, reward rules, and H100 GPU configurations for two Qwen model sizes.

  • GiGPO and HGPO are implemented on a new Verl-agent version and evaluated under the same training settings used for fair comparison.Results are also reported using the old Verl-agent version.
  • ALFWorld experiments use identical method hyperparameters, up to 50 environment steps per episode, and rule-based rewards of 10 for success and 0 for failure.Invalid actions receive a -0.1 penalty.
  • Prompt templates are provided separately for ALFWorld and WebShop agents.
  • WebShop experiments use identical method hyperparameters, up to 30 environment steps per episode, and rule-based rewards of 10 for success and 0 for failure.Invalid actions receive a -0.1 penalty.
  • Qwen2.5-1.5B-Instruct experiments use two NVIDIA H100 GPUs, while Qwen2.5-7B-Instruct experiments use four.Each experiment runs for 160 training iterations with a validation set of 512 examples.

C.4 TRAINING METRICS •

The training metrics track action quality, optimization behavior, policy divergence, update clipping, reward, and task completion. The prompts provide historical context and explicitly separate reasoning from final action selection.

  • Training metrics: Mean advantages measure whether selected actions outperform the group average, with positive stable values indicating better action selection.Large fluctuations suggest unstable training.
  • Training metrics: Policy gradient loss is the main policy-update signal; smooth decreases indicate stable learning, while sharp or excessive changes may harm stability.
  • Training metrics: KL divergence constrains how quickly the new policy departs from the old policy, balancing steady learning against divergence or overly slow updates.
  • Training metrics: Clip fraction measures how many gradients are clipped, with moderate values suggesting stable updates and high values indicating restricted unstable updates.
  • Training metrics: Mean reward and episode success rate directly track average returns and task completion, with increases indicating improving performance.
  • Prompt configuration: Agent prompts dynamically fill task, step, and observation slots, include two steps of history, and separate chain-of-thought reasoning from the final action.

D.2 TRAINING DYNAMICS

Training dynamics compare HGPO, GiGPO, and GRPO across six optimization and performance metrics on ALFWorld and WebShop. The reported results use task-specific success or score measures, while hierarchical group-size distributions are also examined.

  • Training dynamics: Figures 7 and 8 compare HGPO, GiGPO, and GRPO across six training metrics on ALFWorld and WebShop.The metrics are mean advantages, policy gradient loss, KL loss, policy gradient clip fraction, mean reward, and episode success rate.
  • Evaluation metrics: Table 7 reports ALFWorld success rates and WebShop average task scores and success rates, with most results averaged over 3 random testing seeds.
  • Training dynamics: HGPO achieves more stable and efficient policy optimization, with a balanced trajectory compared with the more aggressive GiGPO dynamics.The reported comparison concerns training behavior on WebShop with Qwen2.5-1.5B-Instruct.
  • Reproducibility: The training and evaluation W&B logs are publicly available for the reported dynamics.
  • Hierarchical groups: Hierarchical group-size distributions are reported for K = 4 on both ALFWorld and WebShop.
Loading 2602.22817v1…