Source-linked AI summary
HiPER: Hierarchical Reinforcement Learning with Explicit Credit Assignment for Large Language Model Agents
Jiangweizhi Peng, Yuanxin Liu, Ruida Zhou, Charles Fleming, Zhaoran Wang, Alfredo Garcia, Mingyi Hong
TL;DR
Long-horizon, sparse-reward LLM-agent tasks challenge flat single-time-scale RL and its credit assignment. HiPER separates planning from execution and introduces HAE, achieving state-of-the-art results on ALFWorld and WebShop while providing theoretical guarantees for its estimator.
Problem
Long-horizon interactive tasks with sparse, delayed rewards remain challenging because flat RL must assign credit across extended action trajectories without explicit temporal abstraction.
Method
HiPER factorizes the agent into a high-level planner and low-level executor, using HAE to assign credit across subgoal segments and individual actions.
Results
97.4% ALFWorld success rate and 83.3% WebShop success rate are achieved with Qwen2.5-7B-Instruct, alongside unbiasedness and variance-reduction guarantees for HAE.
Takeaways & Limitations
Explicitly modeling and optimizing multi-timescale agent behavior is presented as important for reliable RL training on long-horizon tasks with sparse feedback.
Takeaways & Limitations
The theoretical guarantees assume conditions including exact value baselines and λ_low = λ_high = 1.
Abstract
from arXiv · showhide
Training LLMs as interactive agents for multi-turn decision-making remains challenging, particularly in long-horizon tasks with sparse and delayed rewards, where agents must execute extended sequences of actions before receiving meaningful feedback. Most existing reinforcement learning (RL) approaches model LLM agents as flat policies operating at a single time scale, selecting one action at each turn. In sparse-reward settings, such flat policies must propagate credit across the entire trajectory without explicit temporal abstraction, which often leads to unstable optimization and inefficient credit assignment. We propose HiPER, a novel Hierarchical Plan-Execute RL framework that explicitly separates high-level planning from low-level execution. HiPER factorizes the policy into a high-level planner that proposes subgoals and a low-level executor that carries them out over multiple action steps. To align optimization with this structure, we introduce a key technique called hierarchical advantage estimation (HAE), which carefully assigns credit at both the planning and execution levels. By aggregating returns over the execution of each subgoal and coordinating updates across the two levels, HAE provides an unbiased gradient estimator and provably reduces variance compared to flat generalized advantage estimation. Empirically, HiPER achieves state-of-the-art performance on challenging interactive benchmarks, reaching 97.4\% success on ALFWorld and 83.3\% on WebShop with Qwen2.5-7B-Instruct (+6.6\% and +8.3\% over the best prior method), with especially large gains on long-horizon tasks requiring multiple dependent subtasks. These results highlight the importance of explicit hierarchical decomposition for scalable RL training of multi-turn LLM agents.
1. Introduction
HiPER addresses long-horizon, sparse-reward LLM-agent training by making implicit subgoal structure explicit and aligning hierarchical credit assignment with planning and execution. It reports state-of-the-art performance on ALFWorld and WebShop.
- Long-horizon sparse-reward tasks remain difficult for flat RL policies that select actions at a single time scale.
- Successful agent trajectories organize into temporally extended subgoal segments, such as locating, cleaning, and storing a cup.
- HiPER separates high-level subgoal planning from low-level action execution through a Plan–Execute interface.
- The interface emits switching decisions, subgoals, and primitive actions within a structured autoregressive LLM policy.
- HAE assigns credit across subgoal segments and individual actions, aligning learning signals with the two-level decision structure.
- 97.4% ALFWorld success rate and 83.3% WebShop success rate are reported for HiPER with Qwen2.5-7B-Instruct.
2. Related Work
Prior work includes classical hierarchical reinforcement learning and recent reinforcement learning methods for interactive LLM agents. The latter generally models agents as flat turn- or token-level policies while targeting optimization and credit-assignment challenges.
- Classical hierarchical RL uses temporally extended options with intra-option policies and termination, forming a semi-Markov hierarchy.
- Recent interactive-agent RL methods typically model LLM agents as flat turn- or token-level policies while improving optimization and credit assignment.
3. Preliminaries
The preliminaries define interactive LLM-agent RL as episodic decision-making over textual observations, textual actions, rewards, and state transitions. The hierarchical formulation adds high-level options and low-level actions, with option boundaries determining planning segments.
- At each environment step, the agent receives a formatted textual state observation, produces a token-sequence action, receives a scalar reward, and transitions to the next state.
- The flat objective treats each interaction as selecting a primitive action at a single time scale under discount factor γ.
- The hierarchical formulation represents each turn with a high-level option, such as a subgoal, and a low-level action.
- Planning segments are delimited by boundary turns where the high-level option terminates or switches, while low-level actions are generated at every turn.
- The hierarchical trajectory records states, subgoals, options, actions, and rewards, with the previous subgoal carried forward when no switch occurs.
- The hierarchical objective explicitly separates high-level planning from low-level execution, providing the basis for the Plan–Execute interface and learning algorithm.
4. Method
HiPER makes hierarchical decisions explicit through a Plan–Execute interface and trains them with coupled advantage estimates at planning and execution timescales. HAE partitions trajectories into subgoal segments, assigns level-specific credit, and provides theoretically justified gradient estimation.
- Plan–Execute framework: HiPER implements hierarchical RL with a persistent high-level subgoal, explicit switching decisions, and low-level primitive actions generated by one autoregressive LLM.The structured output contains SWITCH or KEEP, a subgoal, and an environment action.
- Policy gradient: The Plan–Execute policy gradient decomposes learning into coupled switching, subgoal-selection, and primitive-action components operating at different timescales.The high-level process optimizes switching and subgoal decisions, while the low-level process optimizes actions within each subgoal segment.
- Hierarchical advantage estimation: HAE constructs low-level action advantages within fixed-subgoal segments and high-level planning advantages by aggregating rewards over each segment as a macro-step.Low-level estimates use turn-level residuals, while high-level estimates use segment-level rewards and duration discounts at switching boundaries.
- Hierarchical advantage estimation: The low-level estimator bootstraps its final-step residual to the next boundary’s high-level value, coupling execution credit across adjacent subgoal segments.This keeps low-level value estimates consistent with high-level boundary returns and propagates learning signals across segments.
- Theoretical properties: HAE is unbiased up to standard bootstrapping and critic-approximation errors and achieves variance reduction relative to flat GAE by exploiting Plan–Execute trajectory structure.With Monte Carlo estimation and perfectly learned critics, the resulting stochastic gradient estimator is unbiased.
5. Experiments
Across ALFWorld and WebShop, HiPER delivers stronger performance, faster convergence, and more stable training than flat baselines, particularly on long-horizon tasks with dependent subtasks. Ablations further show that Plan-Execute prompting improves trained baselines, while HiPER remains strongest overall.
- Main results: Standard RL baselines struggle most on tasks requiring multiple sequential subtasks, such as Pick2 and Look.For the 1.5B model, PPO, RLOO, and GRPO remain below 60% on these categories, versus above 70% on simpler Pick tasks.
- Main results: HiPER achieves superior final performance across ALFWorld and WebShop for both model sizes, with especially large gains on challenging task categories.With Qwen2.5-7B, it reaches 97.4% overall success on ALFWorld and 83.3% on WebShop, exceeding GiGPO by around 7% on both benchmarks.
- Training dynamics: HiPER reaches high ALFWorld validation success faster than PPO and GRPO, achieving a 2.8× sample-efficiency gain for the 7B model.HiPER exceeds 80% success in about 50 steps, while PPO and GRPO require roughly 140 steps.
- Training dynamics: HiPER exhibits more stable learning dynamics, with smaller oscillations and fewer sharp regressions than flat baselines during training.The reported comparison is especially pronounced against critic-free GRPO.
- Results analysis: HiPER learns meaningful subgoals from outcome-only rewards through an exploratory phase followed by more stable switching and longer subgoal segments.Switching becomes frequent early in training, then frequency and segment length stabilize as the agent commits to subgoals.
- Ablation on Plan-Execute: Plan-Execute prompting improves trained baseline performance, while HiPER remains best overall with a +4.2% advantage over GiGPO using Plan-Execute.Without RL training, Plan-Execute reduces initial overall success from 8.3% to 2.9%, but training with the structure generally improves final performance.
6. Conclusion
HiPER explicitly separates planning and execution and matches this structure with hierarchical credit assignment. The framework is presented as improving learning stability and benchmark success for long-horizon tasks with sparse feedback.
- HiPER separates high-level planning from low-level execution through a Plan-Execute interface.Its hierarchical policy is optimized with a matching hierarchical advantage estimator.
- Hierarchical advantage estimation couples within-segment credit assignment with boundary-to-boundary progress signals.
- HiPER delivers more stable learning and higher success rates on interactive benchmarks.
- Explicitly modeling and optimizing multi-timescale agent behavior is presented as important for scaling RL to long-horizon tasks with sparse feedback.
A.1. Proof of Theorem 4.1
The proof defines a hierarchical policy factorization with switching, subgoal, and action decisions. Its gradient decomposition uses separate score terms and value functions for these decision levels.
- The Plan-Execute policy factorizes each step into switching, subgoal, and primitive action decisions.The subgoal policy is invoked only when the switching decision indicates a new subgoal.
- The policy-gradient score decomposes into switching, subgoal, and action log-probability gradients.
- The proof removes prefix-return terms using the score-function identity because those terms are independent of the current decision.
- Hierarchical values define Q and V functions separately for switching, high-level subgoal, and low-level action decisions.
A.2. Proof of Theorem 4.2
Theorem 4.2 establishes that HAE is unbiased up to GAE bootstrapping and critic approximation errors, while the variance analysis identifies boundary bootstrapping and option-conditioned baselines as reduction mechanisms.
- HAE’s gradient estimator is unbiased up to GAE bootstrapping and critic approximation errors.
- With Monte Carlo estimation and perfectly learned critics, the HAE estimator equals the true policy gradient.
- Under perfect critics, the switching advantage estimator equals the true switching advantage.
- HAE reduces variance through boundary bootstrapping and an option-conditioned baseline.
- The low-level estimator uses a segment return bootstrapped from the next boundary’s high-level value.
C. Implementation Details
The implementation evaluates HiPER with shared benchmark settings and uses Plan-Execute prompting to expose persistent subgoals. The design adds little memory overhead relative to PPO while contrasting with single-timescale ReAct prompting.
- ALFWorld experiments allow up to 50 environment steps per episode, whereas WebShop experiments allow up to 15.
- HiPER has negligible GPU memory overhead because high- and low-level value estimates use one shared critic backbone with two output heads.
- HiPER uses Plan-Execute prompting, while the baseline methods in Table 1 use ReAct prompting.
- ReAct vs Plan-Execute: ReAct selects an action at each step on a single timescale, whereas Plan-Execute maintains a subgoal across multiple turns and decides when to switch.
- Prompt Templates: The ALFWorld and WebShop Plan-Execute templates define short-term subgoals distinct from both the full task and low-level actions.
D.1. Critic Model Size
HiPER uses a separate learned critic for low-variance advantage estimation, creating additional GPU memory use. Experiments examine whether a smaller critic can preserve competitive performance while reducing this overhead.
- Critic Model Size: HiPER requires a separate critic model because its advantage estimates use a learned value function.This adds GPU memory consumption relative to critic-free baselines such as GRPO and GiGPO.
- Critic Model Size: Table 3 compares actor-critic size pairs on ALFWorld using success rates across six task categories and overall success.Results are averaged over 3 random seeds.
- Critic Model Size: A moderately smaller critic keeps HiPER performance competitive and need not strictly match the actor’s size.The authors identify smaller critics as a potential way to reduce memory overhead from the separate critic.
E.3. WebShop Trajectory
The WebShop trajectory shows the agent refining a product-search subgoal through filtering and search actions. It preserves the subgoal across several actions while changing strategy when needed.
- WebShop Trajectory: The task requires finding a bright-aqua men’s shirt meeting material, fit, size, and price constraints.The stated price requirement is below 60.00 dollars.
- WebShop Trajectory: The agent switches to a direct search query containing the requested color, shirt type, size, material, and price constraint.The resulting page reports 50 total results and lists products with prices below 60 dollars.
- WebShop Trajectory: The agent initially keeps the product-search subgoal while clicking the relevant result, color, size, and buy-now controls.The actions include clicking the item, bright aqua, x-small, and buy now.