Source-linked AI summary
Hierarchical Reinforcement Learning with Stable Temporal Abstraction for Language Model Agents
Shayan Mohajer Hamidi, Yize Cheng, Yuanda Xu, Zhengze Zhou, Alborz Geramifard
TL;DR
An explicit hierarchy does not ensure stable subgoal durations: boundary policies may switch too early or retain stale subgoals. STAC applies constraint costs to sampled boundary decisions while leaving other learning signals unchanged. It improves success over a hierarchical host across benchmarks and backbones, including Qwen3 gains of 8.1 points on ALFWorld and 7.9 on WebShop.
Problem
An explicit hierarchy does not ensure stable temporal abstraction: boundary policies may replace subgoals too often or retain them after they are no longer appropriate.
Source: §1 · p. 2Method
STAC represents premature replanning and stale-subgoal persistence as constraint costs applied only to the sampled boundary decision, leaving rewards, critic targets, and subgoal and action advantages unchanged.
Source: §1 · p. 2Results
Takeaways & Limitations
Constraining the boundary yields longer, valid multi-turn subgoals and accompanies improved ALFWorld and WebShop performance across two backbones.
Source: §1 · p. 2Matched comparisons set HiPER keep penalties to zero and use heuristic duration values, so they do not establish whether tuned penalties recover the gains or whether those values are optimal.
Source: §6 · p. 7
Abstract
from arXiv · showhide
Hierarchical reinforcement learning improves long-horizon control by organizing primitive actions around persistent subgoals and assigning credit at multiple temporal scales. Recent hierarchical language agents bring these benefits to interactive tasks by explicitly separating subgoal planning from action execution. We observe, however, that an explicit hierarchy does not by itself determine how stable the resulting temporal abstraction is: the learned boundary policy may replace the subgoal almost every turn, making it effectively transient, or retain a subgoal after it has stopped being appropriate. We call this temporal abstraction instability. We propose Stable Temporal Abstraction via Constrained Optimization (STAC), a constrained boundary-policy optimization method that represents premature replanning and stale persistence as constraint costs. STAC applies the resulting Lagrangian costs only to the sampled boundary decision, leaving the underlying algorithm's rewards, critic targets, subgoal advantages, and primitive-action advantages unchanged. Across two backbones and two benchmarks, STAC improves success over a strong hierarchical baseline by $8.1$ and $7.9$ points on ALFWorld and WebShop with Qwen3-0.6B, and by $23.5$ and $15.8$ points with Llama-3.2-1B-Instruct.
1 Introduction
Hierarchical language agents explicitly separate subgoal planning from action execution, but explicit structure alone does not ensure stable subgoal durations. STAC constrains boundary decisions, and the introduction reports gains over a released hierarchical host across ALFWorld and WebShop.
1 Introduction
- Hierarchical reinforcement learning uses temporally extended subgoals and lower-level actions, while recent language agents apply this separation to interactive tasks.The decomposition can shorten effective planning horizons and separate credit for high-level planning from low-level execution.Source: §1 · pp. 1–2
- Temporal abstraction instability occurs when a boundary policy replaces subgoals too often or retains them after they are no longer appropriate.The paper identifies this as a problem for multi-phase tasks requiring persistence within a phase and well-timed transitions.Source: §1 · p. 2
- STAC represents premature replanning and stale-subgoal persistence as constraint costs applied only to the sampled boundary decision.The reported experiments optimize a fixed-multiplier relaxation; rewards, critic targets, and subgoal and action advantages remain unchanged.Source: §1 · p. 2
- STAC improves ALFWorld and WebShop performance across two backbones; on ALFWorld, gains are 8.1 points with Qwen3-0.6B and 23.5 with Llama-3.2-1B-Instruct.The comparison uses the publicly released HiPER host.Source: §1 · p. 2
2 Hierarchical Setup and Temporal Abstraction Instability
The section defines an interactive hierarchical policy with subgoals that persist across temporal segments and explicit KEEP/SWITCH boundary decisions. It frames useful abstraction as state-dependent duration and describes the host boundary signal that STAC modifies.
2 Hierarchical Setup and Temporal Abstraction Instability
- The experiments use HiPER’s Plan–Execute agent and hierarchical advantage estimator unchanged as the host optimizer.Its single autoregressive actor emits boundary, subgoal, and action spans and is trained with PPO.Source: §2 · pp. 2–3
- The interactive setup treats each turn as an environment action with a resulting reward, and defines the return using a discount factor over the trajectory.The action may be a tool call with arguments, and the state includes the current observation and interaction history.Source: §2 · p. 2
- A subgoal is a textual intermediate objective, and the boundary policy samples KEEP to retain it or SWITCH to generate a new one.Turns under the same subgoal form a temporal segment; KEEP copies the prior subgoal rather than generating another.Source: §2 · pp. 2–3
- HiPER uses two critic heads for boundary planning and execution under the active subgoal; its boundary signal compares replanning with continuing the previous subgoal.The boundary stream is standardized independently across the update batch.Source: §2 · p. 3
- The normalized boundary advantage is the host quantity STAC reads or modifies; subgoal and action advantages and both critic targets remain untouched.Because batch standardization omits subgoal duration, STAC adds duration-dependent costs to the boundary term.Source: §2 · p. 3
- Useful temporal abstraction requires subgoals to span multiple actions when coordination is needed and end when stale; always switching or always keeping undermines that structure.The duration count includes the subgoal-creating turn and subsequent turns since the last valid switch.Source: §2 · p. 3
3 STAC: Stable Temporal Abstraction via Constrained Optimization
STAC frames temporal abstraction as a constrained boundary-policy problem, penalizing both premature replanning and stale persistence. Its reported fixed-multiplier version applies those costs locally to boundary decisions while leaving task rewards and other learning signals unchanged.
3 STAC: Stable Temporal Abstraction via Constrained Optimization
- STAC models premature replanning and stale-subgoal persistence as bounded constraint costs, without requiring a universal duration interval.The costs may be fixed, state-dependent, or learned.Source: §3.1 · p. 4
- The reported implementation holds both multipliers at 0.1 rather than updating them with the adaptive primal–dual rule.The duration heuristics are dmin = 2 and dmax = 8; dmax is horizon-informed, not a per-segment upper bound.Source: §3.1 · p. 4
- The method combines task and constraint terms into a sampled boundary advantage and uses it in the host’s ordinary actor objective.For PPO, the combined advantage enters the standard clipped surrogate, and only boundary-decision tokens receive the temporal-abstraction terms.Source: §3.1 · p. 4
4 Experiments
The experiments test STAC on ALFWorld and WebShop using the HiPER Plan–Execute architecture as the hierarchical host. They evaluate Qwen3-0.6B and repeat both benchmarks with Llama-3.2-1B-Instruct under identical hyperparameters and seeds.
4 Experiments
- ALFWorld covers long-horizon household tasks, while WebShop tests navigation of a simulated e-commerce site against shopping instructions.Both benchmarks have sparse success rewards and require persistent subgoals alongside timely replanning.Source: §4 · p. 5
- The actor and critic use Qwen3-0.6B by default, and both benchmarks are repeated with Llama-3.2-1B-Instruct under identical hyperparameters and seeds.STAC is instantiated on the publicly released HiPER Plan–Execute implementation; it leaves subgoal/action streams and critic targets unchanged.Source: §4 · p. 5
- Each run uses 128 independent environments per update for 150 updates, with at most 50 environment turns and 128 held-out episodes evaluated every five updates.The evaluation also includes the base checkpoint before training.Source: §4 · p. 5
- Comparisons include flat-policy baselines GRPO and GiGPO, alongside HiPER as the hierarchical host whose boundary advantage STAC modifies.GRPO uses group-normalized trajectory rewards; GiGPO adds step-level relative advantages.Source: §4 · p. 5
5 Results
STAC improves success over released HiPER on both benchmarks with Qwen3-0.6B and Llama-3.2-1B-Instruct. It also produces multi-turn subgoals with high structured-output validity while reducing premature-replanning and stale-persistence violations.
5 Results
- +8.1 points on ALFWorld and +7.9 points on WebShop at the best checkpoint with Qwen3-0.6B; Llama-3.2-1B-Instruct gains +23.5 and +15.8 points, respectively.For Qwen3-0.6B, the reported final-checkpoint gains are +7.8 on ALFWorld and +11.0 on WebShop; for Llama, ALFWorld’s final-checkpoint gain is +23.2 and WebShop’s is +15.2.Source: §5 · pp. 5–6
- Figure 1 tracks the two per-step temporal-abstraction constraint surrogates across training, averaging three seeds per curve.HiPER’s premature-replanning rate stays high, whereas STAC reduces it and holds both rates low; fixed multipliers make these diagnostics rather than enforced targets.Source: Figure 1 · p. 6, §5 · p. 6
- STAC ends with switch rates of 0.150, 0.266, and 0.142 across its three seeds, with structured-output validity of at least 0.996 on every seed.The switch rates correspond to multi-turn subgoals; one released HiPER seed ends with validity 0.016.Source: §5 · p. 6
- STAC reduces premature-replanning violations by 7.6× on ALFWorld and 4.0× on WebShop, while ending lower on stale persistence in both.Mean segment lengths settle at 4.02 and 2.91 turns; with fixed multipliers, these are induced violation levels, not enforced budgets.Source: §5 · p. 6
6 Additional Robustness Checks
The robustness checks include a matched local ALFWorld comparison on the Qwen2.5-1.5B-Instruct HiPER setting and a fixed-penalty sweep on Qwen3-0.6B. The reported scope leaves alternative host controls and duration choices unresolved.
6 Additional Robustness Checks
- +3.9 points at the best checkpoint and +4.7 points at the final checkpoint for duration-based STAC in the matched local Qwen2.5-1.5B-Instruct ALFWorld comparison.This is a controlled local comparison, not a seed-matched reproduction of every upstream aggregate.Source: §6 · p. 7
- Increasing the shared penalty monotonically reduces measured violations over the tested range, while task success peaks at the validation-selected µ = 0.1 and declines under stronger penalties.The selected value is reused unchanged for WebShop and Llama, coupling those reported peak values to the ALFWorld validation sweep.Source: §6 · p. 7
- The comparison does not establish whether a separately tuned HiPER KEEP penalty or a hard first-turn SWITCH block could recover the same gains.The study also does not sweep dmin/dmax; these are prespecified heuristics rather than uniquely justified optima.Source: §6 · p. 7
- The incremental task-success benefit of the stale cost remains unresolved, although it selectively reduces stale persistence.Broader hosts, more seeds, and alternative controls are left for future work.Source: §6 · p. 7
7 Conclusion
STAC addresses temporal abstraction instability with costs applied only to sampled boundary decisions, and improves best-checkpoint success over the released hierarchical host. The reported experiments use fixed multipliers, while adaptive dual optimization remains future work.
7 Conclusion
- Best-checkpoint success gains are 8.1 and 7.9 points with Qwen3-0.6B, and 23.5 and 15.8 points with Llama-3.2-1B-Instruct.The conclusion reports these gains over the released hierarchical host.Source: §7 · p. 7
- STAC prices premature replanning and stale persistence on the sampled boundary decision, leaving rewards, critic targets, and subgoal and action advantages unchanged.The reported runs hold the multipliers fixed; adaptive dual optimization under prescribed budgets is left for future work.Source: §7 · p. 7, A Implementation, Experimental… · p. 10
A Implementation, Experimental Details, and Broader Impact
- STAC can be inserted after advantage normalization in an on-policy hierarchical optimizer using a boundary decision, subgoal-age counter, boundary advantage, and constraint-rate estimates.The behavior-preserving default sets both multipliers to zero.Source: A Implementation, Experimental… · p. 10
- The main results select each seed’s highest-overall-validation checkpoint, while step-150 and late-window summaries expose checkpoint-selection sensitivity.The main result reports every category from the selected checkpoint.Source: A Implementation, Experimental… · p. 10
- Matched local HiPER and STAC runs set HiPER’s two keep penalties to zero while retaining environment rewards and the malformed-output reward term.ALFWorld uses a 2048-token prompt limit and WebShop uses 4096 tokens.Source: A Implementation, Experimental… · p. 10
A.1 Heuristic choice of the two duration thresholds
The duration heuristics discourage uniform one-turn collapse and stale persistence without imposing a universal duration bound. The section also explains the host’s boundary advantage, STAC’s decision-local update, and its relation to classical termination costs.
A.1 Heuristic choice of the two duration thresholds
- The lower threshold sets d_min = 2 to discourage uniform one-turn collapse, while explicitly allowing that some one-turn segments may be appropriate.The upper reference is horizon-informed: T/K̄ estimates an average fair share, not an individual-segment bound; the chosen value 8 is a prespecified heuristic rather than a tuned optimum.Source: §A.1 · pp. 10–11
- With d_max = 8, late-training 90th-percentile segment lengths are 8.27 on ALFWorld and 5.40 on WebShop, while mean lengths are 4.02 and 2.91 turns.The authors treat these as post hoc diagnostics, leave a threshold sweep to future work, and do not evaluate a task- or state-dependent adaptive form.Source: §A.1 · p. 11
B Host Advantage Estimator
- The host’s high-level critic estimates returns from selecting a new subgoal, while its low-level critic estimates returns from continuing the active subgoal; their value difference supplies boundary credit.There is no separately learned boundary-value head.Source: B Host Advantage Estimator · p. 11
C Optimization Details
- STAC adds sampled Lagrangian costs only to the boundary span after host advantage estimation, leaving the reward stream, critic targets, and other turns’ advantages unchanged.The update uses ordinary on-policy actor and critic updates; the reported runs hold multipliers fixed rather than applying the adaptive primal–dual step.Source: C Optimization Details · p. 12
D Classical Termination Regularization
- Unlike one-sided deliberation costs that can favor indefinitely long options, STAC constrains both premature replanning and stale persistence at the boundary-policy level.The method is described as compatible with GAE, PPO, and other on-policy hierarchical objectives.Source: D Classical Termination Regularization · pp. 12–13
E Related Hierarchical Language-Agent Methods
Hierarchical language-agent methods vary in their hierarchy, training, and credit assignment. STAC addresses boundary stability as an orthogonal issue, consuming host-provided boundary credit rather than prescribing a planner, executor, or trajectory representation.
E Related Hierarchical Language-Agent Methods
- ArCHer separates off-policy turn-level temporal-difference learning from token-level optimization, while GLIDER trains a planner–controller hierarchy through behavior cloning, offline RL, and offline-to-online adaptation.ArCHer reports improved sample efficiency relative to fully on-policy multi-turn optimization; GLIDER evaluates transfer and generalization on ScienceWorld and ALFWorld.Source: E Related Hierarchical Language-Agent… · p. 13
- STEP-HRL folds intra-subtask interactions into compact local progress, whereas HiMAC separately optimizes blueprint-level and execution-level relative advantages.STEP-HRL uses shared policy parameters with distinct critics; HiMAC alternates macro exploration and executor adaptation.Source: E Related Hierarchical Language-Agent… · p. 13
- HIPIF folds completed-subgoal interaction history into a compact summary, while Multi2 assigns subgoals and atomic actions to role-specialized agents.HIPIF combines terminal reward with subgoal-oriented process rewards; Multi2 trains its high-level agent with supervised fine-tuning and its low-level agent with offline and online RL.Source: E Related Hierarchical Language-Agent… · p. 13
- DreamPhase improves planning through offline imagination and uncertainty-guided plan selection, a distinct target compatible with STAC’s boundary constraint.DreamPhase concerns what the agent should plan; STAC concerns when an active subgoal should be replaced.Source: E Related Hierarchical Language-Agent… · pp. 13–14
- STAC targets unstable switching or stale persistence independently of how a host defines or trains its hierarchy, and modifies boundary credit rather than prescribing a planner or executor.It also does not prescribe the host’s trajectory representation or base return estimator.Source: E Related Hierarchical Language-Agent… · p. 14
F Additional Results
Across eight benchmark–checkpoint cells, STAC improves on HiPER, with seven comparisons reaching p_W < 0.05. Additional diagnostics show longer segments, stronger multi-phase performance, and differences in hierarchy validity across seeds.
F Additional Results
- STAC’s median segment length rises from 1.55 to 3.03 turns on ALFWorld and from 1.00 to 2.30 on WebShop, while one-turn segment shares fall from 0.55 to 0.27 and 0.80 to 0.36.The 90th percentile also rises from 4.27 to 8.27 on ALFWorld and from 2.08 to 5.40 on WebShop.Source: F Additional Results · p. 14
- Qwen3-0.6B succeeds on 1/384 ALFWorld episodes and 3/256 WebShop episodes before RL training, with binary success of 0.3 ± 0.5% and 1.2 ± 1.7%.The graded WebShop score is 6.0 ± 1.9%, but Table 1 reports binary success consistently.Source: F Additional Results · p. 14
- STAC improves on HiPER in all eight benchmark–checkpoint cells, and seven reach p_W < 0.05; the exception is Qwen3-0.6B ALFWorld final checkpoint at p_W = 0.0547.The comparisons use two-sided unpaired Welch tests derived from three-seed means and standard deviations.Source: F Additional Results · pp. 14–15
- Figure 3 plots Table 1 success rates with per-mean 95% Student-t intervals; overlapping intervals are not themselves tests of differences.At three seeds the intervals are wide, while Table 4’s Welch tests directly compare methods.Source: F Additional Results · p. 14, Figure 3 · p. 15
- Pick-two success rises from 0.231 ± 0.310 to 0.811 ± 0.078 at selected checkpoints, and the largest observed gain is concentrated in the multi-phase category.The wide HiPER spread reflects seed-level bimodality rather than a stable mean.Source: F Additional Results · p. 15
- All STAC seeds retain multi-turn subgoals, nearly perfect structured-output validity, and no final missing boundary masks; one released HiPER seed ends at validity 0.016.That HiPER seed’s validation success is 0.812, so aggregate accuracy alone does not reveal hierarchy state.Source: F Additional Results · p. 15
G Constraint Ablation
The ablation separates two constraint costs: each controls its corresponding boundary violation, while premature replanning is more frequent and suppressing it accounts for most point-estimate gains.
Constraint Ablation
- The short cost reduces premature replanning from 0.553 to 0.085 on ALFWorld and from 0.597 to 0.165 on WebShop; the stale cost reduces stale persistence from 0.147 to 0.045 and from 0.110 to 0.025.Neither cost controls the violation targeted by the other; only the full method is low on both.Source: G Constraint Ablation · pp. 15–16, Table 5 · p. 17, Table 6 · p. 17
- Short only recovers most of STAC’s point-estimate gain, while stale only remains within noise of HiPER on both tasks.Short only reaches 89.2 versus STAC’s 91.4 on ALFWorld and 67.4 versus 69.3 on WebShop; stale only reaches 84.1 versus HiPER’s 83.3 and 62.0 versus 61.4.Source: G Constraint Ablation · pp. 16–17
- The full method has a lower stale-persistence rate than short only, consistent with the stale cost acting as a brake, though this contrast is suggestive at three seeds.Short only has a stale rate of 0.185 ± 0.031 and segments of 4.70 ± 0.61 turns; STAC has 0.098 ± 0.022 and 4.02 ± 0.47 turns.Source: G Constraint Ablation · p. 17
- The violation effects are resolved at three seeds, but task-success differences between arms are not; the ablation supports which cost controls which violation, not separated success outcomes.Short cost changes are significant on both tasks, and the stale cost reduces ALFWorld stale persistence with pW = 0.0095; success comparisons remain unresolved.Source: G Constraint Ablation · p. 17, Table 5 · p. 17
H Final-Checkpoint and Late-Window Robustness
The robustness analysis reports checkpoint-dependent overall outcomes and a Llama Pick2 advantage, while the boundary-mask appendix clarifies that the mask marks boundary-decision tokens at each turn rather than tracking a subgoal.
H Final-Checkpoint and Late-Window Robustness
- On Llama-3.2-1B-Instruct, STAC reaches 80.1 ± 12.8 versus HiPER’s 20.1 ± 9.4 on Pick2 at the best checkpoint, and 61.7 ± 16.9 versus 8.3 ± 10.4 at update 150.Heat is 0.0 for every seed of both methods, while Cool is 0.0 everywhere except one STAC seed at the final checkpoint.Source: H Final-Checkpoint and Late-Window… · p. 18, Table 7 · p. 19, Table 8 · p. 19
- STAC’s final ALFWorld accuracy is 0.893 ± 0.035 versus HiPER’s 0.815 ± 0.036, and its final-overall difference is +0.078; the late-overall difference is +0.095.The late-window diagnostic averages updates 120–150 and is Qwen3-0.6B only; the final-overall comparison has pW = 0.0547.Source: H Final-Checkpoint and Late-Window… · pp. 17–19, Table 8 · p. 19, Table 9 · p. 20
- Best-checkpoint steps differ across methods and seeds, so the discrepancy between main and appendix summaries reflects checkpoint-selection sensitivity rather than a changed method ranking.HiPER’s best-checkpoint steps are 90/145/145, compared with STAC’s 120/145/145.Source: H Final-Checkpoint and Late-Window… · p. 19
I Boundary-Mask Representation
- The boundary mask is reconstructed from every turn’s response and marks where that turn writes its boundary decision; an invalid boundary block produces an all-zero row.It does not identify the active subgoal or persist for that subgoal’s lifetime; subgoal tokens use a separate mask.Source: I Boundary-Mask Representation · pp. 19–20