PaperlayerView exact revision on arXiv

Source-linked AI summary

HiAgent: Hierarchical Working Memory Management for Solving Long-Horizon Agent Tasks with Large Language Model

Mengkang Hu, Tianxing Chen, Qiguang Chen, Yao Mu, Wenqi Shao, Ping Luo

arXiv:2408.09559v1cs.CLcs.AIcs.RO

TL;DR

Long-horizon LLM agents often accumulate redundant action-observation histories, making working-memory management difficult. HiAgent organizes memory around subgoals, summarizes completed subgoals, and selectively retrieves detailed trajectories. Across five long-horizon tasks, it more than doubled overall success rate while using fewer steps, shorter context, and less runtime.

  • Problem

    Existing LLM agents commonly place entire action-observation histories in working memory, creating redundant context that hinders long-horizon decision-making.

    Source: §1 · pp. 1–2

  • Method

    HiAgent uses subgoals as hierarchical memory chunks, retaining detailed history for the current subgoal while summarizing completed subgoals and retrieving past trajectories when needed.

    Source: §1 · p. 2, §3.1 · p. 2, §3.2 · p. 3

  • Results

    HiAgent more than doubled overall success rate across five long-horizon tasks and reduced average completion steps by 3.8 compared with STANDARD.

    Source: §1 · p. 2, §7 · p. 7

  • Takeaways & Limitations

    Hierarchical working-memory management improves both effectiveness and efficiency for long-horizon LLM-agent tasks within the evaluated settings.

    Source: §4.2 · p. 4, §7 · p. 7

Abstract

from arXiv · show

Large Language Model (LLM)-based agents exhibit significant potential across various domains, operating as interactive systems that process environmental observations to generate executable actions for target tasks. The effectiveness of these agents is significantly influenced by their memory mechanism, which records historical experiences as sequences of action-observation pairs. We categorize memory into two types: cross-trial memory, accumulated across multiple attempts, and in-trial memory (working memory), accumulated within a single attempt. While considerable research has optimized performance through cross-trial memory, the enhancement of agent performance through improved working memory utilization remains underexplored. Instead, existing approaches often involve directly inputting entire historical action-observation pairs into LLMs, leading to redundancy in long-horizon tasks. Inspired by human problem-solving strategies, this paper introduces HiAgent, a framework that leverages subgoals as memory chunks to manage the working memory of LLM-based agents hierarchically. Specifically, HiAgent prompts LLMs to formulate subgoals before generating executable actions and enables LLMs to decide proactively to replace previous subgoals with summarized observations, retaining only the action-observation pairs relevant to the current subgoal. Experimental results across five long-horizon tasks demonstrate that HiAgent achieves a twofold increase in success rate and reduces the average number of steps required by 3.8. Additionally, our analysis shows that HiAgent consistently improves performance across various steps, highlighting its robustness and generalizability. Project Page: https://github.com/HiAgent2024/HiAgent .

1 Introduction

LLM agents often retain all historical action-observation pairs, but this creates redundant working memory in long-horizon tasks. HiAgent instead organizes memory around subgoals and summarized observations, improving effectiveness and efficiency across five tasks.

  • 42 vs. 21 average success rate across five long-horizon tasks shows HiAgent’s twofold improvement over STANDARD.HiAgent also improves progress rate and remains more effective as the number of steps increases.Source: Figure 1 · p. 1, §1 · p. 2
  • STANDARD directly appends all historical action-observation pairs to working memory, producing redundant context that hinders coherent strategies and accurate long-horizon predictions.Source: §1 · pp. 1–2
  • The approach is motivated by human problem solving, where complex problems are decomposed into subproblems treated as memory chunks.This chunking focuses attention on completed subproblem results rather than detailed execution histories.Source: §1 · p. 2
  • HiAgent prompts subgoals, retains detailed action-observation pairs for the current subgoal, and replaces completed subgoals with summarized observations.A trajectory retrieval module can recover detailed information from specific past subgoals when necessary.Source: §1 · p. 2
  • HiAgent reduces the average number of steps by 3.8 while also reducing context length and runtime relative to STANDARD.The reported reductions are 35.02% in context length and 19.42% in runtime.Source: §1 · p. 2

2 Preliminary

LLM-based agents interact with environments by mapping instructions and historical action-observation pairs to executable actions. Their working memory contains information needed at the current moment, but storing the full history introduces redundancy.

2.1 Large Language Model based Agent

  • An LLM-based agent operates as a policy that uses instructions and historical action-observation pairs to generate executable actions in an iterative environment interaction loop.Each action changes the state and produces a new observation until completion or a maximum step limit.Source: §2.1 · p. 2

2.2 Working Memory

  • Working memory is the historical information essential for the LLM’s current decision, supporting integration of past experience with present stimuli.Source: §2.2 · p. 2
  • STANDARD stores all historical action-observation pairs in working memory, providing comprehensive information but introducing redundancy that complicates processing.Source: §2.2 · p. 2

3 Methodology

HiAgent hierarchically manages working memory by using subgoals as chunks, summarizing completed subgoals, and retaining detailed history only where needed. It also supports on-demand retrieval of past trajectories.

3.1 Overview

  • HiAgent first formulates a subgoal, generates actions to achieve it, and summarizes the corresponding action-observation pairs when the subgoal is completed.The summary replaces the completed trajectory in working memory while the current subgoal retains detailed context.Source: §3.1 · p. 2, §3.2 · p. 3
  • Figure 2 summarizes the overall HiAgent process of alternating between generating actions for the current subgoal and generating a new subgoal after completion.Source: §3.1 · p. 2, Figure 2 · p. 3, §3.2 · p. 3

3.2 Subgoal-based Hierarchical Working Memory

  • For past subgoals, HiAgent keeps summarized observations, while the current subgoal retains all action-observation pairs for immediate decision-making.This chunking-based organization is motivated by cognitive science accounts of decomposing complex tasks and reducing working-memory load.Source: §3.2 · p. 3

3.3 Observation Summarization

  • Observation summarization synthesizes historical actions and observations in the context of a subgoal and assesses whether that subgoal has been achieved.The resulting representation is intended to preserve informative context while improving efficiency.Source: §3.3 · p. 3

3.4 Trajectory Retrieval

  • Trajectory retrieval recalls complete action-observation pairs for a past subgoal when detailed history is needed for immediate decisions.This supports diagnosing failed executions and reusing successful experiences without continuously carrying the full context.Source: §3.4 · pp. 3–4

4 Experiments

Experiments evaluate HiAgent against STANDARD on five long-horizon tasks using effectiveness and efficiency metrics. HiAgent improves success and progress while requiring fewer steps, shorter context, and less runtime.

4.1 Experimental Setup

  • The evaluation covers Blocksworld, Gripper, Tyreworld, Barman, and Jericho, five tasks that typically require more than 20 steps.Source: §4.1 · p. 4
  • Progress rate, success rate, average steps, context efficiency, and runtime measure task advancement, completion, action count, context usage, and execution time.Source: §4.1 · p. 4
  • STANDARD serves as the baseline by generating one action followed by one observation before the next action.The implementation uses GPT-4 with a maximum of 30 steps and one in-context example per task.Source: §4.1 · p. 4

4.2 Main Results

  • HiAgent increased success rate by 21% and progress rate by 23.94% over STANDARD, while reducing average completion steps by 3.8.It also reduced context tokens by 35% and runtime by 19.42%.Source: §4.2 · p. 4
  • HiAgent achieved more than double the progress-rate improvement on Blocksworld, Barman, and Jericho, while Gripper traded a 1.5% progress-rate decrease for over 50% lower context-token usage.In Tyreworld, success rate improved by 50% and average steps decreased by 9.4.Source: §4.2 · p. 4

5 Analysis

Across five analyses, HIAGENT’s hierarchical working-memory design improves performance over STANDARD, remains stronger as tasks lengthen, and produces statistically significant effectiveness and efficiency gains.

5.1 Answer 1: All Modules in HIAGENT are Effective for HIAGENT

  • Ablating Observation Summarization reduced success rate by 30% and progress rate by 7.6%, showing that summarizing trajectory information supports agent reasoning.The authors state that summarization aggregates detailed trajectory information.Source: §5.1 · p. 4
  • Removing Trajectory Retrieval reduced success rate by 10% and increased average steps by 1.2, despite retrieval lengthening the LLM’s reasoning steps.The authors attribute its value to flexible retrieval of past trajectories for identifying errors in previous actions.Source: §5.1 · p. 4

5.2 Answer 2: HIAGENT is consistently superior to STANDARD at different steps

  • HIAGENT achieved higher progress rates than STANDARD at every evaluated step count, with continued growth where STANDARD plateaued in blocksworld and barman.STANDARD showed no progress-rate increase from steps 15–25 in those tasks, while HIAGENT continued increasing.Source: §5.2 · p. 5, Figure 3 · p. 6

5.3 Answer 3: The improvement in HIAGENT is not solely attributed to task decomposition

  • Task decomposition alone improved success rate by 30% but remained 20% below HIAGENT while increasing runtime by 5.7% and context length by 12.8%.The comparison indicates that HIAGENT’s improvement is not attributable solely to task decomposition.Source: §5.3 · p. 5

5.4 Answer 4: HIAGENT is effective in generating executable actions even under long steps

  • When steps exceeded 20 in blocksworld, STANDARD’s executability fell below 10%, whereas HIAGENT maintained over 80% executability at longer steps.The authors connect this robustness under longer steps with HIAGENT’s strong long-horizon performance.Source: §5.4 · p. 6, Figure 4 · p. 6

5.5 Answer 5: The observed performance improvements in HIAGENT are statistically significant compared to STANDARD

  • Wilcoxon signed-rank tests found significant differences for Progress Rate (statistic 144.0, p-value 2.38 × 10^-5) and Average Steps (statistic 112.5, p-value 0.0016).The tests compare paired HIAGENT and STANDARD results.Source: §5.5 · p. 6

6 Related Work

HIAGENT addresses working-memory management in LLM-based agents, distinguishing itself from planning-only methods by using subgoals as memory chunks. Related work spans agent applications, planning, and long- versus short-term memory management.

Large Language Model based Agent

  • Existing agent research covers applications including code generation, web browsing, robotics, tool use, planning, research, and multi-agent systems.Source: §6 · pp. 6–7
  • HIAGENT can manage trajectories from frameworks such as ReAct and may alleviate information-management challenges in multi-agent frameworks.Source: §6 · p. 7

Planning

  • HIAGENT uses subgoals as memory chunks to manage working memory hierarchically, unlike planning-only methods that retain full context.This design targets context inefficiency in approaches such as Lumos and XAgent.Source: §6 · p. 7

Memory

  • Most memory research emphasizes long-term memory, whereas HIAGENT investigates more efficient management of short-term or working memory used directly as LLM context.The motivation includes evidence that LLMs can suffer attention loss with lengthy texts.Source: §6 · p. 7

7 Conclusion

The paper proposes HIAGENT, a subgoal-based hierarchical working-memory framework for LLM agents and evaluates it across five long-horizon tasks. HIAGENT outperforms the baseline across tasks, improves efficiency, remains effective as steps increase, and is designed for integration with other agent frameworks.

  • HIAGENT achieves an overall success rate more than double the baseline across five long-horizon agent tasks.Source: §7 · p. 7
  • HIAGENT uses subgoals to manage working memory hierarchically and completes tasks with fewer steps, less runtime, and shorter context.Source: §7 · p. 7
  • As the number of steps increases, HIAGENT more effectively generates executable actions and consistently outperforms STANDARD in progress rate.Source: §7 · p. 7
  • Ablation and statistical analyses are used to evaluate individual modules and validate the significance of HIAGENT’s improvements.Source: §7 · p. 7
  • HIAGENT is presented as an effective, flexible framework that can be integrated into other agent frameworks.Source: §7 · p. 7

Reproducibility Checklist

The reproducibility checklist reports computational experiments, dataset use, implementation documentation, evaluation details, and plans for public code release. It also indicates that no theoretical contribution or novel dataset is introduced.

  • The checklist states that preprocessing, experimental, and analysis code is included or will be publicly released, with implementation comments and replication-oriented seed details.Source: Reproducibility Checklist · p. 11
  • The paper reports computational experiments and relies on existing datasets, while introducing no theoretical contribution or novel dataset.Source: Reproducibility Checklist · p. 11
  • The paper specifies computing infrastructure, including hardware, memory, operating system, and relevant software versions.Source: Reproducibility Checklist · p. 11
  • The evaluation reports metric definitions and motivations, algorithm-run counts, distributional information, statistical testing, final hyperparameters, and tuning ranges with selection criteria.Source: Reproducibility Checklist · p. 11

A.1 Blocksworld

The Blocksworld setup defines four robot-arm actions and their state transitions, then provides an example goal, initial observation, and first action. These specifications establish the executable action space and task state representation.

  • The Blocksworld action space includes pickup, putdown, stack, and unstack, each with explicit preconditions and resulting state changes.The transitions specify arm occupancy, block placement, and whether blocks remain clear.Source: §A.1 · p. 12
  • The example goal requires b1 to be on b2 and b2 to be on b3.Source: §A.1 · p. 12
  • The initial observation places all three blocks on the table, leaves the robot arm empty, and marks each block clear.Source: §A.1 · p. 12
  • The example first action is pickup b2.Source: §A.1 · p. 12

A.2 Gripper

The benchmark environments define executable action schemas, state-transition constraints, goals, observations, and example actions across four long-horizon tasks.

A.2 Gripper

  • The Gripper domain specifies move, pick, and drop actions with explicit preconditions and state-changing effects.Its examples describe robot location, object placement, gripper status, and a sample pickup action.Source: §A.2 · p. 12

A.3 Tyreworld

  • Tyreworld models tire replacement through container access, tool handling, wheel and hub manipulation, inflation, and fastening actions.The example goal places w1 in the boot, while observations specify the initial boot, hub, wheel, nut, and tool states.Source: §A.3 · pp. 12–13

A.4 Barman

  • Barman provides hand, glass, dispenser, shaker, ingredient, and cocktail actions for filling, cleaning, mixing, and pouring beverages.Its example goal is for shot1 to contain cocktail1, with observations describing cocktail ingredients and available bar objects.Source: §A.4 · p. 13

A.5 Jericho

  • Jericho defines navigation, inspection, inventory, object manipulation, and valid-action queries for an interactive adventure environment.The example places Link on a path toward a castle with a barrel nearby and demonstrates examining the barrel.Source: §A.5 · p. 13

B Prompt Examples

The prompt examples contrast STANDARD’s flat action-observation history with HIAGENT’s subgoal-based working-memory organization, using Tyreworld trajectories to illustrate both approaches.

B.1 STANDARD

  • The Tyreworld task requires replacing a flat tire with an intact tire, inflating it, tightening the hub nuts, and restoring tools and the closed boot.The goal and initial observation specify the boot, tools, wheel states, hub position, and nut condition.Source: §B.1 · p. 14
  • The prompt examples define Tyreworld’s action semantics through preconditions and effects for opening, fetching, putting away, fastening, jacking, wheel replacement, and inflation.These action definitions are repeated across the STANDARD and HIAGENT environment implementations.Source: §B.1 · p. 14, §B.2 · p. 15
  • The STANDARD example exposes a sequential action-observation trajectory in which the agent opens the boot, retrieves tools, loosens the nut, elevates the hub, and removes w1.A second continuation shows invalid actions after context changes, followed by fetching the wrench.Source: §B.1 · pp. 14–15

B.2 HIAGENT

  • HIAGENT prompts the model to create subgoals, execute actions for unfinished subgoals, summarize completed-subgoal history, and retrieve details when needed.The prompt forbids consecutive subgoals and supports targeted retrieval through retrieve(subgoal_id).Source: §B.2 · p. 16
  • HIAGENT’s worked trajectory demonstrates recovery from an invalid action by retrieving the failed subgoal’s trajectory before checking valid actions and retrying.The corrected retry uses the valid nut identifier and completes the loosening subgoal.Source: §B.2 · p. 16
  • HIAGENT’s example completes the Tyreworld task through seven explicit subgoals, ending when wheel w1 is returned to the boot.The trajectory covers opening the boot, fetching tools, correcting an erroneous nut reference, elevating the hub, removing the wheel, and storing it.Source: §B.2 · p. 16
  • The HIAGENT example also records compressed subgoal outcomes, including tool access, nut loosening, hub elevation, and wheel removal.These summaries retain milestone states rather than every preceding action-observation pair.Source: §B.2 · pp. 16–17

Related papers

Loading 2408.09559v1…