PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning How to Remember: A Meta-Cognitive Management Method for Structured and Transferable Agent Memory

Sirui Liang, Pengfei Cao, Jian Zhao, Wenhao Teng, Xiangwen Liao, Jun Zhao, Kang Liu

arXiv:2601.07470v1cs.AI

TL;DR

Existing LLM-agent memory methods often use fixed representations or abstraction levels, limiting how memory can be adapted to new tasks. MCMA learns memory abstraction and reuse through a memory copilot separate from task execution. Experiments on ALFWorld, ScienceWorld, and BabyAI report improvements in task performance, generalization, and cross-task transfer.

  • Problem

    Existing methods use predefined memory representations or fixed abstraction levels, leaving agents without a learned way to abstract experience or select an appropriate level for unseen tasks.

    Source: §1 · pp. 1–2

  • Method

    MCMA separates task execution from memory management and trains a memory copilot to structure, abstract, and reuse memories, organizing them hierarchically and transferring the copilot when no memory is reusable.

    Source: §1 · p. 2

  • Results

    Experiments on ALFWorld, ScienceWorld, and BabyAI show that MCMA improves task performance, generalization, and cross-task transfer.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The results highlight the importance of learning how to remember in LLM-based agents.

    Source: §5 · p. 8

    MCMA's current memory-reuse process relies on a manually designed strategy to choose the abstraction level for a new task, limiting end-to-end adaptivity.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language model (LLM) agents increasingly rely on accumulated memory to solve long-horizon decision-making tasks. However, most existing approaches store memory in fixed representations and reuse it at a single or implicit level of abstraction, which limits generalization and often leads to negative transfer when distribution shift. This paper proposes the Meta-Cognitive Memory Abstraction method (MCMA), which treats memory abstraction as a learnable cognitive skill rather than a fixed design choice. MCMA decouples task execution from memory management by combining a frozen task model with a learned memory copilot. The memory copilot is trained using direct preference optimization, it determines how memories should be structured, abstracted, and reused. Memories are further organized into a hierarchy of abstraction levels, enabling selective reuse based on task similarity. When no memory is transferable, MCMA transfers the ability to abstract and manage memory by transferring the memory copilot. Experiments on ALFWorld, ScienceWorld, and BabyAI demonstrate substantial improvements in performance, out-of-distribution generalization, and cross-task transfer over several baselines.

1 Introduction

LLM agents need to reuse experience in long-horizon tasks, but existing memory methods rely on fixed representations or abstraction levels. MCMA makes memory abstraction learnable through a memory copilot, and experiments report gains in performance, generalization, and cross-task transfer.

1 Introduction

  • Effective memory reuse matters because it can help agents solve new tasks efficiently and operate robustly in complex, long-horizon environments.The introduction situates this need in agents' accumulation and reuse of procedural memory.Source: §1 · p. 1
  • Fixed or implicit abstraction levels leave agents unable to learn reusable abstractions or adapt the level of abstraction for unseen tasks.Fine-grained representations may overfit to seen environments, while overly abstract ones can lack actionable guidance.Source: §1 · pp. 1–2
  • MCMA treats memory abstraction as a learnable skill, using a memory copilot to structure, abstract, and manage memories while a frozen task model selects actions.This decouples memory management from task execution.Source: §1 · p. 2
  • The copilot learns memory structures and abstraction strategies through DPO, while memories are organized hierarchically from fine-grained details to abstract knowledge.When no prior memory is reusable, MCMA transfers the copilot's learned abstraction capability.Source: §1 · p. 2
  • Experiments on ALFWorld, ScienceWorld, and BabyAI report substantial gains in performance, out-of-distribution generalization, and cross-task transfer.The paper presents memory abstraction as a transferable resource rather than fixed storage.Source: §1 · p. 2

2 Related Work

Prior memory-augmented agents retrieve experience or organize and learn from it, but memory representations and abstraction levels remain largely fixed. These approaches leave open how agents should abstract experience and choose an abstraction level for a new task.

2 Related Work

  • Retrieval-based methods reuse trajectories, subtasks, cases, or execution steps, but surface-similarity matching can mislead agents when tasks or environments change.Related approaches also extract procedural patterns or heuristics from execution histories.Source: §2 · p. 2
  • Hierarchical and control-based methods use multiple memory levels, while reflection-based approaches refine future behavior through error correction.Large-scale training methods instead internalize experience by adjusting model parameters to improve generalization.Source: §2 · pp. 2–3
  • Across these approaches, representations and abstraction levels remain predefined or implicit, leaving agents without a learned way to abstract experience or select a level for a new task.MCMA's figure describes a task model and memory copilot, with memories and the copilot organized hierarchically for adaptive reuse.Source: §2 · p. 3, Figure 2 · p. 3

3 Methodology

MCMA separates task execution from memory management: a task model generates and evaluates trajectories while a learned copilot creates reusable abstractions. The method combines structured trajectory processing, preference-based copilot training, and similarity-based hierarchical reuse, with copilot transfer for novel tasks lacking reusable memories.

3 Methodology

  • The task model gathers trajectories, the copilot turns them into structured memories, and downstream performance supplies preference pairs for copilot training and memory reuse.The system can reuse stored knowledge or the copilot itself.Source: §3 · p. 3

3.1 Trajectory Collection and Preprocessing

  • Trajectory preprocessing prunes noise, segments subtasks, checks their order, and organizes them into goal trees while retaining high-level intent and executable detail.The task model collects both successful and failed trajectories.Source: §3.1 · pp. 3–4

3.2 Multi-Structure Memory Generation

  • The copilot composes multiple structures and tunes abstraction granularity, using downstream utility to favor candidate memories suited to reuse.Structures include trees, chains, key–value pairs, and natural language; smaller α preserves execution detail, while larger α promotes higher-level procedures.Source: §3.2 · p. 4

3.3 Memory Copilot Evolution

  • Downstream scores produce Top-K preference pairs for supervised fine-tuning followed by DPO, with separate copilots for successful-memory summarization and failure reflection.At inference, the system retrieves relevant trajectories by character-level task-description matching and abstracts them before passing them to the task model.Source: §3.3 · p. 4

3.4 Hierarchical Abstraction and Cross-Task Reusing

  • Similarity-based hierarchy selects detailed episodic memories for similar tasks and higher-level abstractions for less similar tasks; under extreme shifts, the copilot’s abstraction strategy can transfer without reusable trajectories.The copilot can generate abstract memories for novel tasks, and higher levels suppress task-specific details to retain shared goals.Source: §3.4 · pp. 4–5

4 Experiment

Across the evaluated benchmarks, MCMA improves task performance and efficiency, while its ablations and transfer analyses show that useful memory depends on learned abstraction, structure, and reuse. The experiments cover long-horizon benchmarks, distribution shifts, and transfer between domains.

4.1 Experimental Setup

  • Experiments use ALFWorld and ScienceWorld with Qwen3-8B and Qwen3-32B task models; the task model is frozen while memory learning stays in the copilot.ALFWorld includes seen and unseen splits; ScienceWorld uses Dev and Test splits, with task success and execution steps reported for ALFWorld and reward for ScienceWorld.Source: §4.1 · p. 5

4.2 Main Results

  • MCMA achieves the strongest performance across environments and model scales, improving task success rates while reducing execution steps.The authors characterize this as more efficient and reliable long-horizon planning.Source: §4.2 · p. 5, Table 1 · p. 6

4.3 Ablation Study

  • On ALFWorld, successful-memory summarization and failure reflection each improve performance, while combining them gives the best results across settings and model scales.The authors describe the two components as complementary: learning what to do and what to avoid.Source: §4.3 · p. 6, Table 3 · p. 7
  • Natural-language and chain-structured memories outperform no memory but remain behind full MCMA, and untrained copilots show lower unseen-setting performance.The reported comparisons support the importance of structured organization and DPO training for transferable abstraction.Source: §4.3 · pp. 6–7, Table 4 · p. 7

4.4 Structured Memory Analyze

  • Providing 4–5 structural knowledge items gives MCMA its best ALFWorld results; too few limit foresight, while too many add contextual overhead.Structured knowledge is more concise than original traces, allowing a higher density of knowledge items.Source: §4.4 · p. 7, Figure 3 · p. 7
  • Tree structures make up approximately 70% of top-level knowledge arrangements and chains about 20%; replacing these structures with alternatives reduces performance.Nested key-value pairs and natural-language descriptions represent finer-grained details within the hierarchy.Source: §4.4 · p. 7, Figure 4 · p. 8

4.5 Cross-Task Knowledge Transfer

  • In transfer from ALFWorld to BabyAI, Level 1 knowledge lowers performance by 3.13%, whereas Level 2 abstractions improve the baseline by 1.04%.The authors attribute the contrast to low-level interaction mismatch and the preservation of reusable task semantics by higher-level abstractions.Source: §4.5 · p. 8, Table 5 · p. 8

4.6 Cross-Domain Copilot Transfer

  • Transferring memory copilots between ALFWorld and ScienceWorld outperforms the baseline, and a mixed-data copilot is effective in both domains.The mixed copilot particularly benefits ScienceWorld, which has more limited training resources.Source: Table 6 · p. 8, §4.6 · p. 8

5 Conclusion

MCMA treats experience reuse as learnable, decouples task execution from memory management, and uses a transferable memory copilot to abstract and organize interaction trajectories. Experiments on ALFWorld, ScienceWorld, and BabyAI show improvements in task performance, generalization, and cross-task transfer.

5 Conclusion

  • MCMA decouples task execution from memory management and uses a transferable memory copilot trained via preference optimization to abstract and organize interaction trajectories.It treats experience reuse as a learnable problem.Source: §5 · p. 8
  • Experiments on ALFWorld, ScienceWorld, and BabyAI show that MCMA improves task performance, generalization, and cross-task transfer.Source: §5 · p. 8

Limitations

MCMA has training and adaptivity limitations: preference supervision adds offline computational cost, and abstraction-level selection is manually designed.

Limitations

  • Preference supervision increases offline training cost because it requires generating and evaluating multiple candidate abstractions for each trajectory.The pipeline remains more expensive than retrieval-based or single-representation memory methods, although the learned copilot can be reused across tasks.Source: Limitations · p. 9
  • The current reuse process manually selects the abstraction level for a new task, limiting full end-to-end adaptivity.The paper identifies jointly learning level selection with memory structuring and reuse as future work.Source: Limitations · p. 9

Ethical consideration

The paper notes that fine-tuning does not remove ethical and social risks inherited from LLM pretraining data, including concerns about private or contentious data in open-source models.

Ethical consideration

A.1 Dataset Details

The appendix describes three interactive benchmarks and, for BabyAI, the observation-to-language and prompting pipeline used to present grid states to the LLM.

A.1 Dataset Details

  • ALFWorld covers six compositional household task types across 120 scenes and evaluates generalization with 140 seen-room tasks and 134 tasks in novel rooms.Source: §A.1 · p. 11
  • ScienceWorld evaluates procedural reasoning in an interactive science environment with 30 tasks across 10 topics and 1,819 test and 1,796 dev tasks.It uses parametric variations to test generalization to novel objects and scenarios.Source: §A.1 · p. 11
  • BabyAI is a partially observable 2D gridworld with 19 curriculum levels, from single-instruction tasks to composite missions.The agent receives a 7 × 7 symbolic local view.Source: §A.1 · p. 12
  • A deterministic translator converts BabyAI’s 7 × 7 symbolic observation into natural-language descriptions, which are combined with the mission goal, action history, and system instructions in the prompt.The system instruction specifies admissible actions and requires structured JSON output.Source: §A.1 · p. 12

A.2 Baseline Details

The section describes ExpeL’s experience-based knowledge extraction and retrieval and TRAD’s step-wise thought retrieval with contextual alignment. It also illustrates BabyAI’s sparse-reward trajectory pattern.

A.2 Baseline Details

  • ExpeL gathers successful and failed trajectories, abstracts cross-task knowledge into natural-language insights, and retrieves similar successful trajectories as few-shot examples at inference.It stores trajectories in a vector database and operates without gradient updates.Source: §A.2 · p. 12
  • BabyAI’s score remains 0.00 until the specific goal condition is met at the final step, illustrating its typically sparse rewards.The figure traces the “Go to the green key” task.Source: Figure 9 · p. 13
  • TRAD retrieves expert demonstration steps using a state-abstraction thought, then restores context with temporal expansion, relative-order marks, and history alignment.Evaluations on ALFWorld and Mind2Web are reported to outperform state-of-the-art baselines.Source: §A.2 · pp. 12–13

A.3 Task Preprocess Details

The appendix describes how trajectories are decomposed into hierarchical trees, gives examples of structured knowledge and task prompts, and outlines copilot transfer and hierarchical abstraction processing.

A.3 Task Preprocess Details

  • Trajectory preprocessing prunes redundant noise, decomposes sequences into sub-goals, verifies segment consistency, and recursively splits segments with three or more actions into a hierarchical tree.Shorter segments are preserved as atomic leaf nodes.Source: §A.3 · p. 13

A.4 Examples of Structured Knowledge

  • Structured knowledge examples span tree, chain, key-value, natural-language, and nested forms, with a prompt specifying the copilot’s role, JSON structures, and a few-shot example.The appendix also shows a raw and processed ALFWorld trajectory.Source: §A.4 · pp. 13, 15, Figure 10 · p. 14, Figure 11 · p. 14

A.5 Task Prompt

A.6 Reuse Memory Copilot on Same Series LLMs

  • The Qwen3-32B-trained memory copilot demonstrates robust performance on other LLMs in the same series, as reported in Tables 7 and 8.The tables concern reuse on Qwen3-4B and Qwen2.5-72B; no numerical results are given in these passages.Source: §A.6 · p. 15, Table 7 · p. 15, Table 8 · p. 15

A.7 Hierarchical Abstraction Processing

  • Hierarchical knowledge processing embeds each entry’s name and description, builds a k-NN graph from the top 10 cosine-similar connections, then clusters and fuses knowledge into higher-level abstractions.Examples contrast lower-level complete task details with higher-level common problems.Source: §A.7 · p. 15, Figure 13 · p. 17, Figure 14 · p. 18

Related papers

Loading 2601.07470v1…