Source-linked AI summary
Agentic Memory: Learning Unified Long-Term and Short-Term Memory Management for Large Language Model Agents
Yi Yu, Liuyi Yao, Yuexiang Xie, Qingquan Tan, Jiaqi Feng, Yaliang Li, Libing Wu
TL;DR
Long-horizon LLM agents need better memory management because finite context windows constrain reasoning, while existing LTM and STM systems are often separate and heuristic. AgeMem integrates both into the agent policy as tool-based actions and trains unified behavior with progressive reinforcement learning and step-wise GRPO. Across five long-horizon benchmarks, it improves task performance and memory quality while using context efficiently, though evaluation remains controlled relative to open-ended real-world deployment.
Problem
Finite context windows constrain long-horizon reasoning, while existing methods commonly manage LTM and STM separately using heuristics or auxiliary controllers.
Method
AgeMem integrates LTM and STM management into the agent policy through tool-based actions, trained with three-stage progressive reinforcement learning and step-wise GRPO.
Results
AgeMem consistently outperforms strong baselines across five long-horizon benchmarks and improves task performance, long-term memory quality, and context efficiency.
Takeaways & Limitations
Unified, agent-centric memory policies are a promising direction for scalable and adaptive LLM agents capable of long-term reasoning.
Takeaways & Limitations
Evaluation uses five representative but relatively controlled benchmarks, leaving persistent dialogue and real-user interaction scenarios for future work.
Abstract
from arXiv · showhide
Large language model (LLM) agents face fundamental limitations in long-horizon reasoning due to finite context windows, making effective memory management critical. Existing methods typically handle long-term memory (LTM) and short-term memory (STM) as separate components, relying on heuristics or auxiliary controllers, which limits adaptability and end-to-end optimization. In this paper, we propose Agentic Memory (AgeMem), a unified framework that integrates LTM and STM management directly into the agent's policy. AgeMem exposes memory operations as tool-based actions, enabling the LLM agent to autonomously decide what and when to store, retrieve, update, summarize, or discard information. To train such unified behaviors, we propose a three-stage progressive reinforcement learning strategy and design a step-wise GRPO to address sparse and discontinuous rewards induced by memory operations. Experiments on five long-horizon benchmarks demonstrate that AgeMem consistently outperforms strong memory-augmented baselines across multiple LLM backbones, achieving improved task performance, higher-quality long-term memory, and more efficient context usage.
1 Introduction
LLM agents’ long-horizon reasoning is constrained by finite context windows, while existing memory systems manage LTM and STM separately through static, trigger-based, or heuristic mechanisms. AgeMem unifies both memory types as tool-based actions within the agent policy and trains them progressively with reinforcement learning.
- Finite context windows constrain the information LLM agents can attend to during long-horizon, multi-step tasks.
- Existing architectures separately optimize LTM and STM, producing fragmented memory construction and suboptimal long-horizon performance.They commonly use static STM with trigger-based or agent-based LTM.
- Unified memory management must coordinate LTM storage, updating, and discarding with STM retrieval, summarization, and removal.The challenge reflects the distinct but complementary functions of the two memory types.
- AgeMem lets the LLM autonomously manage LTM and STM through explicit tool-based operations integrated into the agent’s decision-making policy.The framework replaces external memory components with jointly managed memory actions.
- A three-stage progressive RL strategy teaches LTM storage, STM context management, and joint coordination, with step-wise GRPO supporting end-to-end learning.The approach addresses fragmented experience and sparse, discontinuous rewards across memory decisions.
2 Background and Related Work
Prior LTM research uses modular, structured, graph-based, or extract-update designs, but these approaches largely depend on predefined structures or heuristic update rules. AgeMem instead targets an adaptive policy that learns memory prioritization, updating, forgetting, and context filtering according to task demands.
- Existing LTM systems include modular memory types, linked knowledge units, extract-update pipelines, and temporal knowledge graphs.Examples include LangMem, A-Mem, Mem0, and Zep.
- These approaches largely rely on predefined memory structures or heuristic update rules, limiting adaptive prioritization and forgetting as memory grows.
- AgeMem aims to learn an adaptive memory policy that dynamically decides what to store, update, or forget based on task demands and long-term context.
- The approach also learns when and how to retrieve, summarize, or filter context to balance efficiency with information preservation.
3 Method
AgeMem makes memory management part of the agent’s policy, jointly coordinating persistent long-term memory and active short-term context through tools and staged reinforcement learning. Its trajectories preserve long-term memory across stages while resetting short-term context, and step-wise GRPO assigns delayed outcomes to intermediate memory decisions.
- Unified memory policy: AgeMem integrates LTM and STM management directly into the agent through specialized tools rather than an external memory pipeline.The action space includes language generation and memory operations, making memory control an intrinsic policy component.
- Reward and optimization: AgeMem optimizes a composite trajectory reward covering task performance and memory-management quality, with a KL penalty balancing policy updates.The reward formulation includes task, context, and memory components, while the KL coefficient controls exploration and training stability.
- Three-stage training: Each trajectory has three consecutive stages: LTM construction, STM control under distractors, and integrated reasoning with memory coordination.The stages progressively expose information, require distractor filtering or compression, and evaluate retrieval, context management, and final task solving together.
- Three-stage training: Long-term memory persists across stages, whereas short-term context is reset before Stage 2 to prevent residual-context solutions.The reset forces retrieval from LTM and supports training of memory operations.
- Step-wise GRPO: Step-wise GRPO broadcasts each rollout’s terminal, group-normalized advantage to preceding actions, connecting delayed task outcomes with memory decisions.This supplies a consistent learning signal to memory and reasoning actions across all three stages while retaining the KL-divergence stability term.
- Memory tools: LTM tools ADD, UPDATE, and DELETE modify persistent memory, while STM tools RETRIEVE, SUMMARY, and FILTER control active context.RETRIEVE imports relevant memories, SUMMARY compresses interaction history, and FILTER suppresses irrelevant context.
4 Experiments
Across five benchmarks, AgeMem achieves the strongest average task performance while improving stored-memory quality and controlling prompt-token usage. Ablations and tool-use analyses attribute these gains to unified LTM/STM management and reinforcement-learning optimization.
- Main Results: 41.96% and 54.31% average performance on Qwen2.5-7B-Instruct and Qwen3-4B-Instruct, respectively, with gains over no-memory baselines of 49.59% and 23.52%.AgeMem also improves over the best baselines by 4.82 and 8.57 percentage points on average.
- Main Results: 0.533 and 0.605 Memory Quality scores make AgeMem the top method on both model backbones.The scores measure relevance between stored memories and HotpotQA ground-truth facts.
- Main Results: 3.1% and 5.1% fewer average prompt tokens than AgeMem-RAG demonstrate more efficient STM management on Qwen2.5 and Qwen3.AgeMem uses 2,117 versus 2,186 tokens on Qwen2.5, and 2,191 versus 2,310 on Qwen3.
- Tool Usage Analysis: RL increases LTM ADD and UPDATE usage and raises STM FILTER frequency, indicating more active and balanced memory control.On Qwen2.5, ADD rises from 0.92 to 1.64, UPDATE reaches 0.13, and FILTER rises from 0.02 to 0.31.
- Tool Usage Analysis: 2.31→1.95 and 4.62→4.35 RETRIEVE calls after RL reflect more selective retrieval on Qwen2.5 and Qwen3 rather than undertraining.The shift coincides with improved task performance and Memory Quality.
- Ablation Studies: +13.9%, +21.7%, and +16.1% overall gains make full AgeMem outperform the ablated systems across the three representative benchmarks.Adding STM tools gives the largest component boost on SciWorld (+3.1%) and HotpotQA (+2.4%).
5 Conclusion
AgeMem unifies long-term and short-term memory control through learnable tool-based actions embedded in the agent policy. Across diverse long-horizon benchmarks, it improves task performance and memory quality while using context efficiently.
- AgeMem jointly controls long-term and short-term memory through learnable, tool-based actions integrated into the agent’s policy.
- Progressive reinforcement learning trains the unified memory-management behavior end to end, replacing heuristic memory pipelines.
- Experiments across diverse long-horizon benchmarks show improved task performance, higher memory quality, and efficient context usage.
- The results support unified, agent-centric memory policies as a direction for scalable and adaptive LLM agents capable of long-term reasoning.
Limitations
The study’s evaluation and implementation leave clear scope boundaries. Its tools are fixed, its benchmarks are relatively controlled, and its training trajectories currently come from HotpotQA.
- AgeMem currently uses a fixed set of memory-management tools, limiting the abstraction’s fine-grained control.The authors identify extending the tool set as future work.
- The five evaluated benchmarks remain relatively controlled compared with open-ended real-world deployments.Persistent long-term dialogue and real-user interaction are identified as important next evaluation settings.
- The training curriculum currently relies on HotpotQA trajectories, leaving applicability to richer interaction structures for future study.
A Detailed Design and Implementation of AgeMem
AgeMem implements memory management as structured actions within the agent’s policy, covering both persistent long-term memory and active short-term context. Its tools support retrieval, addition, maintenance, summarization, and filtering, while training examples illustrate learned strategic behavior.
- Unified tool interface: AgeMem exposes memory tools directly in the agent’s action space, allowing reinforcement learning to determine when and how operations are invoked.The policy can output language tokens or structured tool calls for memory operations.
- Long-term memory tools: RETRIEVE accesses semantically relevant long-term memories and inserts them into the active short-term context for reasoning.The number of retrieved memories is controlled by k, typically set to 3–5 in experiments.
- Long-term memory tools: ADD stores new long-term knowledge with content, embeddings, and metadata, while UPDATE and DELETE maintain relevance as information changes.The reward function encourages reusable entries and meaningful maintenance while penalizing redundancy or irrelevance.
- Short-term memory tools: SUMMARY compresses selected short-term context into a replacement summary to prevent overflow while preserving essential information.The agent can summarize all non-system messages or the last N messages, and it can learn to act proactively before overflow.
- Short-term memory tools: FILTER removes context messages whose semantic similarity to a specified criterion exceeds threshold θf, reducing irrelevant or redundant content.The default threshold is θf = 0.6, and criteria may be agent-specified or derived from the task context.
- Learned behavior: After reinforcement learning, the agent selectively stores preferences, recognizes needed updates, and maintains cleaner, more consistent long-term memory than before training.Before training, the baseline either stores information indiscriminately or fails to identify obsolete knowledge.
Analysis: The baseline retains all information
The baseline retains distractors in short-term context, causing irrelevant material to accumulate alongside task-relevant information. After Stage 2 training, AgeMem filters distractors and summarizes large contexts to preserve focus and essential content.
- Baseline behavior: The baseline treats distractors like task-relevant content, bloating context with irrelevant topics and consuming token budget without helping the planning task.Examples include quantum computing and bread-making details retained during machine-learning planning.
- After RL training: AgeMem learns to filter distractors during Stage 2 rollouts while preserving focus on the active task.The demonstrated filter call removes unrelated topics before continuing with course planning.
- After RL training: When context grows large, AgeMem proactively summarizes accumulated details before generating the final answer.The example preserves the three-day structure, session durations, topics, and tools after summarization.
- Comparison: The baseline retains content verbatim, whereas AgeMem combines early filtering with later summarization to prevent context dilution or overflow while preserving essential information.The two operations address different points in the context-management workflow.
B.3 Case 3: Integrated Task Execution with Memory Coordination
The integrated case shows AgeMem coordinating long-term retrieval with short-term context management during personalized task execution. Retrieved preferences are combined with the current request to produce a schedule aligned with the user’s constraints and learning style.
- Baseline comparison: The baseline provides a generic schedule that ignores the user’s 120-minute deep-focus and visual-learning preferences.Its schedule uses shorter, generic time blocks and does not incorporate the stored preferences.
- Integrated coordination: AgeMem’s integrated workflow retrieves relevant user preferences from long-term memory, manages active context, and generates a personalized response.The case is presented as coordination across all three memory-management capabilities.
- AgeMem response: The trained agent retrieves preferences for visual learning, 120-minute focus blocks, Python basics, and interest in computer vision before planning.The retrieved information is explicitly used to tailor the schedule.
- AgeMem response: The resulting schedule uses 120-minute sessions and visual materials such as diagrams, plots, demonstrations, and embedded notebook exercises.The plan includes ML foundations, Python for ML, and optional practice across specified time blocks.
C.2 LLM-based Evaluation Details
The evaluation uses LLM-based judges to score memory quality and answer correctness against HotpotQA ground truth. Baselines are implemented from official codebases, and additional variants isolate reinforcement learning and learned short-term memory management.
- Memory Quality: Memory Quality measures whether predicted supporting facts cover expected facts, remain relevant, and avoid irrelevant content.Qwen-Max assigns a score from 0.0 to 1.0 by comparing stored memories with HotpotQA ground-truth facts.
- Scoring rubric: The MQ rubric ranges from perfect matching at 1.0 to completely incorrect or irrelevant predictions at 0.0–0.1.Intermediate bands represent omissions, irrelevant facts, partial correctness, or significant errors.
- Answer evaluation: The HotpotQA LLM-as-a-Judge metric evaluates answer correctness by comparing the agent’s answer with the ground-truth answer on a 0.0–1.0 scale.Qwen-Max performs the evaluation using a structured question, ground truth, and agent-answer prompt.
- Baselines: Baseline implementations follow official open-source codebases, while AgeMem-noRL isolates the contribution of reinforcement learning.RAG variants replace short-term-memory tools with static retrieval pipelines to compare learned and retrieval-based context management.
- Ablation study: A Qwen3-4B-Instruct ablation shows consistent progressive contributions from LTM, STM, and RL across three representative datasets, supporting generalization across model sizes.The appendix compares these trends with Qwen2.5-7B-Instruct.
D.2 Reward Function Ablation on Qwen3-4B
The Qwen3-4B-Instruct ablation evaluates reward strategies and compares progressively augmented memory configurations against a no-memory baseline.
- The study validates the multi-component reward design on Qwen3-4B-Instruct.The analysis parallels the corresponding Qwen2.5-7B-Instruct results.
- Figure 9 compares No-Memory, LTM-only, LTM-with-RL, and full AgeMem configurations.Green arrows indicate performance gains over the baseline.
- Figure 10 compares All-Returns and Answer-Only reward strategies using solid and dashed convergence curves.
D.2.1 Convergence Analysis
On Qwen3-4B-Instruct, All-Returns remains superior to Answer-Only throughout training, with smoother convergence particularly in later stages.
- All-Returns consistently outperforms Answer-Only throughout the Qwen3-4B-Instruct training process.
- The convergence curve shows smoother progression with lower variance, especially during steps 70-100.
- All-Returns maintains its advantage despite a smaller absolute improvement than on Qwen2.5-7B-Instruct.The authors describe this as validating the robustness of the reward design.
D.2.2 Quantitative Results
Quantitative ablations show that memory-aware rewards improve task performance and memory quality with only marginally higher token use, while integrated memory actions remain advantageous over augmented baselines.
- 0.605 v.s. 0.415 MQ scores show substantially better stored-memory quality with All-Returns.
- 8.67 v.s. 7.21 tool calls indicate more active memory-operation use under All-Returns.
- 2191 v.s. 2164 token counts show that All-Returns gains come with only a marginal increase in token consumption.
- Adding the same ST/RL extensions improves prior LTM-only methods but still does not match full AgeMem.
- The reported advantage stems from an integrated policy over heterogeneous memory actions rather than any single added module.
- Performance remains stable for N = 3 to N = 5 and declines mildly at N = 7 under fixed settings.This suggests Stage 2 learning is robust to distractor intensity.