Source-linked AI summary
Lightweight LLM Agent Memory with Small Language Models
Jiaquan Zhang, Chaoning Zhang, Shuxu Chen, Zhenzhen Huang, Pengcheng Zheng, Zhicheng Wang, Ping Guo, Fan Mo, Sung-Ho Bae, Jie Zou, Jiwei Wei, Yang Yang
TL;DR
Retrieval-based agent memory can be efficient but noisy, while repeated LLM-driven online operations incur runtime overhead. LightMem separates lightweight SLM-driven online memory work from offline consolidation; it reports gains across model scales, including about +2.5 average F1 over A-MEM on LoCoMo.
Problem
Retrieval-based memory can have unstable accuracy because of limited query construction and candidate filtering, while repeated LLM-driven online memory operations add runtime overhead over long interactions.
Source: §1 · p. 1Method
LightMem assigns online query planning, retrieval, and writing to specialized SLMs, organizes memory into STM, MTM, and LTM, and defers long-term consolidation to offline processing.
Source: §1 · p. 2Results
Takeaways & Limitations
The online–offline separation supports efficient memory use under tight compute budgets while allowing long-term memory to evolve.
Source: §1 · p. 2The impact of alternative consolidation strategies and control policies is not fully explored.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Although LLM agents can leverage tools for complex tasks, they still need memory to maintain cross-turn consistency and accumulate reusable information in long-horizon interactions. However, retrieval-based external memory systems incur low online overhead but suffer from unstable accuracy due to limited query construction and candidate filtering. In contrast, many systems use repeated large-model calls for online memory operations, improving accuracy but accumulating latency over long interactions. We propose LightMem, a lightweight memory system for better agent memory driven by Small Language Models (SLMs). LightMem modularizes memory retrieval, writing, and long-term consolidation, and separates online processing from offline consolidation to enable efficient memory invocation under bounded compute. We organize memory into short-term memory (STM) for immediate conversational context, mid-term memory (MTM) for reusable interaction summaries, and long-term memory (LTM) for consolidated knowledge, and uses user identifiers to support independent retrieval and incremental maintenance in multi-user settings. Online, LightMem operates under a fixed retrieval budget and selects memories via a two-stage procedure: vector-based coarse retrieval followed by semantic consistency re-ranking. Offline, it abstracts reusable interaction evidence and incrementally integrates it into LTM. Experiments show consistent gains across model scales, with an average F1 improvement of about 2.5 over A-MEM on LoCoMo, while achieving higher efficiency and low median latency (83 ms for retrieval and 581 ms end-to-end).
1 Introduction
LightMem addresses the trade-off between noisy but efficient retrieval-based memory and costly LLM-driven online operations. It uses specialized SLMs for online memory work and defers heavier consolidation, with reported gains and low latency.
1 Introduction
- Retrieval-based systems can be efficient but produce unstable accuracy from limited query construction and candidate filtering, while repeated LLM-based online operations add runtime overhead over long interactions.These constraints motivate separating lightweight online decisions from heavier offline processing.Source: §1 · p. 1
- LightMem modularizes query parsing, retrieval, writing, and long-term consolidation, assigning high-frequency online tasks to specialized SLMs and heavier consolidation to an offline path.Its stores are organized as STM, MTM, and LTM, with user-identity metadata for per-user logical isolation.Source: §1 · p. 2
- Its two-stage query design first narrows candidates with fast retrieval, then applies semantic-level verification to select relevant memories.The online path also uses intent-conditioned hypothetical queries and a fixed Top-K budget.Source: §1 · p. 2
- Across model scales, LightMem shows gains, including about +2.5 average F1 over A-MEM on LoCoMo, improved semantic consistency on DialSim, and median latency of 83 ms for retrieval and 581 ms end-to-end.The reported results cover both LoCoMo and DialSim.Source: §1 · p. 2
2 Related Work
Prior memory systems use retrieval over compressed interaction entries or LLM-driven operations, with a recurring efficiency–effectiveness trade-off. LightMem positions lightweight SLM control and filtering alongside improved external memory writing and retrieval under a fixed budget.
2.1 LLM Memory Systems
- Retrieval-based systems store and retrieve compressed memories, while LLM-driven systems use model operations for tasks such as working-memory management or note-taking; examples include MemoryBank (Zhong et al., 2024), MemGPT (Packer et al., 2023), and A-MEM (Xu et al., 2025).The passage also describes ReadAgent (Lee et al., 2024) as retrieving evidence from compressed gists and HiAgent (Hu et al., 2025) as summarizing trajectory subgoals.Source: §2.1 · p. 2
- These lines expose a trade-off between lightweight but noisy retrieval and more effective yet costly online LLM operations; LightMem uses SLM-based online control and filtering with improved memory writing and retrieval under a fixed budget.A-MEM typically does not emphasize strict online/offline decoupling under resource constraints.Source: §2.1 · p. 3
3 Method
LightMem separates query-time memory operations from offline long-term consolidation, using specialized SLMs and distinct STM, MTM, and LTM stores. Its retrieval controller plans searches under a fixed budget, and the system writes interaction memories incrementally before consolidating selected evidence into LTM.
3 Method
- LightMem uses specialized SLMs to modularize memory operations, separating lightweight online querying from offline long-term consolidation.The overview depicts online retrieval over STM/MTM and incremental consolidation of MTM into graph-structured LTM.Source: §3 · p. 3, Figure 2 · p. 4
3.1 Problem Setup and Preliminaries
- At each turn, LightMem retrieves at most K memories from a user-scoped store to support response generation, using the available recent dialogue context.The store enforces user-level logical isolation through user identifiers.Source: §3.1 · p. 3
3.2 Memory Stores
- STM buffers recent in-session interaction, MTM carries personalized episodic memory, and LTM stores de-identified semantic knowledge distilled from high-value MTM episodes.LTM is organized as a lightweight graph-structured knowledge base; STM is neither persisted nor retrieved.Source: §3.2 · p. 3
3.3 Intent Modeling and Retrieval Control
- SLM-1 acts as a retrieval controller: it infers coarse intent, rewrites input into hypothetical queries, and specifies search constraints and a fixed Top-K budget.Its intent attributes guide retrieval planning rather than answer generation.Source: §3.3 · p. 3
3.4 Two-Stage Retrieval
- Stage 1 retrieves a fixed 2K candidate pool through metadata-constrained vector search, and Stage 2 uses semantic consistency and relevance judgments to keep the most relevant half.This two-to-one compression provides fixed candidate size and suppresses noise beyond vector similarity.Source: §3.4 · pp. 3–4, Figure 2 · p. 4
3.5 Memory Writing and Update
- After each response, SLM-3 extracts reusable user-relevant information into MTM, merging or rewriting repetitive items and using temporal cues and evidence strength for conflicts.When MTM reaches capacity, it evicts stale, low-utility items and further compresses redundant content.Source: §3.5 · p. 4
3.6 Offline Consolidation
- Offline consolidation processes incremental batches of new, reactivated, or low-utility MTM items, abstracts privacy-preserving knowledge candidates, and links them to nearby LTM anchors.Evidence over time supports merge, update, or drop decisions; confidence decay on weakly supported candidates enables natural forgetting.Source: §3.6 · p. 4
4 Experiment
Experiments compare LightMem with established memory baselines across two dialogue datasets, multiple backbones, and lexical and semantic metrics. LightMem performs best or near-best across the reported settings, and component removals degrade performance.
4.1.1 Datasets
- LoCoMo contains long dialogues averaging 9K tokens across five question categories, while DialSim contains 1,300 multi-party sessions derived from TV shows.The experiments use both datasets to evaluate LightMem.Source: §4.1.1 · p. 5
4.1.2 Baselines
- Baselines include LoCoMo, ReadAgent, MemoryBank, MemGPT, and A-MEM, evaluated with GPT-4o, GPT-4o-mini, Qwen2.5, and Llama 3.2 backbones.The authors state that baselines use the same backbone and retrieval budget for fair comparison.Source: §4.1.2 · p. 5
4.1.3 Metric
- LoCoMo evaluation reports F1 and BLEU-1, while DialSim additionally uses ROUGE-L, METEOR, and SBERT similarity.The metrics cover answer correctness, overlap, and semantic alignment.Source: §4.1.3 · p. 5
4.1.4 Implement Details
- GPT-4o-mini is the fixed response generator, while locally deployed quantized SLMs handle online control and offline consolidation uses a large-context LLM.Coarse retrieval selects the top 10 neighbors; MTM capacity is capped at 104 items.Source: §4.1.4 · p. 5
4.2 Performance
- LightMem achieves the best overall LoCoMo performance and improves on A-MEM in most settings, including GPT-4o multi-hop F1 of 34.52 versus 32.86.It remains best or near-best across diverse backbones.Source: Table 2 · p. 6, §4.2 · pp. 6–7
- On DialSim, LightMem achieves the best results across metrics, with SBERT similarity of 23.40 versus 19.51 for A-MEM.The authors note that answers can have diverse surface forms, so semantic similarity complements overlap metrics.Source: §4.2 · pp. 6–7, Table 3 · p. 7
4.3 Ablation Study
- Removing any component degrades performance; on Llama-3.2-1B, F1 falls from 4.12 to 3.75 without MTM, while disabling offline consolidation yields 3.96.Removing graph structure lowers SBERT similarity from 23.40 to 22.82.Source: §4.3 · p. 7
4.4 Scalability and Latency
The update-gap test finds concurrent MTM/LTM retrieval outperforms using either store alone, while memory-growth and error-injection tests examine retrieval stability and pipeline robustness. LightMem also maintains low retrieval and end-to-end latency.
4.4 Scalability and Latency
- 83/167 ms retrieval latency at P50/P95 and 581 ms end-to-end latency at P50 show LightMem's low retrieval overhead and competitive responsiveness.The comparison uses GPT-4o-mini; LightMem's effective context is around 1K tokens, versus close to 16K for LoCoMo and MemGPT.Source: Table 4 · p. 8, §4.4 · p. 8
4.5 Performance Stability under MTM Growth
- As MTM grows along a dialogue trajectory, LightMem maintains more stable performance than pure vector retrieval, whose retrieval noise increases.The comparison reports cumulative F1 at natural memory-growth stages on DialSim with Llama-3.2-1B.Source: Table 5 · p. 8, §4.5 · p. 8
4.6 Error Injection Stress Test
- Errors introduced across pipeline stages consistently degrade performance, with cascading failures causing the largest collapse.Removing SLM-2 semantic reranking produces a larger decline than query noise, while noisy SLM-3 writes can contaminate future retrieval.Source: Table 6 · p. 9, §4.6 · p. 9
4.7 Update Gap Stress Test
- 28.85 multi-hop F1 is achieved by full LightMem, versus clear drops with only LTM or only MTM; MTM noise saturation lowers multi-hop F1 to 23.10.The study attributes the full system's advantage to combining recent unconsolidated MTM facts with older consolidated LTM knowledge.Source: Table 7 · p. 9, §4.7 · p. 9
5 Conclusion
LightMem separates lightweight online memory operations from offline consolidation and organizes memory across STM, MTM, and LTM. Evaluations report gains across model scales, improved semantic consistency on DialSim, and low median latency.
5 Conclusion
- LightMem decouples high-frequency online query control, retrieval, and writing from offline long-term consolidation.Its memory is organized into STM, MTM, and LTM, with user-scoped isolation.Source: §5 · p. 9
- Online retrieval uses HQ-based routing and a fixed budget, with coarse vector search followed by semantic consistency re-ranking to reduce retrieval noise.Source: §5 · p. 9
- +2.5 average F1 on LoCoMo over A-MEM, improved semantic consistency on DialSim, and median latencies of 83 ms retrieval and 581 ms end-to-end are reported.Source: §5 · p. 9
6 Limitations
The paper focuses on one design of online–offline decoupled memory pipelines driven by specialized small language models; alternative consolidation strategies and control policies remain underexplored.
6 Limitations
- The work focuses on a specific design of online–offline decoupled memory pipelines driven by specialized small language models.Source: §6 · p. 9
- The impact of alternative consolidation strategies is not fully explored.Source: §6 · p. 9
- Alternative control policies remain an open direction for future investigation.Source: §6 · p. 9
8 Appendices
The appendix defines LightMem's memory structures and describes online retrieval, response generation, memory writing, and offline consolidation. It states that online operations use SLMs under strict latency and compute constraints, while consolidation runs asynchronously.
8.1.1 Symbols and Data Structures
- STM is the truncated recent-dialogue context; each user's MTM store and the global LTM store are distinguished, with MTM entries as compressed interaction records and LTM nodes as de-identified knowledge units.MTM entries carry embeddings, temporal metadata, and user identifiers; LTM uses a lightweight graph for associative organization and multi-hop access.Source: §8.1.1 · p. 11
8.1.2 Online Retrieval Procedure
- Online retrieval infers intent and generates HQs and metadata constraints, retrieves MTM/LTM candidates by vector search, then applies semantic filtering and selects up to the fixed Top-K budget.Source: §8.1.2 · p. 11
8.1.3 Response Generation
- The retrieved set is concatenated with short-term context and passed to the fixed response generator, with the same generation setup used for LightMem and baselines.Source: §8.1.3 · p. 11
8.1.4 Online Memory Writing and MTM Maintenance
- SLM-3 extracts and compresses reusable user-relevant interaction information into MTM, then merges redundant entries, resolves conflicts, and enforces the store's capacity bound.Source: §8.1.4 · p. 12
8.1.5 Offline Consolidation into Long-Term Memory
- Offline consolidation into LTM is summarized in Algorithm 3 and runs asynchronously without blocking online interaction.Source: §8.1.5 · p. 12, §8.1.6 · p. 12
8.1.6 Online-Offline Decoupling Guarantee
- Online retrieval and writing use SLMs under strict latency and compute constraints; retrieval cost is bounded by O(K), and LTM evolution adds no overhead to online inference.Source: §8.1.6 · p. 12
9 Experiment
The section reports statistical comparisons with A-MEM and examines a failure case in which ambiguous intent decomposition can add generic advice to retrieval. This focus dilution can affect lexical metrics while semantic similarity remains stable.
9.1 Statistical Significance Analysis
- Significance is assessed against A-MEM under the same backbone and retrieval budget using paired bootstrap resampling over 1,000 evaluation-instance resamples.The analysis reports mean differences, 95% confidence intervals, and two-sided p-values; Table 8 presents the statistical-significance results.Source: §9.1 · p. 12, Table 8 · p. 12
9.2 Failure Analysis: Ambiguous Intent Decomposition in SLM-1
- Under an underspecified recall query, SLM-1 may route a generic timeline-principles query to LTM alongside queries for the user’s decision in MTM.With a fixed Top-K budget, the LTM query can introduce generic content, although the key MTM decision is typically recalled.Source: §9.2 · p. 12
- Focus dilution mainly affects lexical-overlap metrics such as F1 and BLEU, while semantic similarity remains stable and SLM-2 filtering helps preserve relevant MTM evidence.The resulting response typically remains correct but may be less focused or concise because it includes additional general advice.Source: §9.2 · pp. 12–13
10 Full Prompt Templates and Control Interface Definitions
The appendix specifies the prompts and control interfaces for LightMem’s online SLMs and offline consolidation model. It standardizes each prompt’s role, inputs, tasks, constraints, and output format to support structured control and reproducibility.
10 Full Prompt Templates and Control Interface Definitions
- The appendix provides complete prompts for SLM-1, SLM-2, SLM-3, and the offline LLM, specifying roles, inputs, tasks, constraints, and output formats for reproducibility and auditability.The prompts use a unified schema intended to avoid implicit assumptions and undeclared model capabilities.Source: §10 · p. 13, §10.1 · p. 13
10.1.1 Full Prompt Template (SLM-1)
- SLM-1 converts the current query and truncated dialogue context into explicit retrieval queries, resolving ambiguity and routing personalized needs to MTM and general factual needs to LTM.It does not answer the user or perform retrieval itself.Source: §10.1.1 · p. 13
10.2.1 Full Prompt Template (SLM-2)
- SLM-2 checks a fixed-size candidate set for semantic consistency with the query set and selects relevant memories under a fixed budget, rather than generating or rewriting content.Selection prioritizes semantic alignment and direct support for a query over superficial lexical overlap or loose topical relevance.Source: §10.2 · p. 13, §10.2.1 · pp. 13–14
10.2 SLM-2: Semantic Consistency Filtering and Candidate Compression
- SLM-2 receives query representations and a fixed-size candidate list with metadata, retaining only memories that meaningfully support at least one query.Its role is selection only: it does not retrieve, expand, or modify memory items.Source: §10.2.1 · pp. 13–14
10.3 SLM-3: Interaction Summarization and Online MTM Maintenance
- SLM-3 transforms transient dialogue into compact memory representations for future interactions, not user-facing responses or cross-user abstractions.Its inputs include the current utterance, system response, and relevant short-term dialogue context.Source: §10.3.1 · p. 14
10.3.1 Full Prompt Template (SLM-3)
- SLM-3 distills user-relevant information from the current interaction into concise, reusable semantic entries and incrementally maintains MTM rather than retaining full transcripts.It prioritizes information likely to remain useful and self-contained in future interactions.Source: §10.3.1 · p. 14
10.4 Offline LLM: Long-Term Memory Consolidation
- The offline LLM abstracts selected MTM episodes into de-identified, stable knowledge, then merges, updates, discards, or inserts representations in the LTM graph.Consolidation runs asynchronously offline and must not affect online interaction latency.Source: §10.4 · p. 14
11 LTM Graph Schema and Maintenance Dynamics
This section describes LTM as a lightweight graph of de-identified semantic knowledge and outlines its incremental offline maintenance. It distinguishes LTM’s stable, general knowledge from MTM’s episodic interaction summaries.
11 LTM Graph Schema and Maintenance Dynamics
- The section’s scope includes the graph schema and maintenance dynamics, including update frequency, node growth, and downstream reasoning performance.These topics are identified as the section’s coverage; no specific outcomes are stated in the supplied passages.Source: §11 · pp. 14–15
11.1 Graph Schema: Node and Edge Types
- LTM stores stable facts, domain regularities, and cross-user trends in a de-identified graph, unlike MTM’s episodic interaction summaries.The graph uses standardized node and edge types to support cross-task generalization and multi-hop reasoning.Source: §11.1 · p. 15
11.2 Maintenance Dynamics and Performance Impact
- Offline LLM updates are processed in incremental batches so LTM maintenance does not block real-time user interaction.Table 10 summarizes consolidation metrics and their impact on model performance, but the supplied passage gives no specific values.Source: §11.2 · p. 15