LightMem: Lightweight and Efficient Memory-Augmented Generation

Jizhan Fang; Xinle Deng; Haoming Xu; Ziyan Jiang; Yuqi Tang; Ziwen Xu; Shumin Deng; Yunzhi Yao; Mengru Wang; Shuofei Qiao; Huajun Chen; Ningyu Zhang

Source-linked AI summary

LightMem: Lightweight and Efficient Memory-Augmented Generation

Jizhan Fang, Xinle Deng, Haoming Xu, Ziyan Jiang, Yuqi Tang, Ziwen Xu, Shumin Deng, Yunzhi Yao, Mengru Wang, Shuofei Qiao, Huajun Chen, Ningyu Zhang

arXiv:2510.18866v4Oct 21, 2025cs.CLcs.AIcs.CVcs.LGcs.MA

TL;DR

LLMs struggle to use historical interactions efficiently in dynamic, long-context settings, while existing memory systems impose substantial time and computational overhead. LightMem uses three human-memory-inspired stages to filter, organize, and offline-consolidate information, and it consistently outperforms strong baselines on LongMemEval and LoCoMo with lower costs.

Problem
LLMs have difficulty leveraging historical interaction information in dynamic, complex, long-context environments, motivating persistent memory systems with lower overhead.
Method
LightMem combines lightweight sensory filtering and topic grouping, topic-aware short-term consolidation, and sleep-time long-term updates performed offline.
Results
LightMem consistently surpasses strong baselines across LongMemEval and LoCoMo with GPT and Qwen backbones, improving accuracy while reducing token usage and API calls.
Takeaways & Limitations
LightMem maintains strong task performance while sharply reducing computational costs across different models and scenarios.
Takeaways & Limitations
Deployments must address privacy risks from storing sensitive dialogue histories and risks of perpetuating user-provided bias or misinformation.

Abstract

from arXiv · show

Despite their remarkable capabilities, Large Language Models (LLMs) struggle to effectively leverage historical interaction information in dynamic and complex environments. Memory systems enable LLMs to move beyond stateless interactions by introducing persistent information storage, retrieval, and utilization mechanisms. However, existing memory systems often introduce substantial time and computational overhead. To this end, we introduce a new memory system called LightMem, which strikes a balance between the performance and efficiency of memory systems. Inspired by the Atkinson-Shiffrin model of human memory, LightMem organizes memory into three complementary stages. First, cognition-inspired sensory memory rapidly filters irrelevant information through lightweight compression and groups information according to their topics. Next, topic-aware short-term memory consolidates these topic-based groups, organizing and summarizing content for more structured access. Finally, long-term memory with sleep-time update employs an offline procedure that decouples consolidation from online inference. On LongMemEval and LoCoMo, using GPT and Qwen backbones, LightMem consistently surpasses strong baselines, improving QA accuracy by up to 7.7% / 29.3%, reducing total token usage by up to 38x / 20.9x and API calls by up to 30x / 55.5x, while purely online test-time costs are even lower, achieving up to 106x / 117x token reduction and 159x / 310x fewer API calls. The code is available at https://github.com/zjunlp/LightMem.

1 INTRODUCTION

LLM memory systems preserve historical interaction information for long-context and multi-turn reasoning, but existing approaches can incur substantial inefficiency from redundant data and maintenance overhead. LightMem addresses this trade-off and reports stronger accuracy alongside lower resource use across LongMemEval and LoCoMo.

Memory systems give LLMs persistent state across extended interactions, supporting coherence and personalization beyond stateless responses.
Challenges: Existing memory systems face inefficiency because long interactions contain redundant user inputs and model responses that may be irrelevant to later memory construction.
Results and Evaluation: LightMem improves LongMemEval accuracy by 2.09%–6.40% with GPT and up to 7.67% with Qwen while reducing total tokens and API calls.Overall reductions reach 38× tokens and 30× API calls for GPT, and 21.8× tokens and 17.1× API calls for Qwen.
Results and Evaluation: On LoCoMo, LightMem improves accuracy by 6.10%–29.29% and reduces token usage by up to 20.92× and API calls by up to 55.48%.

2 PRELIMINARY

Conventional LLM memory pipelines construct and update persistent memory before retrieving relevant entries for response generation. Their costs arise from redundant sensory processing and difficult granularity trade-offs in short-term memory.

2.1 CONVENTIONAL MEMORY SYSTEMS FOR LLMS: Conventional memory systems construct memory by segmenting raw data, summarizing segments into entries, organizing them in structural backends, and updating conflicts or outdated content.
2.1 CONVENTIONAL MEMORY SYSTEMS FOR LLMS: They retrieve relevant memory entries for each new query, integrate them with the query into a prompt, and invoke the model to generate a response.
2.2 LIMITATIONS OF EXISTING LLM MEMORY SYSTEMS: Current systems often use stronger LLMs for summarization and topic-level processing, so feeding raw data directly wastes resources and can weaken in-context learning through redundancy.
2.2 LIMITATIONS OF EXISTING LLM MEMORY SYSTEMS: Fixed input granularity creates a trade-off: fine-grained data increases latency and underuses short-term memory, whereas coarse-grained data can reduce effectiveness.

3 LIGHTMEM ARCHITECTURE

LightMem uses three lightweight memory modules: sensory compression and topic segmentation, topic-aware short-term consolidation, and sleep-time long-term updates decoupled from online inference.

3 LIGHTMEM ARCHITECTURE: LightMem combines sensory memory, topic-aware short-term memory, and long-term memory updated during sleep time.These modules selectively preserve salient information, process topic-grouped content, and reduce test-time update latency.
3.1 LIGHT1: COGNITIVE-INSPIRED SENSORY MEMORY: Pre-compression removes redundant tokens before topic segmentation forms semantic segments for memory construction.The sensory buffer triggers hybrid segmentation when it reaches capacity.
3.1 LIGHT1: COGNITIVE-INSPIRED SENSORY MEMORY: Tokens are retained when their predicted retention probability exceeds a dynamic percentile threshold.The compression model treats retention as binary token classification and selects tokens above the threshold.
3.1 LIGHT1: COGNITIVE-INSPIRED SENSORY MEMORY: Higher conditional entropy identifies uncertain, less predictable tokens as informationally unique and important for semantic expression.LightMem retains these distinctive tokens for subsequent memory construction.
3.1 LIGHT1: COGNITIVE-INSPIRED SENSORY MEMORY: Topic boundaries combine attention-based boundaries with similarity-based boundaries by taking their intersection.Attention identifies local maxima between consecutive sentences, while low adjacent-turn similarity supplies additional boundaries.
3.2 LIGHT2: TOPIC-AWARE SHORT-TERM MEMORY: Topic segments enter an STM buffer, where threshold-triggered summarization produces topic-indexed LTM entries containing summaries, user turns, and model turns.Topic-constrained inputs reduce API calls while preserving summarization accuracy and stable performance.
3.3 LIGHT3: LONG-TERM MEMORY WITH SLEEP-TIME UPDATE: At test time, LightMem directly inserts incoming entries into LTM, then constructs similarity-based, timestamp-constrained update queues for offline processing.Later entries may update earlier ones, and independent queues can be updated in parallel.

4 COMPLEXITY ANALYSIS ABOUT LIGHTMEM

The complexity analysis contrasts conventional per-turn summarization and updates with LightMem’s compressed, buffered, and selective processing, yielding lower call frequency and runtime complexity.

4 COMPLEXITY ANALYSIS ABOUT LIGHTMEM: Conventional memory systems summarize every turn, requiring N API calls and N(Lsum-in + T + Lsum-out) summarization tokens.Their update phase additionally costs NM1R1(Lup-in + Lup-out) tokens.
4 COMPLEXITY ANALYSIS ABOUT LIGHTMEM: LightMem retains only rxT tokens after x pre-compression iterations and summarizes only when its STM buffer reaches capacity th.This reduces summarization frequency to NrxT/th calls.
4 COMPLEXITY ANALYSIS ABOUT LIGHTMEM: Stricter semantic-similarity and timestamp filters reduce the fraction R2 of LightMem entries that trigger updates.The update phase requires NrxT/th M2R2 calls and costs NrxT/th M2R2(Lup-in + Lup-out) tokens.
4 COMPLEXITY ANALYSIS ABOUT LIGHTMEM: LightMem reduces runtime complexity relative to the O(N) complexity of other memory systems.The analysis attributes the efficiency gain to compressed summarization and selective updates.

5 EXPERIMENTS

LightMem is evaluated on LongMemEval and LoCoMo with multiple LLM backbones and memory baselines, measuring both answer quality and computational efficiency. Results show broad effectiveness and efficiency gains, while analyses examine compression, topic segmentation, STM thresholds, and update behavior.

Datasets, Baselines, and Metrics: LightMem is evaluated on LongMemEval-S and LoCoMo against six conversational-memory baselines using GPT, Qwen, and GLM backbones.Effectiveness is measured by ACC, while efficiency tracks memory-bank construction costs.
Main Results: LightMem demonstrates superior effectiveness and efficiency on both datasets across GPT and Qwen backbones using combined online and offline costs.The reported comparisons combine LightMem’s online and offline efficiency metrics.
LongMemEval: 2.09%–6.40% GPT ACC gains and up to 7.67% Qwen ACC gains over A-Mem are reported on LongMemEval.Total token usage falls by up to 38× for GPT and 21.8× for Qwen, while API calls fall by up to 30× and 17.1×, respectively.
Online Test-Time Costs: 31.4×–105.9× GPT and 30.1×–117.1× Qwen online token reductions accompany API-call reductions of up to 159.4× and 309.9×.These figures refer only to online test-time costs.
LoCoMo: On LoCoMo, ACC improves by 6.10%–18.12% with GPT and 4.41%–29.29% with Qwen, alongside reductions in tokens, API calls, and runtime.GPT total-token efficiency improves 2.87×–20.92×, while Qwen uses 3.33×–18.02× fewer total tokens.

6 RELATED WORK

Prior work addresses prompt compression, document chunking, and memory systems for LLM agents. These approaches reduce redundancy, organize retrievable units, and support adaptation beyond stateless interactions.

Memory Systems for LLM Agents: LightMem’s soft update preserves the full context while appending new information such as a Tokyo trip and Kyoto inquiry.The example illustrates an alternative to hard prompt compression through direct information appending.
Hard Prompt Compression for LLMs: Hard prompt compression removes redundant prompt content to improve LLM efficiency.Approaches have progressed from smaller language models toward query-aware methods that preserve task-relevant information.
Chunking Strategies in RAG Systems: RAG systems divide external documents into smaller retrieval units using rule-based, semantic, or LLM-driven chunking strategies.Semantic methods group content by topic, while other methods use fixed rules or model knowledge.
Memory Systems for LLM Agents: Memory systems help LLM agents move beyond stateless interactions by supporting reasoning and adaptation in complex, changing environments.Existing systems include linear or sequential experience storage, sometimes enhanced with hierarchical structures.

7 CONCLUSION

LightMem is presented as a lightweight memory framework that filters, organizes, and consolidates information while reducing computational costs. The paper also identifies future extensions and ethical safeguards relevant to deployment.

Conclusion: LightMem uses three light modules for sensory filtering, topic-aware short-term processing, and long-term memory with sleep-time updates.The design is inspired by the multi-stage Atkinson-Shiffrin human memory model.
Conclusion: Empirical evaluation indicates that LightMem maintains strong task performance while sharply reducing computational costs.The conclusion characterizes this as the central outcome of the proposed framework.
Conclusion: Planned extensions include offline pre-computed KV caches, lightweight knowledge graphs, and multimodal memory.These extensions target update runtime, explicit multi-hop reasoning, structured retrieval, and visual, auditory, and textual inputs.
Ethics Statement: External memories of user interactions create privacy, bias, and misinformation risks that require safeguards such as anonymization and user consent.The ethics statement also recommends mechanisms to mitigate biased or false memories.
Conclusion: Implementation details are described in Section 3 and Appendix C, while source-code release is planned for future reproducibility support.The passage frames these measures as facilitating verification and replication by other researchers.

A.1 BACKGROUND ABOUT CURRENT MEMORY SYSTEMS

LLM memory pipelines separate memory-bank construction from retrieval and question answering. LightMem differs from mainstream systems by decoupling updates from online interaction, while using additional model components documented alongside shared models.

Memory Pipeline: Memory systems comprise memory-bank construction followed by retrieval and question answering.Construction contains segmentation, summarization or extraction, storage, and updating operations.
Memory Bank Construction: LightMem decouples memory updates from online interaction, reducing test-time latency relative to baseline systems that update during interaction.The construction pipeline distinguishes online test-time processes from offline processes.
Memory Bank Construction: Table 4 compares mainstream and LightMem memory-bank construction workflows, with black processes executed online and red processes offline.This encoding highlights where LightMem moves computation outside online interaction.
Model Mapping: Table 5 maps pipeline functions to their roles and concrete models, distinguishing models shared with baselines from models unique to LightMem.LightMem introduces an additional model, while other models are shared across methods.
Retrieval and Usage: After construction, a new query retrieves relevant memory entries, appends them to the query, and sends the result to the chat model.Retrieval and response generation form the usage stage of the memory system.

C.1 TOPIC SEGMENTATION

LightMem segments dialogue topics using attention patterns over buffered user sentences. Local peaks identify boundaries, and examples show fine-grained alignment with true topic transitions across configurations evaluated by accuracy and efficiency metrics.

Topic Segmentation: The topic-segmentation procedure extracts user sentences and stores them sequentially in a buffer before segmentation.User sentences are selected because they are generally more concise, while LLMLingua-2 has a 512-token maximum input length.
Configuration Analysis: The STM threshold study compares Accuracy, token consumption, API Calls, and Runtime across model configurations and compression ratios.Figure 4 normalizes six metrics for radar-chart visualization, including input, output, and total tokens.
Topic Segmentation: Residual fragments after segmentation carry into the next buffer, allowing processing to continue until the dialogue ends.This preserves unfinished material across iterative buffer operations.
Topic Segmentation: Local peaks in adjacent-sentence attention scores determine topic segmentation points.The method uses the outermost attention layer and marks a sentence when its score exceeds both neighboring positions.
Topic Segmentation: The illustrated segmentation cases closely align with most true boundaries while providing finer-grained segmentation.The reported examples support reliable detection of topic boundaries.
Category-wise Accuracy: Retrieval-augmented and memory-centric methods generally outperform Full Text on Temporal, Multi-Session, and Knowledge-Update categories.Naive RAG is often competitive for Single-User and Single-Assistant questions, while Single-Preference has higher variance.

C.3 DETAILED PARAMETER ANALYSIS

This section reports a parameter analysis of LightMem’s compression ratio and short-term-memory threshold, using numerical results to examine their effects.

The analysis evaluates LightMem’s compression ratio r and STM threshold th.
Table 9 reports the numerical effects of varying compression ratio r and STM threshold th.
The reported parameter effects concern LightMem’s memory configuration rather than a comparison with another memory system.

D.1 DATASETS AND BASELINES

The evaluation uses LongMemEval and LoCoMo to assess conversational memory across long dialogues, with representative memory-modeling baselines and reported parameter-analysis tables.

Datasets: LongMemEval contains 500 evaluation questions based on extended user-assistant dialogues, with LONGMEMEVAL-S averaging approximately 115k tokens per problem.The work adopts LONGMEMEVAL-S because it balances evaluation considerations described in the dataset passage.
Datasets: Five LongMemEval samples with corrupted characters were discarded during LightMem compression and counted as false in accuracy results.The affected sample indices are 74, 183, 278, 351, and 380.
Datasets: LoCoMo evaluates long-range conversational memory using conversations of roughly 300 turns and around 9K tokens on average.Its questions cover Single-hop, Multi-hop, Temporal, and Open-domain categories.
Baselines: The compared conversational-memory baselines include LANGMEM, A-MEM, and MEMORYOS.These baselines represent long-term memory modules, memory-centric knowledge graphs, and OS-inspired conversational-memory hierarchies, respectively.

D.2 IMPLEMENTATION DETAILS

The implementation uses specified GPU, CPU, and memory hardware and evaluates standard, single-session preference, and unanswerable-question tasks with yes-or-no grading prompts.

Hardware: All experiments run on 4 NVIDIA RTX 3090 GPUs, dual Intel Xeon Gold 6133 CPUs, and 256 GB of RAM.The CPUs provide 40 cores and 80 threads.
Tasks: Standard task evaluation covers single-session user-assistant and multi-session settings.
Evaluation: Standard responses are graded yes when they contain the correct answer, an equivalent answer, or all intermediate steps needed to obtain it.Responses containing only a subset of the required information are graded no.
Tasks: Single-session preference tasks judge whether responses correctly recall and use the user’s personal information according to a rubric.The response need not reflect every rubric point if it recalls and utilizes the personal information correctly.
Tasks: Unanswerable-question tasks judge whether the model correctly identifies that the requested information cannot be answered.The response may state that information is incomplete or that only other information is available.