Source-linked AI summary
Rethinking How to Remember: Beyond Atomic Facts in Lifelong LLM Agent Memory
Jingwei Sun, Jianing Zhu, Jiangchao Yao, Tongliang Liu, Bo Han
TL;DR
Existing fact-centric memory systems can lose dialogue detail, limit reasoning over scattered information, and struggle with heterogeneous dialogue styles. TriMem combines raw dialogue, atomic facts, synthesized profiles, and TextGrad prompt optimization, consistently outperforming prior memory approaches across benchmarks and models.
Problem
Existing fact-based agent memory systems face lossy storage, shallow reasoning, and inconsistent extraction granularity across heterogeneous dialogues.
Method
TriMem combines source-linked raw dialogues, extracted facts, synthesized entity profiles, and TextGrad optimization of extraction and profiling prompts.
Results
TriMem consistently outperforms prior memory approaches across benchmarks and high-capability models, while retrieved contexts consume around 1.2k tokens.
Takeaways & Limitations
Maintaining multiple representation granularities balances storage fidelity, retrieval efficiency, and reasoning quality in LLM agent memory systems.
Takeaways & Limitations
TextGrad prompt optimization relies mainly on response-level feedback rather than intermediate retrieval or step-wise reasoning supervision.
Abstract
from arXiv · showhide
To enable reliable long-term interaction, LLM agents require a memory system that can faithfully store, efficiently retrieve, and deeply reason over accumulated dialogue history. Most existing methods adopt an extracted fact based paradigm: handcrafted static prompts compress raw dialogues into atomic facts, which are then stored, matched, and injected into downstream reasoning. Nevertheless, such fact-centric designs inevitably discard fine-grained details in original dialogues and fail to support deep reasoning over scattered isolated facts. Moreover, static prompts cannot maintain consistent extraction granularity across diverse dialogue styles. To address these limitations, we propose TriMem, which maintains three coexisting representation granularities, including raw dialogue segments anchored by source identifiers for storage fidelity, extracted atomic facts for efficient memory retrieval, synthesized profiles that aggregate dispersed facts into holistic semantic understanding for deep reasoning. We further adopt TextGrad-based prompt optimization, which iteratively refines extraction and profiling prompts via response quality feedback, achieving lifelong evolution without any parameter updating. Extensive experiments on LoCoMo and PerLTQA across multiple LLM backbones demonstrate that TriMem consistently outperforms strong memory baselines. The code is available at https://TMLR-TriMem.github.io .
1 Introduction
The introduction argues that fact-centric agent memory loses dialogue details, limits reasoning beyond isolated facts, and cannot reliably maintain extraction granularity across heterogeneous interactions. It presents TriMem as a three-granularity architecture combining raw dialogue, extracted facts, and synthesized profiles, with iteratively optimized prompts.
- Motivation: Long-context, multi-turn interactions exceed LLM context windows, motivating memory modules that store historical information, retrieve relevant memories, and support reasoning.Existing systems typically extract facts, match them against queries, and inject relevant information into prompts.
- Limitations: Static prompts also raise concerns about maintaining consistent and rational extraction granularity across heterogeneous real-world dialogues.This limitation is posed alongside questions about whether extracted facts can beneficially support storage, retrieval, and reasoning.
- Limitations: Fact-centric memory suffers information loss, while many real-world questions require understanding scattered information rather than simple fact matching.The introduction evaluates existing systems across storage fidelity, retrieval efficiency, and reasoning quality.
- TriMem: TriMem maintains verbatim dialogues, extracted facts, and synthesized profiles, using source dialogue identifiers to preserve original segments and incrementally integrating scattered facts into holistic understanding.The architecture targets storage fidelity, efficient retrieval, and deeper reasoning across coexisting representation granularities.
- Contributions: TriMem introduces an entity profile module, preserves raw-dialogue identifiers, and iteratively optimizes system prompts to address shallow reasoning, detail loss, and performance fluctuation.The paper frames these mechanisms as its main technical contributions and reports extensive evaluations across scenarios, benchmarks, model structures, and model sizes.
2 Preliminary and Motivation
Conventional agent memory systems follow storage, retrieval, and reasoning phases centered on extracted facts, which improves evidence localization but introduces lossy storage, shallow reasoning, and prompt-granularity instability. These limitations motivate preserving dialogue details while supporting efficient retrieval and deeper reasoning.
- Conventional Memory Architecture: Conventional memory systems partition historical dialogue into windows, extract factual entries with a fixed prompt, retrieve top-K entries by similarity, and generate responses from retrieved entries plus the question.The architecture comprises storage, retrieval, and reasoning phases.
- Efficient Retrieval: Extracted-fact retrieval localizes relevant memory entries more efficiently and accurately than full-context retrieval, which tends to introduce massive irrelevant information.The comparison evaluates the ratio of successfully retrieved question-related evidence.
- Lossy Storage: Extracted-fact storage loses 14.5% more information than original dialogue, discarding fine-grained details and preventing accurate answers to detail-dependent, high-precision queries.This loss arises from lossy compression during fact extraction and can permanently remove semantic details from stored memory.
- Shallow Reasoning: Reasoning over extracted facts is considerably weaker for multi-evidence questions than for single-evidence questions, revealing a shallow-reasoning bottleneck.The analysis tests reasoning performance on questions whose relevant entries were correctly retrieved.
- Suboptimal Prompt: Fixed hand-written extraction prompts fail to adapt consistently to diverse information styles, expression patterns, and content categories, causing unstable extraction granularity and performance degradation.Realistic long-term interactions expose this limitation because conventional systems rely on static prompts.
3 Method
TriMem uses three linked memory representations—raw dialogue, extracted facts, and integrated profiles—to preserve storage fidelity, enable efficient retrieval, and support deeper reasoning. Its prompts are iteratively optimized with TextGrad feedback so extraction and profiling can evolve without parameter updates.
- Three-level memory architecture: TriMem builds memory by extracting facts, binding them to raw-dialogue and profile identifiers, and retrieving facts before recovering detailed dialogues and integrated profiles.The framework analyzes a question into required information and keywords, matches them against extracted facts, and then obtains raw dialogues and profiles.
- Fact extraction: Overlapping sliding-window segmentation preserves context across boundaries before an agent performs schema-driven, multi-dimensional fact extraction.The schema supports lossless restatement, temporal references, persons, locations, keywords, and named entities.
- Fact extraction and retrieval: The added fsrc extraction dimension records each fact’s source dialogue identifier, addressing lossy storage while query keywords and similarity matching retrieve the top-K relevant entries.The source identifier is defined as ei.src ≜ {rt,ut ∈ wi}.
- Profile synthesis: TriMem groups facts by person and synthesizes structured profiles that capture identity, personality, background, interests, interpersonal dynamics, and behavioral tendencies.Profiles are generated with a profile prompt from person-specific entries, supporting understanding based on scattered facts.
- Lifelong prompt evolution: TextGrad uses answer-quality failure signals to jointly refine extraction and profile prompts, progressively improving detail retention and entity-level understanding without manual prompt engineering.The prompts are treated as joint trainable parameters and updated through natural-language feedback rather than parameter updating.
4 Experiments
Experiments evaluate TriMem against established memory systems on LoCoMo and PerLTQA using both high-capability and efficient models, and analyze the contributions of its components and design choices. TriMem consistently delivers strong performance, while ablations identify effective settings for profiles, raw dialogue, prompt evolution, retrieval, and window size.
- Baselines and Benchmarks: TriMem is compared with Naive RAG and competitive memory systems on the LoCoMo and PerLTQA benchmarks.Compared systems include Mem0, MemoryOS, A-Mem, LightMem, SimpleMem, and xMemory, using their original hyperparameter settings.
- Performance on High-Capability Models: TriMem consistently outperforms prior approaches when integrated with GPT-4o, GPT-4.1-mini, and GPT-5-nano.The experiments also report average retrieved-context token consumption to assess information density.
- Compatibility with Efficient Models: TriMem remains compatible with Qwen3-8B and Llama-3.1-8B-Instruct and achieves substantial performance improvements on efficient models.Unlike xMemory, TriMem does not require model output logits and therefore supports more model configurations.
- Generalization on Different Datasets: TriMem demonstrates strong generalization on PerLTQA, which evaluates personal profiles, social relationships, historical events, and dialogue memories.Table 3 reports performance across the benchmark’s multidimensional long-term agent QA tasks.
- Ablation of Profile and Raw Dialogue: Removing either entity profiles or raw dialogues causes a noticeable performance drop, confirming that both components improve reasoning reliability and storage fidelity.The best results occur when entity profiles and raw dialogue are incorporated together.
- Ablation Studies: Performance improves through 4 evolution steps, but further updates cause excessive prompt refinement; retrieval is optimal at 25 entries, while search queries improve performance despite added retrieval time.A window size of 40 balances memory-construction efficiency with competitive performance.
5 Related Work
Prior LLM-agent memory systems organize around construction, retrieval, and reasoning, commonly extracting and matching atomic facts before supplying them as context. Related lifelong-evolution work instead investigates improving memory management from accumulated experience, including reinforcement-learning formulations.
- Memory Systems for LLM Agents: Most LLM-agent memory systems comprise memory construction, memory retrieval, and memory-supported reasoning stages.Construction extracts and consolidates dialogue information, retrieval matches queries with stored representations, and reasoning uses retrieved information as contextual supplements.
- Memory Systems for LLM Agents: Mem0, A-Mem, and MemoryOS extract atomic facts and consolidate them through dynamic updates or hierarchical stores.These systems represent a prominent construction paradigm for long-term agent memory.
- Memory Systems for LLM Agents: Naive RAG through xMemory retrieves fact embeddings by similarity search and concatenates retrieved facts into prompts for downstream reasoning.The cited systems differ in implementation but share fact-oriented retrieval and contextual supplementation.
- Lifelong Evolution Agents: Lifelong-evolution research seeks to improve LLM agents from accumulated experience over long horizons, with one prominent line framing memory management as reinforcement learning.MemAgent, MemBuilder, and AgentFold exemplify this direction through multi-conversation memory, reward-based construction, and proactive context management.
6 Conclusion · Appendix
The paper identifies lossy storage, shallow reasoning, and suboptimal prompts as limitations of extracted-fact memory systems, then proposes TriMem with three coexisting representation granularities.
- 6 Conclusion: TriMem addresses three limitations of extracted-fact memory: lossy storage, shallow reasoning, and suboptimal prompts for heterogeneous dialogue styles.These limitations motivate the proposed memory-system redesign.
- 6 Conclusion: TriMem maintains verbatim dialogues to preserve storage fidelity.Verbatim dialogue is one of the system’s three coexisting representation granularities.
- 6 Conclusion: TriMem uses atomic facts to support retrieval efficiency.Atomic facts form the retrieval-oriented representation granularity.
- 6 Conclusion: TriMem progressively synthesizes profiles to enable deep reasoning.Synthesized profiles aggregate information beyond isolated extracted facts.
- 6 Conclusion: The proposed design revisits memory systems for long-term LLM-agent interaction.The redesign is framed as a response to limitations in the prevailing paradigm.
- 6 Conclusion: The three representations coexist rather than replacing one another.Their distinct roles jointly target storage fidelity, retrieval efficiency, and deep reasoning.
A Details about Baselines and Benchmarks · B More Experimental Details
The supplied material provides an outline of the paper’s experimental-details section, covering retrieval efficiency, storage completeness, and reasoning performance. It also lists discussion subsections on limitations, future work, and broader impact.
- B More Experimental Details: The experimental-details section includes a subsection on retrieval efficiency.This subsection is listed as B.1.
- B More Experimental Details: The experimental-details section includes a subsection on storage completeness.This subsection is listed as B.2.
- B More Experimental Details: The experimental-details section includes a subsection on reasoning performance.This subsection is listed as B.3.
- B More Experimental Details: The experimental-details material is organized around retrieval, storage, and reasoning dimensions.These are the three subsection titles listed under section B.
- C Discussion: The outline separately identifies a discussion subsection on limitations and future work.This subsection is listed as C.1.
- C Discussion: The outline separately identifies a discussion subsection on broader impact.This subsection is listed as C.2.
D Additional Experimental Results · E System Prompts · Reproducibility Statement
The supplementary material organizes additional experiments, system prompts, and reproducibility information for TriMem. It covers ablations, prompt evolution, retrieval and query examples, window-size effects, prompt specifications, datasets, assumptions, code, and compute environment.
- D Additional Experimental Results: D.1–D.5 cover ablations, prompt-evolution visualization, retrieval-entry content, search-query examples, and different window sizes.These topics are listed as additional experimental results.
- E System Prompts: E.1–E.5 provide prompts for memory extraction, profile construction, target-query generation, key-information analysis, and question reasoning.The system-prompt appendix enumerates these five prompt categories.
- Reproducibility Statement: The authors provide source code through an anonymous link to support reproducibility.The statement identifies the anonymous TriMem repository as a reproducibility resource.
- Reproducibility Statement: The experiments use publicly accessible datasets and evaluate four LoCoMo QA categories: MultiHop, Temporal, OpenDomain, and SingleHop.The categories follow previous work as described in the reproducibility statement.
- Reproducibility Statement: The experimental setup assumes an available pretrained open-source or closed-source LLM that receives information incrementally for memory storage.During memory storage, the agent cannot access relevant problem information and must analyze only the original information.
- Reproducibility Statement: All experiments run on NVIDIA 4090-24GB GPUs using Python 3.10.The environment specification is stated explicitly for reproduction.
A Details about Baselines and Benchmarks … C Discussion
This section defines the baselines and benchmarks used to evaluate long-term conversational memory, then details experiments measuring retrieval efficiency, storage completeness, and reasoning performance. The experiments test whether retrieving individual facts is sufficient for multi-fact reasoning.
- A Details about Baselines and Benchmarks: Full-Context concatenates entire conversations without compression or retrieval, providing an accuracy upper bound but incurring token costs and latency that scale linearly with conversation length.It is therefore impractical for real-world long-horizon deployments.
- A Details about Baselines and Benchmarks: Naive RAG retrieves top-k fixed-size dialogue chunks by cosine similarity without memory consolidation, updates, or deletion.The retrieved context remains raw stored text.
- A Details about Baselines and Benchmarks: Mem0 dynamically extracts candidate facts and applies ADD, UPDATE, DELETE, or NOOP operations against existing memories before retrieval.Mem0 retrieves the resulting memories by prepending them to the inference prompt.
- A Details about Baselines and Benchmarks: MemoryOS uses hierarchical short-, mid-, and long-term memory with FIFO updates, segmented promotion, heat scores, and a persona module.Its storage design balances visit frequency and recency during memory promotion.
- A Details about Baselines and Benchmarks: A-Mem organizes dynamically interlinked Zettelkasten-style notes and evolves existing contextual representations when new memories are integrated.New notes include contextual descriptions, keywords, and tags, while historical memories are analyzed for semantic links.
- A Details about Baselines and Benchmarks: LoCoMo evaluates very long-term conversational memory across conversations of up to 35 sessions, averaging 300 turns and 9K tokens, using three tasks.Its dialogues are generated through a machine-human hybrid pipeline grounded in personas and temporal event graphs, then human-verified and edited.
- A Details about Baselines and Benchmarks: PerLTQA evaluates personalized long-term memory across character profiles, social relationships, events, and dialogues with 8,593 questions spanning 30 characters.Its evaluation includes Memory Classification, Memory Retrieval, and a third subtask described in the benchmark passage.
- B.1 Retrieval Efficiency / B.2 Storage Completeness / B.3 Reasoning Performance: The experiments measure retrieval by Hit@k with k = 5, storage completeness by reference-answer token coverage, and reasoning separately for one-evidence versus multi-evidence questions.For reasoning evaluation, retrieval failures are filtered first; even successfully retrieved scattered facts remain insufficient for complex multi-fact fusion and reasoning.
C.1 Limitation and Future Work · C.2 Broader Impact · D Additional Experimental Results
TriMem identifies storage loss, shallow reasoning, and prompt weaknesses in fact-centric agent memory, while using three representation granularities and evolving prompts to improve lifelong memory. Future work targets richer relational profiles and finer-grained optimization feedback.
- C.1 Limitation and Future Work: TriMem identifies lossy storage, shallow reasoning, and suboptimal prompts as three critical limitations of extracted fact-based agent memory.These limitations arise under heterogeneous dialogue styles.
- C.1 Limitation and Future Work: Future profile construction could extend beyond person-level entities to event chains, social networks, and temporal trajectories for broader semantic understanding.The current entity-profile implementation already provides relatively satisfactory support for deep reasoning.
- C.1 Limitation and Future Work: TextGrad prompt optimization currently relies mainly on response-level feedback signals.Future mechanisms could incorporate intermediate retrieval feedback or step-wise reasoning trajectory supervision.
- C.2 Broader Impact: Reliable memory systems are important for storing, retrieving, and reasoning over dialogue histories in real-world LLM-agent applications.The passage highlights personalized assistants, healthcare consultation, and educational tutoring as long-horizon interaction fields.
- C.2 Broader Impact: Fact-centric systems can lose information and reason shallowly because they use extracted atomic facts across storage, retrieval, and reasoning.This is identified as a crucial and previously overlooked issue in existing agent memory systems.
- C.2 Broader Impact: TriMem’s three-level architecture combines traceable raw-dialogue indices for storage fidelity with synthesized profiles for understanding-aided reasoning.The passage states that comprehensive experiments verified the method’s effectiveness and compatibility.
D.1 Ablation of Profile and Raw Dialogue · D.2 Visualization of Prompt Evolution
D.1 evaluates TriMem against profile- and raw-dialogue-removal variants on LoCoMo across four task types using BLEU and F1-score, while D.2 visualizes TextGrad-driven prompt refinement from concrete failures.
- D.1 Ablation of Profile and Raw Dialogue: D.1 compares full TriMem with variants removing entity profiles or raw dialogue on LoCoMo across MultiHop, Temporal, OpenDomain, and SingleHop tasks.The evaluation uses BLEU and F1-score.
- D.1 Ablation of Profile and Raw Dialogue: The entity-profile ablation study reports detailed results contrasting TriMem with the w/o profile variant.
- D.1 Ablation of Profile and Raw Dialogue: The raw-dialogue ablation study reports detailed results contrasting TriMem with the corresponding raw-dialogue-removal variant.
- D.2 Visualization of Prompt Evolution: D.2 visualizes three TextGrad-style evolution traces in which early-round failure cases prompt additions to extraction or profile-construction prompts.The traces show prompts becoming increasingly detailed.
- D.2 Visualization of Prompt Evolution: The evolved prompts address multi-fact dialogue collapse by preventing several facts from being compressed into a single summary entry.
- D.2 Visualization of Prompt Evolution: The profile-construction prompt is refined to preserve exact adjectives when constructing profiles.
- D.2 Visualization of Prompt Evolution: Prompt evolution adds normalization of relative temporal references to absolute dates after failures left event timestamps unresolved for date-targeted questions.The failure involved expressions such as “last week,” “yesterday,” and “next month.”
- D.2 Visualization of Prompt Evolution: One illustrated case asks when Melanie bought the figurines, with a reference answer of 21 October 2023 and memory stating she bought wooden dolls last Friday.
D.3 Content of Retrieval Entries … E.4 Prompt for Key Information Analysis
TriMem demonstrates precise retrieval and query decomposition, while window-size analysis motivates a 40-dialogue setting. Its system prompts specify structured extraction, profile construction, targeted search, and key-information analysis procedures.
- D.3 Content of Retrieval Entries: TriMem precisely retrieves relevant memory entries for distinct questions, including necklace symbolism and poetry-reading poster content.The necklace example retrieves the reference answer “love, faith, and strength,” while the poetry-reading example retrieves “Trans Lives Matter.”
- D.4 Example of Search Query: Required-information analysis decomposes questions into detailed search targets, enabling more accurate retrieval.For the necklace question, generated targets include symbolism, context, and background; the poster question targets text displayed at the event.
- D.5 Performance of Different Window Size: A larger extraction window can drop low-salience facts or merge atomic facts into bloated entries, so the main text sets the window size to 40.The comparison shows two facts extracted at window size 10 but “None” at window size 50.
- E.1 Prompt for Memory Extraction: The memory-extraction prompt requires factual, source-traceable, disambiguated, temporally normalized, atomic, and detail-preserving entries.It skips social gestures, records source dialogue IDs, replaces pronouns with names, resolves relative dates, separates facts, preserves exact entities, and identifies described-but-unnamed things.
- E System Prompts: The profile-construction prompt updates only affected sections, preserves noncontradicted information, infers traits and preferences, and organizes entity profiles into semantic categories.Its format covers identity, personality, descriptions by others, interests, career, values, spirituality, relationships, life events, and preferences.
- E System Prompts: The target-query prompt analyzes question type, entities, relationships, and minimal required information before generating one to three focused, nonredundant searches.It always includes the original query and prioritizes fewer, more targeted queries for efficiency.
- E.4 Prompt for Key Information Analysis: The key-information prompt extracts query keywords, persons, time expressions, locations, and entities as JSON.It requires JSON-only output with fields for keywords, persons, time_expression, location, and entities.