PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking How to Remember: Beyond Atomic Facts in Lifelong LLM Agent Memory

Jingwei Sun, Jianing Zhu, Jiangchao Yao, Tongliang Liu, Bo Han

arXiv:2605.19952v1cs.CL

TL;DR

Fact-centric agent memories raise questions about preserving dialogue detail, reasoning over scattered facts, and maintaining consistent extraction across dialogue styles. TriMem combines dialogue, facts, and profiles with prompt optimization, and reportedly outperforms prior approaches across tested high-capability models.

  • Problem

    Fact-centric memory raises questions about preserving dialogue details, reasoning beyond fact matching, and maintaining consistent extraction across heterogeneous dialogue styles.

    Source: §1 · p. 2

  • Method

    TriMem retains verbatim dialogue, atomic facts, and synthesized profiles, while using response-quality feedback to refine extraction and profiling prompts.

    Source: §1 · p. 2, §3.4 · p. 6

  • Results

    TriMem consistently outperforms prior approaches across GPT-4o, GPT-4.1-mini, and GPT-5-nano, with retrieved contexts consuming around 1.2k tokens.

    Source: §4.2 · p. 8

  • Takeaways & Limitations

    Prompt optimization without modifying model parameters keeps TriMem applicable to API-only LLMs.

    Source: §6 · p. 11

    TriMem's prompt optimization relies mainly on response-level feedback, with finer-grained retrieval or reasoning supervision left for future work.

    Source: §C.1 · p. 19

Abstract

from arXiv · show

To enable reliable long-term interaction, LLM agents require a memory system that can faithfully store, efficiently retrieve, and deeply reason over accumulated dialogue history. Most existing methods adopt an extracted fact based paradigm: handcrafted static prompts compress raw dialogues into atomic facts, which are then stored, matched, and injected into downstream reasoning. Nevertheless, such fact-centric designs inevitably discard fine-grained details in original dialogues and fail to support deep reasoning over scattered isolated facts. Moreover, static prompts cannot maintain consistent extraction granularity across diverse dialogue styles. To address these limitations, we propose TriMem, which maintains three coexisting representation granularities, including raw dialogue segments anchored by source identifiers for storage fidelity, extracted atomic facts for efficient memory retrieval, synthesized profiles that aggregate dispersed facts into holistic semantic understanding for deep reasoning. We further adopt TextGrad-based prompt optimization, which iteratively refines extraction and profiling prompts via response quality feedback, achieving lifelong evolution without any parameter updating. Extensive experiments on LoCoMo and PerLTQA across multiple LLM backbones demonstrate that TriMem consistently outperforms strong memory baselines. The code is available at https://TMLR-TriMem.github.io .

1 Introduction

TriMem addresses information loss, shallow reasoning, and inconsistent extraction granularity in LLM-agent memory by combining raw dialogue, extracted facts, and synthesized profiles. It also uses feedback-driven prompt optimization to adapt memory construction over time.

1 Introduction

  • LLM agents need robust memory because context-window limits in long, multi-turn interactions can discard key history and break context logic.Existing systems commonly use facts extracted by static hand-written prompts for storage, retrieval, and reasoning.Source: §1 · p. 1
  • Fact-centric memory can lose precise dialogue details, require deeper integration than fact matching for some questions, and extract inconsistently across diverse dialogue styles.Original dialogue can recover unstored details, while integrated profiles support holistic understanding of scattered facts.Source: §1 · p. 2
  • TriMem combines verbatim dialogue, extracted facts, and synthesized profiles, linking compressed entries to source segments and incrementally integrating facts for reasoning.TextGrad-based prompt optimization iteratively refines memory-construction prompts using answer feedback.Source: §1 · pp. 2–3, Figure 1 · p. 2
  • The paper frames agent memory across storage fidelity, indexing efficiency, and reasoning quality, and reports evaluating TriMem across benchmarks, model structures, and sizes.The introduction says the paper also examines ablations, but provides no numerical results here.Source: §1 · pp. 2–3

2 Preliminary and Motivation

Conventional agent memory systems store extracted facts, retrieve relevant entries, and reason over them with the query. Their fact-based design improves retrieval localization but can lose dialogue details, limit multi-evidence reasoning, and extract inconsistently across dialogue styles.

2.1 Conventional Agent Memory Systems

  • Conventional memory systems partition dialogue into windows, extract facts with a fixed prompt, retrieve top-K entries by similarity, and provide them with the query for answer generation.Window granularity may be by turn, session, or topic.Source: §2.1 · p. 3

2.2 Analysis of Existing Systems

  • Extracted-fact retrieval localizes relevant memories more efficiently and accurately than full-context RAG, which can return irrelevant lexical matches and distracting content.Facts summarized across multi-turn dialogue support semantic matching; the passage describes the comparison as involving the ratio of successfully retrieved question-related evidence.Source: §2.2 · p. 3, Figure 2 · p. 4
  • Extracted-fact storage loses 14.5% more information than the original dialogue, potentially removing details needed for accurate, complete answers.The passage gives the omitted modifier “trans” as an example and warns that detail-dependent, high-precision queries may suffer.Source: §2.2 · pp. 3–4, Figure 2 · p. 4
  • Reasoning on correctly retrieved multi-evidence questions is considerably inferior to single-evidence questions, which can often be answered by restating explicit facts.Multi-evidence questions require understanding dispersed facts, including emotional inference and logical induction.Source: §2.2 · p. 4, Figure 2 · p. 4
  • Fixed extraction prompts can produce inconsistent fact granularity and fluctuating reasoning performance across diverse dialogues.The passage contrasts explicitly naming the Pomodoro technique with describing it as 25 minutes on and 5 minutes off.Source: §2.2 · p. 4, Figure 2 · p. 4

3 Method

TriMem combines raw dialogue, extracted facts, and integrated profiles to support faithful storage, efficient retrieval, and deeper reasoning, while refining its prompts through performance feedback.

3.1 Overall Pipeline

  • The pipeline links extracted facts to raw-dialogue and profile identifiers, retrieves relevant facts and their linked context, then uses response failures to suggest prompt revisions.Given a question, it first analyzes required information and keywords to form a search query.Source: §3.1 · p. 5

3.2 Dense Storage with Efficient Retrieval

  • Overlapping windows preserve boundary-spanning coherence; multidimensional facts capture semantic details, while source dialogue IDs anchor top-K retrieval to the original dialogue.The extraction schema includes lossless restatements, temporal references, and persons; additional dimensions such as location, keywords, and named entities are extensible.Source: §3.2 · pp. 5–6

3.3 Scattered Fact with Integrated Profile

  • TriMem groups facts by person and synthesizes them into profiles covering identity, inferred traits, backgrounds, interests, and interpersonal or behavioral patterns for reasoning.Retrieved entries connect agents to the corresponding profiles through person identifiers.Source: §3.3 · p. 6

3.4 Lifelong Evolution with Optimized Prompt

  • TextGrad (Yuksekgonul et al., 2024) uses answer-quality feedback against reference answers to iteratively refine extraction and profiling prompts, improving detail retention and entity-level understanding without manual prompt engineering.The feedback signal comes from comparing generated responses with reference answers; prompt refinement targets semantically valuable details and behavioral patterns.Source: §3.4 · pp. 6–7

4 Experiments

TriMem is evaluated on LoCoMo and PerLTQA against Naive RAG and competitive agent-memory systems, with experiments covering main results, compatibility, and ablations. Across the reported settings, results support its performance and design choices, while analyses examine retrieval and construction efficiency.

4.1 Experimental Setups

  • Experiments use LoCoMo and PerLTQA, a 40-token window with stride 38, up to 25 retrieved entries, and four prompt-optimization rounds.Facts are encoded with Qwen3-embedding-0.6b; prompts are optimized on LoCoMo with Qwen3-8B and applied to other models.Source: §4.1 · pp. 7–8

4.2 Main Results

  • TriMem consistently outperforms prior methods with GPT-4o, GPT-4.1-mini, and GPT-5-nano, using around 1.2k tokens for retrieved contexts.Retrieving raw dialogue and entity profiles slightly increases token overhead but brings substantial performance gains.Source: §4.2 · p. 8
  • TriMem improves performance with Qwen3-8B and Llama-3.1-8B-Instruct and supports varied model sizes without requiring output logits.The paper contrasts this compatibility with xMemory’s limited support for open-source models that provide output logits.Source: §4.2 · p. 8, Table 2 · p. 8
  • On PerLTQA, TriMem achieves the best performance across sub-tasks, supporting generalization across profiles, relationships, historical events, and dialogue memories.The benchmark reports LLM-judged correctness scores, with gpt-4.1-mini evaluating answers against ground truth.Source: Table 3 · p. 8, §4.2 · p. 9

4.3 Ablation and Further Analysis

  • Removing either entity profiles or raw dialogues causes a noticeable performance drop, supporting both components’ roles in reasoning reliability and storage fidelity.Figure 4 reports the comparison on LoCoMo.Source: §4.3 · p. 9, Figure 4 · p. 9
  • Performance improves as prompt-evolution steps increase from one to four, then degrades at five as excessive refinement may exceed model capability.The study therefore sets the number of evolution steps to four.Source: §4.3 · p. 9, Figure 5 · p. 9
  • Retrieving too few entries can miss key information, while too many can introduce irrelevant content; performance is optimal at 25 entries.The authors select 25 as a balance for retrieval.Source: Figure 6 · p. 9, §4.3 · p. 10
  • Search queries improve retrieval accuracy despite added retrieval time, while smaller windows increase construction time, motivating the 40-token window for efficiency.Smaller windows can still yield good performance, but require more agent communications; search queries first analyze required information and keywords before retrieval.Source: §4.3 · p. 10, Figure 7 · p. 10, Figure 8 · p. 10

5 Related Work

TriMem positions memory as a pipeline of linked, multi-level representations and evolves its memory prompts without parameter tuning.

Memory Systems for LLM Agents

  • Most LLM-agent memory systems use extracted facts for construction, retrieval, and reasoning, risking dialogue fidelity and holistic understanding; TriMem links facts to source dialogues and builds entity profiles.The granularity of stored memory constrains retrieval, which in turn shapes downstream reasoning.Source: §5 · pp. 10–11

Lifelong Evolution Agents

  • RL-based evolution methods update parameters, while non-updating methods externalize skills or rules; TriMem uses TextGrad (Yuksekgonon et al., 2024) to evolve memory extraction and profile prompts.Unlike skill- or rule-based approaches targeting the agent’s action space, prompt-level evolution reshapes how raw experience is parsed into memory.Source: §5 · p. 11

6 Conclusion · Appendix

TriMem combines verbatim dialogue, atomic facts, and synthesized profiles to address lossy storage, shallow reasoning, and prompt limitations across heterogeneous dialogue styles. Experiments on LoCoMo and PerLTQA with various LLMs confirm its effectiveness.

6 Conclusion

  • Experiments on LoCoMo and PerLTQA across various LLMs confirm TriMem’s effectiveness.The conclusion reports no specific metrics or benchmark comparisons.Source: §6 · p. 11
  • TriMem addresses three limitations of extracted fact-based memory: lossy storage, shallow reasoning, and suboptimal prompts for heterogeneous dialogue styles.Source: §6 · p. 11
  • TriMem combines verbatim dialogues for storage fidelity, atomic facts for retrieval efficiency, and progressively synthesized profiles for deep reasoning.Source: §6 · p. 11
  • TextGrad-based optimization evolves extraction and profile-construction prompts from accumulated experience, enabling lifelong adaptation without modifying model parameters.Source: §6 · p. 11
  • Because prompt optimization does not modify underlying model parameters, TriMem remains applicable to API-only LLMs.Source: §6 · p. 11
  • The authors hope TriMem will offer insights for future research on agent memory systems.Source: §6 · p. 11

Reproducibility Statement

The paper provides an anonymous source-code link and summarizes the datasets, experimental assumption, and computing environment used to facilitate reproduction.

Reproducibility Statement

A Details about Baselines and Benchmarks

The appendix describes baseline memory systems and the LoCoMo and PerLTQA benchmarks used in the work.

Baselines

Benchmarks

  • LoCoMo (Maharana et al., 2024) covers conversations averaging 300 turns and 9K tokens, with question answering, event-graph summarization, and multimodal dialogue generation tasks.Its conversations span up to 35 sessions, and QA includes single-hop, multi-hop, temporal, open-domain, and adversarial reasoning.Source: A Details about Baselines and Benchmarks · p. 17
  • PerLTQA contains 8,593 questions across 30 characters and evaluates memory classification, retrieval, and synthesis across several memory types.The benchmark probes interactions among character profiles, social relationships, events, and dialogues.Source: A Details about Baselines and Benchmarks · p. 17

B More Experimental Details … C Discussion

The supplementary section specifies how retrieval efficiency and storage completeness are measured, and describes reasoning evaluations that find difficulty with complex questions requiring multiple facts.

B More Experimental Details

B.1 Retrieval Efficiency

  • Retrieval efficiency is measured by whether dialogue IDs covered by the top-5 retrieved entries intersect with ground-truth evidence dialogue IDs.The setup embeds extracted lossless restatements for fact retrieval and 5-turn raw-dialogue chunks for full-context retrieval.Source: §B.1 · p. 18

B.2 Storage Completeness

  • Storage completeness is measured as the coverage of reference-answer content tokens by the full raw dialogue or extracted facts.Stopwords are removed from reference answers before token sets are compared.Source: §B.2 · p. 18

B.3 Reasoning Performance

  • Reasoning performance is evaluated separately for questions with exactly one evidence item and questions with multiple evidence items.Samples with retrieval failures are filtered out first.Source: §B.3 · p. 18
  • Even after retrieval succeeds, models struggle with complex synthetic questions requiring multi-fact fusion and reasoning.The evaluation filters retrieval failures and separates questions with exactly one evidence item from those with more than one.Source: §B.3 · p. 18
  • Reasoning experiments also compare samples constructed from dialogues between different users.The evaluation compares reasoning performance across those samples.Source: §B.3 · p. 18

C.1 Limitation and Future Work · C.2 Broader Impact

TriMem’s three-level memory architecture is presented as a promising approach to long-term agent interaction, while the paper identifies open directions for richer profile structures and finer-grained prompt optimization. The authors also emphasize the importance of reliable memory for high-stakes, long-horizon applications.

C.1 Limitation and Future Work

  • The paper identifies lossy storage, shallow reasoning, and suboptimal prompts for heterogeneous dialogue styles as limitations of the prevailing extracted-fact paradigm.It discusses raw dialogues, extracted facts, and synthesized profiles as coexisting granularities balancing fidelity, retrieval efficiency, and reasoning quality.Source: §C.1 · p. 19
  • Future work could extend entity profiles beyond person-level entities to relational structures such as event chains, social networks, and temporal trajectories.Source: §C.1 · p. 19
  • Because TextGrad optimization mainly uses response-level feedback, future work could incorporate intermediate retrieval feedback or step-wise reasoning supervision to improve lifelong memory evolution.Source: §C.1 · p. 19

C.2 Broader Impact

  • Reliable memory matters for long-horizon applications such as personalized assistance, healthcare consultation, and educational tutoring, where lost history or unintegrated facts can lead to serious consequences.Source: §C.2 · p. 19
  • Fact-centric memory can lose information and produce shallow reasoning over scattered facts by relying on atomic facts across storage, retrieval, and reasoning.Source: §C.2 · p. 19
  • TriMem combines traceable raw-dialogue indices with synthesized profiles to support storage fidelity and understanding-aided reasoning, with experiments verifying effectiveness and compatibility across backbones and benchmarks.The authors describe the architecture as a promising paradigm for long-term agent interactions and future research.Source: §C.2 · p. 19

D Additional Experimental Results … E.4 Prompt for Key Information Analysis

The appendix reports additional experimental results, including the effects of removing entity profiles or raw dialogues across four backbones, and illustrates prompt evolution, retrieval, query analysis, and window-size effects. It also specifies prompts for memory extraction and profile construction, query-key extraction and planning, evidence-grounded question answering, answer evaluation, and iterative refinement.

D.1 Ablation of Profile and Raw Dialogue

  • Removing entity profiles or raw dialogues causes performance declines across four backbones on four LoCoMo task types, supporting both components’ roles in reasoning and storage fidelity.The ablations compare TriMem with variants omitting profiles or raw dialogues using BLEU and F1-score; the supplied passages report declines but do not provide cell values.Source: §D.1 · p. 20, Table 4 · p. 20, Table 5 · p. 20

D.2 Visualization of Prompt Evolution

  • TextGrad evolution traces show prompt refinements addressing temporal normalization, multi-fact dialogue collapse, and preservation of exact profile adjectives.These refinements were prompted by specific failure cases in early rounds.Source: §D.2 · p. 21, Figure 9 · p. 21

D.3 Content of Retrieval Entries

  • TriMem’s visualized retrieval entries show relevant memories marked for use in supporting agent reasoning.The passage describes the retrieved entries as precisely relevant to the questions.Source: §D.3 · p. 22, Figure 10 · p. 22

D.4 Example of Search Query

  • Required-information analysis breaks questions into needed information, enabling more accurate search.The example visualizes both a generated target query and the required key information.Source: §D.4 · p. 23, Figure 11 · p. 23

D.5 Performance of Different Window Size

  • Larger extraction windows can drop low-salience facts or merge atomic facts into bloated entries, motivating the choice of window size 40.The observed failures occur despite an extraction prompt that explicitly forbids merging.Source: §D.5 · p. 24, Figure 12 · p. 24

E.1 Prompt for Memory Extraction

  • The extraction prompt directs the model to omit social gestures, preserve source dialogue IDs, disambiguate people, resolve dates, separate atomic facts, and retain exact details.It also asks the extractor to identify described-but-unnamed things, but that instruction is not detailed in the cited passages.Source: §E.1 · p. 25

E.2 Prompt for Profile Construction

  • The profile prompt updates only affected sections, preserves uncontradicted information, infers traits and preferences, and retains others’ exact descriptions.Its format organizes information into identity, personality, interests, career, values, beliefs, relationships, life events, and preferences.Source: §E.2 · p. 29

E.3 Prompt for Target Query Generation; E.5 Prompt for Question Reasoning

  • Query prompts analyze question type, entities, relationships, and required information before generating a minimal set of targeted, nonredundant searches; the reasoning prompt requires evidence-based inference using context and profiles.The reasoning prompt also instructs the model to identify unnamed things using world knowledge and commit rather than answer that information is unknown.Source: §E.3 · pp. 30–31, §E.5 · p. 33

E.4 Prompt for Key Information Analysis

  • The key-information prompt extracts keywords, person names, time expressions, locations, and entities from a query and requires JSON-only output.Time expressions and locations use null when absent.Source: §E.4 · p. 32

E.5 Prompt for Question Reasoning

  • The reasoning prompt calls for specific evidence, speaker and profile cues, and concise answer formats tailored to question types, including resolving described entities and locations.It distinguishes direct from inferred yes/no answers and directs the model to use profile sections such as others’ descriptions and beliefs when relevant.Source: §E.5 · pp. 33–34

E.6 Prompt for LLM Judge

  • The evaluation criteria require the predicted answer to directly address the specific question asked and capture the “Key Signal” or “Core Entity” described in the criteria.Source: §E.6 · p. 36

E.7 Prompt for Prompts Evolution

  • The TextGrad backward pass diagnoses systematic failures as extraction or profile-synthesis issues, then makes small targeted prompt additions while preserving schemas and leaving prompts unchanged when no eligible pattern appears.The diagnosis uses evaluation-run results and skips unclear cases.Source: §E.7 · pp. 38–39

Related papers

Loading 2605.19952v1…